[Spin-off] The Surprising Truth Behind What Determines the Productivity of Data Annotation? ~Managing Group Psychology in Data Annotation~

Spin-off blog project
- Annotation that supports AI in the DX era. The unexpected truth behind the productivity of annotation in the analog field.
What is the surprising truth that determines the fate of productivity in annotation?
~Managing group psychology in annotation~

Our company has been publishing various blogs about data annotation and AI. In those blogs, we have mainly shared general knowledge and know-how. Data annotation may seem simple at first glance, as it involves putting the content into words, but it is actually a task that cannot be avoided by humans and contains a lot of "ambiguity". Therefore, there is a lot of interaction between people involved in the process. As a result, it requires a lot of experience and know-how to ensure quality and productivity, which cannot be achieved by just following clean theories.

Therefore, we believe that it is helpful to know the specific problems and responses that occur in the actual annotation field as a hint for successful annotation.
In our field, what actually happens and what specific responses and measures are taken? Unlike regular blogs, in our spin-off blog project titled "Annotation that supports AI in the DX era. The real world of analog annotation", we would like to share the reality of our field, including our unique features and commitments.

>>Past Published Blogs (Some)

7 Tips to Successfully Lead Annotations

What is Teacher Data? Explanation from the relationship with AI, machine learning, and annotation to how to create it.

"How to ensure and improve the quality of teacher data? Explaining practical methods!"

Table of Contents

1. Factors Affecting Productivity
2. Time Required for Judgment
3. Impact of Group Psychology on Productivity
4. Despite all the struggles so far, a significant performance improvement has been achieved with ease!?
5. Summary

1. Factors Affecting Productivity

This time, we will deliver a theme about data annotation productivity.

When it comes to productivity, there are various definitions depending on the perspective of management analysis, but this time, I would like to talk about productivity in terms of pure work efficiency and work speed.
For customers who have already outsourced data annotation, they may not pay much attention to how the outsourcing vendor is improving productivity after ordering the work. However, the vendor's productivity greatly affects the cost and expenses when considering outsourcing, and it is especially important to focus on productivity when having the vendor's personnel stationed at the company.

What are the factors that affect productivity? There are five main ones to consider.

1. Work Tools

As can be seen from the fact that many data annotation vendors are promoting productivity improvement with tools, using tools that take into account automation and efficiency of annotation and checking can contribute to productivity improvement. If the tool also includes automation of annotation and checking, utilizing these functions can further contribute to productivity improvement.

2. Operation Manual

Work manuals also have a significant impact on productivity.
If the work manual is not easy for data annotators to understand, it can lead to rework due to mistakes and waste time trying to understand specifications and requirements. In addition, it is important not only to use tools, but also to set up work procedures that flow smoothly and to provide instructions to workers by describing the flow of those procedures in the manual. If this appropriate information is not conveyed, it can lead to inefficient operations and unnecessary screen transitions, resulting in wasted time. Even if a high-performance tool is used, productivity will not improve in this situation.

3. Distribution of Additional Information

In addition, the presence of additional information such as edge cases that increase daily through interactions with workers and customers can also be a factor in reducing productivity. This is a common occurrence when conducting inquiries through chat tools, where past information may be buried, or when information is concluded in personal chats and not disseminated to the entire team, causing it to be dispersed and lost. This can result in wasted time searching for answers to questions such as, "Where was the answer to that question from the other day?" This is an important point to consider when creating manuals, as the problem of not being able to find desired information quickly can also arise.

4. Data Annotation Materials

If the image quality of the materials varies or if there are mixed images from different sources, it can also be a factor in reducing productivity. However, it is necessary to prepare data with such variations for AI learning, so it is a challenging factor.

5. Worker Proficiency

In the initial stages of work, it is common to struggle with using new tools and frequently refer to the work manual, which takes up a lot of time for asking questions. As a result, productivity naturally does not increase. However, if the level of proficiency does not improve, productivity will not improve either, so it is important to accelerate the proficiency and understanding of data annotators.

Now, there is a major factor that is not often focused on besides these five factors. That is the aspect that depends on human psychology. In particular, the mental state of the data annotator and the hesitation in their annotation judgments can greatly affect productivity. In terms of psychology, the group psychology of the organization or team often has a significant impact on productivity. Therefore, for the five factors mentioned earlier - work tools, work manuals, dispersed information, data materials, and worker proficiency - we will leave it to other vendors' blogs and websites for explanation. Instead, we will discuss our experiences with the hesitation of data annotators and the impact of group psychology in the annotation team on productivity.

2. Time Required for Judgment

What takes up the most time during data annotation work? What comes to mind when you are asked this question? Many people may think that the majority of data annotation work is spent on actual hands-on tasks, such as drawing rectangles around images or tracing object outlines for semantic segmentation. Of course, this is true for simple and straightforward annotations (such as drawing a rectangle around an apple in a simple background for image data), but when the annotation task itself is complex or involves high ambiguity, such as language annotation, a significant amount of time may be spent on deciding whether or not to annotate the target data and how to annotate it.

For example, let's consider the annotation to determine the quality of the output text from ChatGPT. Let's say that it is specified to evaluate the degree of goodness (badness) on a 5-point scale. The quality of the text depends on the annotator's subjective opinion and can also be influenced by how they evaluated the text they read before. "Wait, I remember ranking it as B earlier, but now I feel like it fits more into C... What should I do, should I go back and make changes? Wait, at the meeting the other day, it was said that in case of uncertainty, we should annotate according to XX's policy. Maybe this is actually a C..." This kind of hesitation and overthinking is a common occurrence in annotation. When there are many uncertainties, productivity decreases dramatically. I won't go into detail here, but to solve these issues, it can be greatly improved by having productive workers or PMs explain and demonstrate their thought process and decision-making methods.

3. Impact of Group Psychology on Productivity

Next is psychology. What is psychology? You may wonder how such intangible things can affect productivity. This is also a common occurrence in the field of data annotation. Not limited to data annotation, psychological state greatly affects productivity. In particular, group psychology has a significant impact on the overall productivity performance of a team. When the consensus of "no matter how hard we try, we can only make this much progress in an hour. Everyone is the same, right?" spreads within a team, it can be difficult to recover. This is not a laughing matter, and it becomes even more serious when the team's mood maker emits such a psychological state. To prevent such an atmosphere from spreading, it is necessary for the project manager to proactively set appropriate productivity and work pace, and manage in a way that does not decrease productivity.

4. Despite all the struggles so far, a significant performance improvement has been achieved with ease!?

Here, we will introduce a typical example of how group psychology affects productivity.

To respond to a certain client's project, we had to launch two teams in different locations. Since we had a contract with the client based on the hourly rate of each worker, a productivity target was set for each worker. The first team that was launched struggled to meet the productivity target from the beginning of the project. The team's mentality at that time was, "Is it realistically impossible to meet the required productivity while ensuring quality? Is the target too high?" That was the atmosphere. In order to reach the productivity target, the project manager tried various methods and finally managed to meet the target after three months.

In the next team that was formed, we informed them beforehand that "we have already achieved the target productivity in other teams, so it is possible to reach it". As a result, in less than two weeks after starting work, we were able to easily surpass the target productivity while maintaining quality. The team's mindset at this time was, "since it has already been achieved in other teams, we should be able to do it once we get used to it." That was the atmosphere.

There was a similar situation in another project. We were working simultaneously at different locations. Even though we were using the same tools and manuals, and the same person gave the initial instructions, I remember there being a productivity gap of almost 40%. At that time, when we actually showed and explained the work of the team with higher productivity, there was an atmosphere of not accepting the reality, saying "I can't believe they can work that fast. Maybe the person explaining is just naturally fast." This was a project where I deeply realized that the group psychology of the team greatly affects productivity.

5. Summary

As mentioned earlier, there are still many aspects of data annotation that rely on manual labor. In that sense, I also feel that it is quite important for the PM to control this group psychology.

In addition, the way group psychology is formed varies depending on the project and team, so it is necessary to make efforts to advance data annotation each time. By experiencing situations where productivity increases with just one trigger, or takes several months, we have accumulated wisdom and knowledge through success and failure, and have continued to apply them in our company. In order to increase productivity, we sometimes face detours and work every day to find a better path. And we would like to deliver good training data for AI that supports this DX era to our customers.

Author:

Kitada Manabu

Annotation Group Project Manager

Since the establishment of our Data Annotation Group, we have been responsible for a wide range of tasks, from team building and project management for large-scale projects, to creating annotation specifications for PoC projects, and consulting for scalability, with a focus on natural language processing.
Currently, in addition to being a project manager for image and video annotation projects, we also work as a seminar instructor for data annotation and engage in promotional activities such as blogging.