[Spin-off] What Influences the Productivity of Annotations: What is Its Surprising True Identity? ~Management of Group Psychology in Annotations~

Spinoff Blog Project
――Annotations that support AI in the DX era. The reality of that analog field
What is the surprising identity of the thing that influences the productivity of annotations?
~Management of group psychology in annotations~

Until now, our company has been publishing various blogs related to annotation and AI. In those, we have focused on conveying general knowledge and know-how. While the task of annotation may seem simple when expressed in words, it is a task that inevitably involves a lot of human interaction due to its inherent "ambiguity" and the fact that it is a "task that cannot be avoided by humans." Therefore, it often becomes quite messy and cannot be resolved with the neat theories that are commonly found. In order to ensure quality and productivity, a variety of experiences and know-how are actually required.

Therefore, we believe that understanding the actual problems that occur in the field of annotation and how to address them can serve as valuable hints for leading to successful annotation.
In our company, we want to share what actually happens on-site and the specific responses and measures we take. Unlike a regular blog, we aim to convey the realities of the field with our spin-off blog project titled: "Annotation Supporting AI in the DX Era: The Reality of the Analog Field," including our unique characteristics and commitments.

>>Past Published Blogs (Partial)

7 Tips for Successfully Leading Annotations

What is training data? An explanation from its relationship with AI, machine learning, and annotation to how to create it.

How to ensure and improve the quality of training data? Practical methods explained!

Table of Contents

1. Factors Affecting Productivity
2. Time Required for Judgment
3. The Impact of Group Psychology on Productivity
4. After all this struggle, a performance improvement so easily!?
5. Summary

1. Factors Affecting Productivity

This time, we will deliver a theme regarding the productivity of annotations.

When it comes to productivity, there are various definitions depending on the perspective, such as management analysis. However, this time I would like to discuss productivity in terms of pure work efficiency and work speed.
For customers who have already outsourced annotation, they may not particularly care about how the outsourcing vendor is improving productivity after placing the order. Nevertheless, the vendor's productivity is closely related to the costs and expenses considered during the outsourcing evaluation, and it is especially important to focus on productivity when having vendor personnel stationed at your company.

What factors affect productivity? There are five main ones to consider.

1. Work Tools

As can be seen from the fact that many annotation vendors are promoting productivity improvements with their tools, using tools that consider automation of annotation and efficiency of checks contributes to increased productivity. If the tools incorporate automation of annotation and checks, utilizing these features may further contribute to productivity improvements.

2. Work Manual

Work manuals also have a significant impact on productivity.
If the work manual is not easy to understand for annotators, it can lead to rejections due to mistakes and time spent understanding specifications and requirements, which hinders productivity. Additionally, it is important not only to use tools but also to establish work procedures that allow actions and tasks to flow smoothly, and to document these workflows in the manual to instruct workers. If such appropriate information is not conveyed, it can lead to inefficient actions and unnecessary screen transitions, resulting in wasted time. Even with high-functioning tools, productivity will not improve under these circumstances.

3. Distribution of Additional Information

Additionally, the existence of additional information such as edge cases that increase daily through Q&A with workers and clients can also be a factor that lowers productivity. This is often the case when conducting Q&A through chat tools, where past information may become buried, or discussions may conclude in personal chats, preventing the entire team from accessing the information. When information becomes dispersed or lost, it leads to situations like, "Where was the answer to that question from the other day?" which takes time to search for. The issue of not being able to find the information you want quickly is also an important point to consider when creating manuals.

4. Annotation Data Materials

Variations in the quality of materials or the mixing of images from different sources can lead to decreased productivity. However, it is also necessary to prepare data with such variations for AI training, making it a challenging factor.

5. Operator Proficiency

In the initial stages of work, it is common to struggle with operating tools for the first time and to frequently refer to the work manual, which takes up a lot of time for asking questions, so productivity naturally does not improve. However, if proficiency does not increase over time, productivity will not improve, so it is important to accelerate the annotators' proficiency and understanding.

Now, there is a significant factor that is not often highlighted aside from these five factors. It depends on human psychology. In particular, the psychological state of the annotators and their hesitation in making annotation judgments affect productivity. In terms of psychology, the collective mindset of the organization or team often has a significant impact on productivity. Therefore, while we will leave the explanations of the five factors mentioned earlier—work tools, work manuals, information distribution, data materials, and worker proficiency—to the blogs and websites of other vendors, we will share our experiences regarding how the hesitation of annotators in making judgments and the collective psychology of the annotation team affect productivity.

2. Time Required for Judgment

What comes to mind when you are asked what takes up a large amount of time in annotation work? Many may think that the actual time spent working, such as the time spent drawing rectangles around images or tracing the contours of objects in semantic segmentation, occupies the majority of annotation work. Of course, this is true for simple and straightforward annotations (for example, enclosing an apple in a rectangle against a simple background in image annotation), but when the annotation work itself is complex or when there is a high degree of ambiguity, such as in language annotation, the proportion of time spent on deciding whether to annotate the target or how to annotate it becomes particularly significant.

For example, let's consider an annotation to determine the quality of text output by ChatGPT. Suppose we are instructed to evaluate the quality (or lack thereof) on a scale of 1 to 5. The quality of the text is subjective to the annotator and may also be influenced by how they rated previously read texts. "Hmm, I feel like I rated this as a B earlier, but somehow C seems more fitting... What should I do, should I go back and revise it...? Wait, in the last meeting, we decided that if in doubt, we should annotate according to the XX policy for edge cases. I guess this is indeed a C..." Such moments of indecision can lead to a halt in work, which is a common occurrence in annotation tasks. When there are frequent uncertainties in judgment, productivity can suddenly drop. While I won't elaborate on this here, one way to resolve these issues is to dramatically improve the situation by teaching and demonstrating how highly productive workers and PMs think and make decisions.

3. The Impact of Group Psychology on Productivity

Next is psychology. What is psychology? Many may wonder how something as elusive as this can affect productivity, but this is quite common in the field of annotation. While this is not limited to annotation, psychological states greatly influence productivity. In particular, group psychology has a significant impact on the overall productivity performance of the team. When a sentiment like, "No matter how hard we try, we can only make this much progress in an hour. Everyone feels the same way," spreads within the team, it can be quite detrimental. It's not just a laughing matter; if this psychological state is coming from the team's mood maker, it becomes even more critical. To prevent such an atmosphere from spreading, project managers need to proactively set appropriate productivity levels and work paces to ensure that productivity does not decline.

4. After all this struggle, a performance improvement so easily!?

Here, we present a typical example of how group psychology affects productivity.

To respond to a certain client's project, we decided to establish two teams at remote locations. Since the contract with the client was based on the hourly wage of each worker, productivity targets were set for each worker. The first team that was established struggled with productivity right from the start of the project. The team's psychological state at that time was, "Is it realistically impossible to meet the required productivity while ensuring quality? Isn't that an incredibly high target?" There was a palpable sense of this atmosphere. To reach the productivity target, the PM tried various approaches, and after three months, we finally managed to meet the target.

In another team that was established next, we communicated in advance that "other teams have already cleared the target productivity from the start" to instill the awareness that the target productivity is achievable. As a result, within less than two weeks after starting the work, they easily cleared the target productivity while ensuring quality. The psychological state of the team at that time was, "Since it has been achieved by the already established team, we will also be able to do it once we get used to it." There was such an atmosphere.

In another project, we experienced something similar. We were also working in parallel at different locations. Despite using the same tools and work manuals, and having the same person conduct the initial work lecture, I remember that at times there was nearly a 40% gap in productivity. Even when we showed the work of the more productive team and explained it, there was an atmosphere of disbelief, with comments like, "I can't believe they can work that fast. The person explaining must be exceptionally fast." I vividly remember this reality. It was a case that deeply made me realize how much the collective psychology of the entire team can significantly impact productivity.

5. Summary

As mentioned at the beginning, there are still many aspects of annotation that inevitably rely on human effort. In that sense, I feel it is quite important for the PM to control such group psychology.

Additionally, the way group psychology is formed varies by project and team, so it is necessary to devise strategies to advance annotations each time. Through experiences where productivity can either increase with just one trigger or take several months, we have accumulated wisdom and knowledge by learning from both successes and failures. To enhance productivity, we sometimes face these detours and strive daily to find better paths. We also aim to deliver high-quality training data for AI that supports this era of digital transformation to our clients.

Author:

Manabu Kitada

Annotation Group Project Manager

Since the establishment of our Annotation Group, we have been broadly responsible for team building and project management for large-scale projects centered on natural language processing, as well as the formulation of annotation specifications for PoC projects and consulting aimed at scaling.
Currently, in addition to being a project manager for image and video annotation and natural language annotation, I am also engaged in promotional activities such as being a seminar instructor for annotation and writing blogs.