[Spin-off] What Influences the Productivity of Annotations: What is Its Surprising True Identity? ~Management of Group Psychology in Annotations~

1. Factors Affecting Productivity

This time, we will deliver a theme regarding the productivity of annotations.

When it comes to productivity, there are various definitions depending on the perspective, such as management analysis. However, this time, we will proceed with the discussion from the viewpoint of productivity as pure work efficiency and work speed.
For customers who have already outsourced annotation, they may not particularly pay attention to how the outsourcing vendor improves productivity after placing the order. That said, the vendor's productivity is greatly related to the costs and expenses considered when outsourcing, and it is especially important to focus on productivity when personnel from the vendor are stationed in-house.

 

What factors affect productivity? There are five main ones to consider.

 

1. Work Tools

As can be seen from the fact that many annotation vendors are promoting productivity improvements with their tools, using tools that consider automation of annotation and efficiency of checks contributes to increased productivity. If the tools incorporate automation of annotation and checks, utilizing these features may further contribute to productivity improvements.

 

2. Work Manual

The work manual also has a significant impact on productivity.
If the manual is not easy for annotators to understand, productivity will not improve due to rejections caused by mistakes and the time taken to understand specifications and requirements. Moreover, it is important not only to use the tools but also to establish work procedures that allow operations and tasks to proceed smoothly according to the workflow, and to include these workflows in the manual to instruct the workers. Without conveying such appropriate information, inefficient operations and unnecessary screen transitions will cause time loss. In such cases, even using high-functionality tools will not increase productivity.

 

3. Distribution of Additional Information

Additionally, the presence of additional information such as edge cases that increase daily through Q&A with workers and customers also becomes a factor that lowers productivity. This is common when conducting Q&A via chat tools, but if past information gets buried, or if conversations are completed in private chats without reaching the entire team, causing information to become scattered or lost, it takes time to search for answers like "Where was the response to that question the other day?" The problem of not being able to quickly find the information you want to look for is an important point to consider when creating manuals as well.

 

4. Annotation Data Materials

Variations in the quality of materials or the mixing of images from different sources can lead to decreased productivity. However, it is also necessary to prepare data with such variations for AI training, making it a challenging factor.

 

5. Operator Proficiency

In the initial stages of work, it is common to struggle with operating tools for the first time and to frequently refer to the work manual, which takes up a lot of time for asking questions, so productivity naturally does not improve. However, if proficiency does not increase over time, productivity will not improve, so it is important to accelerate the annotators' proficiency and understanding.

 

Now, besides these five factors, there is a major factor that has not been given much attention. That is the aspect dependent on human psychology. In particular, the annotator’s psychological state and hesitation in annotation decisions affect productivity. Psychologically, the group psychology of the organization or team often has a significant impact on productivity. Therefore, while we will leave explanations of the previously mentioned five factors—work tools, work manuals, information dispersion, data materials, and operator proficiency—to other vendors’ blogs and websites, we will share our experiences regarding how annotators’ hesitation in decision-making and the group psychology of annotation teams influence productivity.

2. Time Required for Judgment

What do you think takes up the most time in annotation work? Many might imagine the actual time spent moving their hands, such as drawing bounding boxes around images or tracing object contours in semantic segmentation, as occupying the majority of annotation work. Of course, this is true for straightforward annotations—for example, drawing a bounding box around an apple against a simple background in image tasks. However, when the annotation work itself is complex or involves high ambiguity, such as in language annotation, a significant portion of time is spent deciding whether to annotate the target or how to annotate it.

 

For example, let's consider annotation that evaluates the quality of sentences generated by ChatGPT. Suppose you are asked to rate the degree of quality (or lack thereof) on a 5-point scale. The judgment of the sentence's quality depends on the annotator's subjective view and may also be influenced by how they evaluated previous sentences. Thoughts like, "Hmm, I think I rated that one as Rank B earlier, but now C seems more appropriate... What should I do? Should I go back and revise it...? Wait, in the last meeting, we decided that if we're unsure and it's an edge case, we should annotate according to policy XX. So, this must be a C after all..." often cause annotators to overthink and pause their work—this is a common occurrence in annotation. When such indecision increases, productivity suddenly drops. Although we won't go into detail here, one effective way to resolve this is for highly productive workers or project managers to lecture on their thought processes and decision-making approaches and demonstrate them in practice, which can lead to dramatic improvements.

3. The Impact of Group Psychology on Productivity

Next is psychology. What is psychology? Many may wonder how something as elusive as this can affect productivity, but this is quite common in the field of annotation. While this is not limited to annotation, psychological states greatly influence productivity. In particular, group psychology has a significant impact on the overall productivity performance of the team. When a sentiment like, "No matter how hard we try, we can only make this much progress in an hour. Everyone feels the same way," spreads within the team, it can be quite detrimental. It's not just a laughing matter; if this psychological state is coming from the team's mood maker, it becomes even more critical. To prevent such an atmosphere from spreading, project managers need to proactively set appropriate productivity levels and work paces to ensure that productivity does not decline.

4. After all this struggle, a performance improvement so easily!?

Here, we present a typical example of how group psychology affects productivity.

To respond to a certain client's project, we decided to establish two teams at remote locations. Since the contract with the client was based on the hourly wage of each worker, productivity targets were set for each worker. The first team that was established struggled with productivity right from the start of the project. The team's psychological state at that time was, "Is it realistically impossible to meet the required productivity while ensuring quality? Isn't that an incredibly high target?" There was a palpable sense of this atmosphere. To reach the productivity target, the PM tried various approaches, and after three months, we finally managed to meet the target.

 

In another team that was established next, we communicated in advance that "other teams have already cleared the target productivity from the start" to instill the awareness that the target productivity is achievable. As a result, within less than two weeks after starting the work, they easily cleared the target productivity while ensuring quality. The psychological state of the team at that time was, "Since it has been achieved by the already established team, we will also be able to do it once we get used to it." There was such an atmosphere.

 

A similar situation occurred in another project as well. In that case, work was also carried out concurrently at separate locations. Despite using the same tools, work manuals, and having the same person conduct the initial work briefing, I recall that at worst, there was nearly a 40% gap in productivity. At that time, even when we actually showed and explained the work of the higher-productivity team, there was a clear atmosphere of disbelief, with comments like, "I can't believe they can work that fast. The person explaining must be especially fast." This was a vivid memory. Both cases deeply impressed upon us how the collective psychology of the entire team greatly influences productivity.

5. Summary

As mentioned at the beginning, there are still many aspects of annotation that inevitably rely on human effort. In that sense, I feel it is quite important for the PM to control such group psychology.

 

Additionally, the way group psychology is formed varies by project and team, so it is necessary to devise strategies to advance annotations each time. Through experiences where productivity can either increase with just one trigger or take several months, we have accumulated wisdom and knowledge by learning from both successes and failures. To enhance productivity, we sometimes face these detours and strive daily to find better paths. We also aim to deliver high-quality training data for AI that supports this era of digital transformation to our clients.