[Spin-off] Interview with our seasoned PM: "What is Annotation Work and Its Essence" - "Tackle the Edge Cases!" A shared sentiment among us PMs -

Until now, our company has been publishing various blogs related to annotation and AI. In those, we have focused on conveying general knowledge and know-how. While the task of annotation may seem simple when expressed in words, it is a task that inevitably involves a lot of human interaction due to its inherent "ambiguity" and the fact that it is a "task that cannot be avoided by humans." Therefore, it often becomes quite messy and cannot be resolved with the neat theories that are commonly found. In order to ensure quality and productivity, a variety of experiences and know-how are actually required.

 

Therefore, understanding the specific problems that occur in the actual annotation field and how to address them can serve as helpful hints for leading to successful annotation.

 

What actually happens in our field and what specific responses and measures we take. Unlike a regular blog, we would like to share the realities of our field, including our unique features and commitments, under the spin-off blog project titled: "Annotations Supporting AI in the DX Era. The Reality of the Analog Field."

1. What is annotation work?

Many articles titled "What is Annotation" provide explanations such as "Teaching data to AI..." Our company also publishes such blog posts. However, we feel that simply doing so may lead to a general explanation and fail to convey the difficulties and challenges encountered in actual annotation work.

 

Related Blog Posts

What is annotation? An explanation from its meaning to its relationship with AI and machine learning.

 

In this blog, we are changing our approach and I asked our PMs to share their thoughts on what annotation really means, as they must be feeling it in their daily annotation work. I posed the question, "What is annotation work? What is its essence, or its core?" to draw out their individual thoughts and ideas.

 

Now, let me introduce you right away.

 

This PM articulated the unique challenges from the perspective of annotation work.

 

What is Data Annotation?

To organize and manage the information that serves as the basis for decision-making, so that the entire team can create a large volume of data that meets the quality demanded by the customer.

 

The intention is

The "troublesome cases" that often arise in annotation work involving labeling large amounts of raw data do not easily fit into the criteria outlined in specifications and definitions.
By maintaining close communication with clients to confirm individual judgments, we can organize information by breaking down those criteria, transforming "troublesome cases" into "cases that can be judged" and sharing them within the team.
In cases where the specifications are complex, and there are multiple points to consider during labeling decisions, or when the labeling work involves data with a lot of ambiguity, it is important to capture the "points of concern" from the worker's perspective, organize the information to facilitate a smoother understanding of the specifications, and communicate it clearly to the workers.
Therefore, in management, to accurately understand the quality required by the client, it is essential to anticipate as many "troublesome cases" as possible in advance, accumulate Q&A with the client to elevate the criteria in the specifications and definitions to a level that can be used in actual work, and ensure effective communication of those criteria to each worker.

 

This PM expressed a perspective that focuses more on annotation work.

 

What is Data Annotation Work?

(Logic + Intuition) to the power of ten times Patience.

 

The intention is

Annotation work essentially involves making binary judgments based on the rules set forth in specifications and definitions, requiring logical decisions backed by solid reasoning.
However, within the data, there are many cases that can be considered both white and black (or neither white nor black), known as edge cases. In such instances, while it is necessary to confirm how to make a judgment each time, it is also important to rely on intuition after understanding a certain set of criteria.
Each of these annotation judgments may seem easy at first glance, but it requires accurate judgment criteria and the ability to work consistently and patiently for long periods at a stable speed, truly demanding perseverance.
Furthermore, it is necessary to continue this work for many days, weeks, or even months, making annotation a particularly challenging task that feels like multiplying the effort required.

 

I think this way.

 

What is Data Annotation

Catch the discomfort during work.

 

The intention is

Although the framework is largely determined in specifications and definitions, it contains many data points that do not fit within that framework, which are referred to as "edge cases." Edge cases can only be recognized when they occur, and some may be so trivial that they are overlooked by individuals. Many edge cases start with a feeling of "Huh? Something seems off here?" This feeling is the essence of the discomfort, and there is a task of constructing logic to resolve it. Through this logic, a reconstruction of understanding is achieved, and it is believed that the quality of annotations improves.

2. How to Handle Edge Cases

Looking at it this way, there is something common among each PM. That is how to handle "edge cases" and "troublesome cases." To carry out annotation tasks, it is necessary to manage quality and productivity, and while there are various factors that hinder these, it can be said that the handling of edge cases is particularly important and key.

 

Neglecting edge cases and proceeding with annotation work based on opportunistic criteria, such as considering something as an annotation target yesterday but excluding it today, will lead to inconsistencies in judgment and fluctuations in quality. If such fluctuating quality training data is used for AI learning, it can result in misrecognition by the AI or failure to improve recognition accuracy.

 

Moreover, overthinking edge cases can be a significant factor that hinders the productivity of annotation data. How to handle edge cases in annotation work is a showcase of the PM's skills, and our PMs each devise their own methods, share information, and apply it to daily projects.

 

For more information on these edge cases, please check out this spin-off blog.

 

Dealing with Edge Cases Not Covered by Specifications: Overcoming Edge Cases That Cause Annotation Confusion

3. Summary

Our company accumulates and shares know-how for annotation, ultimately leading to successful AI machine learning, and establishes methods for it. We utilize these methods regardless of the scale, difficulty, or domain of the project. However, in annotation work, there are still instances where things do not go as expected. We are all working together to brainstorm and tackle how to recover from such situations and aim for even higher goals. If the representatives viewing this resonate with the challenges of annotation work, please feel free to consult with us when considering outsourcing.