[Spin-off] Management of Annotation Work in Human Science - To Ensure Quality, Take the Long Way Around. The Detour Becomes the Shortcut.

1. What is management in annotation work?

What do you all think about the annotation work? For example, regarding car annotation. Some may think, "It's just about enclosing the car in a bounding box and assigning the specified label, so it shouldn't be that difficult, right?" Indeed, it is true that in terms of the physical work involved, it often can be done without complex techniques.

 

However, various patterns arise in annotations, such as how to enclose an out-of-focus image in a square, or what to do when it is obscured by obstacles and cannot be fully enclosed. Each task may be simple, but it often involves handling large amounts of data and working with many people. If information management regarding requirements, work methods, work instructions, and Q&A from workers is not properly conducted, it can lead to inconsistencies in quality, making it difficult to redo later tasks or respond to individual inquiries from workers. Additionally, situations may arise where the instructions received from customers are not properly communicated to the workers or reflected in the deliverables. As a result, not only does the variation in quality increase, but the productivity of the workers does not improve, making it difficult to meet deadlines.

 

It is only with appropriate management that we can respond to these. However, I feel that the importance of this management is often overshadowed by the simplicity of the annotation work and tends to be underestimated.

 

Additionally, for customers who request services from annotation vendors, the contents of this management may appear to be a black box, leading them to wonder, "Are the management costs, and consequently the unit price of the annotations that include these management costs, truly appropriate?"

 

Therefore, this time I would like to explain what kind of management we conduct in our company's annotation work.

2. Our Annotation Work/Project Flow

Our company typically proceeds with annotation projects in the following steps.

 

Requirements Definition → Internal Kickoff / Project Review → Sample Delivery → Worker Education / Worker Training → Annotation Work → Delivery → Internal Review → Reflection with the Client

 

As you can see, there are various steps involved beyond just the annotation work itself. You might feel that it seems like a long detour for a simple task, but each step has an important purpose, and we cannot avoid them as they can lead to significant issues later on. Therefore, our company incorporates our unique, somewhat down-to-earth know-how and attention to detail in each step.

 

[Requirements Definition]

We will discuss what you want to achieve through AI development in our meeting. We will work closely with your requests and wishes to determine the detailed specifications for the annotation work that will realize those goals. If you have sample data, the project manager will actually perform the work using the specified (or company-provided) tools to identify any questions. Additionally, in most cases, we will create supplementary materials in addition to the specifications provided by you. Based on these, we will have further meetings to ensure a smooth start to the work.

 

[Internal Kickoff/Project Review]

We gather internal stakeholders to review the project plan, management methods, and more. We conduct discussions and information sharing regarding potential risks identified during the planning stage, and we perform a multifaceted review to ensure there are no omissions or gaps in the plan. This helps facilitate the annotation work and the project more smoothly. It is also possible to identify preparation or risk response omissions at this stage, and we may be able to propose better ways to carry out the work or project, so this is always conducted before starting the work.

 

Sample Delivery (Partial Delivery / Initial Delivery)

We will deliver sample data to the customer in advance or at the beginning of the work to confirm whether the content discussed with the customer is accurately reflected in the work and deliverables. We will check for any deficiencies in the data format, ensure there are no discrepancies or issues with the accuracy of the annotation work and understanding of specifications, and if there are any customer requests, we will incorporate them into the work specifications. This is a very important step to smoothly proceed with the annotation work that meets the customer's requirements.

 

[Operator Education/Operator Training]

Once the requirements definition and sample delivery are successfully completed, it is not enough to just hand over the work specifications and data to the workers with a simple "Good luck!".
We need to create guidance materials for the workers that outline the methods of data exchange and the rules related to the work, as well as detailed procedure manuals that break down the specifications to ensure efficient and stable quality in the work. Using these materials, we will explain the content of the annotation work and, depending on the difficulty and needs, provide training for the workers.

 

[Data Annotation Work]

Once the operator's sufficient understanding of the work has been confirmed, we will begin the actual annotation work. From a management perspective, in addition to general progress management and daily monitoring and checking of the quality of the annotators' deliverables, we will also establish policies and engage in Q&A with clients regarding questions about edge cases and exceptions that arise daily, which cannot be fully covered even with meticulously prepared work specifications in advance, ensuring that work can always be carried out clearly. Additionally, we will compile feedback from clients and share it with team members through Q&A sheets, thoroughly reflecting it in the work. As needed, we will hold team meetings to share methods and know-how for addressing changes, edge cases, and exceptions, making the work diverse and comprehensive.

 

<参考ブログ>

Dealing with Edge Cases Not Covered by Specifications: Overcoming Edge Cases That Cause Annotation Confusion

[Spin-off] Training data starts with good teacher development - What communication is needed in the field?

 

Delivery

In addition to daily quality checks on annotated data, we conduct final confirmations and deliver the data, including specified delivery formats, delivery methods, delivery quantities, and virus checks.

 

[Internal Review]

We will conduct a project review with internal stakeholders. By sharing whether the project progressed as planned, what reflections were made, and the know-how gained through the project, we aim to eliminate knowledge disparities among PMs and ensure that the same management quality can be maintained regardless of which PM is handling it.

 

Customer Reflection

We are working to improve our service levels by receiving feedback from our clients regarding their projects. If our clients are busy and unable to spare time, we may ask for their responses through surveys. Please refer to these case studies for actual feedback from our clients.

 

<参考事例>

Achieving greater efficiency, quality improvement, and cost reduction through data annotation outsourcing ―― Improving efficiency of AI development through division of labor, for a shorter development period (SCSK Corporation)

Ensuring the accuracy and reliability of machine learning systems by outsourcing fast and accurate annotation work (Sumitomo Heavy Industries)

 

3. Summary

In this way, our company strives to maintain close communication with our clients through annotation work and projects, aiming to address their requests and challenges. Additionally, the project management mentioned above is something that is generally expected in overall development. On the other hand, since annotation work is relatively simple, it may seem that such elaborate management is unnecessary. However, there are hidden challenges such as "large data sets," "many people," "long durations," "ambiguous tasks," and "appropriate productivity and deadlines," making it unrealistic to simply hand over specifications, tools, and data to workers and say, "Good luck!" Ultimately, conducting proper requirement definitions, worker training, and information management—essential aspects of general project management—is also crucial in annotation. It may seem labor-intensive and like a detour, but it ultimately becomes the shortcut to success.

 

As mentioned so far, our company is committed to meticulous management from the beginning to the end of a project to properly maintain quality and productivity. Additionally, I hope you have understood how management is carried out behind the scenes of the annotation work. If you choose vendors solely based on low prices, you may end up with annotation services that do not provide proper management to ensure quality, which could lead to failures in AI development.

 

However, on the other hand, our customers have various requests. Some orders prioritize "cost and delivery time above all else!" In such cases, if we conduct meticulous management, it will be reflected in the price and will not meet the customer's needs. In such situations, we explore various strategies for cost reduction, such as simpler management processes, utilizing offshore resources, and management methods that can ensure the required quality at a lower cost. We believe that our strength lies in not being rigid about our management principles, but rather in being flexible and responsive to our customers' requests while discerning their needs, and adapting our management methods according to the situation. At the core of this is our cultural commitment to providing annotation services that align with our customers' requests above all else.