[Spin-off] Management of Human Science's Data Annotation Work ~ Hurry up and take a detour for quality assurance. The detour will become a shortcut ~

1. What is management in data annotation work?

What do you think about data annotation work? For example, when it comes to annotating cars. Some may think, "It's not that difficult, you just have to draw a bounding box around the car and assign the specified label." It's true that when it comes to the actual work, complex techniques are not always necessary and it can be done without much difficulty.

 

However, various patterns of data annotation arise one after another, such as how to enclose images that are out of focus with squares, or what to do when obstacles hide and cannot enclose the entire image. Each task may be simple, but handling large amounts of data and working with a large number of people requires proper management of information such as defining requirements and work methods, giving work instructions, and responding to questions from workers. If this is not done properly, the consistency of quality cannot be maintained, and there may be a need to redo work or respond to individual questions from workers in the future. In addition, situations may arise where instructions from clients are not properly communicated to workers and reflected in the results. As a result, not only does the quality vary greatly, but the productivity of workers also does not increase, making it difficult to meet deadlines.

 

Proper management is necessary in order to handle these tasks. However, the importance of this management is often overlooked due to the simplicity of data annotation work.

 

In addition, even for customers who request data annotation from an annotation vendor, the contents of this management may seem to be black-boxed. They may often wonder, "Is the management cost, and ultimately the unit price of the annotation with the management cost included, really appropriate?"

 

Therefore, I would like to explain what kind of management we are doing for our data annotation work at our company.

2. Our Data Annotation Process/Project Flow

At our company, we typically proceed with data annotation projects in the following steps.

 

Requirements Definition → Internal Kickoff/Project Review → Sample Delivery → Worker Education/Worker Training → Data Annotation → Delivery → Internal Review → Review with Client

 

You can see that there are various steps involved in data annotation, in addition to the annotation process itself. You may feel like it's a roundabout way for a simple task, but each step serves an important purpose and cannot be avoided to prevent future difficulties. That's why our company incorporates our unique and somewhat gritty know-how and attention to detail in each step.

 

[Requirements Definition]

During the consultation, we will listen to what the customer wants to achieve through AI development. We will work closely with the customer's requests and preferences to determine the detailed specifications for the data annotation process. If there is sample data available, our project manager will use the designated (or prepared by our company) tool to actually perform the work and identify any questions. In most cases, in addition to the specifications provided by the customer, we will also create supplementary materials at our company. Based on these, we will continue to have meetings to prepare for a smooth start to the work.

 

[Internal Kickoff/Project Review]

Gather internal stakeholders to review project plans and management methods. Conduct discussions and information sharing regarding response to potential risks during the planning stage, and conduct a comprehensive review to ensure there are no omissions or oversights in the plan. This helps to facilitate data annotation and smooth project progress. At this stage, it is possible to identify preparation or response omissions for potential risks, and there may also be cases where better work or project progress methods can be proposed, so this is always carried out before starting work.

 

[Sample Delivery (Partial Delivery / First Delivery)]

We will deliver a sample data or ask for your confirmation at the beginning of the project to ensure that the contents discussed with you are accurately reflected in the work and deliverables. We will also check for any errors in the data format, discrepancies or issues in the accuracy and understanding of the annotation work, and make any necessary updates to the work specifications based on your requests. This is a crucial step in ensuring smooth progress of the annotation work according to your requirements.

 

[Worker Education/Worker Training]

Once the requirements definition and sample delivery are successfully completed, it is not enough to just hand over the work specifications and work data to the workers and say "good luck!"
It is necessary to create a "guidance material for workers" that describes the method of exchanging data with the workers and the rules surrounding the work, as well as a procedure manual that breaks down the specifications and enables efficient and stable quality work. Using these, we will explain the contents of the annotation work and provide training for the workers as needed based on the difficulty level.

 

[Data Annotation]

Once the worker's "sufficient understanding of the task" has been confirmed, they will begin the actual data annotation work. In terms of management, in addition to general progress management and daily monitoring and checking of the quality of the data annotator's work, we also make decisions on policies and have discussions with customers regarding questions about edge cases and exceptions that cannot be fully covered even with thorough work specifications prepared in advance, in order to always ensure clear work. We also gather feedback from customers and share it with team members through Q&A sheets, and thoroughly reflect it in the work. As needed, we hold team meetings to share methods and know-how for handling changes, edge cases, and exceptions, making the work diverse.

 

<参考ブログ>

[Spin-off] How to deal with edge cases that cannot be covered in the specification document ~ Overcoming edge cases that cause hesitation in data annotation ~

[Spin-off] Teacher Data: From Creating Good Teachers to the Communication Needed in the Field

 

[Delivery]

In addition to daily quality checks on annotated data, we also perform final confirmation and delivery according to specified delivery format, delivery method, delivery quantity, and virus checks on the data.

 

[Internal Reflection]

We will review the project with internal stakeholders. By sharing the knowledge gained through the project, we aim to eliminate knowledge gaps between Project Managers and ensure consistent management quality regardless of who is in charge.

 

[Customer Feedback]

We strive to improve our service level by receiving feedback from our clients. In cases where our clients are busy and unable to provide feedback, we may ask for responses through surveys. Please refer to our case studies for actual feedback from our clients.

 

<参考事例>

Efficiency, quality improvement, and cost reduction achieved through data annotation outsourcing ―― Improved efficiency of AI development through division of labor, contributing to shortened development period (SCSK Co., Ltd.)

Outsourcing for fast and accurate data annotation Ensuring accuracy and reliability of machine learning systems (Sumitomo Heavy Industries, Ltd.)

 

3. Summary

At our company, we closely communicate with our customers through data annotation work and projects, and strive to solve their requests and issues. Additionally, as mentioned above, project management is a common practice in overall development. On the other hand, data annotation work may seem simple, so one might think that such detailed management is not necessary. However, there are hidden challenges such as "large amounts of data," "large number of people," "long periods of time," "ambiguous tasks," and "appropriate productivity and deadlines," and it is not realistic to simply hand over specifications, tools, and data to workers and say, "Good luck!" In the end, it is important to perform management tasks such as appropriate requirement definition, worker education, and information management, which are necessary for general project management, even in data annotation. It may seem time-consuming and roundabout, but it will ultimately lead to a shortcut in achieving results.

 

As mentioned earlier, our company is committed to carefully managing projects from start to finish in order to maintain quality and productivity. Additionally, you may have noticed that this is how management is carried out behind the scenes of data annotation work. Simply choosing a vendor based on low prices may result in failure of AI development due to inadequate management of the data annotation service to ensure quality.

 

However, on the other hand, our clients have various requests. Some may prioritize cost and delivery time above all else. In such cases, if we were to manage in a meticulous manner, the cost would be passed on and it would not meet the client's expectations. In such situations, we explore various measures to reduce costs, such as simpler management processes, utilizing offshore development, and ensuring quality at a lower cost. Rather than stubbornly sticking to our management principles, we strive to understand and respond flexibly to our clients' requests, and use different management methods depending on the situation. We believe that this is our strength. At the core of this is our strong desire to provide data annotation services that meet our clients' needs, which we feel is deeply ingrained in our culture.