Tips for In-House Creation of Training Data: From the Perspective of "4M+I" (Man, Machine, Method, Material, Information)

1. When AI development engineers perform annotation themselves

1-1. Man (Human Resources)

[Issue] Variations in labels due to engineers' subjective judgments

When engineers perform annotations themselves, there is a possibility of subjective judgments such as "this level of damage is fine with this label," or biases influenced by model development considerations like "let's not label this data because the AI model might misclassify it." Furthermore, prolonged work can blur the criteria, increasing the risk of making different judgments on multiple data points that should originally be judged the same.

[Measures & Tips] Preparation of Standardized Specifications

First, document the annotation standards and rules with concrete examples, so that if someone else needs to perform the work later, anyone can make the same decisions. Prepare a specification manual accordingly. The manual is not just created and left as is; accumulate "exception cases" and "common cases where judgment is difficult," and ensure work can be done according to the latest decision criteria.

[Issue] Decreased attention and motivation due to long working hours

Annotation is a relatively monotonous and time-consuming task. When engineers perform it alongside their development duties, there is a risk of decreased attention and motivation, leading to lower quality. Furthermore, processing large amounts of data can cause fatigue to accumulate, reducing concentration and making it more difficult to accurately label according to standards.

[Measures & Tips] Optimizing Work Schedules

By setting an estimated time per data item and using the Pomodoro Technique to take regular breaks, you can reduce fatigue during work. Establishing rules for work and breaks in advance helps maintain focus and motivation, which ultimately leads to improved overall performance.

1-2. Machine (Equipment & Tools)

[Issue] The annotation tool is not intuitive to operate

Some annotation tools have few shortcuts and can be somewhat cumbersome to use until you get accustomed to them. Since repetitive operations mainly rely on the mouse, efficiency tends to be low, and the difficulty in handling the tool until you become familiar with it may affect quality and work pace.

[Measures & Tips] Introducing More Efficient Tools

Let's consider introducing annotation tools that excel in operability and efficiency. For example, "Annofab" features abundant shortcut functions and an intuitive interface, significantly improving work efficiency. Its ease of use reduces worker stress, which in turn leads to more stable quality.

Reference: What is the annotation tool Annofab?

Official Site: Annofab

[Issue] Difficult to manage work history

Many annotation tools do not record information such as "who," "when," and "which data was worked on," making it difficult to identify the data that needs correction during specification changes. This can lead to missed corrections and rework, potentially affecting progress and quality management.

[Measures & Tips] Utilizing Tools with History Management Features

By utilizing annotation tools such as "Annofab," which allow you to grasp the person in charge of the task, the execution date and time, and the current progress status, you can see at a glance who is working on which data and in which phase. When there are specification changes, you can quickly identify the scope that requires attention, preventing rework and making quality control easier.

1-3. Method (Techniques / Workflow)

[Issue] Working with ambiguous annotation standards

If you start working without clear annotation specifications or make frequent changes along the way, there is a risk that the data will become inconsistent even if the engineers themselves handle it. Such fluctuations in specifications can cause quality variations and increase correction work, ultimately leading to a decrease in the overall efficiency of the project.

[Measures & Tips] Organizing Specifications and Using Checklists

As mentioned above, in addition to thoroughly documenting the annotation standards as a specification, utilizing checklists helps prevent oversights and inconsistencies in judgment. In particular, by preparing checklists that organize points prone to confusion and cases where decisions tend to vary, you can proceed with work efficiently while maintaining stable quality.

[Issue] Risk of QA checks not being performed

When working on annotation tasks alone, quality checks by a third party are not conducted, making it difficult to notice judgment errors or work mistakes. Furthermore, as concentration decreases during long hours of work, misrecognition and inconsistent judgments are more likely to occur, which may then be reflected in the data.

[Measures & Tips] Use a checklist to maintain an objective third-party perspective

Summarize annotation rules and decision criteria in a checklist format, and regularly review them before and during work to prevent inconsistencies in judgment. By creating a system to maintain objectivity, you can reduce decision-making errors.

1-4. Information (Information Management)

[Issue] Difficult to notice efficient methods

When working alone, it is easy to fall into the mindset that "my way is the best," and as a result, you may fail to notice more accurate and efficient methods (so-called best practices). This can lead to unnecessary work and inconsistencies in quality.

[Measures & Tips] Sharing Knowledge and Regular Team Reviews

By accumulating knowledge such as best practices and edge cases in tools and shared folders used within the company, and regularly setting up opportunities to review them with team members including other engineers, you can gain insights into improvements and efficiency tips that you might not have noticed on your own. Through such dialogues, it also leads to refining best practices and preventing knowledge silos.

[Issue] Work becomes person-dependent, making handovers difficult

Only certain individuals understand how to set up the tools and environment necessary for annotation, making it impossible for other members to replicate the same work. In such situations, handovers and changes in responsibility do not proceed smoothly, which may lead to work stagnation.

[Measures & Tips] Visualize information with a focus on being "ready to hand over immediately"

To ensure that anyone can reproduce the same environment, briefly summarize the tool installation procedures and the versions being used. There is no need to aim for a perfect manual. The key point is to "release information little by little." Simply keeping information visible significantly reduces the risk of knowledge being confined to individuals.

2. When engineers utilize multiple personnel within the company to perform annotation

2-1. Manpower

[Issue] There is a variation in the skills and experience of the workers

When multiple members perform annotations, it becomes difficult to maintain consistent quality due to differences in skills and experience. In workplaces with high staff turnover, less experienced individuals may carry out tasks, increasing the risk of variability in accuracy.

[Measures & Tips] Training and Continuous Feedback

Before starting the main work, it is important to provide the same training to everyone to ensure a basic understanding of the specifications. After that, regularly checking the work content and sharing issues and countermeasures regarding quality and work pace with the workers themselves helps to improve quality.

[Issue] Inefficiency caused by the absence of dedicated personnel

If you cannot assign a dedicated person for annotation and quality checks, you will need to explain the tasks and provide training each time, which incurs time and costs. Furthermore, if the workers change with each project, it becomes difficult to maintain consistent quality and speed of work.

[Measures & Tips] Conducting Information Sharing Meetings

Let's regularly gather workers and hold information sharing meetings (e.g., daily at the start of a project, then once a week thereafter). By sharing insights gained during work (common judgment errors, efficient work methods, etc.) with everyone in such meetings, we can suppress variations in quality and processing speed even when team members change.

2-2. Machine (Equipment & Tools)

[Issue] Variations in how the tool is used

When multiple workers use the same annotation tool, there tends to be a difference in work speed and the number of errors between those who are familiar with the operation and those who are not. If each person works in their own way, there is a risk of significant variation in productivity and quality.

[Measures & Tips] Creating Operation Manuals and Conducting Workshops

We provide a manual summarizing the basic operations and shortcuts of the tools, ensuring that everyone can work using the same procedures. Additionally, by holding lectures and workshops for those who are uncertain about the operations, we can raise the overall proficiency level.

[Issue] Difficulties in collaborative work within the team

Some annotation tools lack user management and data management features. This can make data exchange and management among workers cumbersome, increasing the burden on administrators.

[Measures & Tips] Introduction of Tools with Enhanced Management Functions

Let's introduce a cloud-based annotation tool equipped with features that support team collaboration, such as user permission management, comment functions, and history tracking. For example, "Annofab" allows role-based permission settings, making it easier for the entire team to work efficiently and with high accuracy.

Reference: 3 Annotation Tools with Management Features Available for Free

2-3. Method (Approach / Workflow)

[Issue] Insufficient Maintenance of Quality Control Documents

When there are changes to the specifications, if quality management documents (specification sheets, Q&A sheets, checklists, etc.) are not properly maintained, some members may continue working based on outdated standards. As a result, inconsistencies in quality may occur, and there is a risk of overlooked corrections.

[Measures & Tips] Thorough Document Management and Coordination

If there are any changes to the specifications, promptly reflect them in the documents and share them with the entire team. Holding a meeting to align everyone on the changes is even more effective. By ensuring that everyone works based on the latest standards, you can prevent misunderstandings and inconsistencies in quality.

[Issue] Frequent occurrence of difficult-to-judge edge cases

In annotation work, many edge cases arise that cannot be determined by the rules described in the specifications alone. This can lead to differences in judgment among workers, resulting in quality variations and rework for corrections.

[Measures & Tips] Unify Decision Criteria with a Collection of Edge Cases

Create a collection of "edge cases" where judgment was difficult, and share the response policies within the team. By visualizing the decision criteria based on case examples, you can reduce uncertainty and maintain consistency in your work.

2-4. Information (Information Management)

[Issue] Knowledge is not shared, and the same mistakes are repeated

When multiple workers handle tasks independently, questions, improvement suggestions, and troubleshooting information tend to be scattered across various places. As a result, similar mistakes and questions are repeated, and organizing the information becomes difficult.

[Measures & Tips] Gather knowledge in one place so that anyone can access it

Let's consolidate FAQs, manuals, and Q&A exchanges into a "knowledge base" on the cloud or chat tools. By centralizing information, you can quickly find what you need when necessary, enhancing the learning and efficiency of the entire team.

[Issue] Without a QA person in charge, mistakes tend to be repeated

When multiple workers proceed with annotation, there is a risk that trends in mistakes and areas for improvement will not be identified, leading to repeated errors. Without a QA person, it becomes difficult to maintain consistent overall quality.

[Measures & Tips] Trend Analysis of Mistakes by QA Personnel and Feedback Mechanism

Assign a dedicated QA personnel to analyze error trends and provide feedback to workers. Sharing improvement points based on the analysis results in regular meetings will lead to stable and improved quality.

3. Summary

In this article, we detailed the common challenges and countermeasures when promoting in-house annotation in AI development sites, dividing the discussion into patterns where engineers work alone and patterns where multiple personnel within the company collaborate. By addressing the four perspectives introduced here—"Man (personnel)", "Machine (equipment and tools)", "Method (techniques and workflows)", and "Information (information management)"—it is possible to stabilize work quality and improve efficiency.

The important thing is to avoid making tasks dependent on specific individuals and to establish a standardized and transparent environment. By gradually implementing measures such as sharing specifications and knowledge, unifying tool usage, and introducing a checking system, annotation work within the company can be operated with high quality. While leveraging the flexibility unique to in-house operations, let's build a system that fits the actual worksite.

4. Human Science Teacher Data Creation and LLM RAG Data Structuring Outsourcing Service

Over 48 million pieces of training data created

At Human Science, we are involved in AI model development projects across various industries, starting with natural language processing, including medical support, automotive, IT, manufacturing, and construction. Through direct transactions with many companies, including GAFAM, we have provided over 48 million high-quality training data. We handle a wide range of training data creation, data labeling, and data structuring, from small-scale projects to long-term large projects with a team of 150 annotators, regardless of the industry.

Resource management without crowdsourcing

At Human Science, we do not use crowdsourcing. Instead, projects are handled by personnel who are contracted with us directly. Based on a solid understanding of each member's practical experience and their evaluations from previous projects, we form teams that can deliver maximum performance.

Not only for creating training data but also supports the creation and structuring of generative AI LLM datasets

In addition to creating labeled and identified training data for data organization, we also support the structuring of document data for generative AI and LLM RAG construction. Since our founding, we have been engaged in manual production as a primary business and service, leveraging our unique know-how gained from extensive knowledge of various document structures to provide optimal solutions.

Secure room available on-site

Within our Shinjuku office at Human Science, we have secure rooms that meet ISMS standards. Therefore, we can guarantee security, even for projects that include highly confidential data. We consider the preservation of confidentiality to be extremely important for all projects. When working remotely as well, our information security management system has received high praise from clients, because not only do we implement hardware measures, we continuously provide security training to our personnel.