How to Outsource Annotation Work? 7 Tips

1. Advantages and Disadvantages of In-House Annotation

Here, we will look at the advantages and disadvantages of in-house annotation.

Advantages
When done in-house, there is less risk of data leakage to external parties, allowing for relatively secure operation from the perspective of security and privacy protection. Additionally, the annotation process can be designed flexibly; for example, even if there are changes to annotation requirements or specifications, it is possible to quickly communicate these to annotators. Furthermore, quality and productivity can be monitored in real time. If resources can be allocated to managing annotation and human resources, the benefits gained from in-house annotation will be significant.

Disadvantages
When AI engineers perform annotation themselves, it not only causes delays in AI development but also results in increased development costs because highly paid engineers are doing the annotation. On the other hand, even if you secure annotation-specialized personnel resources, if the annotation work does not continue, there will be idle capacity, leading to waste, and managing and controlling these personnel resources incurs considerable costs.
Furthermore, to ensure the quality and productivity of annotation using many personnel resources, experience and skills in annotation management, which differ from development, are required. In the case of in-house production, if you do not possess these skills, it is more difficult than expected to guarantee quality and productivity. In fact, we often hear from our customers, "Although we tried in-house annotation, it did not go well, and the lack of these skills and know-how became an internal issue, so we consulted your company."

2. Benefits of Outsourcing Annotations

While this slightly overlaps with what was mentioned above, the annotation work itself does not require much specialization. However, to perform annotations properly, expertise in management related to annotation-specific personnel and tasks is necessary. This is different from the specialization of development engineers, so having engineers in-house perform annotation work alongside their primary duties does not efficiently utilize the engineers' expertise and leads to a decrease in the productivity of their original engineering tasks.

Annotation work itself does not require much specialization, but expertise is necessary for managing the personnel and tasks involved in annotation. Since this expertise differs from that of development engineers, having engineers perform annotation work alongside their primary duties in-house results in inefficient use of their specialized skills and leads to a decline in the productivity of their core engineering tasks.
Therefore, by outsourcing annotation work, engineers can focus their expertise on their primary responsibilities, which can ultimately lead to cost reduction and improved productivity.

No labor costs or management expenses

Even if you gather annotation personnel in-house, managing a large volume of annotations requires a significant number of staff, leading to high costs and effort in management areas such as quality and progress control. Additionally, unless annotation tasks are consistently generated, these resources can become surplus. Furthermore, since annotation requirements vary by project, there is a need for personnel training, which incurs additional educational costs. By outsourcing, you can expect to reduce these labor and management costs. Let's take a detailed look at the benefits of such outsourcing.

Related Blog
>>7 Tips for Successful Annotation

2-1. Cost Reduction and Time Savings

In addition to what has been mentioned so far, the labor cost per unit for development engineers and annotators differs significantly. Therefore, even when including management fees and profits from outsourcing vendors, it is often possible to reduce costs compared to performing annotation in-house, and the cost-saving effect becomes more pronounced as the volume of work requested increases. Additionally, many annotation vendors have specialized experience in annotation tasks, so they possess know-how to improve productivity per person without solely relying on increasing personnel. Thus, by outsourcing, it is possible to not only secure time within the company but also expect a reduction in the overall delivery time of annotation tasks while keeping costs down.

2-2. Reducing Management Burden

Handling large-scale data annotation tasks typically involves a significant number of personnel. To ensure quality while adhering to the schedule, various management tasks are required, such as preparing work manuals and progress management, regardless of the type of annotation. Especially in the early stages of the work, there are many questions and answers with the annotators. This management not only takes more time than expected but also requires experience and expertise to proceed efficiently. If you can entrust this to an experienced external vendor, the management burden can be alleviated.

2-3. Improvement and Stabilization of Quality

Although automation is progressing, annotation is still often done manually, and the experience and skills of annotators, as well as their understanding of specifications, along with variations among annotators, greatly affect the overall quality of the training data. If you have experienced annotators and outsourcing vendors that can manage personnel appropriately, you can achieve high-quality annotations based on specifications and work instructions. By outsourcing to such vendors with experienced annotators, you can expect stable, high-quality training data.

2-4. Comparison Table of In-House and Outsourcing

When proceeding with annotation, many customers struggle with the question of whether to handle it in-house or outsource it. Although there are variations depending on the external vendor, the characteristics can be roughly summarized as follows. We hope this will be helpful when considering outsourcing.

3. Three Common Failures and Cautions When Outsourcing Annotations

While outsourcing annotation has many advantages, proceeding without sufficient preparation can easily lead to failure. In particular, the following cases require caution.

1. Rework Due to Ambiguous Specifications
If the requirements definition is insufficient, frequent corrections may occur during the checking process or after delivery, which can result in increased costs and delayed deadlines.

2. Lack of Shared Quality Standards
If quality standards are not shared with the outsourcing vendor, there may be significant differences in judgment among annotators. This can lead to inconsistencies in quality and potentially affect the model's training.

3. Lack of Communication
If appropriate communication methods and means of information sharing are lacking, delays in responding to questions from the outsourcing vendor regarding specifications that arise after work begins, or failures to properly share information, can occur. As a result, large amounts of data may be created based on incorrect understanding or recognition of the specifications. Especially, understanding and sharing annotation specifications can sometimes be difficult to convey effectively through text alone, so particular caution is necessary.

To prevent such failures, it is important to carefully consider the "7 Points for Choosing an Outsourcing Partner" introduced in the next chapter.

4. Seven Tips for Choosing an Annotation Outsourcing Partner

When outsourcing annotation, choose a company that aligns with your AI development goals, as well as requirements for quality and security. Here, we will explain seven points to consider when selecting an outsourcing partner.

4-1. Suitability for Your Company's Requirements

The types of annotations, data formats, tools you want to use, and the scale of the project can vary greatly. Because this is somewhat obvious, it is easy to overlook confirming these details, so it is important to remember to check whether the outsourcing partner meets your company's requirements.

4-2. Expertise and Experience

In specialized annotations, such as medical image annotation, text, and language annotation, expertise is often required. Be sure to check whether the company has a track record of annotations that match your company's objectives. A company with high expertise and experience in specialized annotations can provide annotations tailored to your requests and offer appropriate advice even if your company lacks the know-how or experience in such annotations.

4-3. Quality Control

It is important to ensure that quality control is conducted properly. This includes not only the project management system, checking methods, and checking structure, but also the thorough management of change information related to specification changes. When feedback such as corrections or changes is provided from our company, it is essential to confirm how that information is communicated to the annotators, how it is reflected, and how it is verified. It is also important to check the processes and methods of information management.

4-4. Security

It is important to ensure that appropriate security measures are in place. Some companies may only accommodate remote work by cloud workers. It is possible that sufficient security measures or compliance with the company's security requirements cannot be ensured through remote annotation. In such cases, check whether they also support on-site work, such as security rooms or client site assignments. Additionally, it is essential to choose a company that implements multifaceted security measures, including not only hardware but also security training for workers and the establishment of information security management systems.

4-5. Scalability

While it is often not given much importance at the PoC stage, let's consider the possibility of expanding the scale of annotation as we move to the next phase, and check the number of personnel that can be secured by the outsourcing partner and the delivery time for the expected scale of annotation. It would also be ideal to confirm whether they can respond in urgent situations.

4-6. Communication

In annotation, even after defining requirements and creating specifications, various exceptions and edge cases may arise as work progresses. Frequent Q&A and feedback with outsourcing partners often occur, making it important not only to ensure smooth communication but also to manage, update, and communicate information efficiently. It is also essential to verify whether information can be shared appropriately and efficiently through communication methods tailored to requests, such as the use of chat tools and centralized information management.

4-7. Price and Pricing Conditions

Depending on the outsourcing partner, not only the fees for annotation services but also the pricing methods for file unit costs, annotation unit costs, hourly rates, and other pricing units may vary. When selecting an outsourcing partner, it is recommended to gather multiple quotes from different companies while aligning the conditions for data quantity, delivery time, checking methods, and the number of objects per file, as well as the conditions for the estimate request. Additionally, since fees often vary based on the working conditions such as remote work or security rooms, it is important to compare prices accordingly.

5. Human Science Achievements

Here, we would like to introduce an example of our past achievements. We hope this serves as a reference when considering outsourcing.

5-1. Project Overview

Conversation Text Classification Annotation

5-2. Issues and Proposals

Issue

When we were consulted, the customer had the following issues.
・It was their first time outsourcing annotation, so they were concerned about pricing, quality control, the checking system, and whether their feedback and requests would be properly reflected.
・The annotation involved high ambiguity with no absolute correct answers. Therefore, significant variability among individuals was expected, and they were struggling with how to perform annotation and how to check it in order to obtain accurate training data.

Proposal Details

Based on these customer concerns, we made the following proposals.
・Proposal of quality control, incorporation of feedback, and management system. Presentation of a clear estimate specifying conditions and unit prices, with flexible adjustments to the estimate content according to requests.
・For annotations with high ambiguity and no absolute correct answer, instead of the usual correctness check, we proposed a triple pass + check method where three annotators (experienced in similar tasks) perform the same annotation.
・Creation of annotation specifications by our company and alignment with the customer.
・Establishment of a rapid feedback system through the opening of a chat with the customer.

5-3. Effects After Implementation and Customer Feedback

"I am very satisfied with the quality of the annotations and your company's response during the project. It was very helpful that you responded promptly to our requests and corrections during the work. From the moment I first approached you, I felt that Human Science was a team of experienced professionals, so I thought it would be safe to rely on you. I truly feel that it was the right decision to trust Human Science."

6. Human Science Annotation, LLM RAG Data Structuring Agency Service

Over 48 million pieces of training data created

At Human Science, we are involved in AI model development projects across various industries, starting with natural language processing, including medical support, automotive, IT, manufacturing, and construction. Through direct transactions with many companies, including GAFAM, we have provided over 48 million high-quality training data. We handle a wide range of training data creation, data labeling, and data structuring, from small-scale projects to long-term large projects with a team of 150 annotators, regardless of the industry.

Resource management without crowdsourcing

At Human Science, we do not use crowdsourcing. Instead, projects are handled by personnel who are contracted with us directly. Based on a solid understanding of each member's practical experience and their evaluations from previous projects, we form teams that can deliver maximum performance.

Generative AI LLM Dataset Creation and Structuring, Also Supporting "Manual Creation and Maintenance Optimized for AI"

Since our founding, our main business and service has been manual creation, and we now also support "the creation of AI-friendly documents to facilitate the introduction of generative AI for corporate knowledge utilization." In sharing and utilizing corporate knowledge and documents using generative AI, current technology still cannot achieve 100% accuracy with tools alone. For customers who absolutely want to leverage their past document assets, we also provide document data structuring. We offer optimal solutions that leverage our unique expertise, deeply familiar with various types of documents.

Secure room available on-site

Within our Shinjuku office at Human Science, we have secure rooms that meet ISMS standards. Therefore, we can guarantee security, even for projects that include highly confidential data. We consider the preservation of confidentiality to be extremely important for all projects. When working remotely as well, our information security management system has received high praise from clients, because not only do we implement hardware measures, we continuously provide security training to our personnel.

In-house Support

We also provide personnel dispatch services for annotation-experienced staff and project managers who match our clients' tasks and situations. It is also possible to organize teams stationed at the client's site. Additionally, we support the training of your workers and project managers, selection of tools tailored to your situation, automation, work methods, and the construction of optimal processes to improve quality and productivity. We assist with any issues related to annotation and data labeling that our clients may face.