Table of Contents

1. Introduction

With the emergence and evolution of generative AI, AI adoption and development in companies have diversified beyond traditional discriminative AI to include those utilizing generative AI. Accordingly, the needs for annotation service vendors have evolved into more diverse demands, transforming from conventional annotation alone to comprehensive "data solutions" involving all data used for AI training. Today, I would like to explain how the requirements for selecting annotation service vendors suitable for long-term partnerships have changed alongside the expansion and evolution of AI adoption and development, and how to choose vendors you can maintain long-term relationships with.

▼Related Blog Posts:

How to Outsource Annotation Work? 7 Tips

Recommended outsourcing services for streamlining annotations! What are the key points for company comparison?

2. Common Troubles Caused by Mistakes in Selecting Annotation Service Vendors

Before explaining the key points for selecting an annotation service vendor, let's look at typical troubles and issues that can arise when a mismatch occurs due to vendor selection failure. Although we are a vendor ourselves, when the volume exceeds what we can handle internally, we sometimes ask partner vendors with the customer's consent. Therefore, we have selected annotation service vendors many times and believe we can speak from the same standpoint as our customers in a way.

●Variations in Quality and Discrepancies in Label Interpretation

These issues stem from insufficient understanding of specifications and misalignment in how exceptions and edge cases are handled. Most often, they are caused by a lack of communication between the client and the vendor, inadequate manualization of work specifications, insufficient information sharing and thoroughness with workers—in other words, annotation-related judgments relying heavily on the individual workers. The likelihood of such problems increases as the difficulty and ambiguity rise. As a result, it is common to hear that many corrections occur during the client’s acceptance checks, or even when sent back to the vendor, the corrections take time and cause delays in delivery schedules.

● Lack of response to specification changes (insufficient flexibility)

While this may be acceptable in the mass production phase of annotation after a PoC, it is rare for annotation specifications to be finalized and proceed exactly as planned from the early stages of AI development. In fact, specification changes are inevitable in AI development.

When performing work beyond what was agreed upon at the time of contract, in some cases cost increases are somewhat unavoidable depending on the situation. However, it is also common for the situation to become rigid with exchanges like "We want to change the specifications now" → "It’s already too late to do that," resulting in a compromise to proceed with the original specifications.

●Handling Variability in Data

The amount of annotation targets within each file or data, such as images, greatly affects the workload and thus the cost. While it is possible to make some predictions based on samples, in most cases, it is not realistic to grasp the exact amount beforehand. As a result, if the volume significantly exceeds the initial estimate at the time of contract, discussions about additional costs and the like often arise.

●Cannot Scale

Although the quality was high during small-scale annotation tasks such as the PoC phase, as soon as it entered the mass production phase, quality variations became frequent and unstable due to the increase in workers, resulting in many checkbacks and rework within the vendor, and it is common to hear about troubles such as failing to meet deadlines.

▼Related Blog Posts:

How to Prevent Personalization of Annotations? Characteristic Causes and Countermeasures of Personalization in In-house Production Sites

[Spin-off] How to Deal with Edge Cases That Cannot Be Fully Covered by Specifications

[Spin-off] What Are the Costs of Annotation?

How to Outsource Annotation Work? 7 Tips

3. Changes in Vendor Requirements Accompanying the Shift in AI Development Needs

We have explained many common examples of mismatches that often occur, but recent changes in the development environment, as described below, hold the potential to further increase mismatches caused by insufficient consideration during vendor selection.

●Shift to Specialized Domains

In AI development environments, general-purpose machine learning models and datasets, such as those for object recognition, have been released, and their scope is continuously expanding while accuracy improves. As a result, identification-type AI development has shifted in recent years toward specialized domains and more complex AI development. Correspondingly, annotation vendors are also increasingly required to provide high-level expertise and specialized personnel.

●Growing Demand for Domain-Specific LLMs (Usable Generative AI)

With the spread of generative AI and LLMs, it has become clear that off-the-shelf LLMs often cannot withstand actual business use. While this may be less of an issue for general tasks such as document creation, the trend is even stronger in work that leverages specialized fields or company-specific know-how. As a result, many companies are increasingly adopting the construction of internal knowledge systems using RAG and developing LLMs that cover greater specialization. However, it remains difficult to prepare datasets and training data solely in-house, leading to more opportunities to request annotation vendors to create datasets with industry knowledge and specialized expertise.

●Security Measures

The increase in LLM development that encompasses corporate know-how means, in other words, that handling confidential data which serves as the source of corporate knowledge is becoming more frequent. Additionally, in the medical and financial industries, there is a large amount of personal information or equivalent data, requiring especially advanced information security management systems. However, an over-spec security management system only increases costs. Therefore, annotation service vendors are required not only to have high-level information security management systems but also to be able to build flexible security management systems according to the security requirements of the data they handle.

●Accompanying the Entire AI Development Process

As the AI being developed becomes more advanced, the specification settings for annotation and the difficulty of the tasks inevitably increase. Additionally, the work specifications greatly influence the labor and cost required for annotation tasks. Therefore, it is no longer sufficient to simply carry out tasks according to the specifications; from the specification formulation stage through the entire AI development process, the involvement of vendors with extensive annotation experience is increasingly required.

▼Related Blog Posts:

[Spin-off] What Is the Unexpected Difficulty of Annotation?

The Growing Demand for Domain-Specific LLMs and Their Background

Enhancing the Reliability of Medical AI: Quality and Security Management of Training Data

4. Key Points for Selecting Annotation Vendors for Long-Term Partnerships

So far, we have discussed various topics, and here I would like to summarize the key points for selecting vendors for long-term partnerships.

●Can they handle highly complex tasks?

As mentioned above, the scope covered by existing learning models, which have seen remarkable performance improvements, is expanding for general-purpose and relatively simple AI development. Therefore, the focus of development requiring annotation is inevitably shifting to more complex and advanced areas. Additionally, in such annotation work, many exceptions and edge cases arise that cannot be fully covered by specifications alone. Hence, it is necessary for the vendor to have a system in place that not only maintains manuals but also manages communication such as Q&A and response policies with the client, properly shares this information with the workers, and ensures thorough adherence.

● Can they reflect expertise in the work? Can they secure specialized personnel?

Among high-difficulty annotation tasks, those requiring particular expertise are increasing. It is possible to have specialized personnel within the customer's company perform the work, but this not only incurs costs, but in some cases, the in-house specialists may be too busy with their primary duties to respond adequately.

Of course, it can be difficult to expect vendors to possess the specialized knowledge that each company has, but it is important whether the vendor can receive training from the customer's experts, pass on know-how and skills to the vendor's PMs and workers, acquire know-how and skills through repeated Q&A with the customer during sample annotation, propose flexible optimal processes such as dividing tasks between parts that require expertise and those that do not with the customer, and whether they are accustomed to such practices.

Also, in specialized fields such as national qualifications for medical professionals and academic knowledge, it is relatively easy to secure specialized personnel, so it is necessary to confirm whether the vendor can accommodate such requirements.

● Can they respond flexibly according to demands and circumstances?

Simply completing tasks and work according to the specifications, no matter how good the quality, is still insufficient to be considered a vendor you can work with long-term. It is important whether they can act and respond as if they are part of the customer's machine learning team.

There are many aspects of machine learning that can only be understood through trial and error. Therefore, an ideal vendor is one who can provide advice and suggestions during the formulation of annotation specifications, flexibly respond to changes in annotation specifications, and if the specifications are not yet finalized and there is a possibility of changes, establish a phase to conduct a small amount of work to reconsider the specifications—essentially, a vendor who can work alongside throughout the entire model development process.

Also, regarding data, if there is likely to be significant variation in the number of annotation targets per file, it is more reassuring to work with a vendor who can propose flexible approaches according to the situation, such as contracting based on the number of annotation targets rather than the number of files, or performing a relatively small amount of work initially to more accurately estimate the workload before moving on to the next phase, and who has experience with such methods.

In terms of flexibility, there is also the balance between cost and quality. Depending on the development phase and requirements, there may be cases where quantity or cost takes priority over quality. No matter the task, a vendor that is "high quality but expensive" cannot be considered a long-term partner. An ideal vendor is one that can flexibly switch between offshore and domestic resources, or hybridize automated annotation with manual work, adapting processes and locations according to the required quality, cost, and delivery deadlines.

●Information Security Management System

Data in highly specialized fields tend to require a high level of information security management systems. Therefore, vendors equipped with such systems and environments are essential. However, not all data used or handled by the AI developed by each company necessarily requires a high-level information security management system. Information security management often involves a trade-off with cost, and demanding more management than necessary will impact expenses. Thus, an ideal vendor is one who can flexibly offer options for management systems according to the security requirements of the data being handled.

●Flexible Scaling System

In some situations, it may be necessary to carry out a large volume of work within a short deadline. It is problematic if the vendor says, "We are busy now, so we cannot respond immediately," or if they require a long delivery time. Therefore, it is essential that the vendor has a sufficient capacity. It is also problematic if the quality becomes unstable as soon as the workforce expands. It is important that the organization has mechanisms and experience to stabilize quality even when the number of workers and the system scale up.

▼Related Blog Posts:

[Spin-off] Creating an Annotation Work Environment Onsite (In Our Company's Security Room)

[Spin-off] What Is the Unexpected Difficulty of Annotation?

How to Outsource Annotation Work? 7 Tips

Recommended outsourcing services for streamlining annotations! What are the key points for company comparison?

5. Summary

We have explained the key points for choosing annotation service vendors with whom you can have a long-term relationship, but there is no vendor that is perfect and all-encompassing in every aspect. Each has its own strengths, weaknesses, and characteristics. It is also necessary to consider using different vendors depending on the purpose and situation, such as whether they have many specialists suited to your industry. However, the most important factor when selecting an annotation service vendor for a long-term partnership is whether they can flexibly respond to your requests with a customer-centric attitude. Additionally, AI is a rapidly advancing industry. It is essential that the vendor not only relies on existing technologies but also keeps up with the evolving trends in the AI industry and continuously strives to provide solutions that meet customer needs. There is no doubt that these are the most important points.

▼Related Blog Posts:

[Spin-off] The Invisible Value Beyond Price That Cannot Be Measured by Cheapness Alone

6. Human Science Teacher Data Creation, LLM RAG Data Structuring Agency Service

Over 48 million pieces of training data created

At Human Science, we are involved in AI model development projects across various industries, starting with natural language processing and extending to medical support, automotive, IT, manufacturing, and construction, just to name a few. Through direct business with many companies, including GAFAM, we have provided over 48 million pieces of high-quality training data. No matter the industry, our team of 150 annotators is prepared to accommodate various types of annotation, data labeling, and data structuring, from small-scale projects to big long-term projects.

Resource management without crowdsourcing

At Human Science, we do not use crowdsourcing. Instead, projects are handled by personnel who are contracted with us directly. Based on a solid understanding of each member's practical experience and their evaluations from previous projects, we form teams that can deliver maximum performance.

Generative AI LLM Dataset Creation and Structuring, Also Supporting "Manual Creation and Maintenance Optimized for AI"

We support not only labeling for data organization and training data creation for identification-based AI, but also the structuring of document data for generative AI and LLM RAG construction. Since our founding, manual production has been our main business and service, and we now also provide support for "organizing business knowledge and manualization toward future generative AI and RAG introduction and utilization." We offer optimal solutions leveraging our unique expertise deeply familiar with the structure of various documents.

Secure room available on-site

Within our Shinjuku office at Human Science, we have secure rooms that meet ISMS standards. Therefore, we can guarantee security, even for projects that include highly confidential data. We consider the preservation of confidentiality to be extremely important for all projects. When working remotely as well, our information security management system has received high praise from clients, because not only do we implement hardware measures, we continuously provide security training to our personnel.

In-house Support

We also provide personnel dispatch services for annotation-experienced staff and project managers who match our clients' tasks and situations. It is also possible to organize teams stationed at the client's site. Additionally, we support the training of your workers and project managers, selection of tools tailored to your situation, automation, work methods, and the construction of optimal processes to improve quality and productivity. We assist with any issues related to annotation and data labeling that our clients may face.