Does repeated checking improve annotation quality? ~Quality control in our company's annotation field~

Spinoff Blog Project
――Annotations Supporting AI in the DX Era. The Reality of the Analog Field
Does repeated checking improve annotation quality?
〜Quality Control in Our Annotation Field～

Until now, our company has been publishing various blogs related to annotation and AI. In those, we have focused on conveying general knowledge and know-how. While the task of annotation may seem simple when expressed in words, it is a task that inevitably involves a lot of human interaction due to its inherent "ambiguity" and the fact that it is a "task that cannot be avoided by humans." Therefore, it often becomes quite messy and cannot be resolved with the neat theories that are commonly found. In order to ensure quality and productivity, a variety of experiences and know-how are actually required.

Therefore, we believe that understanding the actual problems that occur in the field of annotation and how to address them can serve as valuable hints for leading to successful annotation.
In our company, we want to share what actually happens on-site and the specific responses and measures we take. Unlike a regular blog, we aim to convey the realities of the field with our spin-off blog project titled: "Annotation Supporting AI in the DX Era: The Reality of the Analog Field," including our unique characteristics and commitments.

>>Past Published Blogs (Partial)

7 Tips for Successfully Leading Annotations

What is training data? An explanation from its relationship with AI, machine learning, and annotation to how to create it.

How to ensure and improve the quality of training data? Practical methods explained!

Table of Contents

1. Does the quality of annotations improve by conducting multiple checks?
2. Quality Beyond Just Deliverables
3. Considerations from the field of human science before increasing checks
4. Differentiation Based on Situation and Required Quality
5. A Word from Our Customers
6. Summary

1. Does the quality of annotations improve by conducting multiple checks?

This time, it may seem extremely obvious, but I would like to present a deep theme that is actually easy for the field to fall into.

Not limited to annotations, we encounter situations in various industries and locations where quality is emphasized through double-checking and triple-checking. But does increasing the number of checks really improve quality? From the perspective of a service user, it can serve as a reassurance and lead one to think, "Is this company of good quality?" It is true that by increasing checks, the quality of the deliverables provided to customers improves. However, the desired quality varies for different things. Therefore, it is important to pay attention to this when using the service.

Not just with annotations, but in any field, there is a tendency to focus too much on quality, which can lead to excessive checks. However, the more checks are repeated, the higher the costs rise, which are then passed on to the price and delivery time, ultimately reflecting in the costs that customers have to pay.

While this may be a discussion about words and concepts, the definition of quality is "the extent to which a product or service meets its intended purpose." In other words, it refers to "the degree to which customer satisfaction is fulfilled." Therefore, if checks are repeated and costs are passed on, it ultimately leads to a decrease in customer satisfaction. To conclude, while repeated checks improve quality in a narrow sense (the quality of deliverables), they lead to a decrease in quality in a broader sense, which is customer satisfaction.

2. Quality Beyond Just Deliverables

When you hear the word "quality," what comes to mind? Even when we say that something is of good quality, it encompasses various elements beyond just the quality of deliverables like training data. It includes aspects of QCD such as price and delivery time, as well as management skills to achieve those, and the attitude and responsiveness to meet customer needs. I believe it is something that is judged comprehensively.

Our company often receives feedback from clients stating that "the quality of the annotations is good," and it is true that we have many repeat customers. However, upon further discussion, I often feel that this praise is not solely based on the quality of the deliverables, such as training data and annotations. We frequently receive comments from clients like, "You were really flexible in accommodating our requests, and it was a great help." While we take joy in this feedback, it also makes me realize that the evaluation is not based solely on the annotation deliverables, and that quality is a multifaceted concept.

By the way, our company believes that the quality of communication with our clients, which equals the quality of management, is very important. As a policy for our annotation services, we advocate "Management Hospitality" (this is a term we coined...). We value the attitude of being close to our clients and working to solve their challenges, and this has become part of our company's identity, not just in our annotation services.

3. Considerations from the field of human science before increasing checks

Returning to the discussion about specific checks and tasks, the task of checking itself does not inherently create added value, so if it is not necessary, it is better not to do it. However, since annotation is a task that must be performed by humans, careless mistakes can occur, and we cannot stop checking due to the skill level of the workers. Therefore, it is important to develop tools and automation for checking, as well as to establish work processes that minimize errors and to educate workers to prevent errors from occurring.

These do incur costs, but no matter how much we aim for automation and toolization of checks, the task that awaits after the checks, which can be negatively described as "rework," is still present. When considering not only the man-hours required for corrections but also the preparatory work associated with them, as well as the exchange and management of information, it ultimately turns out to be more cost-effective to invest in preventing the occurrence of errors.

In terms of quality, if there are too many errors in the checks, excluding those that can be automated or tool-based, it disrupts the rhythm and concentration of the checker, leading to a decrease in detection capability, which in turn halves the effectiveness and efficiency of the checks. Therefore, our company places great importance on educating people, such as by thoroughly understanding specifications and holding meetings for Q&A. We also emphasize preventive activities, such as preparing procedures in advance and considering processes to suppress the occurrence of errors. Additionally, we believe it is very important to create an environment where workers can maintain their own rhythm and concentration while performing their tasks, thereby preserving the detection capability of the checkers.

4. Differentiation Based on Situation and Required Quality

That said, as I have repeated in previous blogs, annotation work is an unavoidable task that must be performed by humans, so no matter how much ingenuity or improvement is applied, errors cannot be completely avoided. Therefore, it is also nonsensical to pursue error prevention excessively, and we must adapt our responses according to the quality level required by our clients and the specific circumstances.

For example, in short-term projects that are completed within a week, if we focus on training, the project will end while we are still training. In such cases, we shift our focus to checking, and we also adjust the weight of the checks and the frequency of sampling according to the situation. Moreover, the quality levels required by our clients vary significantly. For instance, some clients may say, "Since we are currently in the PoC phase, we just want to input data into the learning model for now. Therefore, the quality can be moderate, and we prioritize price." As the quality levels required by clients differ, it is necessary to adapt our approach accordingly.

To achieve this, it is necessary to determine the direction of the project and tasks before the start of the project. To execute this appropriately, it is important to have project management capabilities that can flexibly respond to customer needs and the situation, as well as a project management system, organization, structure, and the development of project managers that can realize the desired quality and cost.

5. A Word from Our Customers

One time, we received a request from a client with whom we have an ongoing relationship. As usual, we submitted a quote and an annotation sample, and we were told, "You don't need to be this thorough. We plan to check it on our end as well, so can we adjust the price by reducing the precision and the frequency of checks?"

I was once again reminded that what is truly important is to thoroughly understand the client's requirements and to determine the target quality, work, and checking processes in accordance with those requirements. However, when working with a client for the first time, communication may not be smooth enough to reach that level during negotiations, and often the processes are not fully established at that point. Therefore, in our company, during negotiations, we often ask how much annotation accuracy is required and what level of quality is expected. Additionally, our estimates include details on the required annotation accuracy and the frequency of checks, depending on the situation.

6. Summary

What has been described so far is a method that aligns with our organization's culture and practices. It is merely one solution derived from our trial and error in response to customer feedback and our inability to meet customer expectations, which has led to repeated losses in business negotiations. We have seriously faced the challenge of balancing quality and cost, and through continuous improvement, we hope this can serve as a reference for you as one of the possible approaches.

Author:

Kazuhiro Sugimoto

Annotation Department Group Manager

・At my previous job at a Tier 1 automotive parts manufacturer, I focused on quality design and quality improvement guidance for the manufacturing line, and I have experience as a project manager for model line construction and in cross-departmental projects such as business efficiency improvement (lean improvement) consulting teams.
・In my current position, I have been involved in the establishment and expansion of the annotation business, the construction and improvement of the management system for annotation projects, after promoting management systems such as ISO and knowledge management.
・QC Level 1, Member of the Japan Society for Quality Control