[Spin-off] Does double-checking improve data annotation quality? ~Quality control at our data annotation site~

Spin-off blog project
- Annotation that supports AI in the DX era. Will the quality of data annotation improve by repeatedly checking the analog real scene?
Quality management at our annotation site

Our company has been publishing various blogs about data annotation and AI. In those blogs, we have mainly shared general knowledge and know-how. Data annotation may seem simple at first glance, as it involves putting the content into words, but it is actually a task that cannot be avoided by humans and contains a lot of "ambiguity". Therefore, there is a lot of interaction between people involved in the process. As a result, it requires a lot of experience and know-how to ensure quality and productivity, which cannot be achieved by just following clean theories.

Therefore, we believe that it is helpful to know the specific problems and responses that occur in the actual annotation field as a hint for successful annotation.
In our field, what actually happens and what specific responses and measures are taken? Unlike regular blogs, in our spin-off blog project titled "Annotation that supports AI in the DX era. The real world of analog annotation", we would like to share the reality of our field, including our unique features and commitments.

>>Past Published Blogs (Some)

7 Tips to Successfully Lead Annotations

What is Teacher Data? Explanation from the relationship with AI, machine learning, and annotation to how to create it.

"How to ensure and improve the quality of teacher data? Explaining practical methods!"

Table of Contents

1. Does the quality of data annotation improve by double-checking?
2. Quality that goes beyond just deliverables
3. Things to Consider Before Increasing Checks at Human Science
4. Use according to the situation and required quality.
5. Customer Testimonials
6. Summary

1. Does the quality of data annotation improve by double-checking?

This time, it may seem very obvious, but in fact, it is a deep topic that is easy to fall into in the field. I would like to deliver it.

Not only in data annotation, but also in various industries and places, we encounter scenes where quality is emphasized by double or triple checks. However, does quality really improve by repeating checks? From the perspective of using the service, it may be a reassuring factor and make you think, "Is this company really good in quality?" When it comes to the deliverables that are delivered to customers, it is true that quality improves by repeating checks, but the desired quality varies depending on the product. Therefore, when using the service, attention is necessary.

Not only in data annotation, but in any field, it is natural to be concerned about quality, and we tend to do it without thinking. However, the more we check, the more the cost increases, and it is passed on to the price and delivery date, resulting in the cost being passed on to the customer.

While we may talk about words and concepts, the definition of quality is "the extent to which a product or service meets its intended use". In other words, it is the degree to which it satisfies the customer. Therefore, if checks are repeated and costs are transferred, it will ultimately lead to a decrease in customer satisfaction. In conclusion, while repeating checks may improve quality in a narrow sense (the quality of the end product), it can also lead to a decrease in quality in a broader sense = customer satisfaction.

2. Quality that goes beyond just deliverables

When you hear the word "quality", what comes to mind? Even if we say that something has good quality, it's not just about the quality of the end product, such as the teacher's data. There are various elements included in that, such as the QCD aspects including price and delivery time, management skills to achieve them, and attitudes and responsiveness to meet customer needs. I think it is something that is judged comprehensively.

Our company often receives high praise from our customers for the quality of our data annotation, and it is a fact that we have many repeat customers. However, upon closer examination, we realize that it is not just the quality of the final product - the training data and annotations - that our customers are satisfied with. We also receive many comments such as "Thank you for being flexible and accommodating to our requests," which brings us joy. At the same time, it reminds us that quality is a multifaceted concept and cannot be solely evaluated based on the final product of data annotation.

By the way, we believe that the quality of communication with our customers is especially important in terms of quality management. As a policy for our data annotation services, we uphold the concept of "management hospitality" (this is our own term...). We value the attitude of standing by our customers and solving their issues, which not only applies to our data annotation services but also represents our company's identity.

3. Things to Consider Before Increasing Checks at Human Science

Returning to the discussion of specific checks and tasks, the task of checking itself does not necessarily produce added value, so if it is not necessary, it is better not to do it. However, since data annotation is a task that must be done by humans, careless mistakes can occur and it is not possible to stop checking depending on the proficiency of the worker. Therefore, it is important to toolize and automate the checking process, as well as to build a process that is less prone to errors and educate workers to not create errors.

These will inevitably incur costs, but no matter how much automation and tooling is implemented for checks, the task of "fixing" that awaits after the checks is, to put it bluntly, a redo. Not only the man-hours required for fixing, but also the preparation work associated with them, as well as information exchange and management, it is ultimately more cost-effective to invest in preventing errors.

In terms of quality, too many errors in the target of checks can interrupt the rhythm and concentration of the checker, leading to a decrease in detection power and a halving of the effectiveness and efficiency of the checks, except for checks that can be automated or toolized. Therefore, we place great importance on educating people, such as thorough understanding of specifications and meetings for questions and answers. We also prioritize preventive activities, such as preparation of manuals and consideration of processes to suppress the occurrence of errors. In addition, we believe it is very important to create an environment where workers can maintain their rhythm and concentration and maintain the detection power of the checker.

4. Use according to the situation and required quality.

Although it is said, as I have repeated in my blog so far, data annotation work is an unavoidable task that must be done by humans, so no matter how much effort or improvement is made, errors cannot be avoided. Therefore, it is also nonsense to pursue error prevention too much and it is necessary to use different responses depending on the quality level and situation required by the customer.

For example, even if we prioritize education in short-term projects that can be completed in one week, the project will end while we are still educating. In such cases, we will shift our focus to checking and adjust the weight and frequency of checks according to the situation. Additionally, the desired level of quality varies depending on the client. For example, some clients may say, "We are currently in the PoC phase, so we just want to feed data into the learning model. Therefore, the quality can be average. Price is the priority." Therefore, it is necessary to adapt accordingly to these situations.

In order to do this, it is necessary to determine the direction of the project and work before starting the project, but in order to do this properly, it is important to have PM management skills that can flexibly respond to the customer's needs and the quality and cost goals of the project, as well as the project management system, organization, structure, and PM development to achieve this.

5. Customer Testimonials

One time, we received a request from a client with whom we have a continuous relationship. As usual, we submitted an estimate and a data annotation sample, but they said, "You don't have to be so thorough. We plan to check it on our end, so can you lower the accuracy and reduce the frequency of checks to adjust the price?"

What is important is to fully understand the customer's requirements and determine the quality, work, and checking processes that meet those requirements. However, when working with a new customer, communication may not be smooth during negotiations, and the processes may not be fully established at that point. Therefore, in our case, we often ask during negotiations how much data annotation accuracy is required and what level of quality is expected. In addition, our estimates include information on the level of data annotation accuracy and frequency of checks depending on the situation.

6. Summary

Everything we have mentioned so far is a way that is suitable for our organization and culture. We have been making trial and error while repeatedly losing orders in negotiations due to not being able to meet the expectations of our customers and receiving feedback from them. As a result, we have been seriously addressing the balance between quality and cost and making improvements, and this is the only solution that has been derived to be the best for our customers. Therefore, we hope that it can be a reference for one of the ways to do things.

Author:

Kazuhiro Sugimoto

Annotation Department Group Manager

・Previous position: Project Manager for model line construction, quality design and improvement guidance for manufacturing lines, and multiple cross-departmental projects for improving business efficiency (lean improvement) in a Tier 1 automotive parts manufacturer.
・Current position: Engaged in directing the launch and expansion of the data annotation business, as well as the construction and improvement of the data annotation project management system, after promoting management systems such as ISO and knowledge management.
・Holds a QC certification level 1 and is a member of the Japan Association of Public Universities.