For the medical industry
For the automotive industry
For the IT industry
Spin-off Blog Project
――Annotation Supporting AI in the DX Era. The Reality of Analog Field
Good Teacher Data: Building Better Teachers
〜Understanding the Communication Required in the Field〜
We have been publishing various blogs on annotation and AI so far. In those blogs, we have mainly provided general knowledge and know-how. Annotation work may seem simple at first glance when you try to put its content into words, but it is a "task that cannot be avoided and involves a lot of ambiguity," so it inevitably involves a lot of interaction between people. Therefore, many things that cannot be solved with clean logic overflowing in the world happen in a somewhat gritty manner, and in order to ensure quality and productivity, various experiences and know-how are actually required.
Therefore, we believe that it is useful to know specifically about the problems that occur in the actual annotation field and their corresponding measures as hints for successful annotation.
In our field, what actually happens and what specific responses and countermeasures are taken. Unlike regular blogs, we would like to inform you about the reality of the field in a spin-off blog project titled "The Realities of the Analog Field that Supports AI in the DX Era," including our unique features and commitments.
>>Past Published Blogs (Some)
7 Tips to Successfully Lead Annotations
- Table of Contents
1. Can everything be conveyed in specifications and work instructions?
The quality of the teacher data is influenced by the annotator's work quality. Of course, annotation is based on requirements definition, so it is important to first have the requirements determined. However, even if the requirements are determined, a specification document based on them is prepared, and a proper work explanation is provided, it is still not possible to pass through a major trap.
Annotation does not require knowledge or qualifications, and as mentioned earlier, there are misunderstandings because it is a simple task when put into words. For example, when it comes to distinguishing dog breeds, people unconsciously make judgments based on their past experiences and senses. I don't think many people think and judge theoretically like "If this is like this, then it's a Chihuahua." It's a judgment based on intuition... In annotation work, there is inevitably a reliance on human judgment based on experience and intuition. In addition, because a large amount of data is handled, there are many exceptions that cannot be determined by specifications alone. There are big traps hidden in them... (It is not realistic to check all the data in advance and incorporate exceptional cases into the specifications, and it is not practical to describe the characteristics of dog breeds in detail in the specifications, as it would result in a huge amount of text and become unusable.)
No matter how carefully people work, there will always be differences in judgment. I have also experienced differences in judgment and perception while participating in various projects as an annotator. In order to create good training data, it is crucial for the annotators themselves to be good teachers, as this is the most important factor in ensuring quality. Therefore, it is necessary to manage people properly.
There are various types of people in the world, as you would expect. Some people tend to prioritize speed, while others become overly cautious. Additionally, some people struggle with communication, such as asking questions, which can also affect the quality of annotations based on individual personalities. Furthermore, annotations require careful attention to detail and can be a never-ending task. As a result, over time, one's sense of judgment may become numb, leading to inconsistencies and careless mistakes.
In addition to explaining the work such as maintenance of specifications and rules, the PM can prevent many judgment errors by conveying the key points and important points. However, until the work is completed, it is important to keep an eye on the condition and quality of the workers and ensure that the annotators can work smoothly while maintaining quality. In other words, education and support to become a good teacher are important.
2. Education and support through communication
As mentioned earlier, it is possible to improve understanding by creating easy-to-understand specifications and creating supplementary materials and performing maintenance as needed. However, sharing documents alone results in one-way information transmission and does not provide assurance of mutual understanding. In the end, it turns out that the way of communication was poor and the understanding was not obtained... Starting over from scratch... (sigh). This will result in increased costs and time. Depending on the scale and difficulty of the annotation, taking these into consideration, our company has been implementing education and support that focuses on communication according to the situation.
However, there are various types of communication. Group meetings? Contact through chat tools? Emails? Among these various options, what is the best approach? Based on our experience, although it may require more effort, the most effective form of communication is one-on-one meetings.
There are many situations in annotation work where communication is required, such as clarifying the ambiguity specific to annotation or confirming the worker's understanding of the specifications. In such cases, 1-on-1 meetings are effective. Complex nuances that cannot be conveyed through text can be communicated directly through conversation, and using screen sharing can make it even easier to understand. Above all, being able to communicate face-to-face (or through a display in the case of remote communication) is the best. In group meetings, it is easier to convey individualized content that may be difficult otherwise, and annotators can also easily seek advice or express their opinions without worrying about their surroundings.
3. Implementation of 1on1
This is about a natural language processing annotation project. In this project, each annotator receives reviews from clients, and if they continue to have poor performance, they are unable to continue with the project. It is quite strict.
Annotator A has been barely passing for several months, but at one point, they finally fell below the passing score. If we don't make improvements and recover from this, we might end up in a situation where there's no way out. Continuing to barely pass suggests that they may not have a deep understanding of the specifications. After checking the feedback from the client reviewer, it was apparent that A has been annotating in a way that deviates from the explanations in the specifications. Therefore, I felt the need for A to thoroughly understand the specifications, but no matter how much time we have for the chat Q&A sessions we've been doing so far, it's not enough. That's why I decided to have a one-on-one session.
"Your score has dropped. Shall we have a private lesson?" I immediately emailed A-san with this feedback, and right after that, a private message appeared in the chat. "I messed up. Please, go ahead..."
Good things come to those who hustle, so let's get started with the 1on1 right away. We will review each feedback one by one and explain why it is incorrect by comparing it with the specification document. Then, "Oh! So that's the interpretation of this part in the specification document! I've been misunderstanding it all this time...". "...Oh?" (I will regain my composure and continue explaining...). We also went through some examples that are easy to make mistakes and spent about an hour reviewing the feedback together to deepen our understanding.
"If you have any concerns during the work, please refer to the previous feedback or review the specifications. Of course, if you are still unsure, please ask questions in the chat. If it is difficult to explain in writing, let's have a meeting directly," I conveyed.
The next day, there were more questions, possibly due to the effectiveness of the advice, but there didn't seem to be any mistakes in the thinking. A few days later, A-san, who returned to me as the PM, got a passing score. I will send feedback to A-san. "That's great! The number of mistakes has decreased significantly and it's a good score. You understand better than me," I messaged, feeling relieved. No, I reminded myself to be thorough in the future and sent a message to another annotator who scored below the passing score. "You scored below the passing score. Shall we have a private lesson?"
4. Summary
We discussed the importance of educating annotators to create teacher data for quality assurance, and one of the methods, communication, with examples. Among them, 1-on-1 meetings are very effective because they allow for education, advice, and course correction for specific annotators. By actually meeting face-to-face, we can understand if they have understood through their way of speaking and gestures, and we can also grasp their personality. This makes it easier to approach them afterwards, such as "How can we give feedback to this person in a way that is easy to understand?"
To create good teacher data, we cultivate annotators who are teachers. Annotation is a manual process. Communication is an important element that shapes the foundation. Some may think that annotation is easy if you just follow the specifications in the document without putting in much effort. However, in the actual work environment, there are often challenges that cannot be easily overcome. To overcome these problems and create teacher data of higher quality, and to reduce unnecessary costs that may arise from modifications, we are committed to managing people. In addition to ensuring quality, we also value creating a comfortable working environment.
Depending on the scale and continuity of the annotation, this method may not be correct. However, in annotation tasks that often involve unfamiliar data and rules, it is important to provide appropriate education and support throughout the work period to ensure quality. In order to do so, it is necessary to not only provide information in text form, but also to directly communicate with the person (even if it is through a display in remote situations) in order to achieve consensus and accumulate know-how that can only be achieved through face-to-face interaction. I have learned this from my previous experiences, which is somewhat obvious.
These tasks may be laborious and may not be considered smart as a way of working. However, in terms of on-site perception, we believe that this is what annotation is all about. At our company, we are willing to get our hands dirty and are determined to continue assisting you with a proactive attitude.
Author:
Kitada Manabu
Annotation Group Project Manager
Since the establishment of our Annotation Group, we have been focusing on natural language processing. We have been responsible for team building and project management for large-scale projects, as well as annotation specification development for PoC projects and consulting for scalability. Currently, in addition to being a project manager for image and video annotation and natural language annotation, I am also engaged in promotion activities such as being a seminar instructor for annotation and writing blogs.
Related Blogs
Return to the top of the page
+81 35-321-3111[Tokyo]
Contact Us
Request for Information
TOP
SITEMENU