Tips and Insights for Annotation Work

1. What Annotation Work Is

As has been mentioned several times in previous blog posts, annotation is the work of creating data for AI to learn from. Specifically, it involves finding the target that you want the AI to recognize within the data and attaching labels to it. For example, if you want the AI to recognize a person in an image, you enclose the person in a rectangle (bounding box) so that it is identifiable as a person, and attach the tag "person" to that bounding box. The amount of training data required to improve AI recognition accuracy varies depending on the purpose but can range from several thousand to tens of thousands. Therefore, annotation work can sometimes take from several weeks to several months.

Annotation is the repetitive task of identifying subjects within data and labeling them. The work is seemingly simple, as it involves labeling based on specifications. However, it is not widely understood that progressing with annotation while ensuring quality and productivity can be surprisingly challenging.

2. The Challenges of Annotation Work

In annotation work, it is often required to handle thousands to tens of thousands of files, thoroughly reviewing each piece of data to find all targets without omission. Even if automation is implemented, human intervention at the end is unavoidable, so concentration and perseverance are necessary.

Within the data, there are edge cases that are difficult to judge based solely on the specifications, and tasks that should normally take only a few dozen seconds often end up taking several minutes. When such cases accumulate, the work pace naturally slows down. However, if you rely too much on intuition to respond quickly, the basis for your judgments becomes unstable, annotation consistency cannot be maintained, quality declines, and as a result, the AI's recognition accuracy is also affected.

Looking at it this way, annotation work requires efficient execution and quick, logical judgments to maintain accuracy, and since this work continues for weeks to months, having the right mindset and tips is necessary to sustain continuous work.

This time, rather than explaining the specific content of annotation work or from the perspective of management, I will discuss it from the worker's viewpoint, focusing on the mindset and tips necessary to continuously advance work while ensuring productivity and quality, as well as the role of the PM in supporting this.

3. Tips and Insights Necessary for Annotation Work

Make logical judgments to the extent that you can explain them

In annotation work, it is important not to settle for “just a feeling” when you don’t understand something. Not everything is explained or described in the work specifications or manuals. If everything were included in the manual, the volume of content would become enormous, and considering factors such as searchability and the labor required to write it, this is not realistic. Therefore, it is common practice to limit the manual mainly to the basic concepts and representative examples of the objects to be annotated. Accordingly, by understanding and applying the basics and representative examples described, it is necessary to make accurate judgments for various cases that are not explicitly written.

For that reason, it is necessary to have a basis for judgment that allows you to properly explain "why you performed (or did not perform) the annotation in that way." Even if it is a case of "just feeling that way," by logically verbalizing that "just feeling," it becomes possible to perform consistent annotations without deviation. If this cannot be explained, what was annotated as "white" yesterday might be annotated as "black" today. Needless to say, such inconsistencies in judgment affect the quality of the training data.

Take appropriate breaks to reset and pause to objectively assess the results

This is something anyone can imagine, but in image annotation, you need to repeatedly scan every corner of the image to find the target and label it. In some cases, precision down to a few pixels may be required. For text, you must carefully read every sentence without missing anything and accurately label the necessary parts. To continue this work for several hours a day, it is essential to take appropriate breaks without overexerting yourself, reset, and pause.

When you continue working for a long time, no matter how logical your judgment ability is, your senses can become numb and your judgment may become biased. Therefore, it is important to take appropriate breaks to reset, pause for a moment, and objectively assess whether your annotation results are skewed in one direction or another.

Take appropriate breaks to reset and pause to objectively assess the results

This is something anyone can imagine, but in image annotation, you need to repeatedly scan every corner of the image to find the target and label it. In some cases, precision down to a few pixels may be required. For text, you must carefully read every sentence without missing anything and accurately label the necessary parts. To continue this work for several hours a day, it is essential to take appropriate breaks without overexerting yourself, reset, and pause.

When you continue working for a long time, no matter how logical your judgment ability is, your senses can become numb and your judgment may become biased. Therefore, it is important to take appropriate breaks to reset, pause for a moment, and objectively assess whether your annotation results are skewed in one direction or another.

Quickly Decide the Next Action

Even if you have logical thinking skills, "quick decision-making" is indispensable for annotation. As mentioned earlier, the data you encounter in annotation often includes edge cases that are not described in the specifications and have no other examples, causing frequent uncertainty in judgment. If you overthink in such situations, time will pass in the blink of an eye.

Annotation requires a large amount of work. For example, in the case of bounding boxes, except for special cases, one annotation usually needs to be completed in about several tens of seconds. If you stop for several minutes every time you encounter an edge case, productivity will quickly decline. When struggling with an edge case, it is important to keep the time spent "worrying" to a minimum and quickly move on to the next steps, such as "asking questions" or "thinking it through and drawing your own conclusion."

Respond flexibly according to the situation

It is also important to have flexibility without being overly fixated on something. A simple example is that humans make intuitive and instantaneous judgments in all kinds of situations. For instance, when deciding whether an animal is a dog or a cat, we do not judge based on "if the ears have this shape, then it’s a dog or a cat." Instead, based on past experience and the information seen, the brain comprehensively judges using various parameters. If you demand too much theoretical background or justification for such judgments, or hold onto your own particular preferences, you may overthink and arrive at incorrect answers, or embark on a distant journey in search of answers that do not exist.

It may seem contradictory to the logical judgment mentioned earlier, but being able to distinguish between "where logical judgment should be applied" and "where it is pointless to think further and judgment should be based on experience as in the example above" also requires, in a sense, a "logical" decision.

Avoid being overly meticulous (avoid excessive quality)

Pursuing quality and taking responsibility for your own deliverables by working carefully is extremely important in any job, not just annotation. However, whether consciously or unconsciously, doing things too meticulously naturally has a significant impact on productivity. For example, in semantic segmentation of trees, even when there are samples or instructions regarding the accuracy of the coloring, it is common to find cases where "unintentionally, the leaf tips were colored more finely than necessary." When this happens, not only productivity but also variations in quality among different workers become problematic. As mentioned earlier, it is important to pause and objectively confirm "how much accuracy is required" and "whether the requirements are being met."

Carefully read the work specifications and manuals

This may not be limited to annotation work, but thoroughly reading and understanding the contents of work specifications, manuals, and instructions, and proceeding with the work accordingly, is fundamental to everything. Our company has worked with hundreds of annotators so far, but many proceed with their work without carefully reading the work specifications, manuals, or instructions. Especially in annotation, instructions on how to handle frequently occurring edge cases and exceptions often arise. If these materials are not properly read and confirmed, incorrect annotations will be made, which naturally affects quality as well.

4. The Role of the PM

When performing annotation, the knowledge and tips mentioned so far are necessary, but not everyone understands and can put them into practice from the beginning. In many cases, there are some weaknesses, such as "being excellent at detailed work and logical thinking but unable to make quick decisions," or "tending to prioritize productivity, resulting in inconsistent quality."

The PM plays a crucial role here. While other companies may have positions that fulfill this role aside from the PM, our PM's responsibilities include not only managing the quality of annotations, productivity, and work progress, but also guiding workers towards the ideal direction as previously mentioned.

It is ideal if the worker can resolve issues on their own, but there are often times when they are struggling and cannot see a way to solve the problem. By quickly identifying such weaknesses in daily management and conducting one-on-one meetings with workers who have these weaknesses, we aim to address their areas of difficulty. Additionally, sharing the know-how possessed by excellent workers with the entire annotation team is also one of the roles of the PM.

5. Summary

No one can do everything perfectly from the start as mentioned so far. It is important for the annotators, who are the workers, and the PM to work together to elevate the quality to the desired level, and this needs to be pursued by the entire team. To establish teamwork, it goes without saying that a humble and flexible attitude is essential, where we respect each other and openly accept advice and suggestions.

6. Human Science Annotation, LLM RAG Data Structuring Agency Service

Over 48 million pieces of training data created

At Human Science, we are involved in AI model development projects across various industries, starting with natural language processing and extending to medical support, automotive, IT, manufacturing, and construction, just to name a few. Through direct business with many companies, including GAFAM, we have provided over 48 million pieces of high-quality training data. No matter the industry, our team of 150 annotators is prepared to accommodate various types of annotation, data labeling, and data structuring, from small-scale projects to big long-term projects.

Resource management without crowdsourcing

At Human Science, we do not use crowdsourcing. Instead, projects are handled by personnel who are contracted with us directly. Based on a solid understanding of each member's practical experience and their evaluations from previous projects, we form teams that can deliver maximum performance.

Support for not just annotation, but the creation and structuring of generative AI LLM datasets

In addition to labeling for data organization and annotation for identification-based AI systems, Human Science also supports the structuring of document data for generative AI and LLM RAG construction. Since our founding, our primary business has been in manual production, and we can leverage our deep knowledge of various document structures to provide you with optimal solutions.

Secure room available on-site

Within our Shinjuku office at Human Science, we have secure rooms that meet ISMS standards. Therefore, we can guarantee security, even for projects that include highly confidential data. We consider the preservation of confidentiality to be extremely important for all projects. When working remotely as well, our information security management system has received high praise from clients, because not only do we implement hardware measures, we continuously provide security training to our personnel.