Challenges and Solutions in Medical AI Annotation - Utilizing Outsourcing Vendors with Expertise

1. What is Annotation Work

When doctors diagnose diseases based on medical images such as endoscopies and CT scans, it is expected that AI can assist in the diagnosis, helping to prevent medical errors such as missed lesions. To develop such AI, it is necessary to create training data for the AI to learn from. This process of creating data is called annotation.

Image annotation involves marking lesions and body tissues (referred to as objects or targets) within images using specialized tools. Common methods include enclosing objects with rectangles called bounding boxes or tracing the contours of objects in a process known as segmentation. Such annotations are performed on a large number of images (ranging from thousands to tens of thousands).

2. Challenges of Medical Annotation Work

When developing AI for the medical field, it can be broadly divided into cases where medical institutions such as hospitals and universities take the lead, and cases where companies such as medical manufacturers conduct AI development. Let's take a look at the common and individual challenges related to annotation work.

Annotation work requiring expertise

One common challenge is the difficulty in securing workers due to the specialized nature of annotation work.

Annotation requires workers who can accurately discover and identify the objects mentioned earlier. In the case of annotating common objects like cars and people, it is relatively easy to secure workers since a typical person can identify cars. However, this is not the case with medical images. This is because annotators need the ability to correctly discern lesions and body tissues, which is not something that everyone can do easily.

That said, it is not easy to request annotation work from doctors. As mentioned earlier, annotation needs to be performed on a large number of images. Depending on the case, it may require several hours a day, and a substantial amount of time ranging from weeks to months dedicated to annotation work. For doctors to undertake this in addition to their primary duties means doing it outside of regular working hours or allocating part of their working hours to it, which is not something that can be done easily. Furthermore, if doctors perform the work themselves, the costs can become higher compared to general workers.

Furthermore, even if it becomes possible for doctors to perform annotation tasks, management must also be conducted to oversee quality and progress. It is preferable to have a dedicated PM for management, but if the medical institution handles it themselves, it will result in doctors managing the process, which can interfere with their primary duties and ultimately incur additional costs.

The Challenges of Partnering with Medical Institutions

When medical manufacturers engage in AI development, they often form partnerships with medical institutions. By strengthening collaboration with the medical field, they can incorporate opinions and knowledge from a physician's perspective, which is difficult to obtain solely from the manufacturer. Additionally, when acquiring medical images that are not open data, it becomes necessary to partner with medical institutions. In tasks that require advanced expertise and judgment, obtaining endorsement or verification from expert physicians is also essential.

That said, it is not hard to imagine that forming a partnership involves a significant amount of time spent on complex procedures, such as pre-agreements related to contracts. Even if a partnership is established, it is often the case that things do not go as smoothly as expected. Since there are two different organizations, a medical manufacturer and a medical institution, their organizational cultures and ways of working will naturally differ, which may lead to situations where things do not progress as desired.

For example, let's consider how to involve doctors in the series of processes for creating training data (from requirements definition to data completion). There are time constraints due to the doctors' primary occupations, and it is difficult to place doctors under the direction and management of the manufacturers, so various adjustments are necessary. Additionally, the level of expertise and difficulty required varies for each development project, so the extent to which doctors are involved in the work process ultimately becomes a trial-and-error situation for each development project.

We often receive inquiries from medical manufacturers consulting with us regarding concerns about such partnerships.

3. Benefits of Outsourcing Annotations

Have you realized that it is difficult for medical institutions and medical manufacturers engaged in AI development to handle annotation in-house? Nevertheless, the demand for medical AI development is higher than ever. Consequently, it is necessary to proceed with annotation at a faster pace. Therefore, outsourcing to vendors strong in the medical field is one option. While there may be concerns about the difficulty of outsourcing medical annotation, I will explain the benefits and feasibility that can be gained through outsourcing.

●Cost reduction for general workers

It is essential to request a vendor with a proven track record and expertise in high-difficulty and specialized fields, such as medical annotations. However, for tasks like identifying lesions such as tumors or bleeding, depending on the difficulty of the work (for example, if identifying the lesion is relatively easy), it may be possible for individuals who are not doctors to perform annotation tasks. In such cases, by actually carrying out the work and incorporating feedback from doctors and experts, it becomes possible to absorb know-how and appropriately transfer skills, allowing general workers to perform annotations as well.

If you are concerned about completing the entire process from annotation work to checking and acceptance with general workers, there is also the option of physician supervision. For example, general workers can perform the annotation tasks, while physicians can handle the checking tasks. This method allows for high-quality annotations without significantly burdening the physicians' primary duties.

If it can be done by general workers, it is possible to significantly reduce costs compared to work done by doctors or in-house production. This is one of the major advantages of outsourcing annotation work.

●Management by a dedicated PM

One of the benefits is that you can entrust the management tasks of annotation in general, such as building an annotation team structure by gathering and assigning the necessary workers, managing communication such as responding to inquiries during work, and overseeing quality and progress, to a project manager experienced in annotation projects. This eliminates the need for management by doctors or development engineers, making it possible to reduce management costs.

4. When consulting with an annotation vendor

When consulting with an outsourcing vendor specializing in annotation, consider the following points.

●Experience with the intended annotation

Most annotation vendors will likely have experience with image annotation. However, there may be some vendors who have less experience specifically with medical imaging. An experienced annotation vendor is expected to have the management and resource structure in place to handle the intended annotation.

●Annotation work involving doctors, such as supervision by physicians, is possible

There may be cases where supervision or annotation by a physician is required. When consulting with a vendor, let's confirm whether they have resources or can establish a system to accommodate such requests.

●Able to meet high security requirements for on-site work and more

Many medical-related data require high security requirements for data handling, such as anonymized processed information. In cases where vendors secure cloud workers to proceed with work, it inevitably becomes remote work, and sharing or storing data with workers over the network or in cloud environments is not allowed. There may be cases where it is necessary to work in a closed environment, and high-security requirements such as not being able to take data outside are not met. Let's check whether the vendor can meet security requirements, such as having a dedicated security room or being able to accommodate on-site work at designated locations.

5. Medical Annotation Services in Human Sciences

We have realized that there are various challenges in medical annotations, as well as the increasing needs for medical AI development through consultations from our clients. Therefore, we have strengthened our medical annotation services by leveraging our past experiences.

●Extensive annotation experience in medical imaging

Our company has extensive experience in medical image annotation that requires skill transfer, such as surgical images and MRI images, which are of high difficulty and specialization. In addition to experienced project managers in medical image annotation projects, we also have many skilled workers, ensuring high-quality annotation even for projects that are complex, specialized, and require skill transfer.

●Support for physician supervision and annotations by physicians

It can be concerning to have only general workers perform all tasks. In such cases, we receive requests for supervision by a physician for certain check tasks. To meet these requests, we have further strengthened our physician supervision system, allowing us to handle more complex annotations. Additionally, even if you request annotations by physicians instead of general workers, our project managers will provide comprehensive management services, ensuring resource allocation, quality, and progress management.

●Resource management without using crowdsourcing

At Human Science, we do not use crowdsourcing; instead, we advance projects with personnel directly contracted by our company. We form teams that can deliver maximum performance based on a solid understanding of each member's practical experience and their evaluations from previous projects.

●Equipped with a security room in-house

At Human Science, we have a security room that meets ISMS standards within our Shinjuku office. Therefore, we can ensure security even for projects that handle highly confidential data, such as those in the medical field. We consider the protection of confidentiality to be extremely important for all projects. Even for remote projects, we not only implement hardware measures but also continuously provide security training to our staff, which has earned us high praise from our clients for our information security management system.

Supports not only annotation but also the creation and structuring of generative AI LLM datasets.

In addition to labeling and annotation for identification systems for data organization, we also support the structuring of document data for the construction of generative AI and LLM RAG. Since our founding, we have been engaged in manual production as a primary business and service, leveraging our unique know-how gained from a deep understanding of various document structures to provide optimal solutions.