Ensuring the accuracy and reliability of machine learning systems by outsourcing fast and accurate annotation work

Interview Cooperation: Masato Indo, Sumitomo Heavy Industries, Ltd. Technical Research Institute

Overview of Sumitomo Heavy Industries, Ltd.

・Established: November 1, 1934

・Capital: 30.87 billion yen (as of December 31, 2022)

Number of Employees: Consolidated: 25,211 people (as of December 31, 2022)

Business Overview: As a comprehensive machinery manufacturer, we provide integrated solutions ranging from general industrial machinery to cutting-edge precision machinery, as well as construction machinery, ships, and environmental and plant equipment.

Services Utilized

- Annotation Services

Overview of Sumitomo Heavy Industries, Ltd.

HS: First, please tell us about your company's overall business.

Mr. Indou: Yes. Our company is a comprehensive machinery manufacturer that covers everything from general industrial machinery to cutting-edge precision machinery, as well as construction machinery, ships, and environmental and plant equipment. We are involved in various fields from nanotechnology to large structures, utilizing technology that 'moves and controls' to contribute in various areas of social infrastructure.

HS: What role does the technical research institute, to which Mr. Indō belongs, play among comprehensive machinery manufacturers? Also, is there anything in particular that you are focusing on?

Mr. Indō: We focus on research and development based on future prospects by paying attention to the market, customers, and megatrends. Our areas of focus are "Environment & Energy" and "Automation & Digitalization," which are also global trends, and we are working diligently on these topics.

HS: What kind of mission does Mr. Indō have among them?

Mr. Indou: In order to accelerate our efforts in automation and digitalization, we have developed a common platform called "SHICuTe" (Note 1) that enables functional development across our product range. Our mission is to analyze and utilize various operational data collected by this system to meet our customers' diverse needs and develop functions that support their operations.

Challenges Before Implementation

HS: Please tell us what prompted you to outsource the annotation.

Mr. Indō: As explained earlier, our company is currently collecting various data and beginning to consider value creation. Since operational data continues to increase constantly, efficient processing using machine learning is essential. To create this machine learning system, annotation (meaning assignment) of the data is necessary.
The accuracy and reliability of the machine learning system are greatly influenced by the precision of the meaning assignment and the amount of data, so the challenge has been how to perform accurate annotations as quickly as possible. Therefore, we have requested your company, which has experience in annotating data for machine learning.

HS: What specific challenges did you have before making your request?

Mr. Indou: Since the system is still in the development phase, I was doing the annotations myself, but I just couldn't keep up with the speed alone. Also, while annotation is often seen as a simple task, as mentioned earlier, the accuracy directly affects the precision of the machine learning being created, making it difficult to establish a system for checks and training within the company.

HS: Given the volume of work and the specialization of the business, there was an intention to speed things up by using external services.

Reasons for Choosing Human Science

HS: What is the process for selecting a vendor, and what were the reasons for choosing our company?

Mr. Indou: First, we compiled a list of candidates based on information from the internet and exhibitions, and reached out to multiple companies based on their track records. When I received an explanation from your company, the concerns regarding the annotation work, such as man-hours, quality, costs, and structure, matched my own feelings, which was the deciding factor for me.

HS: I see, the importance of the annotations in our materials was the same as what you felt, which is why you wanted to request our services. Thank you.

HS: Did you have any concerns or worries when requesting our services?

Mr. Indō: As mentioned earlier, there was an explanation regarding quality and processes, so I wasn't worried about that. However, since this was my first experience with outsourcing annotations, I had concerns about not having a clear understanding of the cost aspect.

HS: Are you unsure about how much budget you should prepare?

Mr. Indou: That's right. I wasn't sure how much budget to prepare for the required amount of data, but your company provided multiple proposals based on the costs of training and managing annotators, the unit price per data implementation, and the number of checks, which I appreciated as it allowed me to consider within the set budget and objectives. Additionally, the issue of data confidentiality was a concern, and the ability to handle it domestically was also one of the selection points.

HS: Thank you.

Implementation Effects and Evaluation of Human Science Co., Ltd.

HS: I would like to ask about the process from the start to the completion of the work. Please let me know if you felt anything during the flow.

Mr. Indou: At the kickoff, we were able to discuss the communication plan and communication methods, allowing both parties to start in a manageable way. During the process, when there were ambiguous data that did not align with the annotation definition document, and regarding items that tend to vary among annotators, I believe we were able to proceed smoothly thanks to the candid feedback we received, which allowed us to review the definitions.

HS: Thank you. In point annotation of images that project three-dimensional space onto two dimensions (indicating the location of definitions), depth must be considered, which often leads to different judgments among individuals, making it difficult to align our perceptions.

Mr. Indou: I was able to confidently entrust the work to you because you were keenly aware that even minor discrepancies and differences in definitions can impact the accuracy of machine learning.

HS: Thank you.

HS: Could you please let us know if there are any areas for improvement regarding our service level?

Mr. Indou: That's right. For general cases such as image segmentation and classification, it would be appreciated if you could introduce annotation tools or suggest conversions to data formats suitable for annotation, leveraging your know-how to propose cost reductions and speed improvements.

 

 

*1 Introduction of the development and utilization examples of the Sumitomo Heavy Industries Group's common platform "SHICuTe" in construction sites (Press Release)
https://www.shi.co.jp/info/2021/6kgpsq000000li2h.html