Part 2: Explaining Examples of ChatGPT Translation Proofreading Tools! Features of Translation Proofreading Tools and Accuracy Verification Results

In the previous (first) blog post, we discussed the current status and challenges of introducing generative AI in medical translation. To address these challenges, Human Science prototyped an in-house developed ChatGPT translation proofreading tool. This tool, called the "ChatGPT Translation Proofreading Tool," incorporates large language models (LLMs) like ChatGPT into the quality checking of translated texts. The goal was to provide translators with an environment where they can easily and safely leverage generative AI, thereby streamlining the verification work performed by human proofreaders.

This "ChatGPT Translation Proofreading Tool" is designed to automatically check the source text and the translated text side by side using AI. Specifically, it inspects the translation from multiple perspectives as follows.

· Detection of Omitted Translations: Checks whether any content from the original text has been omitted in the translation and points out missing important information.
· Detection of Mistranslations: Identifies errors in the translated text.
· Detection of Unnatural Expressions: Evaluates whether the style and expressions in the translation are inappropriate or unnatural.
· Detection of Formal Errors: Checks for formatting mistakes such as the presence or absence of spaces.

As described above, this tool is characterized by its ability to identify a wide range of errors, including subtle mistakes that human proofreaders tend to overlook. For example, it can automatically detect issues such as "translation omissions" and "inconsistencies in terminology notation," which is expected to have a significant impact from a quality assurance perspective.

In developing this tool, emphasis was placed on enabling safe and secure use of generative AI in translation settings. By implementing the tool within the internal environment and utilizing ChatGPT via API integration, care has been taken to ensure that translated text data does not leak externally. Translators can easily use the tool without needing to be aware of the advanced AI functions, facilitating smooth collaboration between humans and AI.

In the translation of a clinical trial protocol for a medical device, there was a case where the product name "Product X" was omitted from the translated text. The ChatGPT translation proofreading tool automatically detected this missing product name and pointed out that "important information was not translated." Omissions of product names or drug names in specialized documents are critical errors, but this is a good example where AI checks were able to cover human oversights.

In another medical paper, the gene name "PTNPN22" was a typo (error) in the original text. The translation also contained the same mistake accordingly, but the ChatGPT translation proofreading tool pointed out that "PTNPN22 is an error in the original text, and the correct term is PTPN22." In this way, the AI checks and points out errors even on the source text side. The ability to detect errors in the original text that are difficult for translators to notice is a unique strength of generative AI.

In a translation in the field of immunology, there was a case where "IL-3" should have been written, but it was translated as "IL3" in some parts (※ this was also partly due to inconsistencies in the original Japanese text itself). The ChatGPT translation proofreading tool pointed out the lack of formal consistency by stating, "Since the English version includes a hyphen, the Japanese translation should also include the hyphen." In specialized fields, unifying terminology notation is important, and AI can prevent inconsistencies caused by human error by mechanically checking them.

In summary, it was found that the ChatGPT translation proofreading tool provides indications including errors that are easily overlooked by the human eye. Especially, the ability to detect "errors at the original text stage," as in Example 2, is a major strength. This is a part that was difficult for conventional translation support tools and can be said to be a unique value of generative AI.

When the ChatGPT translation proofreading tool was tested on approximately 1,000 words of medical documents, an average of about 20 suggestions were presented by the generative AI. The types of suggestions can be broadly classified into the following three categories.

· Points regarding mistranslations and omissions (43%)
· Points regarding inappropriate/unnatural expressions (14%)
· Points regarding formal errors (such as presence or absence of spaces) (43%)

Among these, only about 17% of the points were judged as "useful" after human scrutiny. The remainder consisted of about 20% clearly incorrect points (such as marking correct parts as mistranslations) and about 60% points that were correct but concerned minor issues that did not affect translation quality (unnecessary points). In other words, at this stage, there are more "unnecessary or incorrect points" than "accurate points."

At first glance, you might think, "Most of AI's suggestions are not useful." However, what deserves attention is that within that 17% of useful suggestions, there were critical errors that could not be overlooked. Detecting errors such as omitted translations of product names, numerical mistakes, and proper noun errors—examples mentioned earlier—that would become major problems if left unaddressed was a significant advantage for quality assurance. Additionally, within the company, efforts are underway to improve the accuracy of suggestions by refining prompts (instructions to AI) and narrowing down the key points for checks.

On the other hand, there are also disadvantages that have become apparent with the introduction of AI proofreading. Since there are many unnecessary or incorrect suggestions besides the useful ones, it requires effort to sort through and select them. In the experiment, adding AI checks increased the overall translation work time, taking about 2 to 3 hours longer than usual for a 10,000-word document. This is due to the time needed for humans to review and verify the AI's suggestions. Summarizing the advantages and disadvantages, they are as follows.

In this verification by Human Science, it was concluded that the current generative AI proofreading contributes to "quality improvement" but its contribution to "work efficiency" is limited. While the quality can be raised through useful suggestions unique to generative AI, time is taken up dealing with false positives, resulting in an overall tendency for efficiency to actually decline.

However, this also means that "if the accuracy of AI's suggestions improves, it will become a very useful tool." Currently, there is still significant room for improvement in the quality of suggestions, and depending on future development, it may be possible to reduce unnecessary suggestions and increase the proportion of useful ones. In fact, Human Science is continuing further research and improvements, such as prompt optimization to elicit more accurate suggestions, based on the results of this evaluation.

In the second installment, we examined the effects and challenges of translation proofreading tools utilizing ChatGPT. In the next, third installment, we will explore the "collaboration model between humans and AI" to effectively leverage this generative AI in practical work. We plan to delve into what tasks should be entrusted to AI and what should be handled by humans, as well as discuss future prospects in detail.

Since its founding in 1985, Human Science has accumulated numerous achievements ranging from manual creation to the translation of specialized documents. In recent years, we have also built up expertise in utilizing AI technologies such as machine translation (MT) and ChatGPT, supporting our clients' operational efficiency through the provision of AI translation services.

One of our strengths is having experienced translators and proofreaders who are well-versed in each specialized field. Professionals with knowledge in areas such as medical, IT, and manufacturing appropriately evaluate and selectively apply AI-generated suggestions, ensuring the final quality. By never relying solely on AI and always performing a final check with human eyes, we deliver accurate and easy-to-read translations.

When incorporating AI into the translation process, consideration for information security is indispensable. At Human Science, we have established an internal system to protect our clients' confidential information, and we have also obtained certification for international standards such as ISMS (Information Security Management System). We support you so that you can safely enjoy the benefits of the latest AI technologies.

Thank you for reading until the end. We hope that the content provided here will offer useful insights for improving efficiency and quality in translation work. For more detailed information or consultations regarding AI utilization and translation services, please feel free to contact us.

>Medical Translation Services
>Post-Editing Services

Free Download

Five Challenges and Solutions in Medical Translation

Introducing common challenges faced by many companies in medical translation and specific solutions.
Additionally, we present multiple success stories of cost and delivery time reduction.