Comparing the Translation Abilities of ChatGPT, Gemini, and Claude

1. Machine Translation

First, let's compare the translation capabilities of these three AIs as machine translation engines.

1-1. Raw translation

I asked the AI tools to translate English text with a simple prompt of "日本語に翻訳して。訳文のみを出力して。" (Translate into Japanese. Output only the translated text.) The translations generated by each AI are as follows. Most notably, the translations provided by both ChatGPT and Gemini for "Leading the pack" and "know a thing or two," were literal translations that are unnatural in Japanese. In contrast, Claude's translations were localized to convey the spirit of the meaning in natural Japanese.

Source	Leading the pack was AAA Media Group, followed by BBB Video, and CCC Technologies. DDD Brothers and EEE Wire came in fourth and fifth, respectively. AAA is a highly successful fully remote company with hundreds of employees, so we know a thing or two about building a remote work environment.
ChatGPT	パックの先頭に立っていたのはAAAメディアグループで、その後にBBBビデオ、CCCテクノロジーズが続きました。DDDブラザーズとEEEワイヤーはそれぞれ4位と5位に入りました。AAAは数百人の従業員を抱える非常に成功した完全リモート会社なので、リモートワーク環境の構築については一つや二つ知っています。
Gemini	先頭集団を率いるのはAAAメディアグループで、BBBビデオ、CCCテクノロジーズがそれに続きました。DDDブラザーズとEEEワイヤーはそれぞれ4位と5位に入りました。 AAAは数百人の従業員を擁する非常に成功した完全リモート企業であるため、リモートワーク環境の構築について1つか2つのことを知っています。
Claude	AAAメディアグループが首位で、BBBビデオ、CCCテクノロジーズと続きました。DDDブラザーズとEEEワイヤーがそれぞれ4位と5位でした。AAAは数百人の従業員を擁する非常に成功したフルリモート企業であり、リモートワーク環境の構築について私たちは多くのことを知っています。

1-2. Use natural Japanese

Next, I tried the prompt, "自然な日本語に翻訳して。訳文のみを出力して。" (Translate into natural Japanese. Output only the translated text.) While ChatGPT's translation remained too literal, Gemini changed its output to a more natural expression. Claude further improved the translation of "know a thing or two" from "多くのことを知っています (I know a lot)" to "知識と経験があります (I have knowledge and experience)," refining the translation to fit the context appropriately.

ChatGPT	パックの先頭を走るのはAAAメディアグループで、その後にBBBビデオ、CCCテクノロジーズが続きます。DDDブラザーズとEEEワイヤーはそれぞれ4位と5位に入りました。AAAは数百人の従業員を抱える大成功を収めた完全リモート会社なので、リモートワーク環境の構築については一つや二つ知っています。
Gemini	首位はAAAメディアグループ、次いでBBBビデオ、CCCテクノロジーズが続きました。DDDブラザーズとEEEワイヤーはそれぞれ4位と5位に入りました。 AAAは数百人の従業員を擁する非常に成功した完全リモート企業であるため、リモートワーク環境の構築について熟知しています。
Claude	AAAメディアグループが先頭に立ち、BBBビデオとCCCテクノロジーズがそれに続きました。DDDブラザーズとEEEワイヤーがそれぞれ4位と5位に入りました。AAAは数百人の従業員を擁する非常に成功した完全リモートの企業であり、リモートワーク環境の構築については私たちも知識と経験があります。

1-3. Use semi-formal style (de aru style)

In some cases, documents may need to be translated in a semi-formal style of Japanese (de/aru style). Therefore, I tried a prompt that said, "そこで常体（だ・である調）で日本語に翻訳して。訳文のみを出力して。" (Translate into polite Japanese (da/de aru style). Output only the translated text.) All AI text generators were able to translate in this semi-formal style.

ChatGPT	パックの先頭を走るのはAAAメディアグループで、その後にBBBビデオ、CCCテクノロジーズが続く。DDDブラザーズとEEEワイヤーはそれぞれ四位と五位に入った。AAAは数百人の従業員を抱える大成功を収めた完全リモート会社であるため、リモートワーク環境の構築については一つや二つ知っている。
Gemini	先頭を行くのはAAAメディアグループ、続いてBBBビデオ、CCCテクノロジーズと続く。DDDブラザーズとEEEワイヤーはそれぞれ4位と5位につけた。AAAは数百人の従業員を抱える非常に成功した完全リモート企業であるため、リモートワーク環境の構築については熟知している。
Claude	AAAメディアグループが先頭に立ち、BBBビデオとCCCテクノロジーズがそれに続いた。DDDブラザーズとEEEワイヤーがそれぞれ4位と5位だった。AAAは数百人の従業員を擁する非常に成功したフルリモート企業であり、リモートワーク環境の構築については私たちも知識がある。

1-4. Keep the company name in English and add "社" (Inc.)

At times, translation instructions or style guides may specify that company names should remain in English with the addition of '社' (Inc.). To test this, I tried the prompt, "日本語に翻訳して。訳文のみを出力して。企業名は英語のままとし「社」を付けること。" (Translate into Japanese. Output only the translated text. Keep the company names in English and add '社' to them.) ChatGPT correctly kept the company names in English and added '社' as instructed, except for the last sentence, where '社' was omitted for 'AAA.' Gemini left all company names in English and properly added '社' to all the companies. Claude ignored the instruction to keep the names in English, but it correctly added '社' to all company names. Among the three generative AIs, Gemini was the most faithful to the instructions.

ChatGPT	パックのリーダーはAAA Media Group社で、その後にBBB Video社、そしてCCC Technologies社が続きました。DDD Brothers社とEEE Wire社はそれぞれ四位と五位に入りました。AAAは数百人の従業員を抱える非常に成功した完全リモートの会社なので、リモートワーク環境の構築については一つや二つ知っています。
Gemini	首位はAAA Media Group社、続いてBBB Video社、CCC Technologies社となった。DDD Brothers社とEEE Wire社はそれぞれ4位と5位にランクインした。AAA社は数百人の従業員を擁する非常に成功した完全リモート企業であるため、リモートワーク環境の構築については熟知している。
Claude	AAAメディアグループ社が首位に立ち、BBBビデオ社とCCCテクノロジーズ社が続きました。DDDブラザーズ社とEEEワイヤー社がそれぞれ4位と5位に入りました。AAA社は数百人の従業員を擁する非常に成功したフルリモート企業であり、リモートワーク環境の構築について私たちは多くのことを知っています。

2. Automatic Revision of Fuzzy Matches in the Translation Memory

"Translation memory" is a database that stores the bilingual data of sentences that were translated in the past. By integrating them with computer-assisted translation (CAT) tools, they help improve the quality and efficiency of translation work. When translating similar texts, such as user manuals and other documents that are frequently revised, using a translation memory can significantly reduce translation time. For more details about CAT tools, check out the following post.

What Are CAT Tools? Introducing the Benefits and Drawbacks

"Fuzzy match" refers to cases where the previous source text in the translation memory is similar but does not completely match the new source text being translated. It is also called a partial match. When a document is revised, small parts of the past source text may be changed. For example, in the table below, "DeepL、" has been added to the revised source text. In such cases, the past translation from the translation memory is retrieved, and any changes are manually reflected, using the old translation as a base.

Previous source text in translation memory	MTrans for Officeとは、Microsoft Office上で、セキュアかつ二次利用の恐れのないGoogle Cloud Translation API等のMTサービスを使って、文書を直接翻訳できるOfficeアドイン及びMTサブスクリプションサービスです。
Revised source text	MTrans for Officeとは、Microsoft Office上で、セキュアかつ二次利用の恐れのないDeepL、Google Cloud Translation API等のMTサービスを使って、文書を直接翻訳できるOfficeアドイン及びMTサブスクリプションサービスです。
Previous translation in translation memory	MTrans for Office is an Office add-in and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as the Google Cloud Translation API, which is secure and has no risk of secondary use.

2-1. Automatic revision of fuzzy matches in the translation memory

For all of the generative AIs that we tested, the word "DeepL" was accurately added to the translation to reflect the revision of the source text. However, because it is more natural in English to conjoin two items in a list with "and," the modifications by Gemini and Claude are more appropriate. As a sidenote, because the original version referred to "Google Cloud Translation API" in the singular, the verbs "is" and "has" were used, but now that the subject has become plural with "DeepL and Google Cloud Translation API," the verbs have been changed to "are" and "have." Such changes are often overlooked when revising translations manually, but all three generative AIs have appropriately modified the text to ensure proper English grammar.

Revised source text	MTrans for Officeとは、Microsoft Office上で、セキュアかつ二次利用の恐れのないDeepL、Google Cloud Translation API等のMTサービスを使って、文書を直接翻訳できるOfficeアドイン及びMTサブスクリプションサービスです。
Previous translation in translation memory	MTrans for Office is an Office add-in and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as the Google Cloud Translation API, which is secure and has no risk of secondary use.
ChatGPT	MTrans for Office is an Office add-in and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as DeepL, Google Cloud Translation API, which are secure and have no risk of secondary use.
Gemini	MTrans for Office is an Office add-in and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as DeepL and the Google Cloud Translation API, which are secure and have no risk of secondary use.
Claude	MTrans for Office is an Office add-in and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as DeepL and Google Cloud Translation API, which are secure and have no risk of secondary use.

2-2. Automatic revision of fuzzy matches in the translation memory + glossary specification

Product names and service names may change. So how might one automatically reflect new terms while utilizing past translation memories? In the following example, we specified that the term "Office add-in" should be written as "Office ADD-IN" and "DeepL" as "DeepLTEST," and automatic revision of fuzzy matches in the translation memory was performed. For all of the generative AIs, the new terms were correctly reflected. However, in Claude, the verbs reverted to singular "is" and "has." It would be necessary to manually correct them to "are" and "have."

Previous translation in translation memory	MTrans for Office is an Office add-in and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as the Google Cloud Translation API, which is secure and has no risk of secondary use.
ChatGPT	MTrans for Office is an Office ADD-IN and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as DeepLTEST, Google Cloud Translation API, which are secure and have no risk of secondary use.
Gemini	MTrans for Office is an Office ADD-IN and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as DeepLTEST and the Google Cloud Translation API, which are secure and have no risk of secondary use.
Claude	MTrans for Office is an Office ADD-IN and MT subscription service that allows you to translate documents directly on Microsoft Office using MT services such as DeepLTEST, Google Cloud Translation API, which is secure and has no risk of secondary use.

3. Summary

This post compared the translation capabilities of the generative AIs ChatGPT, Gemini, and Claude. In machine translation from English to Japanese, Claude excelled in creating natural expressions. On the other hand, when reflecting translation rules that were established through instructions or style guides, Gemini adhered to the rules most appropriately. Additionally, we found that Gemini excels in automatic revisions of translation memory. However, do note that the model of Gemini that we used in testing was Gemini 1.5 Pro, which is currently in private preview and is not available for use in business translations as of the date of posting.

At Human Science, we offer the automatic translation software MTrans for Office and MTrans for Trados, which can utilize ChatGPT. With our tool, you can translate transcriptions with just one click. ChatGPT can be used not only as a translation engine, but also for transcription, rewriting, and proofreading, depending on the prompt. Try out MTrans for Office and MTrans for Trados with our 14-day free trial offer. Please contact us for more information.

Features of MTrans for Office

① Unlimited number of file translations and glossary integration for a fixed fee
② One-click translation from Office products!
③ Secure API connection
・For customers who want further security enhancement, we also offer SSO, IP restrictions, and so on.

④ Japanese language support by a Japanese company
・Response to security check sheets
・Payment by bank transfer available

MTrans for Office is an easy-to-use translation software for Office.

Features of MTrans for Trados

1. ① Simultaneous translation using multiple machine translation engines, such as DeepL and Google.
2. ② Automatic application of specific terminology to machine-translated text. Centrally managed term glossaries, regardless of the machine translation engine
3. ③ String replacement, regular expression replacement, and automatic correction of translation style, notation, and expressions using ChatGPT
4. ④ Automatic revision of fuzzy matches in the translation memory
5. ⑤ Preservation of the original formatting and tags during machine translation

 

What is the Trados-dedicated machine translation solution MTrans for Trados?