没有不值得去解决的问题,也没有不值得去学习的技术!

Re-evaluating A Tour of Go Translation Quality: Codex High Nears ChatGPT High, So I Changed the Default Translation Engine

[Figure 6: Reading the secret key only after all reviews are completed to finalize unblinding]

作者:

在

Today, while continuing work on the A Tour of Go multilingual translation project, I finally re-verified an issue that has already been affecting actual translation progress:

Is Codex’s translation quality truly and noticeably lower than ChatGPT, or was there an uncontrolled variable in the previous experiment’s conditions?

What actually drove me to redo this experiment was not the fact that Codex only recently started supporting higher reasoning effort.

In fact, during the last translation quality comparison between ChatGPT, Codex, and GLM-5.2, Codex already offered the option to select different reasoning efforts. However, at the time I stuck with the default light reasoning, while ChatGPT used high reasoning effort.

Back then, I did not specifically treat “reasoning effort” as the primary experimental variable.

The reason I must re-verify now is due to another, more practical issue:

The current approach of using ChatGPT as the primary TranslationUnit translation engine can no longer stably support the continued progress of actual translation work.

In the previous article, I just documented this adjustment process:

Initially, I wanted ChatGPT to read the officially exported retranslation batch from GitHub, complete the entire TranslationUnit translation, and generate a raw-responses ZIP package for local download and import.

This workflow had been previously verified as executable.

However, when it came to the new official batch, stability became a real problem.

Sometimes, after the batch and inputs in GitHub were read, the subsequent TranslationUnit translation would stall and fail to actually generate. Sometimes the translation process would stop midway and require constant prompting to continue. Even if partial results were eventually generated, continuing to generate the complete ZIP artifact could get stuck again.

Later, to reduce delivery complexity, I further simplified the approach:

Instead of requiring a ZIP, I changed it to only generate a translation-result.json.

The local project then added:

retranslation import --translation-result

responsible for importing the JSON into raw-responses/.

From a workflow design perspective, this was already much simpler than a ZIP.

But after continuing actual testing, the problem still did not disappear.

Even when only asked to generate a JSON, ChatGPT could still remain stuck in a “preparing to generate” or “continuing to generate” state for a long time without actually producing complete TranslationUnit results.

So the problem is no longer:

This workflow is not automated enough.

But rather:

This workflow itself cannot stably support the continued progress of batch translation work.

If ChatGPT could still stably complete every batch, then even if it required downloading a JSON once and importing it locally once, I could have completely accepted it.

But once the actual translation generation process itself frequently gets stuck, I had to re-answer a more practical question:

Is Codex’s translation quality now close enough to ChatGPT to take on official production translation tasks?

And earlier, I had already conducted a dedicated translation quality experiment:

In that experiment, ChatGPT’s overall performance was clearly better.

So even though Codex was always more convenient at the engineering execution level, I previously did not directly switch the default translation engine to Codex.

The reason is simple:

In this project, what I value most is still the final translation quality.

If it just means easier execution but clearly lowers the final translation quality, then this kind of automation doesn’t have much meaning.

What the previous experiment actually proved

Looking back at the previous quality experiment, I found a variable well worth re-verifying.

At that time:

ChatGPT used high reasoning effort.

Codex used the default light reasoning effort.

In other words, what that experiment actually proved was closer to:

ChatGPT High > Codex Light

Rather than:

ChatGPT High > Codex High

And it certainly couldn’t directly deduce that:

Even if Codex also used high reasoning effort, its translation quality would still definitely be clearly lower than ChatGPT.

During the previous experiment, Codex did not lack higher reasoning effort options; it’s just that this point wasn’t isolated and controlled as the main variable at the time.

And today, ChatGPT as the default translation engine has shown persistent stability issues in actual work.

Since the original approach has already affected official translation progress, Codex, which was previously ruled out because its “translation quality was clearly inferior to ChatGPT,” must be re-evaluated.

So this time I actively tested:

  • ChatGPT GPT-5.6 High
  • Codex GPT-5.6 Sol High
  • Codex GPT-5.6 Sol Extra High

To see how much of the previous quality gap remained after increasing Codex’s reasoning effort.

This time I still chose Simplified Chinese instead of directly testing with the ongoing Japanese translation.

The reason is that Chinese is easier for multi-model quality review, and it also allows me to reuse the evaluation methods established in the previous experiment as much as possible.

Selecting 5 TranslationUnits again

This time I selected 5 complete Page TranslationUnits:

  • methods/24
  • concurrency/7
  • concurrency/11
  • generics/1
  • flowcontrol/8

The first three:

  • methods/24
  • concurrency/7
  • concurrency/11

Are exactly the representative pages used in the previous translation quality experiment.

Additionally, I added:

  • generics/1
  • flowcontrol/8

To make the sample cover interfaces, concurrency, generics, control flow, and exercise pages.

The three candidates were:

  • ChatGPT GPT-5.6 High
  • Codex GPT-5.6 Sol High
  • Codex GPT-5.6 Sol Extra High

There is another detail here that is easy to confuse.

Codex’s reasoning effort is not controlled by writing “High” or “Extra High” in the prompt; instead, I select it directly in the Codex interface.

So the difference between Codex High and Extra High this time is an actual difference in model runtime configuration, not different text written in the prompt.

Experiment inputs no longer use the English source files directly

This time, I also did not simply hand the original .article files to the three models for translation.

The minimum translation unit in the project’s official production workflow has been determined to be the TranslationUnit, so the experiment should also simulate the real production environment as closely as possible.

I first exported a zh-CN experiment batch through the official retranslation export workflow:

chatgpt-zh-CN-quality-experiment-001

It contains 5 official TranslationUnit inputs.

These inputs have already been processed by the project’s protected token mechanism, showing things like:

GTI18N_*

These protection markers.

The purpose of this step is to let the three models face inputs that are as consistent as possible with official production translation, rather than preparing a simpler set of plain text specifically for the experiment.

However, right here, I still fell into a very important trap.

The first round of experiments must be discarded: I missed the glossary

At first, I thought:

Official TranslationUnit input + protected token

Was already enough to simulate production translation conditions.

All three models completed the translation smoothly, and the number and order of protected tokens were completely consistent.

I then sent the 15 candidates into the official Engineering Gate.

Results:

  • restore: 15 / 15 successful
  • validation: 12 / 15 passed

All three failed items came from:

concurrency/11

And the errors were exactly the same:

candidate contains forbidden locale translation "幻灯片"

The cause was quickly found.

It is clearly specified in locales/zh-CN/glossary.yaml:

slide => 页面

slides => 页面

At the same time:

幻灯片

Is a forbidden zh-CN translation.

Yet all three models naturally translated slides as “幻灯片” (slide).

This was not some strange simultaneous malfunction across all three models.

The real problem was:

I only gave them the official TranslationUnit input, but did not provide the complete target language glossary as part of the official translation conditions.

In other words, although the structural input in this round was already close to the production environment, the translation context was still incomplete.

So the 12 / 15 Engineering Gate result cannot be simply attributed to the models themselves.

And I certainly couldn’t just manually replace the three “幻灯片” with “页面” (page) to let the experiment continue.

Although that would allow validation to pass, it would have already broken the independence of the candidates.

The only correct way to handle this was:

Discard the first round A / B / C
↓
Continue using the exact same 5 official TranslationUnit inputs
↓
Add the complete locales/zh-CN/glossary.yaml
↓
Regenerate the three candidates
↓
Re-randomize anonymization
↓
Re-run the Engineering Gate

[Figure 1: Redesigning the v2 experiment after discovering the missing glossary condition, and actually selecting GPT-5.6 Sol High in Codex]
[Figure 1: Redesigning the v2 experiment after discovering the missing glossary condition, and actually selecting GPT-5.6 Sol High in Codex]

This meant translating 5 × 3 all over again.

But this round of rework was necessary.

Because what I really wanted to verify was not:

“Which model is better at just translating casually.”

But rather:

Which model is truly suitable to become the TranslationUnit translation engine in the official production workflow of go-tour-i18n.

Codex High and Extra High regenerated separately

After regenerating, I first checked Codex High.

All 5 TranslationUnits were completed and passed:

  • JSON parsing
  • unit_id integrity check
  • GTI18N protected token count, content, and order check
  • Link and .article structure check
  • glossary mandatory / preferred / forbidden check
  • slides => 页面 check
  • Forbidden word check

The 5 TranslationUnits contained a total of 86 protected tokens, all consistent.

The number of occurrences of “幻灯片” (slide) became:

0

[Figure 2: Codex GPT-5.6 Sol High completes v2 translation and full checks]
[Figure 2: Codex GPT-5.6 Sol High completes v2 translation and full checks]

Then, using the exact same TranslationUnits, the exact same glossary, and the exact same task requirements, I ran Codex Extra High.

The only variable I actively changed was setting the reasoning effort in the Codex interface from:

High

To:

Extra High

Extra High also completed the 5 TranslationUnits and passed all pre-checks.

[Figure 3: Codex GPT-5.6 Sol Extra High completes v2 translation and checks]
[Figure 3: Codex GPT-5.6 Sol Extra High completes v2 translation and checks]

At this point, the three new v2 candidates were finally truly comparable:

  • ChatGPT GPT-5.6 High
  • Codex GPT-5.6 Sol High
  • Codex GPT-5.6 Sol Extra High

Independent random anonymization for each TranslationUnit

The next step was not to directly review using the three model names.

I split the 15 candidates from the three JSONs into:

5 TranslationUnits × 3 candidates per page

Then I anonymized them.

Here, I did not simply set:

A = ChatGPT
B = Codex High
C = Codex Extra High

And let this mapping run through all 5 pages.

Instead, I performed independent random A / B / C mapping for each TranslationUnit.

For example, A on one page might come from Codex High, while A on the next page might come from Codex Extra High.

This reduces the chance that the reviewer, after gradually guessing a candidate’s style from the first few pages, will form a prior assumption about subsequent pages.

Ultimately generating:

15 anonymous .article candidates.

The actual mapping was saved separately in a secret key and not read until all reviews were finished.

[Figure 4: Regenerated v2 anonymous package, 5 TranslationUnits split into 15 anonymous candidates]
[Figure 4: Regenerated v2 anonymous package, 5 TranslationUnits split into 15 anonymous candidates]

Second Engineering Gate: 15 / 15 all passed

After anonymization, I re-invoked the project’s official retranslation process, restore, and validator.

This time:

  • Protected token restore: 15 / 15
  • Validation: 15 / 15
  • restore failed: 0
  • validation failed: 0

Engineering Gate:

15 / 15 PASSED

At this point, the 15 candidates were officially frozen.

After this, candidate translations are no longer modified, anonymization rules are no longer changed, and nothing is regenerated.

Freezing the language review prompt

To prevent different review models from receiving different contexts, I generated a unified, frozen anonymous review prompt.

It includes:

  • 5 complete English TranslationUnits
  • Corresponding Candidate A / B / C for each page
  • Complete zh-CN glossary
  • Identical scoring dimensions
  • Identical output JSON format

I also recorded the SHA-256:

94a386eca3680a663409f3be4a7e770310189bb762733cd435ff40f7d91a2add

The check results included:

  • TranslationUnit: 5
  • Candidate: 15
  • GTI18N residue: 0
  • Anonymity leak check: PASSED
  • Candidate hashes consistent before and after generation
  • Secret key not read
  • Repository tracked files unmodified
[Figure 5: Frozen anonymous review prompt, SHA-256, and anonymity leak check results]
[Figure 5: Frozen anonymous review prompt, SHA-256, and anonymity leak check results]

From this moment on, the review models all saw the exact same set of materials.

Four independent models participate in the anonymous review

In the end, four review models were actually used:

  • DeepSeek
  • Doubao 2.1
  • GLM-5.3
  • ChatGPT GPT-5.6 High

This is slightly different from the previous experiment.

The previous one used GLM-5.2, but by the time of this actual review, GLM-5.3 was already being used.

To reduce the impact of potential self-evaluation bias from ChatGPT towards its own candidates, I used:

DeepSeek + Doubao 2.1 + GLM-5.3

As the primary external review results.

The review from ChatGPT GPT-5.6 High served as the fourth auxiliary result.

The score for each TranslationUnit was still out of 100:

DimensionScore
Technical accuracy30
Faithfulness to source20
Chinese naturalness20
Instructional expression15
Terminology consistency10
Readability5
Total score100

At the same time, each reviewer also had to strictly rank the three candidates and record:

  • critical
  • major
  • minor

Three levels of issues.

I did not read the secret key until all reviews were completed.

Lock scores first, then unblind

After all scoring was completed, I finally executed:

cat /tmp/tour-translation-quality-review-20260823-v2-key.json

[Figure 6: Reading the secret key only after all reviews are completed to finalize unblinding]
[Figure 6: Reading the secret key only after all reviews are completed to finalize unblinding]

This step also shows again why you can’t simply tally “A’s average score vs. B’s average score.”

Because each TranslationUnit was independently randomized.

For example:

methods/24

In:

  • A = Codex High
  • B = ChatGPT High
  • C = Codex Extra High

But when it came to:

concurrency/7

It became:

  • A = Codex Extra High
  • B = ChatGPT High
  • C = Codex High

So, after all scores were locked, I had to use the secret key to remap them into the actual model results.

Final results of the three external reviews

First, counting only:

  • DeepSeek
  • Doubao 2.1
  • GLM-5.3

The three external reviews.

The results are as follows:

ModelAverage scoreAverage rankNumber of first places
ChatGPT GPT-5.6 High96.671.735 / 15
Codex GPT-5.6 Sol High95.732.075 / 15
Codex GPT-5.6 Sol Extra High95.872.205 / 15

This was a result that somewhat surprised me.

The number of first places for all three models was actually:

5 / 15

ChatGPT High still had the highest average score.

But compared to Codex High, it only led by:

0.93 points

Compared to Codex Extra High, it only led by:

0.80 points

This is already a noticeable difference from the intuitive impression formed in the previous experiment.

At least on the current sample of 5 official TranslationUnits, I can hardly say anymore:

ChatGPT’s translation quality is clearly higher than Codex’s.

A more accurate description would be:

The three are already in the same high-quality tier, with ChatGPT High still holding a slight edge in the external review average score.

Different pages even had different winners

Looking page by page makes this even more obvious.

Among the three external reviews:

methods/24

Codex High received unanimous first place from the three external reviews.

concurrency/7

ChatGPT High received unanimous first place from the three external reviews.

concurrency/11

Codex High received first place from 2 external reviews, and ChatGPT High received 1.

generics/1

Codex Extra High received unanimous first place from the three external reviews.

flowcontrol/8

Codex Extra High received first place from 2 external reviews, and ChatGPT High received 1.

In other words, this is not a:

ChatGPT > Codex High > Codex Extra High

kind of very neat relationship.

The actual situation is closer to:

The three configurations each have their own advantages on different TranslationUnits.

This is also an important basis for my belief that they have “entered the same quality tier.”

The gap narrows further after adding the ChatGPT review

After adding the fourth review from ChatGPT GPT-5.6 High:

ModelFour-review average scoreAverage rankNumber of first places
ChatGPT High96.901.856 / 20
Codex High96.501.859 / 20
Codex Extra High96.152.305 / 20

Here appeared another very interesting result.

Codex High’s number of first places actually became the highest: 9 / 20.

Moreover, the ChatGPT reviewer itself did not show an obvious bias towards ChatGPT candidates.

Across the 5 pages, the ChatGPT GPT-5.6 High reviewer ranked:

Codex High

First on 4 pages.

This at least shows that the results of this round cannot simply be explained as:

“ChatGPT giving high scores to its own translations.”

Extra High did not bring a stable improvement

Before starting the experiment, I was actually looking forward to Extra High.

If Codex High could already get close to ChatGPT, could Extra High further narrow the gap, or even stably surpass ChatGPT?

The results did not support this hypothesis.

Three external reviews:

  • Codex High: 95.73
  • Codex Extra High: 95.87

Extra High only added:

0.14 points

But the average rank:

  • High: 2.07
  • Extra High: 2.20

Actually dropped slightly.

After adding all four reviews:

  • Codex High: 96.50
  • Codex Extra High: 96.15

Extra High was lower again by:

0.35 points

Moreover, the only issue in the entire experiment judged as major by the external reviews also appeared in the methods/24 candidate of Codex Extra High.

None of the three models had critical issues.

So currently, I haven’t seen sufficient evidence to show that:

Upgrading Codex from High to Extra High can stably improve the translation quality of A Tour of Go.

This means there isn’t a strong necessity to use Extra High by default going forward.

The previous experiment wasn’t wrong, but the conclusion needs conditions

This result doesn’t mean the previous quality experiment was wrong.

What the previous experiment actually answered was:

Under that set of models and reasoning configurations, which candidate performed better?

The result was indeed that ChatGPT had a clear advantage.

But there was a very important condition at the time:

ChatGPT used high reasoning effort.

Although Codex could also choose higher reasoning effort, what was actually used at the time was the default light reasoning effort.

Today’s round actively upgraded Codex to High and Extra High.

The final result became:

ChatGPT High ≈ Codex High

Therefore, looking at it now, Codex’s weaker performance in the previous experiment was likely at least partly due not to the Codex execution environment itself, but to the fact that lower reasoning effort was actually used at the time.

This is also one of my biggest takeaways from re-testing today.

Translation quality is no longer the main reason preventing me from using Codex

If we only compare the average scores of the three external reviews:

ChatGPT High:

96.67

Codex High:

95.73

ChatGPT still leads by about:

0.93 points

But now, engineering execution capability must be considered simultaneously.

The current ChatGPT web interface approach, even after being simplified from ZIP to JSON, still hasn’t truly solved the stability problem.

What actually affects production is no longer an extra download or import, but:

The inability to guarantee the stable completion of the TranslationUnit translation task itself for an official batch.

In contrast, Codex can directly operate the local repository.

If it is used as the official translation engine, the future workflow could completely be:

retranslation export
↓
Codex reads the manifest, inputs, and target language glossary
↓
Complete TranslationUnit translation
↓
Directly write to raw-responses/
↓
retranslation process
↓
Official validation

During the experiment phase, to keep the candidate formats consistent, I still had Codex generate JSON.

But if the subsequent official production switches to Codex, this JSON intermediate layer will no longer be necessary to keep.

I have decided to make Codex High the default translation approach going forward

Based on today’s results, my current choice is:

First choice: Codex GPT-5.6 Sol High

As the default approach for subsequent official TranslationUnit translations.

Its language quality is already very close to ChatGPT High, and it is clearly more suitable for the current project in terms of local batch execution, file writing, and automated workflows.

Second choice: ChatGPT GPT-5.6 High

Looking solely at the average score of the three external reviews, ChatGPT still holds a slight advantage.

So it will still play a very important quality control role in this project.

It’s just that given its actual performance on the web interface, I no longer plan to continue using it for the default production translation tasks of large batches of TranslationUnits.

Not used by default for now: Codex GPT-5.6 Sol Extra High

This time, no stable quality improvement from Extra High over High was found.

Without more evidence, there is no need to incur extra costs for higher reasoning effort.

Next, continue advancing ja-JP

The target of this experiment was Simplified Chinese, not Japanese.

So it cannot directly prove:

That Codex High’s Japanese translation quality is also definitely identical to ChatGPT High.

But it has already solved the most important production decision issue right now:

Is Codex High still unsuitable as the project’s default translation engine due to clearly insufficient translation quality?

At least based on this round of controlled experiments today, the answer is no.

When continuing to advance ja-JP next, I plan to adjust the default translation phase to:

Codex GPT-5.6 Sol High
↓
Read complete TranslationUnit and target language glossary
↓
Directly generate raw-responses/
↓
Official restore / validation
↓
Continue to the next batch of TranslationUnits

After all Page and Example TranslationUnits are completed, we will move into the unified quality acceptance.

Here, I currently still plan to have ChatGPT participate in the Quality Check.

In other words, ChatGPT will no longer handle the large amount of repetitive official translation generation work upfront, but will instead place its superior language quality judgment capabilities into the subsequent unified quality check phase.

The Quality Check will continue to rate complete TranslationUnits according to:

A / B / C / D

My goal is not “mostly reaching A.”

But rather:

All TranslationUnits must ultimately reach A.

If a TranslationUnit gets a B, C, or D, work will continue on it until all reach A, which is the only way to truly pass this round of Quality Check.

After that, we enter:

Final Review
↓
promote

So adjusting the default translation engine does not mean lowering quality requirements.

The new division of labor is closer to:

Codex High is responsible for stably completing the batch complete TranslationUnit translation; ChatGPT is responsible for the subsequent unified Quality Check.

Summary

The starting point of today’s re-experiment was actually not to simply compare model scores once.

It was because after ChatGPT became the default batch translation engine, actual work could no longer advance stably.

After re-controlling reasoning effort, official TranslationUnit input, glossary, anonymization, and the Engineering Gate, the final results obtained were:

  • ChatGPT High external review average score: 96.67
  • Codex High: 95.73
  • Codex Extra High: 95.87
  • All three had 5 / 15 external first places
  • Extra High did not show a trend of being stably superior to High

Therefore, I am now more inclined to interpret this experiment as:

Codex High has entered essentially the same translation quality tier as ChatGPT High.

This is enough to change the subsequent production approach.

Next, the default translation engine for the A Tour of Go multilingual TranslationUnit will be changed to:

Codex GPT-5.6 Sol High.

ChatGPT will shift to the subsequent unified Quality Check.

The final quality gate remains unchanged:

Only when all TranslationUnits achieve an A in the Quality Check will they truly pass the quality acceptance.

The focus of this adjustment is not about who absolutely defeated whom in one experiment.

Rather, when the translation quality gap is small enough, the ability to stably complete real production tasks begins to become the more critical factor in choosing the default translation engine.

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者,自 2013 年起持续运营个人技术博客,记录工作与个人项目中遇到的真实问题,以及解决问题过程中的探索、实践和思考。

除了技术写作,我也在持续开发和维护自己的独立项目,探索新技术和新工具在实际场景中的应用。我始终相信,没有不值得去解决的问题,也没有不值得去学习的技术。

如果这篇文章对你有所帮助,欢迎浏览博客中的其他文章,也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系