Today, while continuing work on the A Tour of Go multilingual translation project, I finally re-verified an issue that has already been affecting actual translation progress:
Is Codex’s translation quality truly and noticeably lower than ChatGPT, or was there an uncontrolled variable in the previous experiment’s conditions?
What actually drove me to redo this experiment was not the fact that Codex only recently started supporting higher reasoning effort.
In fact, during the last translation quality comparison between ChatGPT, Codex, and GLM-5.2, Codex already offered the option to select different reasoning efforts. However, at the time I stuck with the default light reasoning, while ChatGPT used high reasoning effort.
Back then, I did not specifically treat “reasoning effort” as the primary experimental variable.
The reason I must re-verify now is due to another, more practical issue:
The current approach of using ChatGPT as the primary TranslationUnit translation engine can no longer stably support the continued progress of actual translation work.
In the previous article, I just documented this adjustment process:
Initially, I wanted ChatGPT to read the officially exported retranslation batch from GitHub, complete the entire TranslationUnit translation, and generate a raw-responses ZIP package for local download and import.
This workflow had been previously verified as executable.
However, when it came to the new official batch, stability became a real problem.
Sometimes, after the batch and inputs in GitHub were read, the subsequent TranslationUnit translation would stall and fail to actually generate. Sometimes the translation process would stop midway and require constant prompting to continue. Even if partial results were eventually generated, continuing to generate the complete ZIP artifact could get stuck again.
Later, to reduce delivery complexity, I further simplified the approach:
Instead of requiring a ZIP, I changed it to only generate a translation-result.json.
The local project then added:
retranslation import --translation-result
responsible for importing the JSON into raw-responses/.
From a workflow design perspective, this was already much simpler than a ZIP.
But after continuing actual testing, the problem still did not disappear.
Even when only asked to generate a JSON, ChatGPT could still remain stuck in a “preparing to generate” or “continuing to generate” state for a long time without actually producing complete TranslationUnit results.
So the problem is no longer:
This workflow is not automated enough.
But rather:
This workflow itself cannot stably support the continued progress of batch translation work.
If ChatGPT could still stably complete every batch, then even if it required downloading a JSON once and importing it locally once, I could have completely accepted it.
But once the actual translation generation process itself frequently gets stuck, I had to re-answer a more practical question:
Is Codex’s translation quality now close enough to ChatGPT to take on official production translation tasks?
And earlier, I had already conducted a dedicated translation quality experiment:
In that experiment, ChatGPT’s overall performance was clearly better.
So even though Codex was always more convenient at the engineering execution level, I previously did not directly switch the default translation engine to Codex.
The reason is simple:
In this project, what I value most is still the final translation quality.
If it just means easier execution but clearly lowers the final translation quality, then this kind of automation doesn’t have much meaning.
What the previous experiment actually proved
Looking back at the previous quality experiment, I found a variable well worth re-verifying.
At that time:
ChatGPT used high reasoning effort.
Codex used the default light reasoning effort.
In other words, what that experiment actually proved was closer to:
ChatGPT High > Codex Light
Rather than:
ChatGPT High > Codex High
And it certainly couldn’t directly deduce that:
Even if Codex also used high reasoning effort, its translation quality would still definitely be clearly lower than ChatGPT.
During the previous experiment, Codex did not lack higher reasoning effort options; it’s just that this point wasn’t isolated and controlled as the main variable at the time.
And today, ChatGPT as the default translation engine has shown persistent stability issues in actual work.
Since the original approach has already affected official translation progress, Codex, which was previously ruled out because its “translation quality was clearly inferior to ChatGPT,” must be re-evaluated.
So this time I actively tested:
- ChatGPT GPT-5.6 High
- Codex GPT-5.6 Sol High
- Codex GPT-5.6 Sol Extra High
To see how much of the previous quality gap remained after increasing Codex’s reasoning effort.
This time I still chose Simplified Chinese instead of directly testing with the ongoing Japanese translation.
The reason is that Chinese is easier for multi-model quality review, and it also allows me to reuse the evaluation methods established in the previous experiment as much as possible.
Selecting 5 TranslationUnits again
This time I selected 5 complete Page TranslationUnits:
methods/24concurrency/7concurrency/11generics/1flowcontrol/8
The first three:
methods/24concurrency/7concurrency/11
Are exactly the representative pages used in the previous translation quality experiment.
Additionally, I added:
generics/1flowcontrol/8
To make the sample cover interfaces, concurrency, generics, control flow, and exercise pages.
The three candidates were:
- ChatGPT GPT-5.6 High
- Codex GPT-5.6 Sol High
- Codex GPT-5.6 Sol Extra High
There is another detail here that is easy to confuse.
Codex’s reasoning effort is not controlled by writing “High” or “Extra High” in the prompt; instead, I select it directly in the Codex interface.
So the difference between Codex High and Extra High this time is an actual difference in model runtime configuration, not different text written in the prompt.
Experiment inputs no longer use the English source files directly
This time, I also did not simply hand the original .article files to the three models for translation.
The minimum translation unit in the project’s official production workflow has been determined to be the TranslationUnit, so the experiment should also simulate the real production environment as closely as possible.
I first exported a zh-CN experiment batch through the official retranslation export workflow:
chatgpt-zh-CN-quality-experiment-001
It contains 5 official TranslationUnit inputs.
These inputs have already been processed by the project’s protected token mechanism, showing things like:
GTI18N_*
These protection markers.
The purpose of this step is to let the three models face inputs that are as consistent as possible with official production translation, rather than preparing a simpler set of plain text specifically for the experiment.
However, right here, I still fell into a very important trap.
The first round of experiments must be discarded: I missed the glossary
At first, I thought:
Official TranslationUnit input + protected token
Was already enough to simulate production translation conditions.
All three models completed the translation smoothly, and the number and order of protected tokens were completely consistent.
I then sent the 15 candidates into the official Engineering Gate.
Results:
- restore: 15 / 15 successful
- validation: 12 / 15 passed
All three failed items came from:
concurrency/11
And the errors were exactly the same:
candidate contains forbidden locale translation "幻灯片"
The cause was quickly found.
It is clearly specified in locales/zh-CN/glossary.yaml:
slide => 页面
slides => 页面
At the same time:
幻灯片
Is a forbidden zh-CN translation.
Yet all three models naturally translated slides as “幻灯片” (slide).
This was not some strange simultaneous malfunction across all three models.
The real problem was:
I only gave them the official TranslationUnit input, but did not provide the complete target language glossary as part of the official translation conditions.
In other words, although the structural input in this round was already close to the production environment, the translation context was still incomplete.
So the 12 / 15 Engineering Gate result cannot be simply attributed to the models themselves.
And I certainly couldn’t just manually replace the three “幻灯片” with “页面” (page) to let the experiment continue.
Although that would allow validation to pass, it would have already broken the independence of the candidates.
The only correct way to handle this was:
Discard the first round A / B / C
↓
Continue using the exact same 5 official TranslationUnit inputs
↓
Add the complete locales/zh-CN/glossary.yaml
↓
Regenerate the three candidates
↓
Re-randomize anonymization
↓
Re-run the Engineering Gate
![[Figure 1: Redesigning the v2 experiment after discovering the missing glossary condition, and actually selecting GPT-5.6 Sol High in Codex]](https://media.shuijingwanwq.com/2026/08/1-81.png)
This meant translating 5 × 3 all over again.
But this round of rework was necessary.
Because what I really wanted to verify was not:
“Which model is better at just translating casually.”
But rather:
Which model is truly suitable to become the TranslationUnit translation engine in the official production workflow of go-tour-i18n.
Codex High and Extra High regenerated separately
After regenerating, I first checked Codex High.
All 5 TranslationUnits were completed and passed:
- JSON parsing
unit_idintegrity check- GTI18N protected token count, content, and order check
- Link and
.articlestructure check - glossary mandatory / preferred / forbidden check
slides => 页面check- Forbidden word check
The 5 TranslationUnits contained a total of 86 protected tokens, all consistent.
The number of occurrences of “幻灯片” (slide) became:
0
![[Figure 2: Codex GPT-5.6 Sol High completes v2 translation and full checks]](https://media.shuijingwanwq.com/2026/08/2-71.png)
Then, using the exact same TranslationUnits, the exact same glossary, and the exact same task requirements, I ran Codex Extra High.
The only variable I actively changed was setting the reasoning effort in the Codex interface from:
High
To:
Extra High
Extra High also completed the 5 TranslationUnits and passed all pre-checks.
![[Figure 3: Codex GPT-5.6 Sol Extra High completes v2 translation and checks]](https://media.shuijingwanwq.com/2026/08/3-64.png)
At this point, the three new v2 candidates were finally truly comparable:
- ChatGPT GPT-5.6 High
- Codex GPT-5.6 Sol High
- Codex GPT-5.6 Sol Extra High
Independent random anonymization for each TranslationUnit
The next step was not to directly review using the three model names.
I split the 15 candidates from the three JSONs into:
5 TranslationUnits × 3 candidates per page
Then I anonymized them.
Here, I did not simply set:
A = ChatGPT
B = Codex High
C = Codex Extra High
And let this mapping run through all 5 pages.
Instead, I performed independent random A / B / C mapping for each TranslationUnit.
For example, A on one page might come from Codex High, while A on the next page might come from Codex Extra High.
This reduces the chance that the reviewer, after gradually guessing a candidate’s style from the first few pages, will form a prior assumption about subsequent pages.
Ultimately generating:
15 anonymous .article candidates.
The actual mapping was saved separately in a secret key and not read until all reviews were finished.
![[Figure 4: Regenerated v2 anonymous package, 5 TranslationUnits split into 15 anonymous candidates]](https://media.shuijingwanwq.com/2026/08/4-55.png)
Second Engineering Gate: 15 / 15 all passed
After anonymization, I re-invoked the project’s official retranslation process, restore, and validator.
This time:
- Protected token restore: 15 / 15
- Validation: 15 / 15
- restore failed: 0
- validation failed: 0
Engineering Gate:
15 / 15 PASSED
At this point, the 15 candidates were officially frozen.
After this, candidate translations are no longer modified, anonymization rules are no longer changed, and nothing is regenerated.
Freezing the language review prompt
To prevent different review models from receiving different contexts, I generated a unified, frozen anonymous review prompt.
It includes:
- 5 complete English TranslationUnits
- Corresponding Candidate A / B / C for each page
- Complete zh-CN glossary
- Identical scoring dimensions
- Identical output JSON format
I also recorded the SHA-256:
94a386eca3680a663409f3be4a7e770310189bb762733cd435ff40f7d91a2add
The check results included:
- TranslationUnit: 5
- Candidate: 15
- GTI18N residue: 0
- Anonymity leak check: PASSED
- Candidate hashes consistent before and after generation
- Secret key not read
- Repository tracked files unmodified
![[Figure 5: Frozen anonymous review prompt, SHA-256, and anonymity leak check results]](https://media.shuijingwanwq.com/2026/08/5-32-1024x513.png)
From this moment on, the review models all saw the exact same set of materials.
Four independent models participate in the anonymous review
In the end, four review models were actually used:
- DeepSeek
- Doubao 2.1
- GLM-5.3
- ChatGPT GPT-5.6 High
This is slightly different from the previous experiment.
The previous one used GLM-5.2, but by the time of this actual review, GLM-5.3 was already being used.
To reduce the impact of potential self-evaluation bias from ChatGPT towards its own candidates, I used:
DeepSeek + Doubao 2.1 + GLM-5.3
As the primary external review results.
The review from ChatGPT GPT-5.6 High served as the fourth auxiliary result.
The score for each TranslationUnit was still out of 100:
| Dimension | Score |
|---|---|
| Technical accuracy | 30 |
| Faithfulness to source | 20 |
| Chinese naturalness | 20 |
| Instructional expression | 15 |
| Terminology consistency | 10 |
| Readability | 5 |
| Total score | 100 |
At the same time, each reviewer also had to strictly rank the three candidates and record:
- critical
- major
- minor
Three levels of issues.
I did not read the secret key until all reviews were completed.
Lock scores first, then unblind
After all scoring was completed, I finally executed:
cat /tmp/tour-translation-quality-review-20260823-v2-key.json
![[Figure 6: Reading the secret key only after all reviews are completed to finalize unblinding]](https://media.shuijingwanwq.com/2026/08/6-17.png)
This step also shows again why you can’t simply tally “A’s average score vs. B’s average score.”
Because each TranslationUnit was independently randomized.
For example:
methods/24
In:
- A = Codex High
- B = ChatGPT High
- C = Codex Extra High
But when it came to:
concurrency/7
It became:
- A = Codex Extra High
- B = ChatGPT High
- C = Codex High
So, after all scores were locked, I had to use the secret key to remap them into the actual model results.
Final results of the three external reviews
First, counting only:
- DeepSeek
- Doubao 2.1
- GLM-5.3
The three external reviews.
The results are as follows:
| Model | Average score | Average rank | Number of first places |
|---|---|---|---|
| ChatGPT GPT-5.6 High | 96.67 | 1.73 | 5 / 15 |
| Codex GPT-5.6 Sol High | 95.73 | 2.07 | 5 / 15 |
| Codex GPT-5.6 Sol Extra High | 95.87 | 2.20 | 5 / 15 |
This was a result that somewhat surprised me.
The number of first places for all three models was actually:
5 / 15
ChatGPT High still had the highest average score.
But compared to Codex High, it only led by:
0.93 points
Compared to Codex Extra High, it only led by:
0.80 points
This is already a noticeable difference from the intuitive impression formed in the previous experiment.
At least on the current sample of 5 official TranslationUnits, I can hardly say anymore:
ChatGPT’s translation quality is clearly higher than Codex’s.
A more accurate description would be:
The three are already in the same high-quality tier, with ChatGPT High still holding a slight edge in the external review average score.
Different pages even had different winners
Looking page by page makes this even more obvious.
Among the three external reviews:
methods/24
Codex High received unanimous first place from the three external reviews.
concurrency/7
ChatGPT High received unanimous first place from the three external reviews.
concurrency/11
Codex High received first place from 2 external reviews, and ChatGPT High received 1.
generics/1
Codex Extra High received unanimous first place from the three external reviews.
flowcontrol/8
Codex Extra High received first place from 2 external reviews, and ChatGPT High received 1.
In other words, this is not a:
ChatGPT > Codex High > Codex Extra High
kind of very neat relationship.
The actual situation is closer to:
The three configurations each have their own advantages on different TranslationUnits.
This is also an important basis for my belief that they have “entered the same quality tier.”
The gap narrows further after adding the ChatGPT review
After adding the fourth review from ChatGPT GPT-5.6 High:
| Model | Four-review average score | Average rank | Number of first places |
|---|---|---|---|
| ChatGPT High | 96.90 | 1.85 | 6 / 20 |
| Codex High | 96.50 | 1.85 | 9 / 20 |
| Codex Extra High | 96.15 | 2.30 | 5 / 20 |
Here appeared another very interesting result.
Codex High’s number of first places actually became the highest: 9 / 20.
Moreover, the ChatGPT reviewer itself did not show an obvious bias towards ChatGPT candidates.
Across the 5 pages, the ChatGPT GPT-5.6 High reviewer ranked:
Codex High
First on 4 pages.
This at least shows that the results of this round cannot simply be explained as:
“ChatGPT giving high scores to its own translations.”
Extra High did not bring a stable improvement
Before starting the experiment, I was actually looking forward to Extra High.
If Codex High could already get close to ChatGPT, could Extra High further narrow the gap, or even stably surpass ChatGPT?
The results did not support this hypothesis.
Three external reviews:
- Codex High: 95.73
- Codex Extra High: 95.87
Extra High only added:
0.14 points
But the average rank:
- High: 2.07
- Extra High: 2.20
Actually dropped slightly.
After adding all four reviews:
- Codex High: 96.50
- Codex Extra High: 96.15
Extra High was lower again by:
0.35 points
Moreover, the only issue in the entire experiment judged as major by the external reviews also appeared in the methods/24 candidate of Codex Extra High.
None of the three models had critical issues.
So currently, I haven’t seen sufficient evidence to show that:
Upgrading Codex from High to Extra High can stably improve the translation quality of A Tour of Go.
This means there isn’t a strong necessity to use Extra High by default going forward.
The previous experiment wasn’t wrong, but the conclusion needs conditions
This result doesn’t mean the previous quality experiment was wrong.
What the previous experiment actually answered was:
Under that set of models and reasoning configurations, which candidate performed better?
The result was indeed that ChatGPT had a clear advantage.
But there was a very important condition at the time:
ChatGPT used high reasoning effort.
Although Codex could also choose higher reasoning effort, what was actually used at the time was the default light reasoning effort.
Today’s round actively upgraded Codex to High and Extra High.
The final result became:
ChatGPT High ≈ Codex High
Therefore, looking at it now, Codex’s weaker performance in the previous experiment was likely at least partly due not to the Codex execution environment itself, but to the fact that lower reasoning effort was actually used at the time.
This is also one of my biggest takeaways from re-testing today.
Translation quality is no longer the main reason preventing me from using Codex
If we only compare the average scores of the three external reviews:
ChatGPT High:
96.67
Codex High:
95.73
ChatGPT still leads by about:
0.93 points
But now, engineering execution capability must be considered simultaneously.
The current ChatGPT web interface approach, even after being simplified from ZIP to JSON, still hasn’t truly solved the stability problem.
What actually affects production is no longer an extra download or import, but:
The inability to guarantee the stable completion of the TranslationUnit translation task itself for an official batch.
In contrast, Codex can directly operate the local repository.
If it is used as the official translation engine, the future workflow could completely be:
retranslation export
↓
Codex reads the manifest, inputs, and target language glossary
↓
Complete TranslationUnit translation
↓
Directly write to raw-responses/
↓retranslation process
↓
Official validation
During the experiment phase, to keep the candidate formats consistent, I still had Codex generate JSON.
But if the subsequent official production switches to Codex, this JSON intermediate layer will no longer be necessary to keep.
I have decided to make Codex High the default translation approach going forward
Based on today’s results, my current choice is:
First choice: Codex GPT-5.6 Sol High
As the default approach for subsequent official TranslationUnit translations.
Its language quality is already very close to ChatGPT High, and it is clearly more suitable for the current project in terms of local batch execution, file writing, and automated workflows.
Second choice: ChatGPT GPT-5.6 High
Looking solely at the average score of the three external reviews, ChatGPT still holds a slight advantage.
So it will still play a very important quality control role in this project.
It’s just that given its actual performance on the web interface, I no longer plan to continue using it for the default production translation tasks of large batches of TranslationUnits.
Not used by default for now: Codex GPT-5.6 Sol Extra High
This time, no stable quality improvement from Extra High over High was found.
Without more evidence, there is no need to incur extra costs for higher reasoning effort.
Next, continue advancing ja-JP
The target of this experiment was Simplified Chinese, not Japanese.
So it cannot directly prove:
That Codex High’s Japanese translation quality is also definitely identical to ChatGPT High.
But it has already solved the most important production decision issue right now:
Is Codex High still unsuitable as the project’s default translation engine due to clearly insufficient translation quality?
At least based on this round of controlled experiments today, the answer is no.
When continuing to advance ja-JP next, I plan to adjust the default translation phase to:
Codex GPT-5.6 Sol High
↓
Read complete TranslationUnit and target language glossary
↓
Directly generate raw-responses/
↓
Official restore / validation
↓
Continue to the next batch of TranslationUnits
After all Page and Example TranslationUnits are completed, we will move into the unified quality acceptance.
Here, I currently still plan to have ChatGPT participate in the Quality Check.
In other words, ChatGPT will no longer handle the large amount of repetitive official translation generation work upfront, but will instead place its superior language quality judgment capabilities into the subsequent unified quality check phase.
The Quality Check will continue to rate complete TranslationUnits according to:
A / B / C / D
My goal is not “mostly reaching A.”
But rather:
All TranslationUnits must ultimately reach A.
If a TranslationUnit gets a B, C, or D, work will continue on it until all reach A, which is the only way to truly pass this round of Quality Check.
After that, we enter:
Final Review
↓
promote
So adjusting the default translation engine does not mean lowering quality requirements.
The new division of labor is closer to:
Codex High is responsible for stably completing the batch complete TranslationUnit translation; ChatGPT is responsible for the subsequent unified Quality Check.
Summary
The starting point of today’s re-experiment was actually not to simply compare model scores once.
It was because after ChatGPT became the default batch translation engine, actual work could no longer advance stably.
After re-controlling reasoning effort, official TranslationUnit input, glossary, anonymization, and the Engineering Gate, the final results obtained were:
- ChatGPT High external review average score: 96.67
- Codex High: 95.73
- Codex Extra High: 95.87
- All three had 5 / 15 external first places
- Extra High did not show a trend of being stably superior to High
Therefore, I am now more inclined to interpret this experiment as:
Codex High has entered essentially the same translation quality tier as ChatGPT High.
This is enough to change the subsequent production approach.
Next, the default translation engine for the A Tour of Go multilingual TranslationUnit will be changed to:
Codex GPT-5.6 Sol High.
ChatGPT will shift to the subsequent unified Quality Check.
The final quality gate remains unchanged:
Only when all TranslationUnits achieve an A in the Quality Check will they truly pass the quality acceptance.
The focus of this adjustment is not about who absolutely defeated whom in one experiment.
Rather, when the translation quality gap is small enough, the ability to stably complete real production tasks begins to become the more critical factor in choosing the default translation engine.
