On September 24, I ran a Marathi technical translation comparison among GPT-5.6 Sol High, GPT-6 Sol High, and GPT-6 Sol Medium. The detailed process is documented in the previous article.
At that time, GPT-5.6 Sol High was already my official default translation model. The real purpose of the test was to determine whether GPT-6 Sol High or GPT-6 Sol Medium was ready to take over. Ultimately, the four valid AI blind reviews yielded three-page average scores of 93.58, 92.67, and 91.67 respectively. GPT-6 did not prove itself to be better, so I kept GPT-5.6 Sol High.
Now that GPT-6.1 Sol is available in Codex, I naturally have the same question again: Can GPT-6.1 Sol High truly surpass GPT-5.6 Sol High this time?
This post only records the new tests and decisions made since the previous article. I will not repeat the blind review rules, Marathi glossary, six-dimensional percentage scoring, protected token mechanism, or the reasons why automatic validation cannot replace independent reviewers, as they were all detailed in the last round.
1. Continuing with the same Marathi benchmark instead of picking new topics
To make the results as comparable as possible with the previous article, I continued to use the same three complex Pages:
methods/16: type switch and dense terminology;generics/1: type parameters andcomparableconstraint;concurrency/5: blocking and execution semantics ofselect.
Instead of regenerating the output for GPT-5.6 Sol High, I directly reused the original output saved on September 24, 2026. This avoids changing the baseline itself due to resampling.
The models that were actually newly generated in this round are:
- Codex GPT-6.1 Sol High;
- Codex GPT-6.1 Sol Medium.
Both used the byte-identical frozen English source, protected input, Marathi glossary, Translation Task Spec, and Generation instruction from the historical benchmark, with each completing the three Pages in a single independent invocation.
In terms of execution time, GPT-6.1 Sol High took about 91 seconds this time, while GPT-6.1 Sol Medium took about 70 seconds. I did not use the Codex Usage percentage as a core comparison metric this time, because the previous article already proved that subscription quotas have delayed billing, making precise attribution difficult from a single refresh.
2. The Index-Translate API works, but the 4096 context cannot fit the full test
Originally, I planned to add a new contender this time: Index-Translate-35B-A3B.
Its free public API itself works fine. I actually sent short text requests and received normal translation results. The real issue is the context window of the current public preview serving.
After packaging the complete frozen benchmark input, the request contained approximately 12,353 prompt tokens, and the official API returned:
n_ctx = 4096 exceed_context_size_error
Theoretically, I could have deleted the glossary, removed the task spec, compressed the context, or split the three Pages into multiple requests. But if I had done so, the experimental conditions it faced would no longer be the same as those for GPT-5.6 / GPT-6.1.
Therefore, I did not change the test conditions just to “make up four models.” Index-Translate’s status is NOT SCORED: it is not a translation quality failure, but rather that the serving specification of the current public API cannot accommodate this complete frozen input. Once the context window is increased in the future, I can directly use the same benchmark to run a supplementary test.
3. This time I finally collected all five independent Reviewer reports
The Reviewer approach itself was not invented for this round. The previous article also planned to use five independent AI blind reviews, but Qwen failed three times in a row, ultimately leaving only four complete and valid reports from ChatGPT, Doubao, DeepSeek, and GLM.
This time, all five completed successfully:
- ChatGPT GPT-5.6 Sol High;
- DeepSeek;
- GLM 5.3 Ultimate;
- Qwen;
- Doubao.
The scoring rules are exactly the same as in the previous article and will not be reiterated. The only further refinement is that I generated independent A/B/C blind mappings for each of the five Reviewers. Across all 15 “Reviewer × Page” positions, the three real models landed in Candidate A, B, and C exactly 5 times each.
This not only hides the model identities but also further reduces potential bias from the Candidate labels themselves.
4. An old checker nearly killed GPT-6.1 Medium by mistake
The generics/1 for GPT-6.1 Sol Medium once triggered a protected token sequence mismatch in the old benchmark checker.
Upon further inspection, I found that it did not lose, duplicate, or modify any protected tokens; it simply swapped the positions of two complete inline-code pairs to accommodate natural Marathi word order.
The current official Page contract explicitly allows this kind of whole-pair reorder. After restoring it using the historical token mapping, running it through the current official ValidateCandidate also resulted in a normal PASS.
Therefore, this was classified as a legacy checker false positive. GPT-6.1 Medium retained its official scoring eligibility and was not regenerated to “fix the score.”
This is actually consistent with the conclusion from the previous article: mechanical validation must be strict, but the validator itself must also be constrained by the spec. It cannot misjudge a legitimate structural adjustment made for natural word order in the target language as a translation defect.
5. After unblinding, GPT-5.6 Sol High is still first
Only after all five Reviewer reports were completed did I read the local private mapping to restore each Page’s Candidate A/B/C to the real Generation models.
There is a statistical detail here: because A/B/C mappings are independently randomized per Page, the “Candidate A three-page average score” calculated by the Reviewers themselves at the end of their reports cannot directly correspond to any single real model. The final model scores must be reaggregated based on the true mapping of each Page after unblinding.
| Reviewer | GPT-5.6 Sol High | GPT-6.1 Sol High | GPT-6.1 Sol Medium |
|---|---|---|---|
| ChatGPT | 97.67 | 97.67 | 96.00 |
| DeepSeek | 96.00 | 92.67 | 92.67 |
| GLM 5.3 Ultimate | 94.00 | 91.67 | 92.00 |
| Qwen | 81.67 | 85.33 | 82.67 |
| Doubao | 94.00 | 93.67 | 95.00 |
| Five-Reviewer Average | 92.67 | 92.20 | 91.67 |
| Median | 94.00 | 92.67 | 92.67 |
The final ranking remains:
1. GPT-5.6 Sol High 92.67 2. GPT-6.1 Sol High 92.20 3. GPT-6.1 Sol Medium 91.67
GPT-5.6 Sol High is only 0.47 points higher than GPT-6.1 Sol High, and 1.00 point higher than Medium.
So this is absolutely not a case of “5.6 crushing 6.1.” A more accurate conclusion is: The three models are already in a very close, high-quality range, but GPT-6.1 High has still not proven that it should replace the incumbent.
6. Looking by Page, GPT-6.1 High also has clear wins
| Page | GPT-5.6 High | GPT-6.1 High | GPT-6.1 Medium |
|---|---|---|---|
methods/16 | 93.2 | 91.4 | 91.2 |
generics/1 | 93.4 | 90.6 | 91.6 |
concurrency/5 | 91.4 | 94.6 | 92.2 |
GPT-5.6 is more stable on Methods and Generics, but GPT-6.1 High actually holds a clear lead on Concurrency.
This looks more like a slight advantage for different models in different technical expression scenarios, rather than one model comprehensively overpowering another.
Across the 15 independent Page rankings, GPT-5.6 Sol High took first place 6 times, GPT-6.1 Sol Medium 5 times, and GPT-6.1 Sol High 4 times. Again, there was no overwhelming gap.
7. These results also remind me not to blindly trust absolute scores
There is another phenomenon worth noting in this round: the scoring scales of the five Reviewers varied greatly.
ChatGPT often gave 98–99 for excellent translations, while Qwen was noticeably stricter, with many scores only in the 77–90 range. Different Reviewers did not always agree on whether the same minor issue should be classified as Minor or Major.
Even for the exact same historical output from GPT-5.6 Sol High, the overall average score across the four Reviewers in the previous article was 93.58, which became 92.67 after aggregating the five Reviewers this time. The translation itself did not change by a single word; what changed was the Reviewer panel.
This itself illustrates that one should not overinterpret absolute differences of a fraction of a point in cross-round benchmarks. What is truly worth paying attention to is: who consistently stays at the top, whether multiple Reviewers reach a consistent direction, whether there are any critical technical defects, and whether the results flip when swapping in a different set of Reviewers.
As a simple sensitivity check, if I completely remove the noticeably stricter Qwen report, the average scores from the remaining four Reviewers are still:
GPT-5.6 Sol High 95.42 GPT-6.1 Sol High 93.92 GPT-6.1 Sol Medium 93.92
The ranking still did not reverse.
8. So once again, I am not switching the default model
If the question is only “Can GPT-6.1 Sol High do high-quality technical translation?”, the answer is already clear: Yes, and it is already very close to GPT-5.6 Sol High.
But if the question is “Is there enough evidence to make me switch the official Generation default from GPT-5.6 Sol High?”, the answer is still no.
Therefore, the current execution decision remains:
Official Generation default: GPT-5.6 Sol High Independent Reviewer: GPT-5.6 Sol High Trusted Generation fallback / next candidate: GPT-6.1 Sol High Not currently the highest-quality default: GPT-6.1 Sol Medium Index-Translate-35B-A3B: NOT SCORED Current public preview context is insufficient
GPT-6.1 Medium completed the three Pages in about 70 seconds this time, which is indeed faster than High. However, in my project, a few dozen seconds of Generation time is far less important than revision, re-QC, and long-term translation quality, so there is currently no need to lower the reasoning effort for the sake of speed.
The conclusion from the previous article was: GPT-6 Sol did not prove that it should replace 5.6. With GPT-6.1, this gap has narrowed further to the point of being almost negligible, but the final conclusion still has not changed.
A new model does not automatically replace an old one just because its version number is higher; it must wait for actual evidence under the same set of production constraints to prove that it is better.
At least within this current set of Marathi technical translation benchmarks, GPT-5.6 Sol High has retained its position as the default Generation model, while GPT-6.1 Sol High has become a very close backup option that I am ready to enable at any time.