A Tour of Go Translation Experiment: Why More Raw Context Did Not Help—From 157 Protection Markers to 30 Repeated Blind Evaluations

Starting with a question I never fully resolved

Recently, I have been continuing to verify the structural protection strategy in the A Tour of Go multilingual translation project.

The current official translation workflow does not simply hand the entire .article source text over to GLM-5.2. Instead, it first protects certain structures that should not be translated or modified, such as:

• present directives like .play and .image;
• link targets;
• structural boundaries of inline code;
• emphasis markers;
• static preformatted code;
• certain technical content that must be preserved exactly.

These elements are converted into protected tokens, restored after translation, and then fed into unified present parsing, structural comparison, and page validation.

The structural reliability of this approach is already quite mature, but I have always had one question:

If the model does not see the most complete original page, will these protected tokens disrupt the context and thereby degrade translation quality?

This question did not arise out of nowhere.

In previous Chinese-to-English AI translation work with WordPress, I repeatedly confirmed that under the same model, translating an entire article at once is usually significantly better than splitting it into isolated paragraphs.

Complete context helps the model understand preceding and following relationships, terminology, references, article topics, and overall tone.

Therefore, my initial intuition was:

Since more complete context generally benefits translation, reducing protected tokens in A Tour of Go to let the model see more original content should theoretically not degrade quality, at the very least.

But the subsequent experimental results were not that simple.

Starting with Default versus minimal-v1

In earlier experiments, I first designed a minimal protection mode.

Initially, it only protected complete .play directives. Later, based on actual failures, I added emphasis delimiters and introduced more precise retries.

This ultimately formed minimal-v1.

In one round of experiments across 7 representative pages:

Default:
First pass 7/7
Final pass 7/7

minimal-v1:
First pass 5/7
Final pass 7/7

Default:
First pass 7/7
Final pass 7/7

minimal-v1:
First pass 5/7
Final pass 7/7

I then conducted an anonymous translation quality comparison.

The results from the single sample were:

Default：4/7
minimal-v1：3/7

Default：4/7
minimal-v1：3/7

This result made me even more confused at the time.

Since minimal-v1 clearly exposed more original content to the model, why did it not only fail to show a stable quality advantage, but also perform noticeably worse than Default on some pages?

If we simply follow the experience that “more complete context means better translation,” this is not easy to explain.

So I decided not to continue modifying the protection policy for now, and instead to clarify a more fundamental question first:

What exactly did Default’s protected tokens actually hide?

157 protected tokens: how much natural language did they actually obscure?

I re-ran a Default protection information loss audit on the fixed 7 representative pages.

The results were:

Pages: 7
Protected tokens: 157
Replaced source bytes: 1,206

A machine structure: 761 bytes
B code / technical content: 419 bytes
C fixed natural language: 26 bytes
D hidden translatable English: 0 bytes

Hidden translatable English items: 0
Hidden translatable English ratio: 0.00%

Pages: 7
Protected tokens: 157
Replaced source bytes: 1,206

A machine structure: 761 bytes
B code / technical content: 419 bytes
C fixed natural language: 26 bytes
D hidden translatable English: 0 bytes

Hidden translatable English items: 0
Hidden translatable English ratio: 0.00%

This result is crucial.

It turns out I had previously conflated “raw byte integrity” with “natural language context integrity.”

For example:

`comparable`

`comparable`

Default does not hide the entire comparable.

In reality, the backticks on both sides are primarily what gets replaced, while the technically meaningful part in the middle:

comparable

comparable

can still be seen by the model.

Similarly:

*Note:*

*Note:*

The emphasis delimiters are mainly what is hidden, while Note: itself still exists in the model input.

For links:

[[/pkg/image/#Image][Package image]]

[[/pkg/image/#Image][Package image]]

What is primarily hidden is:

/pkg/image/#Image

/pkg/image/#Image

While what actually needs to be understood and translated:

Package image

Package image

remains fully visible.

Therefore, Default and minimal-v1 are not actually:

Complete context
VS
Incomplete context

Complete context
VS
Incomplete context

The more accurate distinction is:

Default:
Complete page-level natural language context
+ Stronger machine structure abstraction

minimal-v1:
Complete page-level natural language context
+ Less machine structure abstraction
+ More raw code and machine syntax

Default:
Complete page-level natural language context
+ Stronger machine structure abstraction

minimal-v1:
Complete page-level natural language context
+ Less machine structure abstraction
+ More raw code and machine syntax

This is actually a different issue from WordPress’s “whole-article translation vs. segmented translation.”

WordPress segmented translation genuinely loses preceding and following natural language context.

However, A Tour of Go’s Default protection does not split the complete course page into multiple isolated translation units.

What might actually lose semantic meaning is the static code

Although no translatable English natural language was completely hidden across the 7 pages, Default does hide some complete static code.

For example, in generics/1:

func Index[T comparable](s []T, x T) int

func Index[T comparable](s []T, x T) int

And in methods/24:

package image

type Image interface {
    ColorModel() color.Model
    Bounds() Rectangle
    At(x, y int) color.Color
}

package image

type Image interface {
    ColorModel() color.Model
    Bounds() Rectangle
    At(x, y int) color.Color
}

This code does not need translation, but it does contain technical semantics.

The question then narrowed down further to:

If Default’s natural language context is already complete, would merely letting the model see this static code additionally improve the translation?

To verify this, I did not disable the existing static code protection. Instead, I added a new mode used only for development experiments:

--dev-static-context

--dev-static-context

This mode still uses the exact same Default protected page.

There is only one difference:

A read-only static code reference is additionally included in the user message.

The code is only used to help the model understand technical relationships; it is not allowed to be copied, modified, or re-output.

This keeps the experimental variable constrained as much as possible to:

Whether the model can see the technical semantics of the static code

Whether the model can see the technical semantics of the static code

But an even larger variable was discovered during the experiment

Before the first round of Static Context experiments could yield quality conclusions, a more worthwhile issue to investigate emerged.

I found that the standard Default for generics/1 passed the validator on the first try in a previous fresh experiment, yet a new round of completely identical Default requests resulted in duplicated protected tokens.

So I directly compared the request.json and response.json from the two existing rounds of experiments.

The compared pages included:

generics/1
flowcontrol/8
methods/20
concurrency/7
methods/24

generics/1
flowcontrol/8
methods/20
concurrency/7
methods/24

The results were very consistent:

REQUEST_IDENTICAL : true
RESPONSE_IDENTICAL: false

REQUEST_IDENTICAL : true
RESPONSE_IDENTICAL: false

This was true for all 5 pages.

The final result:

identical request -> different assistant content = 5/5

identical request -> different assistant content = 5/5

What is compared here is not just that the Prompts are roughly the same.

It was actually confirmed that:

• source is identical;
• model is identical;
• system message is byte-identical;
• user message is byte-identical;
• thinking is identical;
• do_sample is identical;
• max_tokens is identical;
• actual API payload is identical.

Yet the assistant content received for 5/5 pages was all different.

Moreover, the differences were not just in wording.

One round of Default for generics/1 could pass the validator, while another round with the exact same Request duplicated the same set of protected tokens:

protected token count = 21, want 19
token 5 occurrence count = 2, want 1
token 6 occurrence count = 2, want 1

protected token count = 21, want 19
token 5 occurrence count = 2, want 1
token 6 occurrence count = 2, want 1

This means that run-to-run fluctuation can even affect:

validator pass
VS
validator fail

validator pass
VS
validator fail

As for why the GLM-5.2 server-side exhibits this behavior, it is impossible to determine based solely on the artifacts saved by the current project, and I did not continue guessing about specific server-side routing, model deployment, or computational reasons.

There is only one fact the project truly needs to face:

Identical Requests do not exhibit byte-level reproducibility in current actual API calls.

The previous 4:3 can no longer be taken as a modal conclusion

This also directly changed my understanding of the previous anonymous review between Default and minimal-v1.

The original results:

Default：4
minimal-v1：3

Default：4
minimal-v1：3

can only indicate:

When drawing one candidate from each mode this round, these were the results obtained.

It does not prove:

Default's translation quality is consistently higher than minimal-v1's

Default's translation quality is consistently higher than minimal-v1's

Nor does it prove:

minimal-v1 has no quality value

minimal-v1 has no quality value

Because it has now been confirmed that multiple independent runs of the same Default request alone are enough to produce noticeably different translations.

Therefore, if each mode is generated only once per page, it is easy to mistake the model’s own run-to-run fluctuation for the effect of the protection policy.

This is also one of the most important methodological corrections from this round of experiments.

Upgrading the Static Context experiment to 3 independent repeats

To reduce the impact of single-output fluctuation, I did not continue the “one Default vs. one Static Context per page” experiment.

Instead, I fixed 5 pages:

generics/1
flowcontrol/8
methods/20
concurrency/7
methods/24

generics/1
flowcontrol/8
methods/20
concurrency/7
methods/24

Two modes:

Default
Static Context

Default
Static Context

Each mode was run 3 independent times per page:

5 pages × 2 modes × 3 repeats
= 30 planned samples

5 pages × 2 modes × 3 repeats
= 30 planned samples

R2 and R3 also specifically employed interleaved execution:

The Default and Static Context for the same page + repeat were run back-to-back as much as possible, while randomly deciding which mode executed first.

Network failures and validator failures were all retained as-is, with no re-runs and no re-selecting “better-looking results.”

The final structural results were:

               Default     Static Context
generics/1      V / P / V   V / V / V
flowcontrol/8   P / P / P   P / P / P
methods/20      P / P / P   N / P / P
concurrency/7   P / P / N   V / P / N
methods/24      P / P / P   P / P / P

               Default     Static Context
generics/1      V / P / V   V / V / V
flowcontrol/8   P / P / P   P / P / P
methods/20      P / P / P   N / P / P
concurrency/7   P / P / N   V / P / N
methods/24      P / P / P   P / P / P

Where:

P = validator pass
V = API success, validator fail
N = network/API failure

P = validator pass
V = API success, validator fail
N = network/API failure

Overall results:

Default：
planned          15
API success      14
network failure   1
validator pass   12
validator fail    2
usable/planned 12/15

Static Context：
planned          15
API success      13
network failure   2
validator pass    9
validator fail    4
usable/planned  9/15

Default：
planned          15
API success      14
network failure   1
validator pass   12
validator fail    2
usable/planned 12/15

Static Context：
planned          15
API success      13
network failure   2
validator pass    9
validator fail    4
usable/planned  9/15

This sample size is still insufficient to prove that Static Context definitely reduces structural reliability.

This is especially true given the network failures, and the previous reproducibility experiments already confirmed that the validator results for identical requests can also change.

But at least it can be confirmed now:

Static Context did not show a clear structural reliability advantage.

Next, it is more important to look at language quality.

No longer picking the “best result,” but bringing all valid translations for blind review

This time, I did not pick a “representative translation” from each mode.

All candidates that passed the validator entered the anonymous quality materials.

The pages that ultimately entered the blind review were:

flowcontrol/8    6 candidates
methods/20       5 candidates
concurrency/7    3 candidates
methods/24       6 candidates

flowcontrol/8    6 candidates
methods/20       5 candidates
concurrency/7    3 candidates
methods/24       6 candidates

generics/1 had only one valid candidate, so it did not participate in the language quality comparison between the two modes.

All candidates were randomly shuffled within each page, displaying only:

CANDIDATE A
CANDIDATE B
CANDIDATE C
...

CANDIDATE A
CANDIDATE B
CANDIDATE C
...

Completely hidden:

• Default / Static Context;
• R1 / R2 / R3;
• attempt;
• worktree;
• token usage.

During the review, the complete ranking was locked in first, and only then was the separately saved mapping opened.

methods/24 showed very obvious run-to-run fluctuation

methods/24 had a total of 6 valid candidates.

Among them, Candidate D’s expression was:

But we will use the predefined implementations `color.RGBA` and `color.RGBAModel`,
without delving into these interfaces.

These interfaces and types are defined by the image/color package.

But we will use the predefined implementations `color.RGBA` and `color.RGBAModel`,
without delving into these interfaces.

These interfaces and types are defined by the image/color package.

Candidate C was:

But we will use the predefined implementations `color.RGBA` and `color.RGBAModel`
to ignore this.

These interfaces and types are specified by the image/color package.

But we will use the predefined implementations `color.RGBA` and `color.RGBAModel`
to ignore this.

These interfaces and types are specified by the image/color package.

Without knowing the identities at all, the final locked ranking was:

D > B > A > F > E > C

D > B > A > F > E > C

D ranked first, and C ranked last.

If only looking at the translations at this point, it would be easy to guess:

Perhaps D came from some better Prompt or Context design, while C came from another inferior strategy.

But after revealing the identities, the results were completely different.

D = Default R3
C = Default R1

D = Default R3
C = Default R1

That is to say:

Default R3 → 1st place
Default R1 → 6th place

Default R3 → 1st place
Default R1 → 6th place

The two translations came from the same page and the same Default mode.

This result almost encapsulates the most important finding of the entire round of experiments:

A single output from the same translation strategy cannot stably represent the language quality of that strategy.

Did Static Context make the translations better?

After revealing all anonymous mappings, the ranking distributions for several pages were as follows.

flowcontrol/8:

Default: 1, 4, 5
Static Context: 2, 3, 6

Default: 1, 4, 5
Static Context: 2, 3, 6

methods/20:

Default: 1, 3, 4
Static Context: 2, 5

Default: 1, 3, 4
Static Context: 2, 5

concurrency/7:

Default: 2, 3
Static Context: 1

Default: 2, 3
Static Context: 1

methods/24:

Default: 1, 3, 6
Static Context: 2, 4, 5

Default: 1, 3, 6
Static Context: 2, 4, 5

The most obvious characteristic is not that one group leads across the board, but rather:

The two modes are heavily interleaved.

There was no occurrence like:

Static Context: 1, 2, 3
Default: 4, 5, 6

Static Context: 1, 2, 3
Default: 4, 5, 6

this kind of obvious overall upward shift.

For example, in methods/24, Default itself simultaneously occupied:

1st
3rd
6th

1st
3rd
6th

While Static Context was:

2nd
4th
5th

2nd
4th
5th

Therefore, it has not been observed thus far that:

additionally exposing static code can stably improve GLM-5.2’s Chinese translation quality.

“More context isn’t better” is actually the wrong question

Reaching this point, I felt the original question itself needed to be reframed.

Initially, I asked:

Why didn’t giving the model more complete content result in higher translation quality?

The more accurate question now should be:

Was the additionally provided content actually new “useful semantic context”?

Although Default protection used 157 protected tokens, across the 7 representative pages:

hidden translatable English = 0

hidden translatable English = 0

The page-level natural language context that actually needed translation was already completely present.

What minimal-v1 or Static Context added was primarily not narrative flow, sentence relationships, or page topics, but rather:

• more machine syntax;
• link targets;
• directives;
• static code;
• technical structures.

The static code could indeed provide additional technical semantics, so a dedicated single-variable Static Context experiment was conducted.

However, in the current repeated experiments, this additional technical semantics did not show a stable translation quality advantage.

Therefore:

More raw bytes

More raw bytes

cannot simply be equated to:

more context helpful for natural language translation

more context helpful for natural language translation

This is not contradictory to whole-article WordPress translation being better than segmented translation.

WordPress segmentation genuinely loses natural language context.

Here, however, both modes were translating a complete present.Section from the very beginning.

Current engineering decision

After this round of experiments, I will not continue modifying the official translation workflow just to reduce protection for now.

The current official approach continues to use:

Default protected-token

Default protected-token

The reason is not that experiments have already proven:

Default translation quality is higher

Default translation quality is higher

There is no such evidence currently.

The more accurate reasons are:

1. Default already has mature structural protection and restoration capabilities;
2. it did not hide translatable natural language in these 7 representative pages;
3. minimal-v1 showed no stable language quality advantage;
4. separately adding back static code also showed no stable quality advantage;
5. GLM-5.2’s own run-to-run quality fluctuation is significant, so a single output is insufficient to evaluate a protection policy;
6. without clear quality gains, there is no need to weaken verified structural protection just because the “raw content looks more complete.”

minimal-v1 and:

--dev-static-context

--dev-static-context

will continue to be retained as development experiment capabilities.

They are not useless branches.

It is precisely these experimental modes that helped me confirm:

what the protected tokens actually obscured, and which quality differences previously thought to be caused by the protection policy might actually just be the model’s own run-to-run fluctuation.

One last important change: translation strategies can no longer be evaluated using a “single translation”

The impact of this round of experiments on the project’s subsequent testing methodology might be even greater than Static Context itself.

Going forward, if I need to compare two Prompts, two protection policies, or two translation workflows, I will no longer simply adopt:

Call once per page per mode
→ Two translations A/B
→ See which is better

Call once per page per mode
→ Two translations A/B
→ See which is better

Because it has now been actually verified that:

Identical Request
→ 5/5 pages Response different

Identical Request
→ 5/5 pages Response different

Even the same Default could produce both 1st and last place in the blind review within methods/24.

A more reliable approach should be:

Fixed pages
×
Fixed modes
×
Multiple independent runs
→
Tally structural reliability separately
→
Full anonymous review of all validator-passed candidates
→
Reveal modes at the end

Fixed pages
×
Fixed modes
×
Multiple independent runs
→
Tally structural reliability separately
→
Full anonymous review of all validator-passed candidates
→
Reveal modes at the end

This method is more expensive, but at least it won’t easily mistake a single偶然 generated result for an architectural advantage.

This can be considered a more important takeaway from this A Tour of Go multilingual translation experiment than “whether or not to protect a few fewer tokens.”