In designing Go Learning & Documentation v2, there was a problem that seemed straightforward but actually affected the entire translation workflow:
How exactly should a Markdown page be divided into TranslationUnits?
The reason this question deserves its own discussion is that, from an implementation perspective, the answer seems very simple.
Markdown is inherently structured content.
A heading is a block.
A paragraph is a block.
A list item is a block.
A blockquote is a block.
Even a teaching comment within a code block can be extracted individually.
Since the program needs to parse these structures anyway, the most direct approach seems to be:
Turn every small piece of content into a TranslationUnit.
Doing this has many apparent benefits.
Every TranslationUnit is very short.
If translation fails, you only need to retry a small fragment.
Modifying a paragraph doesn’t require reprocessing the entire article.
Machine validation, diffs, and caching all seem simpler.
But after actually bringing Learn and Docs into v2, I ultimately did not adopt this design.
The current formal rule is instead:
One physical Page is one formal Page TranslationUnit.
The Markdown internally is, of course, still parsed by the parser.
But these internal structures are only for:
- Parsing;
- Protection;
- Validation;
- Reconstruction;
- Troubleshooting.
They are not formal language governance units.
Why is this the case?
The reason is actually closely related to translation quality itself.
The segmentation method programs prefer isn’t necessarily suitable for natural language
From a program’s perspective, fine granularity is almost always tempting.
Suppose a document has:
Title
Paragraph A
Paragraph B
- List item 1
- List item 2
Code block
Paragraph CIt could easily be turned into:
TU-001 = Title
TU-002 = Paragraph A
TU-003 = Paragraph B
TU-004 = List item 1
TU-005 = List item 2
TU-006 = Code comment
TU-007 = Paragraph CEach unit gets independent Generation, independent validation, and independent QC.
From a data structure perspective, it is very neat.
But once translation actually begins, another side quickly becomes apparent.
Natural language is not an independent field in a database.
A paragraph often depends on preceding content.
Expressions like “it,” “this approach,” “the previous example,” “next,” and “therefore” inherently require context.
There are also relationships between headings and body text.
List items might share an omitted subject.
A sentence might be perfectly correct when translated in isolation, but when put back into context, it might seem repetitive, abrupt, or even shift the focus of the entire paragraph.
So a significant contradiction emerges here:
Machines want to break content down finer, but language quality often requires a more complete context.
v2 ultimately chose to prioritize protecting the latter.
A correct sentence doesn’t mean the whole page reads correctly
This issue is especially prominent in AI translation.
If each paragraph is submitted to Generation separately, the model can certainly translate every paragraph with correct grammar.
But after they are finally stitched back together, many small issues can arise.
For example, the previous paragraph might have explicitly stated:
“Interfaces in Go are implicitly implemented.”
If the next paragraph is translated in isolation, it might re-explain the entire background.
On its own, it isn’t wrong.
When read consecutively, it feels verbose.
Or consider two consecutive paragraphs in English that respectively use:
this value
itIf the second paragraph is processed separately, Generation might not know which concept it actually refers to, and can only guess based on the current fragment.
Another example is a common structure in tutorial pages:
First, create a new file.
Then add the following code.
This program...If these are processed separately, cohesive relationships like “this program,” “the above code,” and “then” are more likely to have unnatural issues.
Machine validation typically cannot catch these problems.
Because:
- The Markdown structure is correct;
- Not a single protected token is missing;
- All inline code is present;
- No links are lost;
- The placeholders are exactly identical.
But the language itself has lost its continuity.
Therefore, the TranslationUnit boundary cannot be determined solely by:
How far the machine can independently validate
It must also consider:
How much context a Reviewer actually needs to see to judge whether the translation is truly good.
Reviewers also need to see the complete page
This affects not only Generation but also Independent Reviewers.
Suppose a Page is split into 30 TranslationUnits.
A Reviewer might receive:
TU 1
source: ...
candidate: ...
TU 2
source: ...
candidate: ...
TU 3
source: ...
candidate: ...Each one can be individually graded A, B, C, or D.
The problem is:
If all these small units receive an A, it does not mean the reassembled Page is necessarily an A.
For example:
The first paragraph uses “concurrent.”
The third paragraph switches to “parallel.”
The fifth paragraph is written as “simultaneous execution.”
Each small fragment might not have obvious errors when viewed independently.
But if the source is actually discussing the same concept throughout, the entire page has already experienced terminology drift.
Or, in the same page, the text might use “you can…” earlier on, and then suddenly switch to “you could…” later.
When reviewing each paragraph separately, these problems are very easy to miss.
There are also more subtle cases:
Two adjacent paragraphs might repeat the same explanation.
You can only truly see this issue when they are placed together.
Therefore, if the Reviewer’s formal governance unit is the Page itself, the A/B/C/D grades given by the Reviewer truly correspond to the content the end user will actually read.
This is also a principle I have increasingly valued in v2:
Formal Quality Checks should align as closely as possible with the content boundaries actually consumed by end users.
What the user opens is a page.
Not dozens of internal strings.
But internally, the machine still must break things down
This easily leads to another misunderstanding:
Since one Page is one TranslationUnit, does that mean the program treats the entire Markdown as one large block of plain text?
Of course not.
In fact, the more we insist on Page-level TranslationUnits, the more important the internal parser and protection mechanisms become.
Because a single page might simultaneously contain:
- Markdown headings;
- paragraphs;
- ordered / unordered lists;
- inline code;
- fenced code blocks;
- link text;
- link destinations;
- directives;
- HTML;
- placeholders;
- teaching comments;
- other content that must maintain machine semantics.
It is impossible to hand all this content over to the model for free rewriting.
So internally, the Page still needs to be parsed into finer structures.
You can think of it like this:
Physical Page
│
├── translatable slot
├── protected structure
├── translatable slot
├── inline-code binding
├── protected code
├── translatable comment
└── link bindingThese slots are crucial for the program.
They determine what can be translated and what cannot be touched.
They also help the machine re-check after Generation returns results:
- Whether the tokens are intact;
- Whether the code has been broken;
- Whether the link is still bound to the correct position;
- Whether the Markdown can be reconstructed;
- Whether the directive is still valid.
But the existence of these slots does not mean they should become formal TranslationUnits.
This is an important layering in v2.
Internal slots are an implementation detail, not a governance namespace
If the internal parser slices out 20 slots, creating 20 formal TranslationUnits seems natural.
But doing so creates a long-term problem:
The internal implementation starts to control the language workflow in reverse.
Suppose today the parser splits a complex paragraph into 3 slots.
Tomorrow, after a parser improvement, it becomes 5.
If the slots themselves are formal TranslationUnits, then a single parser improvement could lead to:
- TranslationUnit identity changing;
- Historical QC being difficult to reuse;
- Revision evidence needing to be remapped;
- Snapshot structure changing;
- Reviewer scope changing;
- Even a large number of stale states despite no actual language changes occurring.
This is essentially leaking parser implementation into the governance layer.
A physical Page is much more stable.
As long as that real page still exists, its identity as a formal TranslationUnit can continue to exist.
As for how many slots it is split into internally today, or how the parser improves tomorrow, that is the machine layer’s own business.
This is why I later increasingly made a point to separate the two:
The Page is the governance unit.
The slot is the parsing unit.
The two cannot be conflated.
This also allows the parser to continue evolving
This layering also has a very practical benefit.
After Learn / Docs were added, the parser could not be designed perfectly all at once.
Some structures were not fully covered initially.
Some teaching comments within complete code were only later found to need more detailed recognition methods.
Some bindings between inline code and natural language also need continuous adjustment based on real content.
If the formal TranslationUnit were strongly bound to the parser’s internal structure, every parser improvement could affect a large amount of existing history.
But now, as long as:
Page identity remains unchangedThe internal parser can continue to upgrade.
Old candidates can also be re-evaluated by the new parser / validator to determine if they are still valid.
Only the content actually affected needs to be processed.
Unaffected Pages do not need to be retranslated just because of an internal parser refactor.
From a long-term maintenance perspective, this isolation is crucial.
Natural word order is also why I didn’t want to slice TranslationUnits too finely
There is another issue, which is natural word order.
English and Chinese do not always have the same expression order.
Especially when a sentence contains:
- inline code;
- links;
- protected tokens;
- technical terms;
- placeholders;
If the content is sliced too finely, a problematic tendency easily emerges:
Forcing the translation to follow the local English word order to protect machine structures.
This kind of result is structurally very safe.
But the Chinese might be incredibly awkward.
The project has since gradually clarified:
As long as complete pairs, inline code, link bindings, and machine semantics are not broken, Generation should be allowed to adjust their positions according to natural Chinese word order.
For example, the English might be:
Use `go test` to run the tests.A natural Chinese expression could be:
运行测试时，可以使用 `go test`。go test does not need to mechanically remain in the original word order position.
If the TranslationUnit is sliced too finely, even splitting the natural language around inline code into different fragments, this flexibility drops significantly.
Page-level TranslationUnits preserve enough linguistic space for Generation.
The program is responsible for protecting boundaries.
The language model is responsible for natural expression.
Each handles what they are good at.
Exact revision is therefore also easier to keep “locally complete”
Using the Page as the formal TranslationUnit does not mean that upon finding a small issue, the entire page must be retranslated.
These are two different levels of problems.
A Reviewer can point out a specific finding within a Page.
For example:
The terminology in a certain sentence is inaccurate.
Or:
A certain part translated the teaching meaning in reverse.
What Generation receives is this complete Page, along with the corresponding finding.
But the revision requirement is not:
“Retranslate the entire article.”
Rather:
finding-anchored + locally complete.
That is, only modifying the local part actually affected by the issue, while ensuring that after this local modification, it remains coherent with the surrounding context.
This simultaneously preserves two things:
Complete Page context
and
precise revision scope
This is more appropriate than breaking the formal TranslationUnit down sentence by sentence.
Because fine-grained revision does not require fine-grained governance units.
Smaller TranslationUnits aren’t necessarily truly more cost-effective
When first seeing Page-level TranslationUnits, a very natural concern is:
If a page is this long and there’s only one error, isn’t it wasteful to reprocess the entire Page?
But the actual workflow is not simply “regenerate the entire page.”
The places that truly consume costs also include:
- export;
- identity management;
- Reviewer Bundle;
- Quality Check record;
- Snapshot;
- revision eligibility;
- re-QC;
- history;
- pending state;
- finalization.
If a page is split into dozens of formal TranslationUnits, although each unit is smaller, the number of governance objects increases dramatically.
A 20-paragraph Page might become 20 or more TUs.
49 Learn / Docs Pages could quickly become hundreds or even thousands of TranslationUnits.
Then each one needs to be maintained separately:
- identity;
- QC status;
- revision state;
- evidence;
- history.
System complexity does not just disappear.
It simply shifts from “individual TranslationUnits being relatively large” to “a massive number of TranslationUnits.”
For a documentation translation task that requires context, this trade-off is not necessarily worth it.
One Page, one TranslationUnit is not an absolute truth
Of course, it should also be clarified here:
“Page = TranslationUnit” is not a universal truth applicable to all translation systems.
If what is being processed is:
- UI buttons;
- menu items;
- error prompts;
- standalone messages;
- database fields;
- software resource strings;
Small-grained TranslationUnits are often very reasonable.
Because they are inherently independently consumed content.
But Learn / Docs is technical documentation meant for continuous reading.
The natural unit users consume is the page.
So in this project, I ultimately preferred to have the formal TranslationUnit align with the natural boundary of the final content, rather than the parser’s most convenient boundary.
The principles ultimately formed in v2
Looking back now, this design can actually be condensed into a very simple sentence:
The machine can parse content however it needs, but the machine’s internal segmentation method cannot be allowed to dictate the governance boundaries of human language.
For Learn / Docs:
Physical Page
=
Formal Page TranslationUnitAnd:
Internal slots
=
parser / protection / reconstruction / validation
implementation detailsEach bears its own responsibilities.
This allows the machine to do fine-grained structural protection while letting Generation and Reviewer always see the complete linguistic context.
At the same time, as the parser continues to evolve, there is no need to overturn the formal language history just because internal slots have changed.
Allowing natural word order didn’t just start in v2
A special point needs to be made here.
Allowing translations to adjust word order based on the target language is not a capability added only in v2.
In the previous A Tour of Go translation, it was already impossible to mechanically require protected tokens, inline code, or other protected structures in the translation to strictly maintain their original English positions.
As long as necessary tokens are not lost, paired structures and bindings are not broken, and the code and machine semantics remain correct, Generation should prioritize generating natural target language.
For example:
Use `go test` to run the tests.Translated to:
运行测试时，可以使用 `go test`。This kind of order adjustment was inherently allowed.
Therefore, the real change in v2 is not:
Going from “strictly maintaining order” to “allowing natural word order.”
Rather, as Learn / Docs are brought into the system, the representations that need to be handled are much more complex.
A single Page might simultaneously contain:
- Markdown structures;
- inline code;
- link text and link destinations;
- fenced code;
- teaching code comments;
- directives;
- HTML;
- placeholders;
- other machine structures needing protection.
The pre-existing principle:
Protect machine semantics, but do not sacrifice natural language for it.
Here, it needs to be applied to a more complex content model.
This also brings up a new architectural question:
To protect these structures, the parser can slice the Page very finely; but should these internal slices also become formal TranslationUnits?
The final answer v2 gives is no.
Internally, the machine can continue to subdivide the representation to accomplish parsing, protection, validation, and reconstruction.
But formal language governance remains at the complete Page level.
In other words, v2 did not change the natural language principles already adopted during the Tour phase, but further separated the two layers of responsibility:
The machine is responsible for protecting the semantic boundaries that cannot be broken.
Generation can still express things naturally in the target language within these boundaries.
The Reviewer judges the final language quality based on the complete Page.
This is what truly needed to be further clarified after Learn / Docs were added.