【唐诗】《奉和中書舍人賈至早朝大明宮》 岑參：雞鳴紫陌曙光寒，鶯囀皇州春色闌。金闕曉鐘開萬戶，玉階仙仗擁千官。花迎劒珮星初落，柳拂旌旗露未乾。獨有鳳皇池上客，

永夜

没有不值得去解决的问题，也没有不值得去学习的技术！

Markdown Is Not Plain Text: Why Learn / Docs Need a New Parser and Structure Protection

Markdown Is Not Plain Text: Why Learn / Docs Need a New Parser and Structure Protection

作者：

永夜

在

Views: 10

After expanding the translation scope of A Tour of Go to include Learn and Docs, I quickly ran into a problem that couldn’t be solved simply by “handing the text over to the AI.”

Markdown pages look like text, but they are actually not just text.

A technical document might simultaneously contain regular paragraphs, headings, lists, links, inline code, code blocks, instructional code comments, directives, and various machine structures that the translation model cannot arbitrarily modify.

If you hand the entire Markdown document to the AI as-is, the model might very well produce a correct translation.

But “might” is not enough for a long-running automated translation system.

The real question that needs answering is:

How do we give the model enough linguistic freedom while ensuring that code, links, and Markdown structures are not broken?

This is also why, after incorporating Learn / Docs into v2, the parser and representation had to be further refined.

The difficulty of Markdown is not “that it contains some symbols”

If it were just a few #, *, and backticks, it wouldn’t actually be hard to handle.

The real trouble is that a single Markdown document often mixes two completely different types of content:

Natural language that needs translating

and

Structures that must preserve machine semantics

For example:

Plaintext
Use [`go test`](/cmd/go/#hdr-Test_packages) to run the tests.

This contains at least:

  • regular English;
  • inline code;
  • link text;
  • link destination;
  • Markdown link binding.

What really needs translating is the natural language.

go test cannot be translated.

The URL usually cannot be modified either.

But the final Chinese translation cannot be forced to mechanically maintain English word order.

A reasonable translation might be:

Plaintext
运行测试时，可以使用 [`go test`](/cmd/go/#hdr-Test_packages)。

In other words:

The structure must be stable, but the position of the structure is not necessarily fixed.

This issue actually already existed in the previous Tour translation.

The Tour stage did not require all protected tokens to strictly maintain their original English order.

As long as the necessary pair, binding, token integrity, and machine semantics were not broken, the target language should be allowed to express itself in its own natural word order.

So v2 is not the first time “reordering” has been allowed.

The real new problem is:

The structures in Learn / Docs that need to participate in this protection and reconstruction are more complex than in the Tour.

If you protect too little, the AI can easily “help” by modifying things it shouldn’t

Suppose you hand the entire Markdown document directly to the model.

The model might very naturally make modifications that seem harmless to human readers.

For example, adjusting a URL.

Changing the casing in code.

Translating an identifier into Chinese.

Mixing code comments with surrounding explanations.

Changing the boundaries of a fenced code block.

Or reorganizing a Markdown directive as if it were regular text.

Individually, these changes might seem very small.

But to a machine, they could mean:

The page fails to render properly.

Or:

The example code is no longer correct.

Or even:

The document appears to display fine, but links, directives, or other hidden semantics have changed.

So the system first needs to clarify:

What can be left to the language model to decide, and what must be protected by the machine.

But protecting too much also ruins translation quality

The other extreme is also problematic.

The simplest protection method is:

Replace all special structures with tokens, and then require those tokens to remain exactly in place.

For example:

Plaintext
Use ⟪TOKEN_1⟫ to run the tests.

Then require the Chinese result to strictly be:

Plaintext
使用 ⟪TOKEN_1⟫ 来运行测试。

This is indeed very easy for the machine to verify.

But if the natural expression of the target language requires adjusting the position, the rules start to conflict with language quality.

A more natural Chinese phrasing might be:

Plaintext
运行测试时，可以使用 ⟪TOKEN_1⟫。

If the validator flags this change as an error, the system ends up with:

Machine structures that are highly stable, but Chinese that increasingly reads like a projection of English word order.

This is not the result I want.

Therefore, the principle that must continually be upheld in v2 is still:

Protect semantics, not unnecessary positions.

The machine should dictate which relationships cannot be broken.

Not dictate how the translation must be arranged.

What really needs protecting are relationships

As I got further along, I increasingly felt that the name “protected token” actually misleads people into focusing on the wrong thing.

What really matters is not any single token itself.

But the relationship between the token and its surrounding structure.

For example, a link:

Plaintext
[Go documentation](https://go.dev/doc/)

What needs protecting is not just:

Plaintext
https://go.dev/doc/

It also includes:

Which piece of text is still the link text for this URL.

Take inline code as another example:

Plaintext
The `error` value...

error itself must preserve its code semantics.

But the Chinese could be:

Plaintext
`error` 值……

And its position in the sentence can be adjusted based on context.

So what the parser truly needs to retain is:

Plaintext
this is inline code

and:

Plaintext
this code belongs to this surrounding language structure

Not:

Plaintext
它原来在第 17 个字符，所以翻译以后还必须在第 17 个字符附近

This distinction later had a major impact on the validator’s design.

Pairs are also more important than individual tokens

Some structures must appear in pairs.

For example, a Markdown link:

Plaintext
[text](destination)

Or certain directives, placeholders, and internal protection markers.

Just checking:

All tokens still exist in the end

is actually not enough.

You might end up with:

Plaintext
token A 存在
token B 也存在

But:

A and B no longer belong to their original structural group.

This situation is particularly dangerous because a simple token count can still PASS.

So machine validation needs to gradually evolve from:

Plaintext
token exists

to:

Plaintext
pair / binding remains valid

This is also why structural protection in Markdown translation cannot be simply understood as “string replacement.”

It is actually preserving a subset of semantic relationships within the representation.

Link binding is a very typical example

Links are particularly well-suited for illustrating this problem.

Suppose the source has two links:

Plaintext
Read the [language specification](...) and the [memory model](...).

To express this naturally, Chinese can adjust the sentence:

Plaintext
可以继续阅读[语言规范](...)以及[内存模型](...)。

This is fine.

In more complex sentences, the overall position of the two links in the translation might even change.

But one thing cannot change:

“语言规范” cannot end up bound to the memory model’s URL.

Just checking:

  • Both URLs are still present;
  • Both pieces of link text are still present;

does not prove the result is correct.

You must further ensure:

The link text and its original destination binding have not been crossed.

This is the problem the parser / protector should truly be solving.

Complete code is yet another type of representation

Learn / Docs also contains content more complex than standard Markdown:

Complete code.

In a regular paragraph:

Plaintext
`fmt.Println`

versus what actually exists in a fenced Go code block:

Plaintext
fmt.Println("Hello")

Obviously, these cannot be handled the same way.

Complete code must first maintain correct Go semantics.

The vast majority of code should not be translated at all.

But code in instructional documentation might also contain:

Human-readable comments that need translating.

For example:

Plaintext
// Print the result.
fmt.Println(result)

Here:

Plaintext
fmt.Println(result)

must preserve code semantics.

While:

Plaintext
// Print the result.

might be instructional content that needs localizing.

So a complete code block cannot simply be judged as:

“Code, protect entirely.”

Nor can it be judged as:

“It contains English, so allow translating everything.”

It requires further understanding of:

What belongs to Go syntax, and what belongs to translatable teaching comments.

This type of representation is also one of the reasons the Learn / Docs parser needed continuous refinement later on.

Teaching comments easily expose the parser’s boundaries

Code comments seem easy to identify.

In Go there are:

Plaintext
// comment

and:

Plaintext
/* comment */

It seems like you just need to find the comment node and translate the text inside.

But once you get into complete instructional code, the situation is often more complex.

Comments might:

  • Occupy a single line;
  • Be located at the end of a statement;
  • Contain code identifiers;
  • Contain formatting conventions;
  • Be closely tied to the instructional meaning of the surrounding code.

This means the parser needs to know not just:

Here is a comment.

It also needs to know:

Whether this is instructional content that should be handed off to the language model, and how to safely put it back into the original code representation after processing.

If this layer of judgment is wrong, two types of failure easily occur.

One is:

A teaching comment that should have been translated is completely protected, leaving English in the Chinese page.

The other is even more dangerous:

While translating the comment, the complete code is accidentally altered.

Therefore, this kind of structure cannot rely on “the model should know what not to touch.”

The machine must first draw the boundaries of responsibility clearly.

Parser and Generation responsibilities cannot be mixed

There is actually a very important division of responsibilities here.

The Parser should be responsible for:

Determining structure.

Generation should be responsible for:

Determining language expression.

For example, the parser can tell Generation:

Plaintext
这里是一段可以翻译的自然语言。
这里是 inline code。
这里是一组 link binding。
这里是 protected code。
这里是一段可翻译的 teaching comment。

But the parser should not go on to dictate:

That the Chinese must follow the English word order.

Similarly, Generation can decide:

How to express a sentence more naturally in Chinese.

But it cannot decide:

“This URL looks ugly, I’ll go ahead and change it for you.”

If either oversteps its bounds, problems arise.

So the ideal relationship is:

Plaintext
Parser
↓
定义机器边界

Generation
↓
在合法边界内完成自然语言表达

Then reconstruction and the validator check:

Whether Generation has broken through these boundaries.

Reconstruction is just as important as Parsing

When many people think of a parser, they focus on:

How to break down the input.

But for an AI translation system, the other half is equally important:

How to reconstruct the result back into a valid page.

That is, the whole process is not:

Plaintext
Markdown
↓
Parse
↓
AI
↓
结束

It is more like:

Plaintext
Markdown
↓
Parse
↓
Protected / translatable representation
↓
Generation
↓
Reconstruction
↓
Machine validation
↓
Markdown

Only when reconstruction succeeds can we prove that the content returned by Generation can still be reassembled into a valid representation.

So the parser and reconstruction are actually a pair of capabilities.

One side defines:

Which structures are broken down.

The other side proves:

That these structures can ultimately be correctly reassembled.

Round-trip is a very important machine gatekeeper

This is why I later placed so much importance on the machine round-trip.

Before a TranslationUnit enters linguistic review, the machine should at least prove:

That it can be structurally restored.

For example:

  • Protected content is not lost;
  • Pairs are not broken;
  • Link bindings are not crossed;
  • Markdown structure is still valid;
  • Code representation is not corrupted;
  • Placeholders are not missing.

If these basic conditions fail, there is no need to have an Independent Reviewer judge the Chinese quality.

The Reviewer should spend their time on:

“Is this good Chinese?”

Not on:

“Can this Markdown still be parsed?”

Problems that can be determined by a machine should be resolved as much as possible before entering linguistic review.

A machine PASS still does not mean the translation is correct

Of course, we must also stay clear-headed about the other direction.

Assume:

Plaintext
ReconstructWorkflow = PASS

This only proves:

The representation can be correctly restored.

It does not prove:

That the Chinese is good.

It cannot even fully prove:

That the technical meaning is necessarily correct.

A translation could achieve:

  • All inline code correct;
  • All link bindings correct;
  • Markdown intact;
  • Code intact;
  • Placeholders intact;

But completely misunderstand the original English meaning.

In this case, all machine validations might PASS.

So the goal of the parser / protector is never to replace the Reviewer.

It just tries to layer the problems as much as possible:

Plaintext
机器可判断的问题
→ Machine Validation

语言和技术表达问题
→ Independent Reviewer

Each handles the problems best suited to it.

Why an internal slot cannot become a formal TranslationUnit

This brings us back to the previous article about Page TranslationUnits.

To accomplish its work, the parser might internally need to break out many slots.

For example:

Plaintext
slot 1 = paragraph text
slot 2 = inline code
slot 3 = link text
slot 4 = link destination
slot 5 = teaching comment
...

These slots are very useful to the machine.

But they should not become formal linguistic governance units because of it.

Because the parser will evolve.

Today the parser breaks a piece of content into 4 slots.

Later, a more accurate way to represent it might use 6.

If slots were formal TranslationUnits, a pure parser improvement would cause massive changes to linguistic identities.

This is clearly unreasonable.

So v2 gradually clarified:

The physical Page is the formal Page TranslationUnit.

Internal slots are just parsing, protection, reconstruction, and diagnostic tools.

The parser can evolve.

Formal linguistic history should not be meaninglessly rewritten as a result.

Improving the parser also does not mean all Pages need retranslation

This is actually very similar to the earlier thinking about glossary affected scope.

Suppose we discover the parser has an issue representing a certain type of teaching comment.

After fixing the parser, the simplest approach is of course:

Retranslate all Learn / Docs Pages entirely.

But if only a few pages actually contain this representation, there is no need to throw out the entire package and start over.

A more reasonable approach is still to determine:

Which Pages are truly affected.

Existing TranslationUnits that are unaffected and can still be reliably validated under the new parser / validator should be retained.

Behind this is still an increasingly clear principle across all of v2:

Infrastructure changes do not automatically mean all linguistic output is invalidated.

What truly needs handling is the part affected by the change.

The goal of structural protection is not to make the validator easier to write

In this parser work, I felt the easiest place to go wrong was:

Designing overly restrictive translation constraints just to make the validator simple.

For example:

All tokens must maintain their original order.

This kind of rule is of course very easy to verify.

You just compare the sequence.

But that does not mean it is a correct linguistic rule.

If natural Chinese expression requires changing the order, the validator should not prohibit this change just for implementation convenience.

In other words:

The validator should verify semantics that truly cannot be broken, rather than disguising implementation convenience as linguistic constraints.

This statement is very important to me.

Because many machine rules seem perfectly reasonable at first.

It is only when a legitimate natural language expression actually appears but is rejected by the machine that you realize:

Maybe the translation wasn’t wrong, but the validator’s model was too narrow.

But loosening the validator also cannot turn into “anything goes”

Of course, the other extreme is equally dangerous.

After discovering that strict sequence checks harm natural word order, you cannot just switch to:

As long as the tokens are there, it’s fine.

If you do that, pairs, bindings, link destinations, and structural relationships can all become misaligned.

So a more reasonable improvement is not:

Plaintext
strict
→
loose

But rather:

Plaintext
错误的约束
→
更准确的约束

For example, changing from:

Tokens must remain in their original positions.

to:

Tokens can be moved, but valid pair / binding / machine semantics must be maintained.

This is not lowering the validation strength.

In fact, the requirements become more accurate.

It is just that the object of machine validation shifts from:

Surface positions

to:

The semantic relationships that truly need preserving.

Examples will be stricter than regular Pages

Another type of content needs distinguishing here:

Examples.

The natural language in regular Pages often requires a fair amount of word order freedom.

But Examples, especially those containing Go code, expected output, or machine-related tokens, have stricter representation requirements.

So “allowing a Page to adjust the position of valid bindings based on natural language” cannot simply be generalized into:

All content can be moved freely.

You ultimately still need to judge based on the TranslationUnit kind:

What belongs to linguistic expression, and what belongs to program semantics.

That is, structural protection is not a single site-wide string rule.

It must understand content types.

Eventually, the parser becomes more like a machine contract

Initially, I understood the parser as:

A tool for parsing Markdown.

But as v2 went deeper, I felt a more accurate understanding might be:

It establishes a machine contract between the source, Generation, and the final page.

This contract specifies:

What can change.

What cannot change.

What content can be moved.

What bindings must be maintained.

What structures must exist in pairs.

What machine semantics must be fully restored after a round-trip.

Generation has linguistic freedom within this contract.

The Validator is responsible for proving it did not overstep.

In this way, the parser is no longer just a text preprocessor.

It is actually, within the entire AI translation pipeline:

Part of the machine safety boundary.

This is also why the parser will continually iterate

For this kind of system, it is unlikely you will define a perfect representation on the first try.

Real content will constantly expose new edge cases.

At first, you might think:

Just protecting inline code is enough.

Later you realize you need link binding.

Later still, you find there are teaching comments within complete code.

As you keep going, you might discover that some directive or nested structure requires a new representation.

This iteration itself is not frightening.

What truly needs avoiding is:

Every parser improvement forcing all stable linguistic output to be redone.

So the layering between Page identity, internal representation, and affected scope is very important.

The parser can continue to improve.

But it should try to keep truly unaffected TranslationUnits stable.

What we ultimately want is actually a balance

At this point, I think the core issue of the Learn / Docs parser can be condensed into one sentence:

You cannot treat Markdown as plain text, but you also cannot lock down the natural language just to protect the Markdown.

Too free:

Plaintext
AI may break structure

Too strict:

Plaintext
machine-safe but unnatural translation

What is truly needed is:

Plaintext
Protect machine semantics
        +
Allow natural language
        +
Verify round-trip

The machine is responsible for ensuring:

That things that shouldn’t break don’t break.

Generation is responsible for ensuring:

That the Chinese reads like Chinese.

The Independent Reviewer is then responsible for judging:

Whether the final content is truly accurate, natural, and suitable for technical documentation.

These three layers of responsibility cannot substitute for one another.

From Tour to Learn / Docs, what changes is the complexity of the representation

So there is one last point that needs special clarification.

This v2 update is not going from:

Strictly maintaining original order in the Tour stage

to:

Allowing natural word order only in Learn / Docs.

The Tour stage already allowed adjusting the order of expression based on the target language, provided machine semantics were maintained.

The real new challenge brought by Learn / Docs is:

There are more, and more complex, types of representations that need protecting and reconstructing.

Therefore, the pre-existing principle:

Machine protection cannot sacrifice natural language.

In v2, needs to be expanded to cover:

  • Markdown;
  • inline code;
  • links;
  • pair / binding;
  • complete code;
  • teaching comment;
  • directive;
  • and more structured content.

It is also during this process that I became increasingly certain:

Good structural protection should not make the reader feel that structural protection ever existed.

What the end user sees should still be a natural, coherent Chinese technical document.

As for how many slots, tokens, bindings, and reconstruction steps the machine used internally to achieve this, those should remain at the implementation layer.

系列导航

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者，自 2013 年起持续运营个人技术博客，记录工作与个人项目中遇到的真实问题，以及解决问题过程中的探索、实践和思考。

除了技术写作，我也在持续开发和维护自己的独立项目，探索新技术和新工具在实际场景中的应用。我始终相信，没有不值得去解决的问题，也没有不值得去学习的技术。

如果这篇文章对你有所帮助，欢迎浏览博客中的其他文章，也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系

更多文章