Why Visual Inspection Fails for Code Block Comments: How v2 Counts and Verifies Translation Completeness

I previously wrote another post:

Markdown is not plain text: Why Learn / Docs needs a new parser and structure protection

That post mainly discussed a fundamental issue:

Not all content on a page can be handed directly to a translation model.

Titles, body text, and link text can be translated.

Code keywords, identifiers, paths, URLs, directives, and many machine semantics cannot be changed arbitrarily.

Comments inside code blocks are even more special.

Some comments are just machine directives and must be kept as is.

Some comments are natural language genuinely explaining the code to learners, and should be translated.

Once this is done, the problem might seem solved:

Open up the comments that can be translated, and continue protecting the code that cannot be translated.

But later I realized this was far from enough.

Because there is a much harder problem to discover:

How do we prove that all code comments originally supposed to enter the translation workflow were actually found?

If a teaching comment is missed, the translation model will have no idea it exists.

What quality reviewers see might also only be what the parser has already picked out.

The final page could:

• Have a completely correct structure;
• Have runnable Go code;
• Pass all protection rules;
• Have good translation quality for the text that already entered the translation workflow;

But still silently leave an untranslated English explanation inside a code block.

This post organizes why v2 later added another layer:

Code comment completeness audit.

“Being able to translate code comments” and “not missing code comments” are two different problems

These two problems are very easy to conflate.

The first problem is:

Can a certain code comment be translated?

For example:

// Print the result.
fmt.Println(result)

// Print the result.
fmt.Println(result)

Here:

Print the result.

Print the result.

is clearly explaining the code to the learner.

This kind of content can enter translation as a teaching comment.

But something like this:

//go:generate tool

//go:generate tool

is not an ordinary natural language explanation.

It is a Go directive, and just because the target language is Chinese does not mean its content should be translated away.

So the first thing the parser needs to do is classify:

Teaching natural language comments → can be translated
Machine semantic comments → must be protected
Unknown or cannot be reliably determined → do not guess

Teaching natural language comments → can be translated
Machine semantic comments → must be protected
Unknown or cannot be reliably determined → do not guess

This part solves:

Translation boundaries.

But the second problem is completely different:

How many teaching comments that should be translated are actually in the source page?

If the parser was supposed to find 20 comments but only found 19, then no matter how strict the subsequent process is, it will not discover that the 20th comment never entered translation.

So:

“All discovered comments were handled correctly” does not prove “all comments were discovered.”

The most dangerous omissions often do not report errors

The reason this problem deserves its own completeness check is that it often does not cause an obvious failure.

Suppose a page contains:

func main() {
    // Calculate the total.
    total := a + b

    // Print the result.
    fmt.Println(total)
}

func main() {
    // Calculate the total.
    total := a + b

    // Print the result.
    fmt.Println(total)
}

If the parser only recognized the second one:

Print the result.

Print the result.

Then the generated Chinese page might become:

func main() {
    // Calculate the total.
    total := a + b

    // 打印结果。
    fmt.Println(total)
}

func main() {
    // Calculate the total.
    total := a + b

    // 打印结果。
    fmt.Println(total)
}

The code is completely valid.

The page can also render normally.

Machine structure validation might all pass.

If the reviewer mainly looks at translation candidates, they also might not realize:

The first comment actually never entered the translation workflow.

This is the kind of error I now feel most needs to be guarded against:

Silent omission.

It does not fail immediately like a syntax error.

Instead, it can easily follow the content all the way into production.

Only checking the target translation is not enough

At first, it is easy to think of a checking method:

Scan the Chinese page to see if there is still English in the code blocks.

But this method is not reliable.

Because the code itself will inherently contain a lot of English.

For example:

func GetAlbumByID(id string) (*Album, error) {

func GetAlbumByID(id string) (*Album, error) {

Inside it:

• GetAlbumByID
• id
• string
• Album
• error

should all be kept as is.

API names, field names, package names, file names, JSON fields, operating system names, etc., can also legitimately remain in English.

So you cannot simply do:

Target page still has English → definitely missed translation

Target page still has English → definitely missed translation

Similarly, you also cannot assume:

Target page has no obvious English → definitely complete

Target page has no obvious English → definitely complete

A truly reliable determination must start from the source content.

The first step is not checking the translation, but building a unique source comment inventory

The approach v2 later adopted is:

First build a complete code comment inventory from the canonical source content, then compare the target language result against it.

This order is very important.

The source content is responsible for answering:

What code blocks are actually in the page?

Which code blocks actually contain comments?

Which comments are teaching natural language?

Which are machine semantics?

Which code blocks cannot be safely determined yet?

Only after this layer is proven complete does the subsequent “are there any missed translations” become meaningful.

The project now specifically provides a code comment audit entry point, for example:

tour-i18n code-comments unsupported --output-dir /tmp
tour-i18n code-comments coverage --output-dir /tmp
tour-i18n code-comments audit --output-dir /tmp

tour-i18n code-comments unsupported --output-dir /tmp
tour-i18n code-comments coverage --output-dir /tmp
tour-i18n code-comments audit --output-dir /tmp

These commands themselves are read-only engineering audits.

They can output diagnostic JSON and text evidence, but will not modify:

• Candidate translations;
• Candidate Snapshot;
• Quality review;
• authorization;
• Content package archiving;
• Integrated site status;
• activation.

That is to say:

First prove the problem, then decide whether to enter the formal revision process.

Rather than scanning something and immediately starting to modify content.

Why you can’t just directly search for //

If it were only Go code, it seems like you could use a very simple method:

Search for //

Search for //

Then tally up the content following it.

But this method will quickly run into problems.

For example, a string might contain:

url := "https://go.dev/doc/"

url := "https://go.dev/doc/"

Here:

//

//

is of course not a comment.

Similarly, the page might also contain:

• JSON;
• go.mod;
• Command lines;
• Terminal output;
• SQL;
• HTML <pre>;
• Path examples in documentation;
• Plain text code blocks.

Different languages have completely different definitions of comments.

For example:

#

#

might be a comment in some languages.

But in shell parameters, URL fragments, plain output, or text examples, it does not necessarily represent a comment.

So completeness checking cannot rely on:

“Seeing a certain symbol and guessing it is a comment.”

It must rely on:

Proven source language and complete syntax.

The language itself also cannot be guessed by file path

Another shortcut that easily occurs is:

This page is in the Go tutorial, so the code blocks should all be Go.

Or:

// appears in it, so it should be Go.

This is equally unreliable.

The principle v2 currently uses is more conservative.

When determining the code block language, it prioritizes using:

• The language explicitly declared by the Markdown fence;
• Explicit language-* in HTML;
• Formal semantics from an existing source parser or metadata;
• And finally, the complete acceptance of the entire code block by a strict parser.

It cannot use:

• Target translation content;
• locale;
• Preview;
• File path guessing;
• A few ASCII keywords;
• Loose regular expressions;

to prove the source code language.

This point is very important.

Because:

Once the language is guessed wrong, the subsequent “no comments” can also be a false conclusion.

“No comments recognized” cannot automatically equal “there are no comments here”

This is probably the one principle in the entire completeness design that I agree with the most.

Suppose we encounter a code block.

The current parser does not recognize its language.

The simplest implementation might yield:

Found comments count = 0

Found comments count = 0

But what does this 0 actually represent?

It might represent:

The parser fully understood this code and proved there are indeed no comments in it.

Or it might just be:

The parser simply did not know how to analyze it.

These two states are completely different.

So v2 does not allow turning:

Unable to analyze

secretly into:

Proven to have no comments

Unknown languages must be explicitly retained as “unsupported” or “ambiguous”.

As long as there are still code blocks that cannot be proven, you cannot claim:

Code comment coverage is already complete.

This is consistent with the principle the entire project has always adopted:

When it cannot be proven, treat it as a failure, do not guess.

Every code block must receive a clear conclusion

To prevent code blocks from silently disappearing during the tallying process, v2 gives each original code block a clear status.

This can be roughly understood as the following categories.

Comments already extracted

Means the language is supported, and precise comment slots have been established.

Proven no comments via complete syntax

Not “nothing was found by searching”, but the parser has proven there are no comments in it.

Terminal session record

Clearly distinguishes between commands and output, and can no longer treat #, //, or -- in the output as program comments.

Non-code output or literal

The source content itself has already proven this is output, data, or a text example, not program source code that needs further analysis.

Machine semantics fully protected

The content clearly belongs to machine semantics, and the entire block is kept unchanged.

Ambiguous, temporarily unsupported

If it cannot be reliably determined, an unresolved status is explicitly left.

The most important thing here is not the category name itself.

But rather:

Every original code block must appear, and can only receive one conclusion.

You cannot:

• Drop blocks;
• Add extra blocks;
• Change the order;
• Re-guess the type using the target translation;
• Default unknown blocks to zero comments.

The focus of the statistics is not just the “total comment count”

If you really want to avoid omissions, just tallying:

How many comments there are in total

How many comments there are in total

is still not enough.

More valuable statistics should simultaneously include:

• How many code blocks there are in total;
• How many code blocks successfully extracted comments;
• How many were proven to have no comments via syntax;
• How many are terminal records;
• How many are output or literals;
• How many are machine semantics;
• How many still cannot be reliably classified;
• How many code blocks there are for each source language;
• How many teaching comments there are in total;
• How many comments in the target translation are still exactly the same as the source content;
• Whether there are any missing comments;
• Whether there are any extra comments;
• Whether there are any code structure or protected content changes.

Only statistics like this truly answer:

Is the entire source coverage complete?

Rather than just giving a nice-looking number.

Comment identity cannot rely solely on line numbers

There is also another easily overlooked problem:

How do you confirm that the source comment and the comment in the target translation are the same one?

If only line numbers are used, it will be very fragile.

In a Markdown page, as long as a few lines of content are added at the beginning, all subsequent line numbers will change.

So v2’s comment identity is bound to a more stable context, for example:

• Content package;
• source path;
• parent kind / ID;
• parser block coordinate;
• comment ordinal;
• position of the previous lexical token;
• source semantic kind.

The goal here is not:

To give each comment a permanent ID card.

Because after the source code actually undergoes semantic changes, the correspondence may inherently need to be re-established.

What is really needed is:

To be able to precisely prove who this comment is within the current canonical source content.

After the source changes, old statistics cannot automatically continue to be used

This is actually connected again to the invalidation propagation problem I wrote about earlier.

Suppose yesterday we completed:

Code block count = X
Teaching comment count = Y
Unknown code blocks = 0

Code block count = X
Teaching comment count = Y
Unknown code blocks = 0

Today the canonical source content changed.

Then we cannot simply assume:

It was already tallied yesterday, so we will just keep using it today.

Because after the source content or code structure changes:

• Code blocks might be added;
• Comments might be added;
• Comments might be deleted;
• The structure where comments are located might change;
• Old correspondences might be invalidated.

So the source inventory also has its own currentness.

Statistical results are only meaningful when bound to the current source content.

Checking the target language must use the current official translation

After building a complete source inventory, the next step is to compare the target language.

Here, we also cannot just read files from some old directory.

Because the project might simultaneously contain:

• Historical Snapshots;
• Old candidates;
• Invalidated promotions;
• The current official candidate;
• Rendered results in Preview.

A true code comment completeness check must read:

Target content proven by the current official authority.

If it cannot be proven that a certain target translation is still the current version, it should explicitly output:

CURRENT_TARGET_UNAVAILABLE

CURRENT_TARGET_UNAVAILABLE

Instead of:

“There happens to be a file here, so let’s just use it for comparison.”

This is also why completeness auditing and historical recovery cannot be mixed together.

For Chinese, a teaching comment remaining completely unchanged is in itself a strong signal

Suppose the English source comment is:

// Return the vulnerability ID.

// Return the vulnerability ID.

And in the Chinese target page it is still:

// Return the vulnerability ID.

// Return the vulnerability ID.

If this comment has already been officially classified as:

teaching-comment

Then for a non-English target language like Chinese:

Completely retaining the English source text should in itself be treated as a failure by the machine.

Because at this point, a Reviewer is no longer needed to determine:

“Was this sentence missed in translation?”

The machine already knows:

1. It is translatable teaching natural language;
2. The current locale is not English;
3. The target text is exactly the same as the source text.

Therefore it can be determined:

This teaching comment did not actually enter the target language.

But for English variants, such as en-GB and en-CA, this exact carry might be completely reasonable.

So here we also need to know the base language of the target locale, rather than simply maintaining a hand-written list of “which locales are English”.

But “different text” also cannot prove the translation is correct

The reverse is also true.

If:

source comment ≠ target comment

source comment ≠ target comment

It only proves:

The bytes on both sides are not exactly the same.

It cannot prove:

The translation quality is correct.

For example, the original text:

Return the vulnerability ID.

Return the vulnerability ID.

The target content is changed to a completely wrong Chinese sentence.

The machine can only confirm:

It is no longer a verbatim copy of the English.

But naturalness, accuracy, and terminology still need to be judged by an independent Reviewer.

So the division of responsibilities here is very clear:

The machine is responsible for proving completeness and structural correspondence.

The Reviewer is responsible for judging language quality.

“Whether there are missed translations” should not be left for the Reviewer to guess first

I feel this point is particularly important.

If the machine can already determine:

• The source has a teaching comment;
• The corresponding position does not exist in the target;
• The target has an extra comment that cannot be corresponded;
• Protected content has changed;
• The code block structure is inconsistent;

None of these should be given to the Reviewer first to ask:

“Do you think this is a missed translation?”

Because these are deterministic structural facts.

The Reviewer should rather spend their time on:

• Whether the translation is accurate;
• Whether the expression is natural;
• Whether the terminology is appropriate;
• Whether the teaching intent is preserved;

these areas that genuinely require language judgment.

So:

Problems that can be determined by the machine should not consume manual language review.

A completeness failure also should not force the entire content package to be redone

If the statistics find:

Out of 49 pages, only 3 pages have missing teaching comments.

The simplest approach is still:

Regenerate the entire content package
→ Re-review all pages

Regenerate the entire content package
→ Re-review all pages

But this goes back to the very problem the previous posts have been trying to avoid.

A more reasonable approach is:

Identify affected pages
→ Only reopen these pages
→ Precisely revise the missing comments
→ Re-run machine validation
→ Independent re-review
→ Establish a new subsequent official status

Identify affected pages
→ Only reopen these pages
→ Precisely revise the missing comments
→ Re-run machine validation
→ Independent re-review
→ Establish a new subsequent official status

Other pages already proven to have:

• No source changes;
• Complete comments;
• A currently valid candidate;
• A previously still trusted A;

will continue to be inherited.

So code comment completeness ultimately also connects to the same:

affected-only

approach.

It is not creating another special workflow.

This also explains why internal comment slots cannot become new formal translation units

In order to tally comments, the system will definitely generate very fine-grained records internally.

For example:

Page
→ Code Block 3
→ Comment 2
→ teaching-comment

Page
→ Code Block 3
→ Comment 2
→ teaching-comment

But this does not mean every comment should be turned into a formal TranslationUnit.

The formal governance boundary remains:

One physical Page is one formal Page TranslationUnit.

The role of internal comment slots is:

• parser;
• protection;
• restoration;
• validation;
• completeness audit;
• precise impact analysis.

They exist to let the machine see more granularly.

Not to break the formal review process into increasingly smaller pieces.

This remains consistent with the previous design where “Page is the governance unit, slot is the internal parsing unit”.

Future new content can also undergo source-side pre-checking first

This set of statistics also has a use I really like:

Before the content officially enters the TranslationUnit lifecycle, we can already check code comment coverage risks.

For example, in the future we plan to include more /doc/ pages.

We can first do a read-only analysis from the frozen official source:

• How many code blocks there are;
• Which languages are already supported;
• Whether there is any unknown syntax;
• Whether there are any unclassifiable comments;
• Whether the current parser can already fully cover them.

If at the source stage we discover:

A large number of ambiguous code blocks

Then we can completely fix the parser first.

Rather than waiting until the content officially enters Generation to discover:

That there was actually a batch of code comments that were never recognized.

This is actually about pushing the problem as far forward as possible.

A good completeness check should be able to prove “zero omissions”, not report “looks fine”

I think the core distinction of this post lies right here.

The weakest checking method is:

I scanned it once, seems like there are no English comments.

A slightly stronger one is:

All recognized teaching comments have target text.

And what we truly hope to achieve is:

All original code blocks have a disposition established
→ No ambiguous unsupported exists
→ All source teaching comments have a precise identity
→ All current target comments can be precisely corresponded
→ No missing exists
→ No extra exists
→ No protected mutation exists
→ No structure mismatch exists

All original code blocks have a disposition established
→ No ambiguous unsupported exists
→ All source teaching comments have a precise identity
→ All current target comments can be precisely corresponded
→ No missing exists
→ No extra exists
→ No protected mutation exists
→ No structure mismatch exists

Only after reaching this point can we more confidently say:

Within the current scope of support, there are no known code comment omissions.

Note, the most important part of this sentence is actually:

Within the current scope of support.

If there is still unknown grammar, it must be explicitly stated:

It is still partial coverage.

We cannot just count the unknown parts as PASS because all recognized parts passed.

This and ordinary translation quality review solve two different problems

Having reached this point, I think the whole problem can be divided into two tracks.

The first track is quality:

For the text that has already entered the translation workflow, is the translation good?

The second track is completeness:

Has all the text that should have entered the translation workflow actually entered?

The first track mainly relies on an Independent Reviewer.

The second track relies more on:

• source inventory;
• parser;
• precise correspondence;
• currentness;
• deterministic validation.

If we only do the first track, we might get:

Every reviewed translation is good.

But there is still content that never entered review.

So:

High translation quality does not automatically equal translation completeness.

This design made me re-understand the meaning of “statistics”

Previously, when statistics were mentioned, it was easy to think of:

How many pages there are
How many comments there are
How many have been translated

How many pages there are
How many comments there are
How many have been translated

Later I increasingly felt:

Statistics are not for making reports, but for establishing a proof of completeness.

A truly valuable statistic is not a single total number.

But rather one that can answer:

How was the total derived?

Does it cover all source blocks?

Which ones were proven to have no comments?

Which ones are still unknown?

Which target content is missing?

Which ones are just machine semantics and should not be translated?

If these questions cannot be answered, then:

translated_comments = 100%

translated_comments = 100%

a number like this itself does not mean much.

The principles ultimately formed

Looking back now, this part on code comment completeness can be condensed into a few rules.

First:

“Being able to translate teaching comments” does not equal “not missing teaching comments”.

Second:

Completeness checks must build a unique inventory from the canonical source content, and cannot be reverse-engineered from the target translation.

Third:

You cannot rely on //, #, file paths, or a few keywords to guess the code language and comment boundaries.

Fourth:

Unsupported does not equal zero comments; when it cannot be proven, the unknown state must be explicitly retained.

Fifth:

Every original code block must receive a unique, traceable processing conclusion.

Sixth:

The machine is responsible for missing, extra, structure, and protected content completeness; the Reviewer is responsible for translation quality.

Seventh:

Target comparison can only use the current official translation, and cannot be automatically reused just because a historical file exists.

Eighth:

A completeness failure only reopens the truly affected Page or Example, and does not force all unrelated A to be redone.

The ultimate goal is not:

To find as many code comments to translate as possible.

Rather, it is:

To be able to prove that all code comments supposed to enter the translation workflow have not been silently omitted.