【唐诗】《與高適薛據慈恩寺浮圖》 岑參：塔勢如湧出，孤高聳天宮。登臨出世界，磴道盤虛空。突兀壓神州，崢嶸如鬼工。四角礙白日，七層摩蒼穹。下窺指高鳥，俯聽聞驚風。

永夜

没有不值得去解决的问题，也没有不值得去学习的技术！

Surface Passes, but Preview Fails: The Value of Layered Validation After a Wrong Renderer

Surface Passes, but Preview Fails: The Value of Layered Validation After a Wrong Renderer

作者：

永夜

在

Views: 7

The workflow is getting closer to launch.

Generation is complete.

Machine validation passes.

Independent review is complete.

The revised content goes through an independent re-review.

Only Grade A results are kept.

The content package is sealed.

The integrated site layer (Surface) has also passed review.

Reaching this point, it is easy to feel that:

The language content and the full site have already been checked, so Preview should just be about “opening it to see what the page looks like.”

But while pushing forward zh-CN Site v2, I encountered a very typical counterexample.

The Surface gate had already passed, and the full locale Preview started successfully.

Then the automated Preview acceptance failed directly:

Plaintext
[preview-browser] FAILED:
stage=Site shell
check=/
expected=one Site shell main without legacy homepage or Tour SPA
actual=wrong renderer

The problem was not a mistranslated sentence.

Nor was it a translation unit failing review.

It was not even the Surface Reviewer missing a language issue.

The real problem was:

The runtime used the wrong renderer.

This failure made one thing clearer to me:

A PASS at each layer only proves what that layer is responsible for.

A Surface PASS cannot prove the real runtime result in advance for Preview.

This article outlines why Preview cannot be replaced by Surface, and how a wrong renderer actually proved the value of layered validation.

Surface checks “content composition”, Preview checks “real execution”

First, let’s separate the two stages.

The integrated site review primarily answers:

Once this completed language content is placed into the full locale site context, are the compositional relationships correct?

It focuses on issues that individual translation units cannot see.

For example:

  • Whether links between pages are correct;
  • Whether local content coverage is consistent;
  • Whether structured content matches the pages;
  • Whether the local / official fallback complies with current coverage rules;
  • Whether the relationship between canonical and route is sound;
  • Whether there are conflicts when new content is placed alongside inherited and reused content;
  • Whether there are integration issues with the glossary, navigation, and site context.

So Surface is very important.

It solves the problem of:

Just because individual pages are correct does not mean they will remain correct when composed into a site.

But Preview checks something else:

Once the current official state is actually handed over to the runtime, are the HTTP, renderer, route, DOM, and browser behaviors what we expect?

This is no longer just about “content relationships.”

It is about:

The result actually executed by the system.

A simple example: all content correct, but displayed by the wrong renderer

Assume we currently have several correct types of content:

  • The site shell corresponding to Site shell;
  • The homepage content corresponding to Homepage;
  • Learn page content;
  • Docs page content;
  • Tour content;
  • Structured data.

Surface can prove that:

The compositional relationships between these contents are correct.

But actually starting the site requires going through a complete execution chain:

Plaintext
Receive HTTP request
→ Enter runtime
→ Execute route resolution
→ Select page renderer
→ Generate HTML
→ Hand over to browser for display

If any layer in this chain selects the wrong implementation, the final page will still be wrong.

For example, the homepage should originally execute:

Plaintext
Site v2 homepage content
→ Use Site shell renderer to generate the page

But the actual runtime enters:

Plaintext
Wrong path: legacy homepage

Or continues into:

Plaintext
Wrong path: Tour SPA

Then even if the source is correct, the translation is correct, and Surface is correct, the user will still see the wrong page.

This is the nature of problems like wrong renderer.

Why it is hard for the Surface Reviewer to catch this kind of error

Because this is not the Reviewer’s responsibility.

The Surface Reviewer receives:

The current official integration scope, source, target, links, structured content, and related context.

It should determine:

Whether there are issues at the language and integration layer.

If we asked the Reviewer to also prove:

Which handler the Go runtime ultimately calls;

Which renderer the root path request ultimately enters;

Whether the DOM the browser receives is generated by that renderer;

That would essentially make a language and content review stage take on the responsibilities of a runtime integration test.

Not only is this difficult to do, but it also makes the boundaries increasingly blurred.

So it is more reasonable to:

Plaintext
Surface stage
→ Responsible for proving content integration is correct

Preview stage
→ Responsible for proving actual execution is correct

Each solves a different problem.

The most critical failure here was not “the page looks bad”

The wrong renderer error is particularly representative because it is not an ordinary visual issue.

The automated check expected:

Exactly one correct Site shell <main> exists in the root page.

It also required that:

The legacy homepage must not continue to appear.

And also that:

The Tour SPA must not be mistakenly treated as the full site homepage.

The result returned was:

Plaintext
Actual result: wrong renderer

This means the system did not find:

A problem with some page style.

Rather, it found:

What we are looking at is not the expected product implementation at all.

This is completely different from a font being slightly off, inconsistent margins, or a suboptimal image size.

If the renderer is wrong, continuing to check page copy details is practically meaningless.

This also shows Preview should not just be a manual “quick glance”

When Preview is mentioned, it is often understood as:

Starting a local server, opening the page in a browser, and checking it manually.

But this approach has an obvious problem:

People are easily fooled by “the page mostly opens fine.”

For example, the old renderer might also return:

Plaintext
Request result: HTTP 200

The page might also:

  • Have a title;
  • Have navigation;
  • Have body text;
  • Load CSS correctly;
  • Look like it is not completely broken.

If we rely solely on:

The page opened.

It is very likely that the wrong implementation will be mistaken for the correct one.

So Preview needs automated checks for explicit product invariants even more.

For example, it should check that:

The homepage must use the current Site shell.

Instead of just checking:

The homepage can return HTTP 200.

The difference between these two standards is huge.

HTTP 200 is a very weak success condition

This case is actually somewhat similar to the previous “real 404” article.

Looking only at the HTTP status easily leads to false positives.

For example:

Plaintext
Root path request: GET /
Return status: 200

This only proves that:

A handler returned content for this request.

It cannot prove that:

The correct handler returned it.

Similarly:

Plaintext
Learn path request: GET /learn/
Return status: 200

Also cannot independently prove that:

  • The page comes from the currently activated content;
  • The current renderer is used correctly;
  • The canonical is correct;
  • The navigation is correct;
  • The local coverage is correct;
  • The structured data is correct.

So Preview acceptance needs to check not just:

Whether there is a response.

But rather:

Whether the actual response is the one required by the current official product model.

Why Preview happens after Surface passes

There is also a sequencing issue here.

Since Preview can catch real runtime errors, why not do it earlier?

Because if the preceding content state is not yet stable, Preview easily turns into:

Re-running full-site runtime and browser checks every time a translation is changed.

This would be very costly.

A more reasonable sequence is:

Plaintext
First complete TranslationUnit
→ Then complete independent quality review
→ Then complete content package sealing
→ Then complete Integrated Surface Review
→ Then execute Activation
→ Finally enter Preview

That is, first clarify:

Whether the content itself is correct

layer by layer.

Then hand the stabilized activated state over to the runtime.

At this point, Preview assumes the responsibility of:

Final integration execution check.

Rather than repeating a review for the preceding language workflow.

A Preview failure does not mean we should retranslate

This boundary is particularly important.

At the time, seeing:

Plaintext
Actual error: wrong renderer

Without a clear layer division, an erroneous reaction can easily occur:

Preview failed, so do TranslationUnit or Surface need to be redone?

Actually, not necessarily at all.

wrong renderer first points to:

runtime / rendering integration.

If it can already be proven that:

  • The TranslationUnit candidates have not changed;
  • The QC results have not changed;
  • The finalization has not changed;
  • The package closure has not changed;
  • The Surface inputs have not changed;
  • The Surface conclusions still hold;

Then we should not reopen all the language deliverables just because of a renderer engineering defect.

The correct approach should be:

Plaintext
Keep trusted language results
→ Keep trusted Surface evidence
→ Locate renderer issue
→ Fix repository engineering
→ Run targeted tests
→ Start fresh Preview
→ Re-execute Preview acceptance

That is:

Fix whichever layer the failure occurred in first.

Otherwise, it is easy to misdiagnose an engineering error as a translation error

Assume the homepage looks completely wrong.

There could be many reasons.

First: the target translation itself is wrong

In this case, we should return to:

TranslationUnit revision → validation → independent re-QC.

Second: individual pages are correct, but the site integration relationship is wrong

In this case, we should return to:

Affected TranslationUnit or Surface integration correction.

Third: content and Surface are both correct, but the runtime selected the wrong renderer

In this case, we should enter:

repository engineering.

If all issues are uniformly handled as:

Plaintext
Page displays incorrectly
→ Re-execute translation

The outcome will be terrible.

Not only does it waste Generation and Reviewer capacity, but it may also destroy originally trusted translation deliverables.

So the failure information must be specific enough

A good automated acceptance cannot just output:

Plaintext
Preview result: FAILED

Because this provides almost no guidance for recovery.

What is more valuable is:

Plaintext
Failure stage: Site shell
Check path: /
Expected behavior: Only one correct Site shell main, and legacy homepage or Tour SPA must not appear
Actual behavior: wrong renderer

These pieces of information answer different questions respectively.

Failure stage

Indicates which layer the problem occurred in.

Check target

Indicates which specific route or behavior was checked.

Expected result

Indicates what the correct product behavior should be.

Actual result

Indicates which invariant was actually violated.

This way, the Coordinator can determine:

Whether it is a language issue, a structural issue, a runtime issue, or a browser behavior issue.

A good gate does not just “block errors.”

It should also:

Help route the error back to the correct layer of responsibility.

Preview is actually validating the entire execution chain

By the time we reach the Preview step, the object being checked is no longer just files.

It is a complete execution chain:

Plaintext
Currently activated authority state
→ Compute local content coverage
→ Load official content
→ Compose full site
→ Resolve request route
→ Select page renderer
→ Generate HTTP response
→ Generate HTML
→ Browser builds DOM
→ Execute real interactions

This is why Preview discovers issues that Surface could not possibly find.

For example:

  • Whether the HTTP handler is selected correctly;
  • Whether the page renderer is selected correctly;
  • Whether the local route actually exists;
  • Whether a path that should return 404 incorrectly returns 200;
  • Whether in-page fragment targets actually exist;
  • Whether breadcrumb links are correct;
  • Whether navigation clicks are effective;
  • Whether desktop and mobile DOM behaviors are consistent;
  • Whether images actually load;
  • Whether the shared Site chrome meets requirements.

These have all entered the realm of:

Runtime facts.

source and target both being correct does not mean Compose is definitely correct

Here is a more detailed example.

Assume it can already be proven that:

Plaintext
Homepage source: correct
Homepage target: correct
Site shell target: correct

This only proves that the relevant inputs are correct.

During actual execution, it still needs to:

Plaintext
Load official content
→ Execute Compose
→ Select renderer
→ Generate final page

If Compose does not use the current activated package authority, but instead:

  • Reads an old projection;
  • Enters a legacy fallback;
  • Calls the old Tour renderer;

Then the final output could still be wrong.

Therefore:

Verifying input correctness cannot replace verifying execution path correctness.

This is the enduring purpose of integration testing.

Why Preview must use the real activated scope

If Preview creates its own set of:

“I think these pages should be displayed.”

Then it might produce another set of truths.

Therefore, one of Preview’s most important inputs should come from:

The current activated scope and the completed content packages.

Rather than:

  • Some old Snapshot;
  • Manually listed routes;
  • Pages that happen to exist in the file system;
  • Candidates that have not yet been activated.

Only in this way does Preview validate:

If we are about to enter Production, what site the current official activated state will actually compose.

This connects back to “file exists ≠ page exists”

The previous article on local content coverage already discussed:

The existence of a translation file does not mean the local route should exist.

Preview further proves that:

Just because a route can be opened does not mean it is served by the correct content and the correct renderer.

So the states can be broken down more clearly:

Plaintext
source already exists
≠ candidate already exists
≠ TranslationUnit is complete
≠ package is complete
≠ Surface has passed
≠ content is activated
≠ runtime behavior is correct
≠ Production is ready

It looks like there are many layers.

But each layer actually eliminates a different kind of false positive.

Layered validation is not about making the process longer

Looking only at the number of steps, this workflow is indeed getting longer.

But the wrong renderer case made me feel that:

The real issue is not having too many gates, but whether each gate clearly proves one specific thing.

If Surface and Preview are all merged into one giant gate, several problems arise.

First:

Language Reviewers could be blocked by runtime failures.

Second:

After a runtime engineering fix, it is hard to determine whether a language review must be redone.

Third:

Once a failure occurs, it is hard to determine the responsibility boundaries.

Layering them actually makes things clearer.

TranslationUnit is responsible for:

Proving unit language quality.

Package Closure is responsible for:

Proving the content package is fully complete.

Integrated Surface is responsible for:

Proving the content integration quality within the full site context.

Activation is responsible for:

Proving the official activated authority has been safely switched.

Preview is responsible for:

Proving the real runtime behavior matches the current product model.

Production is responsible for:

Proving the final correctness of the official live environment.

Each layer answers only one question with a more clearly defined scope.

PASS cannot inherit across layers what it has not proven

This is the point I want to emphasize most in this case.

For example:

Surface PASS.

Only means that:

The Surface check scope passed.

It does not automatically imply:

Runtime has passed.

Nor does it automatically imply:

Browser has passed.

And it certainly does not automatically imply:

Production has passed.

Similarly:

Preview PASS.

Also cannot prove that:

Production network, deployment, CDN, and service processes are all correct.

So:

A PASS should be strictly bound to its own scope of evidence.

We cannot, just because:

There have been many PASSes already.

Conclude that:

The subsequent stages are probably fine too, so they can be skipped.

wrong renderer is actually a very “clean” failure

Although seeing this failure at the time was annoying, from a system design perspective, it is actually a great outcome.

Because the automation clearly identified before entering Production that:

The homepage is actually served by the wrong renderer.

And it did not misjudge this issue as:

  • A TranslationUnit language quality issue;
  • A Reviewer review failure;
  • A glossary issue;
  • A Surface language defect.

It just accurately pointed out that:

The Preview runtime does not match the current product model.

This shows that the gate at least successfully fulfilled its most important duty:

Blocking runtime errors before Production.

Without Preview, this kind of issue could go straight into Production

Imagine a workflow without this layer:

Plaintext
Integrated Surface Review passed
→ Execute publish directly
→ Execute deploy directly
→ Enter Production

If the runtime is still connected to the old renderer, then ultimately it might result in:

The new Homepage has been translated.

Learn / Docs are complete.

Surface has been reviewed.

But the public homepage still displays:

legacy homepage or Tour SPA.

At this point, all the preceding language work has been genuinely completed.

The problem is just that:

The last mile did not actually consume these deliverables.

This is a very typical integration failure.

It is also why Preview cannot be omitted.

Preview also cannot just test a few pretty pages

Another risk is:

Only picking the homepage, the Learn homepage, and one or two Docs pages to check manually.

This kind of sampling is useful for visual observation.

But machine acceptance should still cover as many official product invariants as possible.

For example, it should check:

  • The current activated route set;
  • The real 404 returned for non-existent local pages;
  • Page canonical links;
  • Sitemap;
  • Shared site chrome;
  • Fragment links;
  • Breadcrumb navigation;
  • Navigation behavior;
  • Image loading status;
  • Tour interaction behavior;
  • Desktop / mobile browser behavior.

Especially those things that:

Should not exist

Also need to be checked.

For example:

  • The legacy homepage should not reappear;
  • The Tour SPA should not occupy the Site homepage;
  • Uncovered local routes should not secretly return 200.

Many integration errors happen precisely in:

Wrong fallbacks.

“No errors” also cannot serve as proof that the renderer is correct

The old renderer itself might be a completely normal, mature, and stable implementation.

So it will not:

  • Panic;
  • Return HTTP 500;
  • Output invalid HTML.

From a traditional operations monitoring perspective, it might even be very healthy.

But semantically for the product, it is still wrong.

For example:

Plaintext
HTTP status: 200
HTML structure: valid
Running process: healthy
Actual renderer: wrong

The system should still fail.

This shows that:

Technical health and product correctness are not the same thing.

This case later also influenced Production design

After Preview caught the wrong renderer, a natural question was:

Even if Preview is fixed, will Production end up using another set of old runtime?

If Preview and Production use completely different:

  • Content models;
  • Route resolvers;
  • Page renderers;
  • Browser validation contracts;

Then a Preview PASS still cannot give much confidence for Production.

So the subsequent release-path convergence increasingly emphasized that:

Preview and Production should share the same Site content model and runtime contract as much as possible.

Delivery can differ.

For example:

Plaintext
Preview environment → localhost
Production environment → public origin / CDN

But the core product rules should not each maintain their own set of truths, for example:

  • Local content coverage;
  • Page renderer;
  • Routes;
  • Canonical links;
  • Sitemap;
  • 404 routing behavior.

These should be shared as much as possible.

The value of Preview is not just “one last check”

If Preview is understood merely as:

Checking what was done before one more time.

Then it does seem redundant.

But what it actually does is:

Taking the previously proven correct static state, putting it into the actual running system, and then proving the dynamic result.

The preceding stages primarily prove that:

These inputs and states are correct.

Preview further proves that:

After the system actually consumes these inputs, the resulting behavior is also correct.

The former focuses on the correctness of inputs and states.

The latter focuses on the correctness of execution results.

These two layers cannot replace each other.

When Preview fails, the recovery principle should also remain affected-only

Even if Preview fails, we cannot retreat to:

Redoing the entire site entirely.

We should still first determine:

Which layer is actually affected.

For a situation like wrong renderer, a judgment like this can be obtained:

Plaintext
TranslationUnit status: trusted
QC status: trusted
Package Closure status: trusted
Surface status: trusted
Activation status: confirmed separately per current authority
Preview runtime status: FAILED

Then the main recovery scope should focus on:

renderer and runtime integration.

After the fix, continue with:

Plaintext
Run targeted tests
→ Start fresh Preview
→ Re-execute automated acceptance

Rather than re-invoking Generation.

This is consistent with the principle the entire project has adhered to:

Trusted deliverables are not redone.

Why a fresh Preview is important

After a repository engineering fix, you also cannot continue using the old Preview service and then say:

The files have been changed, just refresh the browser to check.

Because the old process might:

  • Still be running the old binary;
  • Still hold the old content;
  • Still use the old handler;
  • Still cache the old state.

So after the engineering fix is complete, we need to start:

A new Preview instance.

This is the only way to regain:

Plaintext
Using the new runtime
→ Using the new process
→ Reading the current activated inputs
→ Using the current renderer

And then execute the automated acceptance.

Otherwise, it is very likely that:

The code is correct, but the old process is still being tested.

Automated Preview and the HUMAN gate are not redundant either

Even if the automated Preview completely PASSes, the final HUMAN gate still has value.

Because machines are good at checking:

  • exact route;
  • DOM structure;
  • required links;
  • broken fragments;
  • 404;
  • canonical;
  • Fixed product invariants.

But humans are more likely to notice:

  • The page looks unnatural;
  • A layout is technically valid but provides a poor experience;
  • Fonts, line lengths, or whitespace are noticeably abnormal;
  • The copy reads awkwardly when combined;
  • An interaction passes programmatic checks but actually feels bad to use.

So:

automated acceptance

and:

HUMAN gate

belong to different scopes of proof.

This failure made me even more convinced that “one PASS should not replace another PASS”

Looking back at the entire workflow, it can be clearly stated as:

Plaintext
TranslationUnit passed
→ does not mean Surface has passed

Surface passed
→ does not mean Activation has passed

Activation passed
→ does not mean Preview has passed

Preview passed
→ does not mean Production has passed

This is not because the system distrusts the previous gate.

Rather, it is because:

Entering each new layer introduces a set of new variables that simply did not exist in the previous layer.

Entering Surface adds:

Cross-page relationships.

Entering Activation adds:

Current authority and concurrent state.

Entering Preview adds:

Runtime, renderer, HTTP, DOM, and browser behavior.

Entering Production adds:

Deployment, public origin, CDN, service, and real network environment.

So we cannot expect any single layer to prove all future variables at once.

A few principles I now agree with more

After this wrong renderer, I think the Preview layer can be distilled into a few principles.

First:

Surface PASS proves content integration quality, not that the runtime ultimately selected the correct renderer.

Second:

Preview is not simply “taking a glance at the page”, but an automated acceptance of the real execution chain.

Third:

HTTP 200 only proves there is a response, not that the response comes from the correct product implementation.

Fourth:

source, target, and Surface can all be correct, yet still fail due to a runtime integration error.

Fifth:

A Preview failure must be localized to specific stage, check, expected, and actual as much as possible, so the issue can be routed back to the correct layer of responsibility.

Sixth:

A repository engineering defect should not automatically reopen trusted TranslationUnit and Reviewer conclusions.

Seventh:

After an engineering fix, a fresh Preview must be started; an old process cannot be used to prove new code.

Eighth:

Preview should consume the current activated authority, not reinvent its own page scope.

Ninth:

Automated Preview and the HUMAN gate each prove different things and cannot replace each other.

Tenth:

A gate’s PASS is only responsible for the scope it actually checked.

Ultimately, I think the most memorable takeaway from this case is:

Content being correct does not mean the system is actually using that correct content.

And what Preview truly needs to prove is:

The deliverables completed and activated earlier have finally been consumed as expected by the real running system.

系列导航

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者，自 2013 年起持续运营个人技术博客，记录工作与个人项目中遇到的真实问题，以及解决问题过程中的探索、实践和思考。

除了技术写作，我也在持续开发和维护自己的独立项目，探索新技术和新工具在实际场景中的应用。我始终相信，没有不值得去解决的问题，也没有不值得去学习的技术。

如果这篇文章对你有所帮助，欢迎浏览博客中的其他文章，也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系

更多文章