没有不值得去解决的问题,也没有不值得去学习的技术!

The Command Failed, So Why Did It Still Pass? A Shell Automation Fail-Closed Lesson

The Command Failed, So Why Did It Still Pass? A Shell Automation Fail-Closed Lesson

作者:

在

Recently, in the official workflow for a Go multilingual site, I ran into a seemingly absurd problem:

A command in the middle had clearly failed, yet the script ultimately printed RESULT=PASS.

And this was not because some business logic silently swallowed the error.

The real cause was actually quite mundane.

The first mistake was:

I called a command that does not actually exist:

Bash
generation-bundle check

The correct command should have been:

Bash
generation-bundle current-check

The second mistake was more critical:

To make the script stop early after a failure, I wrote the following in a top-level subshell:

Bash
return

But Bash’s return can only be used inside a function or a script executed with source.

This led to a very dangerous execution chain:

Plaintext
The actual check command does not exist
→ Command exit status is 1
→ Script detects the failure
→ Prints RESULT=FAILED
→ Executes return
→ return itself fails
→ Subsequent commands continue to execute
→ Later checks happen to succeed
→ Finally prints RESULT=PASS again

In other words:

A single script execution produced both FAILED and PASS.

If you only looked at the last few lines in the terminal, you might have only seen PASS.

This incident reaffirmed something for me:

Treating unconfirmed states as failures is not just the responsibility of business validators. The Shell control flow wrapping these validators must also genuinely stop after a failure.

How did this happen?

At the time, I had just created a new translation generation batch for the Simplified Chinese homepage:

Plaintext
chatgpt-zh-CN-059

Then I successfully exported the official Generation Bundle:

Plaintext
zh-CN-chatgpt-zh-CN-059-generation.zip

Corresponding SHA-256:

Plaintext
5b8e2944838ca12e5b79c4ed3fbe1ff49245c10026d176df79e2a73e360b8cb6

After the bundle was exported successfully, we could not immediately start the translation generation.

Because a “successful export” only proves:

A ZIP has just been successfully generated.

It does not prove:

This ZIP is still current and valid relative to the repository state at this moment.

So a currentness check must be performed afterward.

This is where the problem occurred.

I mistakenly executed:

Bash
generation-bundle check

The actual output was very clear:

Plaintext
tour-i18n: unknown command "generation-bundle check"
exit status 1

At this point, the actual currentness check had not been executed at all.

If the script had stopped correctly here, the problem would not have been major.

If the command was wrong, I just needed to figure it out and fix it.

The real danger was that the surrounding Shell control flow did not stop.

A non-functional return

The structure used to wrap this set of commands looked roughly like this:

Bash
(
  some-command
  STATUS=$?

  if [ "$STATUS" -ne 0 ]; then
    printf 'RESULT=FAILED\n'
    return
  fi

  do-more-work

  printf 'RESULT=PASS\n'
)

At first glance, there was nothing wrong.

The logic was also straightforward:

If the previous command failed, print the failure, and then return.

But there was a key issue here.

The outer layer:

Bash
(
  ...
)

creates a subshell.

It is not a Shell function.

So the following:

Bash
return

could not exit the current subshell.

Bash directly threw an error:

Plaintext
bash: return: 只能在函数或者被 source 的脚本中使用 "return"

And what was even easier to overlook:

The fact that return itself failed did not mean the subsequent commands would definitely stop.

So, although the failure branch printed:

Plaintext
===== FINAL =====
RESULT=FAILED

it did not actually create a real stopping boundary.

The subsequent logic continued to execute.

Why could a PASS still appear later?

The original script also scheduled other steps after the currentness check, such as:

  • Viewing the ZIP contents;
  • Counting the number of files;
  • Checking if it contained a manifest;
  • Executing git diff --check.

There was nothing wrong with these steps themselves.

And they did indeed all succeed later.

So the script continued downward, eventually executing the following in the normal success branch:

Plaintext
===== FINAL =====
RESULT=PASS

Ultimately, a result like this appeared in the terminal:

Plaintext
===== FINAL =====
RESULT=FAILED
STAGE=site-v2-shell-homepage-generation-bundle-check
...

Followed immediately by:

Plaintext
===== FINAL =====
RESULT=PASS
STAGE=site-v2-shell-homepage-generation-bundle
...

A single execution actually produced two contradictory final results.

If someone manually read the full log, they could still spot the problem.

But if a subsequent script simply:

takes the last RESULT

then it might very well misinterpret a real failure as a success.

Why shouldn’t you trust that last PASS?

There is an important principle here:

A later success cannot override a prior, unresolved failure of a required check.

At the time, there was already a clear fact on the record:

Plaintext
CHECK_STATUS=1

And the error output explicitly stated:

Plaintext
unknown command "generation-bundle check"

So the true state could only be:

The currentness check did not execute successfully.

Even if all subsequent checks passed, they could not replace it.

For example:

Plaintext
git diff --check = PASS

only proves:

The current Git diff has no whitespace formatting issues.

It does not prove:

The Generation Bundle is still current and valid.

Similarly:

Plaintext
ZIP SHA 匹配

only proves:

The current ZIP bytes match the previously recorded ZIP bytes.

It also does not prove:

This ZIP is still consistent with the current repository state.

Each check has its own responsibility.

One check passing cannot substitute for another check that has not been executed or has already failed.

The “last FINAL” is not inherently trustworthy

In many of my maintenance commands, I uniformly print:

Plaintext
===== FINAL =====
RESULT=PASS

This has a major benefit:

No matter how long the terminal output is, I can quickly find the final status.

But this incident made me realize more clearly:

FINAL is just a summary format, not the authoritative source itself.

Its validity depends on the following premises:

  • All required steps have been executed;
  • The exit status of each step matches expectations;
  • The failure branch can actually stop execution;
  • There is no path that continues execution after a failure;
  • The final result is consistent with all preceding required checks.

If these conditions are not met, then:

Plaintext
RESULT=PASS

is nothing more than a line of plain text.

It cannot prove anything by itself.

In other words:

The process did not succeed because it printed PASS; rather, it only earned the right to print PASS because all preceding conditions had already been proven to hold.

How did I later confirm the real problem?

After discovering that both FAILED and PASS existed in the same output, I did not regenerate the Bundle.

I also did not recreate the translation batch.

The first step did only one thing:

Read the error logs left from the first execution.

The result was very clear:

Plaintext
tour-i18n: unknown command "generation-bundle check"
exit status 1

This immediately distinguished between two completely different problems.

The first scenario was:

The Bundle itself failed the currentness check.

The second scenario was:

The currentness check command did not execute correctly at all.

The recovery methods for these two scenarios are completely different.

If the Bundle had actually become invalid, I would need to continue checking:

  • The current repository state;
  • Input identity;
  • Upstream dependencies;
  • Whether any changes occurred between the Bundle and the current workspace.

But if it was just a typo in the command name, no existing artifacts should be rebuilt.

The correct approach was simply:

Keep the original Bundle, and then use the official currentness check command.

What is the correct command?

Later, I read the current CLI implementation directly and confirmed the official entry point was:

Bash
generation-bundle current-check

It accepts only one required parameter:

Bash
--bundle <generation.zip>

So what should have been executed was something like:

Bash
go run -mod=readonly ./cmd/tour-i18n \
  generation-bundle current-check \
  --bundle /tmp/site-v2-shell-homepage-generation-bundle-20261025/zh-CN-chatgpt-zh-CN-059-generation.zip

Ultimately yielding:

Plaintext
Generation bundle: CURRENT

This showed that the original Bundle was fine all along.

What actually went wrong was:

The surrounding command invocation and Shell control flow.

Why shouldn’t you just rerun upon seeing a failure?

This time, the problem was with a read-only currentness check.

Even if it was re-executed, the actual risk was not high.

But across the project, there are many commands that actually modify the official state, such as:

Plaintext
apply
record
import
promotion
activation

If you develop a habit of:

just running it again if it fails.

you will soon run into more dangerous problems.

Especially when encountering:

Plaintext
mutation=unknown

This does not mean:

absolutely no state change has occurred.

Rather, it means:

it cannot currently be reliably proven whether a state change just occurred.

For example, the following might have occurred:

  • Official files have been written;
  • The current pointer has been replaced;
  • A receipt has been created;
  • But stdout was not returned successfully;
  • Or the process terminated abnormally during the final stage.

In this situation, directly rerunning the original mutation command could cause:

  • Duplicate execution;
  • Use of an already invalid prerequisite SHA;
  • CAS conflicts;
  • Pending marker confusion;
  • Unexplainable historical origins.

Therefore, my recovery principle has always been:

After a failure, first determine whether a state change has occurred, and then decide whether it can be re-executed.

Although this was not an official mutation failure, I applied the same principle.

I did not re-export the Bundle.

I did not recreate chatgpt-zh-CN-059.

Instead, I first read the evidence left from the first execution.

After confirming it was just a command name error, I executed the correct read-only check.

Command names cannot be guessed based on experience

There was another very direct lesson this time.

The following already existed:

Plaintext
generation-bundle export

And many other commands had a similar structure like:

Plaintext
xxx check

So I naturally wrote:

Bash
generation-bundle check

But the actual CLI was not designed this way.

The relevant commands included:

Plaintext
generation-bundle export
generation-bundle current-check
generation-bundle archive-imported
generation-bundle retry-handoff
generation-bundle locale-export
generation-bundle locale-check

As you can see:

The TranslationUnit Generation Bundle uses:

Plaintext
current-check

While the locale Generation Bundle uses:

Plaintext
locale-check

They are not completely unified.

If this were just daily manual operation, checking the help docs after writing a command wrong would not be a big deal.

But in an official workflow, this approach of “guessing a name based on similar commands” carries a higher risk.

Especially when the command might:

  • Create official artifacts;
  • Modify state;
  • Record audit results;
  • Switch the current authoritative state.

In these cases, CLI names, parameters, input types, and failure semantics should all come from the current official contract, not from experiential inference.

Why didn’t I just change it to set -e?

Seeing this, a very natural thought is:

Then just write at the very beginning of the Shell:

Bash
set -e

Won’t it automatically stop after a command fails?

I did not adopt this approach as the main solution.

The reason is that in official workflows, I often need to actively capture exit statuses:

Bash
some-command
STATUS=$?

Then, based on the failure type, decide:

  • What evidence to save;
  • What stop reason to output;
  • Whether the current state change is none, known, or unknown;
  • Whether re-execution is allowed;
  • Whether to keep the pending marker;
  • Whether subsequent steps must be stopped.

And the behavior of set -e across different Shell structures is not as intuitive as one might think.

For example, it is affected by:

  • if
  • &&
  • ||
  • pipeline
  • subshell
  • command substitution

and other syntactic environments.

So I now prefer to explicitly write out the success and failure paths.

For example:

Bash
if some-command; then
  printf 'RESULT=PASS\n'
else
  printf 'RESULT=FAILED\n'
fi

Or:

Bash
if some-command; then
  do-next-step
else
  preserve-evidence
fi

The most important property here is:

After the failure branch ends, the code structure ensures it cannot fall through into the success branch.

Rather than relying on a potentially ineffective:

Bash
return

to stop subsequent execution.

Shell wrapper scripts are also part of workflow correctness

In the past, it was easy to view the Go program as the “official implementation” and the outer Shell commands as temporary aids.

But this incident shows:

As long as the Shell is handling these responsibilities:

  • Chaining multiple official commands;
  • Reading exit statuses;
  • Passing SHAs;
  • Passing prerequisite states;
  • Determining whether to continue;
  • Saving logs;
  • Outputting the final status;
  • Controlling failure recovery;

it is already part of the official workflow.

Even if this Shell script is not permanently stored in the repository, it still directly affects:

whether an official operation continues or stops.

Therefore, the failure path is equally worth reviewing.

At the very least, a few questions should be asked:

  • After the first command fails, will subsequent commands still execute?
  • After a file is missing, is it still possible to enter the success branch?
  • After an SHA mismatch, will execution continue?
  • After a currentness check fails, will execution continue?
  • Could a single execution print both FAILED and PASS?
  • After mutation=unknown, will the next step still execute?

The success path is usually easy to write.

What truly determines whether automation is safe is often the failure path.

Why should long logs be saved to files by default?

This incident also exposed another issue.

Initially, I had the full JSON and check results printed directly to the terminal.

After actual execution, the output was too long to copy back completely.

If I then re-executed an official command just to get shorter output, it could introduce unnecessary duplicate execution risks.

Later, I adjusted the approach to:

Plaintext
Full JSON → evidence file
Full stderr → evidence file
Terminal → only print summary, SHA, identity markers, anomalies, and FINAL

For example, the terminal only retains:

Plaintext
CHECK_STATUS=0
BUNDLE_SHA256=...
MUTATION=none
RESULT=PASS

While the full content continues to be saved under /tmp.

Doing this has several clear benefits.

First, the complete evidence is not lost.

Second, the terminal output is shorter, making it easier to spot anomalies.

Third, it removes the temptation to rerun an official command just because the “output is too long to copy.”

Fourth, when a problem occurs, you can directly read the original results left from the first execution.

This is what ultimately allowed for a safe recovery this time.

No recreated batches.

No re-exported Bundles.

Just reading the original:

Plaintext
export log
check stderr
ZIP SHA

and then continuing the diagnosis.

RESULT=PASS should be the conclusion of a proof, not the default endpoint of the control flow

After this incident, I became more vigilant against a certain way of writing scripts:

Bash
do-step-1
do-step-2
do-step-3

printf 'RESULT=PASS\n'

If any of the three steps has a fall-through path after a failure, this final PASS is very dangerous.

A more reliable approach should be:

Only enter the single success branch after all required conditions have been explicitly met.

In other words, PASS should not just mean:

the script finally reached the last line.

It should mean:

all conditions that must be proven have been individually proven to hold.

These two look similar, but the difference is actually huge.

The former is a process position.

The latter is a state proof.

A few rules left behind by this incident

After this incident, I added a few more rules to my Shell automation.

First:

Do not infer official CLI names and parameters from similar commands.

Second:

The failure branch must genuinely stop, not just print FAILED.

Third:

A subsequent PASS cannot override a prior, unresolved failure of a required check.

Fourth:

FINAL is a status summary, not the state authority itself.

Fifth:

Save long logs to evidence files by default; do not re-execute official mutations just to get short output.

Sixth:

After a failure, prioritize reading the evidence left from the first execution instead of directly rerunning.

Seventh:

The Shell control flow itself is also part of workflow correctness.

Finally, the most intuitive takeaway from this incident:

The most dangerous state of a system is not necessarily when it explicitly tells you “it failed,” but when it has actually failed, yet there remains a path that can reach the end and print a PASS.

True fail-closed is not just about being able to detect errors.

It is:

Once a required condition cannot be proven to hold, the success path is structurally unreachable.

系列导航

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者,自 2013 年起持续运营个人技术博客,记录工作与个人项目中遇到的真实问题,以及解决问题过程中的探索、实践和思考。

除了技术写作,我也在持续开发和维护自己的独立项目,探索新技术和新工具在实际场景中的应用。我始终相信,没有不值得去解决的问题,也没有不值得去学习的技术。

如果这篇文章对你有所帮助,欢迎浏览博客中的其他文章,也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系