没有不值得去解决的问题,也没有不值得去学习的技术!

The A Tour of Go Multilingual Translation Project: From a 203/EXEC Failure to Automated Production Deployment

Figure 3: During the first real run of the production automated deployment script, systemd already showed active, but the first localhost probe still got HTTP 000; subsequently, after 3 consecutive simultaneous passes of active + HTTP 200, the script judged the deployment successful and continued with the public network HTTP 200 acceptance. This real run validated the necessity of continuous application-layer health checks.

作者:

在

The previous article documented an issue with line wrapping in the mobile top bar title on the homepage of the A Tour of Go multilingual translation project.

That issue itself ultimately only required changing a few lines of CSS, but deploying it to production unexpectedly exposed another problem that deserved serious attention:

After switching to the new release, go-tour.service encountered status=203/EXEC due to abnormal directory permissions. systemd continuously attempted to restart it, and the production site temporarily returned 502.

The issue was eventually resolved smoothly, but this experience made me re-examine the production deployment process.

If every future deployment still requires manually uploading files, adjusting permissions, switching current, restarting the service, checking ports, and deciding whether to roll back, then the deployment itself easily becomes a new source of failure.

Therefore, after fixing the mobile CSS this time, I went on to complete another task:

Organizing the first version of a production automated deployment script for the A Tour of Go multilingual translation project, and completing the first real production validation.

However, this automation process itself also went through an important “subtraction”: the first version of the script grew to 722 lines, introducing a complex state machine, deployment lock ownership, multiple exit codes, and automatic inference after SSH interruptions.

This reminded me of a problem I encountered previously when working on a historical article translation project—the automation process itself became so complex that a significant amount of time was ultimately spent maintaining the automation tool rather than completing the core work.

So this time, the final goal became very clear:

An automated deployment script should make deployments simpler and more reliable, not turn into another large project requiring long-term maintenance.

This article documents this entire process.


1. A very small CSS deployment triggered 203/EXEC

After completing the mobile top bar CSS fix in the previous article, I generated a new production release and prepared to switch the production version.

The new release was uploaded to:

Plaintext
/data/go-tour/releases/20260813-zh-CN-a4d4dca

Then switched:

Plaintext
/data/go-tour/current

And restarted:

Plaintext
go-tour.service

But the service did not start normally.

systemd began continuously restarting automatically, repeatedly showing:

Plaintext
status=203/EXEC
Figure 1: After manually switching to the new release, go-tour.service repeatedly encountered status=203/EXEC due to abnormal release directory permissions, and systemd continuously attempted automatic restarts. After fixing the release permissions and recovering, the service finally listened on 127.0.0.1:3999 again. This failure became the direct catalyst for organizing the automated production deployment process.
Figure 1: After manually switching to the new release, go-tour.service repeatedly encountered status=203/EXEC due to abnormal release directory permissions, and systemd continuously attempted automatic restarts. After fixing the release permissions and recovering, the service finally listened on 127.0.0.1:3999 again. This failure became the direct catalyst for organizing the automated production deployment process.

From the logs, you can even see the restart counter continuously increasing:

Plaintext
47
48
49
50
51
52

This shows that systemd itself was actually trying to recover the service according to its configuration all along, but the executable file simply lacked the proper conditions to start.


2. The real problem wasn’t the binary, but the release root directory permissions

After further inspection, the cause was quickly found.

After the new release was uploaded, the directory permissions were inherited as something like:

Plaintext
drwx------ www:www

That is:

Plaintext
0700
www:www

But go-tour.service does not run as the www user.

The service configuration uses:

Plaintext
User=go-tour
Group=go-tour

So even if:

Plaintext
bin/tour

itself has execute permissions, as long as the parent release directory cannot be entered by the go-tour user, systemd still cannot execute the final binary.

This is the direct cause of:

Plaintext
status=203/EXEC

In other words, the problem was not:

A broken Go binary compilation.

Nor was it:

A broken systemd configuration.

But rather:

The go-tour user simply could not traverse the release root directory to access the binary.


3. Roll back first, then fix permissions

At that time, instead of taking risks modifying the failed version, I first switched:

Plaintext
/data/go-tour/current

back to the previously confirmed working release.

After the old version restarted:

Plaintext
go-tour.service

recovered active, 127.0.0.1:3999 returned HTTP 200 again, and the production site was restored accordingly.

Then I handled the new release.

Finally, I unified the permissions to:

  • owner/group: root:root
  • All directories: 0755
  • Regular files: 0644
  • bin/tour: 0755

This means the release does not depend on the original owner, group, and permissions from the upload side, but is uniformly normalized on the production server.

After adjusting the permissions, I switched again, and the new version started successfully.

Although this problem was resolved quickly, it made me realize:

As long as production deployments still rely on a sequence of manual operations, there will always be the possibility of a step being missed.

Today it was permissions.

In the future, it could also be:

  • Incomplete upload;
  • Checksum not verified;
  • current pointing to the wrong directory;
  • Mistakenly judging success just because the service showed active;
  • localhost not actually being ready yet;
  • The public CDN still serving old resources;
  • Forgetting to roll back in time after the new version failed.

So I started considering formalizing the already-validated manual process.


4. Initially trying to “fully protect” the deployment process

The first version of the automated deployment design was quite cautious.

At that time, I wanted the script to handle as many edge cases as possible, including:

  • Strict local release validation;
  • Staging upload;
  • Deployment lock;
  • Lock token;
  • Lock ownership validation;
  • Stale lock handling;
  • Multiple local exit codes;
  • Multi-layer EXIT traps;
  • Automatic recovery from SSH interruptions;
  • Activation phase state machine;
  • Automatic staging cleanup;
  • Critical window judgment for current switch;
  • Automatic rollback;
  • Rollback failure classification;
  • Public network exception classification.

As a result, the script gradually grew to:

Plaintext
722 lines

Looking at each mechanism individually, every one seemed justified.

But after combining them, a new problem began to emerge:

The deployment script itself became increasingly difficult to reason about.

For example, whether an SSH interruption belonged to:

  • Upload phase failure;
  • Staging completed;
  • Final created;
  • Current not yet switched;
  • Current possibly already switched;
  • Restart possibly already executed;
  • Whether rollback needs to be executed;
  • Whether the lock should be deleted.

To cover these cases, I would have needed to keep adding state variables and exit statuses.

At this point, I started to feel the direction was wrong.


5. The historical article translation project already provided a lesson

Previously, when working on the WordPress historical article automated translation project, I encountered a similar problem.

To make the batch translation process reliable enough, I gradually added:

  • State files;
  • Recovery processes;
  • Candidates;
  • Execution evidence;
  • Retries;
  • Resumes;
  • Restart from current;
  • Various error classifications and recovery entry points.

These capabilities were all useful in themselves.

But once the process reached a certain level of complexity, a situation frequently arose:

Originally, I should have been investigating why an article failed to translate, but ultimately it turned into investigating why the translation automation system thought the article had failed.

In other words, the automation tool started stealing time from the core task itself.

If the production deployment script continued developing in the direction of 722 lines, it would very likely become:

Originally just wanting to release a new version, but first having to analyze why the deployment state machine entered a certain branch.

This clearly violated the original intent of automation.

So this time I decided to proactively do subtraction.


6. Redefining the deployment script’s goals

The new goal was no longer:

Building a deployment framework as complete as possible.

But rather:

Automatically executing the already truly validated production deployment steps, and retaining necessary protections only where genuinely dangerous.

The finalized priorities were:

Reliable > Simple > Easy to troubleshoot > Fully automatic recovery for extreme scenarios

The server itself has only one maintainer, so there are no complex team concurrent deployment requirements.

Therefore, there was no need to add a lot of logic for potential future multi-maintainer scenarios.

The deployment lock was retained, but its semantics are very simple:

If it does not exist:

Plaintext
/data/go-tour/.deploy.lock

create it atomically.

If it already exists:

Stop the deployment, prompt that an unfinished previous deployment might exist, and require manual inspection.

Do not analyze:

  • Who created this lock;
  • Whether the token matches;
  • Whether it is stale;
  • Whether it can be automatically taken over.

7. A subtractive refactor of the 722 lines

After reorganizing, the script went from:

Plaintext
722 lines

down to:

Plaintext
478 lines

Later, based on the final production semantic audit, a few necessary boundaries were added, and the final draft was:

Plaintext
494 lines

The main complex mechanisms deleted included:

  • 10/20/21/22/23/30 multi-layer local exit code protocol;
  • Deployment lock token/ownership file;
  • Stale lock judgment;
  • “Another maintainer’s lock” judgment;
  • Multi-layer EXIT traps;
  • Complex remote automatic cleanup state machine;
  • Multiple sets of intersecting boolean states;
  • Automatic guessing and recovery after activation SSH exceptions;
  • Redundant full remote manifest Python validation;
  • Automatic recovery attempts for every microsecond-level switch window.

Ultimately, only the parts directly related to production safety were retained.


8. What protections were ultimately retained

The core processes ultimately retained in the production deployment script include:

  1. Local production bundle check;
  2. Release name safety check;
  3. Deployment lock;
  4. Staging upload;
  5. Not inheriting local owner/group/perms;
  6. Unified permission normalization on the production server;
  7. Remote SHA-256 verification;
  8. go-tour user access permission check;
  9. Forbidding overwriting an existing final release with the same name;
  10. Saving the old current;
  11. Rename staging to final on the same file system;
  12. Temporary symlink + mv -Tf atomic replacement of current;
  13. Restart go-tour.service;
  14. 3 consecutive systemd + localhost HTTP health checks;
  15. Automatic rollback to the old release if the new version explicitly fails;
  16. Checking the public network only after localhost is healthy;
  17. Not rolling back an already healthy origin site due to public network/CDN exceptions.

The final official commit:

Plaintext
3852bc1
feat: Add automated production deployment script
Figure 2: After subtractive refactoring and a final production semantic audit, the automated production deployment script officially entered the repository. The final script is about 494 lines, retaining only core safety mechanisms such as staging, permission normalization, SHA-256, atomic switching, continuous health checks, and failure rollback.
Figure 2: After subtractive refactoring and a final production semantic audit, the automated production deployment script officially entered the repository. The final script is about 494 lines, retaining only core safety mechanisms such as staging, permission normalization, SHA-256, atomic switching, continuous health checks, and failure rollback.

9. A very important design: even if production is already broken, deployment should be allowed

During the final security audit, another critical issue was discovered.

The script previously required the old release to still be executable by the:

Plaintext
go-tour

user.

This seemed reasonable:

The rollback target should ideally be available.

But thinking about it carefully reveals that this requirement has serious problems.

If production currently happens to be failing due to:

Plaintext
203/EXEC

then:

The old release is not executable.

Under this prerequisite, the script would instead refuse to deploy the new fixed version.

This creates a paradox:

Production is broken, so a fixed version needs to be deployed;
but because production is broken, the deployment script does not allow deployment.

Ultimately, this hard threshold was removed.

Now the old current only needs to meet the following at deployment start:

  • It is a symlink;
  • readlink -f can be resolved;
  • The corresponding directory actually exists;
  • It is located under /data/go-tour/releases/.

It does not require:

  • The old binary to be executable;
  • The old release to be currently healthy;
  • systemd to be active;
  • localhost to return HTTP 200.

This way, even if production is currently broken, the automated deployment script can still be used to release the fixed version.


10. If the old version is unhealthy, it cannot falsely report “rollback successful”

After relaxing the pre-deployment conditions, another semantic issue had to be resolved.

Consider this scenario:

Current production is already broken → deploy a fixed version → the fixed version also fails → the script switches current back to the old one.

This only proves that:

The symlink has been switched back to the old release.

It does not prove that:

Production has recovered.

Therefore, after an automatic rollback, the script will still:

  1. Restart go-tour.service;
  2. Check systemd;
  3. Check localhost HTTP;
  4. Only consider the old version healthy after 3 consecutive successes.

Only if it truly passes will it output something like:

Plaintext
rollback completed; old release is healthy

Otherwise, it will:

  • Keep the deployment lock;
  • Preserve the scene;
  • Stop further automatic processing;
  • Prompt for manual inspection.

This boundary ultimately passed the audit as well.


11. First real use of the automated deployment script

After the script was committed and pushed, I did not just consider it “done”.

A real production validation was still needed.

So I generated a new release based on the repository at that time:

Plaintext
/tmp/go-tour-release-20260813-zh-CN-3852bc1

The publish status was:

Plaintext
locale=zh-CN
ready=103
pending=0
blocked=0
pages=103
articles=7

Then, for the first time, I actually executed:

Plaintext
scripts/deploy-production.sh \
    /tmp/go-tour-release-20260813-zh-CN-3852bc1

The entire process completed automatically:

  • Local release preflight;
  • Remote deployment lock;
  • Staging;
  • rsync;
  • Permission normalization;
  • SHA-256;
  • go-tour user permission check;
  • Final release;
  • current atomic switch;
  • Restart;
  • localhost health check;
  • Public network acceptance.

And this first real execution also validated a previously very worthwhile design decision to keep.


12. systemd is already active, but the application is not truly ready yet

After the restart, the first health check showed:

Plaintext
service=active HTTP=000

That is:

systemd already considers the process active.

But:

localhost has not yet received a normal HTTP response.

Only afterward did it show:

Plaintext
active + HTTP 200 (consecutive 1/3)
active + HTTP 200 (consecutive 2/3)
active + HTTP 200 (consecutive 3/3)

Only then did the script judge the deployment successful.

Figure 3: During the first real run of the production automated deployment script, systemd already showed active, but the first localhost probe still got HTTP 000; subsequently, after 3 consecutive simultaneous passes of active + HTTP 200, the script judged the deployment successful and continued with the public network HTTP 200 acceptance. This real run validated the necessity of continuous application-layer health checks.
Figure 3: During the first real run of the production automated deployment script, systemd already showed active, but the first localhost probe still got HTTP 000; subsequently, after 3 consecutive simultaneous passes of active + HTTP 200, the script judged the deployment successful and continued with the public network HTTP 200 acceptance. This real run validated the necessity of continuous application-layer health checks.

This scene was very valuable.

Because during previous manual deployments, it was easy to use:

Bash
systemctl is-active go-tour.service

and upon seeing:

Plaintext
active

assume that:

The service is ready.

But the first real automated deployment already proved:

The process running does not mean the application is ready to serve requests normally.

This is also why the health check was ultimately not simplified to a single systemctl is-active.


13. Why 3 consecutive HTTP 200s are required

The final health standard is:

Each time must simultaneously satisfy:

Plaintext
systemctl is-active == active

and:

Plaintext
http://127.0.0.1:3999/ == HTTP 200

And it must be:

Plaintext
3 consecutive times

If any one fails:

The consecutive count resets to zero.

Maximum checks:

Plaintext
12 rounds

Interval:

Plaintext
3 seconds

The:

Plaintext
active + HTTP 000

in the first real run exactly proved that this mechanism is not redundant defense.


14. Public network/CDN acceptance must be separated from the origin

After localhost is continuously healthy, the script also checks:

Plaintext
https://go-dev.shuijingwanwq.com/

In the first real deployment, it ultimately got:

Plaintext
public acceptance passed: HTTP 200

However, there is another very important design principle here:

Public network failure will not automatically roll back an already healthy origin.

The reason is that the previous mobile CSS fix already encountered an EdgeOne caching issue.

It might happen that:

Plaintext
localhost is normal
Nginx is normal
production release is normal
EdgeOne still caches old CSS

If the script automatically rolled back upon seeing abnormal public network content, then:

A healthy new version would be incorrectly reverted due to a CDN caching issue.

Therefore, the two layers were ultimately explicitly separated:

Origin health

Determines:

Whether the new release can continue running.

Public network/CDN acceptance

Determines:

Whether further investigation of EdgeOne, HTTPS, Nginx, or caching is needed.

The public network check can return non-zero, but it will not automatically roll back an already continuously healthy origin release.


15. The first real automated deployment finally closes the loop

After the script finished executing, I separately performed a final on-site check.

The result was:

Plaintext
CURRENT:
/data/go-tour/releases/20260813-zh-CN-3852bc1

DEPLOY LOCK:
NO LOCK

SERVICE:
active

LOCAL HTTP:
200
Figure 4: The final production state after the first automated deployment. current points to the new release, the deployment lock has been correctly released, go-tour.service remains active, localhost returns HTTP 200, and the entire deployment transaction is completely closed.
Figure 4: The final production state after the first automated deployment. current points to the new release, the deployment lock has been correctly released, go-tour.service remains active, localhost returns HTTP 200, and the entire deployment transaction is completely closed.

This shows that the first real run fully validated:

  • Release uploaded correctly;
  • Permissions correct;
  • SHA-256 correct;
  • current switched correctly;
  • Service started correctly;
  • localhost normal;
  • Public network normal;
  • Deployment lock correctly released;
  • No staging residue;
  • No manual recovery triggered;
  • No erroneous rollback occurred.

16. The current normal production deployment process

After completing this automation, the normal production deployment process has become significantly simpler going forward.

First, generate the release:

Bash
GOOS=linux GOARCH=amd64 go run -mod=readonly ./cmd/tour-i18n publish \
    --locale zh-CN \
    --published-at "$published_at" \
    --output /tmp/go-tour-release-...

Then execute:

Bash
scripts/deploy-production.sh \
    /tmp/go-tour-release-...

The remaining process is handled by the script:

  • Upload;
  • Permission normalization;
  • SHA-256;
  • Staging;
  • Final;
  • current;
  • Restart;
  • Continuous health checks;
  • Necessary rollback;
  • Public network acceptance.

From a daily maintenance perspective, this is the true value of this script.

Not because of how many features it has.

But because:

The steps that need to be manually remembered during deployment are significantly reduced.


17. What was intentionally not automated

This time, some “do not automatically handle” boundaries were also specifically retained.

1. Deployment lock already exists

Stop directly.

Do not automatically judge:

Whether it is a stale lock.

And do not automatically delete it.

Because there is only one maintainer, if they see:

Plaintext
/data/go-tour/.deploy.lock

existing, a more reasonable behavior is:

First confirm what happened in the previous deployment.


2. SSH disconnects exactly during the current switch

Do not try to automatically guess:

Whether it switched or not.

The script will preserve:

  • Lock;
  • Release;
  • Scene.

Then prompt for manual inspection:

Bash
ssh aliyun 'readlink -f /data/go-tour/current'
ssh aliyun 'systemctl status go-tour.service --no-pager -l'
ssh aliyun 'curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3999/'
ssh aliyun 'journalctl -u go-tour.service -n 80 --no-pager'

For this extremely rare edge case, human judgment is more reliable than continuing to build dozens of automatic recovery branches.


3. EdgeOne cache

The script does not call the EdgeOne API.

If:

The origin is correct, but the public network still shows old CSS.

Then separately inspect the CDN manually.

This is a different issue from whether the production release is healthy.


18. The biggest takeaway this time is not the script itself

This time ultimately resulted in a:

Plaintext
494 lines

production automated deployment script.

But I feel what is truly important is not “finally writing a deployment script”.

Rather, it is reconfirming a principle very important for long-term projects:

Automation should eliminate repetitive labor, not create new objects to maintain.

The initial 722-line version was actually more complete from the perspective of “covering more exceptions”.

But what long-term maintenance truly needs to consider is:

When a deployment fails half a year later, can I quickly understand what it is doing?

If the answer is:

Needing to re-study a complex exit status protocol and state machine.

Then it has already started to deviate from the original purpose.

This final version chose:

  • Automatic completion for the normal path;
  • Automatic rollback for explicit failures;
  • Stopping for uncertain states;
  • Manual confirmation for extreme boundaries.

For the current:

Single maintainer + single server + single production service

scenario, it is sufficient.


19. From manual failure to automated deployment, a complete closed loop was formed

Looking back, this incident initially was just:

The last character “目” of the mobile homepage title wrapped to a new line.

But the subsequent chain of events became:

Fix CSS → deploy to production → release permission exception → 203/EXEC → rollback → fix permissions → redeploy → EdgeOne cache → real device verification → re-examine manual deployment → automated deployment script → script overly complex → subtractive refactor → final semantic audit → first real automated deployment.

This is also a very common evolution pattern in long-term maintenance projects.

What truly drives infrastructure improvements is often not:

Deciding to design a deployment system today.

But rather:

A real failure finally proving that this is worth automating.


Summary

The final result of this automated production deployment work for the A Tour of Go multilingual translation project is:

Plaintext
production release:
20260813-zh-CN-3852bc1

ready=103
pending=0
blocked=0
pages=103
articles=7

current:
Switched correctly

deployment lock:
Released

go-tour.service:
active

localhost:
HTTP 200

public:
HTTP 200

And the principles ultimately retained by the automated deployment script are also very simple:

Upload to staging first, uniformly normalize permissions, files must be verified, current atomic switch, do not just trust systemd active, automatically roll back only when the new version explicitly fails, and do not guess when the state is uncertain.

After experiencing the initial complex 722-line version, I now actually agree more with this approach:

The most important thing about a production script is not being able to automatically handle all theoretical exceptions, but being easy enough to manage in normal cases and easy enough to understand in abnormal cases.

Without real deployment failure evidence in the future, this first version of the script will not continue adding features for hypothetical boundaries.

When truly encountering new production problems, I will decide whether it is worth modifying based on the real problem.

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者,自 2013 年起持续运营个人技术博客,记录工作与个人项目中遇到的真实问题,以及解决问题过程中的探索、实践和思考。

除了技术写作,我也在持续开发和维护自己的独立项目,探索新技术和新工具在实际场景中的应用。我始终相信,没有不值得去解决的问题,也没有不值得去学习的技术。

如果这篇文章对你有所帮助,欢迎浏览博客中的其他文章,也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系