The previous article documented an issue with line wrapping in the mobile top bar title on the homepage of the A Tour of Go multilingual translation project.
That issue itself ultimately only required changing a few lines of CSS, but deploying it to production unexpectedly exposed another problem that deserved serious attention:
After switching to the new release,
go-tour.serviceencounteredstatus=203/EXECdue to abnormal directory permissions. systemd continuously attempted to restart it, and the production site temporarily returned 502.
The issue was eventually resolved smoothly, but this experience made me re-examine the production deployment process.
If every future deployment still requires manually uploading files, adjusting permissions, switching current, restarting the service, checking ports, and deciding whether to roll back, then the deployment itself easily becomes a new source of failure.
Therefore, after fixing the mobile CSS this time, I went on to complete another task:
Organizing the first version of a production automated deployment script for the A Tour of Go multilingual translation project, and completing the first real production validation.
However, this automation process itself also went through an important “subtraction”: the first version of the script grew to 722 lines, introducing a complex state machine, deployment lock ownership, multiple exit codes, and automatic inference after SSH interruptions.
This reminded me of a problem I encountered previously when working on a historical article translation project—the automation process itself became so complex that a significant amount of time was ultimately spent maintaining the automation tool rather than completing the core work.
So this time, the final goal became very clear:
An automated deployment script should make deployments simpler and more reliable, not turn into another large project requiring long-term maintenance.
This article documents this entire process.
1. A very small CSS deployment triggered 203/EXEC
After completing the mobile top bar CSS fix in the previous article, I generated a new production release and prepared to switch the production version.
The new release was uploaded to:
/data/go-tour/releases/20260813-zh-CN-a4d4dca
Then switched:
/data/go-tour/current
And restarted:
go-tour.service
But the service did not start normally.
systemd began continuously restarting automatically, repeatedly showing:
status=203/EXEC

From the logs, you can even see the restart counter continuously increasing:
47
48
49
50
51
52
This shows that systemd itself was actually trying to recover the service according to its configuration all along, but the executable file simply lacked the proper conditions to start.
2. The real problem wasn’t the binary, but the release root directory permissions
After further inspection, the cause was quickly found.
After the new release was uploaded, the directory permissions were inherited as something like:
drwx------ www:www
That is:
0700
www:www
But go-tour.service does not run as the www user.
The service configuration uses:
User=go-tour
Group=go-tour
So even if:
bin/tour
itself has execute permissions, as long as the parent release directory cannot be entered by the go-tour user, systemd still cannot execute the final binary.
This is the direct cause of:
status=203/EXEC
In other words, the problem was not:
A broken Go binary compilation.
Nor was it:
A broken systemd configuration.
But rather:
The
go-touruser simply could not traverse the release root directory to access the binary.
3. Roll back first, then fix permissions
At that time, instead of taking risks modifying the failed version, I first switched:
/data/go-tour/current
back to the previously confirmed working release.
After the old version restarted:
go-tour.service
recovered active, 127.0.0.1:3999 returned HTTP 200 again, and the production site was restored accordingly.
Then I handled the new release.
Finally, I unified the permissions to:
- owner/group:
root:root - All directories:
0755 - Regular files:
0644 bin/tour:0755
This means the release does not depend on the original owner, group, and permissions from the upload side, but is uniformly normalized on the production server.
After adjusting the permissions, I switched again, and the new version started successfully.
Although this problem was resolved quickly, it made me realize:
As long as production deployments still rely on a sequence of manual operations, there will always be the possibility of a step being missed.
Today it was permissions.
In the future, it could also be:
- Incomplete upload;
- Checksum not verified;
currentpointing to the wrong directory;- Mistakenly judging success just because the service showed
active; - localhost not actually being ready yet;
- The public CDN still serving old resources;
- Forgetting to roll back in time after the new version failed.
So I started considering formalizing the already-validated manual process.
4. Initially trying to “fully protect” the deployment process
The first version of the automated deployment design was quite cautious.
At that time, I wanted the script to handle as many edge cases as possible, including:
- Strict local release validation;
- Staging upload;
- Deployment lock;
- Lock token;
- Lock ownership validation;
- Stale lock handling;
- Multiple local exit codes;
- Multi-layer
EXITtraps; - Automatic recovery from SSH interruptions;
- Activation phase state machine;
- Automatic staging cleanup;
- Critical window judgment for current switch;
- Automatic rollback;
- Rollback failure classification;
- Public network exception classification.
As a result, the script gradually grew to:
722 lines
Looking at each mechanism individually, every one seemed justified.
But after combining them, a new problem began to emerge:
The deployment script itself became increasingly difficult to reason about.
For example, whether an SSH interruption belonged to:
- Upload phase failure;
- Staging completed;
- Final created;
- Current not yet switched;
- Current possibly already switched;
- Restart possibly already executed;
- Whether rollback needs to be executed;
- Whether the lock should be deleted.
To cover these cases, I would have needed to keep adding state variables and exit statuses.
At this point, I started to feel the direction was wrong.
5. The historical article translation project already provided a lesson
Previously, when working on the WordPress historical article automated translation project, I encountered a similar problem.
To make the batch translation process reliable enough, I gradually added:
- State files;
- Recovery processes;
- Candidates;
- Execution evidence;
- Retries;
- Resumes;
- Restart from current;
- Various error classifications and recovery entry points.
These capabilities were all useful in themselves.
But once the process reached a certain level of complexity, a situation frequently arose:
Originally, I should have been investigating why an article failed to translate, but ultimately it turned into investigating why the translation automation system thought the article had failed.
In other words, the automation tool started stealing time from the core task itself.
If the production deployment script continued developing in the direction of 722 lines, it would very likely become:
Originally just wanting to release a new version, but first having to analyze why the deployment state machine entered a certain branch.
This clearly violated the original intent of automation.
So this time I decided to proactively do subtraction.
6. Redefining the deployment script’s goals
The new goal was no longer:
Building a deployment framework as complete as possible.
But rather:
Automatically executing the already truly validated production deployment steps, and retaining necessary protections only where genuinely dangerous.
The finalized priorities were:
Reliable > Simple > Easy to troubleshoot > Fully automatic recovery for extreme scenarios
The server itself has only one maintainer, so there are no complex team concurrent deployment requirements.
Therefore, there was no need to add a lot of logic for potential future multi-maintainer scenarios.
The deployment lock was retained, but its semantics are very simple:
If it does not exist:
/data/go-tour/.deploy.lock
create it atomically.
If it already exists:
Stop the deployment, prompt that an unfinished previous deployment might exist, and require manual inspection.
Do not analyze:
- Who created this lock;
- Whether the token matches;
- Whether it is stale;
- Whether it can be automatically taken over.
7. A subtractive refactor of the 722 lines
After reorganizing, the script went from:
722 lines
down to:
478 lines
Later, based on the final production semantic audit, a few necessary boundaries were added, and the final draft was:
494 lines
The main complex mechanisms deleted included:
- 10/20/21/22/23/30 multi-layer local exit code protocol;
- Deployment lock token/ownership file;
- Stale lock judgment;
- “Another maintainer’s lock” judgment;
- Multi-layer
EXITtraps; - Complex remote automatic cleanup state machine;
- Multiple sets of intersecting boolean states;
- Automatic guessing and recovery after activation SSH exceptions;
- Redundant full remote manifest Python validation;
- Automatic recovery attempts for every microsecond-level switch window.
Ultimately, only the parts directly related to production safety were retained.
8. What protections were ultimately retained
The core processes ultimately retained in the production deployment script include:
- Local production bundle check;
- Release name safety check;
- Deployment lock;
- Staging upload;
- Not inheriting local owner/group/perms;
- Unified permission normalization on the production server;
- Remote SHA-256 verification;
go-touruser access permission check;- Forbidding overwriting an existing final release with the same name;
- Saving the old
current; - Rename staging to final on the same file system;
- Temporary symlink +
mv -Tfatomic replacement ofcurrent; - Restart
go-tour.service; - 3 consecutive systemd + localhost HTTP health checks;
- Automatic rollback to the old release if the new version explicitly fails;
- Checking the public network only after localhost is healthy;
- Not rolling back an already healthy origin site due to public network/CDN exceptions.
The final official commit:
3852bc1
feat: Add automated production deployment script

9. A very important design: even if production is already broken, deployment should be allowed
During the final security audit, another critical issue was discovered.
The script previously required the old release to still be executable by the:
go-tour
user.
This seemed reasonable:
The rollback target should ideally be available.
But thinking about it carefully reveals that this requirement has serious problems.
If production currently happens to be failing due to:
203/EXEC
then:
The old release is not executable.
Under this prerequisite, the script would instead refuse to deploy the new fixed version.
This creates a paradox:
Production is broken, so a fixed version needs to be deployed;
but because production is broken, the deployment script does not allow deployment.
Ultimately, this hard threshold was removed.
Now the old current only needs to meet the following at deployment start:
- It is a symlink;
readlink -fcan be resolved;- The corresponding directory actually exists;
- It is located under
/data/go-tour/releases/.
It does not require:
- The old binary to be executable;
- The old release to be currently healthy;
- systemd to be active;
- localhost to return HTTP 200.
This way, even if production is currently broken, the automated deployment script can still be used to release the fixed version.
10. If the old version is unhealthy, it cannot falsely report “rollback successful”
After relaxing the pre-deployment conditions, another semantic issue had to be resolved.
Consider this scenario:
Current production is already broken → deploy a fixed version → the fixed version also fails → the script switches
currentback to the old one.
This only proves that:
The symlink has been switched back to the old release.
It does not prove that:
Production has recovered.
Therefore, after an automatic rollback, the script will still:
- Restart
go-tour.service; - Check systemd;
- Check localhost HTTP;
- Only consider the old version healthy after 3 consecutive successes.
Only if it truly passes will it output something like:
rollback completed; old release is healthy
Otherwise, it will:
- Keep the deployment lock;
- Preserve the scene;
- Stop further automatic processing;
- Prompt for manual inspection.
This boundary ultimately passed the audit as well.
11. First real use of the automated deployment script
After the script was committed and pushed, I did not just consider it “done”.
A real production validation was still needed.
So I generated a new release based on the repository at that time:
/tmp/go-tour-release-20260813-zh-CN-3852bc1
The publish status was:
locale=zh-CN
ready=103
pending=0
blocked=0
pages=103
articles=7
Then, for the first time, I actually executed:
scripts/deploy-production.sh \
/tmp/go-tour-release-20260813-zh-CN-3852bc1
The entire process completed automatically:
- Local release preflight;
- Remote deployment lock;
- Staging;
- rsync;
- Permission normalization;
- SHA-256;
go-touruser permission check;- Final release;
currentatomic switch;- Restart;
- localhost health check;
- Public network acceptance.
And this first real execution also validated a previously very worthwhile design decision to keep.
12. systemd is already active, but the application is not truly ready yet
After the restart, the first health check showed:
service=active HTTP=000
That is:
systemd already considers the process active.
But:
localhost has not yet received a normal HTTP response.
Only afterward did it show:
active + HTTP 200 (consecutive 1/3)
active + HTTP 200 (consecutive 2/3)
active + HTTP 200 (consecutive 3/3)
Only then did the script judge the deployment successful.

This scene was very valuable.
Because during previous manual deployments, it was easy to use:
systemctl is-active go-tour.service
and upon seeing:
active
assume that:
The service is ready.
But the first real automated deployment already proved:
The process running does not mean the application is ready to serve requests normally.
This is also why the health check was ultimately not simplified to a single systemctl is-active.
13. Why 3 consecutive HTTP 200s are required
The final health standard is:
Each time must simultaneously satisfy:
systemctl is-active == active
and:
http://127.0.0.1:3999/ == HTTP 200
And it must be:
3 consecutive times
If any one fails:
The consecutive count resets to zero.
Maximum checks:
12 rounds
Interval:
3 seconds
The:
active + HTTP 000
in the first real run exactly proved that this mechanism is not redundant defense.
14. Public network/CDN acceptance must be separated from the origin
After localhost is continuously healthy, the script also checks:
https://go-dev.shuijingwanwq.com/
In the first real deployment, it ultimately got:
public acceptance passed: HTTP 200
However, there is another very important design principle here:
Public network failure will not automatically roll back an already healthy origin.
The reason is that the previous mobile CSS fix already encountered an EdgeOne caching issue.
It might happen that:
localhost is normal
Nginx is normal
production release is normal
EdgeOne still caches old CSS
If the script automatically rolled back upon seeing abnormal public network content, then:
A healthy new version would be incorrectly reverted due to a CDN caching issue.
Therefore, the two layers were ultimately explicitly separated:
Origin health
Determines:
Whether the new release can continue running.
Public network/CDN acceptance
Determines:
Whether further investigation of EdgeOne, HTTPS, Nginx, or caching is needed.
The public network check can return non-zero, but it will not automatically roll back an already continuously healthy origin release.
15. The first real automated deployment finally closes the loop
After the script finished executing, I separately performed a final on-site check.
The result was:
CURRENT:
/data/go-tour/releases/20260813-zh-CN-3852bc1
DEPLOY LOCK:
NO LOCK
SERVICE:
active
LOCAL HTTP:
200

This shows that the first real run fully validated:
- Release uploaded correctly;
- Permissions correct;
- SHA-256 correct;
currentswitched correctly;- Service started correctly;
- localhost normal;
- Public network normal;
- Deployment lock correctly released;
- No staging residue;
- No manual recovery triggered;
- No erroneous rollback occurred.
16. The current normal production deployment process
After completing this automation, the normal production deployment process has become significantly simpler going forward.
First, generate the release:
GOOS=linux GOARCH=amd64 go run -mod=readonly ./cmd/tour-i18n publish \
--locale zh-CN \
--published-at "$published_at" \
--output /tmp/go-tour-release-...
Then execute:
scripts/deploy-production.sh \
/tmp/go-tour-release-...
The remaining process is handled by the script:
- Upload;
- Permission normalization;
- SHA-256;
- Staging;
- Final;
current;- Restart;
- Continuous health checks;
- Necessary rollback;
- Public network acceptance.
From a daily maintenance perspective, this is the true value of this script.
Not because of how many features it has.
But because:
The steps that need to be manually remembered during deployment are significantly reduced.
17. What was intentionally not automated
This time, some “do not automatically handle” boundaries were also specifically retained.
1. Deployment lock already exists
Stop directly.
Do not automatically judge:
Whether it is a stale lock.
And do not automatically delete it.
Because there is only one maintainer, if they see:
/data/go-tour/.deploy.lock
existing, a more reasonable behavior is:
First confirm what happened in the previous deployment.
2. SSH disconnects exactly during the current switch
Do not try to automatically guess:
Whether it switched or not.
The script will preserve:
- Lock;
- Release;
- Scene.
Then prompt for manual inspection:
ssh aliyun 'readlink -f /data/go-tour/current'
ssh aliyun 'systemctl status go-tour.service --no-pager -l'
ssh aliyun 'curl -sS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3999/'
ssh aliyun 'journalctl -u go-tour.service -n 80 --no-pager'
For this extremely rare edge case, human judgment is more reliable than continuing to build dozens of automatic recovery branches.
3. EdgeOne cache
The script does not call the EdgeOne API.
If:
The origin is correct, but the public network still shows old CSS.
Then separately inspect the CDN manually.
This is a different issue from whether the production release is healthy.
18. The biggest takeaway this time is not the script itself
This time ultimately resulted in a:
494 lines
production automated deployment script.
But I feel what is truly important is not “finally writing a deployment script”.
Rather, it is reconfirming a principle very important for long-term projects:
Automation should eliminate repetitive labor, not create new objects to maintain.
The initial 722-line version was actually more complete from the perspective of “covering more exceptions”.
But what long-term maintenance truly needs to consider is:
When a deployment fails half a year later, can I quickly understand what it is doing?
If the answer is:
Needing to re-study a complex exit status protocol and state machine.
Then it has already started to deviate from the original purpose.
This final version chose:
- Automatic completion for the normal path;
- Automatic rollback for explicit failures;
- Stopping for uncertain states;
- Manual confirmation for extreme boundaries.
For the current:
Single maintainer + single server + single production service
scenario, it is sufficient.
19. From manual failure to automated deployment, a complete closed loop was formed
Looking back, this incident initially was just:
The last character “目” of the mobile homepage title wrapped to a new line.
But the subsequent chain of events became:
Fix CSS → deploy to production → release permission exception →
203/EXEC→ rollback → fix permissions → redeploy → EdgeOne cache → real device verification → re-examine manual deployment → automated deployment script → script overly complex → subtractive refactor → final semantic audit → first real automated deployment.
This is also a very common evolution pattern in long-term maintenance projects.
What truly drives infrastructure improvements is often not:
Deciding to design a deployment system today.
But rather:
A real failure finally proving that this is worth automating.
Summary
The final result of this automated production deployment work for the A Tour of Go multilingual translation project is:
production release:
20260813-zh-CN-3852bc1
ready=103
pending=0
blocked=0
pages=103
articles=7
current:
Switched correctly
deployment lock:
Released
go-tour.service:
active
localhost:
HTTP 200
public:
HTTP 200
And the principles ultimately retained by the automated deployment script are also very simple:
Upload to staging first, uniformly normalize permissions, files must be verified,
currentatomic switch, do not just trust systemdactive, automatically roll back only when the new version explicitly fails, and do not guess when the state is uncertain.
After experiencing the initial complex 722-line version, I now actually agree more with this approach:
The most important thing about a production script is not being able to automatically handle all theoretical exceptions, but being easy enough to manage in normal cases and easy enough to understand in abnormal cases.
Without real deployment failure evidence in the future, this first version of the script will not continue adding features for hypothetical boundaries.
When truly encountering new production problems, I will decide whether it is worth modifying based on the real problem.
