Recently, while maintaining a Go multilingual translation project, I have been running into a performance issue more and more visibly:
Looking at each command individually, it seems to have its own job; but looking at the entire workflow connected together, many commands are repeatedly rebuilding almost identical execution environments.
This is quite different from the performance issues I have encountered before.
In the past, when a command was slow, I would usually analyze that command directly:
Where is there too much traversal?
Where is there repeated parsing?
What can be reused within a single execution?
Are there any obvious algorithmic issues?
This optimization approach is of course still effective.
But once the entire workflow reached its current scale, I started to realize:
The truly expensive things increasingly do not belong to any single specific command, but rather to the “current environment” that these commands jointly depend on.
This means that continuing to optimize commands one by one yields diminishing returns.
The next question worth considering becomes:
Can an execution environment that has been reliably built and verified once be safely reused across multiple consecutive commands?
The problem isn’t that a single command is slow, but that an entire group of commands is re-preparing
Recently, while working on the translation process for Learn / Docs, the execution time for several official commands has reached the level of several minutes.
For example, there have been cases where:
- The current-state check took about 475 seconds;
- Result package generation took about 480 seconds;
- Import took about 422 seconds;
- Review package export and inspection even reached about 11 minutes at one point;
- Site integration related commands also once exceeded 8 minutes.
Looking only at these numbers, it is easy to arrive at an intuitive conclusion:
Just optimize these commands separately.
But upon closer analysis, you find that although these commands have different end goals, they perform a lot of similar work upfront.
For example, they might all need to re-confirm:
- Current content package members;
- Current translation units;
- Historical review chain;
- Current candidate state;
- Representation inheritance relationships;
- Glossary and compatibility status;
- Official receipts;
- Current pointers;
- SHAs and identity identifiers;
- Which results are still valid;
- Which historical states have become invalid.
Individually, these checks are all reasonable.
The problem is:
Command A just spent a few minutes building this information, and after Command B starts, it builds it all over again from scratch.
Command C does it again.
Command D will do it yet again.
Thus, the actual execution time of the entire workflow is not just:
The actual business work each command needs to accomplish.
It also includes:
Repeatedly building the same trusted context multiple times.
This is what I now consider more worthwhile to optimize.
The returns from single-command optimization are approaching their limit
In the previous phase, I had already done a lot of internal optimization within individual commands.
The most typical method was:
During the execution of a single command, do not recompute results that have already been calculated.
For example, if a historical state has been successfully parsed and verified once, then multiple subsequent steps within the same invocation can reuse it, rather than re-reading from disk, re-traversing, and re-calculating every time.
This kind of optimization has a major advantage:
Its boundaries are very clear.
After the command exits, all temporary state disappears together.
The next execution still rebuilds the current state from the real repository.
So it is relatively easy to prove that:
- It will not mistake an old state for the current state;
- It will not skip invalidation checks due to cache residue;
- It will not cause cross-process contamination;
- It will not alter official identities;
- It will not make one execution depend on the implicit state left by another.
This “reuse within a single invocation” has always been my preferred optimization method.
But it also has a natural limitation:
It can only solve the redundant work within a single process.
Once the command ends and the next official command restarts, all the environment built up over the previous few minutes is completely lost.
So the same preparation work is executed all over again.
When a complete workflow contains many consecutive commands, this cost starts to become very noticeable.
I started to suspect: what really should be reused is the “environment”, not a specific result
When first thinking about cross-command optimization, it is easy to think of “caching”.
For example:
Write certain results to disk, and the next command reads the cache directly.
But I increasingly feel that the word “cache” makes it too easy to go in the wrong direction.
Because the most important thing in an official translation workflow is not:
Is there an old result we can just use?
Rather, it is:
Why is this result still valid now?
For example, even if a historical review result’s file itself hasn’t changed at all, you cannot just assume it is still valid simply because a cache exists on disk.
You also need to confirm that what it depends on:
- Translation unit identity;
- Content package members;
- Representation;
- Upstream source content;
- Review chain;
- Current pointer;
- Related SHAs;
Is still consistent with the current state.
So I am now more inclined to describe this problem as:
Sharing a verified execution environment.
Rather than:
Sharing old computed results.
The two look similar, but their design focuses are completely different.
An ordinary cache cares about:
Has this value been calculated before?
A shared trusted environment cares more about:
Under what premises was this value proven valid before?
Do these premises still hold true now?
Performance optimization cannot cache away “currentness” along with the rest
The most dangerous optimization method for this kind of system is bypassing the currentness check itself for the sake of speed.
For example, a command just confirmed:
The current content package has 51 pages.
If the next command simply because:
The previous command checked it five minutes ago.
Completely skips current state confirmation, then if an official state change occurs in between, subsequent results could be built on an invalidated environment.
So the truly difficult part of cross-command reuse is not how to keep objects in memory.
The real difficulty is:
How to prove that, at the moment reuse occurs, the original environment still applies to the current command.
This is why I do not simply turn all expensive objects into a long-term global cache.
Performance optimization must preserve several fundamental properties:
Currentness must still hold.
Invalidation propagation must still hold.
Identity identifiers must remain consistent.
SHA verification must remain trustworthy.
Historical origins must remain traceable.
When certainty is impossible, it must still halt.
If, to save a few minutes, any of these properties becomes:
Probably fine.
Then this optimization is meaningless.
Cold starts must always be able to work independently
This is also a boundary I now attach great importance to.
Even if cross-command environment reuse is implemented in the future, I still want:
With absolutely no reused state, any official command can independently build the correct environment from the repository.
That is to say, a cold start must always be a complete, correct execution path.
A shared environment can only be:
A way to reduce redundant computation when it can be proven safe.
And it cannot become:
Without the state left by a previous command, the current command cannot work correctly.
This is a very important distinction.
If the system must depend on “having run Command A first” for Command B to get the correct result, then this is no longer just a performance optimization.
It actually introduces new implicit state.
And implicit state is one of the hardest problems to troubleshoot in an official workflow.
The ideal state should be:
A cold start and a reused start reach the exact same official conclusions.
The only difference is:
The reused start skips some computations that have been proven unnecessary to repeat.
Cross-command reuse is much harder than in-process caching
Caching within a single command is relatively easy.
Because all objects belong to the same process.
You can know for certain that:
- The inputs have not changed;
- The current working directory has not changed;
- The loaded data is the data for this invocation;
- State is automatically destroyed after the command exits.
Once you go cross-command, none of these conditions can be assumed to hold true by default.
For example, after Command A finishes executing and before Command B starts, the following might happen:
- Repository files are modified;
- New official receipts are written;
- The current pointer changes;
- Content package members change;
- A certain representation produces a successor state;
- A candidate snapshot is rebuilt;
- The user switches branches;
- Another process makes concurrent modifications.
Therefore, a shared environment must answer a more complex question:
Since the last time the environment was built, what things have changed?
If it cannot answer this, it can only discard the environment and rebuild it.
This is also a concrete manifestation of “treat as failure when uncertain” in performance optimization:
If you cannot prove the environment is still valid, do not reuse it.
What is truly worth reusing might be a bounded set of verified objects
Based on my current experience, I think a more reasonable direction for the future is not maintaining a giant permanent global state.
It is more likely:
Building an execution environment with a clear lifecycle, clear input identities, and clear invalidation conditions.
This environment can contain a set of verified objects, such as:
- Current content package member resolution results;
- Verified historical review chains;
- Loaded and verified official receipts;
- Current representation relationships;
- Computed identity mappings;
- Read-only indexes that will not change within the same phase.
Then hand it over to multiple consecutive consumers.
For example:
The first command builds the environment.
The second command confirms that the environment’s input identities are still consistent, and thus reuses it.
If the third command finds that the official state has changed, it invalidates the old environment and rebuilds it.
What this truly saves is not a few tens of milliseconds of a single function.
Rather, it is:
The cost of several consecutive commands repeatedly reading, parsing, traversing, and verifying the entire set of historical states from scratch every time.
Why is this more worth researching than continuing to optimize a single 8-minute command?
Assume there are now three commands:
The first takes 8 minutes.
The second takes 7 minutes.
The third takes 7 minutes.
If you invest a lot of time into optimizing them separately into:
7 minutes, 6 minutes, and 6 minutes.
The total time drops from 22 minutes to 19 minutes.
There is, of course, a gain.
But if analysis reveals that each of them spends 4 minutes rebuilding the same type of environment, then the more valuable question becomes:
Can these 4 minutes be spent only once?
If this can be safely achieved, then the return might not be:
A single command being 10% faster.
Rather:
The entire chain eliminating multiple instances of redundant baseline costs.
This is the difference between architectural-level optimization and local optimization.
Local optimization usually asks:
Can this function be even faster?
Architectural-level optimization asks:
Why does this task need to be done three times?
But a shared environment also cannot become a new complex system for the sake of performance
There is another issue here that I am very wary of.
Cross-command reuse can easily evolve step by step into:
- Resident services;
- Global caches;
- Multi-level caches;
- LRU;
- Disk indexes;
- Automatic invalidation;
- Background refreshes;
- Concurrent updates.
Theoretically, all of these solutions can further optimize performance.
But for a low-frequency, official translation workflow that emphasizes recoverability and provability, they are not necessarily worth it.
Because every layer of cache added introduces new problems:
Who is responsible for invalidation?
How do you prove the cache hasn’t expired?
What about concurrent modifications?
How to recover after a crash?
What if the cache is corrupted?
Will it affect official identities?
When debugging, how do you tell if you are getting a real-time result or an old one?
If, in the end, to save a few minutes, you build a cache consistency system more complex than the original workflow, you have likely deviated from the goal.
So I still tend to:
Prefer a simple, bounded reuse method with a clear lifecycle.
For example, letting a group of closely adjacent commands that depend on the same read-only environment share a single environment build.
Rather than designing a permanently existing global cache layer from the very beginning.
Performance testing should also increasingly approximate the real workflow
Another change this time is:
I am increasingly unwilling to run many long “warm cache benchmarks” separately just for performance optimization.
The reason is simple.
If a command itself takes 5 to 10 minutes to run, executing it four or five times consecutively just to compare cold and warm starts incurs a very high testing cost.
Moreover, this kind of testing makes it easy to optimize for a pretty but unrealistic number:
The second run is very fast.
But the reality of the official workflow might be:
Every time is a new process.
The official state has changed before every execution.
The next command will not hit the same cache at all.
Then the value of this “warm start number” to the real lifecycle is very limited.
So now I prefer:
Letting the real next workflow command bear the performance acceptance test.
After optimizing a shared environment, do not fabricate a large number of redundant benchmarks.
Instead, continue running the real process.
If the following:
- Current-state check;
- Result package;
- Import;
- Review package;
- Site integration;
All naturally gain a benefit, then this optimization is truly meaningful.
What you get this way is not a lab number.
Rather, it is:
The entire real lifecycle has genuinely become faster.
This also changes how I judge performance issues
In the past, when I saw a command take 8 minutes, my first reaction might be:
Why is this command so slow?
Now I am more likely to ask first:
Of these 8 minutes it is slow, how much is spent doing work that truly belongs to it?
And how much is spent rebuilding the environment that the previous command just built?
If most of the time comes from the latter, then continuing to drill down inside this command for optimization might be the wrong direction.
At this point, the scope of performance analysis should be expanded to:
The entire workflow.
Even shifting from:
command profile
To:
lifecycle profile
Because what really affects the maintenance experience is never the number of a single command.
Rather, it is:
How long it takes in total from “I started processing this translation unit” to “it can enter the next official phase”.
My current criteria for judging this kind of optimization
After several rounds of performance optimization, I am now more willing to use the following questions to judge whether a cross-command reuse scheme is worth doing.
First:
Is it reusing computation, or is it secretly reusing official conclusions that might already be invalid?
Second:
Without a shared environment, can a cold start still get completely correct results?
Third:
When does the environment invalidate, and can this be clearly determined by a machine?
Fourth:
After reuse, are the currentness check, identity identifiers, SHAs, and historical origins still intact?
Fifth:
After a failure, can it safely fall back to a fully non-reused path?
Sixth:
Is it optimizing a single command, or a common cost that will recur throughout the entire lifecycle?
Seventh:
Is the cost of implementation and verification clearly lower than the time saved from future repetitions?
As long as a few of these questions cannot be clearly answered, I would rather continue to accept a few minutes of slow execution than rush to introduce a long-term, complex caching system.
From “making commands faster” to “making the workflow do less redundant work”
I think this might be a rather obvious shift in thinking during these few months of performance optimization.
Initially it was:
Why is this command slow?
Later it became:
Why is there redundant computation within this command?
Now it has further become:
Why does the next command need to completely rebuild the environment that the previous command already computed?
These three questions represent different levels of performance optimization.
The first level is code-level optimization.
The second level is computation reuse within a single invocation.
The third level is workflow-level execution environment reuse.
The further you go, the greater the potential return.
But at the same time, the requirements for correctness also become higher.
Especially in my multilingual translation project, the official workflow increasingly emphasizes:
- History is immutable;
- Current state is clear;
- Identity is verifiable;
- Independent review;
- Invalidation propagation;
- Recoverability;
- Halting when certainty is impossible.
Performance optimization cannot become an exception to these principles.
Instead, it should be built upon them.
Finally
Looking back now, I increasingly feel:
The most valuable stage of performance optimization is not shaving a few tens of milliseconds off an already fast function.
Rather, it is discovering:
The entire system is repeatedly doing an expensive piece of work that could safely be shared.
For the current translation workflow, this is the kind of problem worth researching in the next phase:
How to let consecutive official commands share a verified execution environment, without letting “reuse” become a new implicit state.
If this can ultimately be achieved, what you truly get is not just:
A single command being a bit faster.
Rather:
The entire translation lifecycle eliminating a large amount of redundant preparation work.
This is also what I now understand performance architecture optimization to be:
Not making the machine repeat the same task faster, but first asking clearly why this task still needs to be repeated.