Preparing for the Next PHP 8.5 Upgrade: From Two Validation Failures to an Automated Rollback Tool

Last time, I upgraded PHP on the server from 8.5.9 to 8.5.12.

Although the upgrade ultimately succeeded, the entire process took about two and a half hours. I also encountered issues like a failed switchover and a rollback exception, and only completed the production deployment after further inspection and another switchover.

For a PHP patch upgrade that only needs to be performed occasionally, this duration is a bit too long.

Therefore, after completing this upgrade, I decided to continue improving my OneinStack custom repository, hoping to complete the upgrade within half an hour the next time PHP 8.5 releases a patch version.

This half-hour is a goal and does not mean it has already been achieved.

What I mainly accomplished this time was the development, verification, and refinement of the upgrade tools.

1. First Determine the Upgrade Plan: Accept Brief Downtime

Initially, I considered running the old and new PHP-FPM simultaneously, then smoothly switching to the new version via Nginx.

This approach could theoretically reduce website downtime, but upon closer consideration, I felt it might not be suitable for my server.

Multiple websites are hosted on the server. If the PHP request forwarding configurations are scattered across different Nginx files, switching versions might require checking and modifying multiple configuration files.

Even if they can be centrally managed via a common configuration, I would still need to verify if any exceptions exist. Otherwise, it would be easy to end up with a situation where some websites have been switched while others still use the old configuration.

What I really wanted to solve was not necessarily achieving zero website downtime.

My primary goal was to shorten the entire upgrade process and ensure reliable recovery in case of errors.

Therefore, I ultimately decided to adopt a simpler approach:

The new PHP version is compiled, assembled, and verified in an isolated directory in advance. Once preparation is complete, stop the old PHP-FPM, retain the old installation directory, switch the new version to /usr/local/php, restart PHP-FPM, and then confirm service recovery through real requests.

If the new version fails to start or verify, attempt to restore the old version.

This entire process does not modify the Nginx configuration, nor does it require reloading Nginx.

As long as the downtime during the final switchover phase is acceptable, there is no need to introduce the extra complexity of running old and new PHP-FPM in parallel with this approach.

2. First Tool the Pre-Upgrade Preparations

This time, I first refined three parts.

The first is PHP configuration migration.

When upgrading PHP, I want to retain the audited production configurations rather than letting the new version’s default settings overwrite the existing ones. At the same time, I also need to check if there are any deprecated or incompatible options in the old configurations.

The second is the distribution directory assembly.

After the main PHP program is installed into the isolated directory, the Imagick, LDAP, and Redis extensions compiled for the target version are added to the distribution directory.

During the assembly process, files, configurations, extensions, and SHA-256 checksums are checked, ultimately generating a distribution package that can be independently verified.

The third is real request health checks.

The PHP-FPM service showing as running does not mean it is already capable of correctly handling PHP requests.

Therefore, I added a FastCGI health probe that requires actual requests to succeed and verifies the PHP version, extension versions, execution user, and key configurations.

After developing these tools, I started using the PHP 8.5.12 build files left from the previous upgrade to conduct real isolated verification on the Alibaba Cloud server.

Problems were quickly discovered.

3. First Failure: Packer Unfit for Real Installation Directory

After passing local tests, I put the packer on the server and ran it using Python 3.6.8.

The candidate configuration validation was normal, but assembling the distribution directory failed.

Upon inspection, the problem lay in the packer’s assumptions about the PHP installation directory structure.

The initial implementation required the PHP extension directory to already exist in the isolated installation directory, for example:

lib/php/extensions/no-debug-non-zts-20250925

But the actual build process does not work this way.

When the main PHP program is installed to the isolated directory via make install, this extension directory does not necessarily exist yet. Imagick, LDAP, and Redis are compiled separately afterwards and then added to the final distribution directory.

Therefore, the absence of an extension directory does not mean the main PHP program installation is incomplete.

The original check condition was too strict and ended up rejecting a normal installation result.

Later, I adjusted the verification method to read php-config from the isolated installation, confirm that the official extension directory it records matches the target ABI, and then create the corresponding directory when assembling the distribution package.

After the modification, I re-ran the local tests and then verified on the server; the distribution package assembly and integrity checks both passed smoothly.

This also shows that even if a locally simulated directory structure passes tests, it may not be completely identical to the real PHP installation result.

4. Second Failure: FastCGI Probe Misjudges Normal Response

Next, I continued testing the standalone PHP-FPM.

This time, I did not use the production socket; instead, I created a standalone configuration in a temporary directory and started a PHP-FPM instance dedicated to verification.

The FPM configuration check passed, and the process started successfully.

But the health probe kept returning failure:

Missing STDOUT stream end

Ultimately, 120 consecutive attempts were made within the 30-second wait time, and all failed.

At first, I was not sure whether the problem was with PHP-FPM or the probe itself.

So, I further inspected the record structure of the real FastCGI response.

It turned out that PHP-FPM had correctly executed the PHP script and returned JSON data. The PHP version, execution user, three extensions, and random checksum value were all as expected.

The real issue was that the probe’s judgment regarding how the response ended was too strict.

The actual order of records received was:

STDOUT → END_REQUEST

But the original probe also required an additional empty STDOUT record to appear before END_REQUEST.

Because this empty record was not received, even though PHP-FPM had successfully processed the request, the probe still judged the result as a failure.

After fixing this judgment condition, I added corresponding positive and negative tests, continuing to check whether the response content, end status, version, extensions, and configurations were correct.

When testing again with the real PHP-FPM, the health probe passed on the 2nd attempt.

This test did not switch the production PHP, nor did it restart the production PHP-FPM.

Compared to simply checking if a socket exists, this real request verification is more meaningful. However, it still cannot replace website functionality checks via Nginx.

5. Implement Brief Downtime Switchover and Automatic Rollback

After completing the previous verifications, I had Codex implement the final switchover script.

The most important part here was not reducing a few commands, but handling exceptions that might occur during the upgrade.

For example, if the old service cannot be stopped normally, the installation directory must not continue to be moved.

If the new PHP-FPM fails to start, it should restore the old installation directory as much as possible and restart the old service.

And if the rollback itself fails, the script must not output a success result; instead, it must preserve the scene and evidence, explicitly requesting manual recovery.

In addition, it also needed to prevent issues like repeated execution, concurrent execution, and accidental overwriting of the backup directory.

Ultimately, php85_fpm_cutover.py was implemented, and a set of tests covering normal switchover, automatic rollback, and fault injection was added.

A restriction was deliberately kept here:

The default execution mode does not actually switch the production PHP.

Even if executing full pre-checks and an actual switchover is required, the corresponding modes must be used separately. An official switchover also requires explicit authorization.

This reduces the risk of website interruptions caused by operational errors.

However, up to this stage, the switchover and rollback had only passed local simulations and had not yet completed a full stop-and-start drill in a real environment.

6. Attempted Docker Isolated Drill, But Did Not Force It Forward

To further verify the switchover script, I originally planned to use local Docker to create a completely independent test environment.

Ideally, I could prepare two PHP 8.5 patch versions inside the container, execute real PHP-FPM stops, directory switches, restarts, and failure rollbacks, and measure the downtime.

This would allow testing real processes without affecting the websites on the Alibaba Cloud server.

But this step encountered a new limitation.

The execution session where Codex resides could not access the local Docker Socket:

/var/run/docker.sock

It should be noted that this does not mean the Docker service itself on the ThinkPad is unavailable.

I later confirmed that the Docker Engine is accessible in a normal local terminal session. What was restricted was the execution environment where Codex was located at the time.

Therefore, no test containers were created this time, and the real switchover and rollback timing were not completed.

Actually, I could have continued adjusting the test environment to let Codex complete the verification in an execution mode with permissions.

But considering that the main goal this time was to prepare for the next PHP patch update rather than immediately deploying a new version, I decided to temporarily stop investing more effort.

For such low-frequency maintenance tasks, there is no need to spend a lot of time dealing with test environment issues just to complete one drill.

7. Finally Complete 138 Tests and Commit Code

Although the Docker drill was not completed, the existing tools and simulation tests still needed a formal wrap-up.

So, I had Codex conduct a final review of the uncommitted PHP 8.5-related changes in the repository, including configuration migration, distribution package, health probe, switchover script, rollback logic, and usage documentation.

The final results are as follows:

Check Item	Result
Total automated tests	138
Passed	133
Skipped	5
Failed	0
Bash syntax check	109 files passed
PHP syntax check	Passed
Python 3.6 compatibility	Static check passed
Sensitive file check	Passed
Git diff check	Passed

Among them, the 5 skipped items are integration tests that still require additional real build inputs, which does not indicate execution failures.

The actual runtime verification on Python 3.6.8 for the newly added switchover code has not yet been completed, and this point has been explicitly noted in the final results.

The code was finally committed to GitHub:

Repository: shuijingwan/oneinstack-custom

Commit: 72c6dd14

Commit message: feat: 完成 PHP 8.5 补丁升级工具并明确验证边界

Afterwards, I pushed the commit to the main branch.

At this point, this round of development is considered finished.

8. Can the Next Upgrade Be Kept Under Half an Hour?

This cannot be confirmed yet.

This time, I have toolized a lot of content that originally required manual inspection and operation, including configuration migration, distribution package assembly, file integrity verification, real FastCGI health checks, as well as brief downtime switchover and automatic rollback.

Among them, the distribution package assembly and standalone PHP-FPM health checks have already been verified on the real server.

However, the full production switchover, failure rollback, and timing of the entire upgrade process still need further verification during the next patch update.

In particular, the compilation time for PHP and extensions themselves can occupy most of the upgrade process. Merely increasing the switchover speed cannot guarantee that the entire upgrade process will definitely be completed within half an hour.

Therefore, during the next upgrade, I plan to continue following the principles established this time:

Prioritize completing compilation, configuration checks, and distribution package preparation without affecting the website, and confirm the recovery conditions for the old version in advance. After all necessary preparations pass, enter the brief downtime switchover phase.

If pre-checks fail, stop the upgrade instead of skipping checks to save time.

If the switchover fails, restore according to the rollback process and retain complete operational evidence.

As for a smooth Nginx switchover, there is still no need to introduce it at this stage.

Closing Thoughts

Looking back at this development effort, I think the most worthwhile thing to record is not how many tests ultimately passed, but the problems discovered during two real verifications.

The first was that the packer’s assumptions about the PHP installation directory did not match the real build results.

The second was that the FastCGI probe’s judgment of the response end records was too strict, misjudging a successfully processed request as a failure.

Neither of these problems was exposed in the initial local tests.

Therefore, automated tests are very useful, but they cannot completely replace real environment verification.

On the other hand, real drills do not mean that everything must be done to the fullest extent every time. For maintenance tasks where no new version needs to be deployed yet, trade-offs can be made between risk, benefit, and time cost.

Tool development and code submission have been completed this time, so we can set it aside for now.

When the next PHP 8.5 patch version is released, we will see if this set of tools can truly reduce the upgrade time from over two hours to under half an hour.

系列导航