没有不值得去解决的问题,也没有不值得去学习的技术!

Alibaba Cloud ECS Disk Usage Exceeded 80%: Drops From 81% to 42% After Cleaning PHP, Nginx, and Redis Upgrade Residuals

[Figure 1: Alibaba Cloud ECS system disk usage exceeding 80% alert email]

作者:

在

Recently, the Alibaba Cloud ECS has been continuously sending system disk usage alerts.

This server’s system disk is only 20GB. Previously, when upgrading PHP, Nginx, and Redis, as well as processing historical WordPress posts, various source directories, build directories, upgrade backups, and historical processing data were left on the server for testing, verification, and easy rollback.

Individually, none of these seemed particularly excessive, but over time they accumulated and ultimately pushed the system disk usage above 80%.

Once the system disk exceeded the alert threshold, notifications were received roughly every hour, so I prioritized resolving this issue today.

Throughout this process, I did not perform any large-scale deletions blindly. Instead, I first identified the directories actually consuming disk space, then verified the current running paths for PHP, Nginx, and Redis, and finally removed the obsolete upgrade residuals one by one.

In the end, the system disk usage dropped from the initial 81% to 42%, and I confirmed that Nginx, PHP-FPM, and Redis all continued to run normally.

Receiving the System Disk Usage Alert

The Alibaba Cloud monitoring email showed that the ECS /dev/vda1 had triggered a “system disk usage too high” alert.

The alert condition in the email specified that the threshold must be met 5 consecutive times with an average reaching or exceeding 80%; the current monitored value at the time was approximately 80.35%.

[Figure 1: Alibaba Cloud ECS system disk usage exceeding 80% alert email]
[Figure 1: Alibaba Cloud ECS system disk usage exceeding 80% alert email]

After logging into the server, I first checked the system disk:

Bash
df -hT /

The result was as follows:

Plaintext
Filesystem     Type  Size  Used  Avail Use% Mounted on
/dev/vda1      ext4   20G   15G  3.7G   81% /

The total capacity of the system disk is only 20GB, with about 15GB already used, leaving only 3.7GB of available space.

[Figure 2: System disk usage reaching 81% before cleanup]
[Figure 2: System disk usage reaching 81% before cleanup]

I also checked the inodes:

Bash
df -ih /

Inode usage was only about 13%, so this was not a case of a large number of small files causing inode exhaustion, but rather a pure disk capacity consumption issue.

Locating the Main Disk-Consuming Directories

I continued by examining the top-level directories on the system disk:

Plaintext
du -x -h --max-depth=1 / 2>/dev/null | sort -h

The notable ones were:

Plaintext
2.3G    /var
4.8G    /root
5.7G    /usr
15G     /

I then took a closer look at /root:

Bash
du -x -h --max-depth=1 /root 2>/dev/null | sort -h

I quickly spotted a batch of very familiar directories:

Plaintext
46M     /root/redis-8.10.0-core-stage
224M    /root/php85-configure-preflight-20260804
226M    /root/php85-configure-preflight-system-iconv-20260804
304M    /root/tools
333M    /root/redis-cutover-backup-20260805-193515
334M    /root/redis-upgrade-backup-20260805-185250
554M    /root/redis-8.10.0
1.8G    /root/nginx-build-1.30.4-openssl-3.5.7
4.8G    /root

These were essentially the test, build, and rollback files left behind from previous upgrades to PHP 8.5, Nginx 1.30.4, OpenSSL 3.5.7, and Redis 8.10.0.

[Figure 3: Numerous PHP, Nginx, and Redis upgrade residual directories in /root]
[Figure 3: Numerous PHP, Nginx, and Redis upgrade residual directories in /root]

Additionally, an old complete PHP directory still existed in /usr/local:

Plaintext
483M    /usr/local/php-8.1.19-before-php85-20260804-182956

As the name suggests, this was the old PHP 8.1.19 version saved prior to upgrading to PHP 8.5.

However, just because a name contains backup, before, or a version number does not mean it can be safely deleted outright.

Before actually cleaning anything up, I verified the versions currently in use in the production environment.

Verifying Current Running Versions of PHP, Nginx, and Redis

PHP:

Bash
/usr/local/php/bin/php -v
readlink -f /proc/$(pgrep -o php-fpm)/exe

The result confirmed that the following is currently running:

Plaintext
PHP 8.5.9
/usr/local/php/sbin/php-fpm

Nginx:

Bash
/usr/local/nginx/sbin/nginx -V 2>&1
readlink -f /proc/$(pgrep -o nginx)/exe

The current version is:

Plaintext
nginx/1.30.4
OpenSSL 3.5.7
/usr/local/nginx/sbin/nginx

Redis:

Bash
/usr/local/redis/bin/redis-server --version
readlink -f /proc/$(pgrep -o redis-server)/exe

Currently running:

Plaintext
Redis server v=8.10.0
/usr/local/redis/bin/redis-server

This means the actual production runtime files are all located in:

Plaintext
/usr/local/php
/usr/local/nginx
/usr/local/redis

The build directories, source directories, and upgrade backups under /root were no longer participating in the current runtime.

Only after confirming this did I begin the actual cleanup.

Unexpected Discovery: PHP-FPM Log Reaching 824MB

While checking for oversized files on the system disk, I discovered an additional issue:

Plaintext
2147483648  /swapfile
863869254   /usr/local/php/var/log/php-fpm.log

/swapfile is a 2GB Swap file, which is a normal system configuration and does not need to be deleted to free up space.

The real anomaly was:

Plaintext
/usr/local/php/var/log/php-fpm.log

This PHP-FPM log had reached 824MB.

I proceeded to check the last 100 lines:

Bash
tail -n 100 /usr/local/php/var/log/php-fpm.log

The log showed almost continuous repetition of:

Plaintext
PHP Deprecated: Method SplObjectStorage::attach() is deprecated since 8.5
PHP Deprecated: Method SplObjectStorage::contains() is deprecated since 8.5

The source was the WordPress Organize Series plugin.

[Figure 4: PHP-FPM log reaching 824MB, with continuous PHP 8.5 Deprecated messages]
[Figure 4: PHP-FPM log reaching 824MB, with continuous PHP 8.5 Deprecated messages]

This PHP 8.5 compatibility issue had already been reported to the plugin developers and will be fixed in a future release, so there was no need to keep these hundreds of megabytes of duplicate logs.

However, since PHP-FPM still had the log file open, I did not delete it directly using rm. Instead, I cleared the file contents:

Bash
truncate -s 0 /usr/local/php/var/log/php-fpm.log

After execution:

Bash
df -hT /

The system disk immediately dropped from:

Plaintext
81%

down to:

Plaintext
77%

This single action freed up approximately 800MB.

Deleting the Nginx Upgrade Build Directory

I first tackled the largest upgrade residual:

Plaintext
/root/nginx-build-1.30.4-openssl-3.5.7

This directory was about 1.8GB.

Since Nginx was confirmed to be running /usr/local/nginx/sbin/nginx, this directory was merely the workspace used at the time to compile Nginx 1.30.4 and OpenSSL 3.5.7.

Deleted:

Bash
rm -rf -- /root/nginx-build-1.30.4-openssl-3.5.7

System disk usage then dropped straight from 77% to 67%.

There was also a Nginx upgrade backup directory of about 5.9MB:

Plaintext
/root/nginx-upgrade-backup-20260805-172623

Since the current Nginx version is running stably, I cleaned this up as well.

Deleting Redis 8.10.0 Upgrade Residuals

The Redis upgrade also left behind quite a few files.

First, the source and compilation directory:

Plaintext
554M    /root/redis-8.10.0

Deleted:

Bash
rm -rf -- /root/redis-8.10.0

Next, I handled two upgrade rollback directories:

Plaintext
334M    /root/redis-upgrade-backup-20260805-185250
333M    /root/redis-cutover-backup-20260805-193515

Deleted them respectively:

Bash
rm -rf -- /root/redis-upgrade-backup-20260805-185250
Bash
rm -rf -- /root/redis-cutover-backup-20260805-193515

As well as the temporary staging directory left from the upgrade process:

Plaintext
46M    /root/redis-8.10.0-core-stage

Deleted:

Bash
rm -rf -- /root/redis-8.10.0-core-stage

After clearing this batch, the system disk had dropped to about 60%.

Deleting PHP 8.5 Upgrade Residual Files

The PHP 8.5 upgrade also left behind two configure preflight working directories:

Plaintext
224M    /root/php85-configure-preflight-20260804
226M    /root/php85-configure-preflight-system-iconv-20260804

These two directories were just temporary workspaces used for configure verification prior to the upgrade, so I deleted them respectively:

Bash
rm -rf -- /root/php85-configure-preflight-20260804
Bash
rm -rf -- /root/php85-configure-preflight-system-iconv-20260804

Then I proceeded to handle:

Plaintext
/usr/local/php-8.1.19-before-php85-20260804-182956

This roughly 483MB directory was the complete PHP 8.1.19 version saved before upgrading to PHP 8.5.

PHP 8.5.9 is now running stably at:

Plaintext
/usr/local/php

Therefore, I proceeded to delete it:

Bash
rm -rf -- /usr/local/php-8.1.19-before-php85-20260804-182956

Once this was done, the system disk had decreased to 55%.

Cleaning Up Two Old OneinStack Archives

After checking /root again, two relatively large compressed archives remained:

Plaintext
432M    /root/oneinstack-backup-20260804-155349.tar.gz
390M    /root/oneinstack-full.tar.gz

The first was a OneinStack backup left around the PHP 8.5 upgrade on August 4, 2026.

The current /root/oneinstack still exists, and the PHP 8.5 upgrade is complete, so there was no need to keep this temporary rollback package:

Bash
rm -f -- /root/oneinstack-backup-20260804-155349.tar.gz

The other oneinstack-full.tar.gz was an old archive left over from 2023.

Before deleting it, I checked its contents:

Bash
tar -tzf /root/oneinstack-full.tar.gz | head -50

It mainly contained:

Plaintext
oneinstack/
oneinstack/install.sh
oneinstack/src/php-8.1.19.tar.gz
oneinstack/src/nginx-1.24.0.tar.gz
oneinstack/src/redis-7.0.11.tar.gz
oneinstack/src/openssl-1.1.1t.tar.gz
...

This was essentially a full archive of the old 2023-era OneinStack source code and installation packages.

The server still retains the currently used /root/oneinstack, so this old archive no longer served any practical purpose:

Bash
rm -f -- /root/oneinstack-full.tar.gz

The system disk dropped further to 51%.

systemd Journal Consuming Another ~2GB

I thought the cleanup was mostly finished at this point, but after checking /var/log again:

Bash
du -ahx /var/log 2>/dev/null | sort -h | tail -30

I found:

Plaintext
2.0G    /var/log
2.0G    /var/log/journal

This meant that almost all the space consumed in /var/log came from the systemd journal.

These files are not suitable for direct manual rm, so I used the built-in cleanup mechanism of journalctl:

Bash
journalctl --vacuum-size=500M

After execution, the system returned:

Plaintext
Vacuuming done, freed 1.4G of archived journals

I then checked:

Bash
journalctl --disk-usage

The result was:

Plaintext
Archived and active journals take up 488.0M in the file system.
[Figure 5: Cleaning systemd journal freed 1.4GB, leaving about 488MB]
[Figure 5: Cleaning systemd journal freed 1.4GB, leaving about 488MB]

This step freed another 1.4GB.

System disk usage dropped straight from 51% down to:

Plaintext
43%

I considered further imposing a fixed disk capacity limit for journald, but since system disk space is now very ample, modifying long-term system configurations might introduce unnecessary complications, so I made no additional adjustments for the time being.

Deleting the Historical Post Processing Directory That Served Its Purpose

Nearing the end of the cleanup, I re-checked /root:

Plaintext
304M    /root/tools
304M    /root/tools/wordpress-ai-excerpt-backfill
304M    /root/tools/wordpress-ai-excerpt-backfill/data
304M    /root/tools/wordpress-ai-excerpt-backfill/data/raw

The wordpress-ai-excerpt-backfill here was the running copy uploaded to the server for processing historical WordPress posts.

The actual Git repository is kept on my local computer; when code modifications are needed, I complete them locally and then upload them to the server via SSH for execution.

The historical post processing phase has now concluded.

If new languages are added in the future, such as Japanese, I do not plan to continue using this legacy migration workflow, as the project itself carries too much past compatibility and legacy logic.

Before deletion, I verified once more:

Bash
pgrep -af 'wordpress-ai-excerpt-backfill|history-migration'

There were no related processes.

I also checked cron and systemd, and nothing referenced:

Plaintext
/root/tools/wordpress-ai-excerpt-backfill

Therefore, I directly deleted the entire server running copy:

Bash
rm -rf -- /root/tools/wordpress-ai-excerpt-backfill

After completion, the system disk further dropped to:

Plaintext
42%

Final Verification of Production Services

After deleting so many upgrade backups and historical working directories, I finally needed to ensure that production services were unaffected.

First, I checked Nginx:

Bash
service nginx status

Result:

Plaintext
Active: active (running)

And the master process was still using:

Plaintext
/usr/local/nginx/sbin/nginx

PHP-FPM:

Bash
pgrep -af php-fpm | head

I could clearly see:

Plaintext
php-fpm: master process (/usr/local/php/etc/php-fpm.conf)
php-fpm: pool www
...

Redis:

Bash
/usr/local/redis/bin/redis-cli ping

Returned:

Plaintext
PONG

Finally, I checked the system disk again:

Bash
df -hT /

Result:

Plaintext
Filesystem     Type  Size  Used  Avail Use% Mounted on
/dev/vda1      ext4   20G  7.7G   11G   42% /
[Figure 6: Final system disk dropped to 42%, with Nginx, PHP-FPM, and Redis all normal]
[Figure 6: Final system disk dropped to 42%, with Nginx, PHP-FPM, and Redis all normal]

Dropped from 81% to 42%

At the start of this cleanup:

Plaintext
/dev/vda1
Total capacity: 20G
Used: 15G
Available: 3.7G
Usage: 81%

After completion:

Plaintext
/dev/vda1
Total capacity: 20G
Used: 7.7G
Available: 11G
Usage: 42%

Overall, about 7GB of system disk space was freed.

This usage was not caused by any single file, but rather was the result of several factors accumulating over time:

  • The Nginx + OpenSSL upgrade left behind about 1.8GB of build workspace;
  • The Redis upgrade left behind source code, staging files, and multiple rollback backups;
  • The PHP 8.5 upgrade left behind preflight working directories and a complete old PHP 8.1.19 version;
  • PHP-FPM accumulated 824MB of logs due to PHP 8.5 Deprecated messages;
  • systemd journal accumulated to about 2GB over time;
  • Historical OneinStack archives continued to occupy hundreds of MB;
  • The concluded WordPress historical post processing directory was still retained on the server.

Most of these items were not “junk files”; they had clear purposes during their respective upgrade, debugging, and rollback phases.

The real issue was that after a phase concluded, the now-useless temporary resources were not cleaned up in a timely manner.

Summary

The biggest takeaway from this process remains that when cleaning up disks on production servers, it is best not to just delete large files blindly upon seeing them.

This is especially true for essential services like PHP, Nginx, and Redis; the source directories and backup files left during upgrades served rollback purposes at the time.

A more reliable sequence is:

First, confirm what is actually consuming the disk space;

Then, verify the versions and paths currently used by production services;

Confirm that the old directories are no longer involved in the runtime;

Delete items one by one, continuously observing disk changes;

Finally, re-verify all core services.

After this round of cleanup, the system disk dropped from 81% to 42%, and the hourly recurring system disk usage alerts lost their trigger condition.

Moreover, compared to simply “clearing a few large files,” this time I also sorted out the abnormal PHP-FPM logs, historical journals, upgrade rollback directories, and the server working directory that had served its purpose.

For a production server that has been running for years, this kind of periodic wrap-up cleanup is probably more valuable than continually hunting for more miscellaneous files.

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者,自 2013 年起持续运营个人技术博客,记录工作与个人项目中遇到的真实问题,以及解决问题过程中的探索、实践和思考。

除了技术写作,我也在持续开发和维护自己的独立项目,探索新技术和新工具在实际场景中的应用。我始终相信,没有不值得去解决的问题,也没有不值得去学习的技术。

如果这篇文章对你有所帮助,欢迎浏览博客中的其他文章,也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系