1. Reference URL: https://www.shuijingwanwq.com/2020/09/30/4552/ . This is a previous blog post.
2. In a cluster deployment, all four queue groups finish executing in about half an hour. However, the execution time is sometimes still uncontrollable and may even exceed an hour.
3. I decided to print log messages at relevant places in the extension’s source code.
4. // Copy the source resource files to the channel publishing resource directory. After the queue job executes successfully, call the corresponding service; otherwise, insert a publishing log (asynchronously). Edit common\services\AssetService::copyAssetsAsync($data, $assets). -167 indicates line 167. The same rule applies to all code below.
PHP
@file_put_contents(Yii::getAlias('@runtime') . '/common-services-AssetService-copyAssetsAsync-data-' . $data['task_id'] . '-' . date("-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-167.txt', print_r($data, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-services-AssetService-copyAssetsAsync-assets-' . $data['task_id'] . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-168.txt', print_r($assets, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-services-AssetService-copyAssetsAsync-copy_queue_info-' . $data['task_id'] . '-' . $copy_queue_info . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-178.txt', print_r($copy_queue_info, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-services-AssetService-copyAssetsAsync-isWaiting-' . $data['task_id'] . '-' . $copy_queue_info . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-180.txt', print_r($copy_queue_info, true), FILE_APPEND | LOCK_EX);
5. // Copy resource file queue job: Copy the source resource files to the channel publishing resource directory. Edit common\jobs\CopyAssetJob->execute($queue)
PHP
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-queue-workerPid-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-41.txt', print_r($queue->workerPid, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-queue-status-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-42.txt', print_r($queue->status(), true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-queue-attempts-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-43.txt', print_r($queue->attempts, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-queue-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-44.txt', print_r($queue, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-empty-assetEnabledItems-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-52.txt', print_r($this->taskId, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-assetEnabledItems-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-77.txt', print_r($assetEnabledItems, true), FILE_APPEND | LOCK_EX);
6. // Copy resource file queue event handler, after the copy resource file queue job executes successfully. Edit common\components\queue\CopyAssetEventHandler::afterExec(ExecEvent $event)
PHP
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterExec-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-36.txt', print_r($taskId, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterExec-serviceClass-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-44.txt', print_r($serviceClass, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterExec-serviceAction-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-45.txt', print_r($serviceAction, true), FILE_APPEND | LOCK_EX);
7. // Copy resource file queue event handler, after the copy resource file queue job fails to execute. Edit common\components\queue\CopyAssetEventHandler::afterError(ExecEvent $event)
PHP
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterError-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-56.txt', print_r($taskId, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterError-serviceClass-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-64.txt', print_r($serviceClass, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterError-serviceAction-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-65.txt', print_r($serviceAction, true), FILE_APPEND | LOCK_EX);
8. Edit /vendor/yiisoft/yii2-queue/src/drivers/redis/Command.php
PHP
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Command-actionListen-' . $timeout . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-70.txt', print_r($timeout, true), FILE_APPEND | LOCK_EX);
9. Edit /vendor/yiisoft/yii2-queue/src/Queue.php
PHP
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-event-' . $event->id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-198.txt', print_r($event, true), FILE_APPEND | LOCK_EX);
vendor-yiisoft-yii2-queue-src-Queue-event-1-2021-01-12-14-11-09-1610431869.6843-476980414-198.txt
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-try-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-247.txt', print_r($this, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-try-event-result-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-249.txt', print_r($event->result, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-catch-Exception-error-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-251.txt', print_r($error, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-catch-Throwable-error-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-255.txt', print_r($error, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-event-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-259.txt', print_r($event, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-trigger-event-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-261.txt', print_r($event, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-297.txt', print_r($event, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-event-attempt-' . $event->attempt . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-299.txt', print_r($event->attempt, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-this-attempts-' . $this->attempts . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-300.txt', print_r($this->attempts, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-event-error-' . $this->attempts . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-303.txt', print_r($event->error, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-event-retry-' . $this->attempts . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-306.txt', print_r($event->retry, true), FILE_APPEND | LOCK_EX);
10. Edit /vendor/yiisoft/yii2-queue/src/drivers/redis/Queue.php
PHP
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-' . $timeout . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-58.txt', print_r($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-payload-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-63.txt', print_r($payload, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-delete-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-65.txt', print_r($payload, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-break-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-69.txt', var_dump($repeat, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-139.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-142.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-if-timeout-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-151.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-elseif-timeout-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-154.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-if-id-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-157.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-return-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand() . '-165.txt', var_dump($message, true), FILE_APPEND | LOCK_EX);
11. I decided to stop the Supervisor-based queue listening first. I planned to manually execute the listen command.
Plaintext
[root@055c3647ea66 /]# supervisorctl status
cronolog RUNNING pid 1002, uptime 0:00:13
nginx RUNNING pid 999, uptime 0:00:13
php-fpm RUNNING pid 1000, uptime 0:00:13
report_client RUNNING pid 1001, uptime 0:00:13
[root@055c3647ea66 /]# cd /mcloud/www/channel-pub-api/
[root@85b422810e4f channel-pub-api]# ./yii copy-asset-queue/listen --verbose=1 --color=0
2021-01-12 16:18:21 [pid: 1815] - Worker is started
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
string(55) "O:24:"common\jobs\CopyAssetJob":1:{s:6:"taskId";i:495;}"
bool(true)
2021-01-12 16:18:22 [61] common\jobs\CopyAssetJob (attempt: 1, pid: 1815) - Started
2021-01-12 16:18:22 [61] common\jobs\CopyAssetJob (attempt: 1, pid: 1815) - Done (0.795 s)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
^Cint(3)
bool(true)
2021-01-12 16:18:34 [pid: 1815] - Worker is stopped (0:00:13)
[root@85b422810e4f channel-pub-api]# ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 61
12. I decided to restore the Supervisor-based queue listening and check the generated log files.
Plaintext
[root@3354317ffbed /]# supervisorctl status
cronolog RUNNING pid 1055, uptime 0:00:04
nginx RUNNING pid 1020, uptime 0:00:04
php-fpm RUNNING pid 1017, uptime 0:00:04
report_client RUNNING pid 1052, uptime 0:00:04
yii-baijia-article-sync RUNNING pid 1121, uptime 0:00:00
yii-channel-app-task-sync STARTING
yii-copy-asset-queue-worker:yii-copy-asset-queue-worker_00 RUNNING pid 1015, uptime 0:00:04
yii-copy-asset-queue-worker:yii-copy-asset-queue-worker_01 RUNNING pid 1016, uptime 0:00:04
yii-douyin-web-video-sync STARTING
yii-log-delete STARTING
yii-netease-tp-article-sync STARTING
yii-pub-article-queue-worker:yii-pub-article-queue-worker_00 RUNNING pid 1010, uptime 0:00:04
yii-pub-article-queue-worker:yii-pub-article-queue-worker_01 RUNNING pid 1009, uptime 0:00:04
yii-qq-cw-transaction-article-sync STARTING
yii-qq-cw-transaction-video-sync STARTING
yii-qq-tp-transaction-article-sync STARTING
yii-qq-tp-transaction-video-sync STARTING
yii-source-callback-queue-worker:yii-source-callback-queue-worker_00 RUNNING pid 1029, uptime 0:00:04
yii-source-callback-queue-worker:yii-source-callback-queue-worker_01 RUNNING pid 1030, uptime 0:00:04
yii-upload-asset-queue-worker:yii-upload-asset-queue-worker_00 RUNNING pid 1011, uptime 0:00:04
yii-upload-asset-queue-worker:yii-upload-asset-queue-worker_01 RUNNING pid 1012, uptime 0:00:04
[root@3354317ffbed runtime]# ls -lrt
total 64
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9494-1819463520-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.9496-794234002-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.953-1837796717-138.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9578-108840870-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.958-592107120-57.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9617-1880485471-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.9619-1414847196-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.9619-1179420058-138.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9628-394649028-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.963-1060811221-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.966-593417676-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.9666-580562356-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-31-1610440351.968-1954959781-141.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9749-573783946-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.9751-211630453-57.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9765-2041102433-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.9767-689870268-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.9781-1224010356-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.9797-903147806-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-31-1610440351.9822-1531347538-141.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-31-1610440351.9864-1375602874-141.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-32-1610440352.0892-1852556723-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-32-1610440352.0894-303473184-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-32-1610440352.0935-1190705523-138.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-32-1610440352.0951-433868058-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-32-1610440352.0953-158476143-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-32-1610440352.0989-1835560487-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-32-1610440352.1096-318569225-141.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-if-id-2021-01-12-16-32-35-1610440355.2908-1182646222-156.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-35-1610440355.2911-1232420494-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-35-1610440355.2944-141684420-141.txt
13. I suspect there is a severe locking issue with listening in a cluster environment, or rather, with multiple processes. For the Docker deployment, I replaced the queue/listen command with the queue/run command. Edit /etc/supervisord.d/yii-copy-asset-queue-worker.ini. The other three queue configuration files were also edited in the same way.
INI
[program:yii-copy-asset-queue-worker]
command = bash -c 'sleep 60 && exec php /mcloud/www/channel-pub-api/yii copy-asset-queue/run'
autorestart = true
startsecs = 60
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-copy-asset-queue-worker-stderr.log
stdout_logfile = /data/logs/yii-copy-asset-queue-worker-stdout.log
INI
[program:yii-upload-asset-queue-worker]
command = bash -c 'sleep 60 && exec php /mcloud/www/channel-pub-api/yii upload-asset-queue/run'
autorestart = true
startsecs = 60
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-upload-asset-queue-worker-stderr.log
stdout_logfile = /data/logs/yii-upload-asset-queue-worker-stdout.log
INI
[program:yii-pub-article-queue-worker]
command = bash -c 'sleep 60 && exec php /mcloud/www/channel-pub-api/yii pub-article-queue/run'
autorestart = true
startsecs = 60
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-pub-article-queue-worker-stderr.log
stdout_logfile = /data/logs/yii-pub-article-queue-worker-stdout.log
INI
[program:yii-source-callback-queue-worker]
command = bash -c 'sleep 60 && exec php /mcloud/www/channel-pub-api/yii source-callback-queue/run'
autorestart = true
startsecs = 60
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-source-callback-queue-worker-stderr.log
stdout_logfile = /data/logs/yii-source-callback-queue-worker-stdout.log
14. I deployed 2 containers in the test environment. As shown in Figure 1
15. I entered container one and executed command lines. I confirmed that this container was receiving requests and running queue jobs. As shown in Figure 2
Plaintext
[root@8d8ff3df26b0 /]# supervisorctl status
cronolog RUNNING pid 1034, uptime 1:47:55
nginx RUNNING pid 1019, uptime 1:47:55
php-fpm RUNNING pid 1014, uptime 1:47:55
report_client RUNNING pid 1033, uptime 1:47:55
yii-baijia-article-sync RUNNING pid 11891, uptime 0:00:00
yii-channel-app-task-sync STARTING
yii-copy-asset-queue-worker STARTING
yii-douyin-web-video-sync STARTING
yii-log-delete STARTING
yii-netease-tp-article-sync RUNNING pid 11703, uptime 0:01:38
yii-pub-article-queue-worker STARTING
yii-qq-cw-transaction-article-sync STARTING
yii-qq-cw-transaction-video-sync STARTING
yii-qq-tp-transaction-article-sync STARTING
yii-qq-tp-transaction-video-sync STARTING
yii-source-callback-queue-worker STARTING
yii-upload-asset-queue-worker STARTING
[root@8d8ff3df26b0 /]# cd /mcloud/www/channel-pub-api/api/runtime/
[root@8d8ff3df26b0 runtime]# ls -lrt
total 668
drwxrwxr-x 5 nginx nginx 4096 Jan 20 16:20 cache
drwxrwxr-x 2 nginx nginx 4096 Jan 20 16:20 logs
-rw-r--r-- 1 nginx nginx 325 Jan 20 16:33 task-group-StandardCreate-requestParams-1611131592.3771-910238203.txt
-rw-r--r-- 1 nginx nginx 1278 Jan 20 16:48 task-group-VideoCreate-requestParams-1611132522.5135-1170578969.txt
-rw-r--r-- 1 nginx nginx 325 Jan 20 16:55 task-group-StandardCreate-requestParams-1611132921.2381-159358239.txt
-rw-r--r-- 1 nginx nginx 325 Jan 20 17:18 task-group-StandardCreate-requestParams-1611134312.9066-273770349.txt
-rw-r--r-- 1 nginx nginx 325 Jan 20 17:20 task-group-StandardCreate-requestParams-1611134415.4961-206630537.txt
-rw-r--r-- 1 nginx nginx 2435 Jan 20 17:32 task-group-VideoCreate-requestParams-1611135156.2789-1685654955.txt
-rw-r--r-- 1 nginx nginx 4335 Jan 20 17:50 task-group-VideoCreate-requestParams-1611136237.5574-1504010220.txt
-rw-r--r-- 1 nginx nginx 3266 Jan 20 17:59 task-group-VideoCreate-requestParams-1611136785.0646-1310530126.txt
-rw-r--r-- 1 nginx nginx 1151 Jan 20 18:21 task-group-VideoCreate-requestParams-1611138109.7108-1428510960.txt
-rw-r--r-- 1 nginx nginx 1133 Jan 20 19:11 task-group-StandardCreate-requestParams-1611141071.2366-451937142.txt
-rw-r--r-- 1 nginx nginx 1161 Jan 20 19:28 task-group-VideoCreate-requestParams-1611142135.7108-1213586108.txt
-rw-r--r-- 1 nginx nginx 1103 Jan 20 19:40 task-group-VideoCreate-requestParams-1611142825.0161-333542923.txt
-rw-r--r-- 1 nginx nginx 505 Jan 20 19:52 vendor-hprose-hprose-src-Hprose-Client-response-1611143571.9702-178210933.txt
-rw-r--r-- 1 nginx nginx 94 Jan 20 19:52 vendor-hprose-hprose-src-Hprose-Client-args-1611143571.9704-862498366.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 20 19:52 vendor-hprose-hprose-src-Hprose-Client-context-1611143571.9704-185265682.txt
-rw-r--r-- 1 nginx nginx 505 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-response-1611143859.2669-528070097.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-context-1611143859.2671-135288224.txt
-rw-r--r-- 1 nginx nginx 94 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-args-1611143859.267-757011716.txt
-rw-r--r-- 1 nginx nginx 505 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-response-1611143876.4248-983204429.txt
-rw-r--r-- 1 nginx nginx 94 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-args-1611143876.4252-311858334.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-context-1611143876.4253-325345408.txt
-rw-r--r-- 1 nginx nginx 505 Jan 20 20:00 vendor-hprose-hprose-src-Hprose-Client-response-1611144013.4502-160577375.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 20 20:00 vendor-hprose-hprose-src-Hprose-Client-context-1611144013.4504-1265475831.txt
-rw-r--r-- 1 nginx nginx 94 Jan 20 20:00 vendor-hprose-hprose-src-Hprose-Client-args-1611144013.4503-2132911066.txt
-rw-r--r-- 1 nginx nginx 1447 Jan 20 20:04 task-group-VideoCreate-requestParams-1611144289.3014-730024961.txt
-rw-r--r-- 1 nginx nginx 36039 Jan 20 20:21 task-group-StandardCreate-requestParams-1611145302.5614-76746077.txt
-rw-r--r-- 1 nginx nginx 16295 Jan 20 20:26 task-group-StandardCreate-requestParams-1611145608.7813-919723561.txt
-rw-r--r-- 1 nginx nginx 5264 Jan 20 20:29 task-group-StandardCreate-requestParams-1611145745.6312-1832504444.txt
-rw-r--r-- 1 nginx nginx 1209 Jan 20 20:29 task-group-StandardCreate-requestParams-1611145763.8136-1359951868.txt
-rw-r--r-- 1 nginx nginx 505 Jan 20 20:34 vendor-hprose-hprose-src-Hprose-Client-response-1611146090.9852-1952201824.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 20 20:34 vendor-hprose-hprose-src-Hprose-Client-context-1611146090.9854-1038382139.txt
-rw-r--r-- 1 nginx nginx 94 Jan 20 20:34 vendor-hprose-hprose-src-Hprose-Client-args-1611146090.9854-122587697.txt
-rw-r--r-- 1 nginx nginx 505 Jan 20 20:35 vendor-hprose-hprose-src-Hprose-Client-response-1611146152.5181-1787102293.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 20 20:35 vendor-hprose-hprose-src-Hprose-Client-context-1611146152.5183-163122809.txt
-rw-r--r-- 1 nginx nginx 94 Jan 20 20:35 vendor-hprose-hprose-src-Hprose-Client-args-1611146152.5183-2040247314.txt
-rw-r--r-- 1 nginx nginx 1287 Jan 21 09:18 task-group-StandardCreate-requestParams-1611191906.7021-1963669870.txt
-rw-r--r-- 1 nginx nginx 3824 Jan 21 09:44 task-group-StandardCreate-requestParams-1611193482.0351-1386708367.txt
-rw-r--r-- 1 nginx nginx 74055 Jan 21 09:47 task-group-StandardCreate-requestParams-1611193642.6311-859693761.txt
-rw-r--r-- 1 nginx nginx 2002 Jan 21 09:53 task-group-VideoCreate-requestParams-1611194033.985-2081486472.txt
-rw-r--r-- 1 nginx nginx 5191 Jan 21 10:00 task-group-VideoCreate-requestParams-1611194444.7456-1390776713.txt
-rw-r--r-- 1 nginx nginx 1162 Jan 21 10:15 task-group-VideoCreate-requestParams-1611195354.4384-150286619.txt
-rw-r--r-- 1 nginx nginx 255286 Jan 21 10:25 task-group-StandardCreate-requestParams-1611195951.7886-179865469.txt
-rw-r--r-- 1 nginx nginx 505 Jan 21 10:26 vendor-hprose-hprose-src-Hprose-Client-response-1611196005.4221-2066255666.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 21 10:26 vendor-hprose-hprose-src-Hprose-Client-context-1611196005.4224-1772338993.txt
-rw-r--r-- 1 nginx nginx 94 Jan 21 10:26 vendor-hprose-hprose-src-Hprose-Client-args-1611196005.4223-1942718750.txt
-rw-r--r-- 1 nginx nginx 505 Jan 21 10:27 vendor-hprose-hprose-src-Hprose-Client-response-1611196052.4831-1616116705.txt
-rw-r--r-- 1 nginx nginx 94 Jan 21 10:27 vendor-hprose-hprose-src-Hprose-Client-args-1611196052.4833-1693396185.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 21 10:27 vendor-hprose-hprose-src-Hprose-Client-context-1611196052.4834-722737667.txt
-rw-r--r-- 1 nginx nginx 505 Jan 21 13:16 vendor-hprose-hprose-src-Hprose-Client-response-1611206183.1099-954326273.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 21 13:16 vendor-hprose-hprose-src-Hprose-Client-context-1611206183.1101-1374021378.txt
-rw-r--r-- 1 nginx nginx 94 Jan 21 13:16 vendor-hprose-hprose-src-Hprose-Client-args-1611206183.11-501590956.txt
-rw-r--r-- 1 nginx nginx 2217 Jan 21 13:44 task-group-VideoCreate-requestParams-1611207852.9096-741188782.txt
-rw-r--r-- 1 nginx nginx 1248 Jan 21 13:47 task-group-VideoCreate-requestParams-1611208057.6225-186751811.txt
-rw-r--r-- 1 nginx nginx 249 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-response-1611211176.7538-1234184004.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-context-1611211176.754-2115515688.txt
-rw-r--r-- 1 nginx nginx 577 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-args-1611211176.754-149978282.txt
-rw-r--r-- 1 nginx nginx 4395 Jan 21 15:00 task-group-VideoCreate-requestParams-1611212401.7912-1032544328.txt
-rw-r--r-- 1 nginx nginx 3252 Jan 21 15:11 task-group-VideoCreate-requestParams-1611213060.6878-1627043027.txt
-rw-r--r-- 1 nginx nginx 1813 Jan 21 15:11 task-group-VideoCreate-requestParams-1611213102.6432-945644833.txt
drwxrwxr-x 2 nginx nginx 4096 Jan 21 15:49 debug
[root@8d8ff3df26b0 runtime]# cd /mcloud/www/channel-pub-api/console/runtime/
[root@8d8ff3df26b0 runtime]# ls -lrt
total 16
drwxrwxr-x 2 root root 4096 Jan 20 18:22 logs
-rw-r--r-- 1 root root 158 Jan 21 09:45 wxArticleContentNodeReplace.txt
-rw-r--r-- 1 root root 231 Jan 21 09:45 channel_output_video_url.txt
-rw-r--r-- 1 root root 105 Jan 21 15:33 api_authorizer_token.txt
[root@8d8ff3df26b0 runtime]# cd logs/
[root@8d8ff3df26b0 logs]# ls -lrt
total 132
-rw-r--r-- 1 root root 129426 Jan 21 10:26 app.log
[root@8d8ff3df26b0 logs]# cd ..
[root@8d8ff3df26b0 runtime]# cd ..
[root@8d8ff3df26b0 console]# cd ..
[root@8d8ff3df26b0 channel-pub-api]# ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 47
[root@8d8ff3df26b0 channel-pub-api]# ./yii upload-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 26
[root@8d8ff3df26b0 channel-pub-api]# ./yii pub-article-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 51
[root@8d8ff3df26b0 channel-pub-api]# ./yii source-callback-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 124
[root@8d8ff3df26b0 channel-pub-api]#
16. I entered container two and executed command lines. I confirmed that this container was receiving requests and running queue jobs.
Plaintext
[root@b49f6b25db29 /]# supervisorctl status
cronolog RUNNING pid 1033, uptime 1:43:35
nginx RUNNING pid 1018, uptime 1:43:35
php-fpm RUNNING pid 1012, uptime 1:43:35
report_client RUNNING pid 1032, uptime 1:43:35
yii-baijia-article-sync RUNNING pid 11401, uptime 0:00:00
yii-channel-app-task-sync STARTING
yii-copy-asset-queue-worker STARTING
yii-douyin-web-video-sync STARTING
yii-log-delete STARTING
yii-netease-tp-article-sync RUNNING pid 11242, uptime 0:01:18
yii-pub-article-queue-worker STARTING
yii-qq-cw-transaction-article-sync STARTING
yii-qq-cw-transaction-video-sync STARTING
yii-qq-tp-transaction-article-sync STARTING
yii-qq-tp-transaction-video-sync STARTING
yii-source-callback-queue-worker STARTING
yii-upload-asset-queue-worker STARTING
[root@b49f6b25db29 /]# cd /mcloud/www/channel-pub-api/api/runtime/
[root@b49f6b25db29 runtime]# ls -lrt
total 48
drwxrwxr-x 5 nginx nginx 4096 Jan 21 14:14 cache
drwxrwxr-x 2 nginx nginx 4096 Jan 21 14:14 logs
-rw-r--r-- 1 nginx nginx 1949 Jan 21 14:36 task-group-VideoCreate-requestParams-1611211008.518-1980950354.txt
-rw-r--r-- 1 nginx nginx 505 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-response-1611211155.5159-746357494.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-context-1611211155.5161-601717535.txt
-rw-r--r-- 1 nginx nginx 94 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-args-1611211155.5161-637322407.txt
-rw-r--r-- 1 nginx nginx 1813 Jan 21 15:11 task-group-VideoCreate-requestParams-1611213090.2354-1640529486.txt
-rw-r--r-- 1 nginx nginx 1813 Jan 21 15:11 task-group-VideoCreate-requestParams-1611213112.5391-1676894084.txt
-rw-r--r-- 1 nginx nginx 3318 Jan 21 15:13 task-group-VideoCreate-requestParams-1611213185.8286-1318629323.txt
-rw-r--r-- 1 nginx nginx 2291 Jan 21 15:27 task-group-VideoCreate-requestParams-1611214022.4444-1384266047.txt
drwxrwxr-x 2 nginx nginx 4096 Jan 21 15:56 debug
[root@b49f6b25db29 runtime]# cd /mcloud/www/channel-pub-api/console/runtime/
[root@b49f6b25db29 runtime]# ls -lrt
total 12
-rw-r--r-- 1 root root 169 Jan 21 14:30 wxArticleContentNodeReplace.txt
-rw-r--r-- 1 root root 231 Jan 21 14:30 channel_output_video_url.txt
-rw-r--r-- 1 root root 105 Jan 21 15:43 api_authorizer_token.txt
[root@b49f6b25db29 runtime]# cd ..
[root@b49f6b25db29 console]# cd ..
[root@b49f6b25db29 channel-pub-api]# ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 47
[root@b49f6b25db29 channel-pub-api]# ./yii upload-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 26
[root@b49f6b25db29 channel-pub-api]# ./yii pub-article-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 51
[root@b49f6b25db29 channel-pub-api]# ./yii source-callback-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 124
[root@b49f6b25db29 channel-pub-api]#
17. When the info command prints information about the queue status, queues in the reserved state generally do not persist for too long. The duration is about 1 minute, which meets expectations. All four queue groups finish executing in about 10 minutes. Furthermore, no new issues have occurred in the cluster environment. When I have ample time later, I plan to carefully analyze the difference between listening and running. For now, I have only solved the problem; I know what happens, but not why.