yii2 queue extension, in the Redis driver, when the info command prints information about the queue status, the analysis of the queue status of the reserved state (2)

by 永夜 · 2021/01/22

浏览量： 1

1. Reference URL:https://www.shuijingwanwq.com/2020/09/30/4552/, this is a previous blog.

2. When the cluster is deployed, the overall 4 groups of queues are all executed. It can be controlled within half an hour. However, sometimes, the length of time is still uncontrollable, and may even exceed an hour.

3. Decide to print log information in the relevant places in the extended source code.

4. // Copy the source resource file to the resource directory published by the channel. After the queue task is successfully executed, call the corresponding service, otherwise, insert the release log (asynchronous). Edit common\services\AssetService::copyAssetsAsync($data, $assets). -167 means on line 167. The following codes are all like this.


@file_put_contents(Yii::getAlias('@runtime') . '/common-services-AssetService-copyAssetsAsync-data-' . $data['task_id'] . '-' . date("-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-167.txt', print_r($data, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-services-AssetService-copyAssetsAsync-assets-' . $data['task_id'] . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-168.txt', print_r($assets, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-services-AssetService-copyAssetsAsync-copy_queue_info-' . $data['task_id'] . '-' . $copy_queue_info . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-178.txt', print_r($copy_queue_info, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-services-AssetService-copyAssetsAsync-isWaiting-' . $data['task_id'] . '-' . $copy_queue_info . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-180.txt', print_r($copy_queue_info, true), FILE_APPEND | LOCK_EX);

5. // Copy the resource file queue job: Copy the source resource file to the resource directory published by the channel. Edit common\jobs\copyassetjob->execute($queue)


@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-queue-workerPid-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-41.txt', print_r($queue->workerPid, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-queue-status-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-42.txt', print_r($queue->status(), true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-queue-attempts-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-43.txt', print_r($queue->attempts, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-queue-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-44.txt', print_r($queue, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-empty-assetEnabledItems-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-52.txt', print_r($this->taskId, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-jobs-CopyAssetJob-execute-assetEnabledItems-' . $this->taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-77.txt', print_r($assetEnabledItems, true), FILE_APPEND | LOCK_EX);

6. // Copy the resource file queue event handler, and the job of copying the resource file queue is successfully executed. Edit common\components\queue\copyAssetEventHandler::afterexec(execEvent $event)


@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterExec-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-36.txt', print_r($taskId, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterExec-serviceClass-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-44.txt', print_r($serviceClass, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterExec-serviceAction-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-45.txt', print_r($serviceAction, true), FILE_APPEND | LOCK_EX);

7. // Copy the resource file queue event handler, and the job of copying the resource file queue fails to execute. Edit common\components\queue\copyAssetEventHandler::afterError(execEvent $event)


@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterError-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-56.txt', print_r($taskId, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterError-serviceClass-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-64.txt', print_r($serviceClass, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/common-components-queue-CopyAssetEventHandler-afterError-serviceAction-' . $taskId . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-65.txt', print_r($serviceAction, true), FILE_APPEND | LOCK_EX);

8. Edit /vendor/yiisoft/yii2-queue/src/drivers/redis/command.php


@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Command-actionListen-' . $timeout . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-70.txt', print_r($timeout, true), FILE_APPEND | LOCK_EX);

9. Edit /vendor/yiisoft/yii2-queue/src/queue.php


@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-event-' . $event->id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-198.txt', print_r($event, true), FILE_APPEND | LOCK_EX);
vendor-yiisoft-yii2-queue-src-Queue-event-1-2021-01-12-14-11-09-1610431869.6843-476980414-198.txt
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-try-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-247.txt', print_r($this, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-try-event-result-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-249.txt', print_r($event->result, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-catch-Exception-error-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-251.txt', print_r($error, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-catch-Throwable-error-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-255.txt', print_r($error, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-event-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-259.txt', print_r($event, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleMessage-trigger-event-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-261.txt', print_r($event, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-297.txt', print_r($event, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-event-attempt-' . $event->attempt . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-299.txt', print_r($event->attempt, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-this-attempts-' . $this->attempts . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-300.txt', print_r($this->attempts, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-event-error-' . $this->attempts . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-303.txt', print_r($event->error, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-Queue-handleError-event-retry-' . $this->attempts . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-306.txt', print_r($event->retry, true), FILE_APPEND | LOCK_EX);

10. Edit /vendor/yiisoft/yii2-queue/src/drivers/redis/queue.php


@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-' . $timeout . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-58.txt', print_r($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-payload-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-63.txt', print_r($payload, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-delete-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-65.txt', print_r($payload, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-break-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-69.txt', var_dump($repeat, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-139.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-142.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-if-timeout-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-151.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-elseif-timeout-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-154.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-if-id-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-157.txt', var_dump($timeout, true), FILE_APPEND | LOCK_EX);
@file_put_contents(Yii::getAlias('@runtime') . '/vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-return-' . $id . '-' . date("Y-m-d-H-i-s") . '-' . microtime(true) . '-' . mt_rand()  . '-165.txt', var_dump($message, true), FILE_APPEND | LOCK_EX);

11. Decide to stop the monitoring of the supervisor-based queue first. Schedule to manually execute monitoring commands.


[root@055c3647ea66 /]# supervisorctl status
cronolog                         RUNNING   pid 1002, uptime 0:00:13
nginx                            RUNNING   pid 999, uptime 0:00:13
php-fpm                          RUNNING   pid 1000, uptime 0:00:13
report_client                    RUNNING   pid 1001, uptime 0:00:13
[root@055c3647ea66 /]# cd /mcloud/www/channel-pub-api/
[root@85b422810e4f channel-pub-api]# ./yii copy-asset-queue/listen --verbose=1 --color=0
2021-01-12 16:18:21 [pid: 1815] - Worker is started
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
string(55) "O:24:"common\jobs\CopyAssetJob":1:{s:6:"taskId";i:495;}"
bool(true)
2021-01-12 16:18:22 [61] common\jobs\CopyAssetJob (attempt: 1, pid: 1815) - Started
2021-01-12 16:18:22 [61] common\jobs\CopyAssetJob (attempt: 1, pid: 1815) - Done (0.795 s)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
int(3)
bool(true)
^Cint(3)
bool(true)
2021-01-12 16:18:34 [pid: 1815] - Worker is stopped (0:00:13)
[root@85b422810e4f channel-pub-api]# ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 61

12. Decide to restore the monitoring of the supervisor-based queue. View the generated log files.


[root@3354317ffbed /]# supervisorctl status
cronolog                                                               RUNNING   pid 1055, uptime 0:00:04
nginx                                                                  RUNNING   pid 1020, uptime 0:00:04
php-fpm                                                                RUNNING   pid 1017, uptime 0:00:04
report_client                                                          RUNNING   pid 1052, uptime 0:00:04
yii-baijia-article-sync                                                RUNNING   pid 1121, uptime 0:00:00
yii-channel-app-task-sync                                              STARTING
yii-copy-asset-queue-worker:yii-copy-asset-queue-worker_00             RUNNING   pid 1015, uptime 0:00:04
yii-copy-asset-queue-worker:yii-copy-asset-queue-worker_01             RUNNING   pid 1016, uptime 0:00:04
yii-douyin-web-video-sync                                              STARTING
yii-log-delete                                                         STARTING
yii-netease-tp-article-sync                                            STARTING
yii-pub-article-queue-worker:yii-pub-article-queue-worker_00           RUNNING   pid 1010, uptime 0:00:04
yii-pub-article-queue-worker:yii-pub-article-queue-worker_01           RUNNING   pid 1009, uptime 0:00:04
yii-qq-cw-transaction-article-sync                                     STARTING
yii-qq-cw-transaction-video-sync                                       STARTING
yii-qq-tp-transaction-article-sync                                     STARTING
yii-qq-tp-transaction-video-sync                                       STARTING
yii-source-callback-queue-worker:yii-source-callback-queue-worker_00   RUNNING   pid 1029, uptime 0:00:04
yii-source-callback-queue-worker:yii-source-callback-queue-worker_01   RUNNING   pid 1030, uptime 0:00:04
yii-upload-asset-queue-worker:yii-upload-asset-queue-worker_00         RUNNING   pid 1011, uptime 0:00:04
yii-upload-asset-queue-worker:yii-upload-asset-queue-worker_01         RUNNING   pid 1012, uptime 0:00:04
[root@3354317ffbed runtime]# ls -lrt
total 64
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9494-1819463520-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.9496-794234002-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.953-1837796717-138.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9578-108840870-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.958-592107120-57.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9617-1880485471-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.9619-1414847196-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.9619-1179420058-138.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9628-394649028-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.963-1060811221-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.966-593417676-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.9666-580562356-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-31-1610440351.968-1954959781-141.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9749-573783946-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.9751-211630453-57.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-31-1610440351.9765-2041102433-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-31-1610440351.9767-689870268-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.9781-1224010356-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-31-1610440351.9797-903147806-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-31-1610440351.9822-1531347538-141.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-31-1610440351.9864-1375602874-141.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-32-1610440352.0892-1852556723-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-32-1610440352.0894-303473184-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-32-1610440352.0935-1190705523-138.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-Queue-actionListen-3-2021-01-12-16-32-32-1610440352.0951-433868058-69.txt
-rw-r--r-- 1 root root 1 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-run-3-2021-01-12-16-32-32-1610440352.0953-158476143-57.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-32-1610440352.0989-1835560487-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-32-1610440352.1096-318569225-141.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-if-id-2021-01-12-16-32-35-1610440355.2908-1182646222-156.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-2021-01-12-16-32-35-1610440355.2911-1232420494-138.txt
-rw-r--r-- 1 root root 0 Jan 12 16:32 vendor-yiisoft-yii2-queue-src-drivers-redis-Queue-reserve-set-2021-01-12-16-32-35-1610440355.2944-141684420-141.txt

13. It is suspected that there is a serious locking problem in the cluster environment, or in multiple processes. Docker deployment, the queue/run command replaces the queue/listen command. Edit /etc/supervisord.d/yii-copy-asset-queue-worker.ini. The other 3-group queue files are also re-edited.


[program:yii-copy-asset-queue-worker]
command = bash -c 'sleep 60 &amp;&amp; exec php /mcloud/www/channel-pub-api/yii copy-asset-queue/run'
autorestart = true
startsecs = 60
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-copy-asset-queue-worker-stderr.log
stdout_logfile = /data/logs/yii-copy-asset-queue-worker-stdout.log


[program:yii-upload-asset-queue-worker]
command = bash -c 'sleep 60 &amp;&amp; exec php /mcloud/www/channel-pub-api/yii upload-asset-queue/run'
autorestart = true
startsecs = 60
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-upload-asset-queue-worker-stderr.log
stdout_logfile = /data/logs/yii-upload-asset-queue-worker-stdout.log


[program:yii-pub-article-queue-worker]
command = bash -c 'sleep 60 &amp;&amp; exec php /mcloud/www/channel-pub-api/yii pub-article-queue/run'
autorestart = true
startsecs = 60
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-pub-article-queue-worker-stderr.log
stdout_logfile = /data/logs/yii-pub-article-queue-worker-stdout.log


[program:yii-source-callback-queue-worker]
command = bash -c 'sleep 60 &amp;&amp; exec php /mcloud/www/channel-pub-api/yii source-callback-queue/run'
autorestart = true
startsecs = 60
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-source-callback-queue-worker-stderr.log
stdout_logfile = /data/logs/yii-source-callback-queue-worker-stdout.log

14. Deploy 2 containers in the test environment. as shown in Figure 1

Figure 1

15. Enter container 1 and execute the command line. Determine this container is receiving requests and running queue jobs. as shown in Figure 2

Figure 2


[root@8d8ff3df26b0 /]# supervisorctl status
cronolog                             RUNNING   pid 1034, uptime 1:47:55
nginx                                RUNNING   pid 1019, uptime 1:47:55
php-fpm                              RUNNING   pid 1014, uptime 1:47:55
report_client                        RUNNING   pid 1033, uptime 1:47:55
yii-baijia-article-sync              RUNNING   pid 11891, uptime 0:00:00
yii-channel-app-task-sync            STARTING
yii-copy-asset-queue-worker          STARTING
yii-douyin-web-video-sync            STARTING
yii-log-delete                       STARTING
yii-netease-tp-article-sync          RUNNING   pid 11703, uptime 0:01:38
yii-pub-article-queue-worker         STARTING
yii-qq-cw-transaction-article-sync   STARTING
yii-qq-cw-transaction-video-sync     STARTING
yii-qq-tp-transaction-article-sync   STARTING
yii-qq-tp-transaction-video-sync     STARTING
yii-source-callback-queue-worker     STARTING
yii-upload-asset-queue-worker        STARTING
[root@8d8ff3df26b0 /]# cd /mcloud/www/channel-pub-api/api/runtime/
[root@8d8ff3df26b0 runtime]# ls -lrt
total 668
drwxrwxr-x 5 nginx nginx   4096 Jan 20 16:20 cache
drwxrwxr-x 2 nginx nginx   4096 Jan 20 16:20 logs
-rw-r--r-- 1 nginx nginx    325 Jan 20 16:33 task-group-StandardCreate-requestParams-1611131592.3771-910238203.txt
-rw-r--r-- 1 nginx nginx   1278 Jan 20 16:48 task-group-VideoCreate-requestParams-1611132522.5135-1170578969.txt
-rw-r--r-- 1 nginx nginx    325 Jan 20 16:55 task-group-StandardCreate-requestParams-1611132921.2381-159358239.txt
-rw-r--r-- 1 nginx nginx    325 Jan 20 17:18 task-group-StandardCreate-requestParams-1611134312.9066-273770349.txt
-rw-r--r-- 1 nginx nginx    325 Jan 20 17:20 task-group-StandardCreate-requestParams-1611134415.4961-206630537.txt
-rw-r--r-- 1 nginx nginx   2435 Jan 20 17:32 task-group-VideoCreate-requestParams-1611135156.2789-1685654955.txt
-rw-r--r-- 1 nginx nginx   4335 Jan 20 17:50 task-group-VideoCreate-requestParams-1611136237.5574-1504010220.txt
-rw-r--r-- 1 nginx nginx   3266 Jan 20 17:59 task-group-VideoCreate-requestParams-1611136785.0646-1310530126.txt
-rw-r--r-- 1 nginx nginx   1151 Jan 20 18:21 task-group-VideoCreate-requestParams-1611138109.7108-1428510960.txt
-rw-r--r-- 1 nginx nginx   1133 Jan 20 19:11 task-group-StandardCreate-requestParams-1611141071.2366-451937142.txt
-rw-r--r-- 1 nginx nginx   1161 Jan 20 19:28 task-group-VideoCreate-requestParams-1611142135.7108-1213586108.txt
-rw-r--r-- 1 nginx nginx   1103 Jan 20 19:40 task-group-VideoCreate-requestParams-1611142825.0161-333542923.txt
-rw-r--r-- 1 nginx nginx    505 Jan 20 19:52 vendor-hprose-hprose-src-Hprose-Client-response-1611143571.9702-178210933.txt
-rw-r--r-- 1 nginx nginx     94 Jan 20 19:52 vendor-hprose-hprose-src-Hprose-Client-args-1611143571.9704-862498366.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 20 19:52 vendor-hprose-hprose-src-Hprose-Client-context-1611143571.9704-185265682.txt
-rw-r--r-- 1 nginx nginx    505 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-response-1611143859.2669-528070097.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-context-1611143859.2671-135288224.txt
-rw-r--r-- 1 nginx nginx     94 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-args-1611143859.267-757011716.txt
-rw-r--r-- 1 nginx nginx    505 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-response-1611143876.4248-983204429.txt
-rw-r--r-- 1 nginx nginx     94 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-args-1611143876.4252-311858334.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 20 19:57 vendor-hprose-hprose-src-Hprose-Client-context-1611143876.4253-325345408.txt
-rw-r--r-- 1 nginx nginx    505 Jan 20 20:00 vendor-hprose-hprose-src-Hprose-Client-response-1611144013.4502-160577375.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 20 20:00 vendor-hprose-hprose-src-Hprose-Client-context-1611144013.4504-1265475831.txt
-rw-r--r-- 1 nginx nginx     94 Jan 20 20:00 vendor-hprose-hprose-src-Hprose-Client-args-1611144013.4503-2132911066.txt
-rw-r--r-- 1 nginx nginx   1447 Jan 20 20:04 task-group-VideoCreate-requestParams-1611144289.3014-730024961.txt
-rw-r--r-- 1 nginx nginx  36039 Jan 20 20:21 task-group-StandardCreate-requestParams-1611145302.5614-76746077.txt
-rw-r--r-- 1 nginx nginx  16295 Jan 20 20:26 task-group-StandardCreate-requestParams-1611145608.7813-919723561.txt
-rw-r--r-- 1 nginx nginx   5264 Jan 20 20:29 task-group-StandardCreate-requestParams-1611145745.6312-1832504444.txt
-rw-r--r-- 1 nginx nginx   1209 Jan 20 20:29 task-group-StandardCreate-requestParams-1611145763.8136-1359951868.txt
-rw-r--r-- 1 nginx nginx    505 Jan 20 20:34 vendor-hprose-hprose-src-Hprose-Client-response-1611146090.9852-1952201824.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 20 20:34 vendor-hprose-hprose-src-Hprose-Client-context-1611146090.9854-1038382139.txt
-rw-r--r-- 1 nginx nginx     94 Jan 20 20:34 vendor-hprose-hprose-src-Hprose-Client-args-1611146090.9854-122587697.txt
-rw-r--r-- 1 nginx nginx    505 Jan 20 20:35 vendor-hprose-hprose-src-Hprose-Client-response-1611146152.5181-1787102293.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 20 20:35 vendor-hprose-hprose-src-Hprose-Client-context-1611146152.5183-163122809.txt
-rw-r--r-- 1 nginx nginx     94 Jan 20 20:35 vendor-hprose-hprose-src-Hprose-Client-args-1611146152.5183-2040247314.txt
-rw-r--r-- 1 nginx nginx   1287 Jan 21 09:18 task-group-StandardCreate-requestParams-1611191906.7021-1963669870.txt
-rw-r--r-- 1 nginx nginx   3824 Jan 21 09:44 task-group-StandardCreate-requestParams-1611193482.0351-1386708367.txt
-rw-r--r-- 1 nginx nginx  74055 Jan 21 09:47 task-group-StandardCreate-requestParams-1611193642.6311-859693761.txt
-rw-r--r-- 1 nginx nginx   2002 Jan 21 09:53 task-group-VideoCreate-requestParams-1611194033.985-2081486472.txt
-rw-r--r-- 1 nginx nginx   5191 Jan 21 10:00 task-group-VideoCreate-requestParams-1611194444.7456-1390776713.txt
-rw-r--r-- 1 nginx nginx   1162 Jan 21 10:15 task-group-VideoCreate-requestParams-1611195354.4384-150286619.txt
-rw-r--r-- 1 nginx nginx 255286 Jan 21 10:25 task-group-StandardCreate-requestParams-1611195951.7886-179865469.txt
-rw-r--r-- 1 nginx nginx    505 Jan 21 10:26 vendor-hprose-hprose-src-Hprose-Client-response-1611196005.4221-2066255666.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 21 10:26 vendor-hprose-hprose-src-Hprose-Client-context-1611196005.4224-1772338993.txt
-rw-r--r-- 1 nginx nginx     94 Jan 21 10:26 vendor-hprose-hprose-src-Hprose-Client-args-1611196005.4223-1942718750.txt
-rw-r--r-- 1 nginx nginx    505 Jan 21 10:27 vendor-hprose-hprose-src-Hprose-Client-response-1611196052.4831-1616116705.txt
-rw-r--r-- 1 nginx nginx     94 Jan 21 10:27 vendor-hprose-hprose-src-Hprose-Client-args-1611196052.4833-1693396185.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 21 10:27 vendor-hprose-hprose-src-Hprose-Client-context-1611196052.4834-722737667.txt
-rw-r--r-- 1 nginx nginx    505 Jan 21 13:16 vendor-hprose-hprose-src-Hprose-Client-response-1611206183.1099-954326273.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 21 13:16 vendor-hprose-hprose-src-Hprose-Client-context-1611206183.1101-1374021378.txt
-rw-r--r-- 1 nginx nginx     94 Jan 21 13:16 vendor-hprose-hprose-src-Hprose-Client-args-1611206183.11-501590956.txt
-rw-r--r-- 1 nginx nginx   2217 Jan 21 13:44 task-group-VideoCreate-requestParams-1611207852.9096-741188782.txt
-rw-r--r-- 1 nginx nginx   1248 Jan 21 13:47 task-group-VideoCreate-requestParams-1611208057.6225-186751811.txt
-rw-r--r-- 1 nginx nginx    249 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-response-1611211176.7538-1234184004.txt
-rw-r--r-- 1 nginx nginx   6151 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-context-1611211176.754-2115515688.txt
-rw-r--r-- 1 nginx nginx    577 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-args-1611211176.754-149978282.txt
-rw-r--r-- 1 nginx nginx   4395 Jan 21 15:00 task-group-VideoCreate-requestParams-1611212401.7912-1032544328.txt
-rw-r--r-- 1 nginx nginx   3252 Jan 21 15:11 task-group-VideoCreate-requestParams-1611213060.6878-1627043027.txt
-rw-r--r-- 1 nginx nginx   1813 Jan 21 15:11 task-group-VideoCreate-requestParams-1611213102.6432-945644833.txt
drwxrwxr-x 2 nginx nginx   4096 Jan 21 15:49 debug
[root@8d8ff3df26b0 runtime]# cd /mcloud/www/channel-pub-api/console/runtime/
[root@8d8ff3df26b0 runtime]# ls -lrt
total 16
drwxrwxr-x 2 root root 4096 Jan 20 18:22 logs
-rw-r--r-- 1 root root  158 Jan 21 09:45 wxArticleContentNodeReplace.txt
-rw-r--r-- 1 root root  231 Jan 21 09:45 channel_output_video_url.txt
-rw-r--r-- 1 root root  105 Jan 21 15:33 api_authorizer_token.txt
[root@8d8ff3df26b0 runtime]# cd logs/
[root@8d8ff3df26b0 logs]# ls -lrt
total 132
-rw-r--r-- 1 root root 129426 Jan 21 10:26 app.log
[root@8d8ff3df26b0 logs]# cd ..
[root@8d8ff3df26b0 runtime]# cd ..
[root@8d8ff3df26b0 console]# cd ..
[root@8d8ff3df26b0 channel-pub-api]# ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 47
[root@8d8ff3df26b0 channel-pub-api]# ./yii upload-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 26
[root@8d8ff3df26b0 channel-pub-api]# ./yii pub-article-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 51
[root@8d8ff3df26b0 channel-pub-api]# ./yii source-callback-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 124
[root@8d8ff3df26b0 channel-pub-api]#

16. Enter the container 2 and execute the command line. Determine this container is receiving requests and running queue jobs.


[root@b49f6b25db29 /]# supervisorctl status
cronolog                             RUNNING   pid 1033, uptime 1:43:35
nginx                                RUNNING   pid 1018, uptime 1:43:35
php-fpm                              RUNNING   pid 1012, uptime 1:43:35
report_client                        RUNNING   pid 1032, uptime 1:43:35
yii-baijia-article-sync              RUNNING   pid 11401, uptime 0:00:00
yii-channel-app-task-sync            STARTING
yii-copy-asset-queue-worker          STARTING
yii-douyin-web-video-sync            STARTING
yii-log-delete                       STARTING
yii-netease-tp-article-sync          RUNNING   pid 11242, uptime 0:01:18
yii-pub-article-queue-worker         STARTING
yii-qq-cw-transaction-article-sync   STARTING
yii-qq-cw-transaction-video-sync     STARTING
yii-qq-tp-transaction-article-sync   STARTING
yii-qq-tp-transaction-video-sync     STARTING
yii-source-callback-queue-worker     STARTING
yii-upload-asset-queue-worker        STARTING
[root@b49f6b25db29 /]# cd /mcloud/www/channel-pub-api/api/runtime/
[root@b49f6b25db29 runtime]# ls -lrt
total 48
drwxrwxr-x 5 nginx nginx 4096 Jan 21 14:14 cache
drwxrwxr-x 2 nginx nginx 4096 Jan 21 14:14 logs
-rw-r--r-- 1 nginx nginx 1949 Jan 21 14:36 task-group-VideoCreate-requestParams-1611211008.518-1980950354.txt
-rw-r--r-- 1 nginx nginx  505 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-response-1611211155.5159-746357494.txt
-rw-r--r-- 1 nginx nginx 6151 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-context-1611211155.5161-601717535.txt
-rw-r--r-- 1 nginx nginx   94 Jan 21 14:39 vendor-hprose-hprose-src-Hprose-Client-args-1611211155.5161-637322407.txt
-rw-r--r-- 1 nginx nginx 1813 Jan 21 15:11 task-group-VideoCreate-requestParams-1611213090.2354-1640529486.txt
-rw-r--r-- 1 nginx nginx 1813 Jan 21 15:11 task-group-VideoCreate-requestParams-1611213112.5391-1676894084.txt
-rw-r--r-- 1 nginx nginx 3318 Jan 21 15:13 task-group-VideoCreate-requestParams-1611213185.8286-1318629323.txt
-rw-r--r-- 1 nginx nginx 2291 Jan 21 15:27 task-group-VideoCreate-requestParams-1611214022.4444-1384266047.txt
drwxrwxr-x 2 nginx nginx 4096 Jan 21 15:56 debug
[root@b49f6b25db29 runtime]# cd /mcloud/www/channel-pub-api/console/runtime/
[root@b49f6b25db29 runtime]# ls -lrt
total 12
-rw-r--r-- 1 root root 169 Jan 21 14:30 wxArticleContentNodeReplace.txt
-rw-r--r-- 1 root root 231 Jan 21 14:30 channel_output_video_url.txt
-rw-r--r-- 1 root root 105 Jan 21 15:43 api_authorizer_token.txt
[root@b49f6b25db29 runtime]# cd ..
[root@b49f6b25db29 console]# cd ..
[root@b49f6b25db29 channel-pub-api]# ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 47
[root@b49f6b25db29 channel-pub-api]# ./yii upload-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 26
[root@b49f6b25db29 channel-pub-api]# ./yii pub-article-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 51
[root@b49f6b25db29 channel-pub-api]# ./yii source-callback-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 124
[root@b49f6b25db29 channel-pub-api]#

17. When the info command prints information about the queue status, the queue of the reserved state will basically not last for too long. The duration is about 1 minute. in line with expectations. The overall 4 groups of queues are all executed. It can be controlled within 10 minutes. And in the cluster environment, there are no new problems. When there is plenty of time in the future, it is necessary to carefully analyze the difference between monitoring and running. Now just solved the problem. Know it but don’t know why.

yii2 queue extension, in the Redis driver, when the info command prints information about the queue status, the analysis of the queue status of the reserved state (2)

You may also like...

Leave a Reply Cancel reply

yii2 queue extension, in the Redis driver, when the info command prints information about the queue status, the analysis of the queue status of the reserved state (2)

You may also like...

Go Concurrency Learning: The Necessity of a Channel from Factorial Cases

Go + Gin actual RESTful API: from environment build (WSL/Docker/VS code) to Git commit and GitHub hosting

Construct a test system in Refactoring: Improve the Design of Existing Code (2nd Edition) – Writing the sample code to be tested

Leave a Reply Cancel reply