1. Reference URL: https://www.shuijingwanwq.com/2021/01/22/4698/ . The queue/run command replaces the queue/listen command. In the local environment, check the status of the queue/run command queue.
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 20
- delayed: 0
- reserved: 0
- done: 0
PS E:\wwwroot\channel-pub-api> ./yii upload-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 0
PS E:\wwwroot\channel-pub-api> ./yii pub-article-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 0
PS E:\wwwroot\channel-pub-api> ./yii source-callback-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 0
2. View the queue data in Redis, cpa:queue:copy:asset.messages. The data volume is 20 entries. See Figure 1
3. Open two command-line windows. After running the queue in window 1, run the same queue in window 2. This is to confirm whether the same queue job will be executed repeatedly when the same queue runs simultaneously (as in a cluster deployment). After running for a while, window 1 has completed 2 queue jobs, and window 2 has completed 3 queue jobs. See Figure 2
4. After running for a while, view the queue data in Redis, cpa:queue:copy:asset.messages. The data volume is 15 entries. See Figure 3
5. After running for another while, run the same queue in window 3. Checking the queue status confirms that among the queue jobs, 10 are waiting, 2 are in progress, and 8 are completed. See Figure 4
6. Finally, check the execution status in the three windows respectively. Window 1 executed 8 queue jobs. Window 2 executed 12 queue jobs. Window 3 executed 0 queue jobs, the reason should be that it timed out and failed when executing the first queue job.
Window 1
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-08-31 17:37:31 [pid: 64508] - Worker is started
2021-08-31 17:37:32 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 17:42:35 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (303.486 s)
2021-08-31 17:42:36 [3] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 17:49:45 [3] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (429.834 s)
2021-08-31 17:49:46 [7] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 17:57:23 [7] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (457.296 s)
2021-08-31 17:57:24 [10] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:06:16 [10] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (531.560 s)
2021-08-31 18:06:16 [14] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:14:02 [14] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (465.649 s)
2021-08-31 18:14:02 [15] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:21:18 [15] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (436.172 s)
2021-08-31 18:21:19 [18] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:27:51 [18] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (392.171 s)
2021-08-31 18:27:52 [20] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:27:54 [20] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (2.273 s)
2021-08-31 18:27:54 [pid: 64508] - Worker is stopped (0:50:23)
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 20
Window 2
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-08-31 17:37:37 [pid: 63068] - Worker is started
2021-08-31 17:37:37 [2] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:44:18 [2] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (400.251 s)
2021-08-31 17:44:18 [4] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:44:20 [4] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (1.969 s)
2021-08-31 17:44:21 [5] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:49:35 [5] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (314.755 s)
2021-08-31 17:49:36 [6] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:56:27 [6] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (411.015 s)
2021-08-31 17:56:27 [8] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:56:29 [8] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (1.854 s)
2021-08-31 17:56:30 [9] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:02:47 [9] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (377.192 s)
2021-08-31 18:02:48 [12] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:02:50 [12] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (2.375 s)
2021-08-31 18:02:51 [13] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:09:42 [13] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (411.228 s)
2021-08-31 18:09:42 [11] common\jobs\CopyAssetJob (attempt: 2, pid: 63068) - Started
2021-08-31 18:17:07 [11] common\jobs\CopyAssetJob (attempt: 2, pid: 63068) - Done (445.086 s)
2021-08-31 18:17:08 [16] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:17:11 [16] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (3.012 s)
2021-08-31 18:17:12 [17] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:22:13 [17] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (300.874 s)
2021-08-31 18:22:13 [19] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:28:40 [19] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (386.913 s)
2021-08-31 18:28:40 [pid: 63068] - Worker is stopped (0:51:03)
Window 3
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-08-31 17:59:31 [pid: 65344] - Worker is started
2021-08-31 17:59:31 [11] common\jobs\CopyAssetJob (attempt: 1, pid: 65344) - Started
2021-08-31 18:09:32 [11] common\jobs\CopyAssetJob (attempt: 1, pid: 65344) - Error
> Symfony\Component\Process\Exception\ProcessTimedOutException: The process "C:\php-7.4.6\php.exe E:\wwwroot\channel-pub-api\yii "copy-asset-queue/exec" 11 600 1 65344 --verbose=1 --color=0" exceeded the timeout of 600 seconds.
2021-08-31 19:07:22 [pid: 65344] - Worker is stopped (1:07:51)
7. Check the operation results of the 20 queue jobs. relative_path is the relative path of the downloaded file, and all queue jobs executed successfully because the relative paths all exist. However, searching for the video files that should have been downloaded, their quantity should equal 30. But the actual number of video files in the download directory is 32. The current problem is that it is unclear how the 2 extra files were generated (one of the files should have already existed, so only 1 extra file was actually added). See Figure 5
8. It has been confirmed that the queue job in window 3 timed out and failed, affecting 2 queue jobs. Although it failed, the files had already been downloaded successfully. See Figure 6
PS E:\wwwroot\channel-pub-api> ./yii source-callback-queue/info --color=0
Jobs
- waiting: 2
- delayed: 0
- reserved: 0
- done: 0
9. The reason why the queue job in window 3 timed out should be that the downstream bandwidth of the local environment is limited. Window 1 and window 2 had already occupied a large proportion of the bandwidth resources, leaving no bandwidth resources available for window 3, resulting in a timeout. Repeat the above steps. I decided to adjust the timeout from 600 seconds to 1800 seconds. All queue jobs in the three windows executed successfully, with the longest execution time being: 708.338 s, no wonder it timed out and failed previously. Because the relative paths all exist. However, searching for the video files that should have been downloaded, their quantity should equal 30. But the actual number of video files in the download directory is 31 (this should be due to internal program reasons, to be analyzed later). See Figure 7
Window 1
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-09-01 10:37:09 [pid: 73336] - Worker is started
2021-09-01 10:37:09 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 10:43:23 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (374.069 s)
2021-09-01 10:43:24 [6] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 10:52:59 [6] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (575.304 s)
2021-09-01 10:53:02 [10] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:03:11 [10] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (609.431 s)
2021-09-01 11:03:12 [14] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:13:11 [14] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (598.621 s)
2021-09-01 11:13:12 [16] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:13:14 [16] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (2.588 s)
2021-09-01 11:13:15 [17] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:21:02 [17] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (466.632 s)
2021-09-01 11:21:03 [20] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:21:05 [20] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (2.528 s)
2021-09-01 11:21:05 [pid: 73336] - Worker is stopped (0:43:56)
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 20
PS E:\wwwroot\channel-pub-api> ./yii source-callback-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 0
PS E:\wwwroot\channel-pub-api>
Window 2
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-09-01 10:38:02 [pid: 67976] - Worker is started
2021-09-01 10:38:03 [3] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 10:49:51 [3] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (708.338 s)
2021-09-01 10:49:52 [8] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 10:49:55 [8] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (3.721 s)
2021-09-01 10:49:56 [9] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 10:57:06 [9] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (429.860 s)
2021-09-01 10:57:06 [11] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 11:08:16 [11] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (669.532 s)
2021-09-01 11:08:17 [2] common\jobs\CopyAssetJob (attempt: 2, pid: 67976) - Started
2021-09-01 11:18:33 [2] common\jobs\CopyAssetJob (attempt: 2, pid: 67976) - Done (616.247 s)
2021-09-01 11:18:35 [19] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 11:25:39 [19] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (424.108 s)
2021-09-01 11:25:40 [pid: 67976] - Worker is stopped (0:47:38)
Window 3
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-09-01 10:38:29 [pid: 75656] - Worker is started
2021-09-01 10:38:30 [4] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 10:38:32 [4] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (2.338 s)
2021-09-01 10:38:33 [5] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 10:46:21 [5] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (468.368 s)
2021-09-01 10:46:22 [7] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 10:58:04 [7] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (702.171 s)
2021-09-01 10:58:05 [12] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 10:58:08 [12] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (2.769 s)
2021-09-01 10:58:09 [13] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 11:06:11 [13] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (482.736 s)
2021-09-01 11:06:12 [15] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 11:16:43 [15] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (631.170 s)
2021-09-01 11:16:44 [18] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 11:24:50 [18] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (485.917 s)
2021-09-01 11:24:55 [pid: 75656] - Worker is stopped (0:46:27)
10. In conclusion, the Yii2 queue extension supports parallel execution, and there are no issues even when running simultaneously in a cluster.