In the local environment, parallel running in the test queue (Yii2 queue extension, queue/run command) mode
1. Reference URL:https://www.shuijingwanwq.com/2021/01/22/4698/. The queue/run command replaces the queue/listen command. In the local environment, check the status of the queue of queue/run commands.
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 20
- delayed: 0
- reserved: 0
- done: 0
PS E:\wwwroot\channel-pub-api> ./yii upload-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 0
PS E:\wwwroot\channel-pub-api> ./yii pub-article-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 0
PS E:\wwwroot\channel-pub-api> ./yii source-callback-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 0
2. Check the queue data in redis, cpa:queue:copy:asset.messages. The amount of data is 20. as shown in Figure 1
Figure 1
3. Open 2 command line windows, run the queue in window 1, and then run the same queue in window 2. to confirm whether the same queue job will be repeated when the same queue is running simultaneously (when clustered). After a period of execution, window 1 has completed two queue jobs, and window 2 has completed three queue jobs. as shown in Figure 2
Figure 2
4. After executing a period of time, check the queue data in redis, cpa:queue:copy:asset.messages. The amount of data is 15. as shown in Figure 3
Figure 3
5. After another period of execution, run the same queue in window 3. Checking the queue status, it has been determined that the queue jobs are 10 waiting to be executed, 2 in progress, and 8 completions. as shown in Figure 4
Figure 4
6. Finally, check the running status in the 3 windows. Window 1 performs 8 queue jobs. Window 2 performs 12 queue jobs. Window 3 executes 0 queue jobs because the timeout failed when executing the first queue job.
window 1
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-08-31 17:37:31 [pid: 64508] - Worker is started
2021-08-31 17:37:32 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 17:42:35 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (303.486 s)
2021-08-31 17:42:36 [3] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 17:49:45 [3] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (429.834 s)
2021-08-31 17:49:46 [7] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 17:57:23 [7] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (457.296 s)
2021-08-31 17:57:24 [10] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:06:16 [10] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (531.560 s)
2021-08-31 18:06:16 [14] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:14:02 [14] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (465.649 s)
2021-08-31 18:14:02 [15] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:21:18 [15] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (436.172 s)
2021-08-31 18:21:19 [18] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:27:51 [18] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (392.171 s)
2021-08-31 18:27:52 [20] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Started
2021-08-31 18:27:54 [20] common\jobs\CopyAssetJob (attempt: 1, pid: 64508) - Done (2.273 s)
2021-08-31 18:27:54 [pid: 64508] - Worker is stopped (0:50:23)
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 20
window 2
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-08-31 17:37:37 [pid: 63068] - Worker is started
2021-08-31 17:37:37 [2] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:44:18 [2] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (400.251 s)
2021-08-31 17:44:18 [4] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:44:20 [4] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (1.969 s)
2021-08-31 17:44:21 [5] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:49:35 [5] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (314.755 s)
2021-08-31 17:49:36 [6] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:56:27 [6] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (411.015 s)
2021-08-31 17:56:27 [8] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 17:56:29 [8] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (1.854 s)
2021-08-31 17:56:30 [9] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:02:47 [9] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (377.192 s)
2021-08-31 18:02:48 [12] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:02:50 [12] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (2.375 s)
2021-08-31 18:02:51 [13] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:09:42 [13] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (411.228 s)
2021-08-31 18:09:42 [11] common\jobs\CopyAssetJob (attempt: 2, pid: 63068) - Started
2021-08-31 18:17:07 [11] common\jobs\CopyAssetJob (attempt: 2, pid: 63068) - Done (445.086 s)
2021-08-31 18:17:08 [16] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:17:11 [16] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (3.012 s)
2021-08-31 18:17:12 [17] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:22:13 [17] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (300.874 s)
2021-08-31 18:22:13 [19] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Started
2021-08-31 18:28:40 [19] common\jobs\CopyAssetJob (attempt: 1, pid: 63068) - Done (386.913 s)
2021-08-31 18:28:40 [pid: 63068] - Worker is stopped (0:51:03)
window 3
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-08-31 17:59:31 [pid: 65344] - Worker is started
2021-08-31 17:59:31 [11] common\jobs\CopyAssetJob (attempt: 1, pid: 65344) - Started
2021-08-31 18:09:32 [11] common\jobs\CopyAssetJob (attempt: 1, pid: 65344) - Error
> Symfony\Component\Process\Exception\ProcessTimedOutException: The process "C:\php-7.4.6\php.exe E:\wwwroot\channel-pub-api\yii "copy-asset-queue/exec" 11 600 1 65344 --verbose=1 --color=0" exceeded the timeout of 600 seconds.
2021-08-31 19:07:22 [pid: 65344] - Worker is stopped (1:07:51)
7. Check the operation results of 20 queue jobs. relative_path is the relative path of the downloaded file, and all the queue jobs are executed successfully. Because there are relative paths. But searching for video files that should be downloaded should be equal to 30 . But the actual number of video files in the download directory is 32 . The problem now is that it is not very clear how the extra 2 files are generated (one of the files should have existed long ago, and only one more file is added). as shown in Figure 5
Figure 5
8. It has been determined that the queue job in window 3 failed to time out, affecting 2 queue jobs. Although it failed, the file has been downloaded successfully. as shown in Figure 6
Figure 6
PS E:\wwwroot\channel-pub-api> ./yii source-callback-queue/info --color=0
Jobs
- waiting: 2
- delayed: 0
- reserved: 0
- done: 0
9. Why does the queue job in window 3 times out? The reason should be that the downlink bandwidth of the local environment is limited. The window 1 and the window 2 have already occupied a large proportion of bandwidth resources, resulting in no bandwidth resources available for window 3, so timeout. Repeat the above steps again. Decided to adjust the timeout from 600 seconds to 1800 seconds. All the queue jobs in the 3 windows are all executed successfully, of which the longest execution time is: 708.338 s. It is no wonder that the timeout fails before. Because there are relative paths. But searching for video files that should be downloaded should be equal to 30 . However, the actual number of video files in the download directory is 31 (it should be due to the internal cause of the program, which will be analyzed in the future). as shown in Figure 7
window 1
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-09-01 10:37:09 [pid: 73336] - Worker is started
2021-09-01 10:37:09 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 10:43:23 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (374.069 s)
2021-09-01 10:43:24 [6] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 10:52:59 [6] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (575.304 s)
2021-09-01 10:53:02 [10] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:03:11 [10] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (609.431 s)
2021-09-01 11:03:12 [14] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:13:11 [14] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (598.621 s)
2021-09-01 11:13:12 [16] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:13:14 [16] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (2.588 s)
2021-09-01 11:13:15 [17] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:21:02 [17] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (466.632 s)
2021-09-01 11:21:03 [20] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Started
2021-09-01 11:21:05 [20] common\jobs\CopyAssetJob (attempt: 1, pid: 73336) - Done (2.528 s)
2021-09-01 11:21:05 [pid: 73336] - Worker is stopped (0:43:56)
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 20
PS E:\wwwroot\channel-pub-api> ./yii source-callback-queue/info --color=0
Jobs
- waiting: 0
- delayed: 0
- reserved: 0
- done: 0
PS E:\wwwroot\channel-pub-api>
window 2
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-09-01 10:38:02 [pid: 67976] - Worker is started
2021-09-01 10:38:03 [3] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 10:49:51 [3] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (708.338 s)
2021-09-01 10:49:52 [8] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 10:49:55 [8] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (3.721 s)
2021-09-01 10:49:56 [9] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 10:57:06 [9] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (429.860 s)
2021-09-01 10:57:06 [11] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 11:08:16 [11] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (669.532 s)
2021-09-01 11:08:17 [2] common\jobs\CopyAssetJob (attempt: 2, pid: 67976) - Started
2021-09-01 11:18:33 [2] common\jobs\CopyAssetJob (attempt: 2, pid: 67976) - Done (616.247 s)
2021-09-01 11:18:35 [19] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Started
2021-09-01 11:25:39 [19] common\jobs\CopyAssetJob (attempt: 1, pid: 67976) - Done (424.108 s)
2021-09-01 11:25:40 [pid: 67976] - Worker is stopped (0:47:38)
window 3
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2021-09-01 10:38:29 [pid: 75656] - Worker is started
2021-09-01 10:38:30 [4] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 10:38:32 [4] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (2.338 s)
2021-09-01 10:38:33 [5] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 10:46:21 [5] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (468.368 s)
2021-09-01 10:46:22 [7] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 10:58:04 [7] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (702.171 s)
2021-09-01 10:58:05 [12] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 10:58:08 [12] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (2.769 s)
2021-09-01 10:58:09 [13] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 11:06:11 [13] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (482.736 s)
2021-09-01 11:06:12 [15] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 11:16:43 [15] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (631.170 s)
2021-09-01 11:16:44 [18] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Started
2021-09-01 11:24:50 [18] common\jobs\CopyAssetJob (attempt: 1, pid: 75656) - Done (485.917 s)
2021-09-01 11:24:55 [pid: 75656] - Worker is stopped (0:46:27)
10. It is concluded that the Yii2 queue extension supports parallel operation, even if it is run at the same time in the cluster, there is no problem.
