The job may have previously timed out.
1. Job runs an error in Laravel 9: Has Been Attempted too many times or run too long. The job may have previously previously timed out.. as shown in Figure 1
Figure 1
2. The total number of records found in the table is 8761297. Based on the chunk method, the entire table is retrieved in units of 10 records at a time. The time of each time is printed in the switch of the closure function. The final result is as follows
[2024-04-09 09:43:57] local.INFO: exportExcelStart {"datetime":"2024-04-09 09:43:57"}
[2024-04-09 09:43:58] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:43:58"}
[2024-04-09 09:44:00] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:00"}
[2024-04-09 09:44:01] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:01"}
[2024-04-09 09:44:03] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:03"}
[2024-04-09 09:44:05] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:05"}
[2024-04-09 09:44:07] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:07"}
[2024-04-09 09:44:09] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:09"}
[2024-04-09 09:44:11] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:11"}
[2024-04-09 09:44:13] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:13"}
[2024-04-09 09:44:14] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:14"}
[2024-04-09 09:44:16] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:16"}
[2024-04-09 09:44:18] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:18"}
[2024-04-09 09:44:20] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:20"}
[2024-04-09 09:44:22] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:22"}
[2024-04-09 09:44:23] local.INFO: exportExcelChunk {"datetime":"2024-04-09 09:44:23"}
3. In the end, it is concluded that the average time required for every 10 records is 1.8 seconds. Decided to adjust 10 to 1000. The time of each time is printed in the switch of the closure function. The final result is as follows. For every 100 records, the average time required to consume is 4.8 seconds. Then the processing time equivalent to every 10 records is reduced to 0.48 seconds.
[2024-04-09 09:53:48] local.INFO: exportExcelStart {"datetime":"2024-04-09 09:53:48"}
[2024-04-09 09:53:53] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:53:53"}
[2024-04-09 09:53:58] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:53:58"}
[2024-04-09 09:54:02] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:02"}
[2024-04-09 09:54:07] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:07"}
[2024-04-09 09:54:11] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:11"}
[2024-04-09 09:54:15] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:15"}
[2024-04-09 09:54:20] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:20"}
[2024-04-09 09:54:24] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:24"}
[2024-04-09 09:54:28] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:28"}
[2024-04-09 09:54:32] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:32"}
[2024-04-09 09:54:37] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:37"}
[2024-04-09 09:54:42] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:42"}
[2024-04-09 09:54:48] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:48"}
[2024-04-09 09:54:57] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:54:57"}
[2024-04-09 09:55:04] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:55:04"}
[2024-04-09 09:55:09] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:55:09"}
[2024-04-09 09:55:14] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:55:14"}
[2024-04-09 09:55:18] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:55:18"}
[2024-04-09 09:55:22] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:55:22"}
[2024-04-09 09:55:27] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:55:27"}
[2024-04-09 09:55:33] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 09:55:33"}
4. Try 10,000 records again, and the average time required for every 10,000 records is 41 seconds. Then the processing time equivalent to every 10 records is reduced to 0.41 seconds. Then in theory, the value of 1000 is appropriate. The memory usage of one processing at one time is less, and the overall time-consuming, compared with 10000, it has not been much longer.
[2024-04-09 10:07:17] local.INFO: exportExcelStart {"datetime":"2024-04-09 10:07:17"}
[2024-04-09 10:08:04] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 10:08:04"}
[2024-04-09 10:09:00] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 10:09:00"}
[2024-04-09 10:09:55] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 10:09:55"}
[2024-04-09 10:10:49] local.INFO: exportExcelChunk1000 {"datetime":"2024-04-09 10:10:49"}
5. In the performance environment, the processing time of the 1000th record is 0.6 seconds, then the processing time of every 10 records is reduced to 0.06 seconds. The final decision limits the maximum number of exported lines to 100000. The code is implemented as follows
$count = 0;
$orderShippingLogQueryBuilder->chunk(1000, function ($orderShippingLogs) use ($params, $excel, $columns, &$count) {
// 限制最大导出行数为 100000
if ($count >= 100000) {
return false;
}
Log::info(
'exportExcelChunk1000',
[
date('Y-m-d H:i:s', time())
]
);
$data = [];
$count = $count + 1000;
});
6. Check Laravel Telescope, the local environment is set to 3000, and it is confirmed that 3 query sql is executed, which is in line with expectations. as shown in Figure 2
Figure 2
7. In the end, in the performance environment, export a 100000-line Excel, with a total execution time: 107 seconds. in line with expectations.
exportExcelStart ["2024-04-10 03:56:32"]
exportExcelChunk1000 ["2024-04-10 03:56:32"]
exportExcelChunk1000 ["2024-04-10 03:56:33"]
exportExcelChunk1000 ["2024-04-10 03:56:33"]
exportExcelChunk1000 ["2024-04-10 03:56:34"]
exportExcelChunk1000 ["2024-04-10 03:56:35"]
exportExcelChunk1000 ["2024-04-10 03:56:36"]
exportExcelChunk1000 ["2024-04-10 03:58:13"]
exportExcelEnd ["2024-04-10 03:58:19"]
