The implementation of Yii 2 based on HTTP client extension, download the file and save as a specific file name (download the remote file to the server), the implementation of memory usage drops from 400 MB to 7 MB
1. Now it has been implemented: Copy the resource file of the source to the resource directory published by the channel, and return the relative path (sync). The code is as follows
/**
* 复制来源的资源文件至渠道发布的资源目录，返回相对路径(同步)
* @param string $source 来源
* 格式如下：spider
* @param array $assets 来源的资源文件的绝对URL
* 格式如下：
* [
* [
* 'type' => 'image',
* 'absolute_url' => 'http://localhost/spider/storage/spider/images/1.png',
* ],
* [
* 'type' => 'video',
* 'absolute_url' => 'http://127.0.0.1/channel-pub-api/storage/spider/videos/7月份北上广深等十大城市租金环比上涨 看东方 20180820 高清_高清.mp4',
* ],
* ]
*
* @return array $channelPubApiAssetAbsolutePaths 渠道发布的资源文件的相对路径
* 格式如下：
* [
* [
* 'type' => 'image',
* 'relative_path' => '/2018/09/20/1537439889.2333.1441541478.png',
* ],
* [
* 'type' => 'video',
* 'relative_path' => '/2018/09/20/1537439889.2403.62871268.mp4',
* ],
* ]
*
* @throws UnprocessableEntityHttpException 如果来源的资源文件不是以来源的 HOME URL + BASE URL 开头，将抛出 422 HTTP 异常
* @throws NotFoundHttpException 如果来源的资源文件不存在，将抛出 404 HTTP 异常
* @throws ServerErrorHttpException 如果创建目录失败，将抛出 500 HTTP 异常
* @throws ServerErrorHttpException 如果复制来源的资源文件至渠道发布的资源目录失败，将抛出 500 HTTP 异常
* @throws \yii\base\Exception if the directory could not be created (i.e. php error due to parallel changes)
*/
public static function copyAssetsSync($source, $assets)
{
// file_put_contents('/mcloud/www/channel-pub-api/console/runtime/copy-assets-sync-source-' . $source . '-' . time() . '.txt', $source);
// file_put_contents('C:/phpStudy/PHPTutorial/WWW/channel-pub-api/douyin/runtime/copy-assets-sync-assets-' . $source . '-' . time() . '.txt', print_r($assets, true));
// 不是以来源的 HOME URL + BASE URL 开头的来源的资源文件
$notAbsoluteUrlStartKeys = [];
// 不存在的来源的资源文件
$notExistsKeys = [];
// 渠道发布的资源文件的相对路径
$channelPubApiAssetAbsolutePaths = [];
foreach ($assets as $key => $asset) {
if ($index = stripos($asset['absolute_url'], '?')) {
$absoluteUrl = substr($asset['absolute_url'], 0, $index);
$asset['absolute_url'] = $absoluteUrl;
$assets[$key]['absolute_url'] = $absoluteUrl;
}
// 检查来源的资源文件的绝对URL是否以来源的 HOME URL + BASE URL 开头
if (!StringHelper::startsWith($asset['absolute_url'], Yii::$app->params['source']['asset'][$asset['type']]['hostInfo'] . Yii::$app->params['source']['asset'][$asset['type']]['baseUrl'])) {
$notAbsoluteUrlStartKeys[] = $asset['absolute_url'];
} else {
// 获取来源的资源文件的绝对路径
$sourceAssetAbsolutePath = Yii::$app->params['source']['asset'][$asset['type']]['basePath'] . str_replace(Yii::$app->params['source']['asset'][$asset['type']]['hostInfo'] . Yii::$app->params['source']['asset'][$asset['type']]['baseUrl'], '', $asset['absolute_url']);
// 检查来源的资源文件是否存在
if (!file_exists($sourceAssetAbsolutePath)) {
$notExistsKeys[] = $sourceAssetAbsolutePath;
}
}
}
if (!empty($notAbsoluteUrlStartKeys)) {
$notAbsoluteUrlStartKeys = implode(",", $notAbsoluteUrlStartKeys);
throw new UnprocessableEntityHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202003'), ['not_absolute_url_start_keys' => $notAbsoluteUrlStartKeys])), 202003);
}
if (!empty($notExistsKeys)) {
$notExistsKeys = implode(",", $notExistsKeys);
throw new NotFoundHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202004'), ['not_exists_keys' => $notExistsKeys])), 202004);
}
foreach ($assets as $key => $asset) {
// 获取来源的资源文件的绝对路径
$sourceAssetAbsolutePath = Yii::$app->params['source']['asset'][$asset['type']]['basePath'] . str_replace(Yii::$app->params['source']['asset'][$asset['type']]['hostInfo'] . Yii::$app->params['source']['asset'][$asset['type']]['baseUrl'], '', $asset['absolute_url']);
// 获取来源的资源文件的路径信息
$pathInfo = pathinfo($sourceAssetAbsolutePath);
// 渠道发布的资源文件的相对路径
$directory = date('Y/m/d');
$channelPubApiAssetRelativePath = '/' . $directory . '/' . microtime(true) . '.' . mt_rand() . '.' . $pathInfo['extension'];
// 渠道发布的资源文件的绝对路径
$channelPubApiAssetAbsolutePath = Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . $channelPubApiAssetRelativePath;
// 创建目录
if (!FileHelper::createDirectory(Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . '/' . $directory)) {
throw new ServerErrorHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202005'), ['directory' => Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . '/' . $directory])), 202005);
}
// 复制来源的资源文件至渠道发布的资源目录
if (!copy($sourceAssetAbsolutePath, $channelPubApiAssetAbsolutePath)) {
throw new ServerErrorHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202006'), ['source_asset_absolute_path' => $sourceAssetAbsolutePath])), 202006);
}
$channelPubApiAssetAbsolutePaths[$key] = [
'type' => $asset['type'],
'relative_path' => $channelPubApiAssetRelativePath,
];
}
return $channelPubApiAssetAbsolutePaths;
}
2. Now it needs to be adjusted and can be realized again: download the file and save it as a specific file name, rather than copy it.
3. Verify the implementation of the first step first. Call this method, print the response result, the code is as follows
$copyAssetsResult = AssetService::copyAssetsSync(
'spider',
[
[
'type' => 'image',
'absolute_url' => 'http://127.0.0.1/channel-pub-api/storage/spider/images/1.png',
],
[
'type' => 'video',
'absolute_url' => 'http://127.0.0.1/channel-pub-api/storage/spider/videos/02018684a82381d9c59bb085e18e1a5d.mp4',
],
]
);
print_r($copyAssetsResult);
exit;
4. Print the response result, as shown in Figure 1
Figure 1
Array
(
[0] => Array
(
[type] => image
[relative_path] => /2020/11/24/1606187993.2607.476899553.png
)
[1] => Array
(
[type] => video
[relative_path] => /2020/11/24/1606187993.2711.210785146.mp4
)
)
5. View the copied files in the resource directory published by the channel. Its absolute paths are: E:\wwwroot\channel-pub-api\storage\channel -PUB-API\images\2020\11\24\1606187993.2607.476899553 .png, e:\wwwroot\channel-pub-api\storage\channel-pub- API\Videos\2020\11\24\1606187993.2711.210785146.mp4. as shown in Figure 2
Figure 2
6. Implement the HTTP model, /common/logics/http/asset_api/download.php
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2020/11/24
* Time: 15:47
*/
namespace common\logics\http\asset_api;
use Yii;
use yii\base\InvalidConfigException;
use yii\httpclient\Client;
use yii\httpclient\Exception;
use yii\web\ServerErrorHttpException;
/**
* 资源接口的下载
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class Download extends Model
{
/**
* HTTP请求，基于来源的资源文件的绝对URL下载文件
*
* @param string $url 来源的资源文件的绝对URL
*
* @return string
* 格式如下：
*
* @throws ServerErrorHttpException 如果响应状态码不等于20x
* @throws InvalidConfigException
* @throws Exception
*/
public function download($url)
{
$client = new Client([
'transport' => 'yii\httpclient\CurlTransport' // 只有 cURL 支持此选项
]);
$response = $client->createRequest()
->setMethod('GET')
->setUrl($url)
->send();
// 检查响应状态码是否等于20x
if ($response->isOk) {
return $response->content;
} else {
throw new ServerErrorHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202218'), ['status_code' => $response->getStatusCode()])), 202218);
}
}
}
7. New implementation: After downloading the resource file of the source, write it to the resource directory published by the channel, and return the relative path (sync). The code is as follows
/* HTTP请求，基于来源的资源文件的绝对URL下载文件 */
$httpAssetApiDownload = new HttpAssetApiDownload();
foreach ($assets as $key => $asset) {
$downloadAsset = $httpAssetApiDownload->download($asset['absolute_url']);
// 渠道发布的资源文件的相对路径
$directory = date('Y/m/d');
$channelPubApiAssetRelativePath = '/' . $directory . '/' . microtime(true) . '.' . mt_rand() . '.' . $asset['extension'];
// 渠道发布的资源文件的绝对路径
$channelPubApiAssetAbsolutePath = Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . $channelPubApiAssetRelativePath;
// 创建目录
if (!FileHelper::createDirectory(Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . '/' . $directory)) {
throw new ServerErrorHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202005'), ['directory' => Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . '/' . $directory])), 202005);
}
// 下载来源的资源文件后写入至渠道发布的资源目录
if (!file_put_contents($channelPubApiAssetAbsolutePath, $downloadAsset, LOCK_EX)) {
throw new ServerErrorHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202220'), ['source_asset_absolute_url' => $asset['source_asset_absolute_url']])), 202220);
}
$channelPubApiAssetAbsolutePaths[$key] = [
'type' => $asset['type'],
'relative_path' => $channelPubApiAssetRelativePath,
];
}
8. Run the code, the size of the video file is: 396 MB, error report: allow memory size of 134217728 bytes exhausted (tried to allocate 62918656 bytes). as shown in Figure 3
Figure 3
{
"name": "PHP Fatal Error",
"message": "Allowed memory size of 134217728 bytes exhausted (tried to allocate 62918656 bytes)",
"code": 1,
"type": "yii\\base\\ErrorException",
"file": "E:\\wwwroot\\channel-pub-api\\vendor\\yiisoft\\yii2-httpclient\\src\\CurlTransport.php",
"line": 40,
"stack-trace": [
"#0 [internal function]: yii\\base\\ErrorHandler->handleFatalError()",
"#1 {main}"
]
}
9. Edit the php.ini file and modify memory_limit = 128m to memory_limit = 1024m. The error is still reported when it is set to 512M. The operation was successful. Check the log, occupying memory: 406 MB. as shown in Figure 4
Figure 4
10. View the copied files in the resource directory published by the channel. Its absolute paths are: E:\wwwroot\channel-pub-api\storage\channel- Pub-API\images\2020\11\25\1606291709.4136.2030184970 .png, e:\wwwroot\channel-pub-api\storage\channel-pub- API\Videos\2020\11\25\1606291735.722.2031039128.mp4. in line with expectations. as shown in Figure 5
Figure 5
11. The problem with this scheme is that the memory usage changes with the size of the downloaded file. May fail because the size of the downloaded file exceeds the memory_limit. Decided to find ways to optimize and improve.
12. Reconstruct the HTTP model, /common/logics/http/asset_api/download.php. How to use: setOutputFile(). Use with yii\HttpClient\CurlTransport to set the file to transfer. as shown in Figure 6
Figure 6
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2020/11/24
* Time: 15:47
*/
namespace common\logics\http\asset_api;
use Yii;
use yii\base\InvalidConfigException;
use yii\httpclient\Client;
use yii\httpclient\Exception;
use yii\web\ServerErrorHttpException;
/**
* 资源接口的下载
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class Download extends Model
{
/**
* HTTP请求，基于来源的资源文件的绝对URL下载文件
*
* @param string $url 来源的资源文件的绝对URL
* @param string $absolutePath 资源文件的绝对路径(下载至的目标路径)
*
* @return bool
* 格式如下：true
*
* @throws ServerErrorHttpException 如果响应状态码不等于20x
* @throws InvalidConfigException
* @throws Exception
*/
public function download($url, $absolutePath)
{
// 打开即将下载的本地文件，在该文件上打开一个流
$handle = fopen($absolutePath, 'w');
if (!$handle) {
throw new ServerErrorHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202220'), ['absolute_path' => $absolutePath])), 202220);
}
$client = new Client([
'transport' => 'yii\httpclient\CurlTransport' // 只有 cURL 支持此选项
]);
$response = $client->createRequest()
->setMethod('GET')
->setUrl($url)
->setOutputFile($handle)
->send();
// 关闭一个已打开的文件指针
fclose($handle);
// 检查响应状态码是否等于20x
if ($response->isOk) {
return $response->content;
} else {
throw new ServerErrorHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202218'), ['status_code' => $response->getStatusCode()])), 202218);
}
}
}
13. Refactoring a new implementation: The resource file of the download source is synchronously written to the resource directory published by the channel, and the relative path (sync) is returned. The code is as follows
/* HTTP请求，基于来源的资源文件的绝对URL下载文件 */
$httpAssetApiDownload = new HttpAssetApiDownload();
foreach ($assets as $key => $asset) {
// 渠道发布的资源文件的相对路径
$directory = date('Y/m/d');
$channelPubApiAssetRelativePath = '/' . $directory . '/' . microtime(true) . '.' . mt_rand() . '.' . $asset['extension'];
// 渠道发布的资源文件的绝对路径
$channelPubApiAssetAbsolutePath = Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . $channelPubApiAssetRelativePath;
// 创建目录
if (!FileHelper::createDirectory(Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . '/' . $directory)) {
throw new ServerErrorHttpException(Yii::t('error', Yii::t('error', Yii::t('error', '202005'), ['directory' => Yii::$app->params['channelPubApi']['asset'][$asset['type']]['basePath'] . '/' . $directory])), 202005);
}
// HTTP请求，基于来源的资源文件的绝对URL下载文件
$httpAssetApiDownload->download($asset['absolute_url'], $channelPubApiAssetAbsolutePath);
$channelPubApiAssetAbsolutePaths[$key] = [
'type' => $asset['type'],
'relative_path' => $channelPubApiAssetRelativePath,
];
}
14. The operation is successful. Check the logs, occupying memory: 6.377 MB. The size of the downloaded file is no longer limited by the memory_limit. as shown in Figure 7
Figure 7
15. The reason why curl is used is that the performance of curl is better than that of file_get_contents and fopen. But it is certain that the memory usage is better than file_get_contents. Untested, this conclusion is described in the official PHP documentation. Link:https://www.php.net/manual/zh/ref.curl.php. as shown in Figure 8
8
