The implementation process of video file sharding upload based on Penguin, including queue, file slice, while loop, etc.
1. The design of the database structure, a resource table, a video file upload table of a Penguin number, and a transaction table of a penguin. The structure is as follows:
28、asset：资源 Asset
id 主键
channel_id 渠道ID
channel_code 渠道代码，qq：企鹅号；wx：微信公众帐号
channel_type_id 渠道的类型ID
channel_type_code 渠道的类型代码，qq_cw：企鹅号的内容网站应用；qq_tp：企鹅号的第三方服务平台应用；wx：微信公众帐号应用
source 来源，xContent：内容库；vms：视频管理系统；cms：内容管理系统；spider：自媒体
type 资源文件的类型，image：图片；video：视频
absolute_url 来源的资源文件的绝对URL
relative_path 渠道发布的资源文件的相对路径
size 文件大小，单位(字节)
task_id 任务ID
channel_article_id 渠道的文章ID
status 状态，0：禁用；1：启用
is_deleted 是否被删除，0：否；1：是
created_at 创建时间
updated_at 更新时间
deleted_at 删除时间
29、qq_video_multipart_upload：企鹅号的视频文件分片上传 QqCwVideoMultipartUpload
id 主键
asset_id 资源ID
qq_app_task_id 企鹅号的应用的任务ID
qq_app_id 企鹅号的应用ID
qq_app_type 企鹅号的应用类型，cw：内容网站应用；tp：第三方服务平台应用
size 视频文件大小，单位(字节)
md5 视频文件MD5值
sha 视频文件SHA-1值
transaction_id 上传的唯一事务ID
mediatrunk 视频 mediatrunk 文件
start_offset 分片的起始位置(从0开始计数)
end_offset 分片的结束位置
vid 视频文件唯一标示ID
status 状态，0：禁用；1：待上传；2：上传中；3：上传中(已失败)；4：已上传
is_deleted 是否被删除，0：否；1：是
created_at 创建时间
updated_at 更新时间
deleted_at 删除时间
30、qq_transaction：企鹅号的事务 QqTransaction
id 主键
group_id 租户ID
qq_app_task_id 企鹅号的应用的任务ID
qq_app_id 企鹅号的应用ID
qq_app_type 企鹅号的应用类型，cw：内容网站应用；tp：第三方服务平台应用
qq_article_id 企鹅号的文章ID
qq_video_multipart_upload_id 企鹅号的应用的视频文件分片上传ID
transaction_id 事务ID
type 类型，1：文章；2：视频
transaction_ctime 事务创建时间
ext_err 扩展的错误
transaction_err_msg 事务的错误信息
article_abstract 文章摘要
article_type 文章类型，取值：普通文章，图文文章，视频文章，直播文章，RTMP直播文章
article_type_code 文章类型代码，normal：文章；multivideos：视频；images：组图；live：直播
article_url 文章快报链接
article_imgurl 文章封面图
article_title 文章标题
article_pub_flag 文章发布状态，取值：未发布，发布成功，审核中
article_pub_time 文章发布时间
article_video_title 视频文章标题
article_video_desc 视频文章描述
article_video_type 视频文章类型，视频
article_video_vid 视频文章的视频唯一ID
task_id 任务ID
status 状态，0：禁用；1：成功；2：失败；3：处理中
is_deleted 是否被删除，0：否；1：是
created_at 创建时间
updated_at 更新时间
deleted_at 删除时间
2. The upload of resources is implemented based on the queue, so the resource data will be stored in the resource table first, and then the queue of upload resources will be executed, and then the job of uploading resources will be performed.
3. Execute an interface request, at this time, the resource table already has the corresponding data of the resource, and enter the queue of the copy resource
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0
Jobs
- waiting: 1
- delayed: 0
- reserved: 0
- done: 0
4. Execute the task command in the replication resource queue, which will copy the corresponding resources and store the relative path of the resources into the resource table
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0
2018-11-15 10:05:56 [pid: 23112] - Worker is started
2018-11-15 10:05:57 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 23112) - Started
2018-11-15 10:06:08 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 23112) - Done (11.394 s)
2018-11-15 10:06:08 [pid: 23112] - Worker is stopped (0:00:12)
5. After the task in the resource queue is successfully executed, it will enter the queue of upload resources
PS E:\wwwroot\channel-pub-api> ./yii upload-asset-queue/info --color=0
Jobs
- waiting: 1
- delayed: 0
- reserved: 0
- done: 0
6. The code for uploading resources is divided into 2 parts, one is a file slicing, and the video resource file needs to be sliced into a small file of size 100M, \channel-pub-api\common\services\AssetService.php
/**
* 文件切片
* @param string $fileAbsolutePath 需要切片的文件的绝对路径
* 格式如下：E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件.mp4
*
* @param int $size 104857600，单位为字节
*
* 生成文件列表如下：
* E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件_0.mp4
* E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件_1.mp4
*/
public static function cut($fileAbsolutePath, $size)
{
// 获取需要切片的文件的路径信息
$pathInfo = pathinfo($fileAbsolutePath);
$i = 0;
$handle = fopen($fileAbsolutePath, "rb");
while (!feof($handle)) {
$cutHandle = fopen($pathInfo['dirname'] . '/' . $pathInfo['filename'] . '_' . $i . '.' . $pathInfo['extension'], "wb");
fwrite($cutHandle, fread($handle, $size));
fclose($cutHandle);
unset($cutHandle);
$i++;
}
fclose($handle);
}
7. HTTP requests, the video files of the content website application of the Penguin account are uploaded shardedly, \channel-pub-api\common\logics\http\qq_api\video.php
* @since 1.0
*/
class Video extends Model
{
const CUT_SIZE = 104857600; //视频分片上传的切片大小：100M
/**
* HTTP请求，申请企鹅号的内容网站应用的视频文件分片上传的唯一事务ID
*
* @param array $data 数据
* 格式如下：
* [
* 'accessToken' => 'KCK0TIV8NDY44ONAEFI2QW', // 企鹅平台企鹅号应用授权调用凭据
* 'size' => 9135849, // 视频文件大小，单位(字节)
* 'md5' => 'd081c619ef7dc62eb2aa29c7c83c6f26', // 视频文件MD5值
* 'sha' => '28a1076b867bed8202df5b3f656b798148e58ced', // 视频文件SHA-1值
* ]
*
* @return array|false
* 格式如下：
* 企鹅号的内容网站应用的视频文件分片上传的唯一事务ID
* [
* 'message' => '', // 说明
* 'data' => [ // 数据
* 'transaction_id' => '780930255958621794', // 上传的唯一事务ID
* ],
* ]
*
* 失败(将错误保存在 [[yii\base\Model::errors]] 属性中)
* false
*
* @throws ServerErrorHttpException 如果响应状态码不等于20x
*/
public function clientUploadReady($data)
{
$response = Yii::$app->qqApiHttps->createRequest()
->setMethod('post')
->setUrl('video/clientuploadready')
->setData([
'access_token' => $data['accessToken'],
'size' => $data['size'],
'md5' => $data['md5'],
'sha' => $data['sha'],
])
->send();
// file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/client-upload-ready_' . $data['size'] . '_' . time() . '.txt', $response->data['data']['transaction_id']);
// 检查响应状态码是否等于20x
if ($response->isOk) {
// 检查业务逻辑是否成功
if ($response->data['code'] === 0) {
$responseData = ['message' => '', 'data' => $response->data['data']];
return $responseData;
} else {
$this->addError('id', $response->data['msg']);
return false;
}
} else {
throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35041'), ['statusCode' => $response->getStatusCode()])), 35041);
}
}
/**
* HTTP请求，企鹅号的内容网站应用的视频文件分片上传
*
* @param array $data 数据
* 格式如下：
* [
* 'accessToken' => 'LQVHLUQDNEKIDBFARJ1OOA', // 企鹅平台企鹅号应用授权调用凭据
* 'transactionId' => '780930287703152921', // 上传的唯一事务ID
* 'mediatrunk' => 'E:/wwwroot/channel-pub-api/storage/channel-pub-api/videos/2018/10/27/1540632234.9842.66697370.mp4', // 视频 mediatrunk 文件
* 'startOffset' => 0, // 分片的起始位置(从0开始计数)
* ]
*
* @return array|false
* 格式如下：
* 企鹅号的内容网站应用的视频文件分片上传
* [
* 'message' => '', // 说明
* 'data' => [ // 数据
* 'end_offset' => 2198151, // 分片的结束位置
* 'start_offset' => 2198151, // 分片的起始位置
* 'transaction_id' => 780930255958621794, // 上传的唯一事务ID
* ],
* ]
*
* 失败(将错误保存在 [[yii\base\Model::errors]] 属性中)
* false
*
* @throws ServerErrorHttpException 如果响应状态码不等于20x
*/
public function clientUploadTrunk($data)
{
$response = Yii::$app->qqApiHttp->createRequest()
->setMethod('post')
->setUrl('video/clientuploadtrunk?access_token=' . $data['accessToken'] . '&transaction_id=' . $data['transactionId'] . '&start_offset=' . $data['startOffset'])
->addFile('mediatrunk', $data['mediatrunk'])
->send();
// file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/client-upload-trunk_' . $data['transactionId'] . '_' . $data['startOffset'] . '_' . time() . '.txt', $response->data['code']);
// 检查响应状态码是否等于20x
if ($response->isOk) {
// 检查业务逻辑是否成功
if ($response->data['code'] === 0) {
$responseData = ['message' => '', 'data' => $response->data['data']];
return $responseData;
} elseif ($response->data['code'] === 40027) { // 无效的事务ID
$this->addError('id', $response->data['code']);
return false;
} else {
$this->addError('id', $response->data['msg']);
return false;
}
} else {
throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35041'), ['statusCode' => $response->getStatusCode()])), 35041);
}
}
}
8. Upload the code of the video file, \channel-pub-api\common\services\qqcwVideoMultiPartUploadService.php
/**
* HTTP请求，企鹅号的内容网站应用的视频文件分片上传
* @param array $data 数据
* 格式如下：
*
* [
* 'accessToken' => 'LQVHLUQDNEKIDBFARJ1OOA', // 企鹅平台企鹅号应用授权调用凭据
* 'transactionId' => '780930287703152921', // 上传的唯一事务ID
* 'mediatrunk' => 'E:/wwwroot/channel-pub-api/storage/channel-pub-api/videos/2018/10/27/1540632234.9842.66697370.mp4', // 视频 mediatrunk 文件
* 'startOffset' => 0, // 分片的起始位置(从0开始计数)
* ]
*
* @return array
* 格式如下：
*
* [
* 'end_offset' => 2198151, // 分片的结束位置
* 'start_offset' => 2198151, // 分片的起始位置
* 'transaction_id' => 780930255958621794, // 上传的唯一事务ID
* ]
*
* @throws ServerErrorHttpException
*/
public function httpUploadTrunk($data)
{
/* HTTP请求，企鹅号的内容网站应用的视频文件分片上传 */
$httpQqApiVideo = new HttpQqApiVideo();
$uploadTrunk = $httpQqApiVideo->clientUploadTrunk($data);
if ($uploadTrunk === false) {
if ($httpQqApiVideo->hasErrors()) {
foreach ($httpQqApiVideo->getFirstErrors() as $message) {
$firstErrors = $message;
break;
}
throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35042'), ['firstErrors' => $firstErrors])), 35042);
} elseif (!$httpQqApiVideo->hasErrors()) {
throw new ServerErrorHttpException('Penguin\'s content website application api HTTP requests fail for unknown reasons.');
}
}
return $uploadTrunk['data'];
}
/**
* 企鹅号的内容网站应用的视频文件分片上传
*
* @param int $assetId 资源ID
* 格式如下：1
*
* @param int $qqCwAppTaskId 企鹅号的内容网站应用的任务ID
* 格式如下：6
*
* @throws ServerErrorHttpException
* @throws \Throwable
*/
public function upload($assetId, $qqCwAppTaskId)
{
// 基于ID查找状态为启用的单个数据模型(资源)
$assetEnabledItem = AssetService::findModelEnabledById($assetId);
// 基于ID查找状态为启用的单个数据模型(任务)
$taskEnabledItem = TaskService::findModelEnabledById($assetEnabledItem->task_id);
// 基于ID查找状态为发布中的单个数据模型(企鹅号的内容网站应用的任务)
$qqCwAppTaskPublishItem = QqCwAppTaskService::findModelPublishById($qqCwAppTaskId);
// 基于ID查找状态为启用的单个数据模型(企鹅号的内容网站应用)
$qqCwAppEnabledItem = QqCwAppService::findModelEnabledById($qqCwAppTaskPublishItem->qq_cw_app_id);
// 基于企鹅号的内容网站应用ID获取有效的 Access Token
$qqCwAccessTokenService = new QqCwAccessTokenService();
$accessTokenValidity = $qqCwAccessTokenService->getAccessTokenValidityByQqCwAppId($qqCwAppEnabledItem->id);
// 基于资源ID、企鹅号的应用的任务ID获取企鹅号的视频文件分片上传的单个数据模型
$qqVideoMultipartUploadItem = $this->getModelByAssetIdAndQqAppTaskId($assetId, $qqCwAppTaskPublishItem->id);
$data = [
'assetId' => $qqVideoMultipartUploadItem->asset_id,
'qqAppTaskId' => $qqVideoMultipartUploadItem->qq_app_task_id,
'qqAppId' => $qqVideoMultipartUploadItem->qq_app_id,
'qqAppType' => $qqVideoMultipartUploadItem->qq_app_type,
'size' => $qqVideoMultipartUploadItem->size,
'md5' => $qqVideoMultipartUploadItem->md5,
'sha' => $qqVideoMultipartUploadItem->sha,
'transactionId' => (string) $qqVideoMultipartUploadItem->transaction_id,
'mediatrunk' => $qqVideoMultipartUploadItem->mediatrunk,
'startOffset' => $qqVideoMultipartUploadItem->start_offset,
'endOffset' => $qqVideoMultipartUploadItem->end_offset,
'vid' => $qqVideoMultipartUploadItem->vid,
'status' => $qqVideoMultipartUploadItem->status,
];
// 文件切片
AssetService::cut(Yii::$app->params['channelPubApi']['asset'][$assetEnabledItem->type]['basePath'] . $qqVideoMultipartUploadItem->mediatrunk, HttpQqApiVideo::CUT_SIZE);
// 获取需要切片的文件的路径信息
$pathInfo = pathinfo(Yii::$app->params['channelPubApi']['asset'][$assetEnabledItem->type]['basePath'] . $qqVideoMultipartUploadItem->mediatrunk);
// 判断分片的起始位置与分片的结束位置是否等于视频文件大小，如果相等则中断分片上传，否则继续执行分片上传
$i = 0;
while ($qqVideoMultipartUploadItem->start_offset != $qqVideoMultipartUploadItem->size && $qqVideoMultipartUploadItem->end_offset != $qqVideoMultipartUploadItem->size) {
// file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/while_' . $assetId . '_' . $qqCwAppId . '_' . $qqVideoMultipartUploadItem->start_offset . '_' . $qqVideoMultipartUploadItem->end_offset . '_' . time() . '.txt', $assetId);
// HTTP请求，企鹅号的内容网站应用的视频文件分片上传
$httpUploadTrunkData = [
'accessToken' => $accessTokenValidity->access_token,
'transactionId' => $qqVideoMultipartUploadItem->transaction_id,
'mediatrunk' => $pathInfo['dirname'] . '/' . $pathInfo['filename'] . '_' . $i . '.' . $pathInfo['extension'],
'startOffset' => $qqVideoMultipartUploadItem->start_offset,
];
$uploadTrunkData = $this->httpUploadTrunk($httpUploadTrunkData);
// 基于资源ID、企鹅号的内容网站应用ID插入/更新数据至企鹅号的内容网站应用的视频文件分片上传
$data['startOffset'] = $uploadTrunkData['start_offset'];
$data['endOffset'] = $uploadTrunkData['end_offset'];
if ($uploadTrunkData['start_offset'] != $qqVideoMultipartUploadItem->size && $uploadTrunkData['end_offset'] != $qqVideoMultipartUploadItem->size) {
$data['status'] = QqVideoMultipartUpload::STATUS_UPLOADING;
} else {
// HTTP请求，基于上传的唯一事务ID获取事务信息
$qqTransactionService = new QqTransactionService();
$qqTransactionServiceHttpTransactionInfoData = [
'accessToken' => $accessTokenValidity->access_token,
'transactionId' => $qqVideoMultipartUploadItem->transaction_id,
];
$qqTransactionServiceHttpTransactionInfoResult = $qqTransactionService->httpTransactionInfo($qqTransactionServiceHttpTransactionInfoData);
// 创建企鹅号的事务
$qqTransactionServiceCreateData = [
'groupId' => $taskEnabledItem->group_id,
'qqAppTaskId' => $qqVideoMultipartUploadItem->qq_app_task_id,
'qqAppId' => $qqVideoMultipartUploadItem->qq_app_id,
'qqAppType' => $qqVideoMultipartUploadItem->qq_app_type,
'qqArticleId' => 0,
'qqVideoMultipartUploadId' => $qqVideoMultipartUploadItem->id,
'transactionId' => $qqVideoMultipartUploadItem->transaction_id,
'type' => $qqTransactionService::getTypeByHttpQqApiTransactionType($qqTransactionServiceHttpTransactionInfoResult['transaction_type']),
'transactionCtime' => $qqTransactionServiceHttpTransactionInfoResult['transaction_ctime'],
'extErr' => $qqTransactionServiceHttpTransactionInfoResult['ext_err'],
'transactionErrMsg' => $qqTransactionServiceHttpTransactionInfoResult['transaction_err_msg'],
'articleAbstract' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_abstract'],
'articleType' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_type'],
'articleTypeCode' => '',
'articleUrl' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_url'],
'articleImgurl' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_imgurl'],
'articleTitle' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_title'],
'articlePubFlag' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_pub_flag'],
'articlePubTime' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_pub_time'],
'articleVideoTitle' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['title'],
'articleVideoDesc' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['desc'],
'articleVideoType' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['type'],
'articleVideoVid' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['vid'],
'taskId' => $assetEnabledItem->task_id,
'status' => QqTransaction::STATUS_PROCESSING,
];
$qqTransactionServiceCreateResult = $qqTransactionService->create($qqTransactionServiceCreateData);
if ($qqTransactionServiceCreateResult['status'] === false) {
throw new ServerErrorHttpException($qqTransactionServiceCreateResult['message'], $qqTransactionServiceCreateResult['code']);
}
$data['status'] = QqVideoMultipartUpload::STATUS_UPLOADED;
}
$result = $this->saveModelByData($data);
if ($result['status'] === false) {
throw new ServerErrorHttpException($result['message'], $result['code']);
}
// 基于资源ID、企鹅号的应用的任务ID获取企鹅号的视频文件分片上传的单个数据模型
$qqVideoMultipartUploadItem = $this->getModelByAssetIdAndQqAppTaskId($assetId, $qqCwAppTaskPublishItem->id);
$i++;
}
}
9. Execute the task command in the upload resource queue, it will slice the file, upload the corresponding resources, and update the video file of the Penguin number to upload the table, and add the transaction of the new Penguin number.
PS E:\wwwroot\channel-pub-api> ./yii upload-asset-queue/run --verbose=1 --isolate=1 --color=0
2018-11-15 11:02:29 [pid: 39632] - Worker is started
2018-11-15 11:02:30 [1] common\jobs\UploadAssetJob (attempt: 1, pid: 39632) - Started
2018-11-15 11:06:48 [1] common\jobs\UploadAssetJob (attempt: 1, pid: 39632) - Done (257.816 s)
2018-11-15 11:06:48 [pid: 39632] - Worker is stopped (0:04:19)
10. View the generated sliced files. Since the slice size is 100M, the file slices of 396 MB (415, 352, 401 bytes) are 4 small files, as shown in Figure 1
Figure 1
11. After the upload is successful, the starting position of the shard and the end position of the shard are equal to the size of the file: 415352401, as shown in Figure 2
Figure 2
12. Check my material in the background of Penguin, the video file has been successfully uploaded, as shown in Figure 3
Figure 3
