基于企鹅号的视频文件分片上传的实现流程，包含队列、文件切片、while 循环等
1、数据库结构的设计，一张资源表，一张企鹅号的视频文件分片上传表，一张企鹅号的事务表，结构如下：
28、asset：资源 Asset id 主键 channel_id 渠道ID channel_code 渠道代码，qq：企鹅号；wx：微信公众帐号 channel_type_id 渠道的类型ID channel_type_code 渠道的类型代码，qq_cw：企鹅号的内容网站应用；qq_tp：企鹅号的第三方服务平台应用；wx：微信公众帐号应用 source 来源，xContent：内容库；vms：视频管理系统；cms：内容管理系统；spider：自媒体 type 资源文件的类型，image：图片；video：视频 absolute_url 来源的资源文件的绝对URL relative_path 渠道发布的资源文件的相对路径 size 文件大小，单位(字节) task_id 任务ID channel_article_id 渠道的文章ID status 状态，0：禁用；1：启用 is_deleted 是否被删除，0：否；1：是 created_at 创建时间 updated_at 更新时间 deleted_at 删除时间
29、qq_video_multipart_upload：企鹅号的视频文件分片上传 QqCwVideoMultipartUpload id 主键 asset_id 资源ID qq_app_task_id 企鹅号的应用的任务ID qq_app_id 企鹅号的应用ID qq_app_type 企鹅号的应用类型，cw：内容网站应用；tp：第三方服务平台应用 size 视频文件大小，单位(字节) md5 视频文件MD5值 sha 视频文件SHA-1值 transaction_id 上传的唯一事务ID mediatrunk 视频 mediatrunk 文件 start_offset 分片的起始位置(从0开始计数) end_offset 分片的结束位置 vid 视频文件唯一标示ID status 状态，0：禁用；1：待上传；2：上传中；3：上传中(已失败)；4：已上传 is_deleted 是否被删除，0：否；1：是 created_at 创建时间 updated_at 更新时间 deleted_at 删除时间
30、qq_transaction：企鹅号的事务 QqTransaction id 主键 group_id 租户ID qq_app_task_id 企鹅号的应用的任务ID qq_app_id 企鹅号的应用ID qq_app_type 企鹅号的应用类型，cw：内容网站应用；tp：第三方服务平台应用 qq_article_id 企鹅号的文章ID qq_video_multipart_upload_id 企鹅号的应用的视频文件分片上传ID transaction_id 事务ID type 类型，1：文章；2：视频 transaction_ctime 事务创建时间 ext_err 扩展的错误 transaction_err_msg 事务的错误信息 article_abstract 文章摘要 article_type 文章类型，取值：普通文章，图文文章，视频文章，直播文章，RTMP直播文章 article_type_code 文章类型代码，normal：文章；multivideos：视频；images：组图；live：直播 article_url 文章快报链接 article_imgurl 文章封面图 article_title 文章标题 article_pub_flag 文章发布状态，取值：未发布，发布成功，审核中 article_pub_time 文章发布时间 article_video_title 视频文章标题 article_video_desc 视频文章描述 article_video_type 视频文章类型，视频 article_video_vid 视频文章的视频唯一ID task_id 任务ID status 状态，0：禁用；1：成功；2：失败；3：处理中 is_deleted 是否被删除，0：否；1：是 created_at 创建时间 updated_at 更新时间 deleted_at 删除时间
2、资源的上传是基于队列实现的，因此会先将资源数据存储至资源表，进而入上传资源的队列，再执行上传资源的作业
3、执行 1 次接口请求，此时资源表已经存在资源的相应数据，且入复制资源的队列
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/info --color=0 Jobs - waiting: 1 - delayed: 0 - reserved: 0 - done: 0
4、执行复制资源队列中的任务命令，会复制相应的资源，且将资源的相对路径存储至资源表中
PS E:\wwwroot\channel-pub-api> ./yii copy-asset-queue/run --verbose=1 --isolate=1 --color=0 2018-11-15 10:05:56 [pid: 23112] - Worker is started 2018-11-15 10:05:57 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 23112) - Started 2018-11-15 10:06:08 [1] common\jobs\CopyAssetJob (attempt: 1, pid: 23112) - Done (11.394 s) 2018-11-15 10:06:08 [pid: 23112] - Worker is stopped (0:00:12)
5、复制资源队列中的任务执行成功后，会入上传资源的队列
PS E:\wwwroot\channel-pub-api> ./yii upload-asset-queue/info --color=0 Jobs - waiting: 1 - delayed: 0 - reserved: 0 - done: 0
6、上传资源的代码分为 2 个部分，一为文件切片，需要将视频资源文件切片为100M大小的小文件，\channel-pub-api\common\services\AssetService.php
/**
* 文件切片
* @param string $fileAbsolutePath 需要切片的文件的绝对路径
* 格式如下：E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件.mp4
*
* @param int $size 104857600，单位为字节
*
* 生成文件列表如下：
* E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件_0.mp4
* E:/wwwroot/channel-pub-api/storage/spider/videos/切片文件_1.mp4
*/
public static function cut($fileAbsolutePath, $size)
{
// 获取需要切片的文件的路径信息
$pathInfo = pathinfo($fileAbsolutePath);
$i = 0;
$handle = fopen($fileAbsolutePath, "rb");
while (!feof($handle)) {
$cutHandle = fopen($pathInfo['dirname'] . '/' . $pathInfo['filename'] . '_' . $i . '.' . $pathInfo['extension'], "wb");
fwrite($cutHandle, fread($handle, $size));
fclose($cutHandle);
unset($cutHandle);
$i++;
}
fclose($handle);
}
7、HTTP请求，企鹅号的内容网站应用的视频文件分片上传，\channel-pub-api\common\logics\http\qq_api\Video.php
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2018/10/26
* Time: 15:33
*/
namespace common\logics\http\qq_api;
use Yii;
use yii\httpclient\Client;
use yii\web\ServerErrorHttpException;
/**
* 企鹅号接口的视频
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class Video extends Model
{
const CUT_SIZE = 104857600; //视频分片上传的切片大小：100M
/**
* HTTP请求，申请企鹅号的内容网站应用的视频文件分片上传的唯一事务ID
*
* @param array $data 数据
* 格式如下：
* [
* 'accessToken' => 'KCK0TIV8NDY44ONAEFI2QW', // 企鹅平台企鹅号应用授权调用凭据
* 'size' => 9135849, // 视频文件大小，单位(字节)
* 'md5' => 'd081c619ef7dc62eb2aa29c7c83c6f26', // 视频文件MD5值
* 'sha' => '28a1076b867bed8202df5b3f656b798148e58ced', // 视频文件SHA-1值
* ]
*
* @return array|false
* 格式如下：
* 企鹅号的内容网站应用的视频文件分片上传的唯一事务ID
* [
* 'message' => '', // 说明
* 'data' => [ // 数据
* 'transaction_id' => '780930255958621794', // 上传的唯一事务ID
* ],
* ]
*
* 失败(将错误保存在 [[yii\base\Model::errors]] 属性中)
* false
*
* @throws ServerErrorHttpException 如果响应状态码不等于20x
*/
public function clientUploadReady($data)
{
$response = Yii::$app->qqApiHttps->createRequest()
->setMethod('post')
->setUrl('video/clientuploadready')
->setData([
'access_token' => $data['accessToken'],
'size' => $data['size'],
'md5' => $data['md5'],
'sha' => $data['sha'],
])
->send();
// file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/client-upload-ready_' . $data['size'] . '_' . time() . '.txt', $response->data['data']['transaction_id']);
// 检查响应状态码是否等于20x
if ($response->isOk) {
// 检查业务逻辑是否成功
if ($response->data['code'] === 0) {
$responseData = ['message' => '', 'data' => $response->data['data']];
return $responseData;
} else {
$this->addError('id', $response->data['msg']);
return false;
}
} else {
throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35041'), ['statusCode' => $response->getStatusCode()])), 35041);
}
}
/**
* HTTP请求，企鹅号的内容网站应用的视频文件分片上传
*
* @param array $data 数据
* 格式如下：
* [
* 'accessToken' => 'LQVHLUQDNEKIDBFARJ1OOA', // 企鹅平台企鹅号应用授权调用凭据
* 'transactionId' => '780930287703152921', // 上传的唯一事务ID
* 'mediatrunk' => 'E:/wwwroot/channel-pub-api/storage/channel-pub-api/videos/2018/10/27/1540632234.9842.66697370.mp4', // 视频 mediatrunk 文件
* 'startOffset' => 0, // 分片的起始位置(从0开始计数)
* ]
*
* @return array|false
* 格式如下：
* 企鹅号的内容网站应用的视频文件分片上传
* [
* 'message' => '', // 说明
* 'data' => [ // 数据
* 'end_offset' => 2198151, // 分片的结束位置
* 'start_offset' => 2198151, // 分片的起始位置
* 'transaction_id' => 780930255958621794, // 上传的唯一事务ID
* ],
* ]
*
* 失败(将错误保存在 [[yii\base\Model::errors]] 属性中)
* false
*
* @throws ServerErrorHttpException 如果响应状态码不等于20x
*/
public function clientUploadTrunk($data)
{
$response = Yii::$app->qqApiHttp->createRequest()
->setMethod('post')
->setUrl('video/clientuploadtrunk?access_token=' . $data['accessToken'] . '&transaction_id=' . $data['transactionId'] . '&start_offset=' . $data['startOffset'])
->addFile('mediatrunk', $data['mediatrunk'])
->send();
// file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/client-upload-trunk_' . $data['transactionId'] . '_' . $data['startOffset'] . '_' . time() . '.txt', $response->data['code']);
// 检查响应状态码是否等于20x
if ($response->isOk) {
// 检查业务逻辑是否成功
if ($response->data['code'] === 0) {
$responseData = ['message' => '', 'data' => $response->data['data']];
return $responseData;
} elseif ($response->data['code'] === 40027) { // 无效的事务ID
$this->addError('id', $response->data['code']);
return false;
} else {
$this->addError('id', $response->data['msg']);
return false;
}
} else {
throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35041'), ['statusCode' => $response->getStatusCode()])), 35041);
}
}
}
8、上传视频文件的代码，\channel-pub-api\common\services\QqCwVideoMultipartUploadService.php
/**
* HTTP请求，企鹅号的内容网站应用的视频文件分片上传
* @param array $data 数据
* 格式如下：
*
* [
* 'accessToken' => 'LQVHLUQDNEKIDBFARJ1OOA', // 企鹅平台企鹅号应用授权调用凭据
* 'transactionId' => '780930287703152921', // 上传的唯一事务ID
* 'mediatrunk' => 'E:/wwwroot/channel-pub-api/storage/channel-pub-api/videos/2018/10/27/1540632234.9842.66697370.mp4', // 视频 mediatrunk 文件
* 'startOffset' => 0, // 分片的起始位置(从0开始计数)
* ]
*
* @return array
* 格式如下：
*
* [
* 'end_offset' => 2198151, // 分片的结束位置
* 'start_offset' => 2198151, // 分片的起始位置
* 'transaction_id' => 780930255958621794, // 上传的唯一事务ID
* ]
*
* @throws ServerErrorHttpException
*/
public function httpUploadTrunk($data)
{
/* HTTP请求，企鹅号的内容网站应用的视频文件分片上传 */
$httpQqApiVideo = new HttpQqApiVideo();
$uploadTrunk = $httpQqApiVideo->clientUploadTrunk($data);
if ($uploadTrunk === false) {
if ($httpQqApiVideo->hasErrors()) {
foreach ($httpQqApiVideo->getFirstErrors() as $message) {
$firstErrors = $message;
break;
}
throw new ServerErrorHttpException(Yii::t('common/error', Yii::t('common/error', Yii::t('common/error', '35042'), ['firstErrors' => $firstErrors])), 35042);
} elseif (!$httpQqApiVideo->hasErrors()) {
throw new ServerErrorHttpException('Penguin\'s content website application api HTTP requests fail for unknown reasons.');
}
}
return $uploadTrunk['data'];
}
/**
* 企鹅号的内容网站应用的视频文件分片上传
*
* @param int $assetId 资源ID
* 格式如下：1
*
* @param int $qqCwAppTaskId 企鹅号的内容网站应用的任务ID
* 格式如下：6
*
* @throws ServerErrorHttpException
* @throws \Throwable
*/
public function upload($assetId, $qqCwAppTaskId)
{
// 基于ID查找状态为启用的单个数据模型(资源)
$assetEnabledItem = AssetService::findModelEnabledById($assetId);
// 基于ID查找状态为启用的单个数据模型(任务)
$taskEnabledItem = TaskService::findModelEnabledById($assetEnabledItem->task_id);
// 基于ID查找状态为发布中的单个数据模型(企鹅号的内容网站应用的任务)
$qqCwAppTaskPublishItem = QqCwAppTaskService::findModelPublishById($qqCwAppTaskId);
// 基于ID查找状态为启用的单个数据模型(企鹅号的内容网站应用)
$qqCwAppEnabledItem = QqCwAppService::findModelEnabledById($qqCwAppTaskPublishItem->qq_cw_app_id);
// 基于企鹅号的内容网站应用ID获取有效的 Access Token
$qqCwAccessTokenService = new QqCwAccessTokenService();
$accessTokenValidity = $qqCwAccessTokenService->getAccessTokenValidityByQqCwAppId($qqCwAppEnabledItem->id);
// 基于资源ID、企鹅号的应用的任务ID获取企鹅号的视频文件分片上传的单个数据模型
$qqVideoMultipartUploadItem = $this->getModelByAssetIdAndQqAppTaskId($assetId, $qqCwAppTaskPublishItem->id);
$data = [
'assetId' => $qqVideoMultipartUploadItem->asset_id,
'qqAppTaskId' => $qqVideoMultipartUploadItem->qq_app_task_id,
'qqAppId' => $qqVideoMultipartUploadItem->qq_app_id,
'qqAppType' => $qqVideoMultipartUploadItem->qq_app_type,
'size' => $qqVideoMultipartUploadItem->size,
'md5' => $qqVideoMultipartUploadItem->md5,
'sha' => $qqVideoMultipartUploadItem->sha,
'transactionId' => (string) $qqVideoMultipartUploadItem->transaction_id,
'mediatrunk' => $qqVideoMultipartUploadItem->mediatrunk,
'startOffset' => $qqVideoMultipartUploadItem->start_offset,
'endOffset' => $qqVideoMultipartUploadItem->end_offset,
'vid' => $qqVideoMultipartUploadItem->vid,
'status' => $qqVideoMultipartUploadItem->status,
];
// 文件切片
AssetService::cut(Yii::$app->params['channelPubApi']['asset'][$assetEnabledItem->type]['basePath'] . $qqVideoMultipartUploadItem->mediatrunk, HttpQqApiVideo::CUT_SIZE);
// 获取需要切片的文件的路径信息
$pathInfo = pathinfo(Yii::$app->params['channelPubApi']['asset'][$assetEnabledItem->type]['basePath'] . $qqVideoMultipartUploadItem->mediatrunk);
// 判断分片的起始位置与分片的结束位置是否等于视频文件大小，如果相等则中断分片上传，否则继续执行分片上传
$i = 0;
while ($qqVideoMultipartUploadItem->start_offset != $qqVideoMultipartUploadItem->size && $qqVideoMultipartUploadItem->end_offset != $qqVideoMultipartUploadItem->size) {
// file_put_contents('E:/wwwroot/channel-pub-api/qq/runtime/while_' . $assetId . '_' . $qqCwAppId . '_' . $qqVideoMultipartUploadItem->start_offset . '_' . $qqVideoMultipartUploadItem->end_offset . '_' . time() . '.txt', $assetId);
// HTTP请求，企鹅号的内容网站应用的视频文件分片上传
$httpUploadTrunkData = [
'accessToken' => $accessTokenValidity->access_token,
'transactionId' => $qqVideoMultipartUploadItem->transaction_id,
'mediatrunk' => $pathInfo['dirname'] . '/' . $pathInfo['filename'] . '_' . $i . '.' . $pathInfo['extension'],
'startOffset' => $qqVideoMultipartUploadItem->start_offset,
];
$uploadTrunkData = $this->httpUploadTrunk($httpUploadTrunkData);
// 基于资源ID、企鹅号的内容网站应用ID插入/更新数据至企鹅号的内容网站应用的视频文件分片上传
$data['startOffset'] = $uploadTrunkData['start_offset'];
$data['endOffset'] = $uploadTrunkData['end_offset'];
if ($uploadTrunkData['start_offset'] != $qqVideoMultipartUploadItem->size && $uploadTrunkData['end_offset'] != $qqVideoMultipartUploadItem->size) {
$data['status'] = QqVideoMultipartUpload::STATUS_UPLOADING;
} else {
// HTTP请求，基于上传的唯一事务ID获取事务信息
$qqTransactionService = new QqTransactionService();
$qqTransactionServiceHttpTransactionInfoData = [
'accessToken' => $accessTokenValidity->access_token,
'transactionId' => $qqVideoMultipartUploadItem->transaction_id,
];
$qqTransactionServiceHttpTransactionInfoResult = $qqTransactionService->httpTransactionInfo($qqTransactionServiceHttpTransactionInfoData);
// 创建企鹅号的事务
$qqTransactionServiceCreateData = [
'groupId' => $taskEnabledItem->group_id,
'qqAppTaskId' => $qqVideoMultipartUploadItem->qq_app_task_id,
'qqAppId' => $qqVideoMultipartUploadItem->qq_app_id,
'qqAppType' => $qqVideoMultipartUploadItem->qq_app_type,
'qqArticleId' => 0,
'qqVideoMultipartUploadId' => $qqVideoMultipartUploadItem->id,
'transactionId' => $qqVideoMultipartUploadItem->transaction_id,
'type' => $qqTransactionService::getTypeByHttpQqApiTransactionType($qqTransactionServiceHttpTransactionInfoResult['transaction_type']),
'transactionCtime' => $qqTransactionServiceHttpTransactionInfoResult['transaction_ctime'],
'extErr' => $qqTransactionServiceHttpTransactionInfoResult['ext_err'],
'transactionErrMsg' => $qqTransactionServiceHttpTransactionInfoResult['transaction_err_msg'],
'articleAbstract' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_abstract'],
'articleType' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_type'],
'articleTypeCode' => '',
'articleUrl' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_url'],
'articleImgurl' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_imgurl'],
'articleTitle' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_title'],
'articlePubFlag' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_pub_flag'],
'articlePubTime' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_pub_time'],
'articleVideoTitle' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['title'],
'articleVideoDesc' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['desc'],
'articleVideoType' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['type'],
'articleVideoVid' => $qqTransactionServiceHttpTransactionInfoResult['article_info']['article_video_info']['vid'],
'taskId' => $assetEnabledItem->task_id,
'status' => QqTransaction::STATUS_PROCESSING,
];
$qqTransactionServiceCreateResult = $qqTransactionService->create($qqTransactionServiceCreateData);
if ($qqTransactionServiceCreateResult['status'] === false) {
throw new ServerErrorHttpException($qqTransactionServiceCreateResult['message'], $qqTransactionServiceCreateResult['code']);
}
$data['status'] = QqVideoMultipartUpload::STATUS_UPLOADED;
}
$result = $this->saveModelByData($data);
if ($result['status'] === false) {
throw new ServerErrorHttpException($result['message'], $result['code']);
}
// 基于资源ID、企鹅号的应用的任务ID获取企鹅号的视频文件分片上传的单个数据模型
$qqVideoMultipartUploadItem = $this->getModelByAssetIdAndQqAppTaskId($assetId, $qqCwAppTaskPublishItem->id);
$i++;
}
}
9、执行上传资源队列中的任务命令，会切片文件，上传相应的资源，且更新企鹅号的视频文件分片上传表、新增企鹅号的事务
PS E:\wwwroot\channel-pub-api> ./yii upload-asset-queue/run --verbose=1 --isolate=1 --color=0 2018-11-15 11:02:29 [pid: 39632] - Worker is started 2018-11-15 11:02:30 [1] common\jobs\UploadAssetJob (attempt: 1, pid: 39632) - Started 2018-11-15 11:06:48 [1] common\jobs\UploadAssetJob (attempt: 1, pid: 39632) - Done (257.816 s) 2018-11-15 11:06:48 [pid: 39632] - Worker is stopped (0:04:19)
10、查看生成的切片小文件，由于切片大小为100M，396 MB (415,352,401 字节)的文件切片为4个小的文件，如图1
图1
11、上传成功后，分片的起始位置与分片的结束位置皆等于文件的大小：415352401，如图2
图2
12、在企鹅号后台查看我的素材，视频文件已经分片上传成功，如图3
图3
近期评论