The command-line script implemented based on Yii 2.0, during the continuous operation, the analysis and solution of excessive memory occupancy
1. When the amount of data in MySQL and Redis is small, the CPU of the Docker container: 0.02%, and the memory: 298MB, as shown in Figure 1 and Figure 2
Figure 1
Figure 2
2. Check the instance monitoring situation of mysql, the memory usage: 2328MB, as shown in Figure 3
Figure 3
3. Check the instance monitoring situation of Redis, the used capacity: 33MB, as shown in Figure 4
Figure 4
4. The command line script \Console\Controllers\CmcConsoleUserController.php , \Console\Consoles\ConfigColumUserController.php The code is as follows:
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/01/09
* Time: 13:55
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\services\CmcConsoleUserService;
use yii\console\Controller;
use yii\console\ExitCode;
use yii\helpers\ArrayHelper;
use yii\web\HttpException;
use yii\web\ServerErrorHttpException;
use common\logics\http\im\IMIndependentMode;
/**
* 框架服务控制台的用户
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class CmcConsoleUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Http)至框架服务控制台的用户模型(Redis)
*
* @throws ServerErrorHttpException
* @throws HttpException 如果登录超时
*/
public function actionSync()
{
// 查询框架服务控制台的用户模型(Redis)的租户ID(以租户 ID 索引结果集)
$redisCmcConsoleUserIndexByGroupIdItems = RedisCmcConsoleUser::find()->indexBy('group_id')->all();
/* 判断 $redisCmcConsoleUserIndexByGroupIdItems 是否为空，如果为空，则成功退出 */
if (empty($redisCmcConsoleUserIndexByGroupIdItems)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($redisCmcConsoleUserIndexByGroupIdItems as $redisCmcConsoleUserIndexByGroupIdItemKey => $redisCmcConsoleUserIndexByGroupIdItemValue) {
$isLockExist = RedisCmcConsoleUser::isLockExist($redisCmcConsoleUserIndexByGroupIdItemKey);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// HTTP请求，获取租户ID下的用户列表
$httpGetUserListData = [
'groupId' => $redisCmcConsoleUserIndexByGroupIdItemKey,
'loginId' => '',
'loginTid' => '',
];
$getUserListData = CmcConsoleUserService::httpGetUserList($httpGetUserListData);
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where([ 'group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey ])->indexBy('id')->all();
// 使用键名比较计算数组的差集，如果不为空，则删除出现在 Redis 中但是未出现在 Http 中的记录
$redisArrayDiffItems = array_diff_key($redisCmcConsoleUserItems, $httpCmcConsoleUserItems);
if (!empty($redisArrayDiffItems)) {
$redisArrayDiffIds = array_keys($redisArrayDiffItems);
if (RedisCmcConsoleUser::deleteAllByIds($redisCmcConsoleUserIndexByGroupIdItemKey, $redisArrayDiffIds) === false) {
continue;
}
}
$im = new IMIndependentMode();
// 遍历框架服务控制台的用户模型(Http)，判断在 Redis 中是否存在，如果不存在，则插入，如果存在且更新时间不相等，则更新
foreach ($httpCmcConsoleUserItems as $httpCmcConsoleUserItemValue) {
$redisCmcConsoleUserItem = RedisCmcConsoleUser::find()->where([ 'id' => $httpCmcConsoleUserItemValue['id'] ])->one();
$attributes = [
'id' => $httpCmcConsoleUserItemValue['id'],
'group_id' => $getUserListData['group_info']['group_id'],
'login_name' => $httpCmcConsoleUserItemValue['login_name'],
'user_token' => $httpCmcConsoleUserItemValue['user_token'],
'user_nick' => $httpCmcConsoleUserItemValue['user_nick'],
'user_pic' => $httpCmcConsoleUserItemValue['user_pic'],
'user_mobile' => $httpCmcConsoleUserItemValue['user_mobile'] ? $httpCmcConsoleUserItemValue['user_mobile'] : '',
'user_email' => $httpCmcConsoleUserItemValue['user_email'] ? $httpCmcConsoleUserItemValue['user_email'] : '',
'user_sex' => $httpCmcConsoleUserItemValue['user_sex'],
'user_type' => $httpCmcConsoleUserItemValue['user_type'],
'user_birthday' => $httpCmcConsoleUserItemValue['user_birthday'],
'user_chat_id' => $httpCmcConsoleUserItemValue['user_chat_id'] ? $httpCmcConsoleUserItemValue['user_chat_id'] : '',
'is_open' => $httpCmcConsoleUserItemValue['is_open'],
'add_time' => $httpCmcConsoleUserItemValue['add_time'],
'update_time' => $httpCmcConsoleUserItemValue['update_time'],
'im_identity' => md5($getUserListData['group_info']['group_id'] . $httpCmcConsoleUserItemValue['login_name']),
];
if (!isset($redisCmcConsoleUserItem)) {
$redisCmcConsoleUser = new RedisCmcConsoleUser();
$redisCmcConsoleUser->attributes = $attributes;
$redisCmcConsoleUser->insert();
} else if (isset($redisCmcConsoleUserItem) && $httpCmcConsoleUserItemValue['update_time'] != $redisCmcConsoleUserItem['update_time']) {
$redisCmcConsoleUserItem->attributes = $attributes;
$redisCmcConsoleUserItem->save();
}
$this->addIMAccount($im, $attributes['im_identity'], $attributes['user_nick'], $attributes['user_pic']);
}
}
// 延缓执行 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
return ExitCode::OK;
}
private function addIMAccount(IMIndependentMode $im, $identifier, $nick, $faceUrl)
{
$im->generateUserSig();
$ret = $im->getAccountProfile($identifier);
if (isset($ret['ActionStatus']) && $ret['ActionStatus'] != 'OK') {
$im->accountImport($identifier, $nick, $faceUrl);
}
}
}
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/03/01
* Time: 13:17
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\models\ConfigColumnUser;
use yii\console\Controller;
use yii\console\ExitCode;
/**
* 栏目人员配置
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class ConfigColumnUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Redis)至栏目人员配置模型(MySQL)
*
*/
public function actionSync()
{
// 查询框架服务控制台的用户模型(Redis)的租户ID(以租户 ID 索引结果集)
$redisCmcConsoleUserIndexByGroupIdItems = RedisCmcConsoleUser::find()->indexBy('group_id')->all();
/* 判断 $redisCmcConsoleUserIndexByGroupIdItems 是否为空，如果为空，则成功退出 */
if (empty($redisCmcConsoleUserIndexByGroupIdItems)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($redisCmcConsoleUserIndexByGroupIdItems as $redisCmcConsoleUserIndexByGroupIdItemKey => $redisCmcConsoleUserIndexByGroupIdItemValue) {
$isLockExist = RedisCmcConsoleUser::isLockExist($redisCmcConsoleUserIndexByGroupIdItemKey);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where([ 'group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey ])->indexBy('id')->all();
// 基于租户ID查询栏目人员配置模型(MySQL)(以 用户ID 索引结果集)
$configColumnUserItems = ConfigColumnUser::find()->where([ 'group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey ])->isDeletedNo()->indexBy('user_id')->all();
// 使用键名比较计算数组的差集，如果不为空，则删除 (软删除) 出现在 栏目人员配置模型(MySQL) 中但是未出现在 框架服务控制台的用户模型(Redis) 中的记录
$diffItems = array_diff_key($configColumnUserItems, $redisCmcConsoleUserItems);
if (!empty($diffItems)) {
foreach ($diffItems as $diffItem) {
/* @var $diffItem \console\models\ConfigColumnUser */
$diffItem->softDelete();
}
continue;
}
}
// 延缓执行 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyNoSleepTime']);
return ExitCode::OK;
}
}
5. The running of the command line script is based on Supervisor Provide support, /etc/supervisord.d/yii-cmc-console-user-sync.ini, /etc/supervisord.d/yii-config-column-user-sync.ini, as shown in Figure 5
Figure 5
[program:yii-cmc-console-user-sync]
command = php /sobey/www/pcs-api/yii cmc-console-user/sync
autorestart = true
startsecs = 0
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-cmc-console-user-sync-stderr.log
stdout_logfile = /data/logs/yii-cmc-console-user-sync-stdout.log
[program:yii-config-column-user-sync]
command = php /sobey/www/pcs-api/yii config-column-user/sync
autorestart = true
startsecs = 0
stopwaitsecs = 10
stderr_logfile = /data/logs/yii-config-column-user-sync-stderr.log
stdout_logfile = /data/logs/yii-config-column-user-sync-stdout.log
6. Import user data in batches under 8 tenants, so that the number of imported users under each tenant is 999, a total of about 8000, as shown in Figure 6
Tenant name: Guangdong Province, Guangzhou City, Shenzhen City, Nanshan District, Luohu District, Shekou Street, China Merchants Street, Default tenant
Figure 6
7. In the Redis of the local environment, the number of users is 8223. In the Redis of the development environment, the connection is the same frame, but the number of users has always been around 3000, and it has never reached 8223, as shown in Figure 7 and Figure 8
Figure 7
8
8. The problem is finally discovered, because there are 2 Each container is running at the same time, and the command line is running in each container: cmc-console-user/sync, the data is conflicted with each other, and the stop Stop the command line script in another container (plan to deploy the command line separately to a container and separate it, so as to prepare for subsequent cluster deployment), as shown in Figure 9
Figure 9
[root@79af01f496bb pcs-api]# supervisorctl status
cronolog RUNNING pid 426, uptime 2:08:17
nginx RUNNING pid 422, uptime 2:08:17
php-fpm RUNNING pid 424, uptime 2:08:17
yii-cmc-console-user-sync RUNNING pid 20685, uptime 0:04:42
yii-config-column-user-sync RUNNING pid 23504, uptime 0:00:45
[root@79af01f496bb pcs-api]# supervisorctl stop yii-cmc-console-user-sync
yii-cmc-console-user-sync: stopped
[root@79af01f496bb pcs-api]# supervisorctl status
cronolog RUNNING pid 426, uptime 2:08:41
nginx RUNNING pid 422, uptime 2:08:41
php-fpm RUNNING pid 424, uptime 2:08:41
yii-cmc-console-user-sync STOPPED Jul 08 01:32 PM
yii-config-column-user-sync RUN
9. In the Redis of the local environment, the number of users is 8223, and in the Redis of the development environment, the number of users is 8223, which is already the same, as shown in Figure 10
Figure 10
10. In the column settings of 8 tenants, add 2 columns under each tenant, and add all users to each column, that is, the number of data added under each tenant is about 2000, a total of about 16000, as shown in Figure 11
Figure 11
11. After the execution of the 6th step, the CPU of the Docker container: 0.05%, memory: 382MB, the memory has increased by about 80MB, and the memory will increase every 1000 pieces of data. Around 10 MB. After the execution of step 10, the CPU of the Docker container: 9.9%, memory: 541MB, the memory has increased by about 160MB, and the memory will increase by 20 MB for every 1000 pieces of data. left and right. Therefore, the focus of optimization is on the second command line script. As shown in Figure 12, Figure 13
Figure 12
Figure 13
12. Edit \Console\Controllers\CmcConsoleUserController.php, \Console\Co NTROLLERS\ConfigColumNuserController.php, check the actual amount of memory used, the total amount of memory allocated by the system
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
13. Run the first command line script, the log is as follows:
8.9912643432617MB
40.378349304199MB
14. Run the second command line script, the log is as follows: Summary: The main difference from the first command line script is about the actual amount of memory used, and the difference is about 17 MB.
26.464874267578MB
40.366722106934MB
15. Analysis and optimization of the first command line script, \console\controllers\cmcConsoleUserController.php, based on unset() implementation, the actual amount of memory used is reduced by 2 MB On the left and right, the total amount of memory allocated by the system does not change
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/01/09
* Time: 13:55
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\services\CmcConsoleUserService;
use yii\console\Controller;
use yii\console\ExitCode;
use yii\helpers\ArrayHelper;
use yii\web\HttpException;
use yii\web\ServerErrorHttpException;
use common\logics\http\im\IMIndependentMode;
/**
* 框架服务控制台的用户
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class CmcConsoleUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Http)至框架服务控制台的用户模型(Redis)
*
* @throws ServerErrorHttpException
* @throws HttpException 如果登录超时
*/
public function actionSync()
{
// 查询框架服务控制台的用户模型(Redis)的租户ID(以租户 ID 索引结果集)
$redisCmcConsoleUserIndexByGroupIdItems = RedisCmcConsoleUser::find()->indexBy('group_id')->all();
/* 判断 $redisCmcConsoleUserIndexByGroupIdItems 是否为空，如果为空，则成功退出 */
if (empty($redisCmcConsoleUserIndexByGroupIdItems)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($redisCmcConsoleUserIndexByGroupIdItems as $redisCmcConsoleUserIndexByGroupIdItemKey => $redisCmcConsoleUserIndexByGroupIdItemValue) {
$isLockExist = RedisCmcConsoleUser::isLockExist($redisCmcConsoleUserIndexByGroupIdItemKey);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// HTTP请求，获取租户ID下的用户列表
$httpGetUserListData = [
'groupId' => $redisCmcConsoleUserIndexByGroupIdItemKey,
'loginId' => '',
'loginTid' => '',
];
$getUserListData = CmcConsoleUserService::httpGetUserList($httpGetUserListData);
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey])->indexBy('id')->all();
// 使用键名比较计算数组的差集，如果不为空，则删除出现在 Redis 中但是未出现在 Http 中的记录
$redisArrayDiffItems = array_diff_key($redisCmcConsoleUserItems, $httpCmcConsoleUserItems);
// 销毁变量
unset($redisCmcConsoleUserItems);
if (!empty($redisArrayDiffItems)) {
$redisArrayDiffIds = array_keys($redisArrayDiffItems);
// 销毁变量
unset($redisArrayDiffItems);
if (RedisCmcConsoleUser::deleteAllByIds($redisCmcConsoleUserIndexByGroupIdItemKey, $redisArrayDiffIds) === false) {
continue;
}
}
$im = new IMIndependentMode();
// 遍历框架服务控制台的用户模型(Http)，判断在 Redis 中是否存在，如果不存在，则插入，如果存在且更新时间不相等，则更新
foreach ($httpCmcConsoleUserItems as $httpCmcConsoleUserItemValue) {
$redisCmcConsoleUserItem = RedisCmcConsoleUser::find()->where(['id' => $httpCmcConsoleUserItemValue['id']])->one();
$attributes = [
'id' => $httpCmcConsoleUserItemValue['id'],
'group_id' => $getUserListData['group_info']['group_id'],
'login_name' => $httpCmcConsoleUserItemValue['login_name'],
'user_token' => $httpCmcConsoleUserItemValue['user_token'],
'user_nick' => $httpCmcConsoleUserItemValue['user_nick'],
'user_pic' => $httpCmcConsoleUserItemValue['user_pic'],
'user_mobile' => $httpCmcConsoleUserItemValue['user_mobile'] ? $httpCmcConsoleUserItemValue['user_mobile'] : '',
'user_email' => $httpCmcConsoleUserItemValue['user_email'] ? $httpCmcConsoleUserItemValue['user_email'] : '',
'user_sex' => $httpCmcConsoleUserItemValue['user_sex'],
'user_type' => $httpCmcConsoleUserItemValue['user_type'],
'user_birthday' => $httpCmcConsoleUserItemValue['user_birthday'],
'user_chat_id' => $httpCmcConsoleUserItemValue['user_chat_id'] ? $httpCmcConsoleUserItemValue['user_chat_id'] : '',
'is_open' => $httpCmcConsoleUserItemValue['is_open'],
'add_time' => $httpCmcConsoleUserItemValue['add_time'],
'update_time' => $httpCmcConsoleUserItemValue['update_time'],
'im_identity' => md5($getUserListData['group_info']['group_id'] . $httpCmcConsoleUserItemValue['login_name']),
];
if (!isset($redisCmcConsoleUserItem)) {
$redisCmcConsoleUser = new RedisCmcConsoleUser();
$redisCmcConsoleUser->attributes = $attributes;
$redisCmcConsoleUser->insert();
} else {
if (isset($redisCmcConsoleUserItem) && $httpCmcConsoleUserItemValue['update_time'] != $redisCmcConsoleUserItem['update_time']) {
$redisCmcConsoleUserItem->attributes = $attributes;
$redisCmcConsoleUserItem->save();
}
}
$this->addIMAccount($im, $attributes['im_identity'], $attributes['user_nick'], $attributes['user_pic']);
}
}
// 延缓执行 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
}
private function addIMAccount(IMIndependentMode $im, $identifier, $nick, $faceUrl)
{
$im->generateUserSig();
$ret = $im->getAccountProfile($identifier);
if (isset($ret['ActionStatus']) && $ret['ActionStatus'] != 'OK') {
$im->accountImport($identifier, $nick, $faceUrl);
}
}
}
8.9912643432617MB //优化前
6.6167755126953MB //优化后
40.378349304199MB //优化前
40.378349304199MB //优化后
16. Analysis and optimization of the second command line script, \Console\Controllers\ConfigColumNuserController.php, based on unset() implementation, the actual amount of memory used is reduced by 9 MB On the left and right, the total amount of memory allocated by the system does not change
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/03/01
* Time: 13:17
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\models\ConfigColumnUser;
use yii\console\Controller;
use yii\console\ExitCode;
/**
* 栏目人员配置
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class ConfigColumnUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Redis)至栏目人员配置模型(MySQL)
*
*/
public function actionSync()
{
// 查询框架服务控制台的用户模型(Redis)的租户ID(以租户 ID 索引结果集)
$redisCmcConsoleUserIndexByGroupIdItems = RedisCmcConsoleUser::find()->indexBy('group_id')->all();
/* 判断 $redisCmcConsoleUserIndexByGroupIdItems 是否为空，如果为空，则成功退出 */
if (empty($redisCmcConsoleUserIndexByGroupIdItems)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($redisCmcConsoleUserIndexByGroupIdItems as $redisCmcConsoleUserIndexByGroupIdItemKey => $redisCmcConsoleUserIndexByGroupIdItemValue) {
$isLockExist = RedisCmcConsoleUser::isLockExist($redisCmcConsoleUserIndexByGroupIdItemKey);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey])->indexBy('id')->all();
// 基于租户ID查询栏目人员配置模型(MySQL)(以 用户ID 索引结果集)
$configColumnUserItems = ConfigColumnUser::find()->where(['group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey])->isDeletedNo()->indexBy('user_id')->all();
// 使用键名比较计算数组的差集，如果不为空，则删除 (软删除) 出现在 栏目人员配置模型(MySQL) 中但是未出现在 框架服务控制台的用户模型(Redis) 中的记录
$diffItems = array_diff_key($configColumnUserItems, $redisCmcConsoleUserItems);
// 销毁变量
unset($redisCmcConsoleUserItems, $configColumnUserItems);
if (!empty($diffItems)) {
foreach ($diffItems as $diffItem) {
/* @var $diffItem ConfigColumnUser */
$diffItem->softDelete();
}
continue;
}
}
// 延缓执行 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyNoSleepTime']);
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
}
}
26.464874267578MB //优化前
17.696464538574MB //优化后
40.366722106934MB //优化前
40.366722106934MB //优化后
17. Upgrade to the development environment, the CPU of the Docker container: 5.4%, memory: 371MB, the memory is reduced (541MB – 371MB) = about 170MB, which is proved based on unset() The implemented scheme is feasible. Figure 14
Figure 14
18. Analysis and optimization of the first command line script, \Console\Controllers\CmcConsoleUserController.php, with the goal of reducing the execution time, from about 30 minutes to about 1 minute. Delete: addDiMaccount() related implementation, which executes HTTP requests during the traversal process, which consumes a lot of time. The actual amount of memory used is reduced by about 0.09 MB, and the total amount of memory allocated by the system is reduced by about 0.002 MB
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/01/09
* Time: 13:55
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\services\CmcConsoleUserService;
use yii\console\Controller;
use yii\console\ExitCode;
use yii\helpers\ArrayHelper;
use yii\web\HttpException;
use yii\web\ServerErrorHttpException;
/**
* 框架服务控制台的用户
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class CmcConsoleUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Http)至框架服务控制台的用户模型(Redis)
*
* @throws ServerErrorHttpException
* @throws HttpException 如果登录超时
*/
public function actionSync()
{
// 查询框架服务控制台的用户模型(Redis)的租户ID(以租户 ID 索引结果集)
$redisCmcConsoleUserIndexByGroupIdItems = RedisCmcConsoleUser::find()->indexBy('group_id')->all();
/* 判断 $redisCmcConsoleUserIndexByGroupIdItems 是否为空，如果为空，则成功退出 */
if (empty($redisCmcConsoleUserIndexByGroupIdItems)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($redisCmcConsoleUserIndexByGroupIdItems as $redisCmcConsoleUserIndexByGroupIdItemKey => $redisCmcConsoleUserIndexByGroupIdItemValue) {
$isLockExist = RedisCmcConsoleUser::isLockExist($redisCmcConsoleUserIndexByGroupIdItemKey);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// HTTP请求，获取租户ID下的用户列表
$httpGetUserListData = [
'groupId' => $redisCmcConsoleUserIndexByGroupIdItemKey,
'loginId' => '',
'loginTid' => '',
];
$getUserListData = CmcConsoleUserService::httpGetUserList($httpGetUserListData);
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey])->indexBy('id')->all();
// 使用键名比较计算数组的差集，如果不为空，则删除出现在 Redis 中但是未出现在 Http 中的记录
$redisArrayDiffItems = array_diff_key($redisCmcConsoleUserItems, $httpCmcConsoleUserItems);
// 销毁变量
unset($redisCmcConsoleUserItems);
if (!empty($redisArrayDiffItems)) {
$redisArrayDiffIds = array_keys($redisArrayDiffItems);
// 销毁变量
unset($redisArrayDiffItems);
if (RedisCmcConsoleUser::deleteAllByIds($redisCmcConsoleUserIndexByGroupIdItemKey, $redisArrayDiffIds) === false) {
continue;
}
}
// 遍历框架服务控制台的用户模型(Http)，判断在 Redis 中是否存在，如果不存在，则插入，如果存在且更新时间不相等，则更新
foreach ($httpCmcConsoleUserItems as $httpCmcConsoleUserItemValue) {
$redisCmcConsoleUserItem = RedisCmcConsoleUser::find()->where(['id' => $httpCmcConsoleUserItemValue['id']])->one();
$attributes = [
'id' => $httpCmcConsoleUserItemValue['id'],
'group_id' => $getUserListData['group_info']['group_id'],
'login_name' => $httpCmcConsoleUserItemValue['login_name'],
'user_token' => $httpCmcConsoleUserItemValue['user_token'],
'user_nick' => $httpCmcConsoleUserItemValue['user_nick'],
'user_pic' => $httpCmcConsoleUserItemValue['user_pic'],
'user_mobile' => $httpCmcConsoleUserItemValue['user_mobile'] ? $httpCmcConsoleUserItemValue['user_mobile'] : '',
'user_email' => $httpCmcConsoleUserItemValue['user_email'] ? $httpCmcConsoleUserItemValue['user_email'] : '',
'user_sex' => $httpCmcConsoleUserItemValue['user_sex'],
'user_type' => $httpCmcConsoleUserItemValue['user_type'],
'user_birthday' => $httpCmcConsoleUserItemValue['user_birthday'],
'user_chat_id' => $httpCmcConsoleUserItemValue['user_chat_id'] ? $httpCmcConsoleUserItemValue['user_chat_id'] : '',
'is_open' => $httpCmcConsoleUserItemValue['is_open'],
'add_time' => $httpCmcConsoleUserItemValue['add_time'],
'update_time' => $httpCmcConsoleUserItemValue['update_time'],
'im_identity' => md5($getUserListData['group_info']['group_id'] . $httpCmcConsoleUserItemValue['login_name']),
];
if (!isset($redisCmcConsoleUserItem)) {
$redisCmcConsoleUser = new RedisCmcConsoleUser();
$redisCmcConsoleUser->attributes = $attributes;
$redisCmcConsoleUser->insert();
} else {
if (isset($redisCmcConsoleUserItem) && $httpCmcConsoleUserItemValue['update_time'] != $redisCmcConsoleUserItem['update_time']) {
$redisCmcConsoleUserItem->attributes = $attributes;
$redisCmcConsoleUserItem->save();
}
}
}
}
// 延缓执行 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
}
}
6.6167755126953MB //优化前
6.5267791748047MB //优化后
40.378349304199MB //优化前
40.37670135498MB //优化后
19. Upgrade to the development environment, the CPU of the Docker container: 5.4%, memory: 280MB, the memory is reduced (371MB – 371MB) = 0MB Left and right, proof that the execution time of the command line is reduced is meaningless for the use of memory optimization. Although in the local development environment, the actual amount of memory used is reduced by about 0.09 MB, and the total amount of memory allocated by the system is reduced by about 0.002 MB, and the amount of memory reduced is very small. as shown in Figure 15
Figure 15
20. Analysis and optimization of the first command line script, \Console\Controllers\CmcConsoleUserController.php, get the data in an array, and call the asArray() method before the query method to obtain The result of the php array form. The actual amount of memory used is reduced by about 0.09 MB, and the total amount of memory allocated by the system is reduced by about 8.96 MB
// 查询框架服务控制台的用户模型(Redis)的租户ID(以租户 ID 索引结果集)
$redisCmcConsoleUserIndexByGroupIdItems = RedisCmcConsoleUser::find()->indexBy('group_id')->asArray()->all();
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey])->indexBy('id')->asArray()->all();
6.5267791748047MB //优化前
6.4357604980469MB //优化后
40.37670135498MB //优化前
31.42064666748MB //优化后
21. Analysis and optimization of the second command line script, \Console\Controllers\ConfigColumNuserController.php, get the data in an array, and call the asArray() method before the query method to obtain The result of the php array form. The actual amount of memory used is reduced by about 12.72 MB, and the total amount of memory allocated by the system is reduced by about 8.95 MB
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/03/01
* Time: 13:17
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\models\ConfigColumnUser;
use yii\console\Controller;
use yii\console\ExitCode;
/**
* 栏目人员配置
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class ConfigColumnUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Redis)至栏目人员配置模型(MySQL)
*
*/
public function actionSync()
{
// 查询框架服务控制台的用户模型(Redis)的租户ID(以租户 ID 索引结果集)
$redisCmcConsoleUserIndexByGroupIdItems = RedisCmcConsoleUser::find()->indexBy('group_id')->asArray()->all();
/* 判断 $redisCmcConsoleUserIndexByGroupIdItems 是否为空，如果为空，则成功退出 */
if (empty($redisCmcConsoleUserIndexByGroupIdItems)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($redisCmcConsoleUserIndexByGroupIdItems as $redisCmcConsoleUserIndexByGroupIdItemKey => $redisCmcConsoleUserIndexByGroupIdItemValue) {
$isLockExist = RedisCmcConsoleUser::isLockExist($redisCmcConsoleUserIndexByGroupIdItemKey);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey])->indexBy('id')->asArray()->all();
// 基于租户ID查询栏目人员配置模型(MySQL)(以 用户ID 索引结果集)
$configColumnUserItems = ConfigColumnUser::find()->where(['group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey])->isDeletedNo()->indexBy('user_id')->asArray()->all();
// 使用键名比较计算数组的差集，如果不为空，则删除 (软删除) 出现在 栏目人员配置模型(MySQL) 中但是未出现在 框架服务控制台的用户模型(Redis) 中的记录
$diffItems = array_diff_key($configColumnUserItems, $redisCmcConsoleUserItems);
// 销毁变量
unset($redisCmcConsoleUserItems, $configColumnUserItems);
if (!empty($diffItems)) {
// 基于租户ID、用户ID查询栏目人员配置模型(MySQL)(待删除)
$toBeDeletedConfigColumnUserItems = ConfigColumnUser::find()->where([
'and',
['group_id' => $redisCmcConsoleUserIndexByGroupIdItemKey],
['in', 'user_id', $diffItems],
])->isDeletedNo()->all();
foreach ($toBeDeletedConfigColumnUserItems as $toBeDeletedConfigColumnUserItem) {
/* @var $toBeDeletedConfigColumnUserItem ConfigColumnUser */
$toBeDeletedConfigColumnUserItem->softDelete();
}
continue;
}
}
// 延缓执行 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyNoSleepTime']);
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
}
}
17.696464538574MB //优化前
4.9719772338867MB //优化后
40.366722106934MB //优化前
31.416854858398MB //优化后
22. Analyze and optimize the first command line script, \console\controllers\CmcConsoleUserController.php, the variable $attributes There is a definition in every traversal, but it is not used in most traversals, so it is only defined when it is used. The actual amount of memory used is reduced by about 0.0007 MB, and the total amount of memory allocated by the system does not change
// 遍历框架服务控制台的用户模型(Http)，判断在 Redis 中是否存在，如果不存在，则插入，如果存在且更新时间不相等，则更新
foreach ($httpCmcConsoleUserItems as $httpCmcConsoleUserItemValue) {
$redisCmcConsoleUserItem = RedisCmcConsoleUser::find()->where(['id' => $httpCmcConsoleUserItemValue['id']])->one();
if (!isset($redisCmcConsoleUserItem)) {
$redisCmcConsoleUser = new RedisCmcConsoleUser();
$attributes = $this->getAttributes($getUserListData['group_info']['group_id'], $httpCmcConsoleUserItemValue);
$redisCmcConsoleUser->attributes = $attributes;
$redisCmcConsoleUser->insert();
} else {
if ($httpCmcConsoleUserItemValue['update_time'] != $redisCmcConsoleUserItem['update_time']) {
$attributes = $this->getAttributes($getUserListData['group_info']['group_id'], $httpCmcConsoleUserItemValue);
$redisCmcConsoleUserItem->attributes = $attributes;
$redisCmcConsoleUserItem->save();
}
}
}
/**
* 获取属性列表
* @param string $groupId 租户ID
* @param array $httpCmcConsoleUser 框架服务控制台的用户模型(Http)
*
* @return array
*/
private function getAttributes($groupId, $httpCmcConsoleUser) {
return [
'id' => $httpCmcConsoleUser['id'],
'group_id' => $groupId,
'login_name' => $httpCmcConsoleUser['login_name'],
'user_token' => $httpCmcConsoleUser['user_token'],
'user_nick' => $httpCmcConsoleUser['user_nick'],
'user_pic' => $httpCmcConsoleUser['user_pic'],
'user_mobile' => $httpCmcConsoleUser['user_mobile'] ? $httpCmcConsoleUser['user_mobile'] : '',
'user_email' => $httpCmcConsoleUser['user_email'] ? $httpCmcConsoleUser['user_email'] : '',
'user_sex' => $httpCmcConsoleUser['user_sex'],
'user_type' => $httpCmcConsoleUser['user_type'],
'user_birthday' => $httpCmcConsoleUser['user_birthday'],
'user_chat_id' => $httpCmcConsoleUser['user_chat_id'] ? $httpCmcConsoleUser['user_chat_id'] : '',
'is_open' => $httpCmcConsoleUser['is_open'],
'add_time' => $httpCmcConsoleUser['add_time'],
'update_time' => $httpCmcConsoleUser['update_time'],
'im_identity' => md5($groupId . $httpCmcConsoleUser['login_name']),
];
}
6.4357604980469MB //优化前
6.4350357055664MB //优化后
31.42064666748MB //优化前
31.42064666748MB //优化后
23. Upgrade to the development environment, the CPU of the Docker container: 0.04%, memory: 283MB, the memory is reduced (371MB – 283MB) = 90MB Left and right, proof that the data is obtained in the form of an array, and call the asarray() method before the query method to obtain the result of the php array form. Optimized solutions are feasible. as shown in Figure 16
Figure 16
24. Analysis and optimization of the first command line script, \Console\Controllers\CmcConsoleUserController.php, based on unset(), the actual amount of memory used is reduced by 0.067 MB On the left and right, the total amount of memory allocated by the system does not change
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 销毁变量
unset($getUserListData['list']);
6.4350357055664MB //优化前
6.3685455322266MB //优化后
31.42064666748MB //优化前
31.42064666748MB //优化后
25. Analysis and optimization of the first command line script, \console\controllers\cmcConsoleUserController.php, it is concluded that the root cause of excessive memory usage is that rediscConsoleUser::find(). It finds 8 records in the Redis AR model records around 8000.
// 查询框架服务控制台的用户模型(Redis)的租户ID(以租户 ID 索引结果集)
$redisCmcConsoleUserIndexByGroupIdItems = RedisCmcConsoleUser::find()->indexBy('group_id')->asArray()->all();
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
3.6807098388672MB //RedisCmcConsoleUser::find() 执行后
6.3685455322266MB //命令行执行后
31.42064666748MB //RedisCmcConsoleUser::find() 执行后
31.42064666748MB //命令行执行后
26. Analysis and optimization of the first command line script, \console\controllers\cmcConsoleUserController.php, delete RedisSCMCConsoleUser::find(), in another form to get a list of tenant IDs.
// HTTP 请求，获取开通有效服务的租户ID列表
$cmcApiGroupIds = CmcApiGroupService::httpGetGroupIds();
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
exit;
3.6807098388672MB //RedisCmcConsoleUser::find() 执行后
3.1283721923828MB //CmcApiGroupService::httpGetGroupIds() 执行后
31.42064666748MB //RedisCmcConsoleUser::find() 执行后
3.1799774169922MB //CmcApiGroupService::httpGetGroupIds() 执行后
27. Analyze and optimize the first command line script, \console\controllers\CmcConsoleUserController.php, delete RedisSCMCConsoleUser::Find(), in another form to obtain the tenant ID list, the actual amount of memory used is reduced by about 1.72 MB, and the total amount of memory allocated by the system is reduced by about 21.17 MB.
/**
* 同步框架服务控制台的用户模型(Http)至框架服务控制台的用户模型(Redis)
*
* @throws ServerErrorHttpException
* @throws HttpException 如果登录超时
*/
public function actionSync()
{
// HTTP 请求，获取开通有效服务的租户ID列表
$cmcApiGroupIds = CmcApiGroupService::httpGetGroupIds();
/* 判断 $cmcApiGroupIds 是否为空，如果为空，则成功退出 */
if (empty($cmcApiGroupIds)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($cmcApiGroupIds as $cmcApiGroupId) {
$isLockExist = RedisCmcConsoleUser::isLockExist($cmcApiGroupId);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// HTTP请求，获取租户ID下的用户列表
$httpGetUserListData = [
'groupId' => $cmcApiGroupId,
'loginId' => '',
'loginTid' => '',
];
$getUserListData = CmcConsoleUserService::httpGetUserList($httpGetUserListData);
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 销毁变量
unset($getUserListData['list']);
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $cmcApiGroupId])->indexBy('id')->asArray()->all();
// 使用键名比较计算数组的差集，如果不为空，则删除出现在 Redis 中但是未出现在 Http 中的记录
$redisArrayDiffItems = array_diff_key($redisCmcConsoleUserItems, $httpCmcConsoleUserItems);
// 销毁变量
unset($redisCmcConsoleUserItems);
if (!empty($redisArrayDiffItems)) {
$redisArrayDiffIds = array_keys($redisArrayDiffItems);
// 销毁变量
unset($redisArrayDiffItems);
if (RedisCmcConsoleUser::deleteAllByIds($cmcApiGroupId, $redisArrayDiffIds) === false) {
continue;
}
}
// 遍历框架服务控制台的用户模型(Http)，判断在 Redis 中是否存在，如果不存在，则插入，如果存在且更新时间不相等，则更新
foreach ($httpCmcConsoleUserItems as $httpCmcConsoleUserItemValue) {
$redisCmcConsoleUserItem = RedisCmcConsoleUser::find()->where(['id' => $httpCmcConsoleUserItemValue['id']])->one();
if (!isset($redisCmcConsoleUserItem)) {
$redisCmcConsoleUser = new RedisCmcConsoleUser();
$attributes = $this->getAttributes($getUserListData['group_info']['group_id'], $httpCmcConsoleUserItemValue);
$redisCmcConsoleUser->attributes = $attributes;
$redisCmcConsoleUser->insert();
} else {
if ($httpCmcConsoleUserItemValue['update_time'] != $redisCmcConsoleUserItem['update_time']) {
$attributes = $this->getAttributes($getUserListData['group_info']['group_id'], $httpCmcConsoleUserItemValue);
$redisCmcConsoleUserItem->attributes = $attributes;
$redisCmcConsoleUserItem->save();
}
}
}
}
// 延缓执行 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
}
6.3685455322266MB //优化前
4.6499099731445MB //优化后
31.42064666748MB //优化前
10.250221252441MB //优化后
28. Analysis and optimization of the second command line script, \Console\Controllers\ConfigColumUserController.php, delete RedisSCMCConsoleUser::Find(), in another form to obtain the tenant ID list, the actual amount of memory used increases by about 0.37 MB, and the total amount of memory allocated by the system is reduced by about 19.39 MB.
/**
* 同步框架服务控制台的用户模型(Redis)至栏目人员配置模型(MySQL)
*
*/
public function actionSync()
{
// HTTP 请求，获取开通有效服务的租户ID列表
$cmcApiGroupIds = CmcApiGroupService::httpGetGroupIds();
/* 判断 $cmcApiGroupIds 是否为空，如果为空，则成功退出 */
if (empty($cmcApiGroupIds)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($cmcApiGroupIds as $cmcApiGroupId) {
$isLockExist = RedisCmcConsoleUser::isLockExist($cmcApiGroupId);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $cmcApiGroupId])->indexBy('id')->asArray()->all();
// 基于租户ID查询栏目人员配置模型(MySQL)(以 用户ID 索引结果集)
$configColumnUserItems = ConfigColumnUser::find()->where(['group_id' => $cmcApiGroupId])->isDeletedNo()->indexBy('user_id')->asArray()->all();
// 使用键名比较计算数组的差集，如果不为空，则删除 (软删除) 出现在 栏目人员配置模型(MySQL) 中但是未出现在 框架服务控制台的用户模型(Redis) 中的记录
$diffItems = array_diff_key($configColumnUserItems, $redisCmcConsoleUserItems);
// 销毁变量
unset($redisCmcConsoleUserItems, $configColumnUserItems);
if (!empty($diffItems)) {
// 基于租户ID、用户ID查询栏目人员配置模型(MySQL)(待删除)
$toBeDeletedConfigColumnUserItems = ConfigColumnUser::find()->where([
'and',
['group_id' => $cmcApiGroupId],
['in', 'user_id', $diffItems],
])->isDeletedNo()->all();
foreach ($toBeDeletedConfigColumnUserItems as $toBeDeletedConfigColumnUserItem) {
/* @var $toBeDeletedConfigColumnUserItem ConfigColumnUser */
$toBeDeletedConfigColumnUserItem->softDelete();
}
continue;
}
}
// 延缓执行 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyNoSleepTime']);
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
}
4.9719772338867MB //优化前
5.3451461791992MB //优化后
31.416854858398MB //优化前
12.024124145508MB //优化后
29. Upgrade to the development environment, the CPU of the Docker container: 92%, memory: 435MB, the CPU has increased (92% – 0.04%) = about 91.96%, the memory has increased (435MB – 283MB) = about 152MB. As shown in Figure 17, Figure 18
Figure 17
18
30. Analyze the specific reasons. The HTTP client extension based on Yii 2 must be added to Linux: setData([]), otherwise the response is 400, as shown in Figure 19, Figure 20
Figure 19
![分析具体原因，基于 Yii 2 的 HTTP 客户端扩展在 Linux 中必须添加：setData([])，否则响应 400](https://www.shuijingwanwq.com/wp-content/uploads/2019/07/20.png)
Figure 20
Figure 21
Figure 22
32. Delay execution for 60 seconds, and decide to stop using sleep(), because during sleep(), the memory has been occupied, which actually prolongs the memory usage time. Although it may reduce the CPU usage percentage.
4.6499099731445MB //延缓执行 60 秒前
4.6499099731445MB //延缓执行 60 秒后
10.250221252441MB //延缓执行 60 秒前
10.250221252441MB //延缓执行 60 秒后
33. Analysis and optimization of the first command line script, \Console\Controllers\CmcConsoleUserController.php, set the cache of the synchronization identifier, if it does not exist, then synchronize, keep the data in the cache 60 seconds. If it exists, it is not synchronized (Note: within 60 seconds after the synchronization is successful, the memory usage will be very small, see the memory usage after the second optimization).
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/01/09
* Time: 13:55
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\services\CmcApiGroupService;
use console\services\CmcConsoleUserService;
use yii\console\Controller;
use yii\console\ExitCode;
use yii\helpers\ArrayHelper;
use yii\web\HttpException;
use yii\web\ServerErrorHttpException;
/**
* 框架服务控制台的用户
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class CmcConsoleUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Http)至框架服务控制台的用户模型(Redis)
*
* @throws ServerErrorHttpException
* @throws HttpException 如果登录超时
*/
public function actionSync()
{
// 设置同步标识、租户ID列表的缓存键
$redisCache = Yii::$app->redisCache;
$redisCacheIdentityKey = 'cmc_console_user_sync';
$redisCacheGroupIdsKey = 'cmc_api_group_ids';
// 从缓存中取回同步标识、租户ID列表
$redisCacheIdentityData = $redisCache[$redisCacheIdentityKey];
$redisCacheGroupIdsData = $redisCache[$redisCacheGroupIdsKey];
if ($redisCacheIdentityData === false) {
if ($redisCacheGroupIdsData === false) {
// HTTP 请求，获取开通有效服务的租户ID列表
$cmcApiGroupIds = CmcApiGroupService::httpGetGroupIds();
// 将租户ID列表存放到缓存供下次使用，将数据在缓存中保留 60 * 60 秒
$redisCache->set($redisCacheGroupIdsKey, $cmcApiGroupIds, Yii::$app->params['cmcConsoleUser']['isEmptyYesSleepTime']);
} else {
$cmcApiGroupIds = $redisCacheGroupIdsData;
}
/* 判断 $cmcApiGroupIds 是否为空，如果为空，则成功退出 */
if (empty($cmcApiGroupIds)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($cmcApiGroupIds as $cmcApiGroupId) {
$isLockExist = RedisCmcConsoleUser::isLockExist($cmcApiGroupId);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// HTTP请求，获取租户ID下的用户列表
$httpGetUserListData = [
'groupId' => $cmcApiGroupId,
'loginId' => '',
'loginTid' => '',
];
$getUserListData = CmcConsoleUserService::httpGetUserList($httpGetUserListData);
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 销毁变量
unset($getUserListData['list']);
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $cmcApiGroupId])->indexBy('id')->asArray()->all();
// 使用键名比较计算数组的差集，如果不为空，则删除出现在 Redis 中但是未出现在 Http 中的记录
$redisArrayDiffItems = array_diff_key($redisCmcConsoleUserItems, $httpCmcConsoleUserItems);
// 销毁变量
unset($redisCmcConsoleUserItems);
if (!empty($redisArrayDiffItems)) {
$redisArrayDiffIds = array_keys($redisArrayDiffItems);
// 销毁变量
unset($redisArrayDiffItems);
if (RedisCmcConsoleUser::deleteAllByIds($cmcApiGroupId, $redisArrayDiffIds) === false) {
continue;
}
}
// 遍历框架服务控制台的用户模型(Http)，判断在 Redis 中是否存在，如果不存在，则插入，如果存在且更新时间不相等，则更新
foreach ($httpCmcConsoleUserItems as $httpCmcConsoleUserItemValue) {
$redisCmcConsoleUserItem = RedisCmcConsoleUser::find()->where(['id' => $httpCmcConsoleUserItemValue['id']])->one();
if (!isset($redisCmcConsoleUserItem)) {
$redisCmcConsoleUser = new RedisCmcConsoleUser();
$attributes = $this->getAttributes($getUserListData['group_info']['group_id'],
$httpCmcConsoleUserItemValue);
$redisCmcConsoleUser->attributes = $attributes;
$redisCmcConsoleUser->insert();
} else {
if ($httpCmcConsoleUserItemValue['update_time'] != $redisCmcConsoleUserItem['update_time']) {
$attributes = $this->getAttributes($getUserListData['group_info']['group_id'],
$httpCmcConsoleUserItemValue);
$redisCmcConsoleUserItem->attributes = $attributes;
$redisCmcConsoleUserItem->save();
}
}
}
}
// 延缓执行 60 秒
// sleep(Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
// 将同步标识存放到缓存供下次使用，将数据在缓存中保留 60 秒
$redisCache->set($redisCacheIdentityKey, $redisCacheIdentityKey, Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
} else {
return ExitCode::OK;
}
}
/**
* 获取属性列表
* @param string $groupId 租户ID
* @param array $httpCmcConsoleUser 框架服务控制台的用户模型(Http)
*
* @return array
*/
private function getAttributes($groupId, $httpCmcConsoleUser) {
return [
'id' => $httpCmcConsoleUser['id'],
'group_id' => $groupId,
'login_name' => $httpCmcConsoleUser['login_name'],
'user_token' => $httpCmcConsoleUser['user_token'],
'user_nick' => $httpCmcConsoleUser['user_nick'],
'user_pic' => $httpCmcConsoleUser['user_pic'],
'user_mobile' => $httpCmcConsoleUser['user_mobile'] ? $httpCmcConsoleUser['user_mobile'] : '',
'user_email' => $httpCmcConsoleUser['user_email'] ? $httpCmcConsoleUser['user_email'] : '',
'user_sex' => $httpCmcConsoleUser['user_sex'],
'user_type' => $httpCmcConsoleUser['user_type'],
'user_birthday' => $httpCmcConsoleUser['user_birthday'],
'user_chat_id' => $httpCmcConsoleUser['user_chat_id'] ? $httpCmcConsoleUser['user_chat_id'] : '',
'is_open' => $httpCmcConsoleUser['is_open'],
'add_time' => $httpCmcConsoleUser['add_time'],
'update_time' => $httpCmcConsoleUser['update_time'],
];
}
}
4.6499099731445MB //优化前
4.6722030639648MB //优化后(有 HTTP 请求，有同步)
2.8900680541992MB //优化后(无 HTTP 请求，无同步)
4.6064605712891MB //优化后(无 HTTP 请求，有同步)
10.250221252441MB //优化前
10.41194152832MB //优化后(有 HTTP 请求，有同步)
2.9416198730469MB //优化后(无 HTTP 请求，无同步)
10.346199035645MB //优化后(无 HTTP 请求，有同步)
34. Analysis and optimization of the second command line script, \Console\Controllers\ConfigColumUserController.php, set the cache of the synchronization ID, if it does not exist, then synchronize, keep the data in the cache 60 seconds. If it exists, it is not synchronized (Note: within 60 seconds after the synchronization is successful, the memory usage will be very small, see the memory usage after the second optimization).
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/03/01
* Time: 13:17
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\models\ConfigColumnUser;
use console\services\CmcApiGroupService;
use console\services\CmcConsoleUserService;
use yii\console\Controller;
use yii\console\ExitCode;
use yii\helpers\ArrayHelper;
use yii\web\HttpException;
use yii\web\ServerErrorHttpException;
/**
* 栏目人员配置
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class ConfigColumnUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Http)至栏目人员配置模型(MySQL)
*
* @throws ServerErrorHttpException
* @throws HttpException 如果登录超时
*/
public function actionSync()
{
// 设置同步标识、租户ID列表的缓存键
$redisCache = Yii::$app->redisCache;
$redisCacheIdentityKey = 'config_column_user_sync';
$redisCacheGroupIdsKey = 'cmc_api_group_ids';
// 从缓存中取回同步标识、租户ID列表
$redisCacheIdentityData = $redisCache[$redisCacheIdentityKey];
$redisCacheGroupIdsData = $redisCache[$redisCacheGroupIdsKey];
if ($redisCacheIdentityData === false) {
if ($redisCacheGroupIdsData === false) {
// HTTP 请求，获取开通有效服务的租户ID列表
// $cmcApiGroupIds = CmcApiGroupService::httpGetGroupIds();
// 将租户ID列表存放到缓存供下次使用，将数据在缓存中保留 60 * 60 秒
// $redisCache->set($redisCacheGroupIdsKey, $cmcApiGroupIds, Yii::$app->params['cmcConsoleUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
} else {
$cmcApiGroupIds = $redisCacheGroupIdsData;
}
/* 判断 $cmcApiGroupIds 是否为空，如果为空，则成功退出 */
if (empty($cmcApiGroupIds)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
foreach ($cmcApiGroupIds as $cmcApiGroupId) {
// HTTP请求，获取租户ID下的用户列表
$httpGetUserListData = [
'groupId' => $cmcApiGroupId,
'loginId' => '',
'loginTid' => '',
];
$getUserListData = CmcConsoleUserService::httpGetUserList($httpGetUserListData);
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 销毁变量
unset($getUserListData);
// 基于租户ID查询栏目人员配置模型(MySQL)(以 用户ID 索引结果集)
$configColumnUserItems = ConfigColumnUser::find()->where(['group_id' => $cmcApiGroupId])->isDeletedNo()->indexBy('user_id')->asArray()->all();
// 使用键名比较计算数组的差集，如果不为空，则删除 (软删除) 出现在 栏目人员配置模型(MySQL) 中但是未出现在 框架服务控制台的用户模型(Http) 中的记录
$diffItems = array_diff_key($configColumnUserItems, $httpCmcConsoleUserItems);
// 销毁变量
unset($httpCmcConsoleUserItems, $configColumnUserItems);
if (!empty($diffItems)) {
// 基于租户ID、用户ID查询栏目人员配置模型(MySQL)(待删除)
$toBeDeletedConfigColumnUserItems = ConfigColumnUser::find()->where([
'and',
['group_id' => $cmcApiGroupId],
['in', 'user_id', $diffItems],
])->isDeletedNo()->all();
foreach ($toBeDeletedConfigColumnUserItems as $toBeDeletedConfigColumnUserItem) {
/* @var $toBeDeletedConfigColumnUserItem ConfigColumnUser */
$toBeDeletedConfigColumnUserItem->softDelete();
}
continue;
}
}
// 延缓执行 60 秒
// sleep(Yii::$app->params['configColumnUser']['isEmptyNoSleepTime']);
// 将同步标识存放到缓存供下次使用，将数据在缓存中保留 60 秒
$redisCache->set($redisCacheIdentityKey, $redisCacheIdentityKey, Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
} else {
return ExitCode::OK;
}
}
}
5.3451461791992MB //优化前
5.5944900512695MB //优化后(有 HTTP 请求，有同步)
2.8861389160156MB //优化后(无 HTTP 请求，无同步)
5.3590927124023MB //优化后(无 HTTP 请求，有同步)
12.024124145508MB //优化前
11.753318786621MB //优化后(有 HTTP 请求，有同步)
2.9376907348633MB //优化后(无 HTTP 请求，无同步)
11.67618560791MB //优化后(无 HTTP 请求，有同步)
35. Upgrade to the development environment, the CPU of the Docker container: 16%, memory: 486MB, the CPU has increased (16% – 0.04%) = about 15.96%, the memory has increased (486MB – 341MB) = about 140MB, it is proved that within 60 seconds after the synchronization is successful, avoid synchronization again. Optimized schemes are not feasible (not as expected). Figure 23, Figure 24
Figure 23
Figure 24
36. The difference between careful comparison and step 31 is that each time the command line is almost finished, Sleep() is used, and it is decided to use sleep() within 60 seconds after the synchronization is successful. After the synchronization is successful, 60 In seconds, the command line will only be executed one more time (and the execution time length is greater than: 60 seconds) and not multiple times. Looking at the Supervisord running status, it is found that the command line is very short and frequent, especially the second command line. as shown in Figure 25
Figure 25
if ($redisCacheIdentityData === false) {
} else {
// 延缓执行 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
return ExitCode::OK;
}
if ($redisCacheIdentityData === false) {
} else {
// 延缓执行 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyNoSleepTime']);
return ExitCode::OK;
}
37. Upgrade to the development environment, the CPU of the Docker container: 0.04%, memory: 283MB, the CPU is unchanged, the memory is reduced (341MB – 283MB) = about 60MB. Avoid sync again within 60 seconds after the synchronization is successful. Optimized solutions are feasible. Figure 26, Figure 27
Figure 26
Figure 27
38. Decide to reduce the execution frequency of synchronization again, and avoid synchronization again within 5*60 seconds after the synchronization is successful. Looking at the supervisord running status, it is found that the command line has been executed for a long time and very low. Upgrading to the development environment, the monitoring data of the Docker container is unchanged. It is proved that when the execution frequency reaches a certain critical value, it decreases again, and the meaning of optimization is not much, at most, the average value of the CPU is lower (theoretically). as shown in Figure 28
Figure 28
39. Avoid in the production environment, when the number of tenants is too many, synchronize the users under all tenants at one time, which leads to too large memory usage and too long running time, and decides to synchronize only one user under one tenant at a time.
40. Analysis and optimization of the first command line script, \console\controllers\cmcConsoleUserController.php, when traversing the tenant ID list, break ends the current foreach The execution of the structure, that is, every run of the command line, only the user list under the tenant is synchronized. In the case of HTTP requests and synchronization, the actual amount of memory used increases by about 1.54 MB, and the total amount of memory allocated by the system decreases by about 0.75 MB.
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/01/09
* Time: 13:55
*/
namespace console\controllers;
use Yii;
use console\models\redis\cmc_console\User as RedisCmcConsoleUser;
use console\services\CmcApiGroupService;
use console\services\CmcConsoleUserService;
use yii\base\InvalidArgumentException;
use yii\console\Controller;
use yii\console\ExitCode;
use yii\helpers\ArrayHelper;
use yii\web\HttpException;
use yii\web\ServerErrorHttpException;
/**
* 框架服务控制台的用户
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class CmcConsoleUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Http)至框架服务控制台的用户模型(Redis)
*
* @throws ServerErrorHttpException
* @throws HttpException 如果登录超时
* @throws InvalidArgumentException if the $direction or $sortFlag parameters do not have
*/
public function actionSync()
{
// 设置同步标识的缓存键
$redisCache = Yii::$app->redisCache;
$redisCacheIdentityKey = 'cmc_console_user_sync';
// 从缓存中取回同步标识
$redisCacheIdentityData = $redisCache[$redisCacheIdentityKey];
if ($redisCacheIdentityData === false) {
// HTTP 请求，获取开通有效服务的租户ID列表
$httpCmcApiGroupIds = CmcApiGroupService::httpGetGroupIds();
/* 判断 $httpCmcApiGroupIds 是否为空，如果为空，则成功退出 */
if (empty($httpCmcApiGroupIds)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
// 设置租户ID列表的缓存键
$redisCacheGroupIdsKey = 'cmc_api_group_ids';
// 从缓存中取回租户ID列表
$redisCacheGroupIdsData = $redisCache[$redisCacheGroupIdsKey];
// 是否设置租户ID列表的缓存，默认：否
$isSetRedisCacheGroupIds = false;
if ($redisCacheGroupIdsData === false) {
$cmcApiGroupIds = [];
foreach ($httpCmcApiGroupIds as $httpCmcApiGroupId) {
$cmcApiGroupIds[] = [
'group_id' => $httpCmcApiGroupId,
'cmc_console_user_last_synced_at' => 0, //上次同步时间
'config_column_user_last_synced_at' => 0, //上次同步时间
];
}
// 是否设置租户ID列表的缓存：是
$isSetRedisCacheGroupIds = true;
} else {
// 获取 group_id 值列表
$redisCacheGroupIds = ArrayHelper::getColumn($redisCacheGroupIdsData, 'group_id');
$cmcApiGroupIds = $redisCacheGroupIdsData;
foreach ($httpCmcApiGroupIds as $httpCmcApiGroupId) {
if (!in_array($httpCmcApiGroupId, $redisCacheGroupIds)) {
$cmcApiGroupIds[] = [
'group_id' => $httpCmcApiGroupId,
'cmc_console_user_last_synced_at' => 0, //上次同步时间
'config_column_user_last_synced_at' => 0, //上次同步时间
];
// 是否设置租户ID列表的缓存：是
$isSetRedisCacheGroupIds = true;
}
}
}
// 判断是否设置租户ID列表的缓存
if ($isSetRedisCacheGroupIds) {
// 将 $cmcApiGroupIds 存放到缓存供下次使用，将数据在缓存中永久保留
$redisCache->set($redisCacheGroupIdsKey, $cmcApiGroupIds);
}
// 基于上次同步时间顺序排列，赋值给：$sortCmcApiGroupIds
$sortCmcApiGroupIds = $cmcApiGroupIds;
ArrayHelper::multisort($sortCmcApiGroupIds, 'cmc_console_user_last_synced_at', SORT_ASC);
foreach ($sortCmcApiGroupIds as $sortCmcApiGroupId) {
$isLockExist = RedisCmcConsoleUser::isLockExist($sortCmcApiGroupId['group_id']);
// 返回 true，表示锁定存在，即已经被其他客户端锁定
if ($isLockExist === true) {
continue;
}
// HTTP请求，获取租户ID下的用户列表
$httpGetUserListData = [
'groupId' => $sortCmcApiGroupId['group_id'],
'loginId' => '',
'loginTid' => '',
];
$getUserListData = CmcConsoleUserService::httpGetUserList($httpGetUserListData);
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 销毁变量
unset($getUserListData['list']);
// 基于租户ID查询框架服务控制台的用户模型(Redis)(以 ID 索引结果集)
$redisCmcConsoleUserItems = RedisCmcConsoleUser::find()->where(['group_id' => $sortCmcApiGroupId['group_id']])->indexBy('id')->asArray()->all();
// 使用键名比较计算数组的差集，如果不为空，则删除出现在 Redis 中但是未出现在 Http 中的记录
$redisArrayDiffItems = array_diff_key($redisCmcConsoleUserItems, $httpCmcConsoleUserItems);
// 销毁变量
unset($redisCmcConsoleUserItems);
if (!empty($redisArrayDiffItems)) {
$redisArrayDiffIds = array_keys($redisArrayDiffItems);
// 销毁变量
unset($redisArrayDiffItems);
if (RedisCmcConsoleUser::deleteAllByIds($sortCmcApiGroupId['group_id'], $redisArrayDiffIds) === false) {
continue;
}
}
// 遍历框架服务控制台的用户模型(Http)，判断在 Redis 中是否存在，如果不存在，则插入，如果存在且更新时间不相等，则更新
foreach ($httpCmcConsoleUserItems as $httpCmcConsoleUserItemValue) {
$redisCmcConsoleUserItem = RedisCmcConsoleUser::find()->where(['id' => $httpCmcConsoleUserItemValue['id']])->one();
if (!isset($redisCmcConsoleUserItem)) {
$redisCmcConsoleUser = new RedisCmcConsoleUser();
$attributes = $this->getAttributes($getUserListData['group_info']['group_id'],
$httpCmcConsoleUserItemValue);
$redisCmcConsoleUser->attributes = $attributes;
$redisCmcConsoleUser->insert();
} else {
if ($httpCmcConsoleUserItemValue['update_time'] != $redisCmcConsoleUserItem['update_time']) {
$attributes = $this->getAttributes($getUserListData['group_info']['group_id'],
$httpCmcConsoleUserItemValue);
$redisCmcConsoleUserItem->attributes = $attributes;
$redisCmcConsoleUserItem->save();
}
}
}
// 从缓存中取回租户ID列表
$cmcApiGroupIds = $redisCache[$redisCacheGroupIdsKey];
// 设置当前租户的上次同步时间
foreach ($cmcApiGroupIds as $cmcApiGroupIdKey => $cmcApiGroupId) {
if ($cmcApiGroupId['group_id'] == $sortCmcApiGroupId['group_id']) {
$cmcApiGroupIds[$cmcApiGroupIdKey]['cmc_console_user_last_synced_at'] = time();
break;
}
}
// 将 $cmcApiGroupIds 存放到缓存供下次使用，将数据在缓存中永久保留
$redisCache->set($redisCacheGroupIdsKey, $cmcApiGroupIds);
// break 结束当前 foreach 结构的执行，即命令行的每一次运行，仅同步成功一个租户下的用户列表
break;
}
// 延缓执行 60 秒
// sleep(Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
// 将同步标识存放到缓存供下次使用，将数据在缓存中保留 60 秒
$redisCache->set($redisCacheIdentityKey, $redisCacheIdentityKey, Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/cmc-console-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
} else {
// 延缓执行 60 秒
sleep(Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
return ExitCode::OK;
}
}
/**
* 获取属性列表
* @param string $groupId 租户ID
* @param array $httpCmcConsoleUser 框架服务控制台的用户模型(Http)
*
* @return array
*/
private function getAttributes($groupId, $httpCmcConsoleUser) {
return [
'id' => $httpCmcConsoleUser['id'],
'group_id' => $groupId,
'login_name' => $httpCmcConsoleUser['login_name'],
'user_token' => $httpCmcConsoleUser['user_token'],
'user_nick' => $httpCmcConsoleUser['user_nick'],
'user_pic' => $httpCmcConsoleUser['user_pic'],
'user_mobile' => $httpCmcConsoleUser['user_mobile'] ? $httpCmcConsoleUser['user_mobile'] : '',
'user_email' => $httpCmcConsoleUser['user_email'] ? $httpCmcConsoleUser['user_email'] : '',
'user_sex' => $httpCmcConsoleUser['user_sex'],
'user_type' => $httpCmcConsoleUser['user_type'],
'user_birthday' => $httpCmcConsoleUser['user_birthday'],
'user_chat_id' => $httpCmcConsoleUser['user_chat_id'] ? $httpCmcConsoleUser['user_chat_id'] : '',
'is_open' => $httpCmcConsoleUser['is_open'],
'add_time' => $httpCmcConsoleUser['add_time'],
'update_time' => $httpCmcConsoleUser['update_time'],
];
}
}
4.6722030639648MB //优化前(有 HTTP 请求，有同步)
2.8900680541992MB //优化前(无 HTTP 请求，无同步)
4.6064605712891MB //优化前(无 HTTP 请求，有同步)
6.2168731689453MB //优化后(有 HTTP 请求，有同步)
2.8925552368164MB //优化后(无 HTTP 请求，无同步)
10.41194152832MB //优化前(有 HTTP 请求，有同步)
2.9416198730469MB //优化前(无 HTTP 请求，无同步)
10.346199035645MB //优化前(无 HTTP 请求，有同步)
9.6655578613281MB //优化后(有 HTTP 请求，有同步)
2.9470977783203MB //优化后(无 HTTP 请求，无同步)
41. Analysis and optimization of the second command line script, \console\controllers\configColumnuserController.php, when traversing the tenant ID list, break ends the current foreach The execution of the structure, that is, every run of the command line, only the user list under the tenant is synchronized. In the case of no HTTP request and synchronization, the actual amount of memory used is reduced by about 0.45 MB, and the total amount of memory allocated by the system has not changed.
<?php
/**
* Created by PhpStorm.
* User: Qiang Wang
* Date: 2019/03/01
* Time: 13:17
*/
namespace console\controllers;
use Yii;
use console\models\ConfigColumnUser;
use console\services\CmcConsoleUserService;
use yii\console\Controller;
use yii\console\ExitCode;
use yii\helpers\ArrayHelper;
use yii\web\HttpException;
use yii\web\ServerErrorHttpException;
/**
* 栏目人员配置
*
* @author Qiang Wang <shuijingwanwq@163.com>
* @since 1.0
*/
class ConfigColumnUserController extends Controller
{
/**
* 同步框架服务控制台的用户模型(Http)至栏目人员配置模型(MySQL)
*
* @throws ServerErrorHttpException
* @throws HttpException 如果登录超时
*/
public function actionSync()
{
// 设置同步标识的缓存键
$redisCache = Yii::$app->redisCache;
$redisCacheIdentityKey = 'config_column_user_sync';
// 从缓存中取回同步标识
$redisCacheIdentityData = $redisCache[$redisCacheIdentityKey];
if ($redisCacheIdentityData === false) {
// 设置租户ID列表的缓存键
$redisCacheGroupIdsKey = 'cmc_api_group_ids';
// 从缓存中取回租户ID列表
$redisCacheGroupIdsData = $redisCache[$redisCacheGroupIdsKey];
if ($redisCacheGroupIdsData === false) {
return ExitCode::OK;
} else {
$cmcApiGroupIds = $redisCacheGroupIdsData;
}
/* 判断 $cmcApiGroupIds 是否为空，如果为空，则成功退出 */
if (empty($cmcApiGroupIds)) {
// 延缓执行 60 * 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyYesSleepTime']);
return ExitCode::OK;
}
// 基于上次同步时间顺序排列，赋值给：$sortCmcApiGroupIds
$sortCmcApiGroupIds = $cmcApiGroupIds;
ArrayHelper::multisort($sortCmcApiGroupIds, 'config_column_user_last_synced_at', SORT_ASC);
foreach ($sortCmcApiGroupIds as $sortCmcApiGroupId) {
// HTTP请求，获取租户ID下的用户列表
$httpGetUserListData = [
'groupId' => $sortCmcApiGroupId['group_id'],
'loginId' => '',
'loginTid' => '',
];
$getUserListData = CmcConsoleUserService::httpGetUserList($httpGetUserListData);
// 框架服务控制台的用户模型(Http)，重建数组索引(以 ID 索引结果集)
$httpCmcConsoleUserItems = ArrayHelper::index($getUserListData['list'], 'id');
// 销毁变量
unset($getUserListData);
// 基于租户ID查询栏目人员配置模型(MySQL)(以 用户ID 索引结果集)
$configColumnUserItems = ConfigColumnUser::find()->where(['group_id' => $sortCmcApiGroupId['group_id']])->isDeletedNo()->indexBy('user_id')->asArray()->all();
// 使用键名比较计算数组的差集，如果不为空，则删除 (软删除) 出现在 栏目人员配置模型(MySQL) 中但是未出现在 框架服务控制台的用户模型(Http) 中的记录
$diffItems = array_diff_key($configColumnUserItems, $httpCmcConsoleUserItems);
// 销毁变量
unset($httpCmcConsoleUserItems, $configColumnUserItems);
if (!empty($diffItems)) {
// 基于租户ID、用户ID查询栏目人员配置模型(MySQL)(待删除)
$toBeDeletedConfigColumnUserItems = ConfigColumnUser::find()->where([
'and',
['group_id' => $sortCmcApiGroupId['group_id']],
['in', 'user_id', $diffItems],
])->isDeletedNo()->all();
foreach ($toBeDeletedConfigColumnUserItems as $toBeDeletedConfigColumnUserItem) {
/* @var $toBeDeletedConfigColumnUserItem ConfigColumnUser */
$toBeDeletedConfigColumnUserItem->softDelete();
}
}
// 从缓存中取回租户ID列表
$cmcApiGroupIds = $redisCache[$redisCacheGroupIdsKey];
// 设置当前租户的上次同步时间
foreach ($cmcApiGroupIds as $cmcApiGroupIdKey => $cmcApiGroupId) {
if ($cmcApiGroupId['group_id'] == $sortCmcApiGroupId['group_id']) {
$cmcApiGroupIds[$cmcApiGroupIdKey]['config_column_user_last_synced_at'] = time();
break;
}
}
// 将 $cmcApiGroupIds 存放到缓存供下次使用，将数据在缓存中永久保留
$redisCache->set($redisCacheGroupIdsKey, $cmcApiGroupIds);
// break 结束当前 foreach 结构的执行，即命令行的每一次运行，仅同步成功一个租户下的用户列表
break;
}
// 延缓执行 60 秒
// sleep(Yii::$app->params['configColumnUser']['isEmptyNoSleepTime']);
// 将同步标识存放到缓存供下次使用，将数据在缓存中保留 60 秒
$redisCache->set($redisCacheIdentityKey, $redisCacheIdentityKey, Yii::$app->params['cmcConsoleUser']['isEmptyNoSleepTime']);
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-usage-' . time() . '.txt', memory_get_usage() / 1024 / 1024 . 'MB');
// file_put_contents('E:/wwwroot/pcs-api/console/runtime/logs/config-column-user-sync-memory-get-peak-usage-' . time() . '.txt', memory_get_peak_usage() / 1024 / 1024 . 'MB');
return ExitCode::OK;
} else {
// 延缓执行 60 秒
sleep(Yii::$app->params['configColumnUser']['isEmptyNoSleepTime']);
return ExitCode::OK;
}
}
}
5.5944900512695MB //优化前(有 HTTP 请求，有同步)
2.8861389160156MB //优化前(无 HTTP 请求，无同步)
5.3590927124023MB //优化前(无 HTTP 请求，有同步)
4.9042816162109MB //优化后(无 HTTP 请求，有同步)
2.8837127685547MB //优化后(无 HTTP 请求，无同步)
11.753318786621MB //优化前(有 HTTP 请求，有同步)
2.9376907348633MB //优化前(无 HTTP 请求，无同步)
11.67618560791MB //优化前(无 HTTP 请求，有同步)
10.693077087402MB //优化后(无 HTTP 请求，有同步)
2.9382553100586MB //优化后(无 HTTP 请求，无同步)
42. Upgrade to the development environment, the CPU of the Docker container: 0.07%, memory: 283MB, the CPU has increased by (0.07% – 0.04%) = about 0.03%, the memory is reduced (283MB – 276MB) = about 7MB. Prove that when the tenant ID list is traversed, break ends the execution of the current foreach structure, that is, each run of the command line, only the user list under the tenant is successfully synchronized. Optimized solutions are not feasible and meaningless. As shown in Figure 29, Figure 30
43. Check the instance monitoring situation of mysql, memory usage: 6166MB, the reference value is not large, because it shares the same database with other products, as shown in Figure 31
44. Check the instance monitoring situation of Redis, the used capacity: 75MB, as shown in Figure 32
45. The implementation scheme of the reduced memory usage is summarized as follows:
(1) The increase in the amount of data will lead to the continuous increase in memory. When the amount of data reaches a certain magnitude in the later stage, it is necessary to support cluster deployment
(2) The CPU of the Docker container: 5.4%, memory: 371MB, the memory is reduced (541MB – 371MB) = about 170MB, which is based on unset() The implemented scheme is feasible
(3) CPU of Docker container: 5.4%, memory: 280MB, memory reduced (371MB – 371MB) = 0MB Left and right, proof that the execution time of the command line is reduced is meaningless for the use of memory optimization.
(4) CPU of Docker container: 0.04%, memory: 283MB, memory reduced (371MB – 283MB) = 90MB Left and right, the proof variable is only defined when used, obtaining data in an array, and calling the asarray() method before the query method to obtain the result of the php array form. Optimized solutions are feasible.
(5) CPU of Docker container: 92%, memory: 435MB, CPU increased (92% – 0.04%) = about 91.96%, memory increased (435MB – 283MB) = about 152MB. When the HTTP request responds to 400, when the command line throws an exception, both the CPU and memory increase a lot
(6) After solving the bug of response 400, upgrade to the development environment, the CPU of the Docker container: 0.04%, memory: 341MB, the CPU has no change, and the memory has increased (341MB – 283MB) = about 60MB. Delete RedisSCMCConsoleUser::find(), get a list of tenant IDs in another form (HTTP request), the actual amount of memory used increases by about 0.37 MB, and the total amount of memory allocated by the system is reduced by 19.39 MB around. Prove that the amount of memory actually used increases, the total amount of memory allocated by the system decreases, and the memory usage of Docker is increasing (mainly affected by the amount of memory actually used)
(7) The CPU of the Docker container: 16%, memory: 486MB, the CPU has increased (16% – 0.04%) = about 15.96%, the memory has increased (486MB – 341MB) = about 140MB, check the running status of supervisord, and find that when the execution time of the command line is very short and the frequency is very high, the CPU and memory are generally increased.
(8) The CPU of the Docker container: 0.04%, memory: 283MB, the CPU is unchanged, and the memory is reduced (341MB – 283MB) = about 60MB. Avoid sync again within 60 seconds after the synchronization is successful. Optimized solutions are feasible
(9) Decide to reduce the execution frequency of the synchronization again, and avoid synchronization again within 5 * 60 seconds after the synchronization is successful. When the execution frequency reaches a certain critical value, it decreases again, and the meaning of optimization is not much. At most, the average value of the CPU is lower (theoretically)
(10) When traversing the tenant ID list, break ends the execution of the current foreach structure, that is, every time the command line runs, only the user list under the tenant is successfully synchronized. After the Docker container is upgraded, the performance metrics have not changed. The meaning of optimization is not much, although the running time of the command line synchronization once is greatly reduced (the reason may be that the number of tenants in the development environment is small, the total data volume is not large, and the meaning of optimization is not reflected.
![执行命令：docker exec -i wshop-mysql mysql -uroot -proot wshop_store < wshop_store.sql ，报错：mysql: [Warning] Using a password on the command line interface can be insecure.。](https://www.shuijingwanwq.com/wp-content/uploads/2022/01/1-3-520x245.png)