在 Rancher 中升级容器时,出现数据库迁移重复执行,进而报错的排查分析

1、在 Rancher 中第一次升级容器时,出现数据库迁移重复执行,进而报错升级失败。如图1

图1

*** applying m210106_122427_add_columns_to_task_table

> add column article_type_id integer NOT NULL DEFAULT 0 COMMENT '文章类型ID' AFTER `type` to table {{%task}} ...Exception: SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'article_type_id'

The SQL being executed was: ALTER TABLE `cpa_task` ADD `article_type_id` int(11) NOT NULL DEFAULT 0 COMMENT '文章类型ID' AFTER `type` (/mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Schema.php:678)

#0 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1304): yii\db\Schema->convertException(Object(PDOException), 'ALTER TABLE `cp...')

#1 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1099): yii\db\Command->internalExecute('ALTER TABLE `cp...')

#2 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(377): yii\db\Command->execute()

#3 /mcloud/www/ccp_api/console/migrations/m210106_122427_add_columns_to_task_table.php(15): yii\db\Migration->addColumn('{{%task}}', 'article_type_id', Object(yii\db\mysql\ColumnSchemaBuilder))

#4 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(114): m210106_122427_add_columns_to_task_table->safeUp()

#5 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(744): yii\db\Migration->up()

#6 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(202): yii\console\controllers\BaseMigrateController->migrateUp('m210106_122427_...')

#7 [internal function]: yii\console\controllers\BaseMigrateController->actionUp(0)

#8 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)

#9 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Controller.php(181): yii\base\InlineAction->runWithParams(Array)

#10 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Controller.php(184): yii\base\Controller->runAction('', Array)

#11 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Module.php(534): yii\console\Controller->runAction('', Array)

#12 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(181): yii\base\Module->runAction('migrate', Array)

#13 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(148): yii\console\Application->runAction('migrate', Array)

#14 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Application.php(392): yii\console\Application->handleRequest(Object(yii\console\Request))

#15 /mcloud/www/ccp_api/yii(23): yii\base\Application->run()

#16 {main}

*** failed to apply m210106_122427_add_columns_to_task_table (time: 0.006s)


2、已经确定数据库结构并未被手动修改过。分析容器的健康检查。周期性向容器发出请求,以检测其健康状态。默认设置下,readiness和liveness使用相同的配置参数。对于应用初始化较长的容器,需要增加就绪检测时间。初步分析结论:由于数据库迁移执行时间过长,导致容器初始化时间过长,导致健康检查失败,再导致容器不断启动运行,数据库迁移重复执行。决定增加就绪检测时间。运行多少时间后开始检测:10秒 -> 100秒。如图2

图2

3、在初始部署时仍然报错:ReplicaSet “api-77dfcc7cfb” has timed out progressing.; Deployment does not have minimum availability.。如图3

图3

4、查看日志,仍然是数据库迁移重复执行的问题。查看 MySQL 中的表结构与迁移表中的记录,确定迁移记录:m210402_052030_add_columns_to_channel_app_source_table 在迁移表中不存在,但是 MySQL 中的表结构已经是此迁移记录执行后的结果。如图4

图4

m210402_052030_add_columns_to_channel_app_source_table

*** applying m201105_020453_add_article_read_count_column_to_channel_app_task_table

> add column article_read_count integer NOT NULL DEFAULT 0 COMMENT '文章的阅读数' AFTER `have_pub_number` to table {{%channel_app_task}} ...Exception: SQLSTATE[42S21]: Column already exists: 1060 Duplicate column name 'article_read_count'

The SQL being executed was: ALTER TABLE `cpa_channel_app_task` ADD `article_read_count` int(11) NOT NULL DEFAULT 0 COMMENT '文章的阅读数' AFTER `have_pub_number` (/mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Schema.php:678)

#0 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1304): yii\db\Schema->convertException(Object(PDOException), 'ALTER TABLE `cp...')

#1 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Command.php(1099): yii\db\Command->internalExecute('ALTER TABLE `cp...')

#2 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(377): yii\db\Command->execute()

#3 /mcloud/www/ccp_api/console/migrations/m201105_020453_add_article_read_count_column_to_channel_app_task_table.php(15): yii\db\Migration->addColumn('{{%channel_app_...', 'article_read_co...', Object(yii\db\mysql\ColumnSchemaBuilder))

#4 /mcloud/www/ccp_api/vendor/yiisoft/yii2/db/Migration.php(114): m201105_020453_add_article_read_count_column_to_channel_app_task_table->safeUp()

#5 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(744): yii\db\Migration->up()

#6 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/controllers/BaseMigrateController.php(202): yii\console\controllers\BaseMigrateController->migrateUp('m201105_020453_...')

#7 [internal function]: yii\console\controllers\BaseMigrateController->actionUp(0)

#8 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)

#9 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Controller.php(181): yii\base\InlineAction->runWithParams(Array)

#10 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Controller.php(184): yii\base\Controller->runAction('', Array)

#11 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Module.php(534): yii\console\Controller->runAction('', Array)

#12 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(181): yii\base\Module->runAction('migrate', Array)

#13 /mcloud/www/ccp_api/vendor/yiisoft/yii2/console/Application.php(148): yii\console\Application->runAction('migrate', Array)

#14 /mcloud/www/ccp_api/vendor/yiisoft/yii2/base/Application.php(392): yii\console\Application->handleRequest(Object(yii\console\Request))

#15 /mcloud/www/ccp_api/yii(23): yii\base\Application->run()

#16 {main}

*** failed to apply m201105_020453_add_article_read_count_column_to_channel_app_task_table (time: 0.006s)

0 from 27 migrations were applied.

Migration failed. The rest of the migrations are canceled.

5、运行多少时间后开始检测。:10秒 -> 100秒。如图5

图5

 

永夜