1. The current implementation principle of the theme editor is: when a configuration item in the editor changes, it requests the backend staging API to cache the data in Redis. When the user clicks the save button, it requests the backend save API, which reads the data from Redis and stores it in MySQL.
2. When a user quickly edits some configuration items, the theme editor’s page data might become inconsistent with the actual page. See Figure 1.
3. Checking the initiation order of network requests, the last request to the backend staging API is confirmed to occur before the request to the backend save API. The deduced conclusion is that when the backend save API is requested and the backend reads data from Redis, the last request to the backend staging API has not yet saved the data to Redis. At this point, the previous staged data is read. An example is illustrated below, with time measured in milliseconds:
Initiation time of the last request to the backend staging API: 2022/11/22 10:38:49.200
Initiation time of the request to the backend save API: 2022/11/22 10:38:49.300
Time when the last request to the backend staging API caches data to Redis: 2022/11/22 10:38:49.800
Time when the request to the backend save API reads data from Redis and stores it to MySQL: 2022/11/22 10:38:49.600
4. In the network requests, the response time of the last request to the backend staging API is: Tue, 22 Nov 2022 02:38:49 GMT. See Figure 2.
5. In the network requests, the response time of the request to the backend save API is: Tue, 22 Nov 2022 02:38:49 GMT. See Figure 3.
6. In the network requests, the duration of the last request to the backend staging API is: 686.69 milliseconds. See Figure 4.
7. In the network requests, the duration of the request to the backend save API is: 380.19 milliseconds. See Figure 5.
8. At this stage, there are two solutions. The first solution is frontend processing: after the user clicks the save button, the backend save API is requested only after all requests to the backend staging API have finished. The second solution is backend processing: the backend save API reads data from Redis and stores it in MySQL only after all requests to the backend staging API have cached the data in Redis.
9. Finally, we decided to adopt the second solution, handled by the backend. (1) When the backend receives a staging API request, it increments the key `update_count` in the current session by 1. After caching the data to Redis, it decrements `update_count` by 1. When the backend receives a save API request, it checks the key `update_count` in the current session. If it is greater than 0, it continues to wait. During the wait, it checks the key `update_count` in the current session again every 500 milliseconds. After checking 3 times, it proceeds with execution. Note: 1 second = 1000 milliseconds; 1 millisecond = 1000 microseconds. (2) It reads the data from Redis and stores it in MySQL.
10. The implementation in the staging API, based on the Redis commands `incr` and `decr`, is as follows:
/**
* 将 更新计数 中储存的数字值增一
* @param string $sessionId
* @return int
*/
public static function incrUpdateCount(string $sessionId): int
{
$key = self::getUpdateCountKey($sessionId);
$incr = Redis::connection('cache')->incr($key);
Redis::connection('cache')->expire($key, ThemePreviewInterface::TTL);
return $incr;
}
/**
* 将 更新计数 中储存的数字值减一
* @param string $sessionId
* @return int
*/
public static function decrUpdateCount(string $sessionId): int
{
$key = self::getUpdateCountKey($sessionId);
$decr = Redis::connection('cache')->decr($key);
Redis::connection('cache')->expire($key, ThemePreviewInterface::TTL);
return $decr;
}
11. The implementation in the save API, based on a `while` check, evaluates a total of 3 times:
/**
* 获取 更新计数 中储存的数字值
* @param string $sessionId
* @return mixed
*/
public static function getUpdateCount(string $sessionId)
{
return Redis::connection('cache')->get(self::getUpdateCountKey($sessionId));
}
$i = 1;
while (ThemePreview::getUpdateCount($sessionId) !== null && ThemePreview::getUpdateCount($sessionId) > 0 && $i <= 3) {
file_put_contents(storage_path() . '/logs/Publish-' . microtime(true) . '-' . mt_rand() . '.txt', print_r($i, true), FILE_APPEND | LOCK_EX);
usleep(500000); // 延迟执行 0.5 秒
$i++;
}
12. The output files are as follows. 1669187163.3346 – 1669187162.8234 = 0.5112. 1669187162.8234 – 1669187162.3034 = 0.52. The difference in creation time is approximately 0.5 seconds, which meets expectations. See Figure 8.
Publish-1669187162.3034-776928128.txt
Publish-1669187162.8234-549007488.txt
Publish-1669187163.3346-885156477.txt
13. Trying to reproduce it again in the local environment, it could no longer be reproduced. However, it was confirmed that the file `Publish-1669188133.9627-720653875.txt` was generated, with a value of 1. This indicates that a wait was executed once in the save API. See Figure 9.
14. In the network requests, the duration of the last request to the backend staging API is 2.79 seconds, due to the poor performance of the local environment. See Figure 10.
15. In the network requests, the duration of the request to the backend save API is 3.37 seconds, which is longer than the duration of the staging API. See Figure 11.
16. However, this solution still has a certain probability of data loss. This happens if the staging API reaches the server later than the time the save API completes. Ultimately, we decided to refer to Shopify’s implementation.
17. Referring to Shopify’s theme editor. Checking its request to the backend save API, the request and response variables are as follows. See Figure 12.
{
"eventMetadata": {
"dirtyTransitionCountInSession": 13,
"durationSeconds": 0,
"lastViewport": "desktop",
"pageType": "JSON"
},
"sessionId": "etPGCT4PQr7YyCr87svY9KJz",
"resourceHashes": [
{
"hashValue": "fe8062f520ac1ab9be9a336c76d9f77517551223",
"resourceId": "gid://shopify/OnlineStoreTheme/130440429753"
},
{
"hashValue": "17183337545547076327",
"resourceId": "gid://shopify/OnlineStoreThemeJsonTemplate/index?theme_id=130440429753"
}
],
"ignoreConflicts": null
}
{
"data": {
"onlineStoreThemeEditorSessionPublish": {
"conflicts": null,
"userErrors": [],
"updatedHash": {
"resourceId": "gid:\/\/shopify\/OnlineStoreTheme\/130440429753",
"hashValue": "fe8062f520ac1ab9be9a336c76d9f77517551223",
"__typename": "OnlineStoreResourceHash"
},
"__typename": "OnlineStoreThemeEditorSessionPublishPayload"
}
},
"extensions": {
"cost": {
"requestedQueryCost": 10,
"actualQueryCost": 10,
"throttleStatus": {
"maximumAvailable": 1200.0,
"currentlyAvailable": 1190,
"restoreRate": 60.0
}
}
}
}
18. Our preliminary speculation is that it is related to `hashValue`. The save operation is executed only when the `hashValue` of the data in the cache equals the `hashValue` in the request parameters. The duration of its save API is approximately 1.5 seconds. See Figure 13.
19. The final designed solution is as follows: in the staging API, calculate the `hashValue` based on the request body content and save it to a Redis key. Then, in the save API, compare the `hashValue` in the request parameters with the `hashValue` saved in Redis. If they are equal, it saves. If they are not equal, but the corresponding `hashValue` exists under `ignoreConflicts`, it can also save. If they are not equal and the corresponding `hashValue` does not exist under `ignoreConflicts`, it does not save, and the returned `conflicts` will contain the corresponding `hashValue`.
20. The existing request and response. See Figure 14.
21. Implementation in the staging API:
/**
* 获取更新时的哈希的缓存键
* @param string $sessionId
* @param string $resourceId
* @return string
*/
private static function getUpdateHashKey(string $sessionId, string $resourceId): string
{
return ThemePreviewInterface::THEME_EDITOR_SESSION_PREFIX . $sessionId . ':' . $resourceId . ':' . ThemePreviewInterface::UPDATE_HASH_KEY;
}
/**
* 保存更新时的哈希
* @param string $sessionId
* @param string $resourceId
* @param string $hashValue
* @return bool
*/
public static function saveUpdateHash(string $sessionId, string $resourceId, string $hashValue): bool
{
return self::getCacheTags($sessionId)->put(self::getUpdateHashKey($sessionId, $resourceId), $hashValue, ThemePreviewInterface::TTL);
}
/**
* 获取更新时的哈希
* @param string $sessionId
* @param string $resourceId
* @return mixed
*/
public static function getUpdateHash(string $sessionId, string $resourceId)
{
return self::getCacheTags($sessionId)->get(self::getUpdateHashKey($sessionId, $resourceId));
}
/**
* 判断哈希值是否与缓存中的相等
* @param string $sessionId
* @param string $resourceId
* @param string $hashValue
* @return bool
*/
public static function isHashEq(string $sessionId, string $resourceId, string $hashValue): bool
{
return self::getUpdateHash($sessionId, $resourceId) == $hashValue;
}
$variables = json_encode($args);
ThemePreview::saveUpdateHash($sessionId, $templateBasename, sha1($variables));
22. Implementation in the save API:
public function __invoke($rootValue, array $args, GraphQLContext $context, ResolveInfo $resolveInfo)
{
$conflicts = null;
$ignoreResourceIds = data_get($args, 'ignoreConflicts.*.resourceId');
$hashValue = $themeId;
foreach ($args['resourceHashes'] as $resourceHash) {
if ($this->isConflict($sessionId, $resourceHash, $ignoreResourceIds)) {
$conflicts[] = [
'resourceId' => $resourceHash['resourceId'],
'hashValue' => $resourceHash['hashValue']
];
continue;
}
if ($resourceHash['resourceId'] == $themeId) {
$hashValue = $resourceHash['hashValue'];
$this->themeConfigLoader->publishSettingsData();
} else {
$this->rawSchemaLoaderAdapter->publishTemplateSettings($resourceHash['resourceId']);
}
}
return [
'conflicts' => $conflicts,
'session' => ['sessionId' => $sessionId],
'updatedHash' => [
'resourceId' => $themeId,
'hashValue' => $hashValue
],
'publishedAt' => $publishedAt
];
}
/**
* 判断哈希值是否冲突
* @param string $sessionId
* @param array $resourceHash
* @param ?array $ignoreResourceIds
* @return bool
*/
private function isConflict(string $sessionId, array $resourceHash, ?array $ignoreResourceIds): bool
{
if (!ThemePreview::isHashEq($sessionId, $resourceHash['resourceId'], $resourceHash['hashValue'])) {
if (!is_array($ignoreResourceIds) || !in_array($resourceHash['resourceId'], $ignoreResourceIds)) {
return true;
}
}
return false;
}
23. On the 1st request, when the `hashValue` hits, the response’s conflict list is null. At this point, the frontend takes no further action. See Figure 15.
24. On the 1st request, when the `hashValue` misses, the response’s conflict list is not null. See Figure 16.
25. When the 1st request’s response conflict list is not null, the frontend needs to make a 2nd request, with the request parameters remaining unchanged. See Figure 17.
26. On the 2nd request, when the `hashValue` hits, the response’s conflict list is null. At this point, the frontend takes no further action.
27. When the 2nd request’s response conflict list is not null, the frontend needs to make a 3rd request, where the value of the request parameter `ignoreConflicts` matches `resourceHashes`. See Figure 18.
28. Reference: In Laravel 6 and Lighthouse 5, the internal order of the parameter `$args` in the `__invoke` method is inconsistent with the frontend request parameters. The final implementation of the staging API is as follows:
ThemePreview::saveUpdateHash($sessionId, $templateBasename, sha1($context->request()->getContent()));