没有不值得去解决的问题,也没有不值得去学习的技术!

Troubleshooting AI Collaboration Issues in the A Tour of Go Multilingual Translation Project: An Analysis of a Long-Term Engineering Session Failure

Figure 5: Inconsistent model identity descriptions

作者:

在

Current Project Workflow

The current Japanese translation task uses:

Plaintext
GitHub inputs
        ↓
ChatGPT reads
        ↓
ChatGPT full-page translation
        ↓
GitHub create/update raw-responses
        ↓
commit

The project has established that:

  • TranslationUnit is the smallest unit of translation;
  • Each page is translated independently;
  • Multiple TranslationUnits belong to the same batch;
  • Once raw-responses are completed, commits are made on a per-batch basis.

The relevant rules are documented in the project files.


Changes in Git Commit History

During the early stages of the Japanese translation:

chatgpt-ja-JP-001

commit messages still adhered to the project’s original conventions.

For example:

Plaintext
fix: 修订 ja-JP concurrency-11 翻译质量问题
Figure 1: chatgpt-ja-JP-001 commit history
Figure 1: chatgpt-ja-JP-001 commit history

However, starting from subsequent batches:

Plaintext
chatgpt-ja-JP-002
chatgpt-ja-JP-003
chatgpt-ja-JP-004
chatgpt-ja-JP-005

The commit messages changed:

Plaintext
fix: process ja-JP revision batch

translation: add ja-JP recovery raw response concurrency page

fix: process ja-JP welcome revision batch

feat: export ja-JP batch 005
Figure 2: Changes in subsequent ja-JP batch commit history
Figure 2: Changes in subsequent ja-JP batch commit history

On its own, a change in commit messages does not prove that the model changed.

However, in a project that had already established conventions for:

  • documentation language;
  • commit message standards;
  • project maintenance habits;

all of which maintained Chinese descriptions, this became an anomaly worth noting.


Core Issue: Execution Workflow Deviation

What actually impacted the work was not the commit messages.

The more important issue was:

The established engineering workflow was no longer being followed.

For example, while continuing to advance chatgpt-ja-JP-005, the conversation reverted to:

Whether ChatGPT can currently write generated results directly to GitHub files.

Figure 3: Revisiting the GitHub write capability issue
Figure 3: Revisiting the GitHub write capability issue

However, this issue was actually already part of the project’s workflow design.

The previous design goal was:

Plaintext
GitHub inputs

↓

ChatGPT translation

↓

GitHub create/update raw-responses

↓

commit

Revisiting this issue led to:

  • translation tasks being paused;
  • the need to re-explain the confirmed workflow;
  • a decrease in engineering efficiency.

Investigating Model State Anomalies

During the investigation, I discovered a phenomenon that could easily cause confusion.

The user interface displayed:

Plaintext
GPT-5.6 Sol
Figure 4: Model selection interface
Figure 4: Model selection interface

However, during the conversation, the following appeared:

Plaintext
GPT-5.5-mini

This was a model identity description.

Figure 5: Inconsistent model identity descriptions
Figure 5: Inconsistent model identity descriptions

This requires careful analysis.

The identity information output by the model itself cannot serve alone as proof of the actual backend model state.

Possible causes include:

  • the model’s self-reported identity being incorrect;
  • information confusion caused by long context;
  • a discrepancy between the system context and the user interface information.

Currently, this phenomenon alone is not enough to confirm that a model switch definitely occurred on the backend.


I will now return to Part Two.


External Model Analysis

To further analyze this anomaly, I also consulted other models.

Different models showed significant differences in their judgments regarding “whether a backend model switch occurred.”


Analysis from GLM 5.3

GLM 5.3 believes that:

From the perspective of AI service operations, adjusting actual computational resource allocation based on factors like cost and load cannot be technically ruled out entirely.

Possible factors include:

  • changes in service load;
  • cost control;
  • using different computational resources for different tasks.

Therefore:

The possibility that the actual execution model on the backend changes after a user selects a specific model cannot be completely ruled out.

GLM 5.3 believes that, to optimize overall resource utilization, AI service platforms might theoretically employ dynamic scheduling mechanisms.

For example, during large-scale AI service operations, a platform might need to consider:

  • the number of users;
  • request pressure;
  • computational resources;
  • service stability.

Therefore, from a technical standpoint, changes in resource scheduling or model routing are not entirely impossible.

However, a single anomalous performance still cannot prove that a model switch definitely occurred.

This is because similar phenomena could also stem from:

  • a decrease in the weight of key constraints due to long context;
  • gradual drift in conversation state;
  • degraded model execution capability;
  • misunderstanding the boundaries of tool calls.

Analysis from DeepSeek

DeepSeek provided a more conservative assessment.

Its main view is that:

If an AI service provider actively switches a user to a lower-performance model without the user’s explicit choice, it would introduce significant business risks.

These include:

  • user trust risks;
  • product experience risks;
  • corporate reputation risks.

Therefore:

The backend directly switching the user to a low-performance model is not the most likely explanation.

DeepSeek leans toward the view that:

This anomaly is more likely to originate from:

  • the model hallucinating its identity description;
  • information confusion caused by long context;
  • a decline in execution constraints due to an overly long conversation history.

Especially in long-term engineering projects, the model needs to process simultaneously:

  • project background;
  • architecture design;
  • workflow constraints;
  • historical decisions.

As the context grows, the model may not be able to consistently maintain its initial execution state.


Comparison of the Two Views

Analysis DirectionGLM 5.3DeepSeek
Possibility of backend model switchCannot be technically ruled outLow probability
Impact of cost factorsPossibleNot the main explanation
Model identity description errorPossibleMore likely
Impact of long contextPossiblePossible
Current assessmentRequires more evidence for validationLeans toward model hallucination or context issues

Current Assessment

Based on all available information so far:

It is currently impossible to prove that:

ChatGPT’s backend actively switched the current session to a lower-performance model.

What can be confirmed currently is:

  1. The user interface displayed the model as GPT-5.6 Sol;
  2. A GPT-5.5-mini identity description appeared during the conversation;
  3. Execution workflow deviation occurred during the long-term engineering session;
  4. The Git commit history format changed from the previous standards.

Specifically:

  • The inconsistent model identity description is an anomaly that needs investigation;
  • The workflow execution deviation is an issue that has actually impacted development progress;
  • However, there is currently insufficient evidence to determine the exact cause.

Therefore, the more reasonable conclusion is:

This incident is more like a state stability issue during long-term AI engineering collaboration. A backend model switch is a possibility that cannot be completely ruled out, but there is currently no direct evidence to prove it.


Project Workflow Improvements

This anomaly also exposed an engineering issue:

Although the project has defined:

  • TranslationUnit;
  • batch;
  • raw response;
  • validation;

the AI collaboration process still requires more explicit state management.


1. Reduce Reliance on Overly Long Continuous Sessions

In long-term engineering projects:

  • new languages;
  • new phases;
  • new batches;

Try to use new sessions.

Restore context through project state summaries rather than relying on the full conversation history.


2. Retain Execution Summaries

When starting a new AI session, only provide:

Plaintext
Project goals

Current state

Immutable rules

Next task

This prevents the model from re-deriving established engineering designs.


3. Do Not Conclude Model Changes Based on a Single Anomaly

Model state issues require more evidence:

  • user interface information;
  • officially provided information;
  • multiple repeated tests.

A single model identity description cannot serve as a final conclusion.


Summary

This anomaly did not prove that ChatGPT definitely experienced a backend model switch.

However, it exposed a more important issue:

In long-term AI-assisted software engineering, the greatest risk is not only code errors, but also the gradual drift of the execution context.

For engineering projects that involve numerous rules and multi-phase workflows, the AI collaboration process needs to be maintained just like a software system:

  • clarify inputs;
  • clarify responsibilities;
  • clarify states;
  • clarify recovery methods.

The A Tour of Go multilingual translation project will continue to move forward.

At the same time, this experience has become part of the project’s engineering insights:

How to enable AI to maintain stable, predictable collaboration capabilities in long-term, complex projects.

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者,自 2013 年起持续运营个人技术博客,记录工作与个人项目中遇到的真实问题,以及解决问题过程中的探索、实践和思考。

除了技术写作,我也在持续开发和维护自己的独立项目,探索新技术和新工具在实际场景中的应用。我始终相信,没有不值得去解决的问题,也没有不值得去学习的技术。

如果这篇文章对你有所帮助,欢迎浏览博客中的其他文章,也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系