没有不值得去解决的问题,也没有不值得去学习的技术!

Troubleshooting Intermittent Mobile Execution and Formatting Failures in A Tour of Go: From Sporadic Errors to Enhanced Nginx Observability

[Figure 1: Error communicating with remote server. when running the Go example in WeChat]

作者:

在

The day before, I had just completed a round of adjustments to the execution chain of the A Tour of Go Chinese production environment Playground, allowing the “Run” and “Format” requests in the browser to directly access the Playground proxy deployed on the ZgoCloud server.

For the previous deployment and adjustment process, you can refer to:

Records related to ZgoCloud server and Playground proxy deployment

Records related to A Tour of Go Playground browser direct connection proxy

After the adjustments, “Run” and “Format” on the desktop side had been working normally, and real tests on the mobile side had also been successful previously.

However, when I accessed the course page on my phone again the next day, a more troublesome problem occurred:

“Run” and “Format” on the mobile side started to fail intermittently, and the problem could not be consistently reproduced.

Compared to continuous failures, this kind of low-frequency, intermittent fault is actually harder to pinpoint.

Run failures first appeared on the mobile side

I first opened the A Tour of Go Chinese course page in WeChat:

https://go-dev.shuijingwanwq.com/tour/welcome/1

After clicking “Run”, I did not get the normal output of the Go program, but instead received:

Plaintext
Error communicating with remote server.
[Figure 1: Error communicating with remote server. when running the Go example in WeChat]
[Figure 1: Error communicating with remote server. when running the Go example in WeChat]

At this point, the first things that are easy to suspect are:

  • WeChat built-in browser compatibility;
  • Mobile network;
  • Differences between Wi-Fi and 5G;
  • Browser failing to directly access ZgoCloud port 8443;
  • Proxy anomaly from ZgoCloud to the official Playground.

But what happened next made these judgments far from simple.

Formatting also fails intermittently

On the same phone, “Format” also experienced failures.

However, the error it displayed was not the runtime communication prompt, but rather:

Plaintext
[object Object]
[Figure 2: Clicking "Format" on the mobile side displays [object Object]]
[Figure 2: Clicking “Format” on the mobile side displays [object Object]]

This indicates at least two things.

First, the remote Playground request chain that both “Run” and “Format” rely on deserves close inspection.

Second, the frontend’s error handling for formatting failures is itself not user-friendly enough. Even if a network anomaly does occur, the end user should not directly see [object Object] converted from a JavaScript object.

However, before the root cause was determined, I did not rush to modify the frontend error messages.

It was more important to investigate the communication issue first.

Strangely, it quickly returned to normal

Afterwards, I retested WeChat, and running returned to normal.

I then continued testing:

  • The phone’s built-in browser;
  • QQ Browser;
  • Baidu APP;
  • Wi-Fi;
  • 5G.

All of these environments experienced successful cases.

For example, I subsequently modified the example code again and then ran it:

Go
fmt.Println("Hello, 世界rrrrr")

It successfully returned:

Plaintext
Hello, 世界rrrrr
[Figure 3: Running the modified Go code again in the mobile browser succeeds]
[Figure 3: Running the modified Go code again in the mobile browser succeeds]

At this point, a very simple conclusion can basically be ruled out:

The problem is not that “WeChat always fails”, nor is it that “a certain mobile browser always fails”.

It also cannot be attributed to:

Plaintext
Wi-Fi fails, 5G normal

Or:

Plaintext
5G fails, Wi-Fi normal

Both network environments can succeed.

Additional verification was also done on the desktop side.

Since this computer might usually use its own network services, I once suspected that the reason the desktop side was always normal was due to a different network path.

So I actively disconnected the relevant network services and retested.

As a result, “Run” and “Format” on the desktop side were still normal.

Therefore, the problem increasingly looks like a genuine intermittent network or client communication anomaly.

Starting to look for evidence in ZgoCloud Nginx logs

Since the browser directly requests the ZgoCloud Playground proxy, the next step is to directly check the server logs.

The current production requests are mainly:

Plaintext
Browser
→ play.go-dev.shuijingwanwq.com:8443
→ ZgoCloud Nginx
→ play.golang.org

Among them:

Plaintext
POST /compile

is responsible for running code,

Plaintext
POST /fmt

is responsible for formatting code.

I first tallied all relevant requests for the day and obtained:

Plaintext
52 POST /compile -> 200
11 POST /fmt -> 200
5 OPTIONS /fmt -> 204
2 GET /compile -> 403
1 GET /fmt -> 403
[Figure 4: Request status statistics for /compile and /fmt on that day]
[Figure 4: Request status statistics for /compile and /fmt on that day]

This set of results is very interesting.

Those genuinely originating from the browser’s normal business flow:

Plaintext
POST /compile
POST /fmt
OPTIONS /fmt

were all successful.

Among them:

Plaintext
52 POST /compile

all returned:

Plaintext
200
Plaintext
11 POST /fmt

also all returned:

Plaintext
200

CORS preflight:

Plaintext
5 OPTIONS /fmt

were all:

Plaintext
204

The remaining 3 403 were not requests generated by actual users running code, but rather GoogleOther using the GET method to access /compile and /fmt.

The current Nginx configuration only allows normal POST and OPTIONS requests anyway, so these GETs returning 403 is expected behavior.

In other words:

On that day, as long as normal Playground requests actually entered the Nginx access log, not a single 4xx or 5xx actual business failure was found.

But the earliest failed requests did not even enter Nginx

I continued checking the logs against the mobile screenshot times.

The first obvious failure occurred around 14:07.

Checking the Playground access log from 14:05 to 14:10, I only found a normal /compile sent by desktop Firefox at 14:05:50:

Plaintext
POST /compile?backend=
200

No corresponding /compile or /fmt from the mobile side were found.

This means that at least this round of failures might have occurred:

Plaintext
Mobile
→ Establishing connection to play.go-dev.shuijingwanwq.com:8443
→ Nginx

before actually entering Nginx.

But things are not that simple.

Near another frontend failure, Nginx actually logged a 200

In another round of testing around 14:26, the Baidu APP page displayed:

Plaintext
Error communicating with remote server.

Yet, in the server logs around the same time, there was:

Plaintext
POST /compile?backend=
HTTP/2.0
200

And the User-Agent clearly came from the Baidu APP.

So currently there are at least two different phenomena:

Plaintext
First type:
Mobile side shows failure
→ Nginx has no corresponding request

And:

Plaintext
Second type:
Mobile side shows failure
→ Nginx has an HTTP 200 request nearby

However, because the current access log does not have a request ID, nor more detailed upstream timing information, it is not yet 100% certain that this second 200 definitely corresponds to that specific failed operation on the page.

This is also the reason for continuing to enhance logging.

No rush to change the proxy architecture, improve observability first

Reaching this point in the investigation, I did not immediately:

  • Change 8443 to 443;
  • Modify CORS;
  • Modify the proxy target;
  • Adjust timeout settings;
  • Modify the frontend request code.

Because there is no evidence right now that these configurations themselves contain stable errors.

If I directly modified the production architecture just to “solve the problem”, it might just turn an intermittent issue into another problem.

What is truly lacking right now is:

The ability to obtain sufficiently complete evidence the next time a failure occurs.

The original Nginx combined log only showed:

Plaintext
Request
Status code
Response size
User-Agent

But it did not show some very critical information, such as:

Plaintext
$request_time
$upstream_response_time
$upstream_status
$upstream_addr

Fortunately, the nginx.conf on this OneinStack server already had a JSON log format defined, which includes:

Plaintext
request_time
upstream_time
upstream_host
upstream_status

So there was no need to re-modify the global log format.

I only needed to add a separate detailed log for the Playground’s 8443 virtual host.

Adding independent debug logs for the Playground

Without changing the original proxy logic, I added:

Plaintext
/data/wwwlogs/play.go-dev.shuijingwanwq.com_debug.log

At the same time, I added a dedicated error log for the Playground:

Plaintext
/data/wwwlogs/play.go-dev.shuijingwanwq.com_debug_error.log

The original:

Plaintext
/data/wwwlogs/play.go-dev.shuijingwanwq.com_nginx.log

was kept.

In other words, this adjustment only added observability capabilities, without modifying:

Plaintext
8443

Without modifying:

Plaintext
proxy_pass

Without modifying CORS, and without affecting other network services on the server.

After the modifications, I first ran an Nginx configuration check:

Plaintext
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

Then I reloaded Nginx.

Verifying the new detailed log

Finally, I actively sent a test request:

Go
package main

func main() {
    println("DEBUG-OK")
}

The returned result was normal:

JSON
{"Errors":"","Events":[{"Message":"DEBUG-OK\n","Kind":"stderr","Delay":0}],"Status":0,"IsTest":false,"TestsFailed":0,"VetOK":true}

The new detailed log simultaneously recorded:

Plaintext
"request_time":0.652
"status":"200"
"upstream_time":"0.652"
"upstream_host":"142.251.45.17:443"
"upstream_status":"200"
[Figure 5: First verification of the Playground debug log, recording request duration and upstream status]
[Figure 5: First verification of the Playground debug log, recording request duration and upstream status]

This means that the complete proxy execution results can now be seen.

In this test:

Plaintext
Client
→ ZgoCloud Nginx
→ play.golang.org

The total request duration:

Plaintext
0.652 seconds

The upstream response duration was also:

Plaintext
0.652 seconds

The official Playground returned:

Plaintext
200

The status that ZgoCloud ultimately returned to the client was also:

Plaintext
200

The dedicated error log produced no anomaly records.

This time, I did not forcibly declare “the problem is fixed”

This investigation ultimately did not yield a root cause that could be consistently reproduced.

If it were just to make the end of the article look good, I could have easily picked the most likely problem, fixed it, and written “the failure has been resolved”.

But for this kind of low-frequency, intermittent production issue, I prefer to keep the current conclusion:

The problem has currently returned to normal, but it cannot yet be proven that the root cause has been found.

The currently confirmed facts include:

  • The mobile side did experience “Run” failures;
  • The mobile side also experienced “Format” failures;
  • WeChat, the built-in browser, QQ Browser, and Baidu APP were all able to run normally afterwards;
  • Both Wi-Fi and 5G can succeed;
  • The desktop can run normally whether or not the original network services are used;
  • All normal /compile and /fmt requests that entered Nginx on that day returned successfully;
  • An early mobile failure had no corresponding Nginx access log;
  • Near another round of failures, there was a record of a mobile request returning 200;
  • There is currently insufficient evidence to attribute the problem to any single layer among the browser, 8443, CORS, ZgoCloud, or the official Playground.

Therefore, the most important outcome this time is not “how much code was modified”, but advancing the production environment from:

Plaintext
Only being able to guess after a problem occurs

to:

Plaintext
Being able to directly see detailed evidence of the request and upstream chain the next time a problem occurs

Next steps

Next, I will no longer continue high-frequency manual click testing.

Letting real user access and daily usage naturally generate requests is sufficient.

If “Run” or “Format” failures occur again in the future, I only need to record the time of the failure and then check the new Playground dedicated log to simultaneously confirm:

Plaintext
Whether the request entered Nginx
Plaintext
Final HTTP status
Plaintext
Actual upstream status
Plaintext
Complete request duration
Plaintext
Upstream response duration
Plaintext
Which upstream address was actually connected to

If a genuine anomalous request can be captured next time, there will be sufficient basis to decide whether further adjustments are needed for:

Plaintext
Port
Proxy
TLS
CORS
Timeout
Frontend request logic

For this kind of intermittent failure, rather than quickly modifying the production environment based on a single screenshot, it is better to first equip the system with the ability to answer questions.

This time, I am turning “why it failed” into a question that can be directly verified via logs next time, and then waiting for a genuine anomalous sample to appear.

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者,自 2013 年起持续运营个人技术博客,记录工作与个人项目中遇到的真实问题,以及解决问题过程中的探索、实践和思考。

除了技术写作,我也在持续开发和维护自己的独立项目,探索新技术和新工具在实际场景中的应用。我始终相信,没有不值得去解决的问题,也没有不值得去学习的技术。

如果这篇文章对你有所帮助,欢迎浏览博客中的其他文章,也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系