没有不值得去解决的问题,也没有不值得去学习的技术!

After Launching the A Tour of Go Chinese Site: Add robots.txt and Sitemap, and Submit to Five Major Search Engines

Figure 1: robots.txt and sitemap.xml are correctly served in the production environment, and the Sitemap contains a total of 104 canonical URLs.

作者:

在

The A Tour of Go Simplified Chinese site officially went live at:

Plaintext
https://go-dev.shuijingwanwq.com/

After that, I started working on search engine submissions for the post-launch phase.

The original plan for this time was simple: reuse the experience gained from adding search engine resources to en.shuijingwanwq.com to quickly complete the integration with five platforms: Google, Bing, 360, Baidu, and Sogou.

In practice, the search engine submission itself was not complicated, but before officially submitting, I first discovered two more fundamental issues:

  • /robots.txt actually returns the Tour homepage HTML;
  • /sitemap.xml also returns the Tour homepage HTML.

In other words, the wildcard routing of the A Tour of Go application took over these two paths that should have been provided to search engines.

Ultimately, this work not only completed the submissions to five platforms, but also filled in the basic SEO entry points for the production site, resolved the verification file access issues for Baidu and Sogou, and cleaned up outdated configuration warnings under the new Nginx version.


1. Before Official Submission, First Check the Search Engine Crawling Entry Points

Initially, I checked from the public internet:

Plaintext
https://go-dev.shuijingwanwq.com/robots.txt
https://go-dev.shuijingwanwq.com/sitemap.xml

It turned out that although both returned HTTP 200, the Content-Type for both was:

Plaintext
text/html; charset=utf-8

The body content was not robots or Sitemap either, but the full HTML of the “A Tour of Go” homepage.

The root cause was straightforward:

The production HTTP router did not register /robots.txt and /sitemap.xml, so the requests ultimately fell through to the Tour’s / wildcard route.

So I did not rush into the various webmaster platforms. Instead, I paused the search engine submissions and added these two entry points in go-tour-i18n.


2. Dynamically Generating the Sitemap from the Official Course Page Set

This time, I did not maintain a separate hardcoded Sitemap list of 103 pages.

The project itself already had the complete set of official course pages, so the Sitemap was generated directly from the officially rendered pages.

The final Sitemap included:

  • Homepage: 1;
  • Official course pages: 103;
  • Total: 104 URLs.

The course URLs uniformly use:

Plaintext
https://go-dev.shuijingwanwq.com/tour/{article}/{section}

At the same time, no data without a reliable source was artificially added:

  • lastmod
  • changefreq
  • priority

Nor did any deprecated old domains appear:

Plaintext
go-tour.shuijingwanwq.com

robots.txt was kept very simple:

Plaintext
User-agent: *
Allow: /

Sitemap: https://go-dev.shuijingwanwq.com/sitemap.xml

After testing the related code, I committed the changes:

Plaintext
7162b98 feat: add search engine crawling entry points

Then I regenerated the production release package.


3. Before the Production Release, Re-verifying the Actual Bundle

The new production bundle still maintained:

Plaintext
locale=zh-CN
ready=103
pending=0
blocked=0
pages=103
articles=7

To avoid a situation where “the source code tests fine, but the actual release package has discrepancies,” I directly started the final production bundle for local acceptance testing.

Final confirmation:

Plaintext
/robots.txt
HTTP 200
Content-Type: text/plain; charset=utf-8

/sitemap.xml
HTTP 200
Content-Type: application/xml; charset=utf-8

Sitemap URL count
104

The course pages themselves also continued to work normally:

Plaintext
/tour/welcome/1
HTTP 200
<html lang="zh-CN">
<title>Go 语言之旅</title>

And /tour/lesson/ was still able to return the Chinese course data.

This showed that adding the new SEO entry points did not affect the way the existing Tour works.


4. Deploying to Production and Handling Stale EdgeOne Cache

After uploading the new production bundle to the server, I first completed:

  • SHA-256 full verification;
  • go-tour service user read permission check;
  • release directory check;
  • current pre- and post-switch confirmation.

After that, I performed the atomic switch for:

Plaintext
/data/go-tour/current

From the old version:

Plaintext
/data/go-tour/releases/20260812-zh-CN-cac4d6e

Switched to:

Plaintext
/data/go-tour/releases/20260812-zh-CN-7162b98

After restarting the service:

Plaintext
go-tour.service
Active: active (running)

Direct access to the origin server was already able to correctly fetch the new robots and Sitemap.

However, public internet access initially still retrieved the old content.

The cause was not the application, nor Nginx, but rather that the EdgeOne nodes still retained the cache from before the deployment.

Therefore, I only purged precisely:

Plaintext
https://go-dev.shuijingwanwq.com/robots.txt
https://go-dev.shuijingwanwq.com/sitemap.xml

I did not flush the entire site.

After the purge, the public internet re-fetched from the origin, ultimately returning:

Plaintext
robots.txt
HTTP/2 200
content-type: text/plain; charset=utf-8

sitemap.xml
HTTP/2 200
content-type: application/xml; charset=utf-8

URL_COUNT=104
Figure 1: robots.txt and sitemap.xml are correctly served in the production environment, and the Sitemap contains a total of 104 canonical URLs.
Figure 1: robots.txt and sitemap.xml are correctly served in the production environment, and the Sitemap contains a total of 104 canonical URLs.

5. A Brief Consideration: Does the Course Text Not Being in the Source Code Affect SEO?

There is another small插曲 worth recording here.

When I opened:

Plaintext
view-source:https://go-dev.shuijingwanwq.com/tour/welcome/1

I found that the current course text was not present in the raw HTML.

The page source mainly consisted of the Tour’s HTML shell; the course text was fetched and rendered via the course API after JavaScript ran.

This led me to briefly wonder:

If the initial HTML fetched by search engines does not contain the course text, how much meaning is there in generating a Sitemap for 103 course pages?

So I specifically checked the current implementation of the official A Tour of Go.

It turned out that the official site itself still uses a similar client-side rendering structure:

HTML
<div table-of-contents></div>

<div ng-view ng-cloak class="ng-cloak"></div>

<script src="/tour/script.js"></script>

The specific course data is likewise fetched after the browser runs JavaScript.

In other words:

This is not a new issue introduced by the Simplified Chinese translation project, but rather the architectural approach of the official A Tour of Go itself.

Therefore, I ultimately did not refactor the entire Tour into server-side rendering just for SEO.

For a continuous tutorial-style website like A Tour of Go, at this stage I value more:

  • The homepage being able to be crawled and indexed normally;
  • Search engines being able to discover the entire set of course URLs;
  • Users entering from the homepage and continuing to learn through the Tour’s own course navigation.

It would of course be better if the course pages could be indexed independently, but right now there is no need to deviate significantly from the official architecture just for this goal.


6. Google Search Console: All 104 Pages Discovered

After the basic SEO entry points were ready, I officially started the search engine submissions.

The first was Google Search Console.

go-dev already existed as an independent URL prefix property, so I went directly to “Sitemaps” and submitted:

Plaintext
sitemap.xml

Google quickly completed reading it:

Plaintext
Status: Success
Discovered pages: 104

This also proved in reverse that the newly generated Sitemap was not only accessible, but could also be parsed correctly by Google.

Figure 2: Google Search Console successfully read the go-dev Sitemap and discovered 104 pages.
Figure 2: Google Search Console successfully read the go-dev Sitemap and discovered 104 pages.

7. Bing: Continuing to Reuse the Experience from Adding the en Subdomain

Next was Bing Webmaster Tools.

I did not re-add go-dev as an independent site here.

When handling en.shuijingwanwq.com previously, I had already verified a simpler method:

Directly submit the subdomain’s full Sitemap URL under the existing shuijingwanwq.com Bing property.

So I continued to reuse this approach this time:

Plaintext
https://go-dev.shuijingwanwq.com/sitemap.xml

It quickly showed:

Plaintext
Status: Success
Discovered URLs: 104
Figure 3: Bing Webmaster Tools successfully read the go-dev Sitemap and discovered 104 URLs.
Figure 3: Bing Webmaster Tools successfully read the go-dev Sitemap and discovered 104 URLs.

This also reiterated a very practical lesson:

For a platform process that has already been successfully navigated once, there is no need to re-explore it from scratch just because the subdomain changed.

If previous successful experience can be directly reused, try to avoid repeating the same pitfalls.


8. 360: Adding a Sub-site, Then Configuring HTTPS and Sitemap

For 360, I also reused the experience from handling en previously.

Because go-dev.shuijingwanwq.com is a subdomain under an existing main domain, I did not select “Add Website” again, but instead:

Plaintext
Site Management
→ Add Sub-site

Added:

Plaintext
go-dev.shuijingwanwq.com

After successful addition, the site showed:

Plaintext
Filing status: Filed
Site permissions: Owner

Then I completed the HTTPS configuration, and entered:

Plaintext
Data Submission
→ Sitemap Submission

Submitted:

Plaintext
https://go-dev.shuijingwanwq.com/sitemap.xml
Figure 4: 360 Webmaster Platform completed the go-dev sub-site, HTTPS, and Sitemap submission.
Figure 4: 360 Webmaster Platform completed the go-dev sub-site, HTTPS, and Sitemap submission.

Right after submission, “Included URLs” and “Update Time” temporarily showed --; there is no need to keep waiting at this stage.

As long as the Sitemap has successfully entered the platform, that is sufficient.


9. Baidu: Continuing to Use File Verification Without Modifying the Tour Application

The Baidu Search Resource Platform requires adding separately:

Plaintext
https://go-dev.shuijingwanwq.com

The site property continued to be set as “Information Technology”.

At the verification stage, Baidu provides:

  • File verification;
  • HTML tag verification.

I had already used file verification when handling en previously, so I continued to use it this time.

The verification file generated by Baidu looks like:

Plaintext
baidu_verify_codeva-XXXXXXXXXX.html

But go-dev is different from a WordPress site.

It is itself a Go Tour application, and does not have a web root directory in the traditional sense where HTML files can be casually dropped.

Initially, I considered adding this verification path to the Tour application, but soon realized that it was completely unnecessary.

Because there is already Nginx in front of the production environment:

Plaintext
EdgeOne
→ Nginx
→ 127.0.0.1:3999
→ Tour

Therefore, I ultimately saved the verification file permanently to:

Plaintext
/data/go-tour/verification/

Then, I used Nginx for exact matching:

Nginx
location = /baidu_verify_codeva-XXXXXXXXXX.html {
    alias /data/go-tour/verification/baidu_verify_codeva-XXXXXXXXXX.html;
    default_type text/html;
}

This way, only the Baidu verification file is returned directly by Nginx:

Plaintext
/baidu_verify_codeva-XXXXXXXXXX.html

All other Tour requests still continue to:

Plaintext
proxy_pass http://127.0.0.1:3999;

Completely avoiding the need to pollute the Tour application itself.


10. Incidentally Cleaning Up Old Configuration Warnings Under the New Nginx Version

After modifying the Baidu verification configuration, I executed:

Bash
nginx -t

I found that the existing Nginx configuration had two historical warnings.

The first was the old HTTP/2 syntax:

Nginx
listen 443 ssl http2;
listen [::]:443 ssl http2;

So I adjusted it to:

Nginx
listen 443 ssl;
listen [::]:443 ssl;
http2 on;

The second was:

Nginx
ssl_stapling on;
ssl_stapling_verify on;

The current certificate no longer has an OCSP responder available for Nginx to use, so the corresponding configuration only continuously generates:

Plaintext
ssl_stapling ignored

warnings.

I deleted these two lines as well.

Finally, I re-executed:

Bash
nginx -t

Only this remained:

Plaintext
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

The original warnings were gone.

Then, following the current production operations manual, I used:

Bash
service nginx reload

Completed the reload.


11. Baidu Verification Succeeded, and Incidentally Found that en Missed “Associated Entity”

After the Baidu verification file passed access tests at both the origin and EdgeOne layers, the site verification was completed.

Finally, in Baidu’s “Site Management”, I could now see all of:

Plaintext
https://go-dev.shuijingwanwq.com/
https://en.shuijingwanwq.com/
https://www.shuijingwanwq.com/
Figure 5: Baidu Search Resource Platform completed the go-dev site verification, while also completing the entity association for existing sites.
Figure 5: Baidu Search Resource Platform completed the go-dev site verification, while also completing the entity association for existing sites.

Here I also incidentally discovered a previously missed issue:

Although en.shuijingwanwq.com had been added to Baidu long ago, it had not been associated with the existing entity.

Later, when checking the Baidu index volume for en, I found that even without an associated entity, Baidu had actually still been indexing it continuously, and the recent index volume had even grown to over 80.

Therefore:

Not having an associated entity does not mean Baidu will completely refrain from indexing a site.

However, since this omission had been discovered, and www, en, and go-dev all inherently belong to the same entity, I took the opportunity to uniformly complete the entity association for all three sites.


12. Sogou: File Verification Succeeded on the First Try This Time

The last platform was the Sogou Resource Platform.

Previously, when performing Sogou verification for en.shuijingwanwq.com, file verification had repeatedly failed.

At that time, en was using Cloudflare CDN. I had always suspected there was some compatibility issue between Sogou’s verification crawler and Cloudflare, but I never had enough evidence to confirm it.

This time, go-dev was using EdgeOne, so it was a perfect opportunity to test again.

The verification file generated by Sogou was:

Plaintext
sogousiteverification.txt

Again, I did not modify the Tour application, but placed it at:

Plaintext
/data/go-tour/verification/sogousiteverification.txt

Then, I used Nginx for exact matching:

Nginx
location = /sogousiteverification.txt {
    alias /data/go-tour/verification/sogousiteverification.txt;
    default_type text/plain;
}

I verified separately:

Plaintext
Origin Nginx
→ HTTP 200

EdgeOne public internet
→ HTTP 200
→ EO-Cache-Status: MISS

The verification file content was also completely correct.

Then I went back to Sogou and clicked “Complete Verification”.

This time:

Succeeded on the first try.

This at least shows that the current:

Plaintext
EdgeOne
→ Nginx
→ Verification file

link can be accessed normally by Sogou.

As for whether the previous en + Cloudflare verification failure was really caused by Cloudflare, I cannot draw a definitive conclusion based solely on this single comparison, but this difference is worth keeping on record.


13. Sogou Does Not Require Full Qualification Review to Submit Resources

After completing the site verification on Sogou, the platform continued to guide me to:

Plaintext
Submit Qualifications
→ Review Qualifications

It required filling in:

  • Website name;
  • Industry;
  • ICP filing number;
  • ICP filing screenshot;
  • Organizer;
  • Entity information;
  • Operator information.

Initially, I was unsure:

If I did not complete this entire qualification process, would it affect Sogou’s indexing?

So I did not rush to continue filling it out, but first entered:

Plaintext
Search Services
→ Resource Inclusion

As a result, I was already able to directly select in the site dropdown:

Plaintext
go-dev.shuijingwanwq.com

This means:

After site verification is completed, even without continuing to submit full qualifications, you can already use the basic resource inclusion feature as a “verified site”.

Therefore, there is currently no need to fill out the full qualifications just for search engine inclusion.


14. Sogou Sitemap Still Uses an Invitation System

The Sogou “Resource Inclusion” page simultaneously features:

Plaintext
URL Submission
Sitemap Submission

So I checked the Sitemap permissions.

The results page explicitly showed:

Plaintext
This site has not enabled sitemap permissions

It also explained that Sitemap uses an invitation system, and will only potentially be opened after Sogou Spider reaches a certain level of crawling on the site.

Figure 6: Sogou has completed the go-dev site verification and opened basic resource inclusion, but Sitemap submission permissions still use an invitation system.
Figure 6: Sogou has completed the go-dev site verification and opened basic resource inclusion, but Sitemap submission permissions still use an invitation system.

This also means that the actual situation across the five platforms is not entirely consistent.


15. Final Status of the Five Platforms

At this point, the search engine integration status for go-dev.shuijingwanwq.com is as follows:

Google

Plaintext
✅ Independent property
✅ Sitemap submitted successfully
✅ 104 pages discovered

Bing

Plaintext
✅ Submitted under existing shuijingwanwq.com property
✅ Sitemap successful
✅ 104 URLs discovered

360

Plaintext
✅ Added sub-site
✅ HTTPS
✅ Sitemap submitted

Baidu

Plaintext
✅ Added go-dev site
✅ File verification passed
✅ Associated with existing entity
⚠ Current account/site does not meet the conditions to directly use Sitemap submission

Sogou

Plaintext
✅ Site file verification passed
✅ Can use resource inclusion as a verified site
⚠ Sitemap permissions still on invitation system

16. Why I Started Considering a Separate “Search Engine Submission Project”

Towards the end, a new requirement became increasingly apparent.

I currently maintain:

Plaintext
www.shuijingwanwq.com
en.shuijingwanwq.com
go-dev.shuijingwanwq.com

If A Tour of Go continues to add new languages in the future, there may be even more sites.

If, for every new site launch or new article publication, I manually log into:

Plaintext
Google
Bing
360
Baidu
Sogou

and operate on each platform individually, it will generate more and more repetitive labor.

Moreover, the capabilities supported by each platform are not the same:

Plaintext
Google / Bing / 360
→ Sitemap is relatively convenient

Baidu
→ Better suited for combining with active push API

Sogou
→ Sitemap invitation system
→ Can currently use URL submission

Therefore, moving forward, I lean towards creating a very lightweight standalone project, for example:

Plaintext
search-engine-submit

Its responsibilities would remain simple:

Plaintext
Read sitemap.xml / URL list
        ↓
Identify new or resubmittable URLs
        ↓
Call official APIs capable of automatic submission
        ↓
Generate list of platform URLs requiring manual submission
        ↓
Save status of each submission

In the first phase, there is absolutely no need to build:

  • A database;
  • A web admin backend;
  • A distributed task queue;
  • A complex scheduling system.

A command-line tool plus a small amount of JSON state files would be sufficient to solve the main repetitive labor.


17. Biggest Takeaway This Time: Reuse Verified Experience Directly

The actual operation time this time was not long, and a very important reason was the extensive reuse of experience accumulated from adding en.shuijingwanwq.com previously.

For example:

For Bing, instead of re-exploring “how to add an independent subdomain site”, I directly:

Plaintext
Existing shuijingwanwq.com
→ Sitemap
→ Submit go-dev full Sitemap URL

For 360, I directly:

Plaintext
Add sub-site
→ HTTPS
→ Sitemap

For Baidu, I continued to:

Plaintext
Add website
→ File verification

For Sogou, I similarly prioritized trying:

Plaintext
File verification

The only difference was that this time go-dev was an independent Go application, so I placed the verification files uniformly in:

Plaintext
/data/go-tour/verification/

And let Nginx handle them precisely.

Throughout the process, there was no need to modify the Tour application for two search engine verification files.

This approach also makes me increasingly certain that:

For long-term maintained projects, recording previously verified production experience is important. When encountering similar problems next time, what truly improves efficiency is often not rediscovering a “smarter” solution, but directly reusing the path that has already been proven to work.


18. Current Results

After this round of processing, the foundational search engine work for the A Tour of Go Simplified Chinese site post-launch has been completed.

The production environment now has:

Plaintext
https://go-dev.shuijingwanwq.com/robots.txt
https://go-dev.shuijingwanwq.com/sitemap.xml

In the Sitemap:

Plaintext
Homepage: 1
Course pages: 103
Total URLs: 104

And:

Plaintext
Google  ✅
Bing    ✅
360     ✅
Baidu   ✅
Sogou   ✅

All five platforms have completed the integration possible at the current stage.

The next step does not require continued frequent adjustments.

I will first let the search engines crawl and index naturally, and then observe based on actual data:

  • Whether the homepage enters the index;
  • Whether it starts appearing when searching for brand terms and Go tutorial-related keywords;
  • How many of the 103 course pages can be independently discovered;
  • How well Baidu, 360, and Sogou crawl these JavaScript Tour pages;
  • Whether it is worth starting a standalone search engine API submission tool project.

The A Tour of Go Simplified Chinese site has taken another step forward from “being officially accessible” to “being reliably discoverable by search engines”.

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者,自 2013 年起持续运营个人技术博客,记录工作与个人项目中遇到的真实问题,以及解决问题过程中的探索、实践和思考。

除了技术写作,我也在持续开发和维护自己的独立项目,探索新技术和新工具在实际场景中的应用。我始终相信,没有不值得去解决的问题,也没有不值得去学习的技术。

如果这篇文章对你有所帮助,欢迎浏览博客中的其他文章,也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系