没有不值得去解决的问题,也没有不值得去学习的技术!

Testing Search Engine Indexing for A Tour of Go: From Google Organic Crawling to Baidu Active Submission and Bing IndexNow

[Figure 3: Baidu's remaining 9 core URLs submitted successfully, remain drops to zero]

作者:

在

After launching the A Tour of Go Simplified Chinese site, completing search engine verification, and submitting the Sitemap, I was ready to continue working on --minimal-protect.

However, before resuming development, I rechecked the indexing status of go-dev.shuijingwanwq.com across several major search engines.

Google, Bing, and 360 had already completed the Sitemap submission. The Sitemap contained 104 URLs in total, comprising the homepage and 103 course pages. Baidu and Sogou did not use the same Sitemap submission process.

My initial thought was: since the new site hadn’t shown significant indexing results yet, should I take this opportunity to proactively submit the URLs once more to minimize the time it takes for search engines to discover and index the pages?

After checking each platform individually, the situation turned out to be more interesting than expected. Google had already started normal indexing; Bing had discovered the site but hadn’t crawled it yet; Baidu allowed proactive submission via API, but new sites only had a very small daily submission quota; Sogou made me further question whether it was worth investing too much maintenance effort into proactive submission.

Ultimately, this practice didn’t turn into “submitting all 104 URLs to every search engine.” Instead, I evaluated the return on investment for each platform individually and then decided which ones were worth handling proactively and which were better left to natural search engine crawling.

1. First, check Google: It has actually started normal indexing

First, I checked Google Search Console.

Initially, the “Page indexing” summary page still showed that data was being processed and asked me to check back later, so looking only at the summary data easily gave the impression that “Google hasn’t started indexing yet.”

So I switched to the URL Inspection tool to check specific pages directly.

Besides the homepage, I also checked the course list:

https://go-dev.shuijingwanwq.com/tour/list

The results showed:

  • The URL is on Google;
  • The page is indexed;
  • Googlebot crawled successfully;
  • The page allows crawling;
  • The page allows indexing.
[Figure 1: Google Search Console shows /tour/list is officially indexed]
[Figure 1: Google Search Console shows /tour/list is officially indexed]

More notably, other course pages had already appeared in the referral sources, such as:

/tour/flowcontrol/2

/tour/moretypes/5

This indicates that Google didn’t just discover the site’s homepage; it had already started following the site’s internal links to discover and process the entire course system.

Therefore, I didn’t continue manual “Request Indexing” on Google’s end.

Since the Sitemap, internal links, and normal crawling pipeline had already started working, continuing manual intervention wouldn’t be very meaningful.

2. Bing’s situation is different: Discovered, but not crawled yet

Next, I checked Bing Webmaster Tools.

For the homepage:

https://go-dev.shuijingwanwq.com

After performing a URL inspection, the status returned by Bing was noticeably different from Google’s.

The page had been discovered by Bing on August 13, but the status was still:

  • Discovered;
  • Not crawled;
  • Not indexed yet.
[Figure 2: Bing has discovered the go-dev homepage, but has not crawled or indexed it yet]
[Figure 2: Bing has discovered the go-dev homepage, but has not crawled or indexed it yet]

This at least shows that the previously submitted Sitemap had served its purpose of “telling Bing these URLs exist,” but after a few days, Bing still hadn’t actually entered the crawling phase.

Therefore, compared to Google which was already working normally, Bing was more worth trying a proactive notification for.

I didn’t continue using the traditional URL Submission, but instead decided to test IndexNow.

3. Baidu’s index count is still 0

In the Baidu Search Resource Platform, the index count for go-dev.shuijingwanwq.com was even clearer.

From August 12 to August 15, the backend index count remained:

Plaintext
0

Baidu hadn’t completed this Sitemap submission like Google, Bing, and 360, so it was indeed worth trying the official standard indexing API here.

The site had a total of 104 URLs, so I first automatically extracted the URLs from the existing:

https://go-dev.shuijingwanwq.com/sitemap.xml

The results confirmed:

Plaintext
========== URL COUNT ==========
104 /tmp/go-dev-urls.txt

Initially, I tried submitting all 104 URLs at once.

Baidu directly returned:

JSON
{"error":400,"message":"over quota"}

This indicated that the issue wasn’t with the URLs, but with the daily submission quota.

To confirm the actual quota, I changed to submitting only the homepage.

This time it succeeded:

JSON
{"remain":9,"success":1}

In other words, this new site actually only had a quota of 10 standard indexing submissions for the day.

4. Baidu’s remaining 9 quotas: No longer mechanically following the Sitemap order

After confirming there were only 9 quotas left, there was no need to continue submitting /tour/basics/1, /tour/basics/2, /tour/basics/3… in Sitemap order.

With only 10 slots per day, it was more worthwhile to prioritize submitting the entry pages of the entire course system.

Therefore, besides the homepage, I ultimately chose:

Plaintext
https://go-dev.shuijingwanwq.com/tour/list
https://go-dev.shuijingwanwq.com/tour/welcome/1
https://go-dev.shuijingwanwq.com/tour/basics/1
https://go-dev.shuijingwanwq.com/tour/flowcontrol/1
https://go-dev.shuijingwanwq.com/tour/moretypes/1
https://go-dev.shuijingwanwq.com/tour/methods/1
https://go-dev.shuijingwanwq.com/tour/generics/1
https://go-dev.shuijingwanwq.com/tour/concurrency/1
https://go-dev.shuijingwanwq.com/tour/welcome/2

Then I submitted these 9 URLs at once:

Plaintext
{"remain":0,"success":9}
HTTP_STATUS: 200
[Figure 3: Baidu's remaining 9 core URLs submitted successfully, remain drops to zero]
[Figure 3: Baidu’s remaining 9 core URLs submitted successfully, remain drops to zero]

At this point, all 10 of Baidu’s proactive submission quotas for the day were exhausted.

However, after getting this far, I actually decided: I will not continue manually submitting the remaining pages every day.

The reason is simple.

Having only 10 quotas per day means I would need to come back and handle it over multiple consecutive days, and proactive submission itself doesn’t equate to final indexing.

For a new site that already has normal internal links, search engine verification, and public accessibility, specifically coming back every day to maintain this task meant the cost was starting to clearly outweigh the benefits.

So this time it was more about verifying:

Baidu’s standard indexing API works normally.

As for the subsequent pages, I plan to leave them to Baidu’s own natural discovery and crawling.

5. Bing switches to IndexNow: Submitting all 104 URLs at once

Bing, on the other hand, was a completely different experience.

Bing Webmaster Tools itself already provides an IndexNow entry, so I generated an IndexNow Key.

I then deployed the corresponding Key file to the go-dev.shuijingwanwq.com root directory and mapped it individually via Nginx to the existing:

Plaintext
/data/go-tour/verification/

directory.

This way, there was no need to modify the Go Tour program itself, and the existing handling of verification files for Baidu, Sogou, etc., could remain unchanged.

After deployment, I verified it on both the origin server and the public network.

Ultimately, accessing the Key file on the public network returned:

Plaintext
HTTP_STATUS=200

The returned content exactly matched the IndexNow Key.

After confirming the Key file was accessible, I read all 104 URLs from the Sitemap and constructed the IndexNow JSON request.

Before submission, I checked again:

Plaintext
url_count=104
payload_check=OK

Then I submitted them all at once:

Plaintext
========== INDEXNOW SUBMIT ==========
HTTP_STATUS=202
[Figure 4: IndexNow submits 104 URLs at once, first request returns HTTP 202]
[Figure 4: IndexNow submits 104 URLs at once, first request returns HTTP 202]

The 202 here does not mean the API call failed.

When using a new IndexNow Key for the first time, the request is received, but Key verification and subsequent processing are still performed asynchronously.

Compared to Baidu’s limit of only 10 quotas per day, the IndexNow approach is clearly more suitable for notifying an entire new site at once.

6. After waiting about 20 minutes, Bing Webmaster Tools still shows no change

After the submission was complete, I didn’t stop immediately but continued observing the IndexNow page in Bing Webmaster Tools.

From completing the IndexNow submission at around 17:13 to checking again at around 17:32, nearly 20 minutes had passed.

The backend still showed the original:

Plaintext
Get Started

page, and the submission statistics for the 104 URLs didn’t appear immediately.

[Figure 5: After waiting about 20 minutes, Bing IndexNow backend still shows Get Started]
[Figure 5: After waiting about 20 minutes, Bing IndexNow backend still shows Get Started]

However, continuing to refresh at this point wasn’t very meaningful.

The existing evidence was already complete enough:

  • IndexNow Key format is normal;
  • Key file deployed successfully;
  • Origin server access is normal;
  • Public network access returns HTTP 200;
  • File content is correct;
  • 104 URLs passed inspection;
  • IndexNow API returned HTTP 202.

The remaining process belonged to the asynchronous processing of the Bing and IndexNow backends.

Therefore, I didn’t repeatedly submit the 104 URLs just to chase an immediate backend result, nor did I regenerate the Key or continue modifying Nginx.

I stopped here this time and let Bing continue processing it on its own.

7. 360: Sitemap submitted, no need for extra operations for now

360’s situation was relatively simple.

The Sitemap submission was complete, but the index count page on the Webmaster Platform temporarily still showed no data.

Since the Sitemap already existed and the site itself was accessible, there was no need to add extra processing for a new site that had only been online for a few days.

Therefore, the current strategy for 360 is:

Continue waiting for natural crawling and backend data updates, and stop fussing over proactive pushing.

8. Sogou’s data made me decide to abandon proactive advancement

Sogou was the best example in this check to illustrate that “proactive submission isn’t necessarily worth the investment.”

go-dev.shuijingwanwq.com currently showed no inclusion or index data.

But I conveniently checked the long-running main site:

www.shuijingwanwq.com

The results as of August 15 were:

Plaintext
索引量:84
收录量:52,936
[Figure 6: Sogou main site inclusion exceeds 50,000, but index count is only 84]
[Figure 6: Sogou main site inclusion exceeds 50,000, but index count is only 84]

The “inclusion count” and “index count” here are not the same concept.

From the backend results, Sogou had discovered or crawled a large number of pages, but the amount of data actually entering the index was much smaller.

This indicates that for Sogou, the problem was no longer just “whether it was proactively told the URLs.”

Even if the search engine knows a large number of pages exist, how many it is ultimately willing to index is another matter entirely.

In this situation, if I were to further research how to manually submit daily or how to batch submit for go-dev, the return on investment would be quite low.

Therefore, I also decided not to proactively advance with Sogou.

The site just needs to continue allowing normal crawling.

9. Ultimately, only Baidu and Bing were truly handled proactively

Before this round began, I was still considering:

Should I proactively submit to all five: Google, Bing, 360, Baidu, and Sogou?

After actually checking the backends, the answer became much simpler.

Google had already started natural crawling and indexing, so no intervention was needed.

Bing had discovered the URLs but hadn’t crawled them, so I notified all 104 URLs at once via IndexNow.

Baidu’s index count was 0, so I verified the standard indexing API and used the day’s 10 quotas to submit the most important entry pages.

360 had already completed the Sitemap submission, so I just need to keep waiting.

Sogou, because the mature main site inherently exhibited a situation of “crawls a lot, actually indexes very little,” I will no longer proactively invest maintenance time into it.

Ultimately, the only ones that truly needed proactive handling were Baidu and Bing.

Moreover, after completing the API capability verification for Baidu this time, I don’t plan to continue submitting daily.

10. Proactive submission isn’t “the more the better”; maintenance costs must also be considered

The biggest takeaway this time wasn’t successfully submitting 10 URLs to Baidu, nor was it successfully submitting 104 URLs to IndexNow at once.

More importantly, it reaffirmed one thing:

Search engine proactive submission is just a way to help search engines discover pages earlier; it doesn’t mean “once submitted, it will definitely be indexed.”

If a website:

  • Has a normally public Sitemap;
  • Has complete internal links between pages;
  • Is not blocked from crawling by robots.txt;
  • Can be accessed stably;
  • Has already had its URLs discovered by search engines;

Then whether it’s worth spending time on manual submissions every day requires considering the actual maintenance cost.

Baidu only has a small daily quota. If pushing the entire site requires coming back to repeat the operation every day, then this maintenance method isn’t suitable for the current project.

IndexNow allows submitting an entire new site at once with low cost, so it’s worth configuring once.

Google is already working normally on its own, so there’s no need to artificially add operations.

360 and Sogou similarly have no need to keep adding maintenance logic just to “see the numbers change as quickly as possible.”

Therefore, after this round of processing, I plan to return to the original development task:

Plaintext
--minimal-protect

The search engine side will temporarily be left to the Sitemap, internal links, and natural crawling to continue working.

11. Future plans

I won’t check these platforms every day going forward.

Currently, I only need to occasionally observe:

  • Whether the number of indexed pages on Google continues to increase;
  • Whether the Bing homepage moves from “Discovered but not crawled” to crawling and indexing;
  • Whether the IndexNow backend eventually shows this submission record;
  • Whether Baidu starts showing a natural index count;
  • Whether 360 starts generating index statistics.

If these data naturally change later, I can just continue recording them.

For www.shuijingwanwq.com and en.shuijingwanwq.com, if there is a definite need to integrate IndexNow in the future, I can deploy independent Keys for each subdomain respectively, and then set up automatic notifications around newly added, updated, and deleted URLs.

But that already belongs to subsequent independent optimization work.

The most important thing right now is still to continue advancing the A Tour of Go multilingual translation project itself, rather than constantly adding manual operations just for the sake of changing numbers in the search engine backends.

This proactive submission practice can come to an end here.

关于作者 · 持续学习与技术实践

我是一名拥有 15 年以上开发经验的技术从业者,自 2013 年起持续运营个人技术博客,记录工作与个人项目中遇到的真实问题,以及解决问题过程中的探索、实践和思考。

除了技术写作,我也在持续开发和维护自己的独立项目,探索新技术和新工具在实际场景中的应用。我始终相信,没有不值得去解决的问题,也没有不值得去学习的技术。

如果这篇文章对你有所帮助,欢迎浏览博客中的其他文章,也欢迎交流技术经验与想法。

关于我  ·  GitHub  ·  邮件联系