In the official Go tutorial, the Web Crawler exercise is a classic concurrent programming case. It requires us to modify a basic recursive crawler so that it can fetch URLs in parallel while ensuring that the same URL is not fetched multiple times.
This article will guide you step by step through analyzing the problems in the original code and using Go’s sync package to implement an efficient, safe concurrent crawler.
1. Problems with the Initial Code
Although the original Crawl function has simple logic, it has two fatal flaws in practice:
1. Serial execution: The code processes child links through recursive calls to Crawl. This means the program must wait for the previous page and all its subpages to be completely fetched before it starts processing the next sibling node. This completely wastes the performance of modern multi-core CPUs and fails to reflect Go’s concurrency philosophy of “don’t communicate by sharing memory; share memory by communicating.”
2. Duplicate fetching and infinite loops: If page A links to B, and B links back to A, the original recursive logic will fall into an infinite loop. Even without a cycle, multiple pages pointing to the same resource will cause duplicate network requests, wasting bandwidth and processing time.
2. Core Solution
To solve the above problems, we need to introduce two key mechanisms:
1. Concurrency control: Use goroutines to initiate network requests in parallel.
2. State synchronization: Use a mutex (Mutex) to protect the shared “visited” record table, ensuring concurrency safety.
3. Code Implementation Details
Below is the complete, improved code implementation:
package main
import (
"fmt"
"sync"
)
type Fetcher interface {
// Fetch returns the body of URL and
// a slice of URLs found on that page.
Fetch(url string) (body string, urls []string, err error)
}
// Crawl uses fetcher to recursively crawl
// pages starting with url, to a maximum of depth.
func Crawl(url string, depth int, fetcher Fetcher) {
// visited 用于记录已经访问过的 URL，避免重复爬取
visited := make(map[string]bool)
var mu sync.Mutex // 互斥锁，保护 visited 的并发访问
var wg sync.WaitGroup // WaitGroup 用于等待所有爬取任务完成
// crawl 是内部递归函数，实际执行爬取逻辑
var crawl func(url string, depth int)
crawl = func(url string, depth int) {
defer wg.Done() // 确保每个 crawl 调用结束时调用 Done，通知 WaitGroup
if depth <= 0 {
return
}
mu.Lock() // 加锁，保护 visited 的访问
if visited[url] {
mu.Unlock() // 已访问过，解锁并返回
return
}
visited[url] = true // 标记当前 URL 已访问
mu.Unlock() // 解锁
body, urls, err := fetcher.Fetch(url)
if err != nil {
fmt.Println(err)
return
}
fmt.Printf("found: %s %q\n", url, body)
for _, u := range urls {
wg.Add(1) // 为每个新的爬取任务增加计数
go crawl(u, depth-1) // 启动新的 goroutine 爬取子 URL，深度减 1
}
}
wg.Add(1) // 为初始 URL 增加计数
go crawl(url, depth) // 启动爬取初始 URL 的 goroutine
wg.Wait() // 等待所有爬取任务完成
}
func main() {
Crawl("https://golang.org/", 4, fetcher)
}
// fakeFetcher is Fetcher that returns canned results.
type fakeFetcher map[string]*fakeResult
type fakeResult struct {
body string
urls []string
}
func (f fakeFetcher) Fetch(url string) (string, []string, error) {
if res, ok := f[url]; ok {
return res.body, res.urls, nil
}
return "", nil, fmt.Errorf("not found: %s", url)
}
// fetcher is a populated fakeFetcher.
var fetcher = fakeFetcher{
"https://golang.org/": &fakeResult{
"The Go Programming Language",
[]string{
"https://golang.org/pkg/",
"https://golang.org/cmd/",
},
},
"https://golang.org/pkg/": &fakeResult{
"Packages",
[]string{
"https://golang.org/",
"https://golang.org/cmd/",
"https://golang.org/pkg/fmt/",
"https://golang.org/pkg/os/",
},
},
"https://golang.org/pkg/fmt/": &fakeResult{
"Package fmt",
[]string{
"https://golang.org/",
"https://golang.org/pkg/",
},
},
"https://golang.org/pkg/os/": &fakeResult{
"Package os",
[]string{
"https://golang.org/",
"https://golang.org/pkg/",
},
},
}4. Key Technical Points
1. Why is sync.Mutex needed?
In Go, the map type is not concurrency-safe. If multiple goroutines simultaneously attempt to read and write the same map, the program will directly panic. Therefore, we must ensure atomicity between the two operations of checking visited[url] and setting visited[url] = true. Through mu.Lock() and mu.Unlock(), we ensure that only one goroutine can modify or check this map at any given moment.
2. The Role of sync.WaitGroup
When we start multiple goroutines, the main function will not automatically wait for them to finish. Without a WaitGroup, the main function might exit immediately after the first request is sent, causing subsequent subtasks to be forcibly terminated.
wg.Add(1): Increments the counter by 1 before starting a new goroutine.
defer wg.Done(): Decrements the counter by 1 when the goroutine finishes executing.
wg.Wait(): Blocks the main thread until the counter reaches zero, meaning all tasks are complete.
3. Combining Closures and Recursion
We defined a local function crawl inside Crawl. The benefit of this is that it can freely access the external visited, mu, and wg variables without needing to pass these parameters down through layers. At the same time, through go crawl(u, depth-1), we transformed recursion into concurrent task distribution.
5. Summary
Through this exercise, we not only implemented a simple crawler but also gained a deeper understanding of several core concepts in Go’s concurrency model:
– Using goroutines to improve the efficiency of I/O-intensive tasks.
– Using a Mutex to protect shared state and prevent data races.
– Using a WaitGroup to coordinate the lifecycle of multiple goroutines.
In an actual production environment, you may also need to consider adding timeout control, error retry mechanisms, and limiting the maximum concurrency (using a buffered channel as a semaphore), but this basic version already covers the core logic of concurrent programming.