Labs

System lab · medium · 28 min

Googlebot: Web crawler

A polite crawler that covers a billion pages a month and skips what it has seen.

A web crawler sounds like a ten-line script: fetch a page, pull out the links, fetch those, repeat. And that script works, right up until it hammers a small blog with 500 requests a second, downloads the same page under 40 different URLs, and spends a week walking an infinite calendar that some site generates on the fly.

The real design is about restraint. Which page next, how fast for each site, and how to know you've seen something before. This lab builds a crawler like the one behind a search engine, from a blank page.

The brief

You're building the crawler that feeds a search index, like Googlebot. It starts from a list of seed URLs, fetches pages, extracts links, and keeps going. Fetched pages are stored for the indexing team to process.

Who uses it?

  • The indexing pipeline consumes the pages you store. It wants fresh, unique, useful pages.
  • Website owners never asked for you. They tolerate you as long as you obey robots.txt and don't slow their sites down.
  • The crawl team sets priorities: crawl news sites often, crawl deep archives rarely.

In scope: fetching HTML pages, extracting links, politeness, deduplication of URLs and content, prioritising what to fetch, and re-crawling pages as they change.

Out of scope: rendering JavaScript (a separate, expensive pool of headless browsers in real life), indexing and ranking, images and video, and crawling behind logins.

Requirements

Functional

  • Start from seed URLs and discover new ones by following links.
  • Fetch pages over HTTP and HTTPS, and store the raw HTML with metadata.
  • Obey robots.txt, including disallowed paths and crawl delay.
  • Never store the same URL twice in one crawl cycle, and detect pages whose content duplicates another page.
  • Re-crawl pages on a schedule that depends on how often they change.

Non-functional

  • Throughput: 1 billion pages a month.
  • Politeness: never more than one request at a time to a host, with a delay between requests that adapts to the site's speed.
  • Robustness: survive malformed HTML, slow servers, redirect loops and crawler traps without getting stuck.
  • Scalable: add machines to crawl more.
  • Extensible: new content types or processors can be added later.

Politeness is a non-functional requirement that directly limits throughput. That tension drives most of the design.