AI InterviewGet started

Design a Web Crawler: System Design Interview Practice

Design a Web Crawler as a system design interview question: Crawl the web from a set of seed URLs, store the pages, and keep crawling newly found links.

How do you answer "Design a Web Crawler" in a system design interview?

Start from seed URLs in a frontier queue. Fetchers take URLs from it, download each page, store it, extract its links, normalise them and add the unseen ones back to the frontier, crawling the link graph breadth first. Per-host queues keep the crawler polite, robots.txt decides what may be fetched, and URL and content hashes stop the same page being crawled twice.

  • Level: Advanced
  • Scored out of: 100
  • Cost: Free to read; a free account gets the review

Crawl the web from a set of seed URLs, store the pages, and keep crawling newly found links.

The scale

1 billion pages a month; pages average 100 KB.

What a strong answer covers

  1. Requirements (20 points). States the functional requirements in scope and the non-functional ones that drive the design (scale, latency, availability, consistency), and leaves out what is out of scope.
  2. Scale estimates (15 points). Turns the stated scale into requests per second (average and peak), storage and bandwidth, with the arithmetic shown, and uses the numbers to make a decision.
  3. API and data model (20 points). Defines the main API calls with their inputs and outputs, and a data model whose keys and store fit the access patterns.
  4. Architecture (25 points). Chooses components that fit the requirements and explains how a request moves through them, for the main read and write paths.
  5. Bottlenecks and trade-offs (20 points). Names the real bottlenecks and failure points at this scale and how the design handles them, weighing the alternatives (consistency vs availability, push vs pull, cost).

How the practice works

Write your design in sections — functional and non-functional requirements, scale estimates, the API, the data model, the request flow and a deep dive on the bottlenecks — and pick the components to draw the architecture. A review scores it on the five areas above in under a minute, then you can defend it in an AI mock interview.

The reference design

Design a Web Crawler: the walkthrough, with animated figures.

Common questions

How do you answer "Design a Web Crawler" in a system design interview?

Start from seed URLs in a frontier queue. Fetchers take URLs from it, download each page, store it, extract its links, normalise them and add the unseen ones back to the frontier, crawling the link graph breadth first. Per-host queues keep the crawler polite, robots.txt decides what may be fetched, and URL and content hashes stop the same page being crawled twice.

What scale is "Design a Web Crawler" designed for here?

1 billion pages a month; pages average 100 KB. Your estimates turn that into requests per second, storage and bandwidth before you pick components.

How is a design scored?

Out of 100 on a fixed rubric: requirements (20), scale estimates (15), api and data model (20), architecture (25), bottlenecks and trade-offs (20). The review names what the design covered and what it missed, then points to the reference design.

Is "Design a Web Crawler" a good first system design problem?

It is rated advanced. Start with the beginner problems — the URL shortener and the rate limiter — and the System Design notes on load balancing, caching and sharding first.

Do I need an account?

Not to read the problem or the reference design. Getting your design reviewed needs a free account.

More design problems