Layered Architecture
Traditional n-tier architecture pattern
Design a Web Crawler
A distributed crawler that discovers, fetches, and extracts text from 10 billion web pages within 5 days while respecting robots.txt and never losing progress.
//The Contract
Requirements
Functionalwhat the system must do
- ✓Operators seed the crawler with an initial list of URLs, and it discovers new pages automatically by following outbound links
- ✓The crawler extracts and stores the clean text content of every page it visits for downstream processing (e.g. LLM training)
- ✓Already-visited URLs and duplicate content are detected and skipped so the same page is never fetched or stored twice
- ✓Crawling automatically respects each site's robots.txt rules, including the Crawl-delay directive
Non-functionalhow well it must do it
- ≫Fault tolerance — a crashed worker or a failed fetch never loses crawl progress; work resumes from the last known state
- ≫Politeness — no more than ~1 request per second per domain, so target servers are never overloaded
- ≫Efficiency — crawl 10 billion pages to completion in under 5 days
- ≫Scalability — the fetch and parse tiers scale horizontally and independently as queue depth grows
//Back of the Envelope
Scale Estimates
Rough, order-of-magnitude numbers. The point is to justify the architecture, not to be exact.
Pages to crawl
10B
the entire crawl must complete inside the 5-day window
Time budget
5 days
≈ 432,000 seconds to discover and fetch every page
Avg. page size
~2MB
includes embedded resources counted toward transfer bandwidth
Required throughput
~23K/s
10B pages ÷ 432,000s — the floor every downstream tier must sustain
Fetcher fleet size
~8 hosts
network-optimized instances at ~30% practical utilization of a 200 Gbps NIC
Per-domain limit
1 req/s
a politeness ceiling enforced independently of total fleet throughput
//Follow the Data
Request Flows
A diagram shows what exists; a trace shows what happens. Press play on a flow, or click any step, and the diagram above lights up the exact path that request takes.
Flow 01
Fetch & Store a Page
From dequeuing a URL to the raw page being durably stored.
Flow 02
Parse, Extract & Rejoin the Frontier
From a fetched-page reference to newly discovered URLs re-entering the crawl.
Flow 03
Handle a Failed Fetch
How the pipeline survives a bad domain without losing the rest of the crawl.
//Component Breakdown
Key Components
Every box on the canvas, and the job it does. Click a card to locate it in the diagram; click the node itself for the full deep dive.
client×1
network×1
compute×4
database×1
cache×2
queue×3
storage×2
external×1
//Deep Dive
Architecture Breakdown
Architecture Overview
This design models a distributed web crawler that discovers, fetches, and extracts text from 10 billion pages in under 5 days, without losing progress and without overloading the servers it crawls.
Key Design Principles
Pipelined stages, not a monolithic worker. Fetching and parsing are split across two queues — the Frontier Queue and the Further Processing Queue — so a failure or slowdown in one stage never forces redoing the other. Fetching is I/O-bound and bandwidth-expensive; parsing is CPU-bound and cheap to redo. Keeping them independent lets each scale on its own bottleneck.
Politeness is a shared, not per-worker, constraint. A Redis-backed per-domain lock and sliding-window counter ensure the entire fleet — not just one thread — respects each domain's crawl-delay. Robots.txt rules are cached in the same layer so the hot path never blocks on a database read.
Two-tier deduplication. A Bloom filter rejects the overwhelming majority of already-seen URLs in O(1) before they ever reach the Metadata DB, while a content hash computed after each fetch catches pages that are byte-identical under different URLs — mirrors, tracking-parameter variants, and syndicated content.
The frontier is a loop, not a line. New URLs discovered during parsing flow back through the Dedup Filter and re-enter the Frontier Queue, expanding the crawl outward from the original seed list until either the frontier drains or the 5-day budget runs out.
Storage is split by purpose. Raw HTML and extracted text live in separate locations. Keeping the raw HTML means extraction logic can be improved and re-run against the whole corpus later without re-paying the bandwidth cost of re-fetching 10 billion pages.
Fault Tolerance
- Fetcher crash mid-fetch: the Frontier Queue message's visibility timeout expires and another worker retries automatically — no operator intervention needed.
- Persistently broken URLs: after 5 failed attempts (tracked via
ApproximateReceiveCount), messages move to a Dead Letter Queue instead of retrying forever. - Parser backlog: the parser fleet autoscales directly on Further Processing Queue depth, so a burst of fetched pages doesn't stall the pipeline.
- Domain-level abuse: the Politeness Rate Limiter's shared lock prevents the fleet from ever exceeding a domain's crawl-delay, even under retry storms.
//Trade-offs
Design Decisions
An architecture is a record of trade-offs. For every major choice here: what won, what lost, and why the constraints made it so.
01
Frontier queue technology
ChoseSQS with per-message exponential backoff
OverKafka topic per priority levelCustom Redis-backed priority queue
The dominant failure mode in a 5-day, 10B-page crawl isn't total system failure — it's individual fetches timing out against slow or dead servers. SQS gives visibility timeouts, ApproximateReceiveCount-driven backoff, and dead-letter routing for free. Kafka would require building retry and DLQ semantics by hand on top of consumer offsets, which is exactly the kind of infrastructure work this deadline can't absorb.
02
Fault-tolerant pipeline shape
ChosePipelined stages — fetch, store, enqueue reference, then parse — instead of one monolithic worker
OverA single worker that fetches, parses, and extracts links in one step
A monolithic worker means any bug in text extraction forces the fetch to be redone too, wasting the exact bandwidth that is this system's scarcest resource (~23K pages/s required). By splitting fetch from parse across two queues, a parser crash only re-runs cheap local computation against HTML that's already safely in blob storage.
03
Politeness enforcement
ChosePer-domain Redis lock and sliding-window counter, TTL matched to each domain's Crawl-delay
OverA static delay baked into each worker threadNo cross-worker coordination — rely on queue randomness to spread requests
With ~8 fetcher hosts running many threads each, multiple workers can independently dequeue URLs from the same popular domain. A per-thread delay only bounds that thread's request rate; it does nothing to stop the fleet collectively exceeding 1 req/s against one domain. Politeness is a property of the whole system, so it needs shared state.
04
URL dedup fast path
ChoseRedisBloom filter as a pre-check in front of the Metadata DB
OverQuery the Metadata DB for every candidate URLAn in-memory hash set per worker
At 10B+ discovered link candidates, hitting DynamoDB for every single one to ask 'have we seen this?' would dominate the DB's read capacity. A Bloom filter answers 'definitely new' in O(1) from memory and rejects the vast majority of candidates — mostly re-discovered links to already-crawled pages — without a network round trip. The DB is only consulted on the rare positive match, where its false-positive tolerance costs nothing but an extra read.
05
Crawler trap prevention
ChoseMax-depth cutoff (15–20 hops from seed), tracked per URL in the Metadata DB
OverUnbounded depth, relying solely on URL-level dedupA fixed total crawl-time budget with no per-branch limit
Session IDs, calendar widgets, and faceted search filters generate effectively infinite unique URLs down a single path — URL dedup alone never triggers because every link really is new. Tracking depth from the nearest seed and cutting off a branch past a threshold bounds the worst case cheaply, without needing to detect the trap pattern itself.
06
Storage split: raw HTML vs. extracted text
ChoseTwo separate blob storage locations — raw HTML and clean text
OverA single blob per URL containing bothDiscard raw HTML immediately after parsing to save space
Storage is cheap; the ~2MB average page fetched over the wire is the expensive resource. Keeping raw HTML around means text-extraction logic can be improved and re-run against the entire 10B-page corpus later without re-crawling a single URL — a rebuild that would otherwise cost another 5-day crawl.
//What Breaks First
Bottlenecks & Failure Modes
Reading a system means sensing where it cracks under 10× load. These are the pressure points of this design, and how it holds.
Risk 01
DNS resolution bottleneck
Failure mode
DNS lookups have historically consumed up to 70% of a naive crawler thread's elapsed time, since every new domain requires a fresh resolution before the first byte can be fetched.
Mitigation
Each fetcher host maintains a local DNS cache, and lookups are round-robined across multiple upstream resolvers so no single resolver becomes the fleet's shared chokepoint.
Risk 02
Frontier queue thundering herd on a popular domain
Failure mode
Many pages across the web link to the same large domain (e.g. a major news site), so a burst of same-domain URLs can land in the Frontier Queue together and race for the same per-domain lock in the Rate Limiter.
Mitigation
The Rate Limiter defers contended fetches via SQS's ChangeMessageVisibility, with jitter added to the delay so deferred workers don't all retry in the same instant and re-create the collision.
Risk 03
Crawler traps / infinite link mazes
Failure mode
Dynamically generated pages — calendar widgets, session-id-bearing links, faceted search — can produce an effectively unbounded number of unique URLs down a single branch, each one passing URL-level dedup because it genuinely is new.
Mitigation
Every URL carries a depth value in the Metadata DB. Branches past 15–20 hops from the nearest seed are simply never enqueued, bounding the worst case without needing to detect the trap pattern itself.
Risk 04
Parser falling behind fetchers
Failure mode
Fetching is I/O-bound and cheap to parallelize; parsing is CPU-heavier. If the parser fleet can't keep pace, the Further Processing Queue backs up and raw HTML accumulates in blob storage faster than it's consumed.
Mitigation
Parser workers autoscale directly on Further Processing Queue depth rather than running a fixed fleet size, so parse capacity grows automatically under sustained fetch pressure.
Risk 05
Metadata DB hot keys from popular domains
Failure mode
Every fetch against a popular domain would otherwise trigger a robots.txt / crawl-delay lookup against the same Domain table row, concentrating read load on a handful of partition keys.
Mitigation
The Rate Limiter caches each domain's robots.txt rules and crawl-delay in Redis with a periodic refresh, so the hot read path never touches the Metadata DB per-request.
//Active Recall
Test Yourself
If you can answer these without scrolling back up, the architecture is yours. Try each one out loud before revealing.
//Go Deeper
Further Reading
- RFC 9309 — Robots Exclusion ProtocolIETF
- Amazon SQS Developer Guide — visibility timeout & dead-letter queuesAWS Docs
- RedisBloom — probabilistic Bloom filter data typeredis.io
- How Google crawlers (User-agents) workGoogle Search Central
- Web Crawler — Problem Breakdownhellointerview.com
Patterns used in this design
//A Word
From the Creator
Every great system starts as a sketch on a whiteboard. The ability to zoom out and see the whole picture (how services connect, where data flows, what breaks and why) is what separates engineers who build features from engineers who build systems. This diagram is more than boxes and arrows. It's a map of decisions, trade-offs, and intentional design.
Studying architectures isn't just about passing interviews. It's about training your intuition. The more systems you take apart, the better you get at sensing where a monolith will crack, where a queue belongs, or when a cache is hiding a deeper problem. You start seeing patterns instead of chaos.
So keep reading, keep tracing those edges, keep asking "why this way and not that way." The engineers who truly understand large systems are the ones who never stop being curious about how things fit together. That curiosity is the only ingredient that really matters.
Atharva Arbat@arbat_atharva
//Explore
More Patterns
Design an API Rate Limiter
A server-side rate limiting system that protects APIs from abuse using sliding window counters and Redis-backed distributed state.
Design Typeahead Suggestion / Autocomplete
A ranked prefix-completion system that returns the top-K suggestions within the keystroke latency budget using an in-memory precomputed trie and an offline popularity build pipeline.