Design a Web Crawler
A distributed crawler that discovers, fetches, and extracts text from 10 billion web pages within 5 days while respecting robots.txt.
Hexagonal Architecture
Ports and adapters pattern — core domain logic isolated from infrastructure concerns.
//The Contract
Requirements
Functionalwhat the system must do
- ✓The domain core can be exercised in complete isolation, with no real database, queue, or HTTP framework involved
- ✓The same business logic can be triggered by an HTTP request or a queued message without duplicating any rule
- ✓Swapping PostgreSQL for a different datastore, or REST for gRPC, never requires touching the domain core
- ✓New driving adapters — a CLI, a scheduled job, a second queue — can be added without modifying existing ones
- ✓The core can call out to multiple secondary systems (database, object storage, payment provider) through explicit, narrow interfaces
Non-functionalhow well it must do it
- ≫Dependency direction is enforced inward — adapters depend on the core's ports; the core never imports adapter code
- ≫Business rules are covered by fast, in-memory unit tests instead of slow integration tests against real infrastructure
- ≫A failure or slow response from one secondary adapter (the payment provider) is isolated behind its port and never leaks a raw infrastructure exception into the domain
- ≫A new technology choice — a different database engine, a different message broker — is an adapter-level concern, not a rewrite of business logic
//Back of the Envelope
Scale Estimates
Rough, order-of-magnitude numbers. The point is to justify the architecture, not to be exact.
Core test coverage
>90%
achievable with pure in-memory unit tests, no infra needed
Adapters per port (typical)
2-3
e.g. Postgres in production, an in-memory fake in tests
Driving adapters here
2
REST and Queue both call the exact same inbound port
Driven adapters here
3
DB, storage, and payment — swappable independently of each other
//Follow the Data
Request Flows
A diagram shows what exists; a trace shows what happens. Press play on a flow, or click any step, and the diagram above lights up the exact path that request takes.
Flow 01
Handle an inbound HTTP command
How a request crosses the boundary into the core and back out through its ports.
Flow 02
Handle the same action from a queue
The whole point of hexagonal architecture: a second entry point, zero duplicated logic.
//Component Breakdown
Key Components
Every box on the canvas, and the job it does. Click a card to locate it in the diagram; click the node itself for the full deep dive.
client×1
network×1
compute×2
database×1
storage×1
external×1
//Deep Dive
Architecture Breakdown
Isolating the Business Rules
Most architectures let business logic quietly absorb infrastructure concerns — a service class that calls an ORM directly, a domain function that constructs SQL. Hexagonal Architecture (ports and adapters) draws a hard line around the Domain Core and forces every dependency to cross it through an explicit, named interface: a port.
Nothing outside the core is allowed to define what the core needs. The core defines its own ports, in its own vocabulary, and it is the adapter's job to satisfy that contract.
Driving Adapters — Getting Requests In
The Inbound Adapter (REST) and Inbound Adapter (Queue) sit on the left side of the hexagon. Both translate their own input format — an HTTP request, a queued message — into the exact same call on the core's inbound port.
This is the pattern's clearest payoff: the core cannot tell which adapter invoked it, so a bug fix or a new rule is written once and applies to both entry points immediately.
Driven Adapters — Calling Out
On the right side, the core depends on three separate outbound ports instead of one shared interface:
- A database port for persistence
- A storage port for files
- A payment port for charging the customer
Keeping these narrow and separate means a slow or failing payment provider is isolated behind its own port — it never has a code path into the database adapter, and swapping any one of the three never touches the others.
What This Buys You
Because the core depends only on ports, it can hit >90% test coverage using in-memory fakes instead of real infrastructure. Because adapters are the only things that know about HTTP, SQL, or message queues, replacing any piece of infrastructure is an adapter-level change, not a domain rewrite.
What It Costs
The pattern adds real ceremony — interfaces, boundary mappers, extra test doubles — that isn't worth paying for trivial CRUD. It earns its keep specifically where business logic is nontrivial, has more than one entry point, or is genuinely expected to outlive today's choice of database or transport.
//Trade-offs
Design Decisions
An architecture is a record of trade-offs. For every major choice here: what won, what lost, and why the constraints made it so.
01
Business logic never imports infrastructure
ChosePorts (interfaces) are owned and defined by the core; adapters implement them
OverDomain services call ORM/HTTP client libraries directly
If the core imported a database driver, replacing that driver would mean editing business logic to do it. Owning the port interface inverts the dependency: the core defines what it needs in its own vocabulary, and it's the adapter's job to satisfy that contract — not the other way around.
02
Two driving adapters, one core
ChoseThe REST adapter and the Queue adapter both call the same inbound port
OverDuplicate the business logic inside the queue consumer for speed
Duplicating logic in a second adapter guarantees the two paths drift apart the first time one gets a bug fix the other doesn't. Funneling both into one port means the >90% unit-tested core is exercised identically no matter which adapter triggered it.
03
Three narrow outbound ports instead of one generic repository
ChoseSeparate database, storage, and payment ports
OverOne broad 'InfrastructureService' interface the core depends on
A single generic interface hides very different failure modes behind one name — a slow payment provider looks the same as a slow disk write. Narrow, named ports let each adapter fail, retry, or time out on its own terms, and a payment provider outage never has a code path to touch the database port.
04
Explicit boundary object mapping
ChoseAdapters translate between wire formats and domain objects at the boundary
OverPass HTTP DTOs or ORM entities straight into the core
An HTTP DTO or an ORM entity is shaped by its framework's conventions, not the domain's. Leaking either into the core means a REST library upgrade or an ORM migration can silently change what the 'domain object' means — mapping at the boundary keeps the core's vocabulary entirely its own.
//What Breaks First
Bottlenecks & Failure Modes
Reading a system means sensing where it cracks under 10× load. These are the pressure points of this design, and how it holds.
Risk 01
Anemic ports that leak infrastructure details
Failure mode
If a port's interface is shaped around the adapter's data model — say, a method that takes a raw SQL row — swapping that adapter still forces a rewrite of the core. The abstraction exists in name only.
Mitigation
Define every port in the domain's own vocabulary ("reserveInventory", not "updateRow"), and have the adapter do all the translation work in both directions.
Risk 02
A slow or unavailable payment provider blocking the core
Failure mode
If the core calls its payment port synchronously and the underlying provider is degraded, the core's own execution stalls waiting on a dependency it was supposed to be isolated from.
Mitigation
Put timeouts and a circuit breaker inside the payment adapter itself, so the core only ever sees a clean domain-level failure (e.g. PaymentDeclined or PaymentUnavailable) — never a raw HTTP timeout.
Risk 03
Overhead for trivial CRUD
Failure mode
Wrapping a simple create-read screen in ports and adapters can be more ceremony — interfaces, mappers, boundary objects — than the feature is worth.
Mitigation
Reserve strict hexagonal boundaries for logic that is genuinely nontrivial, has multiple entry points, or needs its infrastructure swapped; let simple CRUD skip the extra layers.
Risk 04
Testing adapters gets neglected
Failure mode
Teams often unit-test the core exhaustively (hence the >90% figure) but skip integration tests on the adapters, so the seams — SQL mapping, HTTP status codes, message serialization — go unverified until production.
Mitigation
Run a small contract-test suite per adapter against the real dependency (or a faithful test double) in CI, separate from the core's unit tests.
//Active Recall
Test Yourself
If you can answer these without scrolling back up, the architecture is yours. Try each one out loud before revealing.
//Go Deeper
Further Reading
- Alistair Cockburn — Hexagonal Architecture (the original pattern)alistair.cockburn.us
- Netflix Tech Blog — Ready for changes with Hexagonal Architecturenetflixtechblog.com
- Herberto Graça — Ports & Adapters Architectureherbertograca.com
- Domain-Driven Design Communitydddcommunity.org
Patterns used in this design
//A Word
From the Creator
Every great system starts as a sketch on a whiteboard. The ability to zoom out and see the whole picture (how services connect, where data flows, what breaks and why) is what separates engineers who build features from engineers who build systems. This diagram is more than boxes and arrows. It's a map of decisions, trade-offs, and intentional design.
Studying architectures isn't just about passing interviews. It's about training your intuition. The more systems you take apart, the better you get at sensing where a monolith will crack, where a queue belongs, or when a cache is hiding a deeper problem. You start seeing patterns instead of chaos.
So keep reading, keep tracing those edges, keep asking "why this way and not that way." The engineers who truly understand large systems are the ones who never stop being curious about how things fit together. That curiosity is the only ingredient that really matters.
Atharva Arbat@arbat_atharva
//Explore
