Skip to main content
System Design Mafia
Back to patterns

Event-Driven Architecture

Asynchronous event producers and consumers connected through a message broker for decoupled communication.

//The Contract

Requirements

Functionalwhat the system must do
  • ✓A single business action (e.g. an order being placed) can trigger any number of independent downstream reactions without the producer knowing what they are
  • ✓New consumers can subscribe to existing events and start reacting to them without any change to the producer or to other consumers
  • ✓A consumer that is temporarily down does not lose events — it catches up from where it left off once it recovers
  • ✓A message that repeatedly fails processing is isolated so it cannot block every other message behind it
  • ✓Each consumer maintains its own view of state, shaped for what it specifically needs to do
Non-functionalhow well it must do it
  • ≫Producers and consumers are decoupled in time, in deployment, and in failure — one going down never blocks the other
  • ≫The broker durably retains events so consumers can be added later and still process history, not just what happens after they subscribe
  • ≫Throughput scales by adding consumer instances and broker partitions, not by making any single service faster
  • ≫Delivery is at-least-once — consumers must be able to safely process the same event more than once
  • ≫Ordering is guaranteed only within a partition, not globally across the whole event stream
//Back of the Envelope

Scale Estimates

Rough, order-of-magnitude numbers. The point is to justify the architecture, not to be exact.

Events published
50K/sec
peak producer throughput the broker must sustain
Consumer groups
10+
each new feature is a new subscriber, not a producer change
Retention window
7 days
how far back a newly added consumer can replay history
Retry budget
5 attempts
failures beyond this route to the dead-letter queue
Consumer lag alert
>10s
the threshold that pages on-call before staleness becomes visible to users
//Follow the Data

Request Flows

A diagram shows what exists; a trace shows what happens. Press play on a flow, or click any step, and the diagram above lights up the exact path that request takes.

Flow 01

Publish an event and fan it out

From a client action to two independent consumers reacting in parallel.

Flow 02

A poison message and the dead-letter queue

What happens when one event can never be processed successfully.

//Component Breakdown

Key Components

Every box on the canvas, and the job it does. Click a card to locate it in the diagram; click the node itself for the full deep dive.

client×1
network×1
compute×3
database×1
queue×2
//Deep Dive

Architecture Breakdown

Decoupling Through Events

In a request-driven system, a producer that wants two things to happen must call both directly — and now depends on both being available. Event-Driven Architecture breaks that dependency: a producer publishes a fact about what already happened, and any number of consumers react to it independently, without the producer ever knowing they exist.

The Broker Is the Contract

The Message Broker is a durable, partitioned log, not a transient mailbox. Every event a producer publishes is retained for a fixed window (7 days here), which means:

  • A consumer can be added next month and still replay history from before it existed
  • A consumer that crashes for an hour resumes exactly where it left off, losing nothing
  • Multiple consumer groups can each read the entire event stream independently — one group's progress never affects another's

Fan-Out in Practice

Consumer A and Consumer B in this diagram both subscribe to the same events but belong to different consumer groups, so the broker delivers every event to both. Each writes its own resulting state, shaped for its own purpose — this is what makes it safe to keep adding consumers without ever touching the producer.

What This Trades Away

Decoupling isn't free. Delivery is at-least-once, so consumers must tolerate duplicate processing. Ordering is only guaranteed within a partition, so anything that depends on strict global order needs its partition key chosen carefully. And a message that can never be processed successfully needs somewhere to go besides an infinite retry loop — which is exactly what the dead-letter queue is for.

//Trade-offs

Design Decisions

An architecture is a record of trade-offs. For every major choice here: what won, what lost, and why the constraints made it so.

01

Producer publishes instead of calling consumers

ChosePublish one event to a broker
OverProducer calls each consumer's API directly (orchestration)

With 10+ consumer groups and growing, direct calls would mean every new feature requires a code change and a redeploy of the producer. Publishing an event once lets any number of consumers subscribe — including ones that don't exist yet — with zero changes to the producer.

02

A durable, partitioned log instead of a transient queue

ChoseKafka-style broker with consumer groups and a 7-day retention window
OverIn-memory pub/sub with no retentionA queue where each message is consumed exactly once by exactly one worker

Retention means a consumer added next month can replay the last 7 days of history instead of starting blind. Consumer groups mean Consumer A and Consumer B can both process every event independently — a plain queue would hand each message to only one of them.

03

Each consumer owns its own state

ChoseConsumer A and Consumer B write independently, each shaped for its own use case
OverA single shared service owns all writes to the database on consumers' behalf

Coupling consumers to a shared writer reintroduces the exact coordination cost events were meant to remove — a schema change for one consumer risks breaking another. In this diagram they share one database for simplicity; at larger scale each consumer typically owns its own datastore entirely.

04

Dead-letter queue for poison messages

ChoseRoute messages that exceed the retry budget to a DLQ
OverRetry foreverDrop the failing message silently

Retrying forever lets one malformed event stall an entire partition behind it; dropping it silently loses data with no trace. A DLQ (see the poison-message flow) keeps the main stream moving while giving an operator a concrete queue to inspect, fix, and replay from.

//What Breaks First

Bottlenecks & Failure Modes

Reading a system means sensing where it cracks under 10× load. These are the pressure points of this design, and how it holds.

Risk 01

Consumer lag under load

Failure mode

If the broker accepts events faster than a consumer can process them, its backlog grows continuously. Downstream state falls further and further behind — sometimes past the point where it's still useful.

Mitigation

Autoscale consumer instances against a lag metric, not CPU, and set partition count as the hard ceiling on parallelism — the consumer lag alert at >10s is the signal that catches this before it's visible to users.

Risk 02

A poison message blocking a partition

Failure mode

Kafka-style brokers guarantee order only by processing a partition's messages sequentially. A message the consumer can never successfully process would otherwise stall every message queued behind it in that partition indefinitely.

Mitigation

The dead-letter queue (see the poison-message flow) removes the offending message after the retry budget is exhausted, letting the partition continue while the bad message waits for manual inspection.

Risk 03

At-least-once delivery causing duplicate processing

Failure mode

Broker redelivery on failure, timeout, or consumer-group rebalance means the same event can reach a consumer more than once. A consumer that isn't expecting this will double-apply the side effect.

Mitigation

Consumers key writes by the event's unique ID and make the write idempotent (upsert instead of insert-only), so processing the same event twice produces the same end state as processing it once.

Risk 04

Ordering only guaranteed within a partition

Failure mode

Two events for the same entity that land in different partitions can be processed out of order — a 'cancel' could be processed before the 'create' it depends on.

Mitigation

Choose a partition key derived from the entity ID itself, so every event for the same entity always lands in the same partition and is processed strictly in order relative to each other.

//Active Recall

Test Yourself

If you can answer these without scrolling back up, the architecture is yours. Try each one out loud before revealing.

//Go Deeper

Further Reading

Patterns used in this design
CQRS PatternE-Commerce Microservices
//A Word

From the Creator

Every great system starts as a sketch on a whiteboard. The ability to zoom out and see the whole picture (how services connect, where data flows, what breaks and why) is what separates engineers who build features from engineers who build systems. This diagram is more than boxes and arrows. It's a map of decisions, trade-offs, and intentional design.

Studying architectures isn't just about passing interviews. It's about training your intuition. The more systems you take apart, the better you get at sensing where a monolith will crack, where a queue belongs, or when a cache is hiding a deeper problem. You start seeing patterns instead of chaos.

So keep reading, keep tracing those edges, keep asking "why this way and not that way." The engineers who truly understand large systems are the ones who never stop being curious about how things fit together. That curiosity is the only ingredient that really matters.

Atharva Arbat@arbat_atharva

//Explore

More Patterns