Skip to main content
System Design Mafia
Back to patterns

Design an API Rate Limiter

A server-side rate limiting system that protects APIs from abuse by controlling request volume using sliding window counters and Redis-backed distributed state.

//The Contract

Requirements

Functionalwhat the system must do
  • ✓The system limits the number of requests a user, IP address, or API key can make within a configurable time window
  • ✓The system returns HTTP 429 (Too Many Requests) with a Retry-After header when a caller exceeds their limit
  • ✓Rate limit rules are configurable per endpoint, user tier, geographic region, and authentication status
  • ✓The system logs all rate-limited requests for real-time monitoring, dashboards, and retrospective abuse analysis
  • ✓The rate limiter supports multiple algorithms — sliding window log, sliding window counter, token bucket, and fixed window — with per-route selection
Non-functionalhow well it must do it
  • ≫Decision latency must stay under 5ms at P99 — rate limiting adds negligible overhead to every single API call
  • ≫The system must handle 10M+ requests per second at peak without becoming a bottleneck itself
  • ≫Rate limit counters must survive individual middleware instance restarts — state lives in Redis, not in process memory
  • ≫The rate limiter must gracefully degrade under Redis failure — falling back to approximate local counters rather than failing open or closed
//Back of the Envelope

Scale Estimates

Rough, order-of-magnitude numbers. The point is to justify the architecture, not to be exact.

Peak RPS
10M+
rate limiter decides on every request — throughput is the top constraint
Decision latency
<5ms
P99 — users should never feel the rate limiter
Active users
1B
counter keyspace must handle billions of distinct identifiers
Redis ops/sec
20M+
each request is a read + conditional write, doubling the request rate
Rules count
~500
loaded into local cache at startup, refreshed every few minutes
//Follow the Data

Request Flows

A diagram shows what exists; a trace shows what happens. Press play on a flow, or click any step, and the diagram above lights up the exact path that request takes.

Flow 01

Request within rate limit

A user sends an API request that falls within their allowed quota. The rate limiter checks, approves, and forwards it to the backend.

Flow 02

Request blocked (rate limited)

A user or attacker exceeds their rate limit. The middleware rejects the request before it ever reaches the backend.

//Component Breakdown

Key Components

Every box on the canvas, and the job it does. Click a card to locate it in the diagram; click the node itself for the full deep dive.

client×1
network×1
compute×4
cache×1
//Deep Dive

Architecture Breakdown

Core Architecture

The rate limiter sits as middleware between the API Gateway and backend services. Every inbound request passes through it, where a lightweight decision engine checks the caller's current usage against configured rules before deciding to allow, throttle, or block.

  • The Rules Engine stores rate limit definitions — e.g. "100 requests per minute for free-tier users, 1000 for premium" — keyed by endpoint, user tier, or geography
  • The Redis Counter Store holds per-identifier counters with automatic TTL expiry. For sliding window log mode, Redis sorted sets store each request's timestamp; for sliding window counter mode, only two integers per window are needed
  • Allowed requests flow through to the backend API Service. Blocked requests receive HTTP 429 (Too Many Requests) with a Retry-After header
  • A Logging & Analytics pipeline consumes rate-limited events asynchronously for dashboards, abuse detection, and rule tuning

Algorithm Choices

The design supports multiple algorithms, configurable per-route:

  • Sliding Window Log — most accurate; stores every request timestamp in a Redis sorted set, evicts expired entries, then counts remaining entries. O(log N) per check
  • Sliding Window Counter — memory-efficient; approximates the sliding window by weighting the previous window's counter. Only 2 counters per identifier
  • Token Bucket — intuitive; refills tokens at a fixed rate. Burst-friendly but requires tuning bucket depth
  • Fixed Window Counter — simplest; resets a counter at calendar-aligned boundaries. Prone to burst-at-boundary traffic spikes

Distributed Considerations

In a multi-datacenter deployment, all rate limiter instances share the same Redis cluster. Cross-region replication keeps counters eventually consistent — slight over-counting during synchronization windows is acceptable compared to the latency cost of synchronous coordination. Local in-process counters serve as a fallback during Redis degradation, providing approximate rate limiting until the cluster recovers.

//Trade-offs

Design Decisions

An architecture is a record of trade-offs. For every major choice here: what won, what lost, and why the constraints made it so.

01

Server-side vs client-side rate limiting

ChoseServer-side middleware in the API gateway
OverClient-side enforcement in the mobile/web app

Client-side rate limiting is trivially bypassed by modifying client code or disabling JavaScript. Server-side enforcement at the API gateway guarantees every request — regardless of origin — passes through the same gate. This is the only placement that can protect against both malicious attackers and buggy clients.

02

Counter storage: Redis vs relational database

ChoseRedis with atomic counters and sorted sets
OverPostgreSQL with row-level countersIn-process memory within the middleware

Rate limiting demands sub-millisecond reads and writes on the hot path of every API request. Redis delivers this with in-memory storage and atomic ops like INCR and ZREMRANGEBYSCORE. A relational database adds 10–50ms of query latency per check, and its durability guarantees are unnecessary because lost counters only mean a temporary over-allocation. In-process counters can't scale across instances without a shared store.

03

Which rate limiting algorithm

ChoseSliding window log with Redis sorted sets
OverFixed window counterToken bucketLeaky bucket

Fixed window counters suffer from burst-at-boundary spikes — 100 requests at 0:59 and 100 more at 1:00 double the effective rate. Token buckets need careful burst tuning. Sliding window log uses Redis sorted sets (ZSET) to store timestamps and evict expired entries in O(log N), providing precise per-window accounting. The memory cost is acceptable: at 100 req/user with 5M active users, that's roughly 5 GB of timestamp data.

04

Middleware vs standalone service

ChoseMiddleware deployed within the API gateway process
OverDedicated rate limiter service as a separate deployment

Embedding the rate limiter as middleware avoids an extra network hop on the critical path — every request would need two hops instead of one if the limiter were a separate service. For a component that must decide on every single request, that latency penalty is unacceptable. A standalone service only makes sense when the rate limiter must be shared across completely independent API gateways.

05

Throttling response strategy

ChoseHard blocking with HTTP 429
OverThrottling (queue and delay requests)Shaping (deprioritize over-limit requests)

Hard blocking is the simplest and most predictable strategy for API protection. HTTP 429 with Retry-After tells the client exactly what happened and when to retry. Throttling requires buffering and queuing, which adds complexity and can mask capacity problems. Shaping requires priority scheduling across the entire backend — too invasive for most systems.

//What Breaks First

Bottlenecks & Failure Modes

Reading a system means sensing where it cracks under 10× load. These are the pressure points of this design, and how it holds.

Risk 01

Redis hot key contention

Failure mode

A viral user or a botnet sharing a single IP generates millions of requests, concentrating all counter reads and writes on one Redis shard. That shard becomes a hot spot, latency spikes, and the rate limiter itself starts timing out.

Mitigation

Shard the keyspace by user ID or a composite key (user_id + endpoint) so load distributes evenly across Redis cluster nodes. For the hottest keys, use local in-process approximate counters that sync to Redis asynchronously — short-term precision loss is acceptable.

Risk 02

Race conditions in distributed counter updates

Failure mode

In a multi-instance deployment, two middleware processes can read the same counter before either writes, both see the count below threshold, and both allow the request through — letting a burst slip past the limit.

Mitigation

Use Redis atomic operations: INCR for fixed-window counters (atomic increment and check), or Lua scripting for sliding window sorted sets (evict, count, add, and check in one atomic script execution). These guarantee correctness without distributed locks.

Risk 03

Clock skew across data centers

Failure mode

Sliding window and fixed window algorithms rely on accurate timestamps. If middleware instances in different data centers have clock drift of even a few seconds, the counters may systematically over-count or under-count requests across regions.

Mitigation

Use Redis server timestamps via the TIME command instead of the middleware's local clock. For sliding window algorithms, align window boundaries to a server-coordinated epoch rather than local wall-clock time.

Risk 04

Memory exhaustion from sliding window logs

Failure mode

A sliding window log stores every individual request timestamp in a Redis sorted set. At 100 requests per second per user with 10M active users in a 1-minute window, that's 60 billion entries — infeasible for any reasonable Redis cluster.

Mitigation

Switch to the sliding window counter algorithm for high-volume routes: it stores only two integers per window (the previous window's count and the current window's count), reducing memory from O(N) to O(1) per identifier. Reserve sliding window logs for low-volume, high-precision routes like payment APIs.

Risk 05

Cascading failure when Redis degrades

Failure mode

If Redis becomes slow or partially unavailable, every middleware instance may accumulate pending Redis commands. The backlog increases latency for the rate limiter itself, which can cascade into timeouts across all upstream API calls.

Mitigation

Implement a circuit breaker: if Redis P99 latency exceeds 10ms, fall back to approximate local counters that are slightly over-generous (e.g., 110% of the configured limit). This sacrifices precision for availability. Redis Sentinel provides automatic failover for primary nodes.

//Active Recall

Test Yourself

If you can answer these without scrolling back up, the architecture is yours. Try each one out loud before revealing.

//Go Deeper

Further Reading

Patterns used in this design
Design a Web CrawlerDesign Twitter for Millions of Users
//A Word

From the Creator

Every great system starts as a sketch on a whiteboard. The ability to zoom out and see the whole picture (how services connect, where data flows, what breaks and why) is what separates engineers who build features from engineers who build systems. This diagram is more than boxes and arrows. It's a map of decisions, trade-offs, and intentional design.

Studying architectures isn't just about passing interviews. It's about training your intuition. The more systems you take apart, the better you get at sensing where a monolith will crack, where a queue belongs, or when a cache is hiding a deeper problem. You start seeing patterns instead of chaos.

So keep reading, keep tracing those edges, keep asking "why this way and not that way." The engineers who truly understand large systems are the ones who never stop being curious about how things fit together. That curiosity is the only ingredient that really matters.

Atharva Arbat@arbat_atharva

//Explore

More Patterns