Skip to main content
System Design Mafia
Back to patterns

Instagram System Architecture

A photo and video sharing social platform supporting billions of daily media uploads, personalized feed generation, and real-time engagement.

synchronous: caller waits
asynchronous: fire & forget

Denser dots = higher throughput. Hover any edge for its full contract.

//The Contract

Requirements

Functionalwhat the system must do
  • ✓Users can upload photos and videos, which are stored durably with 11 9s of reliability using multipart upload for resumability
  • ✓Users can follow/unfollow other users and view a personalized news feed of recent posts from followed accounts
  • ✓Users can like, comment, and share posts, and receive push notifications for these interactions
  • ✓Users can search for other users and posts based on captions or hashtags with full-text search
  • ✓The system must support ephemeral stories that disappear after 24 hours with separate expiration logic
Non-functionalhow well it must do it
  • ≫High availability — the service must be available 99.99% of the time with no single point of failure and multi-region replication
  • ≫Low latency — News Feed generation must complete within 200ms globally, with Redis absorbing the read path
  • ≫Massive scalability — the system must handle hundreds of millions of daily active users and billions of read/write operations
  • ≫Eventual consistency — it is acceptable for a slight delay in showing new posts or likes, in favor of availability and performance
//Back of the Envelope

Scale Estimates

Rough, order-of-magnitude numbers. The point is to justify the architecture, not to be exact.

Daily Active Users
500M
Generates ~5 billion feed reads per day — every DAU refreshes ~10 times
New Posts / Day
500M
~6,000 writes/sec, with 70% photos and 30% videos — justifies the async pipeline
Read : Write Ratio
~200 : 1
Feed reads and searches dwarf uploads — optimize the read path heavily
Likes / Day
10B
~115,000 writes/sec peak, requiring buffered ingestion via Kafka
Storage / Day
750 TB
Drives the need for a multi-tiered blob storage strategy with CDN caching
//Why It Looks Like This

System Evolution

No system is born with queues and caches. Every box was added because something was about to break. Play a stage and the diagram above shrinks to what existed at that point in the story.

01

The MVP — direct upload, one database, one bucket

Every social platform starts simple. A stateless Post Service issues pre-signed URLs so clients upload photos directly into S3 — media bytes never touch the API tier. Metadata (caption, user_id, timestamp) lands in a sharded PostgreSQL instance.

Timelines are naive: the client requests recent posts from followed users and the database runs a join across the follow graph. This works for a few thousand users, but every feed read is an expensive multi-table query, and every viewer downloads the full-size original photo — a slow, wasteful experience.

02

Feed pre-computation and video transcoding

Scale pressure: Views outnumber uploads 200:1 and every feed read joins across the entire follow graph — database queries take seconds for users with thousands of follows

Stop computing feeds at read time. The Post Service now publishes a 'new post' event to Kafka. A Feed Worker consumes it and writes the post ID to every follower's timeline in Cassandra — partitioned by user_id, clustered by created_at, so reading the top N posts is a single sequential scan.

Video uploads get the same decoupling: a Video Transcoder consumes the same event and processes raw videos into multiple renditions asynchronously. The Feed Service (behind its own API Gateway) serves the pre-computed timeline, and the system now scales read traffic without crushing the relational database.

03

Engagement, search, and notifications

Scale pressure: Likes and comments produce 115K writes/sec at peak — synchronous processing would stall the API servers

Social interactions are write-heavy and latency-tolerant, which makes them a natural fit for the same async pattern. The Engagement Service processes likes, comments, and shares by publishing events to a dedicated Kafka topic. Workers consume these events to update counters in Redis (fast reads) and materialize rows in PostgreSQL (durable storage).

A Search Service backed by Elasticsearch indexes post captions and user profiles for full-text search. The Notification Service consumes engagement events and delivers push notifications via APNs/FCM. Every new subsystem is decoupled by the same queue — the architecture's async backbone.

04

Scale with caching and hybrid fan-out

Scale pressure: Celebrities with 100M followers cause a write explosion — fanning out a single post to every follower's timeline writes 100M rows

Two optimizations finish the architecture. First, a Redis cache in front of the feed timeline stores each user's top 100 post IDs in memory — the hottest reads never touch Cassandra. Second, a hybrid fan-out strategy switches celebrity accounts to fan-in-on-read: their posts are NOT pre-written to followers' timelines; instead, the Feed Service merges them from a dedicated Posts DB at read time.

With these additions, the system handles the full Instagram scale: 500M DAUs, 500M new posts per day, 10B likes, and sub-200ms feed generation for every user — whether they follow 200 people or 200 million.

//Follow the Data

Request Flows

A diagram shows what exists; a trace shows what happens. Press play on a flow, or click any step, and the diagram above lights up the exact path that request takes.

Flow 01

Upload a photo

The write path for a new photo — from client to object storage, with metadata persistence and async jobs.

Flow 02

Load the home feed

The read path — how a user's personalized feed is generated and served with sub-200ms latency.

//Component Breakdown

Key Components

Every box on the canvas, and the job it does. Click a card to locate it in the diagram; click the node itself for the full deep dive.

client×3
network×3
compute×8
database×4
cache×3
queue×2
storage×1
//Deep Dive

Architecture Breakdown

1. Media Upload & Processing Pipeline

Handles the ingestion and storage of user-uploaded photos and videos. The client requests a pre-signed URL from the Post Service and uploads the media directly to Object Storage (S3). The Post Service saves the metadata (caption, user_id, etc.) in a sharded relational database (PostgreSQL) and publishes a 'new post' event to Kafka to trigger asynchronous processes like feed generation and video transcoding. Video transcoding is handled by a dedicated async worker service that processes the raw video into multiple renditions.

2. Feed Generation & Read Path

Focuses on serving a user's personalized news feed with low latency (<200ms). A hybrid fan-out approach is used: for normal users with a moderate number of followers, posts are pre-computed and pushed to a Feed Timeline table (Cassandra) and cached in Redis (fan-out-on-write). For celebrity users with millions of followers, posts are fetched on-demand (fan-in-on-read) and merged with the cached feed for the user. The Feed Service retrieves post IDs from the timeline and then fetches the full post metadata, with results served from the Redis cache.

3. Engagement & Notification Service

Manages user interactions (likes, comments) and notifications. The Engagement Service asynchronously processes likes and comments via a message queue (Kafka) to update databases and caches. The Notification Service consumes events from Kafka and sends push notifications to users via APNs/FCM. The system uses a sharded relational database for core data (posts, users, comments) and a wide-column store (Cassandra) for feed timelines, optimized for fast sequential reads. Elasticsearch is used for full-text search and indexing of posts and user profiles, with results cached in Redis for performance.

//Trade-offs

Design Decisions

An architecture is a record of trade-offs. For every major choice here: what won, what lost, and why the constraints made it so.

01

Clients upload media directly

ChosePre-signed URLs + direct S3 upload
OverProxy uploads through API servers

Video and image uploads can be gigabytes in size. If these bytes flowed through the API tier, it would have to scale with media traffic instead of request traffic, making it expensive and slow. Pre-signed URLs offload the bandwidth and latency cost to the object storage, while the API tier only handles small metadata requests. Multipart upload adds resumability for free: a failed chunk retries alone.

02

Feed generation strategy

ChoseHybrid fan-out (write for normal, read for celebrities)
OverFan-out on write for all usersFan-in on read for all users

Fan-out on write for all users would cause massive write amplification for celebrities with millions of followers — 100M writes per post. Fan-in on read for all users would make feed reads too slow due to many joins per request. The hybrid approach balances this: most users get fast pre-computed feeds from Cassandra/Redis, while celebrity accounts use on-demand reads to avoid the 'write explosion' problem.

03

PhotoID generation

ChoseEpoch time + auto-increment in a dedicated key generation service
OverDatabase auto-increment in each shard

To uniquely identify photos and avoid collisions across shards, we need a globally unique ID. Using epoch time as the most significant bits allows sorting photos by time naturally, and an auto-increment sequence ensures uniqueness. This also enables efficient retrieval of recent posts from a specific user through a sequential index scan.

04

Data sharding strategy

ChosePartition based on PhotoID for even distribution
OverPartition based on UserID

Sharding by UserID keeps all photos of a user on one shard, but creates 'hot spots' for popular users. Sharding by PhotoID with a consistent hash ensures even data distribution and avoids any single shard being overwhelmed by a celebrity's uploads. It also simplifies adding new shards for future growth without rebalancing.

05

Async video processing and engagement

ChoseDedicated worker services consuming from Kafka queues
OverProcess videos synchronously during uploadProcess engagement inline in the API

Video transcoding is CPU-intensive, and engagement writes peak at 115K/sec. Processing either synchronously would block the API server and degrade user experience. By decoupling through Kafka, uploads and engagement actions return instantly, the worker fleets scale independently on queue depth, and failed jobs retry without affecting the user-facing API.

06

Cassandra for feed timelines

ChoseWide-column store partitioned by user_id
OverSharded PostgreSQLRedis as the primary store

The feed timeline workload is write-heavy (fan-out) and requires fast sequential reads of the top N posts — a natural fit for Cassandra's partitioned row storage. PostgreSQL would require complex sharding for the same throughput. Redis lacks the durability and capacity for all users' full timelines, so it works as a cache atop Cassandra, not a replacement.

//What Breaks First

Bottlenecks & Failure Modes

Reading a system means sensing where it cracks under 10× load. These are the pressure points of this design, and how it holds.

Risk 01

Feed: Celebrity post 'write explosion'

Failure mode

When a celebrity with 100 million followers posts a photo, fan-out-on-write would require 100 million writes to the followers' timelines. This could overwhelm Cassandra and Kafka queues simultaneously, causing backpressure across the entire async pipeline.

Mitigation

The hybrid fan-out approach switches celebrities to fan-in-on-read: their posts are fetched from the Posts DB at read time and merged with the cached feed. This avoids the write spike for popular accounts while keeping read latency under 200ms.

Risk 02

Engagement: Thundering herd on viral posts

Failure mode

A viral post can receive millions of likes and comments within minutes. The engagement service, notification service, and databases all see a massive surge — enough to saturate connections and cause cascading timeouts.

Mitigation

Kafka decouples ingestion from processing. The Engagement Service writes each action to the queue almost instantly, then workers drain it asynchronously with batching. Redis caches running counts so reads never hit the database during the storm.

Risk 03

Feed: Cassandra hot partitions

Failure mode

Partitioning the feed timeline by user_id spreads load for most users, but a celebrity with millions of followers still creates a single hot partition when their followers read simultaneously.

Mitigation

The Redis cache absorbs nearly all reads for hot keys. For celebrity fan-in, the read path goes to the Posts DB rather than the timeline at all — Cassandra only stores pre-computed timelines for normal users.

Risk 04

Media: High cost of storage and egress

Failure mode

With 500M new posts per day, storing and serving petabytes of media becomes extremely expensive, especially serving content directly from blob storage to millions of viewers.

Mitigation

A multi-tiered storage strategy: hot tier (frequently accessed content) and cold tier (older content). A global CDN caches media closer to users, reducing egress costs and latency by 90-95% for cache hits. Direct S3 URLs serve as the fallback for rarely-viewed content.

//Break Things on Purpose

Failure Lab

Most of this architecture exists because any one box can die. Take a component offline and the diagram above shows the blast radius: what goes down, what degrades, and what the user feels.

Post Service

REST API

No new posts can be created — the control plane for uploads is down. Existing posts and feed reads continue working because the read path is independent.

Metadata DB (PostgreSQL)

PostgreSQL

Core functionality breaks: no new posts, no likes, no comments, no follows. Existing feed data cached in Cassandra and Redis remains readable but cannot be updated.

Message Queue (Kafka)

Kafka

Uploads complete and metadata persists, but videos are never transcoded, feeds never update, and search indexes stall. Creators see confirmation but followers never see the new post.

Feed Service

REST API

Every feed request fails — the app shows stale cached content or an empty state. Users cannot discover new posts from followed accounts.

Feed Timeline (Cassandra)

Cassandra

Feeds cannot be read from the primary timeline store. Users see either a degraded empty feed or whatever remains in the Redis cache before entries expire.

Feed Cache (Redis)

Redis

Nothing breaks for most users — but every feed read now lands on Cassandra, increasing p99 latency from 1ms to ~12ms. Redis is a cache, not a source of truth.

Message Queue (Kafka)

Kafka

Likes, comments, and shares appear to succeed (the client gets a 200) but never materialize — counters don't increment, notifications don't send. The system accepts writes but processing halts.

Search Index (Elasticsearch)

Elasticsearch

Search returns empty results or errors. Users cannot find posts by caption, hashtag, or search for other users by name.

//Active Recall

Test Yourself

If you can answer these without scrolling back up, the architecture is yours. Try each one out loud before revealing.

//Go Deeper

Further Reading

Patterns used in this design
Event-Driven Architecture
//A Word

From the Creator

Every great system starts as a sketch on a whiteboard. The ability to zoom out and see the whole picture (how services connect, where data flows, what breaks and why) is what separates engineers who build features from engineers who build systems. This diagram is more than boxes and arrows. It's a map of decisions, trade-offs, and intentional design.

Studying architectures isn't just about passing interviews. It's about training your intuition. The more systems you take apart, the better you get at sensing where a monolith will crack, where a queue belongs, or when a cache is hiding a deeper problem. You start seeing patterns instead of chaos.

So keep reading, keep tracing those edges, keep asking "why this way and not that way." The engineers who truly understand large systems are the ones who never stop being curious about how things fit together. That curiosity is the only ingredient that really matters.

Atharva Arbat@arbat_atharva

//Explore

More Patterns