Scalability

Scalability in System Design: The Complete Interview Guide

Vertical vs. horizontal scaling, load balancing, caching, sharding, queues — and how to talk about all of it in a 45-minute interview.

Almost every system design interview is a scalability interview wearing a costume. The prompt sounds like a product question — "design a URL shortener," "design Instagram's feed," "design a rate limiter" — but the follow-up is always the same: now make it work for 100 million users. Candidates who have memorized a list of buzzwords stall at that point. Candidates who understand where systems actually break keep going.

This guide covers what scalability means, the handful of techniques that do almost all the work, and a repeatable way to structure your answer.

What scalability actually means

Scalability is the ability of a system to handle increased load by adding resources. That last clause matters. A system isn't scalable because it's fast today; it's scalable because doubling the hardware roughly doubles the work it can do.

Two words get confused constantly in interviews:

  • Performance — how fast a single request is served under a fixed load. Fixing performance means making the code, query, or algorithm cheaper.
  • Scalability — how gracefully throughput grows as you add machines. Fixing scalability means removing shared bottlenecks.

A single-threaded service with a global lock can be extremely fast with one user and completely unscalable with ten thousand. Conversely, a slow-but-stateless service can be scaled by throwing servers at it. Interviewers listen for whether you know which problem you're solving.

The three numbers you should always name

Metric Definition Typical target
Latency Time to serve one request. Always quote percentiles (p50, p95, p99), never averages. p99 < 200 ms for user-facing reads
Throughput Requests (or bytes, or messages) handled per second. Derived from your traffic estimate
Availability Fraction of time the system serves correct responses. 99.9% ≈ 43 min downtime/month

Why percentiles: if 1% of requests take 5 seconds and a page makes 20 backend calls, roughly 1 in 5 page loads is slow. The average hides that entirely. Say "p99" out loud at least once in your interview.

Vertical vs. horizontal scaling

This is the fork in the road, and it's the first design decision an interviewer expects you to make explicitly.

Vertical (scale up) Horizontal (scale out)
Approach Bigger machine — more CPU, RAM, faster disk More machines behind a load balancer
Complexity Low — no code changes High — needs statelessness, coordination, distributed data
Ceiling Hard limit at the largest instance available Effectively unbounded
Fault tolerance Single point of failure Redundant by construction
Cost curve Superlinear — big iron is expensive Roughly linear on commodity hardware

The mature answer is not "always scale horizontally." Vertical scaling is the correct first move surprisingly often: it buys months of runway for one afternoon of work, and relational databases in particular scale up very well. The right framing is:

"I'd scale the stateless web tier horizontally from day one, since it's cheap to do. For the database I'd scale vertically first, add read replicas next, and only shard when a single primary can't absorb the write volume — sharding is a one-way door."

The scalability toolkit

Six techniques cover the vast majority of interview scenarios. Learn what each one costs, not just what it does.

1. Load balancing

A load balancer sits in front of a server pool, spreads requests across healthy instances, and removes dead ones from rotation. Common algorithms:

  • Round robin — simple rotation. Fine when requests are uniform.
  • Least connections — routes to the least busy server. Better when request cost varies a lot.
  • Consistent hashing — maps a key (user ID, cache key) to a specific server so the same key lands on the same node, and only a small fraction of keys move when the pool changes size. This is the workhorse behind sharded caches.

Two details that impress: health checks (how does the LB know a node is dead?) and the load balancer itself is a single point of failure — you need at least an active-passive pair with a floating IP, or DNS-level distribution across several.

2. Statelessness

Horizontal scaling only works if any server can serve any request. That means no session data in local memory and no reliance on sticky routing. Push state outward:

  • Session data → Redis, or a signed token (JWT) held by the client
  • Uploaded files → object storage (S3 and equivalents), never local disk
  • In-process caches → acceptable, but they must be disposable

Statelessness is the enabling condition for autoscaling, blue-green deploys, and casually terminating misbehaving instances. It's the single highest-leverage property in the whole list.

3. Caching

Caching is the cheapest way to add read capacity, because most systems are read-heavy by a wide margin. Layers, from client to origin:

Browser cache
  → CDN edge
    → Load balancer / reverse proxy cache
      → Application cache (Redis, Memcached)
        → Database buffer pool
          → Disk

Write strategies:

  • Cache-aside (lazy loading) — app checks cache, falls back to DB on a miss, then populates. Default choice.
  • Write-through — write to cache and DB together. Consistent, higher write latency.
  • Write-behind — write to cache, flush to DB asynchronously. Fast, risks data loss.

Eviction and invalidation are where the real questions live. LRU is the standard eviction policy. For invalidation, TTLs are simple and usually sufficient; explicit invalidation on write is precise but easy to get wrong. Name these failure modes before the interviewer does:

  • Thundering herd / cache stampede — a hot key expires and thousands of requests hit the database simultaneously. Fix with request coalescing (a single in-flight fetch per key), a short lock, or jittered TTLs.
  • Hot key — one celebrity's profile overwhelms a single cache node. Fix by replicating that key across nodes or adding a local in-process cache.
  • Cold start — a freshly deployed cache passes all traffic to the origin. Warm it, or roll out gradually.

4. Database scaling

The database is where nearly every real system hits its wall. Apply these roughly in order:

Indexing and query tuning. Unglamorous, and frequently worth a 10× improvement for zero architectural cost. Mention it first; skipping straight to sharding reads as inexperience.

Read replicas. One primary takes writes and streams changes to replicas that serve reads. Excellent for read-heavy workloads. The cost is replication lag: a user posts a comment, the read hits a stale replica, and the comment appears to vanish. Handle it by routing reads-after-write to the primary, or by pinning a user's session to the primary for a few seconds.

Vertical partitioning (federation). Split by feature area — users on one database, orders on another. Simple and effective until you need a join across them.

Horizontal partitioning (sharding). Split rows of one table across databases by a shard key. This is the real scaling lever for writes, and the most expensive to reverse.

Shard key choice drives everything:

  hash(user_id) % N   → even spread, but range queries fan out
                        to every shard, and resizing reshuffles data
  consistent hash     → even spread, cheap to add/remove nodes
  range (created_at)  → efficient time-range scans, but the newest
                        shard becomes a write hotspot
  geography / tenant  → natural isolation, risk of a whale tenant
                        outgrowing a single shard

What sharding costs you: cross-shard joins (do them in the application layer, or denormalize), distributed transactions (avoid; reach for the saga pattern instead), globally unique IDs (use Snowflake-style IDs or a dedicated ID service rather than auto-increment), and rebalancing. A common trick is to shard into many logical shards up front and map several to each physical node, so growth is a matter of moving logical shards rather than rehashing keys.

Denormalization and NoSQL. When a query is too expensive across a normalized schema, precompute it — store data in the shape it's read in. This is the core insight behind wide-column and document stores: accept duplicated data and weaker consistency to get single-partition reads.

5. Asynchronous processing

Not every task needs to finish before the response is sent. A message queue (Kafka, RabbitMQ, SQS) between the web tier and workers gives you three things at once:

  • Lower user-facing latency — return 202 Accepted, do the work later. Image transcoding, email, report generation, search indexing.
  • Load smoothing — the queue absorbs traffic spikes that would otherwise topple the workers.
  • Failure isolation — if the email service is down, messages wait instead of erroring out the checkout flow.

The costs are real: eventual consistency in the user experience, the need for idempotent consumers (most queues guarantee at-least-once delivery, so duplicates will happen), dead-letter queues for poison messages, and queue depth as a metric you must alarm on.

6. CDN and edge delivery

A CDN caches static assets — images, video, JS, CSS — at points of presence near users. It cuts latency by shortening the physical distance, and it removes the bulk of bandwidth from your origin. For media-heavy products this is often the largest single win available, and it's frequently the first thing candidates forget to mention.

The limits: why scaling isn't linear

Adding machines helps less than you'd hope, for two reasons worth naming in an interview.

Amdahl's law: the speedup from parallelism is capped by the fraction of work that must run serially. If 5% of a request is inherently sequential, you cannot exceed a 20× speedup no matter how many cores you add.

The Universal Scalability Law adds a second, nastier term: coordination cost. Beyond a certain point, throughput doesn't just plateau — it declines, because nodes spend more time coordinating (locks, consensus, chatter) than working. This is why "just add servers" eventually makes a system slower, and why removing shared state matters more than adding capacity.

The practical takeaway: find and eliminate the serial resource. Usually it's a single primary database, a distributed lock, a global counter, or a shared queue partition.

Back-of-the-envelope estimation

Interviewers use estimation to check whether your design is anchored to reality. The method is always the same: start from users, derive requests per second, then storage, then bandwidth.

Example: a photo-sharing service

Users:        100M DAU
Writes:       each user uploads 0.2 photos/day  → 20M uploads/day
              20M / 86,400s                     → ~230 writes/sec average
              peak = 3× average               → ~700 writes/sec
Reads:        100:1 read/write ratio            → ~23,000 reads/sec average

Storage:      avg photo 2 MB + thumbnails ~0.5 MB → 2.5 MB
              20M × 2.5 MB                       → 50 TB/day
              × 365                              → ~18 PB/year  → object storage + CDN,
                                                     never a relational DB

Metadata:     ~500 bytes/photo × 20M            → 10 GB/day → 3.6 TB/year
                                                   → fits comfortably in a sharded DB

Bandwidth:    egress 23,000 reads/sec × 2.5 MB  → ~57 GB/sec → CDN is mandatory

Numbers worth memorizing: 1 day ≈ 86,400 seconds (round to 100,000 for speed), 1 million writes/day ≈ 12/sec, a modern server handles thousands of requests/sec, a single relational primary comfortably handles low thousands of writes/sec, and an SSD read is ~100 µs while a cross-continent round trip is ~150 ms.

Protecting the system under load

Scalability isn't only about growing. It's about degrading gracefully when you can't grow fast enough.

  • Rate limiting — token bucket or sliding window per user/API key, so one client can't consume the whole system.
  • Backpressure — reject or shed load at the edge rather than letting queues grow without bound. A fast 503 is better than a 30-second timeout.
  • Circuit breakers — stop calling a failing dependency for a cooldown period; retrying into an overloaded service is how a partial outage becomes a total one.
  • Retries with exponential backoff and jitter — naive fixed-interval retries synchronize clients into a stampede.
  • Bulkheads — separate connection pools and thread pools per dependency, so one slow downstream can't exhaust every worker.
  • Graceful degradation — serve stale cache, drop the recommendation carousel, disable non-essential writes. Keep the core path alive.

How to structure the interview answer

Use a fixed skeleton so you never freeze. Roughly 45 minutes:

  1. Clarify requirements (5 min). Functional scope, then non-functional: expected DAU, read/write ratio, latency target, consistency needs, data retention. Write them down.
  2. Estimate (5 min). QPS, storage, bandwidth. This is what justifies every later decision.
  3. Draw the simple version (5 min). Client → load balancer → stateless app servers → database. Get agreement before adding anything.
  4. Define the data model and API (5–10 min). Key entities, access patterns, endpoints. Access patterns determine the storage choice, not the reverse.
  5. Scale it deliberately (15 min). Walk the bottlenecks in order: cache the hot reads, add replicas, move heavy work to a queue, put static content on a CDN, shard when writes exceed one primary. State the tradeoff for each step.
  6. Address failure and operations (5 min). What happens when a node, a region, or the cache dies? What do you monitor — p99 latency, error rate, queue depth, replication lag?

The one habit that separates strong candidates: introduce every component as a response to a specific bottleneck, and immediately name what it costs. "I'm adding Redis in front of the profile read path because it's 23,000 QPS against a table that changes rarely. The cost is that a profile edit may take up to 60 seconds to appear, which I think is acceptable here." That sentence pattern is the whole interview.

Common mistakes

  • Sharding on slide one. Distributed complexity without a demonstrated need reads as cargo-culting.
  • Naming technologies instead of properties. "I'd use Cassandra" is weak. "I need high write throughput and can accept eventual consistency, so a wide-column store like Cassandra fits" is strong.
  • Ignoring the write path. Caches and replicas scale reads. If writes are the bottleneck, they don't help at all.
  • Forgetting the load balancer, cache, or CDN is itself a distributed system that can fail, fill up, or become a hotspot.
  • No numbers. Without estimates, every decision is unfalsifiable, and interviewers notice.
  • Silence. Think out loud. An unstated tradeoff scores zero.

Quick revision checklist

Bottleneck First move Tradeoff to state
App CPU saturated More stateless instances behind the LB Requires externalized session state
Repeated expensive reads Redis cache-aside with TTL Staleness; stampede on hot keys
DB read-bound Read replicas Replication lag; read-after-write anomalies
DB write-bound Shard by a well-chosen key No cross-shard joins or transactions
Slow request path Queue + async workers Eventual results; needs idempotency
Bandwidth / global latency CDN + object storage Cache invalidation across edges
Abusive or spiky clients Rate limit + circuit breakers Some legitimate traffic gets rejected

Frequently asked interview questions

Is horizontal scaling always better than vertical? No. Vertical scaling is simpler, requires no code changes, and is often the right first step — especially for databases. Horizontal scaling wins on fault tolerance and has no ceiling, which is why stateless tiers should be built for it from the start.

How do you keep a load balancer from being a single point of failure? Run redundant load balancers in active-passive with a floating IP or health-checked DNS, and distribute across availability zones.

What breaks first as traffic grows? Almost always the database, and usually the write path. Reads can be cached and replicated; writes must eventually be partitioned.

How do you pick a shard key? Choose one that spreads load evenly and matches your dominant access pattern so most queries hit a single shard. Beware of monotonically increasing keys (hotspots on the newest shard) and low-cardinality keys.

Does scalability conflict with consistency? Often, yes. Distributing data forces a choice under network partitions between staying available and staying consistent. Most large consumer systems pick availability with eventual consistency, and reserve strong consistency for narrow paths like payments and inventory.

Wrapping up

Scalability is a discipline of finding the one resource everything else waits on, and removing it — then doing it again at the next order of magnitude. The techniques are few and well understood: keep the compute tier stateless, cache aggressively, replicate reads, partition writes, push slow work into queues, serve bytes from the edge, and shed load deliberately when you must.

In an interview, what earns the offer isn't reciting that list. It's starting simple, measuring before adding, and naming the cost of every component you introduce. Practice on three or four classic prompts — a URL shortener, a news feed, a chat service, a rate limiter — and narrate your estimates out loud each time. The vocabulary becomes automatic faster than you'd expect.

Next in this series: availability and fault tolerance, the CAP theorem in practice, and how to design for multi-region deployments.