Design an LLM Inference Serving Platform

Your company runs several open-weights language models on its own GPU fleet: a large model for hard reasoning, a small fast model for high-volume tasks, and dozens of fine-tuned variants for individual teams. Product services call a shared internal endpoint and expect streamed tokens back.

GPUs are the most expensive thing the company buys, and they sit idle at night while queues form at noon. Product teams care about two numbers: how quickly the first token arrives and how fast tokens keep coming. Finance cares about tokens per GPU-hour.

Design the serving platform. In 45 minutes, cover how requests are routed to the right model, how a single GPU serves many sequences at once, how memory for attention state is managed, how the fleet scales up and down when a cold replica takes minutes to load, and how you meet per-tier latency targets.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 15 of 18

Ask before you design. Try answering each yourself, then compare.

1. Which models and sizes?
   Typical answer

   One ~70B-class model, one ~8B-class model, and about 50 LoRA fine-tunes of the small model. Weights are open, so we control the serving engine.

2. What hardware?
   Typical answer

   Assume 80 GB data-centre GPUs, 8 per node with fast interconnect inside the node. About 1,500 GPUs in two regions.

3. What traffic?
   Typical answer

   About 100 M requests/day across internal callers, ~3,500/s at peak. Average 2,000 prompt tokens and 300 output tokens; prompts up to 32k.

4. What are the latency targets?
   Typical answer

   Interactive tier: time to first token p95 under 500 ms and at least 40 tokens/s per stream. Batch tier (offline jobs): throughput only, finish within hours.

5. Do callers share prompt prefixes?
   Typical answer

   Yes. Most products send a long fixed system prompt, and chat follow-ups resend the same history.

6. Is quality allowed to change, for example with quantization?
   Typical answer

   Per model owner decision. Assume fp8 weights are approved for the large model after evaluation; small model runs fp16.

Requirements

Functional

• Serve streamed completions for a named model and version, including LoRA variants.
• Support priority tiers (interactive, standard, batch) with different SLOs.
• Roll out new model versions gradually and roll back quickly.
• Report per-caller token usage for chargeback.

Non-functional

• Interactive: TTFT p95 < 500 ms for prompts up to 4k tokens; >= 40 output tokens/s per stream.
• ~3,500 requests/s peak, ~7 M prompt tokens/s and ~1 M output tokens/s.
• 99.9% availability per model; a node failure must not fail more than in-flight requests on that node.
• Average GPU utilisation above 60% over a day (assumption: target set by finance).

Out of scope

• Training and fine-tuning pipelines.
• Writing custom GPU kernels (assume an existing serving engine with paged attention).
• Product-level safety filtering.

Back-of-the-envelope estimates

• Requests: 100 M/day ≈ 1,200/s average, ~3,500/s peak.
• Tokens at peak: 3,500 × 2,000 = 7 M prompt tokens/s (prefill) and 3,500 × 300 ≈ 1 M output tokens/s (decode).
• KV cache size (assumed 8B-class architecture: 32 layers, 8 KV heads, head dim 128, fp16): 2 × 32 × 8 × 128 × 2 B ≈ 128 KB per token. An 80 GB GPU minus 16 GB of weights and overhead leaves ~58 GB ≈ 450k tokens, i.e. ~200 concurrent 2k-token sequences.
• Large model (assumed 80 layers, same head layout): ~320 KB per token; in fp8 the weights take ~70 GB, so a replica spans 4 GPUs to leave room for KV.
• Fleet (assumptions: small model ~4k decode tokens/s and ~30k prefill tokens/s per GPU; large model ~2.5k decode and ~15k prefill tokens/s per 4-GPU replica): with 70% of traffic on the small model, roughly 350 GPUs for small and ~1,000 for large at peak, about 1,350 GPUs.

KV cache memory, not compute, caps how many sequences a GPU can run at once, and prefill tokens outnumber decode tokens 7:1. Both point the design at memory management, prefix reuse and separating prefill from decode load.

API

rpc Generate(GenerateRequest) returns (stream TokenChunk)
  GenerateRequest {model, version?, adapter?, priority, prompt | messages,
                   max_tokens, temperature, stop[], cache_hint?, deadline_ms}
  TokenChunk      {text, token_ids[], finish_reason?, usage?}
rpc Cancel(request_id)
rpc ListModels() returns (ModelInfo[])        // context length, tiers, status
POST /admin/models/{name}/rollout {version, percent}

Callers pass a priority and a deadline_ms instead of picking replicas, so the platform can reorder, shed or spill work centrally and the same API serves interactive and batch traffic.

Data model

models(name, version, weights_uri, dtype, tensor_parallel, max_context, status) and routes(model, version, percent, pools[]) sit in a small relational registry; changes are rare and must be consistent, and routers cache them with a watch.

Weights live in an object store and are pre-staged onto each node's local NVMe so a restart reads at disk speed instead of across the network. Replica state (replica_id, model, free_kv_blocks, queue_len, cached_prefix_hashes) is soft state pushed by each engine every ~200 ms to the routers' in-memory tables; losing it only costs a few bad routing decisions. Usage records stream to Kafka and land in a columnar store for chargeback by caller and model.

High-level design

A product service sends a Generate call to the inference gateway, which authenticates the caller, checks its token quota and attaches a priority and deadline. The model router resolves the model name and rollout percentage from the model registry, then chooses a replica: among healthy replicas of that model it prefers one that already holds the request's prompt prefix in its KV cache, as long as that replica has free KV blocks; otherwise it picks the least loaded. If every replica is full, the request waits in an admission queue ordered by priority and deadline rather than overloading a GPU.

On the replica, the serving engine runs continuous batching: each forward pass decodes one token for every running sequence and folds in chunks of new prompts, so finished sequences leave and new ones join every iteration. Tokens stream back through the router and gateway to the caller. Large-model replicas span 4 GPUs with tensor parallelism; the LoRA pool shares one base model and swaps small adapters per request.

Engines publish queue length, KV utilisation, TTFT and tokens/s to metrics. The autoscaler adds replicas when queue wait or KV pressure rises and drains them when idle, pulling weights from the weights store onto local disk first. Batch-tier work fills spare capacity at night.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Continuous batching and chunked prefill

Decode is memory-bandwidth bound: reading all weights to produce one token costs nearly the same for 1 sequence as for 64, so throughput comes from large batches. Static batching wastes slots because short replies wait for the longest one. Continuous batching schedules per iteration: finished sequences leave, queued ones join. Prefill is the opposite, compute-bound and bursty; a 16k-token prompt can stall every running stream for hundreds of milliseconds. Split long prompts into chunks (say 512–2,048 tokens) mixed into decode iterations to bound inter-token latency. At very large scale, run prefill and decode on separate pools and ship the KV cache between them, so each pool can be sized and tuned for its own bottleneck.

Managing KV cache memory

Allocate KV memory in fixed-size blocks (for example 16 tokens) through a page table per sequence instead of one contiguous region per maximum length; this removes fragmentation and lets sequences grow on demand. Hash full blocks of the prompt so sequences that share a system prompt point at the same physical blocks (prefix caching), with reference counts and LRU eviction for unreferenced blocks. When memory runs out mid-decode, preempt the lowest-priority sequence: either swap its blocks to host memory or drop them and recompute later. The scheduler should admit a new sequence only if its prompt plus a reasonable output estimate fits, otherwise it causes preemption churn.

Autoscaling when cold starts take minutes

CPU utilisation is meaningless here; scale on queue wait time, KV block utilisation and TTFT against the SLO. Cold start is the enemy: pulling 70 GB of weights, loading them to GPU memory and warming kernels can take 2–5 minutes. Mitigations: keep weights on local NVMe of every node in the pool, keep a small warm buffer of idle replicas per interactive model, scale ahead using the daily traffic curve, and fill the gap with batch work that can be preempted in seconds. Scale down slowly, by draining (stop admitting, let streams finish). Fragmentation matters too: a 4-GPU replica needs 4 free GPUs on one node, so bin-pack small models onto partial nodes.

Multi-model routing and tiers

The router keeps a live view of each replica's free KV blocks, queue length and cached prefix hashes. Routing by a consistent hash of the prefix improves cache reuse, but pure affinity creates hot replicas, so use bounded load: prefer the affine replica unless it is over a threshold, then fall back to least loaded. Interactive requests may preempt batch sequences; batch traffic is admitted only when interactive queues are empty. LoRA fine-tunes share a base model pool, with adapters cached on GPU and loaded in milliseconds, which is far cheaper than a dedicated replica per team. New versions get a route percentage and are compared on latency and offline quality before ramping.

Failure modes

• If a GPU or node fails, its in-flight sequences are lost; the router retries them on another replica from the prompt plus tokens already emitted, and the node is cordoned.
• If traffic spikes beyond capacity, the gateway sheds batch and low-priority work first and returns a retry-after; interactive requests wait in bounded queues with deadlines instead of timing out everywhere.
• If a new model version regresses quality or latency, the registry route is set back to the previous version in seconds because old replicas are drained, not deleted, during rollout.
• If the weights store is unavailable, existing replicas keep serving from local disk; only scale-up is blocked.
• If routers lose replica stats, they fall back to round-robin with per-replica concurrency caps until fresh reports arrive.

Core algorithm in code

import random


def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_value=2):
    return 2 * layers * kv_heads * head_dim * bytes_per_value  # K and V


def make_requests(n, seed=7):
    rng = random.Random(seed)
    reqs, t = [], 0
    for i in range(n):
        t += rng.randint(0, 2)
        reqs.append({"id": i, "arrive": t, "prompt": rng.randint(200, 2000),
                     "out": rng.choice([20, 40, 80, 300, 600])})
    return reqs


def simulate(reqs, max_batch, kv_budget_tokens, continuous):
    pending = sorted(reqs, key=lambda r: r["arrive"])
    queue, running, latency = [], [], {}
    step = i = busy_slots = 0
    while len(latency) < len(reqs):
        while i < len(pending) and pending[i]["arrive"] <= step:
            queue.append(dict(pending[i], left=pending[i]["out"]))
            i += 1
        if continuous or not running:
            kv = sum(r["prompt"] + r["out"] for r in running)  # reserve worst case
            while (queue and len(running) < max_batch
                   and kv + queue[0]["prompt"] + queue[0]["out"] <= kv_budget_tokens):
                r = queue.pop(0)
                kv += r["prompt"] + r["out"]
                running.append(r)
        step += 1
        busy_slots += len(running)
        for r in running:
            r["left"] -= 1
        for r in [r for r in running if r["left"] == 0]:
            latency[r["id"]] = step - r["arrive"]
            running.remove(r)
    return step, sum(latency.values()) / len(latency), busy_slots / (step * max_batch)


# Assumed 8B-class model: 32 layers, 8 KV heads (grouped-query), head dim 128, fp16.
per_tok = kv_bytes_per_token(32, 8, 128)
free_gb = 80 - 16 - 6          # 80 GB GPU minus fp16 weights minus activations/overhead
budget = int(free_gb * 1e9 / per_tok)
print("KV per token: %d KB, KV budget: ~%dk tokens" % (per_tok // 1024, budget // 1000))

reqs = make_requests(300)
for name, cont in (("static", False), ("continuous", True)):
    steps, lat, occ = simulate(reqs, max_batch=64, kv_budget_tokens=budget, continuous=cont)
    print("%-10s steps=%5d  mean latency=%6.1f steps  slot occupancy=%3.0f%%" % (name, steps, lat, occ * 100))

Trade-offs

• Larger batches give more tokens per GPU-hour but cost per-stream speed and tail latency for interactive users.
• Prefix-affinity routing raises cache hit rates and cuts prefill cost but risks hot replicas and uneven load.
• Disaggregated prefill and decode pools tune each phase separately but cost KV transfer bandwidth and more operational complexity.
• fp8 quantization nearly halves memory and raises throughput but costs an evaluation gate and a small quality risk.
• Warm idle replicas protect TTFT during spikes but cost GPUs that do no useful work.

What a strong answer shows

• Separates prefill and decode and explains why each is bound by different resources.
• Explains continuous batching and why static batching wastes GPU time.
• Computes KV cache size per token and uses it to bound concurrency.
• Designs autoscaling around cold-start time with the right signals.
• Routes with prefix awareness, priorities and LoRA sharing.
• Talks in TTFT, tokens/s and cost per token, with numbers.

Mistakes to avoid

• Treating an LLM replica like a stateless web server and load balancing by request count.
• Autoscaling on CPU or GPU utilisation percentage rather than queue wait and KV pressure.
• Ignoring KV cache memory and claiming a GPU can run hundreds of long sequences.
• Letting long prompts block decode for everyone, causing stutter in every stream.
• Giving every fine-tune its own full replica.

Follow-ups to expect

• How would speculative decoding help, and when would it not? (A small draft model proposes tokens; helps at low batch, less at high batch.)
• How do you serve 128k-token contexts? (Dedicated pool, chunked prefill, KV offload, tight per-caller limits.)
• How do you do chargeback fairly? (Meter prefill and decode tokens separately, discount cached prefix tokens.)
• How would you run across two regions? (Each region serves its own callers; overflow cross-region for batch only.)

Practice next