Design a Consumer LLM Chat Assistant

Design a consumer chat assistant backed by a large language model. Tens of millions of people open it every day on web and mobile, type a question, and watch the answer appear word by word. They come back to old conversations, continue them days later, and expect the assistant to remember what was said earlier in the thread.

There is a free tier and paid tiers. Paid users get a stronger model and higher limits; free users must not be able to run up an unbounded GPU bill. Every prompt and every answer must pass safety checks, yet the first words of a reply should show up within about a second.

In 45 minutes, agree on scope, estimate the load in tokens and GPUs rather than only requests, sketch the API and data model, draw the architecture, and go deep on streaming, fitting long conversations into a finite context window, safety filtering, and per-tier limits and cost control.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 14 of 18

Ask before you design. Try answering each yourself, then compare.

1. Do we build and host the model, or call one?
   Typical answer

   Assume we self-host open-weights models on our own GPU fleet for most traffic, with a hosted LLM API available as overflow. Training is out of scope.

2. What scale should we plan for?
   Typical answer

   About 50 million daily active users sending around 10 messages each, so roughly 500 million turns per day.

3. How long can conversations get, and what is the model context window?
   Typical answer

   Threads can run to hundreds of turns. Assume a 32k-token window on the paid model and 8k on the free model.

4. What tiers exist and how do limits differ?
   Typical answer

   Free, Plus and Team. Limits are expressed in tokens per time window and in messages to the premium model, not raw requests.

5. What latency matters most?
   Typical answer

   Time to first token under about 1 second at p95, then a steady 30+ tokens per second while streaming.

6. Must every answer be filtered before the user sees it?
   Typical answer

   Yes for clearly disallowed content. A small buffer of delay is acceptable; blocking the whole answer until it finishes is not.

7. Do we need file uploads, images or web browsing?
   Typical answer

   Not in this round. Text chat only; mention tools as a follow-up.

Requirements

Functional

• Send a message in a conversation and receive a streamed reply.
• List, open, rename and delete past conversations; continue any of them.
• Regenerate or stop a reply mid-stream.
• Apply safety policy to prompts and replies.
• Enforce per-tier usage limits and show users their remaining allowance.

Non-functional

• Time to first token p95 under 1 s; streaming rate at least 30 tokens/s per reply.
• 500 M turns/day, about 17k turns/s at peak, ~140k concurrent streams.
• 99.9% availability for chat; conversation history durable (no lost acknowledged turns).
• Cost per free-tier user per month capped by design (assumption: under $0.30).

Out of scope

• Model training and fine-tuning.
• Images, voice, file uploads and web browsing (follow-ups).
• Payments and subscription billing internals.

Back-of-the-envelope estimates

• Turns: 50 M DAU × 10 = 500 M turns/day ≈ 5,800/s average, about 17,000/s at a 3× peak.
• Tokens per turn (assumption): ~1,500 prompt tokens after history packing and ~400 output tokens.
• Output throughput at peak: 17k × 400 ≈ 7 M output tokens/s. A reply at ~50 tokens/s lasts ~8 s, so about 140k streams are open at once.
• GPUs (assumption): one 8-GPU node serving a mid-size model with continuous batching sustains ~20k output tokens/s, so peak needs ~350 nodes (~2,800 GPUs) for decode, plus headroom for prefill. At an assumed $2 per GPU-hour that is ~$5,600/hour at peak.
• Storage: ~2 KB of text per turn → ~1 TB/day, ~365 TB/year before compression; text compresses ~4×.

The number that drives the design is GPU time per turn. Every decision (history packing, model routing by tier, caching shared prompt prefixes, token-based limits) exists to keep it bounded.

API

POST   /v1/conversations                      -> {conversationId}
POST   /v1/conversations/{id}/messages          {text, parentId?, model?}
       -> 200 text/event-stream
          event: delta {turnId, text}
          event: done  {turnId, usage:{in, out}, finish}
POST   /v1/turns/{turnId}/stop
GET    /v1/turns/{turnId}/stream?offset=N        (resume after disconnect)
GET    /v1/conversations?cursor=...              -> titles, updatedAt
GET    /v1/conversations/{id}/messages?cursor=...
GET    /v1/me/usage                             -> {tier, tokensLeft, resetsAt}

Replies stream over server-sent events on a plain HTTP response: it passes through proxies and CDNs easily, is one-directional like the data, and a turn id plus offset lets a dropped client resume instead of paying for a second generation.

Data model

conversations(user_id, conversation_id, title, model, summary, summary_upto_turn, updated_at) and turns(conversation_id, turn_id, parent_id, role, text, tokens, safety_flags, created_at) live in a wide-column store partitioned by user_id (conversations) and conversation_id (turns), clustered by time. Reads are always "this user's recent threads" or "this thread in order", so there are no cross-partition queries. parent_id allows regenerate and edit as branches.

Hot state lives in Redis: per-user token buckets, and an in-flight turn buffer (turn_id → streamed text so far, 10-minute TTL) that powers resume. Usage events go to Kafka and are aggregated into a relational usage_daily(user_id, day, model, in_tokens, out_tokens, cost) table, which billing and support can query with SQL. Turns older than 90 days are compacted to blob storage.

High-level design

The app opens an HTTPS request to the API gateway, which authenticates the user and asks the tier rate limiter whether the user has token budget left (an estimate of prompt plus maximum output tokens is reserved up front). The request then reaches a stateless chat orchestrator.

On the write path, the orchestrator runs the prompt through the fast safety classifiers while the context builder loads the thread from the conversation store and packs the system prompt, a rolling summary and the newest turns into the model's token budget. The model router picks a model pool from the user's tier, current load and the request, and sends it to the self-hosted inference fleet; if that pool is saturated or unhealthy, paid traffic can spill to a hosted LLM API. Tokens flow back through the router and orchestrator, pass an incremental output check, and are pushed to the client as SSE events while also being appended to the in-flight buffer.

When the reply finishes, the orchestrator writes both turns to the store and emits a usage event. The metering worker converts tokens to cost, settles the reservation against actual usage, updates the quota database and refreshes limiter settings. Reading history is a simple partition scan on the conversation store and never touches the GPUs.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Streaming, stop and resume

Each turn gets an id before generation starts. The orchestrator forwards token deltas over SSE and also appends them to a Redis buffer keyed by turn id. If the phone switches networks, the client reconnects with offset=N and receives the rest from the buffer, while generation continues server-side; without this, users hit regenerate and you pay twice. Stop is a small RPC that sets a cancel flag; the router aborts the sequence so its KV cache slot is freed immediately, which matters because abandoned streams otherwise burn GPU time nobody reads. Gateways need long idle timeouts and heartbeat comments every 15 s so proxies do not cut quiet streams.

Fitting long threads into the context window

Treat the window as a budget: window minus reserved output tokens minus system prompt minus the new message. Fill it newest-first with whole turns. Anything older is represented by a rolling summary, regenerated asynchronously by a cheap model every N turns and stored with summary_upto_turn, so building a prompt never waits on summarization. Count tokens with the exact tokenizer of the target model, since routing can switch models mid-thread. Keep the stable prefix (system prompt, summary) at the front and unchanged between turns so the serving layer can reuse its cached attention state; that cuts prefill cost on every follow-up message.

Safety without killing time to first token

Run a small input classifier in parallel with prompt prefill instead of before it; if it flags the prompt, cancel the generation and return a refusal. Prefill costs a little wasted GPU on the rare bad prompt but saves ~100–200 ms on every good one. For output, buffer a short window (say 40–80 tokens) and classify sliding chunks before releasing them, so the user sees a small fixed lag rather than waiting for the whole answer. A late violation retracts the partial message with a replacement event. Log decisions with policy version for appeals and tuning, and keep thresholds per tier and region in config, not code.

Tiered limits and cost control

Meter tokens, not requests. Each user has buckets per model class (for example, free: 40k tokens per 3 hours on the small model; paid: a quota of premium messages, then fallback to the standard model). The limiter reserves an estimated cost before generation and the metering worker settles the difference afterwards. Cost levers beyond limits: route by tier and request difficulty, cap max_tokens per tier, reuse cached prompt prefixes, summarize instead of resending full history, and shed free traffic first under load by downgrading the model rather than failing. A global spend circuit breaker per hour protects against abuse spikes.

Failure modes

• If an inference node dies mid-reply, the router retries the sequence on another node from the prompt plus tokens already sent, and the client sees a short pause rather than an error.
• If the GPU pool is saturated, free users are downgraded to a smaller model or queued with a visible wait, while paid traffic spills to the hosted API.
• If the conversation store is slow, the orchestrator can still answer using the Redis copy of recent turns and writes the turn asynchronously with retries.
• If the safety service is down, fail closed for flagged categories on free tier (refuse) and log for review; never silently skip checks.
• If the metering pipeline lags, limiter reservations still bound spend; reconciliation catches up from the Kafka log.

Core algorithm in code

from dataclasses import dataclass


@dataclass
class Turn:
    role: str
    text: str
    tokens: int


def build_context(system_tokens, summary, turns, new_msg_tokens, window, reserve_output):
    """Pack newest-first until the budget runs out; older turns are covered by the summary."""
    budget = window - reserve_output - system_tokens - new_msg_tokens
    if budget < 0:
        raise ValueError("message too long for this model")
    kept = []
    for t in reversed(turns):
        if t.tokens > budget:
            break
        kept.append(t)
        budget -= t.tokens
    kept.reverse()
    dropped = len(turns) - len(kept)
    use_summary = None
    if dropped and summary and summary[1] <= budget:
        use_summary = summary
    return use_summary, kept


class TierBucket:
    def __init__(self, tokens_per_min, burst):
        self.rate = tokens_per_min / 60.0
        self.cap = burst
        self.level = burst
        self.t = 0.0

    def try_spend(self, now, tokens):
        self.level = min(self.cap, self.level + (now - self.t) * self.rate)
        self.t = now
        if tokens > self.level:
            return False
        self.level -= tokens
        return True


turns = [Turn("user" if i % 2 == 0 else "assistant", "msg %d" % i, 300 + 50 * (i % 3)) for i in range(20)]
summary = ("User is planning a trip to Lisbon; prefers trains.", 120)
s, kept = build_context(system_tokens=400, summary=summary, turns=turns,
                        new_msg_tokens=200, window=4096, reserve_output=800)
print("kept", len(kept), "of", len(turns), "turns; summary used:", s is not None)
print("tokens in prompt:", 400 + 200 + sum(t.tokens for t in kept) + (s[1] if s else 0))

free = TierBucket(tokens_per_min=6000, burst=8000)
paid = TierBucket(tokens_per_min=60000, burst=80000)
for name, b in (("free", free), ("paid", paid)):
    ok = sum(b.try_spend(now=i * 2.0, tokens=2500) for i in range(10))
    print(name, "tier: allowed", ok, "of 10 requests of 2,500 tokens in 20 s")

Trade-offs

• Self-hosting open-weights models gives lower unit cost and control over batching but costs capacity planning, GPU operations and idle spend at night.
• Rolling summaries keep prompts short and cheap but lose detail from early turns, which users sometimes notice.
• Streaming output through a buffered safety check catches bad content before display but adds a fixed delay of a few hundred milliseconds to every reply.
• Token-based limits are fair to the GPU bill but are harder for users to understand than "N messages per day".
• Spilling to a hosted LLM API absorbs peaks without buying GPUs but costs more per token and changes model behaviour slightly between replies.

What a strong answer shows

• Estimates in tokens and GPUs, and shows GPU time is the dominant cost.
• Chooses a streaming transport and handles disconnect, resume and stop.
• Has a concrete context-window strategy (budget, recent turns, summaries, prefix reuse).
• Places safety checks so they add little latency while still blocking bad output.
• Designs limits by tier in tokens with reservation and reconciliation.
• Explains degradation under GPU shortage instead of just "autoscale".

Mistakes to avoid

• Sizing the system by requests per second and never converting to tokens or GPUs.
• Sending the entire conversation history on every turn until it overflows the window.
• Waiting for the full reply before running safety checks, making time to first token several seconds.
• Treating the LLM call as a normal 200 ms RPC with standard timeouts and retries.
• Rate limiting by request count so a single giant prompt bypasses cost controls.

Follow-ups to expect

• How would you add web browsing or tools? (An agent loop in the orchestrator with tool calls, per-tool timeouts and limits.)
• How do you support long-term memory across conversations? (Extract facts to a per-user store, retrieve a few into the system prompt.)
• How would you roll out a new model safely? (Shadow traffic, then a small percentage with quality and safety metrics, then ramp.)
• How do you make the app work in a region with no GPUs? (Route to the nearest GPU region; keep history and auth local.)

Practice next