Design an Internal LLM Gateway

Hundreds of internal teams now call large language models: some through two or three hosted LLM API vendors, some through the company's own GPU cluster running open-weights models. Each team has its own API keys, its own retry logic and no idea what it spends. Security worries that customer emails and card numbers are being sent to outside vendors. When one vendor has an outage, a dozen products break at once.

Design a single internal gateway that every LLM call goes through. It should give teams one API, route each request to a suitable model, fail over when a provider is slow or down, cache repeated questions, enforce rate limits and per-team budgets, track cost precisely, and strip personal data before anything leaves the company.

In 45 minutes, cover the request path and its latency overhead, routing and fallbacks, caching (including semantic caching), quotas and cost accounting with streaming responses, and PII redaction.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 57 of 70

Ask before you design. Try answering each yourself, then compare.

1. What traffic should we plan for?
   Typical answer

   About 50 M requests/day from ~2,000 applications across ~500 teams, peaking at ~1,700 requests/s. Most responses are streamed.

2. Which providers?
   Typical answer

   Two hosted LLM API vendors with different APIs and price lists, plus an internal self-hosted inference cluster. Each vendor gives us account-level rate limits.

3. How much latency may the gateway add?
   Typical answer

   Under ~20 ms p99 on top of model time; streaming must start as soon as the provider starts.

4. What counts as PII and what should happen to it?
   Typical answer

   Emails, phone numbers, card and account numbers, government ids and names in some flows. Mask before sending to external vendors and restore in the response when the caller is allowed to see it.

5. How are budgets set?
   Typical answer

   Monthly dollar budgets per team, with per-app token-per-minute limits. Exceeding a hard budget blocks; a soft budget alerts.

6. Can the gateway change which model answers?
   Typical answer

   Only within an alias the team chose (for example "chat-large" maps to equivalent models); never silently to a weaker class.

Requirements

Functional

• One API for chat/completions and embeddings, with model aliases mapped to providers.
• Automatic fallback across providers on errors, timeouts and rate limiting.
• Exact and semantic response caching, opt-in per app.
• Per-app rate limits (requests and tokens per minute) and per-team monthly budgets.
• Accurate cost tracking per team, app and model; dashboards and exports.
• PII detection and masking before external calls; audit logs without raw PII.

Non-functional

• Gateway overhead < 20 ms p99 (excluding model time); 1,700 requests/s peak with headroom to 5,000.
• 99.99% availability for the gateway itself; it must never be the outage.
• Cost accuracy within 1% of vendor invoices.
• No raw PII leaves the company for external providers on masked routes.

Out of scope

• Running the self-hosted inference cluster.
• Prompt management and evaluation tooling.
• Content safety moderation (could plug in as another filter).

Back-of-the-envelope estimates

• Requests: 50 M/day ≈ 580/s average, ~1,700/s at peak.
• Tokens (assumption): 1,500 in and 300 out per request → 75 B input and 15 B output tokens/day.
• Spend (assumed blended prices: $1.5 per M input, $6 per M output): ~$110k + ~$90k ≈ $200k/day, ~$6 M/month. A 1% metering error is $60k/month, so accounting precision matters.
• Cache: if 15% of requests are served from cache, that saves ~$30k/day and the corresponding latency.
• Gateway compute: the gateway mostly proxies streams; ~1,700 rps with long-lived streams (~5 s each) means ~8,500 concurrent connections, a handful of instances per region. PII detection with patterns plus a small NER model costs ~2–8 ms per request on CPU.
• Logs: ~6 KB of redacted metadata per request ≈ 300 GB/day.

Dollars, not QPS, drive this design: routing, caching, quotas and metering all exist to control a seven-figure monthly bill, while the gateway must stay thin enough to add almost no latency.

API

POST /v1/chat/completions   {model: "chat-large", messages, stream, max_tokens,
                             cache: {mode: off|exact|semantic, ttl_s}, pii: mask|block|allow}
     headers: Authorization: Bearer <app key>, X-Request-Id
     -> SSE chunks (same shape for every provider) + final usage {in, out, cost_usd, provider}
POST /v1/embeddings          {model, input[]}
GET  /v1/usage?team=&from=&to=&groupBy=app,model
PUT  /admin/routes/{alias}   {targets:[{provider, model, weight}], fallback_on:[429,5xx,timeout]}
PUT  /admin/teams/{id}/budget {monthly_usd, hard: true}

The gateway exposes one provider-neutral schema and returns the actual provider, tokens and cost on every response, so teams can move between models without code changes and see what each call cost.

Data model

Configuration is relational and small: teams(id, monthly_budget, hard), apps(key_hash, team_id, rpm, tpm, allowed_aliases, pii_policy), routes(alias, targets[], fallback_on) and a versioned prices(provider, model, in_per_m, out_per_m, cached_in_per_m, valid_from). Gateways cache all of it in memory and refresh on change.

Hot counters live in Redis: token buckets per app and per provider account, and month-to-date spend per team, updated by reservation and settlement. Every request emits a usage event (request_id, team, app, alias, provider, model, in_tokens, out_tokens, cached, latency_ms, status) to Kafka; the aggregator writes cost rows to a columnar billing table partitioned by day. The semantic cache stores (scope_hash, embedding, response, created_at) in a vector index, where the scope covers team, model alias, system prompt and parameters.

High-level design

An app calls the LLM gateway with its key and a model alias. The gateway validates the key against the keys and quotas service (cached locally), checks the team's month-to-date spend against its budget, and asks the rate limiter to reserve tokens: the prompt token count plus max_tokens. The PII redactor scans the prompt, replaces sensitive values with placeholders and keeps the mapping in request memory. If the app enabled caching, the request is looked up by exact hash and then in the semantic cache; a hit returns immediately.

On a miss, the model router expands the alias into an ordered list of targets (for example Provider A, then Provider B, then the self-hosted pool), skips targets whose circuit breakers are open or whose shared account limits are exhausted, translates the request to that provider's format and streams the response back. If the provider fails before the first token, the router retries on the next target; after tokens have flowed it surfaces the error. Placeholders in the output are restored for callers allowed to see them.

When the stream ends, the router emits a usage event. The cost aggregator multiplies tokens by the price table, settles the reservation in Redis with the actual count and writes cost rows to the usage and billing DB that drives dashboards and chargeback.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Routing, fallbacks and streaming

Teams pick an alias that names a capability class ("chat-large", "embed-small"), and the platform maps it to equivalent models from different providers, checked for context length, tool calling and JSON mode support. Each target has a health score from recent error rate and time to first token, and a circuit breaker that opens after repeated failures and probes again after a cool-down. Retrying is safe only before the first token reaches the client; after that, switching providers would produce a stitched answer, so the error is returned. Hedging (sending a second request if the first has not started streaming within, say, 2 s) cuts tail latency at extra cost, so it is enabled only for latency-critical aliases.

Exact and semantic caching

The exact cache keys on a hash of the normalized request (model, messages, parameters) and is safe for deterministic settings. The semantic cache embeds the last user message and returns a stored answer when similarity exceeds a high threshold (for example 0.95 cosine), but only inside a scope of team, alias, system prompt and parameters, and only for apps that opt in, such as FAQ bots. Its risk is a confident wrong answer for a question that looks similar but differs in one key detail ("refund within 30 days" vs "60 days"), so measure false-hit rate on labelled samples, use short TTLs, and never cache responses that contain user-specific data or restored PII.

Quotas and precise cost tracking

Token limits must be enforced before the output exists. Reserve prompt tokens plus max_tokens from the app's bucket at admission, then refund the unused part when the stream ends. Counting uses the provider's own tokenizer or its reported usage, never character estimates, because invoices are computed that way. Budgets are hierarchical: an organisation cap, team monthly budgets and app limits. Provider accounts have their own shared limits, enforced by a global token bucket per provider account so one team's batch job cannot starve everyone. Prices are versioned with effective dates; cached-input discounts and batch pricing are separate line items. Daily reconciliation against vendor invoices catches drift.

PII redaction that keeps answers useful

Detect PII with fast patterns (emails, card numbers with a checksum, phone numbers, ids) plus a small named-entity model for names and addresses, running on CPU in a few milliseconds. Replace each value with a typed placeholder such as <EMAIL_1>, keep the mapping only in request memory, and restore placeholders in the streamed output, buffering a few characters so a placeholder split across chunks is still recognised. Policy is per app and per destination: masking for external vendors, allow for the self-hosted pool inside the network, block for categories like government ids. Logs and caches store only redacted text, and detections are counted per app so security can see where sensitive data flows.

Failure modes

• If a provider has an outage, its breaker opens within seconds and traffic shifts to the next target; aliases without a fallback return a clear 503.
• If Redis (limits and spend counters) is down, gateways fall back to local per-instance limits sized as a share of the global limit and log usage for later settlement, failing open on budgets for a bounded time.
• If the usage pipeline lags, budgets use Redis counters; billing catches up from Kafka without loss.
• If the PII detector fails, requests on masking routes to external providers are blocked rather than sent unredacted; internal routes continue.
• If a gateway region fails, DNS and client SDKs shift traffic to another region; gateways are stateless so this needs no data migration.

Core algorithm in code

import re

PATTERNS = [("EMAIL", r"[\w.+-]+@[\w-]+\.[\w.]+"),
            ("CARD", r"\b(?:\d[ -]?){13,16}\b"),
            ("PHONE", r"\+?\d[\d -]{8,}\d")]


def redact(text):
    mapping = {}
    for kind, pat in PATTERNS:
        def sub(m, kind=kind):
            key = "<%s_%d>" % (kind, len(mapping) + 1)
            mapping[key] = m.group(0)
            return key
        text = re.sub(pat, sub, text)
    return text, mapping


def restore(text, mapping):
    for k, v in mapping.items():
        text = text.replace(k, v)
    return text


class Breaker:
    def __init__(self, threshold=3, cooldown=30.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.fails, self.opened_at = 0, None

    def allow(self, now):
        return self.opened_at is None or now - self.opened_at >= self.cooldown

    def record(self, ok, now):
        if ok:
            self.fails, self.opened_at = 0, None
        else:
            self.fails += 1
            if self.fails >= self.threshold:
                self.opened_at = now


class Gateway:
    def __init__(self, routes, providers, prices, budgets):
        self.routes, self.providers, self.prices = routes, providers, prices
        self.budgets = dict(budgets)
        self.spent = {t: 0.0 for t in budgets}
        self.breakers = {p: Breaker() for p in providers}

    def complete(self, team, alias, prompt, now):
        if self.spent[team] >= self.budgets[team]:
            return "429 budget exhausted for " + team
        safe, mapping = redact(prompt)
        for target in self.routes[alias]:
            if not self.breakers[target].allow(now):
                continue
            try:
                out, tin, tout = self.providers[target](safe)
            except RuntimeError:
                self.breakers[target].record(False, now)
                continue
            self.breakers[target].record(True, now)
            pin, pout = self.prices[target]
            self.spent[team] += (tin * pin + tout * pout) / 1e6
            return "[%s] %s" % (target, restore(out, mapping))
        return "503 all providers unavailable"


def provider_a(prompt):
    if provider_a.down:
        raise RuntimeError("timeout")
    return "A says: noted " + prompt.split()[-1], 1200, 300
provider_a.down = False


def provider_b(prompt):
    return "B says: noted " + prompt.split()[-1], 1200, 300


gw = Gateway(routes={"chat-large": ["prov-a/large", "prov-b/large"]},
             providers={"prov-a/large": provider_a, "prov-b/large": provider_b},
             prices={"prov-a/large": (3.0, 15.0), "prov-b/large": (2.5, 10.0)},
             budgets={"search-team": 0.04})

print(redact("Refund jane.doe@corp.com, card 4111 1111 1111 1111, call +44 20 7946 0958")[0])
print(gw.complete("search-team", "chat-large", "Contact jane.doe@corp.com", now=0))
provider_a.down = True
for t in range(1, 8):
    res = gw.complete("search-team", "chat-large", "Contact jane.doe@corp.com", now=t)
    print("t=%d A-breaker=%-6s %s" % (t, "open" if not gw.breakers["prov-a/large"].allow(t) else "closed", res))
print("spent $%.4f of $%.2f" % (gw.spent["search-team"], gw.budgets["search-team"]))

Trade-offs

• A central gateway gives one place for policy, cost and failover but becomes a critical dependency that must be more available than any provider.
• Semantic caching saves money and latency on repeated questions but risks returning a wrong answer for a subtly different question.
• Reserving max_tokens up front makes token limits strict but under-uses quota for apps that set large max_tokens.
• Masking PII protects customers but can reduce answer quality when the model needs the real value to reason.
• Falling back across providers improves availability but changes model behaviour between requests.

What a strong answer shows

• Keeps the gateway thin and stateless with a clear overhead budget.
• Explains fallback rules, including why retries stop after streaming starts.
• Scopes semantic caching carefully and discusses false hits.
• Enforces token limits with reservation and settlement, plus shared provider limits.
• Tracks cost precisely with versioned prices and reconciliation.
• Designs reversible PII masking with per-destination policy.

Mistakes to avoid

• Rate limiting by requests per minute only, ignoring tokens and cost.
• Retrying a half-streamed response on another provider and stitching outputs.
• A global semantic cache shared across teams and system prompts.
• Estimating tokens from characters, so cost reports drift from invoices.
• Logging raw prompts for debugging, which defeats PII redaction.

Follow-ups to expect

• How would you route by cost and quality automatically? (A classifier sends easy prompts to a cheaper model; validate with offline evals.)
• How would you support batch jobs? (A batch endpoint with lower priority and discounted provider batch APIs.)
• How do you add safety moderation? (Another pluggable filter stage on input and output, with per-app policy.)
• How would you charge back shared prompt caching discounts fairly? (Attribute cached-token savings to the app whose prefix was cached.)

Practice next