Design an LLM Evaluation and Monitoring Platform

A company has about 40 product features built on large language models: chat assistants, summarisers, extraction pipelines and a few agents that call tools. Teams change prompts, swap models and tweak retrieval every week, and they mostly find out about quality drops from angry users. Finance also wants to know what each feature costs.

Design a platform that lets every team trace production LLM calls, run versioned evaluation sets against a proposed change before it ships, score outputs with automatic checks, model-based judges and human labels, collect user feedback, and watch quality, drift, latency and cost on dashboards with alerts.

Outputs are free text and non-deterministic, so "did this get worse?" is a statistics question as much as an engineering one. In 45 minutes, size the trace volume, design ingestion and the eval workflow, and go deep on trustworthy automated judging and regression gating.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 58 of 70

Ask before you design. Try answering each yourself, then compare.

1. How much production traffic?
   Typical answer

   About 50 M LLM calls a day across 40 apps, ~4 spans per request (retrieval, model, tool calls), ~8 KB of payload per span.

2. Do we keep every trace?
   Typical answer

   Keep all metadata (tokens, latency, cost, versions). Keep redacted payloads for 30 days. Run online quality scoring on a 2-5% sample.

3. How big are eval sets and how fast must a run be?
   Typical answer

   200 to 5,000 cases per app. A run gates a merge, so it should finish in under 20 minutes.

4. Which models can act as judges?
   Typical answer

   A strong hosted model and a cheaper self-hosted one. Judge spend is budgeted.

5. Is there sensitive data?
   Typical answer

   Yes. Redact PII before storage, isolate teams’ data, and keep raw payloads only 30 days; aggregates for a year.

6. What overhead can the SDK add to apps?
   Typical answer

   Under 1 ms per call. If the platform is down, apps must be unaffected.

7. Who uses it?
   Typical answer

   App engineers, product managers and on-call engineers.

Requirements

Functional

• SDK to record traces and spans for LLM, retrieval and tool calls, with prompt, model and config versions.
• Versioned eval datasets built by hand, from production traces and from failures.
• Eval runs of a candidate configuration with heuristic, model-judge and human scorers, compared with a baseline.
• CI gate that blocks a change on statistically meaningful regressions.
• Collect explicit and implicit user feedback attached to traces.
• Dashboards and alerts for quality, input drift, latency, error rate and cost per app.

Non-functional

• Ingest 7 k spans/s at peak with < 1 ms SDK overhead and no app impact on outage.
• Traces queryable within 60 s of the call.
• Gate runs of 2,000 cases complete in < 20 minutes.
• PII redacted before persistence; per-team access control.
• Judge cost under an agreed daily budget.

Out of scope

• Training or fine-tuning models.
• Serving the LLM apps themselves.
• Prompt authoring UI beyond versioning.

Back-of-the-envelope estimates

• Calls: 50 M/day ≈ 580/s average, ~1,800/s at peak. With ~4 spans each, ~7 k spans/s at peak.
• Storage: 200 M spans × 8 KB ≈ 1.6 TB/day raw; 30 days compressed ~4× ≈ 12 TB of payloads. Metadata rows (~300 bytes) are only ~60 GB/day and go to a columnar store for a year.
• Online judging: 3% of 50 M = 1.5 M judge calls/day × ~1,500 input tokens ≈ 2.3 B tokens/day. At an assumed $1 per million input tokens that is ~$2.3 k/day, so judging everything is not affordable; sampling and a cheap first tier are.
• Offline gates: 40 apps × 5 changes/day × 2,000 cases × 3 calls (app + 2 judges) ≈ 1.2 M calls/day in bursts. 2,000 cases in 20 minutes at ~10 s each needs ~20 cases in flight per run.

Judge cost and rate limits, not storage, are what shape the design.

API

sdk.span(kind="llm|retrieval|tool", name, input, output, model, prompt_version,
         tokens_in, tokens_out, latency_ms)            # buffered, exported async
POST /v1/ingest/spans          [batch of spans]
POST /v1/feedback              {trace_id, signal:"thumbs_down"|"regenerate"|..., comment?}
POST /v1/datasets/{id}/versions {cases:[{input, reference?, tags[]}]}
POST /v1/eval-runs             {app, dataset_version, candidate:{prompt_version, model, params},
                                baseline_run_id, scorers:["groundedness@v3", "json_valid"]} -> {run_id}
GET  /v1/eval-runs/{id}/compare -> {per_scorer:[{delta, ci_low, ci_high}], slices, verdict}

Every score and every piece of feedback attaches to a trace ID, and every trace records the exact prompt, model and retrieval versions, so any quality signal can be traced back to the configuration that produced it.

Data model

Spans in a columnar analytics store: (app, day, trace_id, span_id, parent_id, kind, model, prompt_version, tokens, latency_ms, cost, status), partitioned by app and day so dashboards scan little data.

Payloads (redacted inputs and outputs) in object storage keyed by trace_id, with a 30-day lifecycle rule; the span row stores only a pointer.

Datasets (relational): immutable dataset_versions and cases(case_id, input, reference, tags), so runs months apart are comparable.

Scores: (run_id or trace_id, case_id, scorer, scorer_version, score, rationale). Scorers registry pins judge model and judge prompt versions. Feedback rows keyed by trace_id.

High-level design

Production path. Each LLM app embeds the SDK, which buffers spans in memory and ships them in batches to the ingest API; user feedback comes in the same way. Ingest validates, authenticates the team and writes to the span stream, which absorbs bursts. Redact + enrich workers scrub PII, compute cost from token counts and a versioned price table, then write metadata to the trace warehouse and payloads to the payload store. They also forward a sample of traces, including all with negative feedback or errors, to the scorer pool, which runs cheap checks and calls the judge models; scores go back to the warehouse. Dashboards and alerts read rollups by app, version and slice.

Change path. When a prompt or model change is proposed, CI asks the eval orchestrator for a run. It loads the pinned dataset version from datasets and runs, replays every case against the candidate app build with bounded concurrency, sends outputs to the scorer pool, compares against the baseline with paired statistics and returns pass or block to CI.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Making model judges trustworthy

Each judge is a scorer with a pinned model, prompt and rubric version. Before use, it is calibrated on a few hundred cases graded by humans; ship it only if agreement is acceptable (for example, kappa above 0.6). Narrow rubrics ("does the answer contain a claim not in the provided context?") work far better than "rate quality 1-10". For pairwise comparisons, judge twice with the order swapped and treat disagreement as a tie, because judges favour the first or longer answer. Changing a judge version invalidates old scores, so the baseline is re-scored with the new judge rather than compared across judges. A cheap tier (format checks, a small classifier) handles easy cases; the strong judge only sees uncertain ones.

Detecting regressions in noisy outputs

Run baseline and candidate on the same cases and compare per case, which removes most of the variance from case difficulty. Compute the mean delta and a bootstrap confidence interval; block only when the interval’s lower bound falls below an agreed tolerance such as -2 points. Repeat the evaluation on every tag slice, since a change that helps most traffic can break refunds or one language. Mark some cases must-pass (safety, legal), where any failure blocks. Sample each case two or three times when temperature is above zero. Report the minimum change a set of this size can detect, so teams know when they need more cases.

Ingesting traces without hurting apps

The SDK never blocks a request: spans go into a bounded in-memory buffer flushed every second or 500 spans, and when the buffer is full it drops and counts. Span IDs are generated client-side, so retried batches deduplicate. Metadata is kept for every call, but payloads can be sampled per trace, deciding after the trace ends so that errors, slow calls and negative feedback are always kept. Redaction runs before anything is persisted, and the stream gives a buffer of hours if the warehouse is slow. Teams are isolated by an app key that is checked on ingest and enforced on every query.

Drift, feedback and cost dashboards

Explicit thumbs are rare and skewed, so add implicit signals: regenerate clicks, copying the answer, abandonment and follow-up rephrasing. To spot drift, embed a sample of inputs weekly, cluster them, and compare the cluster mix with last month and with the eval set; growing clusters the eval set does not cover become candidates for new cases after human review. Cost is computed per span from tokens and the price table and rolled up by app, feature, model and team. Alerts fire on cost per request, judge score drops by version, and latency percentiles, each compared with the same hour last week.

Failure modes

• If ingest is down, SDK buffers fill and drop spans with a counter; apps keep working and dashboards show a data gap, not false zeros.
• If the judge provider rate-limits or fails, offline runs pause and resume from the last scored case; online sampling falls back to cheap scorers only.
• If a judge silently changes behaviour after a provider update, nightly re-scoring of the calibration set catches the agreement drop and freezes that scorer.
• If redaction misses PII, the 30-day payload lifecycle limits exposure, and a purge job can delete by trace ID or pattern.

Core algorithm in code

import random

random.seed(7)


def paired_bootstrap(base, cand, rounds=1000):
    """Mean score delta (cand - base) on the same cases, with a 95% bootstrap interval."""
    diffs = [c - b for b, c in zip(base, cand)]
    n = len(diffs)
    means = sorted(sum(diffs[random.randrange(n)] for _ in range(n)) / n for _ in range(rounds))
    return sum(diffs) / n, means[int(0.025 * rounds)], means[int(0.975 * rounds)]


def gate(cases, base, cand, max_drop=0.02):
    """Block if the interval's lower bound is worse than -max_drop overall or on any tagged slice."""
    report, ok = [], True
    slices = {"ALL": range(len(cases))}
    for tag in sorted({c["tag"] for c in cases}):
        slices[tag] = [i for i, c in enumerate(cases) if c["tag"] == tag]
    for name, idx in slices.items():
        mean, lo, hi = paired_bootstrap([base[i] for i in idx], [cand[i] for i in idx])
        bad = lo < -max_drop and mean < 0
        ok = ok and not bad
        report.append(f"  {name:8} n={len(idx):3}  delta={mean:+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]  {'REGRESSION' if bad else 'ok'}")
    return ok, report


# 300 eval cases; scores are judge grades scaled to 0..1
cases = [{"tag": random.choice(["billing", "refunds", "shipping"])} for _ in range(300)]
base = [min(1, max(0, random.gauss(0.78, 0.15))) for _ in cases]

noise_only = [min(1, max(0, b + random.gauss(0, 0.05))) for b in base]
refund_bug = [min(1, max(0, b + random.gauss(0, 0.05) - (0.10 if c["tag"] == "refunds" else -0.045)))
              for b, c in zip(base, cases)]

for name, cand in [("prompt v12 (reworded)", noise_only), ("prompt v13 (new tool)", refund_bug)]:
    ok, report = gate(cases, base, cand)
    print(f"{name}: {'PASS' if ok else 'BLOCK'}")
    print("\n".join(report))

Trade-offs

• Model judges scale to thousands of cases cheaply but cost bias and drift, so a human-graded calibration set must be maintained.
• Sampling payloads cuts storage but costs the ability to inspect every bad response, which tail-based keep rules partly restore.
• Strict CI gating stops regressions but costs slower merges and occasional false blocks on noisy scorers.
• One shared platform gives comparable metrics across teams but costs per-team flexibility in custom scorers, which a plugin interface allows.
• A columnar store makes aggregate dashboards fast but costs slow single-trace lookups, handled by the payload store keyed by trace ID.

What a strong answer shows

• Uses trace IDs and version metadata so scores, feedback and cost all join to a configuration.
• Treats regression detection statistically with paired comparison and slices.
• Validates and versions judges instead of trusting them blindly.
• Keeps the SDK off the apps’ critical path and plans for sampling and redaction.
• Sizes judge cost and designs a cascade or sampling to control it.
• Connects production signals back into eval datasets.

Mistakes to avoid

• Comparing average scores between runs on different cases or with different judges.
• Asking a judge for a vague 1-10 quality score and treating it as ground truth.
• Logging synchronously from the request path.
• Storing raw prompts with customer PII indefinitely.
• Judging 100% of production traffic with the most expensive model.

Follow-ups to expect

• How would you evaluate multi-step agents? (Score the trajectory: tool choice, arguments, final outcome, step count.)
• How do you run online A/B tests of prompts? (Assign by user hash, log variant on the trace, compare feedback and judge scores.)
• How do you stop eval sets from leaking into fine-tuning data? (Tag and hash eval cases; filter them from training exports.)
• How would you support human review at scale? (Queue disagreements and low-confidence judge scores to annotators with a rubric.)

Practice next