Design a RAG Question-Answering System

Employees at a 50,000-person company waste hours hunting through wikis, shared drives, ticket systems and chat exports. Build an assistant that answers their questions in plain language using the company's own documents, and shows exactly which documents each statement came from.

The hard parts are not the language model. Documents change every minute, and an answer based on last month's policy is worse than no answer. Many documents are restricted: an engineer must never get an answer built from a finance-only spreadsheet, even indirectly. Search must work for both vague questions and exact terms like error codes and project names. And you need a way to know whether answers are actually getting better or worse.

In 45 minutes, cover ingestion and chunking, embedding and indexing, retrieval and reranking, answer generation with citations, permissions, freshness and evaluation.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 49 of 70

Ask before you design. Try answering each yourself, then compare.

1. How big is the corpus and how fast does it change?
   Typical answer

   About 20 M documents (~100 M chunks); around 1% change per day. Edits should be searchable within 5 minutes, and permission changes within 1 minute.

2. How many users and queries?
   Typical answer

   20k daily users asking ~8 questions each, ~160k queries/day, peaking at maybe 10–20 queries/s during work hours.

3. Where do permissions come from?
   Typical answer

   Each source system has its own ACLs. We sync them as lists of user and group ids; the identity service maps a user to their groups.

4. Which models can we use?
   Typical answer

   A hosted LLM API under an enterprise agreement that does not retain data, or self-hosted open-weights models; embeddings can be self-hosted.

5. What latency is acceptable?
   Typical answer

   First answer token within ~2 s at p95; full answer streamed in under ~10 s.

6. What if the documents do not contain the answer?
   Typical answer

   Say so clearly and show the closest sources. Never invent policy.

Requirements

Functional

• Ingest documents from several sources continuously, including edits and deletions.
• Answer natural-language questions with inline citations linking to source passages.
• Only use documents the asking user is allowed to read.
• Say "not found" when sources do not support an answer.
• Collect user feedback and run offline evaluations.

Non-functional

• First token p95 < 2 s; retrieval + rerank budget ~400 ms.
• Freshness: content edits searchable in < 5 min, permission revocations in < 1 min.
• 100 M chunks, ~20 queries/s peak, 99.9% availability in working hours.
• Zero tolerance for cross-permission leaks.

Out of scope

• Training or fine-tuning the LLM.
• Multi-step agent actions (filing tickets etc.).
• Image and video understanding.

Back-of-the-envelope estimates

Here a "write" is a document change and a "read" is a question.

• Corpus: 20 M docs × ~5 chunks of ~500 tokens = 100 M chunks.
• Vectors: 100 M × 768 dims × 4 B ≈ 300 GB in float32; ~75 GB with int8 quantization, plus HNSW links (~100 GB total). It fits a handful of memory-heavy shards, replicated 3×.
• Chunk text and metadata: ~2 KB per chunk ≈ 200 GB.
• Change rate: 1% of docs/day = 200k docs ≈ 1 M chunks to re-embed per day, only ~12/s. The full backfill is 50 B tokens; at an assumed 50k tokens/s per embedding GPU that is ~280 GPU-hours, which matters when you change the embedding model.
• Queries: 160k/day, ~20/s peak. Each prompt carries ~8 chunks ≈ 4k tokens plus ~400 output tokens; at an assumed $3 per million input tokens that is ~$2,000/month of input tokens, small next to engineer time saved.

Query volume is modest. The design is driven by corpus size, freshness and permission correctness, plus the cost of re-embedding everything when models change.

API

POST /v1/ask   {question, conversationId?, filters?:{source, after}}
     -> text/event-stream: delta{text} ... done{citations:[{n, docId, chunkId, title, url, snippet}], noAnswer}
POST /v1/feedback {answerId, rating, comment?}

// internal
POST /ingest/events {source, docId, op: upsert|delete, version, acl[], uri}
POST /search       {query, userGroups[], k, mode: hybrid}   -> chunks[] with scores

Retrieval is a separate internal API that requires the caller's group ids, so every consumer (chat, IDE plugin, agents) gets permission filtering for free and cannot skip it.

Data model

documents(doc_id PK, source, uri, title, version, content_hash, acl_groups[], updated_at, deleted) and chunks(chunk_id PK, doc_id, ord, text, heading_path, token_count, content_hash, embed_model) live in a relational store: we need transactions when a document version replaces its chunks, and joins for debugging.

The vector index stores chunk_id → vector plus filterable attributes (acl_groups, source, updated_at, doc_version), sharded by chunk_id hash. The keyword index stores the same chunks with the same attributes for BM25. Both are derived data and can be rebuilt from the relational store. A separate acl_versions(doc_id, version) lets a permission change patch only attributes without re-embedding.

High-level design

Ingestion path: source connectors watch each system (webhooks where available, polling otherwise) and put change events on a Kafka topic keyed by document id, so edits to one document stay in order. The ingest pipeline fetches the document, extracts text, splits it into chunks along headings and paragraphs, and compares each chunk's content hash with the stored one. Only new or changed chunks go to the embedding service, which batches them onto GPUs and upserts the vectors into the vector index; the same chunks go to the keyword index, and document metadata and ACLs go to the docs and ACL store. Deletes and permission changes skip embedding entirely and just update attributes in both indexes.

Query path: the query API authenticates the user and asks the identity service for their group ids. The RAG orchestrator rewrites follow-up questions into standalone queries, then runs dense and BM25 searches in parallel with the user's groups as a filter inside each index. The candidates are fused and sent to the reranker, which scores each (question, chunk) pair and keeps the best 6–10. The answer LLM receives numbered chunks and instructions to cite them and to answer "not found" when they do not support an answer. Citations are checked against the supplied ids before streaming to the user, and every trace goes to eval and feedback.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Chunking and incremental ingestion

Chunk along the document's structure (headings, paragraphs, table rows) to roughly 300–800 tokens with a small overlap, and prepend the heading path and document title to each chunk's embedded text so a chunk saying "it takes 10 days" still knows what "it" is. Keep chunk boundaries stable across edits by splitting at structure, then hash each chunk: a typo fix re-embeds one chunk, not the whole document. Write a new document version's chunks first, flip the version pointer, then delete old chunks, so readers never see a half-updated document. Record the embedding model on each chunk so a model upgrade can be backfilled into a parallel index and switched atomically.

Hybrid retrieval and reranking

Dense vectors capture meaning ("time off" matches "annual leave") but blur exact tokens such as error codes, ticket numbers and product names, which BM25 nails. Run both and merge with reciprocal rank fusion, which needs no score calibration between systems. Take ~50 from each, fuse to ~100, then run a cross-encoder reranker that reads the question and the chunk together; it is far more accurate than vector similarity but costs ~10–30 ms per batch of candidates on a GPU, so it only runs on the short list. Boost recent versions and authoritative sources slightly, and collapse many chunks from one document so a single long page does not crowd out others.

Permissions that cannot leak

Store each chunk's allowed group ids as an indexed attribute and pass the user's groups as a filter inside the ANN and BM25 searches. Post-filtering the top 50 is wrong twice: restricted users get few or zero results, and any bug in the filter sends forbidden text to the LLM, which can paraphrase it. Permission revocations bypass the normal pipeline on a priority topic and patch attributes in seconds. Resolve group membership at query time from the identity service (cached for a minute) rather than baking users into chunks, so team changes need no reindexing. Caches of answers must be keyed by the user's group set, or not used at all for restricted sources.

Citations and evaluation

Number the chunks in the prompt and require the model to cite [n] after each claim; after generation, verify that every cited id was supplied and drop or flag sentences without support. Evaluate in layers. Retrieval: a golden set of a few hundred real questions with known source documents, tracking recall@k and MRR on every chunking, embedding or ranking change. Generation: an LLM judge scores faithfulness to the supplied chunks and answer relevance, spot-checked by humans. Online: thumbs, citation click-through and "not found" rate per source. Gate deploys on the offline suite, then A/B the change on a slice of users.

Failure modes

• If the vector index is down, fall back to keyword search only and label answers as lower confidence rather than failing.
• If a connector stops syncing, stale documents keep serving; track per-source lag and show "last synced" on citations, alerting past the freshness SLO.
• If the LLM API is down or rate-limited, return the reranked passages as a search result page instead of an answer.
• If an embedding model upgrade degrades quality, the old index is still live; switch the alias back.
• If the identity service is unreachable, deny access to restricted sources (fail closed) and answer from public-to-company documents only.

Core algorithm in code

import hashlib
import math
import re

STOP = {"the", "a", "an", "of", "to", "is", "in", "for", "and", "how", "do", "i", "what", "on", "my", "we"}


def tokenize(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]


def embed(text, dims=64):
    v = [0.0] * dims
    for w in tokenize(text):
        h = int(hashlib.md5(w.encode()).hexdigest(), 16)
        v[h % dims] += 1.0 if (h >> 8) & 1 else -1.0
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]


def cosine(a, b):
    return sum(x * y for x, y in zip(a, b))  # both already unit length


def bm25_scores(query, chunks, k1=1.2, b=0.75):
    docs = {c["id"]: tokenize(c["text"]) for c in chunks}
    avg = sum(len(d) for d in docs.values()) / len(docs)
    scores = {}
    for q in set(tokenize(query)):
        df = sum(1 for d in docs.values() if q in d)
        if not df:
            continue
        idf = math.log(1 + (len(docs) - df + 0.5) / (df + 0.5))
        for cid, d in docs.items():
            tf = d.count(q)
            if tf:
                scores[cid] = scores.get(cid, 0) + idf * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len(d) / avg))
    return scores


def retrieve(query, chunks, user_groups, k=3, rrf_k=60, min_cos=0.2):
    allowed = [c for c in chunks if c["acl"] & user_groups]          # filter BEFORE ranking
    qv = embed(query)
    dense = sorted((c for c in allowed if cosine(qv, c["vec"]) >= min_cos), key=lambda c: -cosine(qv, c["vec"]))
    kw = bm25_scores(query, allowed)
    sparse = sorted((c for c in allowed if c["id"] in kw), key=lambda c: -kw[c["id"]])
    fused = {}
    for ranking in (dense[:20], sparse[:20]):
        for rank, c in enumerate(ranking):
            fused[c["id"]] = fused.get(c["id"], 0) + 1.0 / (rrf_k + rank + 1)
    by_id = {c["id"]: c for c in allowed}
    return [(by_id[i], s) for i, s in sorted(fused.items(), key=lambda kv: -kv[1])[:k]]


DOCS = [
    ("hr/leave.md#2", "Employees accrue 25 days of paid annual leave per year.", {"all"}),
    ("hr/leave.md#3", "Unused annual leave up to 5 days carries over to next year.", {"all"}),
    ("it/vpn.md#1", "Install the VPN client and sign in with SSO to reach internal tools.", {"all"}),
    ("eng/oncall.md#4", "The on-call engineer must acknowledge a page within 10 minutes.", {"eng"}),
    ("fin/comp.md#1", "Salary bands for 2026 and the annual leave payout formula.", {"finance"}),
    ("eng/deploy.md#2", "Deploys are frozen during the last week of the quarter.", {"eng"}),
]
chunks = [{"id": i, "text": t, "acl": acl, "vec": embed(t)} for i, t, acl in DOCS]

for user, groups, q in (("alice (eng)", {"all", "eng"}, "how many days of annual leave do I get"),
                        ("alice (eng)", {"all", "eng"}, "page acknowledge on-call minutes"),
                        ("bob (sales)", {"all"}, "on-call page acknowledge")):
    print("%s asks: %s" % (user, q))
    hits = retrieve(q, chunks, groups)
    for c, s in hits:
        print("   [%s] rrf=%.4f cos=%.2f  %s" % (c["id"], s, cosine(embed(q), c["vec"]), c["text"]))
    if not hits:
        print("   no permitted source supports an answer -> reply 'not found'")

Trade-offs

• Smaller chunks give more precise retrieval and citations but cost context, so the LLM sees fragments without surrounding meaning.
• A cross-encoder reranker substantially improves the top results but adds GPU cost and ~100 ms to every question.
• Hybrid search handles both concepts and exact identifiers but doubles the indexes to keep fresh and consistent.
• Filtering by ACL inside the index is safe but makes ANN search slower and less accurate for users with very narrow permissions.
• A hosted LLM API is fast to adopt and strong but costs per token and needs contractual data guarantees; self-hosting keeps data inside but costs GPU operations.

What a strong answer shows

• Treats ingestion, chunking and metadata as first-class design, not an afterthought.
• Uses hybrid retrieval plus a reranker and explains why each stage exists.
• Enforces permissions inside retrieval and handles revocation latency.
• Keeps the index fresh incrementally and plans for embedding model changes.
• Produces verifiable citations and a concrete evaluation loop with offline and online metrics.

Mistakes to avoid

• Saying "put the documents in a vector DB" and spending the interview on the LLM prompt.
• Filtering permissions after retrieval or, worse, asking the LLM to ignore restricted content.
• Fixed-size chunking that splits tables and loses headings.
• Re-embedding entire documents (or the whole corpus) on every change.
• No way to measure quality, so every change is judged by anecdotes.

Follow-ups to expect

• How would you handle questions that need many documents, such as "summarise all Q3 incidents"? (Map-reduce over a filtered set, or structured queries, not top-k.)
• How do you support multiple languages? (Multilingual embedding model; per-language BM25 analyzers.)
• How do you handle conflicting documents? (Prefer recency and authority signals; show both with dates.)
• How would you cache? (Embedding cache by query text; answer cache keyed by normalized question plus group set, invalidated by doc versions.)

Practice next