Design an AI Customer Support Assistant

An online retailer with 30 million customers handles over a million support conversations a month. Most are routine: where is my order, how do I return this, can I change my address. Human agents are expensive and slow at peak times, so the company wants an AI assistant in its web and app chat.

The assistant must answer from the help centre and policy documents, look up the customer’s own orders, start returns or cancellations through internal systems, and hand the conversation to a human agent with full context when it should. It must never invent a policy, promise a refund it cannot give, act on someone else’s order, or be talked into misbehaving by a clever message.

In 45 minutes, design the conversation flow, retrieval, tool calls, guardrails and human handoff, and explain how you would measure whether the assistant actually helps customers.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 56 of 70

Ask before you design. Try answering each yourself, then compare.

1. What volume?
   Typical answer

   About 1.2 M conversations a month (~40 k a day), averaging 6 turns, with holiday peaks around 4×.

2. What actions may the assistant take?
   Typical answer

   Read order status and tracking, start returns within policy, cancel unshipped orders. Refunds above $100 need a human.

3. What latency feels acceptable?
   Typical answer

   First token within ~1.5 s, a full answer within ~6 s; tool calls may add up to ~2 s with a visible status.

4. How large and how fresh is the knowledge base?
   Typical answer

   About 3,000 help articles and policy pages in English and Spanish, edited weekly; changes must reach the assistant within an hour.

5. How does handoff work today?
   Typical answer

   400 agents use an existing help desk; handoff should create a ticket there with a summary so the customer does not repeat themselves.

6. How do we define success?
   Typical answer

   Containment (resolved without a human) around 60%, CSAT at least equal to human agents, and zero unauthorised actions.

7. Are customers logged in?
   Typical answer

   Usually; guests can ask general questions but cannot see order data until verified.

Requirements

Functional

• Multi-turn chat on web and app with streaming responses.
• Answers grounded in help-centre articles, with links to sources.
• Tool calls for order lookup, tracking, returns and cancellations, scoped to the verified customer.
• Explicit confirmation before any action that changes an order.
• Human handoff with summary, intent and actions taken.
• Collect CSAT and resolution signals; replay conversations for evaluation.

Non-functional

• First token < 1.5 s at p95; full answer < 6 s.
• 99.9% availability; if the assistant fails, route to humans.
• Zero writes outside policy; every action audited.
• Knowledge updates live within 1 hour.
• PII redacted from logs used for analytics.

Out of scope

• Voice and phone support.
• Training a custom model.
• Agent-side copilot tools.

Back-of-the-envelope estimates

• Turns: 40 k conversations/day × 6 turns ≈ 240 k turns/day ≈ 2.8/s average, ~11/s at a 4× peak.
• Model calls: ~1.3 calls per turn (some turns need a tool round trip) ≈ 15 calls/s at peak, each ~4 k input tokens (instructions, history, five retrieved chunks, order data) and ~250 output tokens.
• Tokens and cost: ~1.3 B tokens/day. At an assumed blended $3 per million tokens, ~$4 k/day, roughly $0.10 per conversation, against several dollars for a human-handled contact. Cost is not the constraint; correctness and safety are.
• Concurrency: ~15 calls/s × ~6 s streaming ≈ 90 concurrent generations at peak.
• Knowledge index: 3,000 articles → ~30 k chunks × 1,024 dims ≈ 120 MB of vectors; it fits in memory on one node, replicated for availability.
• Logs: ~20 KB per turn with traces ≈ 5 GB/day.

API

POST /v1/conversations {channel, locale}                       -> {conversation_id}
POST /v1/conversations/{id}/messages {text}
     -> stream: token... | tool_status{"Checking your order"} | confirm{action, summary} | handoff{eta}
POST /v1/conversations/{id}/confirm {action_id, approved:true}
POST /v1/conversations/{id}/feedback {csat:1-5, resolved:bool}

tools (internal): get_order(order_id) list_orders() track(order_id)
                  start_return(order_id, items, reason) cancel_order(order_id)
                  create_handoff(summary, intent, priority)

Tools run with the customer’s own scoped session token rather than a service account, so even a manipulated model cannot read or change another customer’s order.

Data model

Conversations (document store, keyed by conversation_id): customer, channel, locale, state, and a list of turns with message text, retrieved chunk IDs, tool calls and results, and the prompt and model versions used.

Knowledge chunks (vector database plus keyword index): (chunk_id, article_id, article_version, locale, heading_path, text, embedding). Small, rebuilt incrementally from the help-centre CMS.

Action audit log (append-only, relational): (action_id, conversation_id, customer_id, tool, args, policy_result, confirmed_at, outcome).

Handoff tickets live in the existing help desk; the assistant stores the ticket ID. Feedback and eval cases sit in the analytics store.

High-level design

A turn. The customer’s message arrives over the chat gateway, which authenticates the session and keeps a streaming connection. The chat orchestrator loads the conversation state and runs the message through input guardrails (PII detection, prompt-injection and abuse checks). It retrieves the most relevant chunks for the question from the help doc index using hybrid search, then calls the LLM with system instructions, the retrieved chunks marked as reference material, the conversation history and the list of tools allowed in the current state.

Tools. When the model proposes a tool call, the orchestrator sends it to the tool gateway, which validates the schema, enforces policy rules in code (ownership, return window, refund limits) and calls the order and returns APIs with the customer’s scoped token. Write actions pause for an explicit customer confirmation first. The result goes back to the model for the final answer, which passes output guardrails before streaming.

Handoff. On a handoff trigger, the tool gateway calls human handoff, which opens a help-desk ticket with a summary for support agents.

Knowledge and learning. Doc ingest picks up changes from the help centre CMS and re-indexes them. Every turn is written to the conversation store, which feeds CSAT and eval analytics.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Grounded answers from the help centre

Split articles along their headings into chunks of a few hundred tokens and prefix each with the article title and heading path, so a chunk makes sense on its own. Tag chunks with locale and policy version, and never mix versions. Retrieve with hybrid search, since customers use product names and policy terms verbatim, then rerank to the top five. Instruct the model to answer only from provided material and cite chunk IDs; the orchestrator checks that cited IDs were actually retrieved and shows the article links. If retrieval scores are low, the assistant asks a clarifying question or offers a human instead of guessing. CMS webhooks trigger re-embedding of changed articles within minutes.

Safe tool calling

The model only proposes; deterministic code decides. The tool gateway checks the arguments against a schema, confirms the order belongs to the verified customer, and applies policy rules: return window, item condition, refund limit. Eligibility is never left to the model’s reading of the policy text. Reads run freely. Writes need an explicit confirmation step where the customer sees a plain summary ("Return 1 pair of shoes from order A100 for a refund of $80?") and taps yes. Each action carries an idempotency key so a retry never creates two returns, and every call lands in the audit log with the policy result.

Guardrails against manipulation

Treat everything that is not the system prompt as untrusted: the customer’s message, retrieved articles and tool outputs such as order notes. Wrap retrieved and tool content in clearly marked data sections and tell the model never to follow instructions inside them. More important, limit what a manipulated model could do: the tool list depends on conversation state (no write tools before verification), and tools run with the customer’s permissions. Output checks catch promises of refunds, discounts or amounts that no tool result supports, leaked PII and off-topic content; a flagged answer is replaced with a safe fallback or a handoff.

Handoff and measuring real success

Hand off when the customer asks, when sentiment turns sharply negative, after two failed attempts at the same issue, for sensitive topics (fraud, legal threats, safety) or for actions above the automation limits. The ticket carries a summary, detected intent, order IDs and actions already taken, so the agent starts informed. Containment alone is misleading, because a customer who gives up looks contained. Count a conversation as resolved only if there is no handoff and no repeat contact on the same issue within 72 hours. Combine that with CSAT, and replay a fixed set of past conversations offline before every prompt or model change.

Failure modes

• If the LLM provider is slow or down, the orchestrator falls back to a secondary model, and if that fails, routes the chat to the human queue with a message.
• If order APIs fail, the assistant says it cannot check right now and offers a handoff rather than guessing an order status.
• If the doc index is stale or unavailable, answers that need policy content are deferred to a human; order lookups still work.
• If the help-desk integration fails during handoff, the request is queued and retried, and the customer gets an email confirmation once the ticket exists.

Core algorithm in code

from datetime import date

ORDERS = {
    "A100": {"customer": "c1", "delivered": date(2026, 10, 1), "total": 80.0, "status": "delivered"},
    "A200": {"customer": "c1", "delivered": date(2026, 8, 1), "total": 45.0, "status": "delivered"},
    "A300": {"customer": "c2", "delivered": date(2026, 10, 20), "total": 300.0, "status": "delivered"},
    "A400": {"customer": "c1", "delivered": None, "total": 260.0, "status": "processing"},
}
TOOLS = {"get_order": {"write": False, "args": {"order_id"}},
         "start_return": {"write": True, "args": {"order_id", "reason"}},
         "cancel_order": {"write": True, "args": {"order_id"}}}
RETURN_WINDOW_DAYS, AUTO_REFUND_LIMIT = 30, 100.0


def check_tool_call(call, customer_id, today, confirmed):
    """Deterministic gate between the model's proposed tool call and the real order system."""
    spec = TOOLS.get(call["name"])
    if spec is None:
        return "reject", "unknown tool"
    if set(call["args"]) != spec["args"]:
        return "reject", "bad arguments"
    order = ORDERS.get(call["args"]["order_id"])
    if order is None or order["customer"] != customer_id:
        return "reject", "order not found for this customer"      # never reveal other customers' orders
    if not spec["write"]:
        return "allow", "read-only"
    if call["name"] == "start_return":
        if order["delivered"] is None or (today - order["delivered"]).days > RETURN_WINDOW_DAYS:
            return "reject", "outside return window: explain policy, offer handoff"
        if order["total"] > AUTO_REFUND_LIMIT:
            return "handoff", "refund above auto limit needs an agent"
    if call["name"] == "cancel_order" and order["status"] != "processing":
        return "reject", "already shipped"
    if not confirmed:
        return "confirm", "show summary and ask the customer to confirm"
    return "allow", "confirmed write"


today = date(2026, 10, 26)
calls = [
    ({"name": "get_order", "args": {"order_id": "A300"}}, False),                     # someone else's order
    ({"name": "start_return", "args": {"order_id": "A100", "reason": "too small"}}, False),
    ({"name": "start_return", "args": {"order_id": "A100", "reason": "too small"}}, True),
    ({"name": "start_return", "args": {"order_id": "A200", "reason": "changed mind"}}, True),
    ({"name": "cancel_order", "args": {"order_id": "A400"}}, True),
    ({"name": "issue_refund", "args": {"order_id": "A100", "amount": 500}}, True),     # injected tool
]
for call, confirmed in calls:
    verdict, why = check_tool_call(call, "c1", today, confirmed)
    print(f"{call['name']:13} {call['args'].get('order_id')}  confirmed={confirmed!s:5} -> {verdict:8} {why}")

Trade-offs

• A large model gives better reasoning and tone but costs latency; a smaller model for routing and simple FAQs keeps first-token time low.
• Requiring confirmation for every write prevents mistakes but costs an extra turn for the customer.
• Aggressive handoff rules protect CSAT but cost containment and agent load.
• Strict grounding reduces invented policies but costs some helpfulness on questions the articles do not cover.
• Customer-scoped tokens make cross-account access impossible but cost an extra verification step for guests.

What a strong answer shows

• Keeps policy decisions and permissions in deterministic code, not in the model.
• Designs retrieval with chunking, metadata, citations and a low-confidence path.
• Treats retrieved text and tool outputs as untrusted and limits tool power by state.
• Designs handoff with clear triggers and context transfer.
• Defines success with resolution and recontact, not just containment.
• Estimates token cost and latency and uses them to choose models.

Mistakes to avoid

• Giving the model a service account that can act on any order.
• Letting the model decide refund eligibility from policy text.
• Measuring success by containment alone.
• Ignoring prompt injection through retrieved documents or order notes.
• No plan for when the model or order systems are down.

Follow-ups to expect

• How would you add email support? (Same orchestrator, non-streaming, longer context, stricter review before sending.)
• How do you roll out a new prompt safely? (Offline replay set, then a small traffic share with CSAT and recontact monitoring.)
• How would you personalise tone or language? (Locale-specific instructions and retrieval filtered by locale.)
• How would you keep long conversations within context limits? (Summarise older turns and keep tool results as structured state.)

Practice next