Design a Real-time Payment Fraud Detection System

A payment processor authorises card payments for thousands of online merchants. Each authorisation must get a fraud decision (approve, decline, ask for extra authentication, or send to manual review) while the shopper waits at checkout.

Fraud is rare, about one payment in a thousand, but each miss costs the full amount plus fees, and every wrong decline loses a sale and annoys a real customer. Fraudsters test stolen cards with bursts of small payments, then cash out fast. The truth about whether a payment was fraud often arrives as a chargeback weeks later. Risk analysts need to add rules within minutes when a new attack appears, and declined customers and regulators expect a reason.

In 45 minutes, design the scoring path within its latency budget, the streaming features, how rules and the model combine, how labels flow back into training, and how decisions are explained.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 50 of 70

Ask before you design. Try answering each yourself, then compare.

1. What volume and peaks?
   Typical answer

   About 300 M transactions a day, ~3.5 k/s average and ~12 k/s at holiday peaks.

2. What is the latency budget?
   Typical answer

   The fraud decision gets 50 ms at p99 inside a ~300 ms authorisation.

3. What decisions are possible?
   Typical answer

   Approve, decline, challenge (step-up authentication such as a one-time code) or queue for manual review on high-value orders.

4. How quickly do we learn the truth?
   Typical answer

   Chargebacks arrive 30-90 days later; analyst confirmations and customer reports arrive in days.

5. What happens if the fraud service is down?
   Typical answer

   Payments must not stop. Fall back to a rules-only decision with conservative limits.

6. Who writes rules?
   Typical answer

   Risk analysts, without a code deploy, after testing on historical traffic.

7. Do decisions need explanations?
   Typical answer

   Yes: top reason codes for analysts and merchants, stored with each decision.

Requirements

Functional

• Score each authorisation and return approve, decline, challenge or review with reason codes.
• Compute streaming velocity features per card, device, IP, email and merchant.
• Combine analyst-authored rules with an ML model; rules deploy in shadow then active mode.
• Ingest labels from chargebacks, analysts and customers and retrain regularly.
• Case management queue for analysts reviewing held payments.

Non-functional

• p99 decision latency < 50 ms; 99.99% availability with a degraded fallback.
• 12 k decisions/s at peak.
• Velocity features reflect a payment within 1 s for critical counters.
• Decisions are idempotent per transaction ID and stored for 7 years for audits.
• Rule changes live within 5 minutes after approval.

Out of scope

• Card network messaging and settlement.
• Merchant onboarding risk.
• Anti-money-laundering investigations.

Back-of-the-envelope estimates

• Decisions: 300 M/day ≈ 3.5 k/s average, ~12 k/s at peak.
• Feature reads: each decision reads ~8 entities (card, device, IP, email, account, merchant, card BIN, shipping address) → ~100 k key reads/s at peak.
• Feature writes: ~30 window aggregates per payment → ~360 k counter updates/s at peak. This write rate drives the online store choice.
• Model: a gradient-boosted tree model over ~500 features takes ~1 ms on a CPU core; 12 k/s needs ~15 cores, ~50 with headroom and redundancy.
• Storage: ~2 KB per decision with its feature snapshot ≈ 600 GB/day, ~220 TB/year, mostly in a cold tier for the 7-year retention.
• Latency budget: ~5 ms network and parsing, ~15 ms parallel feature fetch, ~3 ms model, ~2 ms rules, leaving ~25 ms slack for tail latency.

API

POST /v1/score {txn_id, card_token, amount, currency, merchant_id, mcc, device_id, ip,
                email_hash, channel, ts}
  -> {decision: approve|decline|challenge|review, score, reason_codes[],
      model_version, rules_hit[]}               # idempotent on txn_id
POST /v1/labels {txn_id, label: fraud|legit, source: chargeback|analyst|customer, ts}
POST /v1/rules  {expression, action, priority, mode: shadow|active}
GET  /v1/decisions/{txn_id}                     -> decision, features, reasons

Scoring is idempotent on txn_id: the payment switch retries on timeout, and a retry must return the stored decision instead of re-scoring with counters that now include the first attempt.

Data model

Online features (in-memory key-value store with replication): key (entity_type, entity_id), value = ring buffers of minute and hour buckets holding counts, sums and small distinct-count sketches. Hash-partitioned by entity; one read returns all windows for that entity.

Decision log (append-only, wide-column, partitioned by day and txn_id): request, full feature snapshot, score, decision, rule hits, model and rule versions. Mirrored to the data lake.

Labels: (txn_id, label, source, received_at), joined to the decision log for training.

Rules (relational, versioned): expression, action, mode, author, approval. Small and read through a cache by every scoring node.

High-level design

Scoring path. The payment switch calls the scoring API with the transaction. The API checks for an existing decision for that txn_id, then hands off to the decision service. In parallel, the decision service fetches aggregates for every entity in the payment from the online features store, with a 15 ms timeout and defaults for anything missing. It builds the feature vector, calls the model server for a fraud probability and per-feature contributions, and evaluates the rules engine, which holds hard rules (blocked cards, sanctioned countries) and analyst rules. The final decision combines both: hard rules win, otherwise cost-based thresholds on the score choose approve, challenge, review or decline. The response carries reason codes.

Learning path. The decision, with its exact feature snapshot, goes to the decision events stream. The stream aggregator updates velocity counters from it, and the events land in the decision lake. The chargeback feed and analyst outcomes attach labels there. The training pipeline builds datasets from logged features plus matured labels and ships new models to the model server in shadow mode first. Fraud analysts write and backtest rules against the lake.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Holding the 50 ms budget

All feature reads go out in parallel with per-call timeouts; a missing entity gets a default plus a "missing" indicator that the model was trained with, so a slow shard lowers accuracy slightly instead of failing the payment. Keep the online store in the same zone as the decision service, and send a hedged read to a replica when the first is slower than its p95. The model is a tree ensemble on CPU, loaded in memory and warmed at start. If the whole scoring path is unavailable, the switch applies a local fallback: approve low amounts on known cards, challenge the rest, decline only hard-rule matches.

Streaming velocity features

Features like "payments on this card in the last 10 minutes" or "distinct cards used on this device today" are kept as rings of minute buckets per entity, so updates and reads are constant time, with approximate distinct counts from small sketches. Card-testing attacks fire bursts faster than an async pipeline can catch up, so the few most important counters are incremented synchronously during scoring, while the long tail is updated from the stream within a second. Define exactly whether the current payment is counted, and log the feature snapshot with every decision; training on logged values removes any mismatch between the online and offline computations.

Delayed and biased labels

Chargebacks take up to 90 days, so a payment only becomes a reliable negative after that. Train on payments old enough to have mature labels, and use early signals (analyst confirmations, customer reports) for recent ones. Declined payments never get a chargeback, so the model never learns whether they were fraud. Keep a small, controlled exploration set: for a tiny random slice of low-amount payments in the decline band, challenge instead of declining and treat the outcome as a label. New models run in shadow on live traffic, then go to a small share as a challenger before taking over.

Rules, model and explanations

Rules handle hard policy and give analysts a fast response to a new attack; the model generalises across hundreds of weak signals. Each new rule runs first in shadow mode and is backtested on the last 30 days of logged decisions, showing how many payments it would have hit and how many were fraud. The threshold is cost-based: decline when probability × amount × expected loss is much larger than the cost of a false decline, challenge in the middle. Reason codes come from per-feature contributions for this prediction, mapped to readable codes such as "unusual amount for this card", and stored with the decision.

Failure modes

• If the online feature store is slow or partitioned, reads time out to defaults with missing flags; decisions continue with lower accuracy, and alerts fire on the default rate.
• If the model server fails, the decision service uses rules plus a conservative amount limit and challenges more payments.
• If the stream aggregator lags, critical synchronous counters still work; long-window features go stale and their age is logged with the decision.
• If a bad rule is activated and declines spike, a per-rule decline-rate monitor auto-reverts it to shadow mode.

Core algorithm in code

import math


class Velocity:
    """Count and sum per entity over a sliding window, kept as one-minute buckets."""
    def __init__(self, window_s, bucket_s=60):
        self.window, self.bucket, self.b = window_s, bucket_s, {}

    def add(self, ts, amount):
        k = ts // self.bucket
        c, s = self.b.get(k, (0, 0.0))
        self.b[k] = (c + 1, s + amount)
        for old in [x for x in self.b if x <= (ts - self.window) // self.bucket]:
            del self.b[old]                       # evict buckets that left the window

    def get(self, ts):
        lo = (ts - self.window) // self.bucket
        vals = [v for k, v in self.b.items() if k > lo]
        return sum(c for c, _ in vals), sum(s for _, s in vals)


WEIGHTS = {"txn_10m": 0.45, "amount_vs_avg": 0.9, "new_device": 1.4, "foreign_ip": 0.8}
BIAS = -6.0
BLOCKED_BINS = {"999999"}


def score(features):
    z = BIAS + sum(WEIGHTS[k] * v for k, v in features.items())
    contrib = sorted(((WEIGHTS[k] * v, k) for k, v in features.items()), reverse=True)
    return 1 / (1 + math.exp(-z)), [k for c, k in contrib[:2] if c > 0]


def decide(txn, features, loss_rate=0.9, false_decline_cost=4.0):
    if txn["bin"] in BLOCKED_BINS:                         # hard rule first
        return "decline", 1.0, ["rule:blocked_bin"]
    p, reasons = score(features)
    expected_loss = p * txn["amount"] * loss_rate
    if expected_loss > 5 * false_decline_cost:
        return "decline", p, reasons
    if expected_loss > false_decline_cost:
        return "challenge", p, reasons                     # step-up auth is cheaper than a decline
    return "approve", p, reasons


card = Velocity(window_s=600)
avg_amount = 40.0
stream = [(0, 25, False, False), (60, 30, False, False), (420, 900, True, True),
          (450, 880, True, True), (470, 870, True, True), (500, 15, False, False),
          (520, 60, True, False)]
for ts, amount, new_dev, foreign in stream:
    count, _ = card.get(ts)                                # read before this txn is added
    feats = {"txn_10m": count, "amount_vs_avg": amount / avg_amount / 5,
             "new_device": int(new_dev), "foreign_ip": int(foreign)}
    d, p, why = decide({"bin": "411111", "amount": amount}, feats)
    card.add(ts, amount)
    print(f"t={ts:>3}s ${amount:>4}  prior_10m={count}  p={p:.3f}  -> {d:9} {why}")
print(decide({"bin": "999999", "amount": 5}, {"txn_10m": 0, "amount_vs_avg": 0, "new_device": 0, "foreign_ip": 0}))

Trade-offs

• Synchronous counter updates catch fast card-testing bursts but cost write latency on the scoring path, so only a few critical counters get them.
• Tree models on CPU are fast and explainable but cost some accuracy compared with deep sequence models over a card’s history.
• Cost-based thresholds that grow with the amount reduce losses but cost more friction on large legitimate purchases.
• Exploration on declines fixes label bias but costs a small, budgeted amount of fraud loss.
• Failing open with rules-only keeps payments flowing but costs higher fraud during outages.

What a strong answer shows

• Breaks the 50 ms budget into parallel steps with timeouts and a fallback.
• Designs streaming velocity features with clear windows and storage.
• Explains how rules and model combine and how rules are safely deployed.
• Handles delayed labels and the bias from declined payments.
• Uses cost-based thresholds rather than a single fixed cut-off.
• Stores explanations and feature snapshots for audits and training.

Mistakes to avoid

• Computing velocity features by querying the transaction database at request time.
• Training on recomputed features that differ from what was served.
• Treating unlabelled recent payments as legitimate.
• Failing closed and declining every payment when the service is down.
• Ignoring idempotency, so retries double-count or flip decisions.

Follow-ups to expect

• How would you detect fraud rings? (Build a graph of cards, devices, emails and addresses; run offline community detection and serve ring scores as features.)
• How would you add a sequence model? (Run it on GPU asynchronously for review cases, or on a short recent-events window within budget.)
• How do you support merchant-specific risk appetite? (Per-merchant threshold profiles applied after scoring.)
• How would you go multi-region? (Region-local scoring with per-card affinity; replicate counters asynchronously and accept small gaps.)

Practice next