Design a Payment System

Design the payments backend for an online marketplace. Buyers pay by card at checkout; the money goes through an external card payment provider; the marketplace keeps a fee and owes the rest to the seller. Finance needs books that balance to the cent, and support needs to answer "was I charged?" with certainty.

The hard part is that the network lies. A call to the provider can time out after the card was actually charged, clients retry, webhooks arrive late, twice or out of order, and the provider's end-of-day report sometimes disagrees with what we recorded. Charging a customer twice, or losing track of money we collected, is not acceptable.

In 45 minutes, cover requirements and scale, the payment flow and its states, how you make retries safe, how money is recorded, and how you find and fix discrepancies.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · Commerce & Payments · problem 33 of 70

Ask before you design. Try answering each yourself, then compare.

1. Do we store card numbers?
   Typical answer

   No. The provider tokenises cards in the browser; we only see tokens, which keeps us out of most card-data compliance scope.

2. What volume?
   Typical answer

   About 10 million payments a day, peaking around 600 per second during sales events.

3. Is authorisation separate from capture?
   Typical answer

   Yes: authorise at checkout, capture when the order ships (or immediately for digital goods). Refunds are supported.

4. Do we pay out sellers in this design?
   Typical answer

   We record what we owe them in the ledger; the actual bank payout job is a follow-up.

5. Single currency?
   Typical answer

   Multiple currencies are possible, but each payment is in one currency; no currency conversion inside our system.

6. How fast must the buyer see a result?
   Typical answer

   Within a few seconds normally; in rare unknown cases we show 'processing' and settle it asynchronously.

7. One provider or several?
   Typical answer

   One primary provider, with the design open to adding a second for failover.

Requirements

Functional

• Create a payment for an order, authorise it with the provider, and capture it later or immediately.
• Refund fully or partially.
• Record every movement of money in a double-entry ledger (buyer, provider, seller payable, fees).
• Process provider webhooks for asynchronous outcomes (3-D Secure, delayed failures, disputes).
• Reconcile our records against the provider's settlement reports and flag differences.

Non-functional

• Exactly-once effect per payment: retries never double-charge.
• Strong consistency and durability for payment state and ledger (no lost writes, RPO 0).
• p99 under 3 s for a payment including the provider call; 99.99% availability for the API.
• ~600 payments per second at peak; ledger keeps 7+ years of history for audit.
• All changes auditable: append-only records, no in-place edits of money amounts.

Out of scope

• Building a card network or acquiring bank; we use a provider.
• Fraud scoring models (we call a risk check as a black box).
• Seller bank payouts and tax reporting.

Back-of-the-envelope estimates

• Writes: 10 M payments/day ≈ 115 per second on average, ~600 at sale peaks. Each payment writes a payment row, one to three attempt rows and three or four ledger entries, so ~3,000 row writes per second at peak, easily handled by a sharded relational database.
• Reads: ~5 status reads per payment (client polling, support, order service) ≈ 50 M/day, ~3,000 per second at peak.
• Storage: ~4 KB per payment including attempts and ledger entries: 10 M × 4 KB × 365 × 7 years ≈ 100 TB, ~300 TB replicated. Older years move to cheaper archival storage but stay queryable for audit.
• Provider latency: 300 ms–2 s per call, with a tail of timeouts. At 600 per second and 2 s each, ~1,200 calls are in flight, so the service must be asynchronous or have generous connection pools.

Volume is modest; the design is driven by correctness under partial failure, not throughput. Every number above fits on a few database shards, which is why we can afford synchronous, transactional writes everywhere.

API

POST /v1/payments   Idempotency-Key: <uuid>
     {orderId, amount: 2500, currency: "EUR", paymentMethodToken, captureMode: "manual"}
     -> 201 {paymentId, status: AUTHORIZED | REQUIRES_ACTION | PROCESSING | FAILED}
POST /v1/payments/{id}/capture   Idempotency-Key  {amount?}
POST /v1/payments/{id}/refunds   Idempotency-Key  {amount, reason}
GET  /v1/payments/{id}                        -> {status, amount, attempts[], ledgerTxnIds[]}
POST /internal/webhooks/provider  (signed)    -> 200 once stored

Every mutating call takes an idempotency key, stored with the request hash and the final response, so a retry returns the original answer and a reused key with a different body is rejected.

Data model

Payments DB (PostgreSQL, sharded by payment_id): payment(payment_id, order_id, merchant_id, amount, currency, status, provider_ref, version); payment_attempt(attempt_id, payment_id, op, provider_idempotency_key, status, request, response, created_at); idempotency(key, client_id, request_hash, response, status) with a unique constraint on (client_id, key); and an outbox table. Relational because we need transactions, constraints and exact decimal math.

Ledger DB (append-only, sharded by account): ledger_txn(txn_id, source_ref, created_at) and ledger_entry(txn_id, account_id, amount_minor, currency). Each transaction's entries sum to zero per currency. Balances are derived by summing entries, with periodic snapshot rows so a balance read does not scan years of history. Entries are never updated; a correction is a new reversing transaction.

Settlement files: provider reports land in object storage and are parsed into a settlement_line table for reconciliation.

High-level design

Paying. The checkout client sends a payment request with an idempotency key through the API gateway. The payment service inserts the idempotency record and a PENDING payment in one transaction; if the key already exists it returns the stored response. It then writes an attempt row, and the PSP adapter calls the card provider, passing a provider idempotency key derived from the attempt. On a clear answer the service moves the payment to AUTHORIZED or FAILED and, in the same transaction, writes an outbox event.

Unknown outcomes. If the provider call times out, the payment stays PROCESSING. The recovery worker scans for attempts stuck longer than a minute, asks the provider for the status by idempotency key (or safely retries with the same key), and finishes the state transition. The webhook receiver verifies the provider's signature, stores the raw event, and feeds it through the same state machine, which ignores transitions that have already happened.

Recording money. Outbox events are relayed to the payment events stream. The ledger service consumes them and posts balanced double-entry transactions (receivable from provider, payable to seller, fee revenue), keyed by event id so redelivery posts nothing new.

Checking. Each day the provider publishes settlement files. The reconciliation job matches every settled charge, refund and fee with our ledger, and raises breaks: money the provider moved that we did not record, or the reverse.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Idempotency end to end

There are two layers. Client to us: the client generates a key per logical payment; we insert (client_id, key, request_hash) under a unique constraint before doing anything. A concurrent duplicate blocks on the constraint and then reads the stored outcome; a key reused with a different body gets a 422. Us to provider: each attempt has its own key that we send to the provider, so a retried network call after a timeout is deduplicated on their side too. Never generate a fresh provider key on retry of the same attempt; that is exactly how double charges happen. Keys expire after a window (say 7 days) that is longer than any client retry policy.

Payment state machine and unknown outcomes

States: CREATED → PROCESSING → AUTHORIZED → CAPTURED, with FAILED, CANCELED, REFUNDED and PARTIALLY_REFUNDED as exits, and REQUIRES_ACTION for 3-D Secure. Every transition is a conditional update on (status, version), so the synchronous response, a webhook and the recovery worker can all race safely: the first one wins, the others become no-ops. A timeout is not a failure; it is PROCESSING with an unknown result. Tell the buyer 'processing', never 'failed', until the recovery worker or a webhook resolves it, and only then release the order. Retries use exponential backoff with jitter and a cap, and only for errors the provider marks as retryable.

The double-entry ledger

A captured €25.00 payment with a 3% fee becomes one ledger transaction: +2500 provider receivable, −2425 seller payable, −75 fee revenue. Entries sum to zero, are immutable, and reference the payment event that caused them. When the provider later pays out, another transaction moves the receivable into our bank account; a refund posts the reverse. Because nothing is overwritten, any balance at any past time can be recomputed, auditors can trace each cent, and a bug is fixed by posting corrections rather than editing history. Use integer minor units per currency, never floats. Shard by account, and keep balance snapshots per account per day so reads are fast.

Reconciliation

Even with idempotency, things drift: a webhook we never received, a refund issued in the provider's dashboard by a support agent, a fee schedule change. Each day, load the provider's settlement report and match line by line on provider reference and amount against ledger entries and payment rows. Classify results: matched, missing on our side, missing on theirs, amount mismatch. Most breaks auto-resolve (for example, a late webhook arriving within 24 hours); the rest go to a finance queue. Also run lighter hourly checks between payments DB and ledger, so an internal bug in the consumer is caught within the hour rather than at month end.

Failure modes

• If the provider times out, the payment stays PROCESSING; the recovery worker queries or retries with the same key and resolves it, so the customer is charged at most once.
• If the payments DB primary fails, writes pause until a synchronous replica is promoted; no acknowledged payment is lost because commits wait for the replica.
• If the ledger consumer is down, events accumulate in the stream; when it recovers it posts them in order, idempotently by event id.
• If webhooks arrive twice or out of order, the state machine ignores transitions it has already made and rejects ones that would move backwards.
• If the provider is fully down, a circuit breaker fails fast with a clear error (or routes to a secondary provider) instead of piling up threads.

Core algorithm in code

import random


class ProviderTimeout(Exception):
    pass


class FlakyProvider:
    """Card provider stand-in. It dedupes on the idempotency key we send, and sometimes the
    response is lost after the charge went through - the classic 'unknown' outcome."""
    def __init__(self, seed=3):
        self.charges = {}                      # key -> (charge_id, amount)
        self.rng = random.Random(seed)

    def charge(self, key, amount):
        if key not in self.charges:
            self.charges[key] = ("ch_%d" % (len(self.charges) + 1), amount)
        if self.rng.random() < 0.6:
            raise ProviderTimeout()            # charged, but we never heard back
        return self.charges[key][0]

    def lookup(self, key):
        """Status query by idempotency key (what the recovery worker calls)."""
        return self.charges[key][0] if key in self.charges else None

    def settlement_report(self):
        return [{"charge_id": cid, "amount": amt} for cid, amt in self.charges.values()]


class Ledger:
    """Double-entry, append-only. Amounts are integer cents; each transaction sums to zero."""
    def __init__(self):
        self.entries, self.txns = [], set()

    def post(self, txn_id, entries):
        if txn_id in self.txns:
            return                              # idempotent posting
        assert sum(a for _, a in entries) == 0, "unbalanced transaction"
        self.txns.add(txn_id)
        self.entries += [(txn_id, acct, amt) for acct, amt in entries]

    def balance(self, account):
        return sum(a for _, acct, a in self.entries if acct == account)


class PaymentService:
    def __init__(self, provider, ledger, max_attempts=5):
        self.provider, self.ledger, self.max_attempts = provider, ledger, max_attempts
        self.payments = {}                      # idempotency key -> payment record (DB row)

    def pay(self, key, merchant, amount):
        p = self.payments.get(key)
        if p and p["status"] in ("SUCCEEDED", "FAILED"):
            return p                            # replay: same answer, no second charge
        p = p or {"key": key, "merchant": merchant, "amount": amount, "status": "PENDING", "attempts": 0}
        self.payments[key] = p
        while p["attempts"] < self.max_attempts:
            p["attempts"] += 1
            try:
                charge_id = self.provider.charge(key, amount)    # same key on every retry
            except ProviderTimeout:
                continue                        # outcome unknown: retrying with the key is safe
            return self._succeed(p, charge_id)
        p["status"] = "UNKNOWN"                 # left for the recovery worker / reconciliation
        return p

    def _succeed(self, p, charge_id):
        p.update(status="SUCCEEDED", charge_id=charge_id)
        fee = p["amount"] * 3 // 100
        self.ledger.post(charge_id, [
            ("provider_receivable", p["amount"]),
            ("merchant_payable:" + p["merchant"], -(p["amount"] - fee)),
            ("fee_revenue", -fee),
        ])
        return p

    def recover(self, key):
        """Recovery worker: ask the provider what happened instead of guessing."""
        p = self.payments[key]
        charge_id = self.provider.lookup(key)
        if charge_id:
            return self._succeed(p, charge_id)
        p["status"] = "FAILED"                  # never reached the provider: safe to fail
        return p


def reconcile(ledger, report):
    ours = {}
    for txn, acct, amt in ledger.entries:
        if acct == "provider_receivable":
            ours[txn] = amt
    theirs = {r["charge_id"]: r["amount"] for r in report}
    issues = []
    for cid in sorted(set(ours) | set(theirs)):
        if ours.get(cid) != theirs.get(cid):
            issues.append((cid, "ledger=%s" % ours.get(cid), "provider=%s" % theirs.get(cid)))
    return issues


provider, ledger = FlakyProvider(), Ledger()
svc = PaymentService(provider, ledger, max_attempts=3)
keys = ["order-%d" % i for i in range(1, 9)]
for k in keys:
    p = svc.pay(k, "m_books", 2500)
    print(k, p["status"], "after", p["attempts"], "attempt(s)")
print("client retries order-1:", svc.pay("order-1", "m_books", 2500)["status"],
      "| provider charges for order-1:", sum(1 for k in provider.charges if k == "order-1"))
print("owed to merchant (credit balance):", -ledger.balance("merchant_payable:m_books"), "cents | ledger sums to",
      sum(a for _, _, a in ledger.entries))
issues = reconcile(ledger, provider.settlement_report())
print("reconciliation breaks:", issues or "none")
for k, p in svc.payments.items():
    if p["status"] == "UNKNOWN":
        print("recovery worker looks up", k, "->", svc.recover(k)["status"])
print("after recovery:", reconcile(ledger, provider.settlement_report()) or "ledger matches provider")

Trade-offs

• Synchronous provider calls give buyers an immediate answer but cost tail latency and held threads; async with polling scales better but complicates the UX.
• An append-only ledger gives auditability and easy correction but costs storage and needs balance snapshots for fast reads.
• Posting ledger entries from events (eventual) decouples the ledger from the payment path but costs a short window where payment state and ledger disagree.
• Retrying aggressively improves success rates but risks hammering a degraded provider; backoff and circuit breakers trade recovery time for stability.
• A second provider for failover improves availability but doubles reconciliation work and complicates idempotency across providers.

What a strong answer shows

• Uses idempotency keys at both boundaries (client to us, us to provider) and explains the unknown-outcome case.
• Defines a payment state machine with guarded transitions that tolerate races between response, webhook and recovery.
• Records money in an immutable double-entry ledger with integer minor units.
• Includes reconciliation against provider reports as a first-class component.
• Uses a transactional outbox so state changes and their events cannot diverge.
• Chooses strong consistency and durable storage, and explains why throughput is not the bottleneck.

Mistakes to avoid

• Treating a timeout as a failure and letting the client pay again with a new key.
• Keeping a mutable balance column updated in place instead of a ledger.
• Using floating-point numbers for money.
• Publishing events to the stream outside the database transaction, so a crash loses or invents them.
• Skipping reconciliation because 'idempotency makes it unnecessary'.

Follow-ups to expect

• How would you add a second provider for failover? (Routing layer per payment, provider id stored on the attempt, reconciliation per provider.)
• How do you handle chargebacks? (Webhook event moves the payment to DISPUTED and posts ledger entries for the held funds.)
• How would you pay sellers? (Daily payout job sums seller payable balances, creates bank transfers, and posts ledger entries.)
• How do you support multiple currencies? (Separate ledger accounts per currency; FX only at explicit conversion transactions.)
• How would you prevent a single merchant's spike from starving others? (Per-merchant rate limits and separate worker pools.)

Practice next