Design a Content Moderation System

A social app with 300 million daily users receives about 500 million posts a day, many with images. The trust and safety team must find posts that break the rules (hate, harassment, nudity, graphic violence, spam, self-harm content) and act on them: remove, limit reach, add a warning, or strike the account.

Machine-learning classifiers can catch most of it, but they make mistakes in both directions, so a few thousand human reviewers handle uncertain cases. Users can appeal. Policies change often, some content is legal in one country and illegal in another, and adversaries constantly tweak content to slip past filters.

Design the moderation pipeline. In 45 minutes, cover what is checked before and after publishing, how classifier scores become actions, how review queues are prioritised, how appeals and policy changes work, and how you measure whether moderation is any good.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 47 of 70

Ask before you design. Try answering each yourself, then compare.

1. Do we block before publishing or act after?
   Typical answer

   Both. A fast check under 150 ms blocks only clear, severe cases and known-bad media. Everything gets a full async scan within a couple of minutes.

2. What volume and media mix?
   Typical answer

   500 M posts/day, about 30% with images (1.5 images on average). Video is out of scope for now.

3. How many human reviewers?
   Typical answer

   About 5,000 across time zones and languages, each making roughly 400 decisions a day.

4. What actions exist?
   Typical answer

   Remove, reduce distribution, age-gate or add a warning label, and account strikes that escalate to suspension.

5. Do rules differ by country?
   Typical answer

   Yes. Some content is restricted only in certain countries, which means geo-blocking rather than global removal.

6. Are there known-bad media lists?
   Typical answer

   Yes: our own bank of previously removed media plus industry hash lists for the most severe categories, which must be matched and reported.

7. What is the appeals SLA?
   Typical answer

   Decisions on appeals within 24 hours, by a different reviewer than the original.

Requirements

Functional

• Pre-publish fast check for severe content and known-bad media.
• Async full scan of every post (text and images) with per-policy classifiers.
• Turn scores into actions via versioned, per-region policy rules.
• Prioritised human review queues by policy, language and urgency.
• User reports and appeals with re-review.
• Full audit trail of every decision and the policy version behind it.

Non-functional

• Fast check p99 < 150 ms; full scan within 2 minutes at p95.
• 17 k posts/s and ~8 k images/s at peak.
• Automated removals at ≥ 98% precision per policy, measured weekly on sampled audits.
• High-severity reports reviewed within 1 hour; appeals within 24 hours.
• Posting stays available if moderation is degraded (fail open except for hash matches).

Out of scope

• Video and live streams.
• Training the classifiers in detail.
• Law-enforcement request workflows.

Back-of-the-envelope estimates

• Posts: 500 M/day ≈ 5.8 k/s average, ~17 k/s at peak.
• Images: 225 M/day ≈ 2.6 k/s, ~8 k/s at peak. With four image models at an assumed ~5 ms of GPU time each when batched, that is ~160 GPU-seconds per second, so ~160 GPUs busy at peak; plan ~250 for headroom. A shared multi-head text model at ~2 ms per post adds ~35 GPUs.
• Human review: 5,000 reviewers × 400 = 2 M decisions/day, which is only 0.4% of posts. Reviewer capacity, not compute, sets the review band.
• Storage: scores and decisions at ~1 KB per post ≈ 500 GB/day; two years ≈ 365 TB in columnar storage for audits and retraining.

The key number is the 0.4%: thresholds have to be tuned so the uncertain band fits the people available.

API

POST /v1/moderate/precheck {content_id, author_id, text, media_hashes[], surface, country}
      -> {verdict: allow|hold|block, policy_id?, decision_id}          # sync, < 150 ms
event  content.created {content_id, ...}                              # triggers full scan
GET  /v1/review/tasks?queue=hate.es&n=10        -> leased tasks (10 min lease)
POST /v1/review/tasks/{id}/decision {label, policy_id, action, notes}
POST /v1/appeals {decision_id, statement}       -> {appeal_id, eta}
POST /v1/policies/{id}/versions {thresholds, actions, regions, effective_at}

Every verdict carries a decision ID and the policy version that produced it, so any action can be explained, audited and reversed.

Data model

Signals (columnar store): (content_id, model, model_version, policy, score, ts), for audits, threshold tuning and retraining.

Decisions (wide-column, keyed by content_id, append-only): (decision_id, policy_version, action, regions, source=auto|human|appeal, reviewer_id, ts). Current state is the latest row; history is kept for audits.

Review tasks (relational): (task_id, content_id, queue, priority, lease_owner, lease_expires, status). Needs transactional leasing so two reviewers never get the same task.

Media hash index: perceptual hashes of known-bad media in an index that supports nearest-neighbour search by Hamming distance.

Policy config (relational, versioned): thresholds, actions and regions per policy, read through a cache.

High-level design

Before publishing. The posting service calls the fast-path check, which computes a perceptual hash of each image and looks it up in the media hash index, and runs a small text model on the classifier fleet. Only a hash match or a very confident severe score holds or blocks the post; everything else is published.

After publishing. The posting service emits a content event. Scan workers fetch text and media, batch them into the full set of text and image classifiers on the GPU fleet, and pass calibrated scores to the policy engine. The engine applies the current policy config for the post’s regions: scores above a policy’s automation threshold produce an action immediately; scores in the uncertain band create a task in the review queues with a priority based on severity and expected reach. Every outcome is written to the decision log, and enforcement acts on the content.

People. Human reviewers lease tasks, decide, and their verdicts go to the decision log and to training labels for retraining. The appeals service sends contested decisions back to the queues for a different reviewer.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

From scores to actions: thresholds as a product decision

Raw model outputs are calibrated to probabilities so thresholds survive model upgrades. Each policy and region gets three bands. The top band acts automatically and is set from a precision target, such as 98%, measured on a fresh human-labelled sample. The bottom of the review band is set by reviewer capacity: if the hate queue can absorb 200 k tasks a day, lower the floor until volume matches. Below it, posts are allowed but still count toward account-level signals. New model versions run in shadow first, and thresholds are re-tuned before switching. Severity matters too: for the worst categories, accept lower precision and more review load.

Prioritising human review

Order tasks by expected harm, roughly probability × severity × projected reach, so a borderline post that is going viral beats a likely violation seen by three people. Queues are split by policy and language and routed to trained reviewers. Tasks are leased with a timeout so abandoned work returns to the pool. Quality comes from gold tasks with known answers mixed into the stream and from double review when the stakes are high; disagreements go to senior reviewers. Protect reviewers: blur graphic media by default, cap daily exposure to the worst categories, and rotate assignments.

Known-bad media and adversaries

Exact hashes break with any edit, so use perceptual hashes that change little under resizing, recompression or small crops. Matching means finding hashes within a small Hamming distance, which can be done by splitting the hash into chunks and indexing each so that any near match shares at least one exact chunk. Adversaries also attack text with spacing, lookalike characters and deliberate misspellings; normalise text before classification and train on obfuscated examples. Because single posts are easy to disguise, add account and network signals: new accounts posting near-identical content quickly, or clusters of accounts sharing the same media.

Policy changes, appeals and measurement

Policies are versioned config with an effective date. When a rule tightens, a backfill job re-scores recent, high-reach content at a capped rate rather than the whole history. Appeals go to a different, more senior reviewer; the overturn rate per policy and per model version is a key quality signal, and an overturn restores the content and removes the strike. Measure prevalence (the share of views that land on violating content) by labelling a random sample of viewed posts each week. It captures misses, which precision on actioned posts cannot show.

Failure modes

• If the classifier fleet is overloaded, the full scan backlog grows in the stream; the posting path stays up, and high-reach and reported content is scanned first.
• If the fast check times out, the post is published (fail open) except when the hash index already matched, and the async scan still runs.
• If a bad model version starts over-removing, the overturn and appeal rate spikes; roll back the model and re-run affected decisions from the signal log.
• If a review queue site goes offline, tasks lease-expire and flow to reviewers in other time zones with the same language skills.

Core algorithm in code

import heapq
import math

# Per-policy bands (calibrated probabilities). auto: act without a human. review: send to a person.
POLICY = {
    "hate":     {"auto": 0.97, "review": 0.60, "severity": 3, "action": "remove"},
    "violence": {"auto": 0.98, "review": 0.70, "severity": 4, "action": "remove"},
    "spam":     {"auto": 0.95, "review": 0.85, "severity": 1, "action": "reduce_reach"},
}


def route(scores, projected_views):
    """Return (decision, policy, priority). Highest-severity triggered policy wins."""
    best = ("allow", None, 0.0)
    for name, p in scores.items():
        rule = POLICY[name]
        harm = p * rule["severity"] * math.log10(projected_views + 10)   # expected harm
        if p >= rule["auto"]:
            cand = (rule["action"], name, harm)
        elif p >= rule["review"]:
            cand = ("review", name, harm)
        else:
            continue
        rank = {"allow": 0, "review": 1, "reduce_reach": 2, "remove": 3}
        if (rank[cand[0]], cand[2]) > (rank[best[0]], best[2]):
            best = cand
    return best


def review_floor(scores, auto, capacity):
    """Lowest review threshold whose volume still fits reviewer capacity."""
    band = sorted((s for s in scores if s < auto), reverse=True)
    return band[capacity - 1] if len(band) >= capacity else 0.0


posts = [
    ("p1", {"hate": 0.99, "spam": 0.10}, 200),
    ("p2", {"hate": 0.72}, 1_500_000),
    ("p3", {"spam": 0.90}, 50),
    ("p4", {"violence": 0.75}, 3_000),
    ("p5", {"hate": 0.65}, 40),
    ("p6", {"spam": 0.20, "hate": 0.05}, 900),
]
queue = []
for pid, scores, views in posts:
    decision, policy, harm = route(scores, views)
    print(f"{pid}: {decision:12} {policy or '-':8} harm={harm:5.2f}")
    if decision == "review":
        heapq.heappush(queue, (-harm, pid))
print("review order (most expected harm first):", [heapq.heappop(queue)[1] for _ in range(len(queue))])

hate_scores = [0.99, 0.95, 0.91, 0.88, 0.84, 0.80, 0.74, 0.66, 0.61, 0.55, 0.40]
print("hate review floor for 5 reviews/hour:", review_floor(hate_scores, auto=0.97, capacity=5))

Trade-offs

• Blocking before publishing stops harm early but costs posting latency and false blocks, so only the most confident severe cases are held.
• A wider review band catches more violations but costs reviewer capacity and slower decisions for urgent cases.
• Per-region rules respect local law but cost complexity and the risk of inconsistent enforcement.
• Perceptual hashing catches edited copies but costs some false matches, so matches in low-severity banks go to review instead of auto-removal.
• Account-level signals catch coordinated abuse but cost the risk of penalising legitimate users who share popular content.

What a strong answer shows

• Splits a fast pre-publish check from a thorough async scan with clear reasons.
• Ties thresholds to precision targets and reviewer capacity with numbers.
• Prioritises review by severity and reach, not arrival time.
• Handles known-bad media with robust hashing and plans for adversaries.
• Designs appeals, audit trails and policy versioning.
• Measures quality with prevalence and overturn rates, not just model metrics.

Mistakes to avoid

• Running every classifier synchronously before publishing and blowing the latency budget.
• Picking one global threshold per model without considering precision targets or review capacity.
• Treating the review queue as first-in, first-out.
• Using exact file hashes for known-bad media.
• Measuring success only by the number of removals.

Follow-ups to expect

• How would you extend this to video? (Sample frames and audio transcripts, scan progressively, prioritise by views.)
• How would you use a large language model here? (As a reviewer assistant or for new policies with few labels, behind cheaper classifiers.)
• How do you handle a sudden coordinated attack? (Temporary stricter thresholds, hash-bank the attack media, rate-limit new accounts.)
• How would you produce transparency reports? (Aggregate the decision log by policy, action, source and region.)

Practice next