Design a Metrics Monitoring and Alerting System

An infrastructure team runs about 100,000 hosts and the services on them. Every host reports numbers such as CPU, memory, request rates, error counts and latency histograms, each tagged with labels like service, region and status code.

Engineers want dashboards that load in about a second over the last hour or the last year, and on-call staff want to be paged within a minute when something breaks, but not woken up by a single noisy sample. Monitoring must keep working during the very outages it is meant to detect.

Design the system that collects, stores, queries and alerts on these metrics. In 45 minutes, size the write load, choose a storage layout for time series, explain downsampling and retention, and show how alert rules are evaluated reliably.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · Data & Infrastructure · problem 13 of 18

Ask before you design. Try answering each yourself, then compare.

1. How many series per host and how often do we collect?
   Typical answer

   About 1,000 active series per host, sampled every 10 seconds, so roughly 100M active series and 10M samples per second.

2. Push from hosts or pull by scraping?
   Typical answer

   Either is acceptable; assume a local agent that scrapes local endpoints and pushes batches, which works through firewalls and short-lived jobs.

3. How long do we keep data and at what resolution?
   Typical answer

   Raw 10-second data for 15 days, 1-minute rollups for 13 months, 1-hour rollups for 3 years.

4. How fresh must data be for alerts?
   Typical answer

   A sample should be queryable within 10–20 seconds; alerts should fire within a minute of a condition becoming true.

5. Can a little data be lost?
   Typical answer

   Losing a few seconds of samples on a crash is tolerable. Losing an alert notification is not.

6. Is there a limit on label cardinality?
   Typical answer

   Yes. We must protect the system from a team that puts user IDs or request IDs into labels.

7. Do we need logs and traces too?
   Typical answer

   No, metrics only. Correlation with logs is a follow-up.

Requirements

Functional

• Agents on every host send labelled numeric samples (counters, gauges, histograms).
• Query a metric over a time range with label filters, aggregation and rate functions for dashboards.
• Define alert rules (expression, threshold, duration) and notify on-call channels with grouping and silencing.
• Downsample old data and expire it according to retention tiers.

Non-functional

• 10M samples/s sustained, 100M active series, ingestion lag under 20 s.
• Dashboard queries over 1 hour under 500 ms p95; over 30 days under 3 s using rollups.
• Alert evaluation every 30 s with notification p99 under 60 s after the condition holds.
• 99.9% write availability and an alerting path that survives a single zone loss.

Out of scope

• Log search and distributed tracing.
• Anomaly detection with machine learning (follow-up).
• Billing and per-team cost allocation.

Back-of-the-envelope estimates

• Ingest: 100k hosts × 1,000 series ÷ 10 s = 10M samples/s, about 864B samples per day. Raw that is 16 bytes each (timestamp + float), so 160 MB/s on the wire before compression.
• Compression: timestamp delta-of-delta and XOR-ed float encoding brings a sample to ~1.5 bytes, so raw data is ~1.3 TB/day, about 20 TB for 15 days, ~60 TB with three copies.
• Rollups: 100M series × 1,440 minutes × ~5 bytes (min, max, sum, count compressed) ≈ 0.7 TB/day, ~280 TB for 13 months. Hourly rollups add only ~5 TB per year.
• Memory: keeping the latest 2 hours of every series in memory costs ~1–3 KB per series, so 100–300 GB per copy spread over a few dozen ingesters.
• Queries: 50k alert rules every 30 s ≈ 1,700 queries/s, plus a few hundred dashboard queries/s.

The write rate and series count drive the design: per-sample writes to a general database are impossible, so samples are buffered per series in memory and flushed as compressed blocks.

API

POST /v1/write        [ {labels:{__name__,host,service,...}, samples:[[ts,val],...]} ]
GET  /v1/query_range  ?expr=sum by (service)(rate(http_errors[5m]))&start&end&step
GET  /v1/series       ?match=cpu_usage{region="eu"}   -> label sets
PUT  /v1/rules/{id}   {expr, threshold, for: "5m", severity, labels, runbook}
POST /v1/silences     {matchers, startsAt, endsAt, comment}

Writes are batched per agent (thousands of samples per request, compressed) so the cost per sample is dominated by encoding, not by HTTP overhead.

Data model

Series: a unique, sorted label set hashed to a 64-bit series ID. An inverted index maps each label=value pair to a sorted list of series IDs, so a filter becomes an intersection of posting lists.

Chunks: per series, compressed runs of ~2 hours of (ts, value) pairs. Ingesters hold the open chunk in memory with a write-ahead log, then flush immutable blocks (chunks + index for that time range) to object storage, which is cheap and durable.

Sharding: series are spread over ingesters by hashing the series ID on a consistent-hash ring with 3 replicas; blocks are partitioned by time and tenant. Rules and silences live in a small SQL database; alert state is kept in memory and checkpointed.

High-level design

Write path. An agent on each host scrapes local endpoints every 10 seconds, batches samples and pushes them to the ingest gateway. The gateway authenticates the team, enforces per-tenant series and sample limits, and appends batches to a sample stream partitioned by series hash. The stream absorbs bursts and lets ingesters restart without dropping data. Each ingester consumes its partitions, appends samples to the in-memory chunk of each series and to a local write-ahead log, and every two hours cuts an immutable, compressed block that it uploads to object storage. A downsampler later merges small blocks into larger ones and writes 1-minute and 1-hour rollups, deleting data past retention.

Read path. A dashboard sends a range query to the query frontend, which splits long ranges into day-sized pieces, serves pieces it has already computed from a results cache and fans the rest out: recent hours go to the ingesters, older ranges go to block storage at the coarsest resolution that still gives enough points for the chart. Partial results are merged and aggregated.

Alert path. The rule evaluator runs each rule every 30 seconds against the freshest data, tracks pending versus firing per label set, and sends firing alerts to a notifier that deduplicates, groups related alerts, applies silences and routes pages to the on-call rotation.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Storing samples in a byte or two

Samples within one series arrive at nearly fixed intervals with slowly changing values. Store the first timestamp in full, then the change in the gap between consecutive timestamps, which is usually zero and fits in a single bit. For values, XOR each float with the previous one; similar numbers share sign, exponent and most of the mantissa, so only a short run of meaningful bits needs storing. Together this brings a 16-byte sample down to about 1.5 bytes. Chunks are append-only and decoded sequentially, which suits range scans perfectly and makes per-sample random access unnecessary.

Downsampling and retention tiers

A 1-year chart of 10-second data would mean 3M points per series, far more than any screen shows. The downsampler reads finished raw blocks and writes per-minute and per-hour buckets holding min, max, sum and count, not just the average, so averages stay correct when buckets are combined and spikes remain visible in the max. The query layer picks the tier from the requested step: a 30-day chart with a 1-hour step reads hourly rollups. Raw blocks are deleted after 15 days and minute rollups after 13 months, which keeps long-term storage roughly ten times smaller than raw.

Cardinality control

Cost scales with the number of distinct series, not samples. One label carrying user IDs can create millions of series, blow up ingester memory and the index, and slow every query. Enforce per-tenant limits on active series and on series per metric at the gateway, reject or drop offending labels with a clear error, and publish each team's top metrics by cardinality. Histograms deserve special care: each bucket is its own series, so fixed coarse bucket layouts or native sparse histograms keep them affordable.

Reliable alert evaluation

Rules are sharded across evaluator replicas by rule ID, with two replicas per shard in different zones; the notifier deduplicates the doubled alerts by rule and label set, so a crashed evaluator costs nothing. A rule must hold for its duration (for example 5 minutes) before firing, which filters one-off spikes. Evaluation should wait for ingestion to catch up past the evaluation timestamp, otherwise lag looks like a drop to zero. Absence of data is its own alert, and a dead-man's switch alert that always fires proves the whole pipeline, including paging, is alive.

Failure modes

• If an ingester crashes, its replicas on the hash ring still hold the series; on restart it replays its WAL and the stream offsets it had not yet flushed.
• If object storage is slow or down, ingesters keep data in memory and on local disk longer and retry uploads; only long-range queries degrade.
• If the ingest pipeline lags, evaluators hold back rules instead of firing false 'no data' alerts, and a lag alert fires through a separate, minimal path.
• If the monitoring system itself fails, a small independent watchdog in another region checks a heartbeat alert and pages when it stops arriving.
• If one team floods the system with new series, per-tenant limits at the gateway reject their excess writes while other tenants are unaffected.

Core algorithm in code

from collections import defaultdict


def downsample(samples, step):
    """Roll raw (ts, value) samples into buckets keeping min, max, sum and count."""
    buckets = defaultdict(lambda: [float("inf"), float("-inf"), 0.0, 0])
    for ts, v in samples:
        b = buckets[ts - ts % step]
        b[0], b[1], b[2], b[3] = min(b[0], v), max(b[1], v), b[2] + v, b[3] + 1
    return {t: {"min": b[0], "max": b[1], "avg": round(b[2] / b[3], 1), "n": b[3]}
            for t, b in sorted(buckets.items())}


class AlertRule:
    """Fires when the windowed average stays above a threshold for `for_s` seconds."""

    def __init__(self, name, threshold, window_s, for_s):
        self.name, self.threshold, self.window_s, self.for_s = name, threshold, window_s, for_s
        self.state, self.pending_since = "ok", None

    def evaluate(self, samples, now):
        window = [v for ts, v in samples if now - self.window_s < ts <= now]
        if not window:
            return self.state                      # no data: keep state, alert on absence separately
        breaching = sum(window) / len(window) > self.threshold
        if not breaching:
            self.state, self.pending_since = "ok", None
        elif self.state == "ok":
            self.state, self.pending_since = "pending", now
        elif self.state == "pending" and now - self.pending_since >= self.for_s:
            self.state = "firing"
        return self.state


# CPU of one host, one sample every 10 s: normal, a short spike, then a sustained climb
cpu = []
for i in range(60):
    ts = i * 10
    v = 40 + (i % 3)
    if 12 <= i <= 16:
        v = 97                                     # 50 s spike: goes pending, never pages
    if i >= 35:
        v = 92                                     # sustained problem
    cpu.append((ts, v))

print("1-minute rollups (first 4):")
for t, agg in list(downsample(cpu, 60).items())[:4]:
    print("  t=%3ds %s" % (t, agg))

rule = AlertRule("HighCPU", threshold=85, window_s=30, for_s=120)
last = None
for now in range(0, 600, 30):                      # evaluator runs every 30 s
    state = rule.evaluate(cpu, now)
    if state != last:
        print("t=%3ds %s -> %s" % (now, rule.name, state))
        last = state

Trade-offs

• Pushing through a stream gives buffering and replay but adds a hop and some lag; scraping straight into ingesters is simpler but loses data while an ingester restarts.
• Keeping 2 hours in memory makes recent queries and alerts fast but costs hundreds of GB of RAM and a WAL replay on restart.
• Rollups make long-range charts cheap but lose detail; a spike that lasted 20 seconds survives only in the max, not in percentiles.
• Object storage for blocks is cheap and durable but slower per read, so it needs a block cache and index caching on query nodes.
• Strict cardinality limits protect the system but reject data teams want; soft limits with alerts are friendlier but risk an outage from one bad deploy.

What a strong answer shows

• Turns host count, series per host and interval into a sample rate and sizes memory and disk from it.
• Proposes a time-series layout: per-series compressed chunks, immutable blocks and a label inverted index.
• Explains downsampling with min/max/sum/count and choosing resolution by query range.
• Designs alert evaluation with durations, deduplication, grouping and protection against pipeline lag.
• Addresses cardinality explosions and noisy tenants.
• Considers how monitoring stays alive when the infrastructure it watches is failing.

Mistakes to avoid

• Storing each sample as a row in a relational or generic key-value database.
• Downsampling to averages only, which hides spikes and makes rollups impossible to combine correctly.
• Paging on a single sample above threshold, producing alert storms and fatigue.
• Ignoring label cardinality until it takes down the ingesters.
• Running alerting on the same failure domain as the systems it monitors with no external heartbeat.

Follow-ups to expect

• How would you compute p99 latency across 500 hosts? (Aggregate histogram buckets, not per-host percentiles.)
• How would you support global queries across regions? (Regional stores plus a federating query layer; alert regionally.)
• How would you add anomaly detection? (Seasonal baselines computed offline, compared in the evaluator.)
• How do you backfill historical data? (Write separate out-of-order blocks and let the compactor merge them.)

Practice next