Design a Centralized Logging System

Engineers at a large company debug production by reading logs, but today those logs sit on thousands of machines and vanish when containers are replaced. Build a central place where every service's logs land within seconds and can be searched.

Around 50,000 hosts emit a mix of plain text and structured JSON lines. An on-call engineer at 3 a.m. wants to filter by service, time range and level, search for an error string or an order ID, and watch new lines arrive live. Security wants a year of retention for audits; finance wants the storage bill to stop growing faster than traffic.

In 45 minutes, design collection, ingestion, indexing, retention and search. Be ready to explain what happens when ingest spikes during an incident, which is exactly when people need logs most.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · Data & Infrastructure · problem 37 of 70

Ask before you design. Try answering each yourself, then compare.

1. What volume are we handling?
   Typical answer

   About 2M log lines per second on average, roughly 500 bytes each, so ~1 GB/s and ~86 TB per day raw. Incidents can triple the rate.

2. How quickly must a line be searchable?
   Typical answer

   Within 10 seconds for recent data. Live tail should show lines within 2–3 seconds.

3. What retention do we need?
   Typical answer

   7 days fast and fully searchable, 30 days searchable more slowly, 1 year in cheap archive that can be rehydrated for audits.

4. What kinds of queries are common?
   Typical answer

   Mostly a narrow time range plus service and level filters, then a free-text term or an ID. Aggregations like error counts per minute are useful but secondary.

5. Is losing some log lines acceptable?
   Typical answer

   Under normal operation no; during extreme overload we may drop or sample debug-level lines, but never error-level or audit lines.

6. Are logs multi-tenant?
   Typical answer

   Yes, per team, with quotas and access control. Some fields contain personal data that must be redacted.

Requirements

Functional

• Agents collect logs from files, stdout and syslog on every host and ship them centrally.
• Parse and enrich lines (service, host, level, trace ID) and redact sensitive fields.
• Search by time range, labels and free text; tail live logs for a service.
• Apply per-team retention, quotas and access control; archive and rehydrate old logs.

Non-functional

• Ingest 2M lines/s (~1 GB/s), bursts of 3×, with lines searchable in under 10 s.
• Search over 1 hour of one service under 2 s p95; over 7 days under 15 s.
• No loss of error or audit lines during a 30-minute central outage.
• Storage cost should grow slower than raw volume through compression and tiering.

Out of scope

• Metrics storage and distributed tracing (assume they exist separately).
• SIEM-style security correlation rules.
• Log-based machine learning anomaly detection.

Back-of-the-envelope estimates

• Ingest: 2M lines/s × 500 B ≈ 1 GB/s, ~86 TB/day raw; incident peaks reach ~3 GB/s.
• Compression: log text compresses ~8–10×, so ~10 TB/day stored. A full-text index typically costs another 0.5–1× of the compressed data.
• Hot tier (7 days, full-text indexed, 2 replicas): ~10 TB/day × 1.7 × 7 × 2 ≈ 240 TB on SSD.
• Warm tier (8–30 days, label index + Bloom filters only): ~10 TB/day × 23 days ≈ 230 TB in object storage.
• Archive (1 year): ~3.6 PB compressed in cold object storage, the cheapest per byte.
• Search: a few thousand engineers issue maybe 50 queries/s at peak; reads are rare but each may scan gigabytes.

Write volume and storage cost drive everything: indexing every word for a year would cost more than the systems being logged, so only recent data gets a full index.

API

POST /v1/ingest        (agent) gzip batch of {ts, service, host, level, msg, fields}
GET  /v1/search        ?q=level:error AND "timeout" &service=checkout&from&to&limit
                       -> {lines[], nextCursor, scannedBytes}
GET  /v1/aggregate     ?q=...&groupBy=level&interval=1m
WS   /v1/tail          ?service=checkout&filter=...
POST /v1/rehydrate     {service, from, to}  -> jobId

Search returns a cursor and reports bytes scanned, so the UI can stream results progressively and teams see the cost of broad queries.

Data model

Log event: {ts, service, env, host, level, trace_id, message, fields{}}. Low-cardinality fields (service, env, level, region) are labels; high-cardinality ones (request ID, user ID) stay in the body.

Chunks: lines grouped by label set and time window (for example 5 minutes), compressed in columnar form. Each chunk carries a Bloom filter of its tokens and min/max timestamp. Chunks live in object storage, partitioned by tenant/day/service.

Hot index: a sharded full-text inverted index for the last 7 days, sharded by day and by hash of service, so each index shard is time-bounded and can be dropped whole at expiry. Chunk catalog: a key-value table (tenant, service, hour) -> chunk list used to prune queries.

High-level design

Collection and ingest. A lightweight agent on each host tails files and container output, adds host and service labels, batches and compresses lines, and keeps a disk buffer of up to a few GB so it survives central outages. It sends batches to the ingest gateway, which authenticates the team, checks quotas and appends to a log stream partitioned by tenant and service. The stream is the shock absorber: during an incident, consumers fall behind briefly but nothing is lost.

Processing and indexing. Parser workers consume the stream, parse JSON or known text formats, extract level and trace ID, redact sensitive fields and forward events to indexers. Indexers build two outputs: full-text index segments for the hot tier on SSD, and compressed chunks with Bloom filters that go to object storage and are registered in the chunk catalog. A separate consumer turns logs into counters such as errors per minute for alerting.

Search. The UI sends a query to the query service, which uses the time range and labels to pick shards and chunks from the catalog. Recent ranges hit the hot index; older ranges scan only the chunks whose Bloom filter might match, in parallel across many workers. Results are merged by timestamp and streamed back. Live tail subscribes directly to the stream with a filter.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Full-text index versus label index plus scanning

A complete inverted index over every token makes any search fast but roughly doubles storage and burns CPU on ingest for data that is mostly never read. The alternative indexes only a handful of low-cardinality labels and finds text by brute-force scanning compressed chunks, which is cheap to write but slow for broad queries. A tiered hybrid works well: full-text index for the last week, when most debugging happens, and label index plus per-chunk Bloom filters afterwards. Bloom filters let a query for a rare order ID skip most chunks without decompressing them.

Backpressure and surviving incident spikes

Log volume spikes exactly when things break, because failing services log stack traces in tight loops. Every layer needs a buffer and a policy. Agents keep a bounded disk buffer and back off when the gateway returns throttling responses. The gateway enforces per-tenant rates, so one service flooding errors cannot starve others. The stream holds hours of data, so indexers can lag and catch up. When a tenant exceeds quota, drop or sample debug and info lines first, deduplicate identical repeating lines with a counter, and always keep error and audit lines.

Retention and tiering

Organise storage by time so expiry is a cheap delete of whole shards or prefixes, never a row-by-row purge. Day 0–7 lives on SSD with full indexes; day 8–30 lives in standard object storage with label indexes; beyond that, chunks move to an archive storage class that is cheap to keep but slow to read. Audit requests trigger a rehydrate job that copies the needed time range back into a temporary searchable tier. Per-team retention settings and per-tenant prefixes also make it simple to delete one team's data on request.

Privacy and access control

Logs leak secrets and personal data more often than any database does. Redact known patterns (card numbers, tokens, emails) in the parser before anything is stored, because deleting it later from immutable chunks is expensive. Tag each event with its owning tenant and enforce access in the query service, not in the UI. Keep an audit log of who searched what, since the logging system itself becomes a sensitive data store.

Failure modes

• If the ingest gateway or stream is down, agents buffer to local disk and resend with backoff; when buffers fill they drop debug lines first and emit a counter of what was dropped.
• If indexers fall behind, recent logs become searchable late but are not lost; the UI shows the current ingest lag so engineers know results may be incomplete.
• If a hot index shard node fails, its replica serves queries while a new copy is rebuilt from the stream or from chunks in object storage.
• If one query tries to scan a month of all services, per-query byte limits and timeouts stop it from starving other users; it can run as an asynchronous job instead.
• If a parser deploy breaks parsing, lines are stored raw with a parse-error label rather than dropped, and can be reprocessed from the stream.

Core algorithm in code

import hashlib
import re


class Bloom:
    """Tiny Bloom filter: 'definitely not here' or 'maybe here'."""

    def __init__(self, bits=512, k=3):
        self.bits, self.k, self.v = bits, k, 0

    def _pos(self, word):
        h = hashlib.sha1(word.encode()).digest()
        return [int.from_bytes(h[i * 4:i * 4 + 4], "big") % self.bits for i in range(self.k)]

    def add(self, word):
        for p in self._pos(word):
            self.v |= 1 << p

    def maybe(self, word):
        return all(self.v >> p & 1 for p in self._pos(word))


def tokens(line):
    return set(re.findall(r"[a-z0-9_]+", line.lower()))


class Chunk:
    """A compressed block of lines for one stream (service) and time window."""

    def __init__(self, service, start):
        self.service, self.start, self.lines, self.bloom = service, start, [], Bloom()

    def append(self, ts, line):
        self.lines.append((ts, line))
        for t in tokens(line):
            self.bloom.add(t)


class LogStore:
    def __init__(self, window=60):
        self.window, self.chunks = window, {}          # (service, window_start) -> Chunk

    def ingest(self, service, ts, line):
        key = (service, ts - ts % self.window)
        self.chunks.setdefault(key, Chunk(*key)).append(ts, line)

    def search(self, service, start, end, term):
        term = term.lower()
        hits, opened, skipped = [], 0, 0
        for (svc, w), chunk in sorted(self.chunks.items()):
            if svc != service or w + self.window <= start or w > end:
                continue                                # pruned by label and time: free
            if not chunk.bloom.maybe(term):
                skipped += 1                            # pruned by Bloom filter: no decompression
                continue
            opened += 1
            hits += [(ts, l) for ts, l in chunk.lines if start <= ts <= end and term in tokens(l)]
        return hits, opened, skipped


store = LogStore()
for ts in range(0, 600, 5):
    store.ingest("checkout", ts, f"GET /cart status=200 latency_ms={20 + ts % 7}")
    store.ingest("payments", ts, f"charge ok id=p{ts}")
store.ingest("checkout", 312, "ERROR timeout calling payments order=o981")
store.ingest("checkout", 447, "ERROR timeout calling inventory order=o1204")

hits, opened, skipped = store.search("checkout", 0, 599, "timeout")
print("chunks in store:", len(store.chunks))
print(f"checkout chunks opened: {opened}, skipped by Bloom filter: {skipped}")
for ts, line in hits:
    print(f"  t={ts}s  {line}")

Trade-offs

• A full-text index gives fast arbitrary search but roughly doubles storage and ingest CPU; label index plus scanning is cheap to write but slow for broad text queries.
• A durable stream between agents and indexers adds a few seconds of latency and an extra system, but turns spikes and indexer outages into lag instead of data loss.
• Sampling noisy logs keeps cost and ingest stable but can drop the one line someone needs; restrict it to low-severity levels.
• Small chunks make searches more selective and lines visible sooner but create many small objects, which hurts object storage cost and listing speed; compaction merges them later.
• Schema-on-write parsing makes queries on fields fast but rejects or mangles unexpected formats; schema-on-read is flexible but slower to query.

What a strong answer shows

• Quantifies ingest volume and storage per tier, and lets cost shape the indexing decision.
• Puts durable buffering between hosts and storage and explains backpressure and load shedding by severity.
• Lays out data by tenant, service and time so queries prune most chunks before scanning.
• Compares full-text indexing with label indexing plus Bloom-filtered scans and picks a tiered answer.
• Covers retention and archive with cheap expiry and rehydration.
• Mentions redaction and access control for sensitive data in logs.

Mistakes to avoid

• Writing logs synchronously from applications to the central store, so a logging outage takes down production.
• Full-text indexing a year of data on SSD and never looking at the bill.
• Making high-cardinality fields like request ID into index labels, exploding the number of streams.
• Deleting expired data row by row instead of dropping whole time-partitioned shards.
• Ignoring that log volume spikes during incidents, the moment the system matters most.

Follow-ups to expect

• How would you link logs to traces and metrics? (Propagate a trace ID, index it as a field, and deep-link between tools.)
• How do you support aggregations like p95 latency parsed from logs? (Extract fields at ingest into columnar storage or emit metrics.)
• How would you handle a tenant asking to delete all logs mentioning one user? (Search, then rewrite affected chunks in a background job; prevention through redaction is far cheaper.)
• How would you make live tail scale to many viewers? (Fan out from the stream through a pub/sub layer with server-side filtering.)

Practice next