Design a Feature Store

A company runs dozens of ML models: ranking, fraud scoring, recommendations, churn prediction. Each team computes its own features in its own pipelines, so the same "orders in the last 7 days" exists in five slightly different versions, and the value a model sees in production often differs from the one it was trained on.

Design a shared feature platform. Teams declare a feature once; the platform computes it in batch or from streams, keeps its full history for building training sets, and serves the latest values to online models in a few milliseconds.

The hard parts are building training data without leaking information from the future, backfilling years of history when someone adds a feature, keeping streaming features fresh, and serving hundreds of thousands of lookups per second. In 45 minutes, cover requirements, sizing, the offline and online paths, and go deep on point-in-time correctness and serving latency.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · AI & ML Systems · problem 48 of 70

Ask before you design. Try answering each yourself, then compare.

1. Who uses the platform and how big is the catalogue?
   Typical answer

   About 50 production models from 15 teams, roughly 200 feature views holding 5,000 features in total.

2. What latency do online models need?
   Typical answer

   p99 under 10 ms to fetch ~200 features for one entity, under 25 ms for a batch of 100 entities.

3. How fresh must features be?
   Typical answer

   Batch features can be a day old. Streaming features (counts, sums over recent windows) must reflect an event within about 30 seconds.

4. How many entities and how much traffic?
   Typical answer

   Around 500 M users and 50 M items. Peak online traffic is about 200 k entity lookups per second.

5. Does the platform run the transformations or only store results?
   Typical answer

   Both: it runs declared SQL-style and windowed aggregations itself, and also accepts values pushed from teams’ own pipelines.

6. How much history do training jobs need?
   Typical answer

   Two years, queried as of arbitrary past timestamps.

7. Multi-region?
   Typical answer

   Online serving in two regions; the offline store and compute live in one.

Requirements

Functional

• Register versioned feature definitions (entity, source, transformation, TTL, owner).
• Compute features in batch and from event streams, and accept pushed values.
• Generate point-in-time correct training datasets for a list of (entity, timestamp) rows.
• Serve the latest feature values online by entity key, grouped by a named feature set per model.
• Backfill history for new or changed features.
• Monitor freshness, null rates and training/serving skew.

Non-functional

• Online read p99 < 10 ms for one entity, 99.95% availability.
• Streaming feature freshness < 30 s at p99; batch freshness < 24 h.
• 200 k entity lookups/s at peak; 550 M entities online.
• Two years of offline history; training-set builds over a billion rows finish in hours, not days.

Out of scope

• Model training and model serving themselves.
• Data labelling.
• A general-purpose BI warehouse.

Back-of-the-envelope estimates

• Online size: 550 M entities × ~2 KB of packed feature values ≈ 1.1 TB; with 3 replicas ≈ 3.3 TB. Too big for one box, fine for a sharded SSD-backed key-value store with a memory tier for hot keys.
• Online reads: 200 k lookups/s at peak, each touching ~4 feature views → ~800 k key reads/s and ~400 MB/s of payload.
• Streaming writes: ~2 B events/day ≈ 23 k/s average, ~70 k/s at peak, each updating a handful of window aggregates.
• Batch writes: materialising 500 M rows nightly within one hour is ~140 k writes/s, a burst larger than the read traffic.
• Offline history: ~1 TB/day of feature rows before compression; columnar at ~5× gives ~150 TB for two years, in object storage.

Two numbers drive the design: the nightly bulk-load burst that must not hurt 10 ms reads, and point-in-time joins over hundreds of terabytes of history.

API

apply(FeatureView(name, entity, source, transform, ttl, owner))   # declarative, versioned in a repo
apply(FeatureService(name="fraud_v3", features=[...]))

POST /v1/online/get   {service:"fraud_v3", entities:[{user_id:42}, ...]}
      -> {rows:[{values:{...}, event_ts:{...}, status:{...}}]}
POST /v1/push         {view, entity_key, values, event_ts}
get_historical_features(entity_rows=[(user_id, label_ts), ...], service="fraud_v3") -> table
POST /v1/backfill     {view, version, start, end}

Models ask for a named, versioned feature service rather than a list of columns, so the training job and the online model are pinned to exactly the same definitions and the platform knows which consumers a change affects.

Data model

Registry (relational): feature_views(name, version, entity, schema, source, transform_ref, ttl, owner) and feature_services(name, version, views[]). Small, strongly consistent, read through a cache by every component.

Offline store: columnar tables in object storage, one per view and version, rows (entity_key, event_ts, created_ts, feature columns), partitioned by event date and clustered by entity key so as-of joins can merge sorted runs. Append-only, so history is never rewritten in place.

Online store: key-value, key (view_id, entity_key), value = packed latest feature values plus their event_ts. Hash-partitioned by entity key. One key per view keeps a lookup to a few reads, and a key-value store gives predictable single-digit-millisecond gets at this size.

High-level design

Definition. An engineer commits feature views and feature services to a repo; CI applies them to the registry, which generates batch and streaming jobs from the same definition.

Batch write path. Scheduled batch compute jobs read the lake, run the transformation, and append rows with event_ts and created_ts to the offline store. The materializer then takes the newest row per entity and bulk-loads it into the online store.

Streaming write path. The stream aggregator consumes app events, keeps windowed aggregates in state, writes updated values straight to the online store within seconds, and also appends them to the offline store so history matches what was served.

Online read path. A product request reaches the model service, which calls the feature server with a feature service name and entity keys. The server resolves the views from its cached registry, issues parallel multi-gets to the online store, fills defaults for missing or expired values, and returns values plus their ages.

Training path. A training job sends its labelled rows to the point-in-time join engine, which scans the offline store and returns, for each row, the values that were known at that timestamp.

The freshness monitor watches value ages, null rates and served-versus-offline distributions.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Point-in-time correct training data

For each training row (entity, label_ts) the join must pick the newest feature row with event_ts ≤ label_ts, ignore rows older than the view’s TTL, and, for strict correctness, also require created_ts ≤ label_ts so that late corrections do not leak into the past. Joining on "latest value" silently trains the model on the future and inflates offline metrics. At scale, avoid a range join: bucket both sides by entity hash, sort each bucket by time, and do a single merge pass per bucket. Because the offline tables are already clustered by entity and partitioned by date, the engine only reads the partitions between min(label_ts) - ttl and max(label_ts).

Backfills and one definition for two engines

Adding a feature means computing two years of history. Run the same transformation in batch over historical sources and write it into a new versioned table; never overwrite an existing version, because trained models depend on it. Windowed aggregations are the tricky part: if streaming code and batch code are written separately they will drift. Compile both from one definition that produces partial aggregates per small time tile (say one minute and one hour). A 7-day count is then the sum of tiles, computed identically whether the tiles came from a nightly batch or a live stream. Backfills run at low priority with a throughput cap and only flip the online pointer after validation.

Serving in single-digit milliseconds

Store one packed row per (view, entity) so a 200-feature request is about four key reads, issued in parallel. Keep slow-changing item features in a small in-process cache on the feature server. Hedge a read to a second replica if the first has not answered within the p95 time. The biggest threat is the nightly bulk load: writing 140 k rows/s triggers compactions that spike read latency. Either throttle the materializer against a read-latency signal, or build the new snapshot as a fresh table on the side and atomically switch a pointer, then drop the old table. Streaming updates are small and continuous, so they can write in place.

Freshness and skew monitoring

Every served value carries its event_ts, so the server can report age per view and the monitor can alert when a stream job falls behind its 30-second target. Log a sample of exactly what was served, keyed by request ID, back into the offline store. That log lets teams train on the precise values the model saw, and lets the monitor compare served distributions with recomputed offline values: a gap means the online and offline paths disagree. Track null and default rates too; a silently failing upstream join usually shows up as a jump in defaults long before model metrics move.

Failure modes

• If the stream aggregator stalls, online values age past their TTL; the feature server returns defaults with a status flag, models trained with missing-value indicators degrade gracefully, and the monitor pages on age.
• If a batch materialization writes bad data, the pointer swap is reverted to the previous snapshot within minutes because old versions are kept for a few days.
• If an online store shard is lost, replicas serve reads; if a whole region fails, traffic shifts to the second region, whose store is replicated asynchronously and a few seconds behind.
• If the registry is down, feature servers keep using their cached definitions; only new registrations are blocked.

Core algorithm in code

from bisect import bisect_right

# Feature history for one feature view: entity -> rows sorted by event_ts.
# Each row is (event_ts, created_ts, value). created_ts is when the platform
# actually learned the value; a late correction has created_ts > event_ts.
HISTORY = {
    "u1": [(100, 101, 3), (200, 260, 5), (300, 301, 9)],
    "u2": [(150, 151, 1)],
}


def as_of(rows, label_ts, ttl):
    """Point-in-time lookup: newest value that existed at label_ts and is not older than ttl."""
    times = [r[0] for r in rows]
    i = bisect_right(times, label_ts) - 1
    while i >= 0 and label_ts - rows[i][0] <= ttl:
        event_ts, created_ts, value = rows[i]
        if created_ts <= label_ts:      # was it known back then?
            return value, label_ts - event_ts
        i -= 1
    return None, None                   # missing -> model sees a default


def naive_latest(rows):
    """What a careless join does: take today's value. Leaks the future."""
    return rows[-1][2] if rows else None


def build_training_set(labels, ttl):
    out = []
    for entity, label_ts, y in labels:
        rows = HISTORY.get(entity, [])
        value, age = as_of(rows, label_ts, ttl)
        out.append((entity, label_ts, y, value, age, naive_latest(rows)))
    return out


labels = [("u1", 250, 1), ("u1", 350, 0), ("u2", 500, 1), ("u3", 300, 0)]
print("entity label_ts y | point-in-time (age) | naive latest")
for e, ts, y, v, age, naive in build_training_set(labels, ttl=300):
    print(f"{e:>6} {ts:>8} {y} | {str(v):>5} ({age}) | {naive}")
print("u1@250 gets 3, not 5: the value 5 for t=200 only arrived at t=260.")
print("u2@500 is None: the last value is older than the 300s TTL.")

Trade-offs

• Writing streaming features directly to the online store gives sub-minute freshness but costs a second write path that must be kept consistent with the offline history.
• Packing all features of a view into one value makes reads cheap but costs read-modify-write work when a single feature changes.
• Strict created_ts filtering prevents leakage from late corrections but costs some training rows that end up with missing values.
• Snapshot-and-swap bulk loads protect read latency but cost double storage during the swap window.
• A central platform removes duplicated pipelines but costs teams some flexibility, so a push API is kept as an escape hatch.

What a strong answer shows

• Separates the offline (history, throughput) and online (latest value, latency) stores and explains why each store type fits.
• Explains point-in-time correctness concretely, including TTLs and late-arriving data.
• Keeps one definition for batch and streaming to avoid training/serving skew.
• Protects online latency from bulk loads and sizes the online store with numbers.
• Treats backfills and versioning as first-class, not an afterthought.
• Adds monitoring for freshness, nulls and skew.

Mistakes to avoid

• Joining training labels to the current feature value, which leaks future information.
• Writing separate batch and streaming code for the same feature and hoping they match.
• Using the analytical warehouse for online lookups and missing the latency target by 100×.
• Overwriting feature history in place during a backfill, breaking reproducibility of earlier models.
• Ignoring the nightly bulk-load burst when sizing the online store.

Follow-ups to expect

• How would you support on-demand features computed from request data? (Run a registered transform inside the feature server, using the same code in the training-set builder.)
• How do you handle a feature that is very expensive to compute per entity? (Precompute in batch, accept staleness, expose its age.)
• How would you delete a user’s data for a privacy request? (Tombstone by entity key online; rewrite offline partitions in a periodic compaction job.)
• How do you make embeddings first-class features? (Store vectors as a view; serve via key lookup, and sync to a vector index for similarity queries.)

Practice next