Design Dropbox

Design a file sync service. People install a desktop client that watches a folder; whatever they put there appears on their other laptops and phones within seconds, and they can share folders with colleagues who see the same files. Files range from tiny text notes to 50 GB video projects.

The tricky parts: a one-line change to a large file should not re-upload the whole thing, the same file often exists in thousands of accounts, devices go offline for days and then edit the same document as someone else, and a sync bug that silently loses a file destroys trust in the product.

In 45 minutes, cover how files are split, stored and deduplicated, how changes propagate to other devices, how you detect and resolve conflicts, and how sharing works.

Sketch on the HLD board, write your answer in the Design doc, and size it with the Estimator. Start the mock interview from the timer for a 45-minute run.

Design Path · Media & Storage · problem 18 of 70

Ask before you design. Try answering each yourself, then compare.

1. What scale should we assume?
   Typical answer

   500 M registered users, 50 M daily active, about two devices each, and on average 10 file changes per active user per day.

2. How fast must changes appear on other devices?
   Typical answer

   Within about 5 seconds when both are online, not counting upload time for large files.

3. What happens when two devices edit the same file?
   Typical answer

   Never lose either version. Keep one as the file and save the other as a clearly named conflicted copy.

4. Do we keep file history?
   Typical answer

   Yes, previous versions and deleted files are recoverable for 30 days.

5. What can be shared?
   Typical answer

   Folders shared with other users (view or edit) and read-only links to files or folders.

6. Is real-time co-editing inside a document required?
   Typical answer

   No. We sync whole files; collaborative editors are a separate product.

Requirements

Functional

• Upload, download, rename, move and delete files and folders from any device.
• Sync changes to all of a user's devices and to collaborators on shared folders.
• Resume interrupted transfers and upload only changed parts of a file.
• Detect concurrent edits and keep both versions.
• Version history and undelete for 30 days; share folders and links.

Non-functional

• Durability of 11 nines for committed files; never acknowledge a commit before its blocks are durable.
• Change notification to online devices within 5 s at p95.
• 99.95% availability for metadata operations; block downloads served at line rate.
• Scale: ~500 M file commits per day, ~100 M connected devices.

Out of scope

• In-document real-time collaboration.
• Full-text search over file contents and previews (mention as follow-ups).
• Billing and quota purchase flows.

Back-of-the-envelope estimates

• Commits: 50 M × 10 = 500 M file changes per day, ~5,800 per second, ~17,000 at peak. Each is a small metadata transaction.
• New bytes: after chunk-level dedupe the average change uploads ~100 KB of new data, so ~50 TB per day, roughly 90 PB over five years. Erasure coding at 1.5× overhead brings that to ~140 PB raw instead of ~270 PB with three full copies.
• Downloads: each change is pulled by about two other devices, so ~100 TB/day out, ~10 Gbps average plus bursts when a large shared folder updates.
• Connections: ~100 M devices online, each holding an idle long-poll or socket for change notifications.
• Metadata: ~100 B file entries at ~500 bytes ≈ 50 TB, which must be sharded.

Storage volume decides the block layer (dedupe, erasure coding, cold tiering); correctness of metadata commits decides user trust.

API

POST /v1/blocks/missing  {hashes[]}                         -> {missing[]}
PUT  /v1/blocks/{sha256}  <chunk bytes>                     -> 201
POST /v1/commit {ns, path, baseRev, blocks[], size, mtime}   -> 200 {rev} | 409 {latestRev}
GET  /v1/changes?ns=..&cursor=c123                          -> {entries[], cursor, hasMore}
GET  /v1/notify?cursors=..   (long-poll, 60 s)               -> {changed: [ns]}
GET  /v1/blocks/{sha256}                                     -> chunk bytes
POST /v1/shares {path, members[], role} | /v1/links {path}   -> {shareId | url}

A commit carries the revision it was based on, so the server can reject a stale edit with 409 instead of silently overwriting someone else's change.

Data model

Namespaces: each user's root and each shared folder is a namespace with its own monotonically increasing journal. journal(ns_id, seq, path, rev, blocks[], size, deleted, author_device) is the source of truth; files(ns_id, path, latest_seq) is a materialised view of current state. Both live in a sharded relational store partitioned by ns_id, so a commit is a single-shard transaction.

mounts(user_id, ns_id, mount_path, role) maps shared namespaces into each member's tree; links(token, ns_id, path, expires) backs public links.

Blocks: block_index(sha256 PK, location, size, refcount) in a key-value store; chunk bytes in an erasure-coded object store, cold blocks moved to a cheaper tier after months without reads.

High-level design

Upload: the desktop client notices a changed file, splits it into content-defined chunks of roughly 4 MB, and hashes each one. It asks the block service which hashes are missing, and uploads only those. The block service writes each chunk to the erasure-coded block store and records it in the block index. Only then does the client call the metadata service to commit: the new list of chunk hashes for the path, plus the revision it started from.

Commit: the metadata service checks that every chunk exists, and in one transaction on the namespace's shard verifies that the base revision is still current, appends a journal entry and updates the file row. It then publishes a change event, which the notification service turns into a wake-up for every device that has this namespace mounted.

Download: a woken device calls /changes with its last cursor, receives the new journal entries, diffs the chunk lists against what it has locally, and fetches only missing chunks, often from a CDN or nearby block cache. It writes them to a temporary file and atomically renames it into place.

A background garbage collector decrements refcounts as versions expire and deletes or tiers unreferenced blocks.

Open the HLD board and press Reference to see the architecture diagram.

Deep dives

Chunking and deduplication

Fixed 4 MB blocks are simple, but inserting one byte near the start shifts every later boundary and changes every hash. Content-defined chunking picks boundaries with a rolling hash over a small window, so boundaries move with the content and an insertion changes only one or two chunks. Each chunk is addressed by its SHA-256, which gives dedupe for free: across versions of a file, across a user's files and, if policy allows, across users. Cross-user dedupe leaks information (an upload that finishes instantly proves someone already has that file), so either limit dedupe to a user or team, or make the client prove possession. Compress chunks before upload and keep a local hash cache so unchanged files are never re-read.

Commit protocol and conflicts

Bytes and metadata are separated: chunks are uploaded first and are harmless if never referenced, then a small commit makes them visible. The commit is a compare-and-set on the file's revision inside one shard transaction. If two devices edited the same base revision, the first commit wins and the second gets 409 with the latest revision. The client then downloads the winner and saves its own edit next to it as report (Ana's laptop conflicted copy).docx. No data is lost and no merge is attempted on opaque binary files. Renames and moves are journal entries too, so a rename on one device and an edit on another apply cleanly by tracking a stable file id, not just the path.

Notifying devices and catching up

Each namespace's journal has a sequence number, and every device stores a cursor per namespace. Online devices keep a long-poll or WebSocket open to the notification service; when a commit lands, the service tells subscribers which namespaces changed, but sends no data. Devices then pull /changes from their cursor. This keeps notifications tiny and idempotent: a missed wake-up is fixed by the next poll, and a device back from a week offline simply pages through the journal. For namespaces with very long histories, the server returns a snapshot instead of millions of entries. A shared folder with 10,000 members fans one event out to many connections, so subscriptions are grouped by namespace.

Sharing and permissions

A shared folder is its own namespace mounted into each member's tree, so a file is stored and journaled once no matter how many people see it, and a single ACL check on the namespace covers all of its files. Unsharing removes the mount (and optionally leaves a copy for the ex-member). Public links store a random token pointing to a namespace and path, checked on every download, with optional expiry and password. Keeping ACLs at namespace granularity avoids per-file permission checks on hot paths.

Failure modes

• If a client crashes mid-upload, already uploaded chunks remain and the retry skips them; unreferenced chunks are garbage-collected after a grace period.
• If a metadata shard is unavailable, commits for its namespaces fail and clients queue changes locally; other namespaces are unaffected.
• If the notification service drops connections, devices reconnect with jitter and poll /changes from their cursor, so no change is missed.
• If a block store node or rack fails, erasure coding reconstructs data from the remaining fragments and a repair job restores redundancy.
• If a buggy client starts deleting en masse, version history and per-namespace rate alarms allow a bulk restore.

Core algorithm in code

import hashlib
import random

rng = random.Random(42)
GEAR = [rng.getrandbits(32) for _ in range(256)]   # random value per byte


def fixed_chunks(data: bytes, size: int) -> list:
    return [data[i:i + size] for i in range(0, len(data), size)]


def cdc_chunks(data: bytes, mask: int = 0x0FFF, min_size: int = 1024, max_size: int = 16384) -> list:
    """Content-defined chunking with a gear rolling hash.

    The boundary depends only on nearby bytes, so inserting data early in a
    file shifts offsets but leaves later boundaries (and chunk hashes) intact.
    """
    chunks, start, h = [], 0, 0
    for i, b in enumerate(data):
        h = ((h << 1) + GEAR[b]) & 0xFFFFFFFF
        length = i - start + 1
        if (length >= min_size and (h & mask) == 0) or length >= max_size:
            chunks.append(data[start:i + 1])
            start, h = i + 1, 0
    if start < len(data):
        chunks.append(data[start:])
    return chunks


class BlockStore:
    def __init__(self):
        self.blocks = {}

    def put_missing(self, chunks: list) -> int:
        uploaded = 0
        for c in chunks:
            key = hashlib.sha256(c).hexdigest()
            if key not in self.blocks:        # client asks "which hashes do you lack?"
                self.blocks[key] = c
                uploaded += len(c)
        return uploaded


original = bytes(rng.getrandbits(8) for _ in range(300_000))
edited = original[:5000] + b"a small insertion near the start" + original[5000:]

for name, chunker in (("fixed 4 KB", lambda d: fixed_chunks(d, 4096)), ("content-defined", cdc_chunks)):
    store = BlockStore()
    first = store.put_missing(chunker(original))
    second = store.put_missing(chunker(edited))
    print("%-16s v1 uploaded %7d bytes, edit uploaded %7d bytes (%d chunks stored)"
          % (name, first, second, len(store.blocks)))

Trade-offs

• Content-defined chunking survives insertions and dedupes far better but costs CPU on every client and produces variable-size chunks that are harder to pack.
• Cross-user deduplication saves a lot of storage but leaks whether a file exists elsewhere and couples users' data lifetimes through refcounts.
• Conflicted copies never lose data but push merging onto the user; automatic merges are friendlier but only safe for formats you understand.
• Erasure coding cuts storage overhead from 3× to about 1.5× but makes repairs and small reads more expensive than plain replication.
• Notify-then-pull keeps notifications cheap and reliable but adds a round trip before each device sees a change.

What a strong answer shows

• Splits files into hashed chunks and uploads only what the server lacks.
• Separates block storage from metadata and commits metadata atomically after blocks are durable.
• Uses revisions or version vectors with compare-and-set to detect conflicts, and keeps both versions.
• Designs change propagation with a per-namespace journal and cursors, plus a lightweight notification channel.
• Models sharing so a shared folder is stored once and permissions are checked cheaply.
• Discusses durability, garbage collection and recovery from client bugs.

Mistakes to avoid

• Re-uploading whole files on every change.
• Writing metadata before blocks are durable, so other devices see files they cannot download.
• Last-writer-wins on file contents, silently discarding one device's edit.
• Pushing full file contents through the notification channel instead of a 'something changed' signal.
• Forgetting that deleting a chunk is unsafe while any version or user still references it.

Follow-ups to expect

• How would you support LAN sync between devices in the same office? (Peers advertise chunk hashes locally and fetch from each other first.)
• How would you add full-text search? (Async extraction worker indexing new versions, ACL-filtered queries per namespace.)
• How would you recover a user hit by ransomware that encrypted all files? (Detect mass rewrites, roll the namespace back to a journal point.)
• How would you migrate blocks to a new storage backend without downtime? (Dual-write new blocks, background copy, flip index locations.)

Practice next