AI System Design
← Learn
Level 3advanced

Key-Value Store

Two replicas just died. Keep reads and writes flowing.

Depth:
1

Mission

You're building a Dynamo-style key-value store: data partitioned on a hash ring, N replicas per key. Machines fail all the time. Tune quorums, conflict resolution and repair so the store stays available without silently losing data.
2

Interactive Simulation

Before we explain anything — play. Push it until it breaks, then fix it.

R + W = 2 ≤ N = 3: fast and available, eventually consistent.
Consistency
Eventual
R+W=2 vs N=3
Stale reads
4.0%
Lost updates
No
last-write-wins
Replica drift
none
Requests
20K/s
Latency
2ms
p95 7ms
Error rate
0.0%
CPU
50%
Accepted
20K/s
Latency (ms)
Error rate (%)

Quorum

Replicas per key (N)

R + W > N means read and write sets always overlap. Lower numbers are faster and more available.

Repair & conflicts

Conflict resolution

Break it

System Score99
3

What just happened?

Read and write quorums (R and W out of N) decided whether requests survived failed replicas and whether reads were guaranteed fresh. Last-write-wins quietly discarded one of two concurrent updates, while vector clocks kept both as siblings to merge. Replicas that came back after an outage stayed stale until hinted handoff or Merkle-tree anti-entropy repaired them.

4

The concept

A distributed key-value store combines several building blocks: consistent hashing to partition keys across nodes, replication to N nodes clockwise on the ring, quorum consensus (R + W > N for strong consistency), versioning with vector clocks to detect conflicting writes, gossip to detect failures, sloppy quorums with hinted handoff for temporary failures, and Merkle-tree anti-entropy for permanent drift. Writes go to a commit log and memtable, flushed to immutable SSTables; reads check memory, then Bloom filters, then SSTables.

5

Trade-offs

Nothing is free. Here's what this solution costs you.

Availability vs consistency
Lower quorums and sloppy quorums keep serving during failures at the cost of fresh reads.
Conflict handling cost
Vector clocks avoid lost updates but push merge logic onto clients.
Repair overhead
Anti-entropy and hinted handoff consume bandwidth and disk in the background.
6

In the real world

Conceptually similar to Amazon Dynamo, Apache Cassandra and Riak.

7

Mini quiz

Question 1 of 30 correct

With N = 3, which (R, W) pair guarantees reads see the latest write?

8

Interview me

The app becomes your interviewer. One question, in your own words.

9

Boss challenge

Survive a 2-node outage

5 replicas per key, 2 of them down, and two clients writing the same key.

Goal: Keep reads and writes available, reads strongly consistent, no lost updates, and repair recovered replicas.

Use the simulator above with no hints. These checks update live as you play.

10

Interview question

“Design a distributed key-value store that is highly available and scalable. Cover partitioning, replication, consistency (quorums), conflict resolution, failure detection and handling of temporary and permanent failures.”

On to Challenges