Design S3-like object storage

Do the numbers before the boxes

An interviewer gives request rates and a data size. Before any box is drawn, work out what reaches the store each second.

A 45-minute version of this answer is at the end.

We assume one AWS Region, us-east-1. Every count on this page is our estimate unless sourced.

The callers are our own services, so no answer crosses the internet. Public downloads go through a CDN and are out of scope.

The entry is client requests a second: one GET or one PUT of one object.

PUTs are machine traffic, flat over the day: 1,250 a second, all day.

Each storage node keeps recently read objects in memory, its page cache, and serves some GETs from it.

At the peak, 27,500 GETs a second miss the page cache and reach the drives. GETs the cache serves add to that.

So the total GET rate is 27,500 plus the page-cache hits. A beat after Stage 1 says how many hits the design can take.

New objects are read most. Of the 27,500, 26,500 are for objects from the last 7 days, and 1,000 for older ones.

The entry is 27,500 + 1,250 = 28,750 client requests a second.

A request carries 256 KiB on average. Most objects are small, and a few large ones lift the mean (our size mix).

Check in: Close the page and derive the entry. What does it leave out?

Three copies on one tier

Stage 1 is the first answer: one tier of storage nodes, and three copies of every object, one in each zone.

The store runs in 3 availability zones, separate data centres in one Region. Every count on this page splits evenly over the 3.

The bytes live on storage nodes: 48 EC2 d3en.12xlarge instances, 16 a zone. EC2 is AWS's virtual servers.

Each has 24 hard drives (HDD) sold as 14 TB, 13,980 GB each. They are instance store: their data is lost when the instance stops.

Drives are sold in decimal GB, and this page counts binary GiB: 24 x 13,980 x 10^9 / 2^30 = 312,477.5 GiB a node.

PB is decimal too: 4 PB is 4 x 10^15 bytes, 3,725,290 GiB. At 256 KiB an object, that is about 15.3 billion objects.

One copy a zone means each zone holds the whole 3,725,290 GiB. Over 16 nodes: 3,725,290 / (16 x 312,477.5) = 74.5% of their space.

A lost node loses all 24 drives at once. Its data is rebuilt on its own zone's 15 others, to keep one copy a zone.

They then hold 3,725,290 / (15 x 312,477.5) = 79.5%. Every tier on this page is sized to stay under 80% after one rebuild.

A hard drive does about 120 random reads a second, so a seek costs 1/120 s, 8.333 ms.

The node streams 6,200 MiB/s over 24 drives, 258.33 MiB/s each. A 256 KiB read adds 0.25 / 258.33 s = 0.968 ms: 9.301 ms in all.

Small objects are packed into append-only volumes, so a PUT streams with no seek: 0.968 ms a copy.

Drive time: 27,500 x 9.301 ms + 1,250 x 3 x 0.968 ms = 259.4 drive-seconds a second. Over 48 x 24 = 1,152 drives, that is 22.5%.

The data card shows the larger of drive time and bytes: 74.5%. Nothing saturates, and the last tile names Data nodes as busiest.

So the bytes set the node count, and the drives sit mostly idle. 14 TB drives are the cheapest bytes in the catalog.

A node costs $6.30864 an hour. Per raw GiB: $6.30864 x 720 / 312,477.5 = $0.0145 a month.

The cost tile shows $236k/mo. The storage nodes are 48 x 720 hours x $6.30864 = $218,026.60; the API and the metadata table add the rest.

On the canvas there are no load buttons; the next beat says why. The bytes stay at 74.5% at any load.

The trade-off:

Three copies on 14 TB drives buy the cheapest bytes and survive a zone. In exchange, the store holds every byte three times.

Stage 1: Three copies on one tier

• Load: 28,750 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $236k/mo

Check in: What sets the node count in Stage 1, the drives or the bytes?

The API and the metadata table

Every request passes the API, and every one reads or writes a metadata row. Size both before the twists.

The Object API runs 48 tasks on Fargate, AWS's serverless containers, 16 a zone. Each task has 2 vCPUs, virtual processors.

A request costs about 2 ms of CPU (our estimate). So the fleet serves 48 x 2 x 1,000 / 2 = 48,000 requests a second.

The entry takes 28,750 / 48,000 = 59.9% of it.

The metadata is one row an object in DynamoDB, AWS's managed key-value database: about 1 KB a row.

A GET reads its row with a strongly consistent read, which always sees the last write. A row under 4 KB costs 1 read unit.

A PUT commits its row with 1 write unit, for a row under 1 KB. So the table takes 27,500 read units and 1,250 write units a second.

On demand, a table takes 40,000 read units a second by default, a quota AWS can raise. 27,500 / 40,000 = 68.75%.

Before cutover, pre-warm the table (warm throughput): a new on-demand table starts at 12,000 reads a second, and this one needs 27,500.

The drives see the reads the page cache misses. Reads the cache serves come on top: up to about 4,500 a second keep the API and the table clean.

With 4,500 more, the table reads 32,000 / 40,000 = 80%, its warning line. The API reads 33,250 / 48,000 = 69.3%.

That is 4,500 / (26,500 + 4,500) = 14.5% of the reads of new objects.

Past that, more tasks and a quota raise fix both. A higher hit share costs money, not correctness.

A GET reads the row, then the bytes.

The latency tile shows 11ms: the API's 2 ms, plus about 8.9 ms on the drives for the mix of GETs and PUTs.

It leaves out the row read. A GET takes 2 ms on the API, about 2 ms for the row and 9.3 ms on a drive: about 13 ms.

That is server time. A caller in the Region also waits its own round trip.

On the canvas, the API card checks CPU only. Its network, 27,500 x 256 KiB = 6,875 MiB/s, about 1.2 Gbps a task, is worked by hand.

The trade-off:

A fixed fleet and an on-demand table carry the peak with room. In exchange, page-cache hits spend that room, about 4,500 a second of it.

Check in: The page cache serves 6,000 GETs a second. What changes, and what does not?

Reading the result tiles

Stage 1 showed a canvas and four result tiles. Five rules hold on every stage.

Load: the entry counts client requests a second, GETs and PUTs, and the tile labels them req/sec. A storage card applies the copies or shards itself.

A card turns red at 90% of its limit. A part that saturates, past 100%, is named in the result; otherwise the last tile notes the busiest.

A storage card warns when its drives are over 70% busy, or its bytes or network over 80%. A full drive does not slow a read.

Latency is server time on the slowest path. Past saturation it is a floor, which is why it prints "≥".

Cost: each card bills the stage load all month. The node fleets are fixed, so only the table's request charges would be lower in a quiet hour.

No stage has load buttons. The table sits at 68.75% of its default quota, and warns past about 1.16 times this load.

Each stage ends with a paragraph that starts "On the canvas". It says where a card counts differently from AWS, and why that stage has no buttons.

Check in: Stage 1's data card reads 74.5%. Which line is it nearest, and what moves it closer?

One GET and one PUT, from client to drive

Walk one GET and one PUT through their states. A senior interviewer probes each one.

A row holds bucket, key, version, size, checksum and a logical volume id. The partition key is the bucket, and the sort key the object key.

A placement map turns a volume id into nodes and a tier. A small consensus group, servers that agree on each change, holds it. Each API task caches it.

So moving or compacting a volume changes the map and the node's index, never the rows.

This is not consistent hashing of objects. A volume moves as a unit, and only the map records where it went.

Routed. The API reads the row, then finds the volume's nodes in its cached map: no call to the placement service.

Read. The copy in the client's zone finds the object in its in-memory index of the volume: one seek.

Written. The bytes are appended to an open volume on 3 nodes, one a zone. Group commit flushes many PUTs to the drive together.

Each drive keeps one open volume, so new objects land on every drive.

Committed. Then a conditional write of the row, which applies only if the row is as expected: 1 write unit.

The row is the commit point: bytes first, then the row.

Crashed before the commit. The bytes are orphans that no row points at, so the object does not exist.

Compaction, which copies a volume's live objects to a fresh volume, drops them later.

A crash can never leave a row that points at nothing. A GET or LIST after the commit sees the object.

Two PUTs to one key at once: the later commit wins, as on S3.

Objects over 8 MiB are split into 8 MiB chunks on different drives (our estimate), so no single drive streams a huge object.

Step through the figure to follow both.

The trade-off:

Two writes, bytes then row, make every PUT all or nothing. In exchange, a crash leaves orphan bytes that compaction must find.

One GET and one PUT, from client to drive

1. GET, the row: the API reads the metadata row with a strongly consistent read. The row names the volume.
2. GET, the bytes: the cached placement map names the nodes. The copy in the client's zone finds the object in its index: one seek.
3. PUT, the bytes: appended to an open volume on 3 nodes, one a zone. Group commit flushes many PUTs together.
4. PUT, the commit: a conditional write of the row. The row is the commit point.
5. A crash before the commit: the bytes are orphans. No row points at them, so the object does not exist.

Check in: A PUT crashes after its bytes reach 3 nodes, before the row commits. What does a GET of that key see?

Why DynamoDB, and when sharded SQL wins

Magic Pocket, Dropbox's own store, keeps its index on sharded MySQL. An interviewer asks why this design picks DynamoDB.

The composite key gives an ordered LIST in a bucket: one Query on the bucket's partition, sorted by object key, with no listing table.

Partitions split by load with no resharding project. A strong read is 1 read unit, and a conditional write is the commit.

GET and LIST read the base table with strongly consistent reads. Never a global secondary index (GSI): its reads are eventually consistent.

And never a cache of rows, which could serve a row older than the last PUT.

Sharded SQL wins when a change spans many rows at once: DynamoDB transactions stop at 100 items.

It also wins at a high, flat rate, where owned or reserved database nodes cost less than request units.

It loses on LIST. Hash sharding needs a separate listing table; range sharding needs a manager that splits ranges.

Time-ordered keys in one bucket send every new write to one partition, which takes 1,000 writes a second.

Prefix the partition key with a hash of the key, and merge the sorted Queries of each prefix for LIST.

Say it like this: "DynamoDB for an ordered LIST and splits by load. Sharded SQL when changes span many rows, or the rate is high and flat."

The trade-off:

DynamoDB gives LIST and splits with no project. In exchange, a change across more than 100 items needs another design.

Check in: Why not read rows from a GSI, to spare the base table?

Reed-Solomon 8 + 4 on every object

The interviewer asks: Three copies is 200% overhead. Cut it.

Stage 2 codes every object with Reed-Solomon 8 + 4: 8 data shards and 4 parity shards. Any 8 of the 12 rebuild the object.

This is erasure coding. The shards go to 12 drives, at most 4 a zone, so the object survives a lost zone.

A 256 KiB object becomes 12 shards of 32 KiB.

Per-object coding needs a fleet sized for seeks; ours is sized for bytes. S3 codes each object and copes because millions of drives spread the seeks.

The bytes fall to 12 / 8 = 1.5 times the data: 1.5 x 3,725,290 = 5,587,935 GiB, 1,862,645 a zone.

9 nodes a zone hold 1,862,645 / (9 x 312,477.5) = 66.2%, and 74.5% after a rebuild. That is 27 nodes, not 48.

But a GET now reads 8 shards on 8 drives: 8 seeks, not 1. Each shard costs 8.333 + 0.121 = 8.454 ms.

Drives have grown since 2006, but a drive still does about 120 random reads a second (Warfield, AWS).

A GET takes 8 x 8.454 = 67.6 ms of drive time. 27,500 GETs need 1,859.9 drive-seconds a second, and the PUTs add 1.8.

Over 27 x 24 = 648 drives, that is 1,861.8 / 648 = 287.3%.

The result names Data nodes, past 100% on drive time. Its warning gives 287%.

GETs queue on the drives: they need almost 3 seconds of work every second. The tile's time is a floor.

The latency tile shows "≥ 34ms". The ≥ marks that floor.

The claim holds even if a drive did twice the cited 120 seeks: about 146%.

The cost tile shows $141k/mo. The storage nodes are 27 x 720 hours x $6.30864 = $122,639.96, against Stage 1's $218,026.60.

Coding halves the bytes. On small objects it multiplies the seeks by 8.

On the canvas, the card reads each GET as 8 shard reads, as this design does. There are no load buttons; the drives run out at about 9,580 GETs a second.

Step through the figure to compare the layouts.

The trade-off:

Coding every object halves the nodes. In exchange, each small GET costs 8 seeks, and a fleet sized for bytes cannot keep up.

Copies against 8 + 4 on one object

1. Three copies: a 256 KiB object sits whole on one drive in each zone. A GET reads one copy: one seek.
2. Reed-Solomon 8 + 4 on the object: 12 shards of 32 KiB on 12 drives, at most 4 a zone.
3. A GET reads the 8 data shards: 8 seeks on 8 drives. 27,500 GETs need 287.3% of 648 drives.
4. Code a sealed volume instead: each small object sits whole in one shard, so a healthy GET is one seek.

Stage 2: Reed-Solomon 8 + 4 on every object

• Load: 28,750 req/sec
• First to saturate: Data nodes
• Estimated cost: $141k/mo

• Latency past saturation is a lower bound, not an estimate. Data nodes is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Data nodes: Drives busy 287% of the time on d3en.12xlarge x27. A read costs 1 seek (8.3 ms at 120 a drive) plus 32 KiB at 258.3 MiB/s a drive, on 24 drives.

Check in: Why does halving the bytes make Stage 2 fail?

A hot tier and a cold tier

New objects are read most. Stage 3 keeps them on three copies, and codes their volumes once they go cold.

f4, Facebook's warm blob store, found week-old blobs get an order of magnitude fewer reads than blobs under a day old.

The hot tier holds 7 days of new objects. PUTs land 1,250 x 256 KiB = 312.5 MiB/s, 26,367 GiB a day.

Seven days is 7 x 26,367 = 184,570 GiB. Every PUT lands on the hot tier as 3 copies, one a zone.

The hot drives see 26,500 GETs a second, after the page cache. The other 1,000 GETs are for older objects, on the cold tier.

21 nodes, 7 a zone, hold 184,570 x 3 / 21 = 26,367 GiB each: 8.4% of their space.

Drives: 26,500 x 9.301 ms + 3,750 x 0.968 ms = 250.1 drive-seconds a second. Over 21 x 24 = 504 drives, 49.6%.

7 a zone keeps the drives near half busy. New objects' reads come in bursts, and a lost node's 6 zone-mates take its share: 57.9%.

A full volume seals. After 7 days a background job codes it 8 + 4 onto the cold tier, and updates the placement map.

The rows keep their volume id. The cold tier holds 3,725,290 - 184,570 = 3,540,720 GiB, coded to 1.5 times: 5,311,080 GiB, 1,770,360 a zone.

9 nodes a zone hold 1,770,360 / (9 x 312,477.5) = 63.0%, and 70.8% after a rebuild.

Nothing saturates, and the last tile names Metadata (bucket, key) as busiest.

At 4 PB this split costs the same 48 nodes as Stage 1, $218,026.60. The cost tile shows $236k/mo again; the split pays at scale.

The latency tile shows 11ms, as in Stage 1. A GET reads the row, then the bytes.

On the canvas, the API's two edges split the entry: 0.965 to the hot tier, every PUT and 26,500 GETs, and 0.035 to the cold.

There are no load buttons: the 26,500 is a premise about the page cache, and the table quota holds as in Stage 1.

The trade-off:

Two tiers keep new objects at one seek a GET, and old ones at 1.5 times their bytes. In exchange, every object is written twice.

Stage 3: A hot tier and a cold tier

• Load: 28,750 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $236k/mo

Check in: At 4 PB the split runs the same 48 nodes as Stage 1. What does it buy?

Why code volumes, not objects

Stage 2 coded objects and failed on seeks. Stage 3 codes again, and an interviewer asks what changed.

Code sealed volumes, not single objects.

A small object sits whole in one shard, so a healthy read is one seek.

A degraded read, when that shard is lost, takes 8.

The trade-off:

Coding volumes keeps a healthy read at one seek. In exchange, a lost shard turns a read into 8.

Check in: When does a cold GET cost 8 seeks?

Four years on: 40 PB

The interviewer asks: Four years on, you hold 40 PB.

Stage 4 holds 40 PB at the same request rate. Reads follow new objects, and they arrive at the same rate.

40 PB is 37,252,903 GiB. The hot tier still holds 7 days, 184,570 GiB, so it is unchanged.

At 1,250 PUTs a second, four years would add about 38.5 million GiB. So about one byte in eight of everything ever written, launch data included, is deleted by then.

The cold tier holds 37,068,333 GiB, coded to 55,602,500 GiB: 18,534,167 GiB a zone.

76 nodes a zone hold 18,534,167 / (76 x 312,477.5) = 78.0%, and 79.1% after a rebuild. That is 228 nodes.

Its drives are 1.2% busy: the same 1,000 GETs over 228 x 24 = 5,472 drives.

Three copies of 40 PB would need 151 nodes a zone, 453 in all.

The storage nodes are 249 x 720 hours x $6.30864 = $1,131,012.98. Three copies: 453 nodes, $2,057,626.02, so $926,613.04 a month more.

Nothing saturates, and the last tile notes "Busiest: Cold tier (sealed volumes) at 78%".

The cost tile shows $1,181k/mo. The table now stores 142,100 GiB, about 150 billion rows of 1 KB: $35,525 a month at $0.25 a GiB.

At scale the bytes are the whole bill, and the code halves them. The request rate, and the hot tier, did not change.

The latency tile shows 11ms, as before.

On the canvas there are no load buttons. The cold tier is bound by bytes, which no load changes, and the table quota holds as in Stage 1.

The trade-off:

Two tiers use 204 fewer nodes than three copies. In exchange, a degraded cold read takes 8 seeks, and the move job runs all day.

Stage 4: Four years on: 40 PB

• Load: 28,750 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $1,181k/mo

Check in: Why did the hot tier not grow with the data?

When copies win, and when a zone is lost

Skip on a first read; come back before the interview. Stages 3 and 4 run both schemes, and an interviewer asks which wins where.

Copies win for small, busy objects. A GET is one seek, and a rebuild reads one surviving copy.

A code wins for sealed, cold bytes: 1.5 times the data, not 3. At 40 PB that is 249 nodes against 453.

The cold tier is sized for degraded reads: when the shard that holds an object is lost, a GET reads 8 shards.

On Stage 3, the cold tier's 1,000 GETs a second keep its 648 drives about a tenth busy, even so.

A coded volume spans 12 nodes. Two of them keep its index in memory, and the API asks one: one hop, no seek.

At 40 PB that index holds about 150 billion entries. At about 32 bytes an entry, kept twice, that is about 40 GiB a node (our estimate).

A node has 192 GiB of memory.

Lose a zone, and both schemes still serve. Three copies keep two, one of them spare.

A stripe is one set of 12 shards. With at most 4 a zone, 8 + 4 keeps exactly 8 of each stripe.

One more lost drive then makes its stripes unreadable until the zone returns. A cold rebuild reads 8 shards, most of them in other zones.

With a zone down, the API runs 28,750 x 2 ms over 32 tasks of 2 vCPUs: 89.8%. The hot drives reach 49.6% x 3 / 2 = 74.4%.

The requirement holds, with a thin margin.

f4, the store from earlier, codes Reed-Solomon 10 + 4 inside one data centre, plus an XOR parity copy in another: 2.1 times the bytes.

f4's 2.1 times survives the loss of a whole region; the page's 1.5 survives one zone. They buy different things.

Local reconstruction codes, which add parity for small groups of shards, cut repair reads. Choose them when repair must stay in a zone.

Say it like this: "Copies for hot small objects, one seek. 8 + 4 on sealed volumes, 1.5 times. Both survive a zone; the code has no spare."

Step through the figure to compare a zone outage on each.

The trade-off:

8 + 4 halves the bytes of three copies. In exchange, a zone outage leaves it no spare, and a rebuild reads across zones.

A zone lost, with copies and with 8 + 4

1. Three copies, one a zone. A zone goes down: two copies remain, and one is spare.
2. 8 + 4, at most 4 shards a zone. A zone goes down: 8 shards remain, exactly enough, with no spare.
3. f4: Reed-Solomon 10 + 4 in one data centre, plus an XOR parity copy in another. Its 2.1 times survives a whole region.

Check in: A zone is down. Which scheme can lose one more drive and still serve every read?

Why two tiers, and when one tier wins

Two tiers cost a background job and a second write of every object. An interviewer asks when one tier is the better answer.

At 4 PB the split saves no nodes: Stage 1 and Stage 3 both run 48. At 40 PB it runs 249, against 453 for copies alone.

A single coded tier still needs copies. A volume is coded only once it seals, so new objects sit in replicated open volumes.

The split decides how long a volume stays replicated. Here it is 7 days, because reads fall sharply after that (f4).

One replicated tier wins when the data is small, or when reads do not fall with age. Then the move and its second write buy nothing.

The move is not free: it reads 312.5 MiB/s of sealed objects and writes them again as shards.

Magic Pocket does the same: it replicates new blocks and codes them once they are gathered into larger buckets.

Say it like this: "Replicate open volumes, and code sealed ones after 7 days. One tier wins when the data is small or reads do not age."

The trade-off:

Two tiers cut the bytes at scale. In exchange, every object is written twice, and a job must move it.

Check in: Reads of an object stay flat for a year. Is a 7-day hot tier still correct?

Why build it, and when S3 wins

"Why not just use S3?" is the first thing a senior interviewer asks. Compare committed with committed, and tiered with tiered.

Prices are from AWS's S3 price file of 2026-10-15. The reserved node price is from a third party.

Build, on demand: the Stage 3 and Stage 4 tiles plus about $56,700 of zone traffic. About $293,000 a month at 4 PB, $1,238,000 at 40 PB.

Build, on 3-year reserved nodes at $2.725 an hour: about $169,000 at 4 PB, $595,000 at 40 PB.

S3 Standard, storage and requests: about $123,500 at 4 PB, $827,600 at 40 PB.

A lifecycle rule can move objects after 7 days to Glacier Instant Retrieval, an S3 class with millisecond reads at $0.004 a GB.

That costs about $306,000 a month at 40 PB, at least: it bills 128 KB and 90 days at minimum.

So S3 with tiering is cheapest at 40 PB. The reserved build beats S3 Standard at 40 PB, but not at 4 PB.

People to run the build are in no figure here.

So: rent, with tiering, unless you own the hardware.

Dropbox moved off S3 onto its own hardware, Magic Pocket, at hundreds of PB. That is when building wins.

In the interview, build it anyway: the question is how S3 works. Then say when you would not.

Say it like this: "At 40 PB, S3 tiered to Glacier Instant Retrieval beats my reserved build. I build only on my own hardware, at hundreds of PB."

Step through the figure to compare the four bills at 40 PB.

The trade-off:

A build needs a team and a 3-year commitment to compete. Renting with tiering costs less at 40 PB, unless you own the hardware.

Build or rent: one month at 40 PB

1. Build on demand: the Stage 4 tile plus about $56,700 of zone traffic, about $1,238,000.
2. Build on 3-year reserved nodes at $2.725 an hour: about $595,000.
3. S3 Standard, storage and requests: about $827,600.
4. S3 moved to Glacier Instant Retrieval after 7 days: about $306,000, at least.

Check in: Is S3 cheaper than the build?

One object goes viral

The interviewer asks: One object goes viral: 5,000 GETs a second.

Stage 5 is Stage 4 with one viral object: 5,000 GETs a second for it.

The viral object's 5,000 GETs come on top of the usual traffic. Its bytes come from the page cache, so the drives see no more.

The entry is 28,750 + 5,000 = 33,750 client requests a second. The drives still see 26,500 hot GETs and 1,000 cold ones.

Every GET still reads its row: the table reads 32,500 of 40,000, past its warning line, and the row gets 5,000 against 3,000.

A DynamoDB item lives on one partition, which serves at most 3,000 strong reads a second. 5,000 / 3,000 = 166.7%.

The result names Metadata (bucket, key). One warning gives 5,000 read units a second on one partition, against 3,000.

The other gives the table at 32,500 / 40,000 = 81.25%. That quota can be raised; the limit of one item cannot.

A table scales by partitions, but one item lives on one, and no table setting raises it.

Eventually consistent reads would allow 6,000 an item, but break read-after-write. DAX cannot help: it passes strong reads through to the table.

These 5,000 hits also pass the 4,500 that keep the API and the table clean. The API reads 33,750 / 48,000 = 70.3%.

The viral row's reads are throttled and retried by the clients. The tile shows the slower data path, so it leaves those retries out.

The latency tile shows 11ms. The cost tile shows $1,183k/mo: 5,000 more table reads a second add $1,620.00 to Stage 4.

On the canvas, the hot edge still carries 27,750 of the 33,750, a share of 0.822. The table card carries the viral share, 5,000 of 32,500 reads.

Those shares hold only for this object, so there are no load buttons.

The trade-off:

Strong reads keep read-after-write for every GET. In exchange, one popular object is capped by one partition.

Stage 5: One object goes viral

• Load: 33,750 req/sec
• First to saturate: Metadata (bucket, key)
• Estimated cost: $1,183k/mo

• Metadata (bucket, key): Read capacity 81% used (on-demand): 32500 of 40000 RCU/s at 33750 RPS.
• Metadata (bucket, key): Hot-partition read throttling: hottest key draws 5000 RCU/s, above the 3000 RCU/s per-partition ceiling.

Check in: Why does a bigger table not help the viral row?

Route the row lookup by key, and coalesce it

Stage 6 keeps the viral object. Its row reads must merge before they reach the table, without serving an old row.

Each GET stays on an API task in the client's zone, which reads the bytes from that zone's copy.

Only the row lookup, about 1 KB, goes by key to a small row service. It runs 6 Fargate tasks, 2 a zone, at about 0.1 ms a lookup (our estimate).

One owner task handles each key, single-flight. With no read of the row in flight, it sends one now.

With one in flight, it waits for it to end, then sends one read for every lookup that arrived after it started.

A read takes about 2 ms, so the viral row gets at most about 500 reads a second: a sixth of its 3,000.

A lookup never joins a read that began before it arrived, so a GET after a PUT still sees the PUT.

The table now takes 32,500 - 5,000 + 500 = 28,000 reads a second: 28,000 / 40,000 = 70.0%.

Coalescing brings the table back to 70%, under its warning line.

The row service takes all 32,500 lookups, against 6 x 2 x 1,000 / 0.1 = 120,000: 27.1%.

The viral key's owner takes 5,000 + 27,500 / 6 = 9,583 lookups a second. At 0.1 ms each, that is 958 ms over 2 vCPUs: about 48% of one task.

The API stays at 70.3%. Nothing saturates, and the last tile notes "Busiest: Cold tier (sealed volumes) at 78%".

The cost tile shows $1,181k/mo. The row service adds $426.56, and 4,500 fewer table reads a second save $1,458.00: about $1,031 less than Stage 5.

The latency tile shows 11ms.

Route the row read, not the GET. Sending whole GETs to the key's owner would put 5,000 GETs and about 10.5 Gbps on one task.

It would also send two GETs in three across zones.

A cache of rows would break read-after-write, unless every PUT cleared it at once. It wins only where a read may be stale.

On the canvas, the edges carry this object's shares: 0.963 of the entry to the row service, and 0.862 of that to the table.

Those shares hold only for this object, so there are no load buttons. Step through the figure to follow the viral row.

The trade-off:

A row service adds a hop and a small fleet. In exchange, one row gets one read in flight, and read-after-write holds.

The viral row, routed and merged

1. Stage 5: 5,000 page-cache hits on top, and every GET reads its row. The viral row gets 5,000 reads; one item serves 3,000.
2. Stage 6: each GET stays in its zone. Only the 1 KB row lookup goes by key to its owner task.
3. Single-flight: while a read is in flight, new lookups wait, then share one new read.
4. At about 2 ms a read, the row gets about 500 reads a second. The bytes come from page caches.

Stage 6: Route the row lookup by key, and coalesce it

• Load: 33,750 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $1,181k/mo

Check in: Why not let each API task merge its own reads of the viral row?

The object store chain on one screen

Skip on a first read; come back before the interview. Every number on Stages 1 to 6 comes from the ones before it.

Units: drives and PB are decimal, and this page counts binary KiB, MiB and GiB. 4 PB = 3,725,290 GiB; 40 PB = 37,252,903 GiB.

Entry: 27,500 GETs that reach the drives + 1,250 PUTs = 28,750 client requests a second. Page-cache hits add to it.

Stages 5 and 6: the viral object's 5,000 page-cache hits are added, 33,750 a second. The drives still see 27,500.

Node: 24 x 13,980 GB = 312,477.5 GiB. A drive streams 6,200 / 24 = 258.33 MiB/s.

Drive time: a GET 8.333 + 0.968 = 9.301 ms, and a copy append 0.968 ms. A coded GET is 8 x (8.333 + 0.121) = 67.6 ms.

Stage 1: 3,725,290 / (16 x 312,477.5) = 74.5%, 79.5% after a rebuild. Drives 259.4 / 1,152 = 22.5%.

Stage 2: 1,862,645 / (9 x 312,477.5) = 66.2%. Drives 1,861.8 / 648 = 287.3%.

Stage 3, hot: 184,570 x 3 / 21 = 26,367 GiB a node, 8.4%. Drives 250.1 / 504 = 49.6%.

Stage 3, cold: 1,770,360 / (9 x 312,477.5) = 63.0%, 70.8% after a rebuild.

Stage 4, cold: 18,534,167 / (76 x 312,477.5) = 78.0%, 79.1% after a rebuild. Three copies: 151 a zone.

API: 28,750 / 48,000 = 59.9%; in Stages 5 and 6, 33,750 / 48,000 = 70.3%.

Table: 27,500 / 40,000 = 68.75%. Stage 5: 32,500 / 40,000 = 81.25%. Stage 6: 28,000 / 40,000 = 70.0%.

Viral row: 5,000 / 3,000 = 166.7%. Single-flight at about 2 ms a read: about 500 reads a second.

Row service: 32,500 / 120,000 = 27.1%; the owner task 9,583 x 0.1 ms over 2 vCPUs, about 48%.

Storage nodes, at nodes x 720 hours x $6.30864: 48 nodes $218,026.60; 27 nodes $122,639.96; 249 nodes $1,131,012.98.

Table: reads 27,500 x 2,592,000 s / 1 million x $0.125 = $8,910; writes $2,025; storage $3,550 at 4 PB, $35,525 at 40 PB.

API: 48 x 720 x $0.09874 = $3,412.45. Row service: 6 tasks, $426.56.

Stage 5 reads 32,500 x 2,592,000 s / 1 million x $0.125 = $10,530 from the table; Stage 6 reads 28,000, $9,072.

To the cent, the cost tiles are $235,924.12, $140,537.48, $235,924.12, $1,180,885.50, $1,182,505.56 and $1,181,474.17.

The lines here sum to 7 to 18 cents under those, because the bill adds unrounded prices.

Check in: Close the page and rebuild the cold tier's node count at 40 PB.

Repair, durability and the zone bill

Three questions usually follow. None changes a canvas, so we reason by hand.

On EC2 the unit of loss is the instance. A host failure, a retirement or a stop loses all 24 drives at once, 336 TB raw.

Repair: the placement service gives the lost node's volumes new homes on its zone-mates. They write at their drives' spare rate, 25 MiB/s a drive (our estimate).

A hot node's 26,367 GiB onto 6 zone-mates' 144 drives: 26,367 x 1,024 / (144 x 25) = 7,500 s, about 2.1 hours.

Stage 1's node holds 232,831 GiB, onto 15 zone-mates' 360 drives: about 7.4 hours.

A cold node at 40 PB holds 243,871 GiB, but each lost shard reads 8. That is about 1.95 million GiB read, mostly across zones.

Over the cold fleet of 5,472 drives at 25 MiB/s, that read takes about 4 hours.

Durability: this page gives no nines. Data is lost only when failures land inside one repair window, so repair time is the lever.

AWS publishes no instance failure rate, and correlated failures dominate any independent model: a host, a rack, a firmware batch.

Scrub: a background read checks a checksum on every object and shard. At 5% of a drive's 258 MiB/s, one pass takes about 12 days.

The zone bill: data between zones is not on the canvas. This design keeps GETs in the client's zone, a stated premise.

PUT copies, the move to the cold tier, cold GETs and row lookups still cross zones: about 1,120 MiB/s.

Over 2,592,000 s that is about 2,835,000 GiB. AWS charges $0.01 a GB in each direction: about $56,700 a month, on the build only.

Without zone-local GETs, two hot GETs in three would cross as well: 26,500 x 256 KiB x 2/3, about $224,000 a month more.

The trade-off:

Fast repair and zone-local reads keep the risk and the bill down. In exchange, every tier carries spare space for a rebuild.

Check in: Why does this page give no durability nines?

LIST, multipart, versions and deletes

An interviewer often ends on the API. Each answer here is one row design.

LIST is one Query on the bucket's partition, 1,000 keys a page: 250 read units a page at 1 KB a row.

With a delimiter such as "/", a Query cannot skip the keys under a common prefix. So each common prefix costs a new Query that starts past it.

Multipart upload: parts of 5 MiB to 5 GiB, up to 10,000 of them, an object up to 48.8 TiB.

Each part gets its own item, with sort key key#uploadId#part. 10,000 parts in one row would pass the 400 KB item limit.

Complete writes the object row once, with a condition, naming the upload id. A GET of a multipart object Queries its parts.

Versions: sort key key#inverted-version, so the latest sorts first. A GET of the latest is a Query with limit 1.

A DELETE writes a delete marker in a versioned bucket, and marks the row otherwise.

Compaction copies a volume's live objects to a fresh physical volume under the same logical id. No row is rewritten.

It drops any object no row points at, once it is older than the longest PUT. A freed volume keeps a grace period for GETs in flight.

Volumes that are never merged only shrink, so the map grows. Small volumes are merged rarely, and only then are rows rewritten.

Coded volumes are not rewritten for a delete. f4 deletes an object by deleting its encryption key, so its space stays used.

The trade-off:

Volume ids in rows keep moves and compaction free of row writes. In exchange, deleted cold bytes keep their space.

Check in: Why does compaction keep a volume's logical id?

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a caller see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

A node dies: its zone-mates rebuild its volumes and stay under 80% full. Reads use another copy meanwhile.

A zone is lost: copies keep a spare, and 8 + 4 keeps exactly 8. The API runs at 89.8%.

A PUT crashes before its commit: no row, so no object. Compaction drops the orphan bytes.

The page cache serves more than 4,500 GETs a second: add API tasks and raise the table quota.

One object goes viral: its row passes 3,000 reads. Route the row lookup by key, single-flight.

Time-ordered keys fill one partition: prefix the partition key with a hash, and merge sorted Queries for LIST.

Say it out loud: "Metadata in DynamoDB, and the row is the commit point. Bytes on HDD nodes, in packed volumes, one seek a GET."

And: "Three copies for 7 days, then 8 + 4 on sealed volumes. At 40 PB, 249 nodes, not 453."

Minutes 0 to 5: requirements. 4 PB to 40 PB, 28,750 requests a second, a zone lost, read-after-write.

Minutes 5 to 12: Stage 1, the API and the table. One request's states, and why DynamoDB.

Minutes 12 to 22: Stages 2 and 3. Coding objects fails on seeks; two tiers code volumes instead.

Minutes 22 to 30: Stage 4 at 40 PB, a zone lost, one tier or two, and build against rent.

Minutes 30 to 38: Stages 5 and 6. The viral row and single-flight.

Minutes 38 to 45: repair, durability, the zone bill, LIST and multipart.

The trade-off:

Each answer has a price. Copies cost bytes, a code costs seeks and its spare in an outage, and a row service costs a hop.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.