Design YouTube

Do the numbers before the boxes

An interviewer gives a scale, not a load. This system has two loads, uploads and views, and they differ by a factor of 1,000.

A 45-minute version of this answer is at the end.

We assume 86,400 uploads a day, about one for every 1,000 views. 86,400 / 86,400 = 1 upload per second on average.

The evening peak runs about 3 hours at twice the average, our assumption: 2 uploads per second.

Views: about 17.28 million daily viewers watch 5 videos each. That is 86.4 million a day: 1,000 views per second, 2,000 at the peak.

This page is about 1% of YouTube. YouTube takes in over 500 hours of video a minute: about 8.6M five-minute uploads a day, against our 86,400.

An average upload is 5 minutes of 1080p at about 8 Mbps: 300 s x 8 / 8 = 300 MB.

We cut each video every 4 seconds into chunks: 300 / 4 = 75 chunks, each encoded on its own.

A vCPU is one virtual processor. A chunk takes about 24 vCPU-seconds: 6 for each second of video, our estimate for 5 resolutions at a fast preset.

So one upload is 75 x 24 = 1,800 vCPU-seconds of encoding.

A view watches about 150 s, half an average video, at about 2.5 Mbps delivered: 150 x 2.5 / 8 = 46.875 MB.

Out of scope: live streaming, recommendations, search and comments. Each is a system of its own.

Check in: Close the page and derive the two peak rates.

Upload straight to S3, in parts

A 300 MB upload over a home link takes minutes, and phones drop connections. A creator who loses the link at 90% should not start again.

The client opens an upload session.

The upload API writes the video row and starts an S3 multipart upload: one file sent as numbered parts, joined by S3 at the end.

It returns 30 signed URLs, one for each 10 MB part. A signed URL is a short-lived link that lets one request write one part, valid for an hour.

The client PUTs the parts straight to S3. The upload API never carries the upload: 2 uploads per second x 300 MB is 600 MB per second it never sees.

A client sends about 4 parts at a time (our estimate), so a drop loses up to 4 parts in flight, 40 MB. ListParts shows which finished.

The client then sends only the rest. Expired URLs are signed again, and a lifecycle rule aborts uploads still unfinished after 7 days.

The client then calls complete. The API asks S3 to join the parts, marks the video uploaded, and starts the planner without waiting.

S3 allows up to 10,000 parts, so 10 MB parts cover 100 GB. For bigger files the client picks bigger parts.

The trade-off:

The client does more work: it splits, tracks and resends parts. In exchange, a dropped link costs a few parts, and the upload API carries no video.

Check in: A phone loses signal during part 27 of 30. What does it send when it reconnects?

Split, encode, merge

A 5-minute video must play in every resolution within 10 minutes of upload, at the evening peak too.

One 2-vCPU worker would take 1,800 / 2 = 900 s, 15 minutes, already past the target. And a crash starts it over.

To transcode is to decode a video and encode it again, at other resolutions and bitrates.

A codec is the compression format; H.264 plays on nearly every device. A rendition is one codec at one resolution: our ladder has 5, 1080p to 240p.

A keyframe is a frame that decodes on its own; the frames after it need the ones before them.

Split: the planner cuts at fixed times every 4 s, 75 chunks. It reads the source's keyframe index, so each job starts decoding at the keyframe before its cut.

Encode: an SQS queue holds one message per job. A job decodes its chunk once, encodes all 5 resolutions, and forces a keyframe at the cut.

Merge: every chunk starts on a keyframe at the same instant in every resolution, with identical encoder settings. So the encoded chunks are already the streaming segments.

The planner encodes the audio once, into 75 segments of its own. A manifest, or playlist, lists a rendition's segments in order, so the merge is just writing manifests.

Each finished job adds its number to the plan's set of done jobs, in DynamoDB. Any worker that reads back 75 writes the manifests and publishes.

With fleet headroom, a 5-minute video plays about 30 to 40 s after the upload completes.

Step through the figure to follow one upload.

The trade-off:

Chunk edges can show small quality steps, and every chunk must use identical settings. In exchange, 75 jobs run at once, and a crash redoes one 24-s job, not the video.

One upload, from parts to playable

1. The upload API signs 30 part URLs. The client sends the parts straight to S3, so the API never carries the upload.
2. The planner cuts at fixed times every 4 s: 75 chunks. Each job starts decoding at the source keyframe before its cut.
3. 75 jobs run at once. Each decodes its chunk once and encodes all 5 resolutions, with a keyframe at the cut.
4. Every chunk starts on a keyframe at the same instant in every resolution. So the encoded chunks are the streaming segments.
5. Each job adds its number to the done set. The worker that reads back 75 writes the manifests and publishes.

Check in: A worker adds job 75 to the done set, then dies before it publishes. What happens?

Reading the tiles

Each stage has a live canvas and four result tiles. Here is how to read them on this page, once, so the stages can stay short.

Load: the buttons count uploads per second on Stages 1 to 3, and views per second on Stages 4 and 5. The tile labels both req/sec.

A part saturates when it gets more work than it can serve. A card turns red at 90% of its limit; only past 100% does the result name it.

With nothing saturated, the last tile notes the busiest part instead.

Latency: the slowest path, in milliseconds. On the upload stages it ends when the upload lands in S3.

The S3 card adds latency by object size, a size step: 150 ms for a 10 MB part. A real 300 MB upload takes minutes on a home link.

Encoding runs later, from the queue, so its wait is not in that number. A short fleet shows as a saturated card instead: jobs waiting, uploads publishing late.

When a saturated part sits on the slowest path, the tile shows a floor, such as "≥ 4ms". The real figure is higher, by an amount no model can know.

Past 70% busy, a card's latency climbs, to 4 times its base at full use. A load balancer adds 1.5 ms: 1 ms of its own and 0.5 ms of TLS.

A database card's capacity is our CPU model, 0.5 ms of CPU per statement. Real throughput also depends on commit latency and connections.

Cost: each card bills its peak load for the whole month. A real fleet follows the daily curve, so its real month costs about the average.

The cost tile also bills the API's replies to users as data sent to the internet, per GB after 100 GB a month free.

The video bytes come from CloudFront, which is not on the canvas. The bill near the end of the page prices them by hand.

The fleet card adds its job queues together, with no priority, and each queue card's warning gives the whole fleet's shortfall. The prose gives the priority.

Check in: The latency tile on an upload stage reads 150ms. How late does a new upload publish?

Chunked H.264 on a Fargate fleet

Stage 1 is the launch design at 2 uploads per second, the evening peak. A queue feeds a fleet of containers, one job at a time per vCPU.

The fleet is an ECS service on Fargate: AWS runs the containers, and no server of ours needs managing.

Each task is a container with its own vCPUs and memory, billed for both.

Jobs: 2 uploads x 75 = 150 per second. CPU: 150 x 24 = 3,600 vCPUs busy.

The fleet adds 2-vCPU tasks to stay near its 70% target: 3,600 / (2 x 0.7) = 2,571.4, so 2,572 tasks at 69.98%.

Nothing saturates; the note reads "Busiest: Transcode fleet at 70%".

One ECS service runs at most 5,000 tasks, a limit AWS does not raise. So the fleet finishes at most 5,000 x 2 / 24 = 416.67 jobs per second.

That is about 5.6 uploads per second. The account's Fargate vCPU quota starts at 6 per Region; this fleet needs 5,144, so raise it early.

A long upload holds at most 200 jobs in flight. The planner sends the first 200, and each worker that finishes one sends the next.

The latency tile shows 150ms, the part upload into S3. The cost tile shows $197k/mo, 92.6% of it the fleet.

The fleet reaches S3 through a gateway endpoint, a private route that carries its bytes free. A NAT gateway would charge about $77,000 for the same bytes in an average month.

Press 1, the daily average: 1,286 tasks, nothing saturates, and the cost tile shows $99k/mo.

The trade-off:

Transcoding is the compute. Size it from the work: vCPU-seconds for each second of video, times the video uploaded per second.

Stage 1: Chunked H.264 on a Fargate fleet

• Load: 2 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $197k/mo

Check in: Why not one worker per video?

Players adapt; a CDN carries the bytes

Viewers watch on phones, TVs and browsers, over networks that change by the minute. A stall or a slow start sends them away.

ABR, adaptive bitrate, means the player picks each next segment's resolution from its measured throughput and buffer.

It starts low, so the first segment arrives fast, then climbs. Aligned keyframes let it switch at any segment boundary.

The segments are CMAF: fragmented MP4, with video and audio as separate tracks. HLS and DASH are two manifest formats over the same segments, so one copy serves every player.

The ladder: 1080p at 5 Mbps, 720p 2.5, 480p 1.2, 360p 0.8, 240p 0.4, and audio 0.128. About 10 Mbps together, 375 MB for 5 minutes.

Most views are 480p to 720p on phones and 1080p on TVs: about 2.5 Mbps delivered on average, our estimate.

A CDN, a content delivery network, caches segments at edges near viewers. Here it is CloudFront, which fetches from the Renditions bucket on a miss.

Moving data from S3 to CloudFront is free, so a miss costs one S3 GET. The bytes to viewers are egress: data sent out to the internet, billed per GB.

The watch API signs CloudFront cookies for one video's path, for about an hour. Private and unlisted videos stay private at the edge.

At the peak, 2,000 views per second x 150 s = 300,000 viewers. At 2.5 Mbps that is 750 Gbps, 5 times CloudFront's default 150 Gbps per distribution.

Requests peak at about 160,000 per second, under the default 250,000. Ask for the bandwidth quota early, or split the traffic over several distributions.

CloudFront's cost is worked by hand in the bill near the end of the page.

Step through the figure to follow one view's segments.

The trade-off:

Every view moves about 47 MB. The servers are cheap next to the bytes, so delivery is where this system's money goes.

The ladder and the player

1. Five video renditions and one audio track, about 10 Mbps together. A 5-minute video stores about 375 MB.
2. The player starts low, then picks each next segment's resolution from its throughput and buffer. A dip steps it down.
3. A CDN edge serves most segments. On a miss, CloudFront fetches from S3, and that transfer is free.
4. At the peak, 750 Gbps leaves the edges, 5 times the default quota. Ask for more early, or split distributions.

Check in: Why does a player start at a low resolution?

AV1 for every video

The interviewer asks: AV1 saves bandwidth. Encode everything in AV1 too.

Stage 2 encodes every video in AV1 as well as H.264, in the same queue as new uploads.

AV1 is a newer codec, at about 5 times H.264's CPU (our estimate). It encodes the same picture in at least about 30% fewer bits than H.264; published comparisons run about 30 to 50%.

A player turns only part of that into fewer bytes, because adaptive streaming spends some of it on a sharper picture. We use 30% fewer bytes, a conservative stand-in.

So an AV1 job covers one chunk at one resolution: 375 AV1 jobs per upload, 24 vCPU-seconds on average.

The 1080p resolution has about 55% of the ladder's pixels, so its job takes about 67 vCPU-seconds; the 240p job about 3.

With H.264, an upload brings 450 jobs, 6 times the work. At 2 uploads per second that is 900 jobs per second.

The fleet finishes at most 416.67 jobs per second. The Transcode fleet saturates.

Its warning reads "Scaled to the 5000-task ceiling and 100% used at 900 RPS"; here that is jobs per second.

Holding 70% would take 900 x 24 / (2 x 0.7) = 15,428.6 tasks: more than three times what one service may run.

Nothing is refused. Jobs wait in the queue, and new uploads' H.264 jobs wait behind AV1 work. The backlog grows by 483 jobs per second, about 1.7 million after an hour.

Even the daily average, 450 jobs per second, is about 8% past what the fleet finishes, at our estimates. The queue grows about 2.9 million jobs a day.

SQS keeps a message 4 days by default and 14 at most, then deletes it silently. A dead-letter queue does not catch that.

So the job queues keep messages 14 days, and an alarm watches the age of the oldest message. Even at 14 days, a backlog growing about 2.9M jobs a day loses jobs, and some uploads never publish.

The latency tile still shows a plain 150ms, not a ≥ floor: the fleet sits behind the queue, after the upload. Stage 2 has no load buttons; the cost tile shows $372k/mo.

The trade-off:

AV1 for everything spends 6 times the encode CPU, most of it on videos almost nobody watches.

Stage 2: AV1 for every video

• Load: 2 req/sec
• First to saturate: Transcode fleet
• Estimated cost: $372k/mo

• Encode jobs: Transcode fleet drains 417/s of the 900/s offered, so the backlog here grows by 483/s - about 1.7M messages after an hour.
• Transcode fleet: Scaled to the 5000-task ceiling and 100% used at 900 RPS.

Check in: Is the fix more tasks?

Spend encode CPU where the views are

AV1 saves bytes on every view, but it costs CPU once, at encode. It pays only on a video that gets watched enough.

Views are uneven. About 2% of uploads carry about 70% of watch time, our estimate: the head. The other 98% are the tail.

With 1,000 views per upload, a head video averages 0.7 x 1,000 / 0.02 = 35,000 views. A tail video averages about 306, and the median far fewer.

One AV1 encode is 375 x 24 = 9,000 vCPU-seconds: about 3.57 vCPU-hours at the fleet's 70%, priced as if it bought its own capacity.

A view saves 46.875 MB x 30% x the share of views on devices that play AV1: 30%, Netflix's real share, to 60%, our estimate.

That is 4.2 to 8.4 MB per view. At $0.020 per GB, CloudFront's list price past 5 PB a month, it is worth about $0.00008 to $0.00016.

Big customers rarely pay list: they commit to a volume for a lower price, here an illustrative $0.004 a GB (our estimate).

Spot capacity is spare AWS capacity at up to 70% off, which AWS can reclaim at 2 minutes' notice.

The 1,000-view trigger is a predictor, not the break-even. A video with 1,000 views in its first week is usually in the head, and head videos average about 35,000 views over their life.

One AV1 encode costs about $0.18 on demand. At list prices it pays back after about 1,100 to 2,200 views; at a committed $0.004 a GB, after about 5,600 to 11,200 views.

On Spot both fall by up to 70%. At the committed price, the average head video pays the encode back about 3 to 6 times over on demand. On Spot it is 10 to 20 times.

AV1 for the tail would cost 3.7 to 7.3 times what it saves at list, on demand. At the committed price it would cost 18 to 37 times.

So every video gets H.264 at once, and AV1 goes only where the views predict a payback.

Step through the figure to compare the head and the tail.

The trade-off:

A late bloomer waits for its AV1, and workers must honour the queue priority. In exchange, AV1 runs only where it pays back.

AV1 where the views are

1. A few videos carry most of the watching: about 2% of uploads carry about 70% of watch time, our estimate.
2. An AV1 encode costs about $0.18 once. It pays back after about 1,100 to 2,200 views at list, 5,600 to 11,200 committed.
3. The 1,000-view trigger is a predictor, not the break-even. Videos past it are usually in the head, which averages about 35,000 views.
4. Every upload gets H.264 at once. AV1 goes to proven channels at upload and to videos past 1,000 views, and runs in the fleet's spare.

Check in: Why not predict popularity at upload?

AV1 only for videos that earn views

Stage 3 gives every upload H.264 at once. AV1 goes to proven channels at upload, and to other videos once they pass 1,000 views in their first week.

Jobs per upload: 75 H.264 + 2% x 375 AV1 = 82.5. At 2 uploads per second: 150 H.264 and 15 AV1 jobs, 165 in all.

The AV1 plan is a conditional write on the video's row: it succeeds only if AV1 is not yet planned. A count that crosses 1,000 twice plans nothing more.

The fleet scales on the H.264 queue's backlog alone and keeps about 30% spare, since it holds 70% busy.

AV1 jobs, about a tenth of the H.264 work, run in that spare, so they never buy capacity or delay a new upload. At a peak with no spare, AV1 waits.

The tile adds both queues, so it sizes the fleet about 10% larger: 2,829 tasks against 2,572. That is 165 x 24 / (2 x 0.7) = 2,828.6, at 69.99%.

Nothing saturates; the note reads "Busiest: Transcode fleet at 70%". The cost tile shows $216k/mo, against Stage 2's $372k that could not keep up.

At the average month the tile's fleet is 1,415 tasks, about $100,600. Sized on H.264 alone, the real one is 1,286 tasks, about $91,400.

Spot, and Graviton (AWS's ARM processors, 20% less per vCPU), would cut that further.

Even a 67-s 1080p job finishes inside the queue's 120-s hold and Spot's 2-minute notice. A job lost anyway is redone from the queue.

Press 1, the daily average: nothing saturates, and the cost tile shows $109k/mo.

Press 6: a creator fund triples uploads, every day. The Transcode fleet saturates at 5,000 tasks. That is your turn, at the end of the page.

At that load the 2-vCPU fleet is at the wall unless our CPU estimate is more than 15.8% too high.

Through a 3-hour peak, AV1 waits, and H.264 alone still falls behind. New uploads publish about 14 minutes late, against the 10-minute target.

That delay holds only within 2.3% of our CPU estimate: a thin consequence near this load, not the proof.

The trade-off:

On the tile AV1 adds 129 tasks at the average, about $9,200 a month; in the spare it buys none. It saves up to about $142,600 to $285,200 a month at list, $28,500 to $57,000 committed.

Stage 3: AV1 only for videos that earn views

• Load: 2 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $216k/mo

Check in: What does the AV1 queue's low priority buy?

The upload chain on one screen

Skip on a first read; come back before the interview. Every number on Stage 3 comes from the one before it, at the 2-upload peak.

Uploads: 86,400 a day / 86,400 = 1 per second, x 2 at the peak = 2.

Upload load balancer: 2 x 2 = 4 calls per second, $17.15. Upload API: 4 calls, 2 tasks, $71.09.

Statements: 4 per second, from opening and completing each upload, $973.44. Publishing adds 2 more, not on the canvas: about 6 at the peak.

Parts: 2 x 30 = 60 PUTs per second, $777.60. Plans: 2 per second, $426.17.

Jobs: 2 x 75 = 150 H.264 per second, $466.16; 2 x 7.5 = 15 AV1, $46.26.

Fleet: 165 x 24 = 3,960 vCPUs busy; at 70%, 2,829 tasks of 2 vCPUs, $201,121.53.

Video segments: 150 H.264 jobs x 5 + 15 AV1 jobs x 1 = 765 per second. Audio: 2 x 75 = 150. Together 915 PUTs per second, $11,858.40.

Job counts: 2 x 1.1 plan items (1, plus 5 for 2% of uploads) + 165 dones = 167.2 writes per second, $271.11.

Replies to creators: 11 KiB of answer, plus a 4 KiB TLS handshake, the setup that opens a secure connection.

The two calls of an upload are minutes apart, so each opens a new connection: 15 KiB a reply.

4 calls per second x 15 KiB x 2,592,000 s in a 30-day month = 148.3 GiB, which AWS bills as GB.

Past the free 100 GB, at $0.09 a GB, that is $4.35.

Total: $216,033.26 a month at the peak, the tile's $216k.

Check in: Close the page and rebuild the chain from 2 uploads per second to the fleet's task count.

Every view event updates the video's row

The interviewer asks: Creators want watch time, and ranking uses it.

The page turns to watching. Stages 4 and 5 count views per second, not uploads: 1,000 views for every upload.

The player calls the watch API for the page. It sends a view event at 30 s, then a heartbeat every 30 s: a small "still watching, at this position" message.

A 150-s view sends 5 heartbeats: 7 calls per view. It is an upper estimate, as views shorter than 30 s send none.

The Watch API spends 2 ms on the page and 0.5 ms per event: 5 ms over 7 calls, 0.7143 ms per call (our estimates).

In this design each event is one UPDATE on the video's row: 6 writes per view, plus 2 reads for the page.

At 2,000 views per second: 14,000 calls, 16,000 statements, 12,000 of them writes.

The writer is a db.r6g.xlarge. Each of its vCPUs runs 1,000 / 0.5 = 2,000 statements per second, 8,000 in all. The Metadata database saturates.

Its warning reads "Writer 150% used: 12,000 writes/s against 8,000 on the writer alone (readers take no writes)". The reader is at 50%.

The latency tile shows "≥ 4ms", a floor: 1.5 + 0.7143 + 0.5 x 4 = 4.21 ms. The database's 0.5 ms counts 4 times at full use.

Really, updates queue on the writer until they time out, and events are lost or retried.

Not on the tile: a viral video at 10% of views takes 1,200 writes per second on one row.

Each waits for that row's lock, about 1,000 updates per second at most, our estimate. A bigger writer does not lift that.

The cost tile shows $8,146/mo. About $4,688 of it is replies to viewers, data sent to the internet.

A reply averages about 1 KiB. A view's 7 calls are 30 s apart, so they share one connection.

Its setup, a 4 KiB TLS handshake, adds about 0.6 KiB a call: 1.6 KiB a reply.

14,000 calls per second x 1.6 KiB is about 55,371 GiB a month.

AWS prices it in tiers: 100 GB free, then $0.09, $0.085 and $0.07 a GB as the month grows.

Press 1,000, the daily average: 6,000 writes, 75% of the writer. Nothing saturates; the latency tile shows 3ms.

The note reads "Busiest: Metadata database at 75%", and the cost tile shows $4,638/mo.

The trade-off:

The writes grow with watch time, not with videos. And a viral video's writes all wait on one row.

Stage 4: Every view event updates the video's row

• Load: 2,000 req/sec
• First to saturate: Metadata database
• Estimated cost: $8,146/mo

• Latency past saturation is a lower bound, not an estimate. Metadata database is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Metadata database: Writer 150% used: 12,000 writes/s against 8,000 on the writer alone (readers take no writes).

Check in: Why not a bigger writer?

Count in memory, log every event

Stage 4 put every heartbeat on one writer. The screen needs a number now; creators and ranking need one they can trust.

Each view event and heartbeat becomes one HINCRBY on the video's live counters: an add to one field, in Valkey, an in-memory store.

The watch page reads the live count there: 7 counter commands per view, 6 adds and 1 read.

Each event is also one record in the view log, a Kinesis stream. A Kinesis shard takes 1,000 records per second.

The log is keyed by view id. Keyed by video, a viral video's 1,200 records per second would overflow one shard; keyed by view, they spread over every shard.

Every event carries its view id and a sequence number. A view counts once per view id, at 30 s.

An hourly consumer reads the log, drops repeats and bot traffic, and adds each video's validated counts to its row.

About 2 million videos are watched in an hour, our estimate: about 556 row updates per second, about one batched statement per second.

The live count is approximate; a retried event can count twice until the hourly totals correct it. The hourly history stays in S3.

Step through the figure to follow one view's events.

The trade-off:

Two more stores to run, and a live count that can be briefly wrong. In exchange, view events never touch the writer.

One view's events

1. A 150-s view makes 7 calls: the page, a view event at 30 s, and 5 heartbeats.
2. In Stage 4 each event updates the video's row. That is 6 writes a view, 12,000 a second against the writer's 8,000.
3. In Stage 5 each event adds to the live counter and appends to the log. Keyed by view, a viral video spreads over every shard.
4. Once an hour, a consumer drops repeats and bots, and adds each video's validated counts to its row.

Check in: Why key the view log by view, not by video?

Counts in memory, every event in a log

Stage 5 moves view events off the database, at the same 2,000 views per second.

Counter commands: 2,000 x 7 = 14,000 per second. The live counters are a cache.r7g.large, which runs 160,000 commands per second: 8.8%.

160,000 is AWS's measured figure for one Graviton2 core. This node is a newer Graviton3, which has no published figure, so the card uses the Graviton2 figure as a safe minimum.

Log records: 2,000 x 6 = 12,000 per second. The view log has 18 shards, 18,000 records per second, so it runs at 66.7%.

The database takes only the page's 2 reads per view: 4,000 per second, 50%.

Nothing saturates; the note reads "Busiest: Watch API at 67%". The latency tile shows 3ms: 1.5 + 0.7143 + 0.5 = 2.71 ms.

The cost tile shows $9,029/mo, about $882 more than Stage 4's $8,146 that could not carry the load.

The first real wall is now the view log, at 18,000 / 6 = 3,000 views per second. More shards move it.

Press 1,000, the daily average: nothing saturates, and the cost tile shows $5,302/mo.

The trade-off:

A number that changes thousands of times a second is not a row. Count it in memory, log every event, and write the database in batches.

Stage 5: Counts in memory, every event in a log

• Load: 2,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $9,029/mo

Check in: What if the counter node dies?

Delivery is the bill

Skip on a first read; come back before the interview. The canvases show servers. Most of this system's money is bytes.

The bill mixes two times on purpose. Delivery, compute and requests are an average month at today's rates; storage is the stock after a year, with sources already archived.

At the average, 1,000 views per second x 46.875 MB is about 113 million GiB a month, AWS's billed GB.

At CloudFront's list prices, that data is about $2.30 million a month.

Requests add about $207,350: about 80 per view, mostly 37.5 video and 37.5 audio segments, plus playlists (our estimate).

We assume about 5% of requests miss the CDN and go to S3. For a long tail of videos with a few hundred views each, that is optimistic.

A miss costs little money, about $4,100 a month, but it slows a tail video's first segment. Origin Shield, a cache layer in front of S3, helps.

Serving straight from S3 would cost about $5.74 million.

At about 110 PB a month, nobody pays list. At the committed $0.004 a GB, our estimate and not a quote, data is about $452,600.

Storage grows by about 1.64 million GiB a month. After a year, about $414,400 a month in S3 Standard; about $255,100 with sources archived after 30 days.

The next storage lever is tiering the renditions, about $120,000 a month (our estimate).

A safety classifier checks 1 frame a second of each upload on GPUs before it publishes: about $4,300 a month.

Every line at list: about $2.88 million a month. Delivery is 87.4%, storage 8.9%, the transcode fleet 3.2%.

The canvases' lines come to about $103,700 of it, the replies to users included. The fleet, sized on H.264 alone at 1,286 tasks, is about $91,400.

At the committed price the month is about $1.03 million, and delivery is still the largest line, at 64.6%.

At far larger scale, the answer is caches inside viewers' ISPs, such as YouTube's Google Global Cache.

The trade-off:

Delivery is the bill. Spend compute where it saves bytes, and buy the bytes at a committed price.

Check in: What are the two largest lines at list, and what shrinks each?

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a user see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

A worker dies mid-job: the job reappears after its 2-minute hold, and another worker redoes it. Fixed segment keys overwrite, not duplicate.

A job fails 3 times: it moves to a dead-letter queue, kept for failed jobs. The video is marked failed, and the creator is told.

A plan is stuck short of 75 with nothing dead-lettered: a sweeper replans the missing jobs.

The fleet is short: jobs wait in the queue and the publish delay grows. A Fargate vCPU quota that is hit stops new tasks starting.

A Spot task is reclaimed: on the 2-minute notice the worker finishes its job, or the queue hands it to another.

A premiere: Origin Shield collapses simultaneous misses into one fetch, and the log keyed by view spreads its events.

The origin Region fails: cached segments keep playing, misses fail. Cross-region replication of renditions covers it.

A takedown: delete the segments, then invalidate the video's path in CloudFront. Cookies already issued expire within the hour.

At 100 times, 200 uploads per second at the peak need about 257,000 2-vCPU tasks for H.264. That is 52 ECS services at the 5,000-task limit, or custom encode hardware.

Delivery reaches about 12 EB a month, so it moves into caches inside ISPs, and the system spans many Regions from day one.

Say it out loud: "Cut every 4 seconds, force a keyframe at each cut, and the chunks are the segments. AV1 for the head, counters off the database."

Minutes 0 to 5: requirements, and the two rates: 2 uploads and 2,000 views per second at the peak.

Minutes 5 to 10: upload. Signed multipart parts, resume, and no upload through the API.

Minutes 10 to 20: Stage 1. Split, encode, merge; size the fleet; ECS's 5,000 tasks.

Minutes 20 to 27: Stages 2 and 3. AV1 for everything breaks the fleet; AV1 for the head, in the fleet's spare.

Minutes 27 to 35: delivery. HLS and DASH, the CDN, the bill at list and committed, the long tail, the quota.

Minutes 35 to 42: Stages 4 and 5. Heartbeats on the writer, then counters and a log keyed by view.

Minutes 42 to 45: storage tiers, safety, failures.

The trade-off:

Each answer has a price: a planner, two queues, a done set and a log are more to run than one worker and one row. They cost far less than the bytes.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.