Design an ad click aggregation system

Do the numbers before the boxes

An interviewer gives a scale, not a load. If we draw before we do the math, we cannot say why any box has its size.

We assume 864 million clicks a day on 1 million active ads: 864,000,000 / 86,400 = 10,000 clicks a second on average.

The evening peak is twice that, our assumption: 20,000 clicks a second. That is the entry rate.

Each link carries a click id, signed by the ad server with an HMAC, a keyed hash that shows the link was not altered.

A click event is 512 bytes: ids, time, address, user agent, device and referrer.

An ordinary top ad gets 1% of clicks. A viral ad can get 10%: 2,000 clicks a second on one ad at the peak.

A sports final is 3 times the peak for 30 minutes: 60,000 clicks a second.

In a peak minute 150,000 distinct ads are clicked. One total per ad per minute is 150,000 / 60 = 2,500 rows a second, 0.125 a click.

Check in: Close the page and derive 20,000 clicks a second and 2,500 rows a second.

Count each click in the database

The simplest answer: every click adds 1 to a counter item for its ad and minute. Stage 1 runs it at 20,000 clicks a second.

A click receiver checks the click, adds 1 to the counter, then redirects the user. It takes about 1 ms of CPU a click, our estimate.

At 1,000 clicks a second a task and a 70% target, 20,000 / 700 = 28.6, so 29 tasks run, at 69%.

DynamoDB bills writes in WCU, write capacity units: 1 WCU is one write a second of an item up to 1 KB.

An increment per click is 20,000 writes a second, 1 WCU each. At a 70% target, 20,000 / 0.7 = 28,571, rounded up to 28,600 WCU: 69.9%.

Read the four tiles. "First to saturate" reads "Nothing at this load"; its note names the Click counters, at 70%.

A part saturates when it gets more requests than it can serve. A card turns red at 90% of its limit; only past 100% does the result name it.

The latency tile shows 5ms, a click accepted: load balancer 1.5 + receiver 1 + counter write 2 = 4.5 ms.

The cost tile shows $32k/mo: $32,406.44. The counter table is $13,584.74 of it, and the answers to browsers are $15,009.86.

Data sent out to the internet is called egress, and data coming in is free. AWS bills egress per GB: 100 GB a month free, then $0.09 a GB, falling to $0.05 at this volume.

Each click is answered with a 302 of about 0.5 KB. Each click usually opens its own encrypted connection too.

Setting one up, the TLS handshake, sends about 4 KB more, mostly certificates. So 4.5 KB goes out per click, and the handshake is 8 times the redirect.

If browsers reused or resumed connections, a click would send about 0.5 KB, and egress would be $2,143.84.

Press 10,000, the average hour. Nothing saturates.

Press 30,000. The Click counters saturate: 30,000 writes a second against 28,600 WCU.

The trade-off:

Every card is green at 20,000, and three problems hide behind them. A retried increment counts twice, one ad is one item, and nothing is kept to recount.

Stage 1: Count each click in the database

• Load: 20,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $32k/mo

Check in: What breaks first as clicks grow? Find the share of clicks at which one ad breaks its item before the table breaks.

A table is not an item

The interviewer asks: One ad goes viral: 10% of all clicks. What breaks?

The viral ad has one counter item a minute, and every one of its clicks writes to it.

Its item takes 10% x 20,000 = 2,000 writes a second. One partition of a table serves at most 1,000 WCU, and one item lives on one partition.

The card turns that into a ceiling for the whole table: 1,000 / 0.1 = 10,000 clicks a second, where the hot item reaches 1,000.

Stage 2 runs at 20,000. The Click counters saturate, and the latency tile shows "≥ 5ms", a lower limit. The cost tile shows $32k/mo, as in Stage 1.

What really happens: 1,000 of the viral ad's 2,000 writes a second are throttled.

Almost all of the other ads' 18,000 succeed. Ads sharing the hot item's partition are throttled too, until DynamoDB moves the item to its own partition.

Receivers stuck retrying delay every redirect on their tasks, then drop the click. The ad that matters most is under-counted.

More WCU does not help. The table already has 28,600; the limit is per item.

Press 5,000: the whole stage at a quarter. The viral item takes 500 writes a second of its 1,000, and nothing saturates.

The trade-off:

Only fewer writes to that item help. So count first, and write totals.

Stage 2: One ad goes viral

• Load: 20,000 req/sec
• First to saturate: Click counters
• Estimated cost: $32k/mo

• Latency past saturation is a lower bound, not an estimate. Click counters is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Click counters: Hot-partition write throttling: hottest key draws 2000 WCU/s, above the 1000 WCU/s per-partition ceiling.

Check in: Why not provision more WCU?

Count first, then write totals

We want one write per ad per minute, not one per click. Something has to count in between, and keep the clicks while it does.

Receivers put each click on a Kinesis stream and wait for its acknowledgement before the redirect. A click the user saw succeed is never lost.

The stream is keyed by click id, so the viral ad spreads over all 32 shards. A shard takes 1,000 clicks a second: 32,000 in all.

A Flink job, the aggregator, counts per ad per minute. It runs 15 slots, each one Managed Flink processing unit, or KPU: 1 vCPU and 4 GB.

AWS's starting figure is 1 MB a second per KPU: 1,048,576 / 512 = 2,048, rounded to 2,000 clicks a second, 0.5 ms each. 15 slots take 30,000.

It writes one total per ad per minute. Rows follow distinct ads, not clicks: the viral ad is now 1 put a minute.

Every click also goes to a raw archive in S3: 412 GiB a day, about 82.4 GiB as Parquet, 31.4 TiB for 13 months. Sizes are binary, as AWS bills.

Step through the figure to compare an increment with a total.

The trade-off:

Writes now scale with ads, not clicks. In exchange the counts arrive about a minute late, not at once.

An increment per click, or a total per minute

1. Counting by increment, every click is one write to its ad's counter item. 20,000 clicks a second are 20,000 writes.
2. A write times out but lands, and the retry adds 1 again. The click is counted twice.
3. The viral ad's item takes 2,000 writes a second. One item gets at most 1,000, so half are throttled.
4. With totals, the aggregator counts each minute first, then writes one row per ad. That is 2,500 rows a second, and the viral ad is 1 put a minute.
5. A total is put with a version. A repeat writes the same value, and an older total is refused, so retries are safe.

Check in: Why key the stream by click id rather than ad id?

Run the stream design

Stage 3 runs the stream design at 20,000 clicks a second, with the viral ad still at 10%.

The stream runs at 20,000 / 32,000 = 62.5%, the aggregator at 66.7%, and Minute counts at 2,500 rows a second of 3,600 WCU, 69.4%.

"First to saturate" notes the Minute counts, at 69%.

The latency tile shows 73ms: 1.5 + 1 + 70 = 72.5 ms. The stage assumes 70 ms for the stream put; measure it.

AWS publishes no typical put latency. The 100 ms target holds if a put is acknowledged within 100 - 1.5 - 1 = 97.5 ms, so watch its p99.

Past a timeout, the receiver redirects anyway and spools the click locally. Everything after the stream is off the path.

The cost tile shows $24k/mo: $23,679.70. Egress is the same $15,009.86 as in Stage 1: every click still gets its answer.

Without egress, Stage 3 costs about $8,670, half of Stage 1's $17,397.

The tile holds the peak all month. An average month, at 10,000 clicks a second, is $15,024.06, with $8,598.76 of egress.

Press 10,000, the average hour. Nothing saturates, and the cost tile shows $15k/mo.

Press 29,000. Minute counts saturates: 29,000 x 0.125 = 3,625 rows a second against 3,600 WCU.

The Aggregator (Flink) card is red at 96.7%, 29,000 of 30,000 clicks, and the Click stream at 90.6%, 29,000 of 32,000. Neither is named.

That 0.125 holds when growth means more users reaching more ads. If growth piles onto a few ads, the first real wall is the aggregator, at 30,000.

The trade-off:

The bill for the parts halves against Stage 1; the answers to browsers cost the same. In exchange there are three more parts to run: a stream, a job and an archive.

Stage 3: Stream the clicks, count them, write totals

• Load: 20,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $24k/mo

Check in: At 29,000 the table is named first. When is that not the real first wall?

A key is never split, so count in two stages

Flink groups by ad to count. A keyed step sends every click of one ad to one slot, so the viral ad is back on one place.

Single-stage, the viral ad's slot takes 2,000 + 18,000 / 15 = 3,200 clicks a second, against its 2,000.

That slot's backpressure slows every slot upstream, so the whole job runs at its pace. More slots do not help: a key is never split.

So the job counts in two stages. First it drops duplicates, keyed by click id, which spreads evenly.

Then, in the same slot, it pre-counts: a partial for each ad and minute of the click. Partials go out every second, and before each checkpoint, a saved copy of the job.

Then partials are summed, keyed by ad. The viral ad's summing slot gets 15 partials a second, not 2,000 clicks.

A long-tail ad seen once in a second sends one partial for one click. So every summing slot gets about (18,000 + 15) / 15 = 1,200 records a second.

Partials stay inside the job and write nothing. In Flink SQL this is mini-batch, which buffers rows briefly before updating state, with two-phase aggregation.

The pre-count has no published cost. We keep 0.5 ms a click for the whole job; the load test that replaces it must run this two-stage job.

The stage cards spread load evenly, which is true here only because of the two stages.

Step through the figure to watch the viral ad split.

The trade-off:

Two stages add a step and a partial per ad per second. In exchange, no ad can hold the job at one slot.

One viral ad, split across every slot

1. Single-stage, a step keyed by ad puts every click of the viral ad on one slot. That slot takes 2,000 + 1,200 = 3,200 clicks a second.
2. Its backpressure slows every slot upstream, so the whole job runs at its pace. More slots never split one key.
3. Two-stage: first, dedup is keyed by click id, which is random. Every slot gets an even share, about 1,333 clicks a second.
4. In the same slot, each click adds 1 to a partial for its ad and minute. The viral ad now has 15 partials, one in each slot.
5. The viral ad's summing slot gets 15 partials a second, not 2,000 clicks. Every summing slot gets about 1,200 records a second.
6. At the event, the viral ad alone sends 6,000 clicks a second to one slot, 3 times its 2,000, however many slots run. Two-stage sends it one partial a second from each slot.

Check in: Why does adding slots not fix a viral ad in a single-stage job?

One click through every state

We follow one click from the link to the bill. Time is not in the simulation, so this is by hand.

Emitted: when the ad server shows the ad, it signs the impression id, the click id, the ad id and the time into the link.

Received: a receiver checks the signature and drops an unsigned or expired click. It stamps the event time, when the click happened, from its own clock.

Made durable: the receiver puts the click on the stream, keyed by click id, waits for the acknowledgement, then answers 302.

Deduplicated: the aggregator keeps each click id for 10 minutes of event time and drops a second copy.

Filtered: known bots, data-centre addresses, and addresses a per-slot sketch finds too busy are flagged as invalid and counted apart. Nothing is deleted.

Repeat clicks by one user on one ad are found later, by the nightly recount.

Pre-counted, then windowed: partials are summed per ad into a 1-minute window of event time.

Aggregated: the window fires once the minute is over, plus 10 seconds for clicks that arrive out of order.

Stored: its total goes to the counts table by a conditional put, written only if no newer version is already stored.

The click itself goes into a Parquet file under its event hour, committed at the checkpoint, every 60 seconds.

Queried: a dashboard reads one ad's minute rows. Reconciled: a nightly job recounts the day from the archive. Expired: rows after 30 days, raw clicks after 13 months.

The trade-off:

Every state keeps something: a signature, a stream record, an id in memory, a file. That is the price of a count that billing can trust.

Check in: Follow one click from the link to the bill. Say where it lives at each step.

Counted once, in four parts

The interviewer asks: How do you count each click exactly once?

Copies come from three places: double clicks, browser retries and producer retries. A restart re-reads the stream too. Each needs its own answer.

Identity: a billable click is one per click id. A second click on the same impression is not billed.

Dedup catches copies that are new stream records: double clicks, browser retries, and a put that timed out but landed.

It keeps each id for 10 minutes, so it is an approximation. A re-click after 10 minutes is counted twice in the stream.

The nightly recount counts distinct click ids, so it corrects that. The recount is the billable number.

Ten minutes of ids is 20,000 x 600 = 12 million at the peak, and 36 million at the event. With their timers that is about 1.4 and 4.3 GB.

Checkpoints handle restarts. Every 60 seconds Flink saves its state, dedup ids included, together with the stream positions.

Each checkpoint uploads only what changed: about 144 MB at the peak. A restore rolls state and positions back together, so the re-read records are not copies.

Idempotent sinks, safe to repeat, finish it. The counts table gets a conditional put of the total with a version: a repeat writes the same value, and an older total is refused.

The stock Flink DynamoDB sink cannot do this: its batch writes take no condition. So the sink is a custom writer using PutItem.

The raw archive commits files only at a checkpoint. So the total is correct even when every step retries: effectively once.

The trade-off:

A refused conditional write still uses write capacity. They happen only on out-of-order retries, which are rare.

Check in: A Flink task crashes and restarts from its checkpoint. Why are the re-read clicks not counted twice?

Event time, watermarks and late clicks

Clicks arrive out of order and sometimes late. A minute needs a rule for when it is done, and for what comes after.

Windows use event time, the receiver's stamp, not the time the aggregator sees the click. A late click still lands in the minute it happened.

A watermark says how far event time has got: the newest time seen minus 10 seconds, the minimum across shards.

Where lateness comes from: a receiver retrying a stalled or throttled put keeps the original time. A throttled stream can delay clicks by minutes.

A shard idle for 1 minute is ignored. The timeout runs on processing time, so a stalled shard can be ignored too, and the watermark passes its clicks.

Clock skew between receivers is only milliseconds.

A window fires at its end plus 10 seconds. Clicks up to 5 minutes late update it; ours fires again at most once a minute, so at most 5 late puts.

Later than 5 minutes, a click goes to a late-clicks table that the nightly recount folds in.

Freshness: at the peak, 150,000 rows land as the minute closes. At 3,600 a second they take 42 seconds, so a minute is in within about 1 minute.

Windows are 1-minute tumbling windows, so each click is in exactly one, which billing needs. "Last 5 minutes" is 5 rows summed at query time.

Step through the figure to follow a late click.

The trade-off:

Waiting for late clicks costs freshness and some extra puts. If 2% of windows refire 5 times, that is 250 puts a second: 2,750 of 3,600.

Event time, the watermark and a late click

1. A click from minute 10, processed at minute 20, still counts in minute 10. By processing time it would land in the wrong minute and bill the wrong hour.
2. A window fires when the watermark, 10 seconds behind the newest click, passes its end. A click up to 5 minutes later fires it again.
3. A receiver retries a throttled put for 2.5 minutes. The click keeps its time, 10:00:30, so it arrives late but still counts in 10:00.

Check in: A click from 10:00:30 reaches the aggregator at 10:03. Which minute does it count in, and why is it late?

A sports final, with nothing raised

The interviewer asks: A sports final: 3 times the clicks for 30 minutes. What breaks, and what does it cost?

Stage 4 is Stage 3's design at the event's 60,000 clicks a second.

Rows do not triple: the extra clicks go mostly to the final's few ads. Distinct ads rise to about 180,000 a minute: 3,000 rows a second, 0.05 a click.

Two cards saturate: the Click stream and the Aggregator (Flink). "First to saturate" names one "and 1 more".

Each passes on only its ceiling. The stream passes 32,000 clicks and the aggregator 30,000, so the table gets 30,000 x 0.05 = 1,500 rows a second.

Minute counts reads 1,500 of 3,600 WCU, 41.7%. The table is not a wall at the event.

The latency tile shows "≥ 73ms", a lower limit: the saturated stream is on the path. The cost tile shows $55k/mo, $37,257.17 of it egress.

Every click still gets an answer, even one the stream refuses, so egress counts all 60,000.

The one that loses money is the stream. It is before the acknowledgement: 28,000 clicks a second are refused.

Receivers retry them with their original times, so users wait, some clicks arrive minutes late, and some are lost.

The aggregator is after the stream, which keeps 7 days. It only delays counts.

The trade-off:

A wall before the acknowledgement loses clicks. A wall after it only makes them late.

Stage 4: A sports final, with nothing raised

• Load: 60,000 req/sec
• First to saturate: Click stream
• Estimated cost: $55k/mo

• Latency past saturation is a lower bound, not an estimate. Click stream is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Click stream: Write-bound at 100% of 32,000 records/s (32 shards x 1,000 records/s at 0.5 KB per record).
• Click stream: Aggregator (Flink) drains 30.0k/s of the 32.0k/s offered, so the backlog here grows by 2.0k/s - about 7.2M messages after an hour.
• Aggregator (Flink): Fargate 1vCPU-4GB ×15 at 100% of a fixed 15-task ceiling.

Check in: Which wall loses clicks, and which only delays them?

Raise what the click waits for, let the counter lag

The event is scheduled, so capacity is raised the day before. The question is which parts to raise.

The stream goes from 32 to 96 shards in two calls, as one call can at most double. That is 32 to 64, then 64 to 96, a few minutes each.

Minute counts goes from 3,600 to 5,400 WCU: 3,000 rows a second plus up to 300 late puts is 3,300, 61.1%. The minute burst takes 33 seconds.

The aggregator stays at 15 slots, on purpose. Resizing a Flink job means a restart from a savepoint, a snapshot taken by hand.

Stage 5 runs at 60,000. Only the Aggregator (Flink) is named: that is the design working. The stream at 62.5% takes every click.

While the aggregator lags, Minute counts reads 1,500 rows of 5,400 WCU, 27.8%.

The latency tile shows 73ms and is bounded: the aggregator is behind the stream, off the path.

The cost tile shows $58k/mo, as if the event ran all month. Its real price is about $144 for the day.

The totals are correct, only late: windows use event time, so each click lands in its own minute.

Two stages keep the viral ad spread across the 15 slots while they catch up.

The trade-off:

Letting the stateful part lag costs fresh dashboards for about 2 hours. It saves a rescale during the event, and the totals do not change.

Stage 5: Raise what the click waits for, let the counter lag

• Load: 60,000 req/sec
• First to saturate: Aggregator (Flink)
• Estimated cost: $58k/mo

• Click stream: Aggregator (Flink) drains 30.0k/s of the 60.0k/s offered, so the backlog here grows by 30.0k/s - about 108.0M messages after an hour.
• Aggregator (Flink): Fargate 1vCPU-4GB ×15 at 100% of a fixed 15-task ceiling.

Check in: Why not let Managed Flink auto-scale for the event?

How far behind, and for how long

Time is not in the simulation, so we work the lag by hand.

For 30 minutes the aggregator processes 30,000 of 60,000 clicks a second. The backlog grows by 30,000 x 1,800 = 54 million clicks.

At the event's end it is 15 minutes behind, and 30 minutes behind another 30 minutes later.

After minute 30 the backlog shrinks by 10,000 a second. But the aggregator is still in the event's minutes, at half an event-second a second, so it falls further behind until minute 60.

Then it has 30,000 - 20,000 = 10,000 a second spare: 54,000,000 / 10,000 = 5,400 seconds. It catches up 90 minutes after the event.

At 100% for 2 hours, aligned checkpoints queue behind full buffers. So the catch-up uses unaligned checkpoints, which let the checkpoint marker pass queued data.

The raw archive files clicks under their event hour. The default would file late clicks under the hour they were processed.

Afterwards, shrinking also takes two calls, as one call can at most halve: 96 to 48, then 48 to 32.

Step through the figure to follow the lag and the catch-up.

The trade-off:

The stream holds the backlog for 7 days, so a lag of 30 minutes costs nothing but freshness.

The aggregator falls behind, then catches up

1. The sports final runs 30 minutes at 60,000 clicks a second. The aggregator processes 30,000.
2. The backlog grows by 30,000 a second for 1,800 seconds: 54 million clicks, held in the stream.
3. At the event's end the aggregator is 15 minutes behind. Still working through the event's clicks, it is 30 minutes behind at minute 60.
4. After the event, 10,000 clicks a second are spare. 54 million / 10,000 = 5,400 seconds: caught up 90 minutes after the event.

Check in: The backlog starts shrinking at minute 30. Why does the lag keep growing until minute 60?

Rollups, and which store for which question

The interviewer asks: An advertiser opens a 30-day chart. And why DynamoDB, and why Kinesis?

A minute row suits the last hour, not the last month. The store has to fit the question.

A query is billed on the bytes it reads. 30 days of minute rows is 43,200 rows, about 2.6 MB: 316.5 read units for one chart.

So the job also writes hourly and daily rollups. 30 days of hourly rows is 720 rows: 5.5 read units.

About 400,000 hourly rows land as each hour closes, and drain in the spare WCU in 6 to 8 minutes. Kept 13 months, they are 558 GiB.

DynamoDB answers one ad over a time range. Slicing by country, device and campaign needs an analytics store, such as Druid, Pinot or ClickHouse.

That is a second store for a second question, fed from the stream. It does not replace the counts table, and it is not drawn.

Kafka on Amazon MSK instead of Kinesis: you choose partitions, keep data longer and get a transactional sink, but you run brokers.

At 9.8 MiB a second either fits. A rough MSK bill is $1,318.80 a month, against the stream's $1,601.28.

The trade-off:

Rollups cost one more write per ad per hour. In exchange, a month-long chart reads about 58 times fewer units.

Check in: Why does a 30-day chart need hourly rows?

The API, the tables and the dashboards

The interviewer asks: Walk me through the API and the data model.

None of these parts breaks under load, so no stage is about them. An interviewer asks about each one, so we answer each in a line or two.

Click: GET /c with the signed payload returns a 302 to the advertiser.

Counts: GET /ads/{adId}/clicks with from, to and grain=minute reads minute rows. Top ads: GET /top with a minute and n=100 reads one row.

The counts table: partition key ad_id, sort key minute, with valid, invalid and version. Rows expire after 30 days.

Top 100: each slot keeps exact counts for its own ads, so its local top 100 is exact. One step merges 15 x 100 = 1,500 candidates a minute.

Dashboards: 2,500 advertisers online x 6 panels / 30 seconds = 500 reads a second, 250 read units of the table's 400. They are not drawn.

Acknowledge first, or redirect first? Redirecting first with a 100 ms buffer loses about 690 x 0.1 = 69 clicks when a receiver task dies.

For billing we wait for the acknowledgement.

Fraud: the stream runs only rules that need no new key. A per-slot sketch of addresses sends its heaviest to one step a minute, about 1 MiB a slot.

Models over the raw archive find the rest later, and credit it back at reconciliation.

The trade-off:

Waiting for the acknowledgement adds the put to every redirect. In exchange, a click the user saw succeed is never lost.

Check in: Is the top 100 exact or approximate, and why?

Replay, reconcile, retain

The interviewer asks: A bug wrote wrong totals for two days. What do you do?

A streaming count is only as good as its last deploy. We need a way to redo it, and a way to check it.

This is Kappa: one streaming job serves the counts. Lambda would run a batch path beside it and merge both, two copies of the logic.

The stream keeps 7 days. Deploy the fixed job as a new consumer from a time before the bug, writing to a new table version.

It starts with empty dedup state, so start it at least 10 minutes before the first minute you keep. Switch dashboards when it catches up.

Two days is 1.728 billion clicks. At 15 slots it gains 20,000 a second on the live stream: 24 hours.

With the fleet sized for the event, which you do in the challenge, it takes about 7.3 hours.

Older than 7 days: run the same job in batch mode over the raw archive. Never restore state written by the buggy job.

Reconciliation: nightly, per ad and day, count distinct valid click ids in the archive and compare with the minute rows.

The stream can only be higher, from a re-click after 10 minutes, or lower, from a click over 5 minutes late. The recount is what billing uses.

Over 0.1% of difference opens an investigation, our tolerance. Scanning 3 columns, about 30% of a day's 82.4 GiB, costs about $0.12 a day.

The trade-off:

The recount is a check, not a second serving path. It costs a nightly scan and catches what the stream approximates.

Check in: Why start a replay at least 10 minutes before the first minute you keep?

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does an advertiser see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

A receiver dies: unacknowledged clicks fail, and the browser may retry with the same click id, which dedup drops.

The aggregator dies: it restores the last checkpoint, at most 60 seconds old, and re-reads from the saved positions. Nothing counts twice.

The counts table throttles: the sink backs up and the job slows, and the stream holds the backlog.

Cost: an average month is $15,024.06 over 25.92 billion clicks, $579.63 per billion.

Counting by increment is $930.37 per billion. Egress is $331.74 of each, so the $350.74 difference is all in the parts.

Corrected by hand, with Managed Flink, a connection a click and AWS's retention price: $16,589.53, $640.03 per billion.

The event day: extra shards $53.76, extra WCU $28.08 and extra slots $32.16 for 24 hours. With the 30 minutes' load, egress included, about $144, or $173 on Managed Flink.

Say it out loud: "Clicks go to a stream keyed by click id, and a two-stage count runs by event time. Each total is a versioned put, and billing uses a nightly recount."

Minutes 0 to 5: requirements. Per-ad minute counts, exact for billing, freshness, replay.

Minutes 5 to 10: the numbers. 864 million a day, 10,000 average, 20,000 peak, 512 bytes a click.

Minutes 10 to 15: an increment per click. Its cost, and why a retry double counts.

Minutes 15 to 22: the viral ad. 1,000 WCU an item, aggregate first, a stream keyed by click id, two stages.

Minutes 22 to 30: exactly once. Click id, dedup state, checkpoints and idempotent puts.

Minutes 30 to 37: event time, watermarks and 5 minutes of lateness. Tumbling windows, rollups and the top 100.

Minutes 37 to 42: the event. Raise the stream, let the aggregator lag; event time keeps the totals correct.

Minutes 42 to 45: reconciliation, replay, retention and cost per billion clicks.

The trade-off:

Each answer has a price. A stream, dedup state and a nightly recount are more parts to run than a counter, and they buy a bill you can defend.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.