Design a gaming leaderboard

Do the numbers before the boxes

An interviewer gives players, not loads. A leaderboard has one load that sizes everything: runs finished a second.

A 45-minute version of this answer is at the end.

The game is a single-player mobile arcade game. A run is one game, about 3 minutes long. The phone plays it and sends the result.

Points add up over a calendar month in UTC, and the board resets on the 1st.

We assume 25M monthly players in one AWS Region, us-east-1. Every count on this page is our estimate unless sourced.

About 7.2M of them play on a given day, 29%, and each plays about 12 runs: 86.4M runs a day.

86.4M / 86,400 s = 1,000 runs a second on average. The evening peak is 2.5 times that: 2,500 runs finished a second.

The game launches with 2M monthly players and the same habits: 2,500 x 2M / 25M = 200 runs a second at the peak.

A player's rank is their place on the board: 1 plus the number of players with more points.

After every run the results screen shows the player's monthly total, their rank, and the two players ahead and the two behind.

About 1 run in 4 is followed by a board open: the top N, the first N players, here 10, and the player's neighbourhood.

The median run's player has about 30% of the board ahead of them. Active players rank high and play most.

The biggest mobile games have hundreds of millions of monthly players. This page sizes a hit, not a giant; the last beat covers 10 times.

Check in: Close the page and derive the peak rate, at launch and at 25M players.

Reading the result tiles

Each stage has a live canvas and four result tiles. Four things hold on every stage.

Load: the buttons count runs finished a second, and the tile labels them req/sec.

A card turns red at 90% of its limit, and a database card also warns over 80%. A part saturates past 100%, and only then does the result name it. Otherwise the last tile notes the busiest part.

Latency is server time on the slowest path. Past saturation it is a floor, which is why it prints "≥".

Cost: each card bills its peak load all month; the bill beat uses the average month. Where a card reads differently from AWS, the stage that shows it says so.

Check in: The latency tile on Stage 1 reads 3ms. Is that what a player waits for the results screen?

A leaderboard in SQL, at launch

Stage 1 is the launch: 2M monthly players, 200 runs finished a second at the evening peak.

The phone posts each result through an ALB, AWS's application load balancer, to a stateless Score API.

The API runs on Fargate, AWS's managed containers. Each task is one container; here a task has 1 vCPU, one virtual processor, and 2 GB.

The API first checks the run token: a ticket the server signed when the run began. Its run id is single use.

Then one transaction, a group of writes that commit together, stores the result: insert the run row, and add its points to the monthly total.

The Scores database is Aurora, AWS's managed PostgreSQL: a writer and two readers, copies that serve queries. Each sits in its own availability zone, a separate data centre in one Region.

The rank is a count on a reader: how many totals this month have more points than mine. An index on month and points serves it.

An index is a sorted copy of those columns, so a query reads only the entries it needs. The neighbours are one short read around the player, and the top 10 one more.

A count reads every index entry ahead of the player. The median player has 30% of 2M ahead: 600,000 entries. At about 50 ns an entry (our estimate), that is 30 ms of CPU.

The database card has one cost for every query. So on Stages 1 and 2 this page counts in units of 0.1 ms of CPU. In an interview, say milliseconds.

A count is 300 units. Each run brings 1.25 counts: one for the results screen, and a quarter for the board open that follows 1 run in 4.

It also brings 1.5 short reads of 1 unit: 1.25 neighbour reads the same way, and 0.25 top-10 reads. With 2 rows on the writer, a run is 1.25 x 300 + 1.5 + 2 = 378.5 units.

Each vCPU serves 10,000 units a second, and each instance has 8 vCPUs: 80,000 units a second, 160,000 across the two readers.

At 200 runs a second that is 75,700 units. The readers carry 75,300 of 160,000: 47.1%. The writer carries 400 of 80,000: 0.5%.

Nothing saturates; the note reads "Busiest: Scores database at 47%". The cost tile shows $3,071/mo, and the three database instances are $2,920.32 of it.

Data out to the internet, called egress, is $35.49 of the tile: the result's answer, about 1 KiB a run.

The latency tile shows 3ms: 1.5 ms at the load balancer, 1 ms in the API and 0.1 ms on the database, 2.6 ms. The database card prints one query's time for the whole run. Really, the count alone takes 30 ms, so the results screen takes about 35 ms of server time.

Press 80, the daily average: the readers are at 18.8%, and the cost tile shows $3,027/mo.

The trade-off:

Already at launch, 99.5% of the database's work is one query, the count. Two big readers carry it today, with one familiar query.

Stage 1: A leaderboard in SQL, at launch

• Load: 200 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $3,071/mo

Check in: The players grow 12.5 times. What happens to one count's cost?

The same SQL at 25M players

The interviewer asks: The game is a hit: 25 million monthly players.

Nothing in the design changes. The players are 12.5 times as many, so two things grow at once.

Runs grow 12.5 times: 2,500 a second at the evening peak.

And each count reads 12.5 times as many entries: 30% of 25M is 7.5M, about 375 ms of CPU.

A count is now 3,750 units. A run is 4,691: 1.25 x 3,750 + 1.5 = 4,689 on the readers, and 2 on the writer.

2,500 x 4,691 = 11,727,500 units a second. The readers get 11,722,500 of their 160,000: 73 times their capacity.

The result names the Scores database. Its warning begins "Readers 7327% used".

Nothing is refused. Counts queue on the readers and time out, and the API's retries add more load.

The latency tile shows "≥ 3ms", a floor: the database card still counts one 0.1 ms query, at most 4 times that when full. Really, each count takes 375 ms before it waits at all.

The cost tile shows $3,971/mo. Only the load balancer, the API and the egress grew, and the readers are 73 times short.

Press 1,000, the daily average: the readers are still 29 times short.

How much database would it take? At the 80% line (our headroom choice, the same as the card's warning), the readers need 1,465 vCPUs.

The largest class in the card's family, r6g, has 64 vCPUs, and Aurora takes at most 15 readers: 960.

It takes 8 readers of AWS's largest Aurora class, 192 vCPUs each. With the writer, that is about $231,600 a month.

Even at a fifth of our count estimate, it takes 10 readers of 32 vCPUs and a writer: about $42,800 a month.

In practice it is worse. Totals change 2,500 times a second, so the count also checks table rows, and every update moves an index entry.

The trade-off:

A rank by counting costs more with every player, twice: more counts, and each one longer.

Stage 2: The same SQL at 25M players

• Load: 2,500 req/sec
• First to saturate: Scores database
• Estimated cost: $3,971/mo

• Latency past saturation is a lower bound, not an estimate. Scores database is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Scores database: Readers 7327% used: 11,722,499 reads/s against 160,000 across 2 readers (the writer takes no reads while readers exist).

Check in: Players grew 12.5 times. Why did the readers' load grow about 156 times?

A count against a sorted set

The wall is the requirement: an exact rank for every player, live. A candidate usually tries three things first.

Cache each player's rank for a minute: still a count a player a minute, and the rank is a minute old.

Bucket counts: keep a count of players in each of 10,000 score buckets, about 2,500 players each.

A rank is then the players in the buckets ahead, plus a 0.1 ms count inside one bucket. It is exact and cheap to read.

But it hand-builds an index of counter rows that many runs update at once. So the counters batch once a second, and ranks lag a second.

Ranks rebuilt every minute, by one sort of 25M rows into a table: cheap, but a minute old.

A rank by counting is O(rank): its cost grows in step with the number of players ahead.

PostgreSQL's index, a B-tree, stores no counts under each branch, so it cannot do better.

Valkey is an in-memory key-value store, which AWS runs as ElastiCache.

Its sorted set keeps members ordered by a score, and answers a rank in O(log N). The cost grows very slowly with N, the number of players.

The sorted set is a skiplist: a sorted linked list with express lanes. Each link stores how many members it skips.

For 25M players it is about 12 levels high. A rank sums the skips in 40 to 50 steps, about 5 µs (our estimate).

Step through the figure to compare one count with one rank.

The trade-off:

The rank moves out of the database into a structure built for it. In exchange, there is a second store to keep in step with the truth.

A count against a sorted-set rank

1. A SQL rank counts every total ahead of the player. At launch the median player has 600,000 ahead: 30 ms.
2. At 25M players the same player has 7.5M ahead. Each count takes 375 ms, and there are 12.5 times as many.
3. A skiplist keeps members in score order, with express lanes. Each link stores its span: how many members it skips.
4. A rank sums the spans along the path. For 25M players that is 40 to 50 steps, about 5 µs.

Check in: Why can a skiplist give a rank without counting?

One run, from result to rank

After every run the phone needs the new total, the rank and the neighbours, in one reply, within a few ms of server time.

The database stays the truth: the same transaction, 2 rows a run. It returns the new total.

The board is one sorted set a month, with every player who played that month, scored by their total.

The API writes the total with ZADD GT: set the score only if it is greater than the stored one. A retry or a late reply can never lower a player.

Then it reads the rank, then the two players either way, then their display names. Each step needs the reply to the one before.

A pipeline sends commands without waiting for replies, so it cannot do that.

So the four commands run as one Valkey Function: a script stored on the server and called by name.

One call, one round trip, and nothing else runs on the board in between. It also returns the board's own score.

That differs from the total only when GT refused a stale write.

The board runs on a primary and a replica, a copy in another zone. The results-screen call runs on the primary.

A board open is a second function, run on the replica: the rank, the neighbours and their names.

The top 10 is the same for everyone, so each API task caches it for 1 s.

A friends view follows about 1 run in 4 and also reads the replica. Two commands fetch up to 200 friends' scores and names, and the API sorts them.

So a run costs 4 commands on the primary, and 0.25 x 3 = 0.75 for board opens on the replica. That makes 4.75, the figure on the canvas.

Friends views add 0.25 x 2 = 0.5 a run on the replica, by hand: 5.25 commands a run in all.

Step through the figure to follow one run.

The trade-off:

Every run writes twice: the database, then the board. In exchange, the results screen is one transaction and one round trip.

One run, from result to rank

1. The phone posts its result with the run token. The API checks the token's signature and that its run id is new.
2. One transaction inserts the run row and adds the points to the monthly total. It returns the new total, T.
3. One function call on the primary: ZADD GT writes T, then reads the rank, the two players either way, and their names.
4. One round trip back. The reply carries the total, the rank and the neighbours: about 5 ms of server time.
5. Board opens and friends views read the replica. The top 10 comes from each API task's 1 s cache.

Check in: Why write the stored total with ZADD GT, and not add the run's points with ZINCRBY?

Ranks from a sorted set in Valkey

Stage 3 keeps 25M players and 2,500 runs a second. The rank moves to the sorted set.

The database keeps the same transaction, 2 rows a run. The rank count and its index are gone, so the total's update moves no index entry.

Two rows a run is a design choice, and an interviewer may push on it. The run row is the run's receipt: support can read it back at once.

That answers a player who says a run did not count. One transaction gives retry safety, through the run id, and the receipt together, with no second system to feed.

The database is now a writer and one reader, each a db.r6g.large with 2 vCPUs. At our 0.1 ms a row, 2,500 x 2 = 5,000 rows a second is 25% of the writer's 20,000.

The reader sits in another zone as the failover target. If the writer fails, it takes over in about 30 s, and phones retry their results.

Leaderboard is Valkey: a primary and a replica, each a cache.r7g.large. The card counts 2,500 x 4.75 = 11,875 commands a second, 7.4% of its 160,000 a second.

160,000 is AWS's measured figure for one Graviton2 core. Graviton is AWS's ARM processor. This node is a newer Graviton3, which has no published figure, so the card uses the Graviton2 figure as a safe minimum.

By hand, Valkey runs commands on one command thread. Per run the primary does 4 commands at 5 µs, one round trip at 5 µs and 3 µs of function work: 28 µs.

So 2,500 runs a second take 70 ms of each second: 7.0%. At 12,000 they take 33.6%, or 63.6% if a command and a round trip cost 10 µs.

On the replica a board open costs 23 µs: 3 commands, a round trip and the function. A friends view costs about 55 µs, and applying each write 5 µs. At 12,000: 0.25 x 23 + 0.25 x 55 + 5 = 24.5 µs a run, 29.4%. At 10 µs a step and 100 µs a friends view, it is 57.9%.

Nothing saturates, and the note names the Score API as busiest. The cost tile shows $1,790/mo, 45% of Stage 2's bill; the rank chain lists its lines.

The latency tile shows 3ms: 1.5 ms at the load balancer, 1 ms in the API and Valkey's fixed 0.2 ms, 2.7 ms. The tile treats the API's two calls as parallel.

Really, a run is a chain: the commit, a few ms while Aurora waits on storage, then the function call, about 0.3 ms. That is about 5 ms, before the phone's 50 to 150 ms.

Press 1,000, the average: the writer is at 10%, and the cost tile shows $1,178/mo.

Press 12,000: the season's last evening. The result names the Scores database, and the warnings show its load. The Valkey card reads 35.6%, and every hand figure in this beat is under 70%.

Fixing the database is your turn, at the end of the page, after the quiz.

The trade-off:

Two stores to keep in step. In exchange, the database answers "what happened" and the sorted set "where am I", each with what it is fast at.

Stage 3: Ranks from a sorted set in Valkey

• Load: 2,500 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $1,790/mo

Check in: An interviewer asks why a run writes 2 rows, not 1. What do you say?

Ties, retries and a lost board

Two players with equal points need a fixed order. A retried result must not count twice, and the board itself can be lost.

Valkey orders equal scores by the member's bytes, which is arbitrary for players. So the score is a composite: points, then when the total was reached.

Score = points x 2^22 + (2^22 - 1 - t), with t the seconds since the month began. A month has at most 2,678,400 s, under 2^22 = 4,194,304.

Equal points: the earlier player ranks higher. Points only grow, so the composite only grows, and ZADD GT stays correct.

Valkey stores a score as a double, a 64-bit floating-point number. It holds whole numbers exactly up to 2^53, so points can reach 2^31.

A retried result finds its run row by run id and returns the stored total. Writing that total with ZADD GT again changes nothing.

If the API crashes between the commit and the board write, the player's next run writes a higher total. A nightly job repairs the rest.

Memory, by hand, in binary GiB: about 90 B a member, so the month's board is 2.10 GiB.

Last month's board stays a week while the new one fills, another 2.10 GiB at most. Display names add 1.86 GiB.

That is at most 6.06 GiB: 62% of the 9.80 GiB usable once AWS reserves its default 25%.

If both Valkey nodes are lost, results still commit. A node is replaced first, about 10 to 20 minutes (our estimate; AWS publishes no figure).

Then the board reloads from the 25M monthly totals in 1 to 3 minutes. Ranks say "updating" for about 15 to 25 minutes in all.

Never fall back to the SQL count meanwhile: that is Stage 2's wall, at the worst moment.

The rebuild can run during live writes, because its ZADD GT never lowers a newer score.

The trade-off:

A composite score and a stored "reached at" time. In exchange, ties have one order, retries are harmless, and the board can always be rebuilt.

Check in: Two players both reach 9,000 points. Who ranks higher, and how does the score know?

The rank chain on one screen

Skip on a first read; come back before the interview. Every number on Stages 1 to 3 comes from the one before it.

Load balancer: an LCU is its billing unit; one covers 25 new connections a second. The card assumes 0.6 new connections a call.

At 200 runs a second: 200 x 0.6 / 25 = 4.8 LCU. At $5.76 an LCU-month ($0.008 an hour): $16.20 + 4.8 x $5.76 = $43.85.

API: 1 ms of CPU a run, so a 1-vCPU task serves 1,000 a second, 700 at its 70% target.

200 / 700 is under one task, so the API sits at its floor of 2 tasks: $71.09.

Database: 200 x 378.5 = 75,700 units. Readers 75,300 of 160,000, 47.1%; writer 400 of 80,000, 0.5%. Three instances, $2,920.32.

Answers: 200 x 1 KiB a second x 2,592,000 s = 494.38 GiB a month, which AWS bills as GB.

Egress: the first 100 GB are free, then $0.09 a GB: $35.49.

Stage 1: these lines, each rounded to the cent, come to the tile's $3,071 a month at the peak.

Stage 2 at 2,500: 2,500 x 0.6 / 25 = 60 LCU, $16.20 + 60 x $5.76 = $361.80. API: 2,500 / 700 = 3.6, so 4 tasks, $142.19.

Database: 2,500 x 4,691 = 11,727,500 units. Readers 11,722,500 of 160,000, saturated. Still $2,920.32.

Answers: 2,500 x 1 KiB a second is 6,179.81 GiB a month. Egress: $547.18.

Stage 2: $3,971.49, the tile's $3,971.

Stage 3 at 2,500: the load balancer, the API and the answers' $547.18 of egress are unchanged.

Rows: 2,500 x 2 = 5,000 a second, 25% of the writer's 20,000. Two instances at $243.36 each.

Commands on the card: 2,500 x 4.75 = 11,875 a second, 7.4% of 160,000 commands a second. Two nodes at $0.1752 an hour for 720 hours: $252.29. Friends views add 0.5 a run by hand.

Stage 3: $1,790.18 a month at the peak, the tile's $1,790.

Check in: Close the page and rebuild Stage 2's reader load from 2,500 runs.

Replay every run

The interviewer asks: Someone posted 2 billion points from a modified app. How do you stop cheating?

The phone computes the score, so the server cannot trust it. Anti-cheat is everything that stops a modified app posting a false score.

The safe first answer is to replay every run: re-run the game on a server from the phone's recorded inputs, and check the points.

The game must be deterministic: the server replaying a run must reproduce the phone exactly, from the same seed, version and inputs.

The seed is the random start the token carries. Compilers and maths libraries round floating point differently, so the game uses fixed-point physics, maths on whole numbers.

That is a choice made before launch, and every build replays a set of recorded runs on the server before release.

Every phone uploads its run's input log, about 20 KiB, to S3, AWS's object storage. A presigned URL in the reply allows that one upload.

Each upload sends an event to SQS, AWS's message queue, and a fleet of workers replays the run.

A 3-minute run replays at about 90 times real time: 2,000 ms of CPU (our estimate, for a physics-heavy game).

A worker task replays 2 a second, 1.4 at its 70% target. 2,500 / 1.4 = 1,786 tasks, 7,144 vCPUs. ECS, AWS's container service, allows 5,000 tasks a service, but AWS's Fargate vCPU quota must be raised.

Nothing saturates; the note reads "Busiest: Replay workers at 70%". The latency tile still shows 3ms: the upload and the replay are async, so no player waits on them.

The cost tile shows $296k/mo: replay workers $253,943.48, uploads $32,400.00, the queue $7,775.60, plus Stage 3's $1,790.18.

The workers also write 2,500 verdicts a second, and release about 5 held-back runs a second, which the hold-back beat explains. Neither is on the canvas.

So the database writer really takes 5,000 + 2,505 = 7,505 rows a second: 37.5%, not 25%.

Press 1,000, the average: 715 replay tasks, and the cost tile shows $119k/mo.

By hand, a replay of 200 ms, as in a simple 2D game, needs 179 tasks, and Stage 4 comes to about $67,400 a month. The uploads would then be the biggest line.

The trade-off:

Replaying everything needs a fleet sized to all play: 7,144 vCPUs, 86% of the bill.

Stage 4: Replay every run

• Load: 2,500 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $296k/mo

Check in: Why does Stage 4's latency tile not move, with 7,144 vCPUs of replay behind it?

Cheap checks for every run

A replay for every run costs a fleet. Most runs can be checked for almost nothing, inside the API's 1 ms.

At the start of a session the app asks for run tokens. Attestation is the platform vouching for the app and the device.

On iOS, App Attest attests a key once per install, and every result carries an assertion, a signature made with that key.

On Android, the session's first call gets a Play Integrity verdict, and results are signed with a Keystore key registered then. Keystore holds it in the phone's secure hardware.

It shows the app and the device passed integrity checks when the session began. A hooked app on a passing device can still sign a false result.

The points cap, the elapsed-time check and replay are what catch that.

Tokens come 3 at a time, numbered 1 to 3, with one issue time. The next 3 ride on each result's reply, so a phone can play 3 runs offline.

The k-th result must show time since issue of at least the first k runs' claimed lengths, minus 5 s of slack. So no run is posted faster than real time.

A points-a-second cap rejects an impossible run, like the 2 billion, before anything is stored.

The trade-off:

An attestation per install or session and a signature per result. In exchange, a modified app cannot post faster than real time or past the cap.

Check in: A phone posts three 3-minute results together, 4 minutes after its tokens were issued. Which pass?

Replay 2%, and hold contenders back

Checks catch the impossible, not the merely too good. Some runs must still be replayed, and some must be replayed before anyone sees them.

After a result arrives, the API picks runs to replay, so a cheater cannot know which. Only those replies ask for the log.

It picks 1% at random, and every run the checks flag, about 0.8%.

A confirmed failed replay voids the player's whole month and flags the account. Every later run of a flagged player is replayed.

A mismatch alone never voids a month: it goes to review, and every build is tested for determinism before release. A voided player can appeal, and support replays the archived log.

So a cheater posting 100 fake runs is caught with probability 1 - 0.99^100 = 63%, and 95% by 300.

It also picks every contender: a run whose new total would put its player in the top 1,000, about 0.2% of runs. The API compares against a cached 1,000th score.

A contender's run commits as usual, but its points also go into pending points, a column on the monthly total. The board carries points minus pending points.

So the run stays off the public board until its replay passes. The player sees a provisional rank on their own screen only.

On a pass, the worker moves the points out of pending in one transaction, then writes the board with ZADD GT. Points minus pending points only grows, so GT stays correct.

No log within 10 minutes drops only that run, since phones lose signal. Three missing logs in a month void it.

On the month's first day the public top 1,000 is labelled provisional; the last beat explains why.

Step through the figure to follow one contender.

The trade-off:

A contender waits for a replay before their public place. In exchange, no unverified run reaches the public top 1,000 after the month's first day.

Hold a contender, then release it

1. A run's new total would pass the 1,000th player's score. The run is a contender.
2. The run commits as usual, and its points also go into pending points. The board carries points minus pending points.
3. The player sees a provisional rank on their own screen. Everyone else sees the board as it was.
4. The phone uploads its input log. A worker replays the run from the same seed and version, in about 2 s.
5. A pass clears pending in one transaction, then writes the board. A confirmed fail voids the month; a lone mismatch goes to review.
6. No log within 10 minutes drops only that run. Three missing logs in a month void it.

Check in: Why hold a contender's points off the board, instead of showing them as "verifying"?

Check every run, replay 2%

Stage 5 carries the same 2,500 runs a second. Every run gets the cheap checks, and 2% are replayed.

Replays: 1% + 0.8% + 0.2% = 2% of runs. 2,500 x 0.02 = 50 a second.

Only those phones upload a log: 50 uploads and 50 queue messages a second.

The replay fleet is 50 / 1.4 = 35.7, so 36 tasks. Nothing saturates; the note reads "Busiest: Replay workers at 69%".

The month comes to $7,711.98 at the peak: replay workers $5,118.68, uploads $648.00, the queue $155.12, plus Stage 3's $1,790.18.

The bill falls 38 times from Stage 4. The latency tile shows 3ms, unchanged: replays stay async.

The workers' writes, about 55 rows and 5 board writes a second, are not on the canvas.

Press 1,000, the average: 15 replay tasks, and the cost tile shows $3,632/mo.

By hand, at a 200 ms replay, Stage 5 would be about $3,200 a month, and replaying every run still about 21 times as much.

Replay proves the log matches the score, not that a person played. A bot makes a genuine log.

So the top 1,000 and every reward tier also get behavioural checks on the log, such as reaction times and timing spread.

A human reviews them before any prize.

A multiplayer game is different: its game servers decide the result and sign it, so there is nothing to replay.

The trade-off:

Cheap checks for every run, a replay for a sample and for every contender. In exchange, the bill falls 38 times.

Stage 5: Check every run, replay 2%

• Load: 2,500 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $7,712/mo

Check in: Why replay every contender, when 1% at random catches a persistent cheater anyway?

Friends, the month close and other Regions

Three questions usually follow. None changes a canvas, so we reason by hand.

Friends board: one command reads up to 200 friends' scores from the replica, and one more their names. About 50 µs together (our estimate).

A friends view follows about 1 run in 4: 625 a second at the peak, 3.1% of the replica's command thread.

A board per player, written on every run, would hold 25M x 200 = 5B entries, about 419 GiB, and take 500,000 writes a second.

Month close: at 00:10 UTC on the 1st, after pending replays finish, a job reads the old board in pages of 10,000 players.

Those pages give the final ranks for rewards.

A sample of every reward-eligible player's runs is replayed before prizes.

The old board is kept a week, then removed with UNLINK, which frees its memory away from the command thread.

Other Regions: one global order needs one writer. The home Region keeps the database and the primary board.

Other Regions forward results home, about 70 to 200 ms more. They read from Global Datastore, ElastiCache's copies in other Regions.

AWS says those copies lag typically under one second.

A cheaper option is a board per Region, and a global top 100 merged every minute from each Region's top 100. It is exact for the top 100.

The trade-off:

A friends view reads; nothing fans out on write. In exchange, a friends board costs 3.1% of the replica's thread, not 419 GiB of memory.

Check in: Why not keep a friends board per player, updated on every run?

The audit chain on one screen

Skip on a first read; come back before the interview. Stages 4 and 5 at the 2,500 peak, from one run to the bill.

A 30-day month has 2,592,000 s.

Stage 4: 1 log, 1 message and 1 replay a run.

Uploads: 2,500 x 2,592,000 = 6.48B PUTs, at $0.005 a 1,000: $32,400.00.

Queue: the card bills 3 requests a message, 19.44B, at $0.40 a million after the first million: $7,775.60.

Replays: a task replays 2 a second, 1.4 at its 70% target. 2,500 / 1.4 = 1,785.7, so 1,786 tasks, $253,943.48.

Stage 4: $253,943.48 + $32,400.00 + $7,775.60 + $1,790.18 = $295,909.26.

Stage 5: 0.02 a run, so 50 logs, 50 messages and 50 replays a second.

Uploads: 129.6M PUTs, $648.00. Queue: 388.8M requests, $155.12. Replays: 50 / 1.4 = 35.7, so 36 tasks, $5,118.68.

Stage 5: $5,118.68 + $648.00 + $155.12 + $1,790.18 = $7,711.98. That is 38 times less.

Step through the figure to compare the two.

Replay every run, or 2%

1. Stage 4 replays every run: 2,500 logs, messages and replays a second. The replay fleet is 86% of the bill.
2. Stage 5 checks every run cheaply and replays 2%: 1% at random, 0.8% flagged, and 0.2% contenders.
3. The bill falls 38 times. Every contender is still replayed, so no unverified run reaches the public top 1,000.

Check in: Close the page and size Stage 5's replay fleet from 2,500 runs.

The bill in an average month

Skip on a first read; come back before the interview. The canvases bill the peak all month and leave out storage and real connections.

This is Stage 5 at the average 1,000 runs a second, at AWS prices. The canvas there comes to $3,631.81.

Load balancer: a result comes every 3 minutes, past the 60 s idle timeout, so each result opens a new connection.

A board open follows its result within seconds and reuses it. With a session's token call, a third of one a run, that is 4/3 new connections a run.

1,000 x 4/3 / 25 = 53.3 LCU: about $323, not the card's $154.44.

Database storage: runs kept 90 days at about 190 B each, 1,477 GB, $332.42. Totals for 13 months add $5.85: $338.27.

Logs: 1.728M a day at 20 KiB, kept 30 days, 988.8 GiB: $22.74. The workers' reads add $20.74.

Queue: receives and deletes batched by 10 make 1.2 requests a message: $24.48, not $61.81.

Egress: the canvas bills the result's answer, 1 KiB a run, $213.47. Board opens and token calls add 0.58 KiB a run.

So 1,000 x 1.58 KiB x 2,592,000 s = 3,905.64 GiB a month. Past the first 100 GB free, at $0.09 a GB, egress is $342.51 in all.

Friends views carry up to 200 names and scores each, so they add more; this bill leaves them out.

The month comes to about $4,274. The replay workers are $2,132.78 of it, about 50%.

Fargate Spot, spare capacity at a discount that AWS can take back, could take up to about 70% off the replay line.

Stage 4 in an average month would be 715 replay tasks, $101,662.70, plus $12,960 of uploads.

The trade-off:

The replay workers are about half the real bill. Spot is the biggest lever still unused, and a faster replay the next.

Check in: Which line of the average month is largest, and what shrinks it?

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a player see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

The Valkey primary fails: ElastiCache promotes the replica in another zone. Plan for under a minute, detection included, and phones retry.

Both Valkey nodes fail: results still commit. Ranks say "updating" for about 15 to 25 minutes while a node is replaced and the board reloads.

The Aurora writer fails: the reader takes over, usually in about 30 s. Phones keep their results and retry with the same run id.

The replay fleet stops: audits wait in the queue for up to 14 days. Contenders stay off the public board, and their 10-minute timeout pauses.

A cheater is found: the database voids the month and flags the account. A correction flag on the board then stops any late write, and the player comes off it.

A result arrives after its month's close: it counts in the new month, so holding one back gains nothing. A token expires after 2 hours.

On the month's first day nearly every run would be a contender. So a contender must also pass a floor: last month's 1,000th total x the share of the month gone.

At 10 times the players a board is 21 GiB. The primary takes 100,000 commands a second at the peak: 25,000 x 28 µs is 70% of its thread.

A finale of 120,000 runs a second brings 480,000 commands, 336%. So the board must split.

Hash sharding makes writes lighter: with S shards, each takes 1/S of them.

Exact rank and neighbour reads fan out to every shard, so each shard still carries 2.5 reads a run, however many shards there are.

Score-range sharding splits the board by score. A rank is the players in higher ranges, refreshed every second, plus one local rank.

Ranges are cut at equal shares of runs, not of players. So the narrow highest range may get its own shard: in effect, a contender board.

A player who crosses a boundary is added on the new shard first, then removed from the old. A read that finds two copies keeps the higher score.

A common pattern, by our reading and not from a cited studio: exact ranks for the top 100,000.

Everyone else gets a percentile, such as "top 12%", from a histogram rebuilt every minute.

Say it out loud: "The database is the truth: one transaction, 2 rows a run. Ranks come from one sorted set a month, written with ZADD GT."

And: "Every run gets cheap checks. 2% are replayed, and a contender stays off the board until its replay passes."

Minutes 0 to 5: requirements, one run's path, and the counts: 2,500 runs a second at the peak, 200 at launch.

Minutes 5 to 12: Stage 1 and the cost of a count.

Minutes 12 to 20: Stage 2's O(rank) wall; Stage 3, the sorted set, ZADD GT and ties.

Minutes 20 to 30: anti-cheat, Stages 4 and 5.

Minutes 30 to 40: friends, the month close, a rebuild and other Regions.

Minutes 40 to 45: 10 times the players, and why an exact rank does not hash-shard.

The trade-off:

Each answer has a price: two stores to keep in step, a fleet for 2% of runs, and contenders who wait for their public place. Counting every rank costs far more.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.