Design a distributed message queue

Do the numbers before the boxes

An interviewer gives an event rate. Before any box is drawn, turn it into bytes a second, the unit every limit on this page uses.

A 45-minute version of this answer is at the end.

We assume one AWS Region, us-east-1. Every count on this page is our estimate unless sourced.

Producers are our own order services, in our own network. So no event crosses the internet.

The entry is events a second: one order event, published once. The peak is 25,000 events a second.

An event is 2 KiB on average. So producers write 25,000 x 2 KiB = 50,000 KiB a second.

That is 50,000 x 1,024 = 51,200,000 bytes a second.

AWS states broker rates in decimal MB/s, 10^6 bytes a second. So the stream is 51.2 MB/s.

In binary units it is 51,200,000 / 2^20 = 48.83 MiB/s.

Sizes on disk are binary GiB, and the page bills a GiB at AWS's price for a GB.

Every stage carries the same 51.2 MB/s. What changes is how many times the brokers store it and send it.

Check in: Close the page and derive the entry in MB/s and in MiB/s.

One topic, three brokers, seven days on local disks

Stage 1 is the first answer: a replicated log of order events on three brokers, keeping Kafka's default 7 days.

Kafka keeps a log: producers append events, and readers read them at their own pace. Amazon MSK is AWS's managed Kafka.

The events go to one topic, a named log, in 48 partitions. A partition is an ordered piece of the topic.

The key is the order id. All events of one order go to one partition, so they stay in order.

A broker is one Kafka server. Stage 1 runs 3 kafka.m7g.large brokers, one in each of 3 availability zones, separate data centres in one Region.

Each partition has 3 copies, one a zone: a replication factor (RF) of 3. One copy is the leader, and two followers fetch from it.

One copy of a day is 50,000 KiB x 86,400 s / 2^20 = 4,119.873 GiB.

So 7 days x 3 copies x 4,119.873 GiB = 86,517 GiB.

A Standard broker, the MSK kind with its own disks, holds at most 16,384 GiB at every size. That is an MSK quota.

3 brokers hold 49,152 GiB, and 86,517 / 49,152 = 176%.

This retention fits only up to 49,152 / 86,517 x 25,000 = 14,203 events a second.

A card turns red at 90%. A part is saturated past 100%: the result names it, the card stops at 100%, and its warning gives the full figure.

Here the result names Kafka cluster. Its warning gives 86,517 GiB needed and 49,152 GiB held.

A full Kafka disk does not slow down. The broker stops taking writes and goes offline. Leadership fails over to the in-sync followers, which fill at the same rate. Replicas do not move.

A bigger broker does not help, since the quota is the same at every size. A log is sized by what it holds before what it moves.

Four tiles go with each canvas: the load in events a second (req/sec), the latency, the cost a month, and what saturates first.

The latency tile shows 10ms, a fixed 10 ms the card adds to a produce. A produce also waits up to 5 ms to fill a batch, which the tile does not count.

The cost tile shows $5,782/mo. Brokers: 3 x 720 hours x $0.204 an hour = $440.64. A full cluster bills its full disks: 49,152 GiB x $0.10 = $4,915.20.

One consumer group reads the topic: 12 tasks on Fargate, AWS's serverless containers. They cost 12 x 720 hours x $0.04937 = $426.56.

The tile adds them: $440.64 + $4,915.20 + $426.56 = $5,782.40.

On the canvas, each connection out of the cluster is one consumer group. The card applies the copies and the groups itself, so no connection carries a multiplier. There are no load buttons: at the peak this stage is past its wall.

The trade-off:

Three copies in three zones survive a lost zone. In exchange, a week of them needs 176% of the disks, and a bigger broker holds no more.

Stage 1: One topic, three brokers, seven days on local disks

• Load: 25,000 req/sec
• First to saturate: Kafka cluster
• Estimated cost: $5,782/mo

• Kafka cluster: 168 h x RF 3 of 48.83 MiB/s needs 86,517 GiB, and 3 brokers hold 49,152 GiB. When a broker's log directory fills, it stops taking writes. 14,203 messages/s is the most this retention sustains.

Check in: Why does a bigger broker not fix Stage 1?

One event, from producer to delete

An interviewer walks one event through its states, and asks what each promise means. Each state is a probe.

Batched. The producer groups events by partition and waits up to 5 ms, linger.ms, for a fuller batch. One request carries many events.

Appended. The leader writes the batch to its log in memory, the page cache. The two followers fetch it.

Acked. Producers send with acks=all: the ack waits for every copy in the in-sync replicas (ISR), the copies that are caught up.

min.insync.replicas is 2, so a write fails when fewer than 2 copies are in sync. An acked write is then in at least 2 zones.

An ack is not a flush to disk. Kafka leaves that to the operating system, so durability is copies in 3 zones.

Visible. Consumers read up to the high watermark, the last offset every in-sync copy holds. An offset is an event's position in its partition.

Committed. A group commits its offset after it processes the events. A restart replays from the last commit, not from where it stopped.

Deleted. After 7 days, the file that holds the event is deleted.

Retries. Idempotence is on by default with acks=all: the broker drops a retried batch it already has, per producer session and partition.

With IAM, AWS's access control, the producer also needs the kafka-cluster:WriteDataIdempotently permission.

A restarted producer, or an app that resends after its delivery timeout, writes the event again at a new offset.

So each sink, the store a consumer writes to, writes keyed on an event id the producer sets.

The event id is the order id plus the order's event number. A replay or a resend then writes nothing twice.

An offset only finds where a replay starts. A sink keyed on the offset would write a resend twice.

Leaders. MSK sets unclean leader election to true on a cluster without tiered storage. It lets a copy that is not in sync become leader.

Set it false. Then a lost ISR means a pause, not lost acked writes.

Step through the figure to follow one event.

The trade-off:

acks=all and min.insync.replicas 2 put every acked write in 2 zones. In exchange, each write waits for a follower in another zone.

One event, from producer to delete

1. Batched: the producer groups events by partition, and waits up to 5 ms for a fuller batch.
2. Appended: the leader writes the batch to its page cache. Two followers, one in each other zone, fetch it.
3. Acked: every in-sync copy has it, and at least 2 are in sync. An ack is not a flush to disk.
4. Visible: consumers read up to the high watermark, the last offset every in-sync copy holds.
5. Committed: a group commits its offset after it processes the event. A restart replays from there.
6. Deleted: after 7 days. From Stage 2, a closed segment is copied to the tier first.

Check in: A producer restarts and sends an event again. What stops the payments ledger from booking it twice?

Why a Kafka log, and when SQS or Kinesis wins

An interviewer asks why a log and not a queue, and why Kafka and not Kinesis. Each has a fair case.

A log keeps events after they are read. Many groups read at their own pace, and any group can replay.

A queue such as SQS, AWS's message queue, wins for work items read once. Each message has its own ack and retries.

Messages that keep failing go to a dead-letter queue, and there are no partitions to size.

Kinesis Data Streams is AWS's own log. A shard takes 1 MiB/s or 1,000 records a second in, and gives 2 MiB/s out.

Readers share the 2 MiB/s, or each gets its own with enhanced fan-out. Fan-out is billed per reader-shard-hour and per GB read.

The stream is 48.83 MiB/s, so it needs at least 49 shards. At 80% of each shard, it needs about 62.

Records need only 25,000 / 1,000 = 25 shards, so bytes set the count.

With six readers, the number Stage 3 reaches, each needs enhanced fan-out.

Prices are the examples on AWS's Kinesis pricing page (read 2026-10-18). Sized at 80%, about 62 shards, that is about $16,100 a month.

Most of it is the per-GB fan-out fee. A month of the stream is 30 x 4,119.873 = 123,596 GiB.

Fan-out reads it 6 times at $0.013 a GB: about $9,640.

The rest is for 62 shards: shard-hours $669.60, 7-day retention $892.80, PUT units $907.20 and fan-out hours $4,017.60. That is $6,487.20.

Kinesis wins with no brokers or disks to run, an on-demand mode that reshards for you, and native triggers for Lambda, AWS's functions.

Kafka wins when many groups read the same bytes: they cost broker network, not a fee a GB.

Kafka also keeps long replay cheap in a tier, which Stage 2 adds. Its transactions give exactly once inside Kafka.

Say it like this: "A log, because six teams read every event and can replay. Kafka, because readers cost broker network, not a fee a GB."

The trade-off:

Kafka needs brokers, disks and partitions that we size. In exchange, each extra reader costs network, not a fee for every GB it reads.

Check in: Six teams each read every event. Why does that favour Kafka over Kinesis?

Tiered storage: one day on local disks

The interviewer asks: Keep a week of history, and do not lose data.

Stage 1's disks cannot hold a week of 3 copies. Stage 2 moves the old bytes off the brokers.

A partition's log is stored in files called segments. Tiered storage copies each closed segment to a low-cost tier, as one copy.

The tier keeps it for the full 7 days: 7 x 4,119.873 = 28,839.11 GiB.

Broker disks keep only 24 hours of 3 copies: 3 x 4,119.873 = 12,359.62 GiB. That is 12,359.62 / 49,152 = 25.1% of what they hold.

Tiered storage also sets unclean leader election to false by default.

Now the disks have room. Nothing saturates, and the last tile notes "Busiest: Kafka cluster at 65%". The next beat says which limit that is.

The latency tile shows 10ms. The cost tile shows $3,834/mo.

MSK: brokers $440.64, local disks 12,359.62 x $0.10 = $1,235.96, the tier 28,839.11 x $0.06 = $1,730.35. That is $3,406.95.

With the consumers, $3,406.95 + $426.56 = $3,833.51. A week now fits, for less than Stage 1.

Step through the figure to compare the two layouts.

The trade-off:

The tier holds one copy at $0.06 against three at $0.10. In exchange, reads older than a day come from the tier, slower and billed a GB.

A week on broker disks, or one day and a tier

1. Stage 1: 7 days x 3 copies on broker disks, 86,517 GiB against 49,152. The disks need 176%.
2. Stage 2: broker disks keep 24 hours x 3 copies, about 12,360 GiB, 25.1% of what they hold.
3. The tier keeps the full 7 days as one copy, about 28,839 GiB, at $0.06 a GiB-month.

Stage 2: Tiered storage: one day on local disks

• Load: 25,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $3,834/mo

Check in: The tier holds the week, and broker disks hold a day. Why does that cut the bill as well as the disks?

A broker's three limits: Equation 1

Stage 2's disks have room, so what does the 65% on the card measure? AWS sizes a Standard broker on three limits, its Equation 1.

A broker has three limits on the bytes it moves. They are its storage volume (250 MB/s), its EBS link (78.75 MB/s on kafka.m7g.large) and its network (119.94 MB/s).

Each byte in is written 3 times, and a leader sends it once to each group and once to each follower. AWS sizes to 80% of the lowest.

The storage volume is the disk itself, on EBS, the network disks of AWS. It takes 250 MB/s by default.

The EBS link is the pipe from the instance to that disk. On a kafka.m7g.large it carries 78.75 MB/s.

At RF 3 on 3 brokers, each broker holds every partition, so each writes all 51.2 MB/s.

The volume reads 51.2 / 250 = 20.5%. The EBS link reads 51.2 / 78.75 = 65.0%.

For the network term, a leader sends each byte c + r - 1 times. That is once to each of c groups, and once to each of r - 1 followers.

With 1 group and RF 3, that is 3 copies, 51.2 MB/s out of each broker. AWS's sizing post gives kafka.m7g.large 119.94 MB/s: 42.7%.

So the EBS link binds, at 65.0%, and that is the 65% on the card. The card warns past 80% on any limit.

The buttons count events a second. On Stage 2's canvas, press 50,000, twice the peak: the EBS link reads 102.4 / 78.75 = 130%.

The result names Kafka cluster. Its network warning reads 85%: 102.4 / 119.94, past the 80% line.

Growth hits the EBS link first while one group reads.

The latency tile shows "≥ 10ms". The ≥ marks a floor: produce requests queue, so the true time is higher than any figure shown.

The cost tile shows $5,430/mo. The bill is clamped at the EBS wall, 78.75 MB/s / 2 KiB = 38,452 events a second, so it is not double.

Step through the figure to see the three limits at both loads.

The trade-off:

Equation 1 sizes a broker on the bytes it writes and sends. In exchange, it has no CPU term, so the peak still needs a test.

One broker's three limits

1. Stage 2 at 25,000: volume 20.5%, EBS link 65.0%, network 42.7%. The EBS link is the tallest, under the 80% line.
2. At 50,000: volume 41.0%, EBS link 130%, network 85.4%. Two bars pass the 80% line, and the EBS link passes 100%.

Check in: Stage 2 runs at 50,000 events a second. Which limit breaks first, and why that one?

Why tiered storage, and when local disks win

Stage 1 had too little disk. An interviewer asks why not buy bigger disks, or keep the week on local disks with more brokers.

Bigger disks are not on offer. A Standard broker holds at most 16,384 GiB at every size.

More brokers can hold the week, but every GiB of it sits there 3 times, at $0.10 a GiB-month.

The disk alone for 7 days x 3 copies is 86,517.33 x $0.10 = $8,651.73 a month. All of Stage 2 is $3,406.95.

The tier holds one copy at $0.06 a GiB-month, against three copies at $0.10 on broker disks.

Local disks win when consumers often replay more than a day. First bytes from the tier are slower, and a tier read costs $0.0015 a GB.

Local disks also win for compacted topics, which keep only the latest value for each key. MSK tiered storage does not support them.

Tiered storage needs MSK 3.6.0 or later, or 2.8.2.tiered. On a tiered cluster, a topic's cleanup policy cannot change after creation.

A topic can turn the tier off, but cannot turn it on again.

Why kafka.m7g.large: it is the family of AWS's sizing post, and large is its cheapest size.

A kafka.m5.large has a 3% higher EBS baseline and a 20% lower network baseline, for $0.006 more an hour.

Say it like this: "Retention is a bytes problem. Tiered storage keeps one copy of the week, and 24 hours of three copies on the brokers."

The trade-off:

The tier moves old bytes off the brokers. In exchange, a replay older than a day reads slower, and a compacted topic stays local.

Check in: A team replays three days of events every morning. Does tiered storage still suit it?

Six consumer groups read the whole topic

The interviewer asks: Five more teams want every order event.

Stage 3 keeps Stage 2 and adds five consumer groups. The producers do not change.

The six groups are fulfilment, the payments ledger, fraud, the search indexer, notifications and the warehouse loader. Each needs every event (our estimate).

Each group reads every event at its own offset. Fraud scores every change of state: address edits, payment retries, cancels.

The loader copies every event to the data warehouse. Other systems cannot read the Kafka tier, so the loader is a group too.

A leader now sends each byte c + r - 1 = 6 + 3 - 1 = 8 times. That is once to each group, and once to each follower.

That is 51.2 x 8 = 409.6 MB/s out of 3 brokers, 136.5 MB/s a broker.

Against 119.94 MB/s of network a broker, 136.5 / 119.94 = 113.8%. The warning prints 114%.

The result names Kafka cluster, on the network. The EBS link is still at 65.0%.

The producers did not change. The readers moved the wall.

Kafka filters nothing on the broker. Each consumer group pays for the whole stream.

A group that needs one type of event should read a topic of that type. Here each of the six needs every event.

Follower fetching lets a consumer read a follower in its own zone. The same 8 copies still leave some broker, so the network term does not fall.

The latency tile shows "≥ 10ms", a floor: produce requests queue at the brokers.

The cost tile shows $5,606/mo. The six groups are 72 tasks: 72 x 720 hours x $0.04937 = $2,559.34.

MSK reads $3,046.36, less than Stage 2, because the card bills storage only on the bytes the brokers carry.

That is not a saving: the rest of the stream waits in the producers. $3,046.36 + $2,559.34 = $5,605.70.

On the canvas there are no load buttons: at the peak this stage is past its wall.

Your turn, after the quiz, starts from this stage. Step through the figure to count the copies.

The trade-off:

Six teams read one stream with no extra producer work. In exchange, every group adds one copy of the stream to the leaders' network.

Each group adds a copy of the stream

1. One group: a leader sends each byte 3 times, to the group and to 2 followers. 42.7% of the network.
2. Six groups: each reads the whole stream. A leader sends each byte 8 times, 6 groups plus 2 followers.
3. 51.2 x 8 = 409.6 MB/s out of 3 brokers: 136.5 MB/s each, against 119.94. That is 113.8%.

Stage 3: Six consumer groups read the whole topic

• Load: 25,000 req/sec
• First to saturate: Kafka cluster
• Estimated cost: $5,606/mo

• Latency past saturation is a lower bound, not an estimate. Kafka cluster is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Kafka cluster: Leaders send 409.6 MB/s: 51.2 MB/s in x (6 groups + RF 3 - 1), over 3 brokers. That is 114% of Equation 1's network term, 119.94 MB/s a broker on kafka.m7g.large. AWS sizes to 80%.

Check in: The producers did not change. Why does Stage 3 saturate when Stage 2 did not?

Why six groups, and how a group keeps order

An interviewer asks whether six groups are needed, and how a group processes 48 partitions in order.

Six groups suit six owners who each need the whole stream, at their own pace.

Split the topic when a group needs only a subset.

A fan-out service, one group that writes many sinks, suits groups with one owner and one pace.

A group's tasks share the partitions, and each partition goes to one task. 12 tasks own 4 partitions each.

So a group's parallelism stops at the partition count: 48 tasks at most.

A task runs 1 vCPU, a virtual processor, with 10 threads. At 2 ms an event (our estimate), it takes 10 x 1,000 / 2 = 5,000 events a second.

12 tasks take 60,000, so a group's card reads 25,000 / 60,000 = 41.7% when the brokers keep up.

Order within a key means one key at a time. Each task runs different keys in parallel.

A partition takes 25,000 / 48 = 521 events a second. At 2 ms each, one thread in sequence would need 104% of its time.

So each task runs a key-ordered consumer: events of one key in order, different keys at once.

Why 48 partitions: a group can run up to 48 consumers. Kafka can add partitions but not remove them, and adding them moves keys.

AWS recommends at most 1,000 partition copies on a kafka.m7g.large broker. Here each broker holds 48.

Why not one partition, for one global order? Even with one group, one leader's network share is 119.94 / 3 = 39.98 MB/s.

That is 19,521 events a second, so 25,000 reads 128%. And nobody needs a global order of all orders.

Say it like this: "Each group pays the whole stream. Order is per key, so a group runs keys in parallel across 48 partitions."

The trade-off:

Order per key lets a group scale to 48 tasks. In exchange, one key runs at the speed of one thread.

Check in: A group runs 60 tasks on 48 partitions. What do the extra 12 do?

Why more brokers, and when bigger ones win

Stage 3's wall can move two ways: more brokers of one type, or bigger brokers. An interviewer asks how to choose.

Choose on what one failure costs, not only on a load line.

Kafka does not copy a lost broker's replicas again by itself. Partitions that had a copy there run on 2 copies until it is replaced.

Their leaders then send c + r - 2 copies: 7 at six groups, not 8.

On Stage 3's 3 brokers, one lost leaves 2 survivors at 51.2 x 7 / 2 / 119.94 = 149.4% of their network.

The hole. A lost broker takes its share of the network: a third on 3 brokers. The more brokers, the smaller the share.

The patch. A patch restarts brokers one at a time, so the network hole opens on every patch.

The rebuild. The disk usually comes back with the broker, so only the gap is fetched.

The full refetch is the worst case, a lost volume. On 3 brokers that is a third of 12,359.62 GiB, about 4,120 GiB.

AWS's sizing post warns that too few very large brokers increase the blast radius.

Bigger brokers win when the team wants fewer machines: a rolling update restarts fewer brokers, and finishes sooner.

A bigger broker raises the EBS link and the network a broker. It does not raise the 16,384 GiB disk quota.

Both are sound designs.

Say it like this: "More small brokers make each failure a smaller hole. Bigger ones mean fewer machines and shorter rolling updates."

The trade-off:

More brokers shrink each hole and each worst-case rebuild. In exchange, there are more machines to run, and a rolling update takes longer.

Check in: Why does a lost broker's hole open even when no broker fails?

Rent the broker: Express

The interviewer asks: We do not want to size broker disks.

Stage 4 makes one change: the cluster moves to 3 express.m7g.2xlarge brokers, one a zone. The key stays the order id.

Express brokers are an MSK broker family where AWS manages the storage. Storage is unlimited, so there are no disks to size and no retention wall.

Replication is always 3, and min.insync.replicas always 2.

AWS publishes a throttle for each Express size: data in and out a broker, sustained and maximum, and 15 MB/s a partition.

The quotas count client traffic only. Each broker sends 6 x 51.2 / 3 = 102.4 MB/s to the six groups.

An express.m7g.2xlarge sends 125 MB/s sustained and 234.2 MB/s at most. 102.4 MB/s is 81.9% of the first and 43.7% of the second.

The card warns past the sustained figure and saturates past the maximum. So it reads 43.7%, with no warning.

Data in is 51.2 / 3 = 17.07 MB/s a broker, 18.2% of its 93.7 MB/s maximum.

The hottest partition takes 51.2 / 48 = 1.07 MB/s, 7.1% of 15.

Nothing saturates. The last tile notes "Busiest: Kafka cluster (Express) at 44%". The latency tile shows 10ms.

The cost tile shows $10k/mo. Brokers: 3 x 720 hours x $1.632 = $3,525.12.

Storage: 7 days of one copy, 28,839.11 GiB x $0.10 = $2,883.91. Data in: 123,596 GiB a month x $0.01 = $1,235.96.

MSK is $7,644.99. With the six groups, $7,644.99 + $2,559.34 = $10,204.33.

On the canvas there are no load buttons: the lesson is the disks, and they are gone at any load.

The trade-off:

Express takes the disks away. In exchange, the brokers cost $3,525.12 a month, and every GB in costs $0.01.

Stage 4: Rent the broker: Express

• Load: 25,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $10k/mo

Check in: Each Express broker sends 102.4 MB/s to consumers. Why does the card not warn?

Why Express, and when Standard wins

Express rents the storage at a price. An interviewer asks what that buys, and when Standard brokers are the better answer.

Express gives no disks to size and no retention wall. Storage grows with the bytes, and AWS sizes it.

AWS publishes a hard throttle a broker and a partition, so each ceiling is a published number.

Express also takes over the partition moves. Intelligent rebalancing is on by default for new Express clusters, since 20 November 2025.

It moves partitions when you add or remove brokers, and while it is on, MSK blocks manual reassignment tools. Clusters created before that date start with it paused.

Why the 2xlarge: each broker sends 102.4 MB/s to consumers. An express.m7g.xlarge sends 62.5 MB/s sustained, so it would warn.

Standard costs less for a steady load, and lets you choose RF and disks.

Express wins when the team will not size disks or move partitions.

A steady load like this one goes back to Standard once the team will size disks. Your turn sizes it.

Say it like this: "Express rents the disks and publishes the throttle. For a steady load, I would size Standard brokers instead."

The trade-off:

Express takes disk sizing off the team. In exchange, a steady load pays more than on Standard brokers sized for it.

Check in: Does Express remove partition moves when the cluster grows?

Key by merchant

The interviewer asks: The ledger now keeps a balance for each merchant, with an overdraft check.

Stage 5 makes one change: the key. An overdraft check reads the balance before it posts, so the ledger needs each merchant's events in order.

So events are keyed by merchant. The largest merchant sends 30% of them (our estimate).

A key lives on one partition, and a partition has one leader. A cluster of 3 brokers or 30 gives it the same leader.

The hottest partition takes the 30%, plus 1/48 of the other 70%: 0.3 + 0.7 / 48 = 31.46% of the stream.

That is 0.3146 x 51.2 = 16.11 MB/s, against AWS's 15 MB/s a partition: 107.4%.

The result names Kafka cluster (Express). The card stops at 100%, and its warning prints 107%.

The brokers have room: data out at 43.7% of the maximum, data in at 18.2%. One partition is full.

The same key fails on Standard. On kafka.m7g.large at six groups, one leader's network share is 119.94 / 8 = 15.0 MB/s at any broker count.

16.11 MB/s is 107.4% there too.

The ledger cannot keep up either: 7,500 events a second on one key, at 2 ms each, is 15 seconds of work a second.

On the canvas, the consumer cards spread each group's load evenly over its tasks, so the ledger's card does not show its hot key.

The latency tile shows "≥ 10ms", a floor: one leader queues produce requests.

The cost tile shows $9,921/mo, less than Stage 4, because the card bills the bytes the partition lets in.

That is not a saving: the rest queues in the producers. MSK is $7,361.92, and $7,361.92 + $2,559.34 = $9,921.26.

Express reads AWS's 15 MB/s as data in. AWS does not state a direction; if it counts data out too, this stage is further past its line.

On the canvas there are no load buttons: at the peak this stage is past its wall. Step through the figure to follow the hot key.

The trade-off:

Keying by merchant gives the ledger merchant order. In exchange, the largest merchant gets one partition and one leader, whatever the cluster size.

One hot key, one partition

1. By order id: each of 48 partitions takes 1/48 of the stream, 1.07 MB/s against 15.
2. By merchant: the largest takes 30% plus 1/48 of the rest. 16.11 MB/s on one leader, 107.4%.
3. Split over 24 sub-keys: the hottest partition takes 1.39 MB/s, 9.2%, read at 8.3 MB/s. The card shows 11.3%, an upper bound.

Stage 5: Key by merchant

• Load: 25,000 req/sec
• First to saturate: Kafka cluster (Express)
• Estimated cost: $9,921/mo

• Latency past saturation is a lower bound, not an estimate. Kafka cluster (Express) is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Kafka cluster (Express): The hottest partition takes 16.11 MB/s (48 partitions, hot key 30%) against Express's 15 MB/s a partition: 107%. A partition has one leader, so the broker count does not raise this ceiling.

Check in: The cluster grows from 3 Express brokers to 30. What happens to the hottest partition?

Split only the hot merchant over 24 sub-keys

Stage 6 makes one change: the largest merchant's key gains a sub-key, 0 to 23, picked from the order id. Every other merchant keeps its own key.

One order's events share one sub-key, so they stay on one partition, in order.

The ledger keeps 24 sub-balances for that merchant, each with its own overdraft check, in its own order.

The 30% spreads over 24 partitions: 30% / 24 = 1.25% each.

The other 23 sub-keys sit on their own partitions, so the hottest takes 1.25% plus 1/48 of the other merchants' 70%.

That is 0.0125 + 0.70 / 48 = 2.71% of the stream: 0.027083 x 51.2 = 1.387 MB/s, 9.2% of 15.

Six groups read it at 6 x 1.387 = 8.32 MB/s, 55.5% of 15. So it is clean whichever direction the 15 MB/s counts.

The card spreads all the rest evenly, so it shows 1.69 MB/s, 11.3%. That is an upper bound, and still clean.

Why 24: 8 sub-keys would put 2.67 MB/s on a partition, read at 16.0 MB/s. That is clean only if the 15 MB/s counts writes alone.

The 24 sub-keys must land on 24 different partitions. Kafka's default hash does not do this: it spreads 24 keys over about 19 of 48.

So the producers use a custom partitioner, the code that maps a key to a partition. Or they pick sub-key values that hash to different partitions.

The ledger keeps up again: each sub-key takes 7,500 / 24 = 312.5 events a second, 62.5% of one thread at 2 ms.

Premise: no other merchant sends more than a few percent of the events (our estimate).

Nothing saturates. The last tile notes "Busiest: Kafka cluster (Express) at 44%".

The latency tile shows 10ms. The cost tile shows $10k/mo, the same $10,204.33 as Stage 4: the split changes keys, not machines.

On the canvas there are no load buttons: this stage is about the key, and the brokers sit at 81.9% of their sustained line.

The trade-off:

Splitting one key frees the partition and the ledger. In exchange, the merchant has 24 balances to keep, and a custom partitioner to run.

Stage 6: Split only the hot merchant over 24 sub-keys

• Load: 25,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $10k/mo

Check in: Why must the hot merchant's 24 sub-keys use a custom partitioner?

Why a split key, and when the order id wins

The split keeps merchant order inside each sub-key only. An interviewer asks what that costs the overdraft check, and when a split is wrong.

A sub-balance can refuse a debit that the merchant's total covers. The ledger moves funds between sub-balances to fix it. A payout is 24 debits, one on each sub-key.

Each debit is checked in its own order.

A sum of the 24 read at one moment is not a consistent balance: each sub-balance sits at its own offset.

This is the bucketed stock pattern: one hot counter split into buckets, each checked on its own.

The order id wins when no consumer needs merchant order. The merchant dashboard sums events, and a sum does not depend on order.

The split wins when one consumer needs merchant order, as the overdraft check does.

A third choice is a merchant-keyed topic that only the ledger reads. It still needs the split: 30% of 51.2 MB/s is 15.36 MB/s on one partition.

Split only the key that is hot, and only as far as its consumer can merge.

Say it like this: "One key, one partition, one leader. I split only the hot merchant, 24 ways, and the ledger keeps 24 sub-balances."

The trade-off:

24 sub-balances keep each check in order. In exchange, a debit can be refused while the total covers it, and a payout takes 24 debits.

Check in: Why is a sum of the 24 sub-balances not a safe balance to show?

Rebalancing, exactly once and offsets

Three consumer questions usually follow. None changes a canvas, so we reason by hand.

A consumer dies. Its partitions stall until its session times out, 45 s by default.

A rebalance gives its partitions to other tasks. The eager protocol stops the whole group while it runs.

Cooperative rebalancing (KIP-429, Kafka 2.4) moves only the partitions that change owner. The newer consumer protocol, KIP-848, moves assignment to the broker.

A KIP is a Kafka improvement proposal. KIP-848 is in Kafka 4.0.

Static membership gives each task a fixed id, so a rolling restart skips the rebalance.

Exactly once. Idempotence drops retries inside one producer session and partition, and nothing more.

Transactions, read by consumers set to read_committed, give exactly once from Kafka to Kafka.

A sink outside Kafka is at least once: DynamoDB, AWS's managed database, or S3, its object store.

Writes keyed on the producer's event id make a replay and a resend harmless.

On MSK, do not set read_committed for reads from the tier unless the client uses transactions.

Offsets. Commits go to a Kafka topic, __consumer_offsets, not once an event.

Auto-commit runs every auto.commit.interval.ms (5 s by default), checked at a poll. A manual commit runs wherever the code calls it.

The commit cadence sets how much a restart replays.

Say it like this: "At least once into the sinks, with writes keyed on the producer's event id. Exactly once only from Kafka to Kafka."

The trade-off:

An auto-commit every 5 s keeps offsets cheap. In exchange, a restart replays the events since the last commit, so every sink must be idempotent.

Check in: Is this pipeline exactly once?

Compaction, CPU and the zone bill

Three more questions turn on things no stage draws: compaction, broker CPU and data between zones.

Compaction. A compacted topic keeps the latest value for each key, so its disk is key count x value size, not rate x retention.

Tiered storage does not support compacted topics, so one stays on broker disks. This page uses the delete policy, 7 days.

Broker CPU. Equation 1 has no CPU term. AWS keeps CPU User + System under 60%.

MSK encrypts client traffic by default with TLS. TLS removes Kafka's zero-copy send, which hands file bytes straight to the network.

So CPU follows the bytes out: test at the peak.

Data between zones. MSK does not charge for replication between brokers. Clients pay standard transfer, $0.01 a GB each way.

A GB that crosses zones pays both ways: $0.02. A leader is in another zone for 2 of every 3 partitions.

Six groups: 6 x 123,596 GiB x 2/3 x $0.02 = about $9,888 a month.

Producers send to leaders in other zones too: 123,596 x 2/3 x $0.02 = about $1,648.

Together that is about $11,536, 3.4 times Stage 2's MSK line. The cost tiles do not include it.

Follower fetching, with each zone set as a rack, lets each consumer read the copy in its own zone. That removes the consumer part.

Two weeks of retention doubles the tier: 57,678.22 GiB x $0.06 = $3,460.69 a month. The local disks do not change.

Say it like this: "Consumers fetch in their own zone, or six groups pay about $9,900 a month to cross zones."

The trade-off:

Fetching from a follower in the same zone cuts the zone bill. In exchange, a consumer can see new events a little later.

Check in: Why does follower fetching cut the zone bill but not the network term?

The number chain on one screen

Skip on a first read; come back before the interview. Every number on Stages 1 to 6 comes from the ones before it.

Units: broker rates in decimal MB/s, sizes in binary GiB. The page bills a GiB at the price of a GB.

Entry: 25,000 x 2 KiB = 50,000 KiB a second, 51.2 MB/s, 48.83 MiB/s.

One copy a day: 50,000 x 86,400 / 2^20 = 4,119.873 GiB. A month of 30 days: 123,596 GiB.

Stage 1 disks: 7 x 3 x 4,119.873 = 86,517.33 GiB, against 3 x 16,384 = 49,152 GiB: 176%.

Stage 2 disks: 3 x 4,119.873 = 12,359.62 GiB, 25.1%. The tier: 7 x 4,119.873 = 28,839.11 GiB.

Equation 1 at 3 brokers and RF 3: the volume 51.2 / 250 = 20.5%, the EBS link 51.2 / 78.75 = 65.0%.

Network at 1 group: 51.2 x 3 / 3 / 119.94 = 42.7%. At 6 groups: 51.2 x 8 / 3 / 119.94 = 113.8%.

Express at 6 groups: 6 x 51.2 / 3 = 102.4 MB/s out a broker, 43.7% of 234.2 and 81.9% of 125.

Partitions: by order id, 51.2 / 48 = 1.07 MB/s. By merchant, 0.3146 x 51.2 = 16.11 MB/s, 107.4%. Split 24 ways, 1.387 MB/s, 9.2%.

Consumers: 12 tasks x 5,000 = 60,000 events a second a group, 41.7% at the peak.

Prices for 720 hours: kafka.m7g.large $0.204 an hour, express.m7g.2xlarge $1.632, a consumer task $0.04937.

Storage: $0.10 a GiB-month local, $0.06 in the tier, $0.10 on Express plus $0.01 a GB in.

Stage 1: $440.64 + $4,915.20 + $426.56 = $5,782.40. Stage 2: $3,406.95 + $426.56 = $3,833.51.

Stage 3: $3,046.36 + $2,559.34 = $5,605.70. Stages 4 and 6: $7,644.99 + $2,559.34 = $10,204.33.

Stage 5: $7,361.92 + $2,559.34 = $9,921.26. Stages 3 and 5 bill storage only on the bytes the brokers carry.

Check in: Close the page and rebuild Stage 3's 113.8% from the entry.

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a caller see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

A broker dies: its leaders move to the survivors. Partitions with a copy on it run on 2 copies until it is replaced.

A zone goes down: acked writes survive on 2 copies, since min.insync.replicas is 2.

A disk fills: the broker stops taking writes. So retention is sized in bytes, and the tier holds the week.

A consumer dies: its partitions stall for the 45 s session timeout, then move.

A producer resends: the sink writes nothing twice, because it keys on the event id.

A key gets hot: one partition and one leader carry it at any cluster size. Split only that key.

Say it out loud: "A Kafka log on MSK, RF 3 across 3 zones, acks=all and min ISR 2. Each group pays the whole stream."

And: "Size by what the log holds and how many groups read it, not only by what producers write."

Minutes 0 to 5: requirements. 25,000 events a second, 2 KiB each, a week of history, a zone lost.

Minutes 5 to 12: Stage 1 and one event's states. A log, the bytes held, and the disk wall.

Minutes 12 to 20: Stage 2, tiered storage and Equation 1.

Minutes 20 to 28: Stage 3, six groups and the network wall. More brokers or bigger ones.

Minutes 28 to 34: Stage 4, Express, and when Standard wins.

Minutes 34 to 41: Stages 5 and 6. The hot merchant, the split, and the sub-balances.

Minutes 41 to 45: rebalancing, exactly once, and the zone bill.

The trade-off:

Each answer has a price. Copies cost disk and network, every group costs network, and a split key costs the ledger simple balances.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.