Design a chat system

Do the numbers before the boxes

An interviewer gives a scale, not a load. A chat system has several loads: messages sent, connections held and connects. Each sizes a different part.

A 45-minute version of this answer is at the end.

We assume 20M daily users in one AWS Region, us-east-1, serving North America and Europe. Every count on this page is our estimate unless sourced.

Each daily user sends about 13 messages a day: 20M x 13 = about 260M a day, 3,000 a second on average.

The evening peak runs at twice the average: 6,000 messages sent a second.

Phones hold a connection open while the app is open. A sixth of daily users are online at the evening peak (3.33M), with 1.2 devices each: 4M connections.

That is about 2 hours of connected time a daily user a day, a desktop-heavy assumption: Slack-like clients stay connected through the working day.

A phone-only chat app would be nearer 1 to 2M at the peak, and every connection figure on the page scales with it. On average 2M are connected.

A connection lasts about 8 minutes on average. So 4M / 480 s = 8,333 connects a second at the peak, 4,167 on average.

Each connected device also sends a heartbeat every 30 s: a tiny message that says the connection is still alive. That is 4M / 30 = 133,333 a second at the peak.

Scope: this design is not end-to-end encrypted. The server reads messages and keeps history, as Slack, Discord and Telegram's cloud chats do.

The last beat says what changes with end-to-end encryption.

Slack reports tens of millions of connected clients, and Messenger has over a billion accounts. This page sizes a mid-sized product.

Check in: Close the page and derive the peak message and connect rates.

Hold a WebSocket, send over HTTPS

A message must reach an online phone at once. The server cannot open a connection to a phone, so the phone keeps one open.

A WebSocket is a two-way connection that a phone opens over HTTPS and then holds. Either end can send a frame, one small message, at any time.

Phones hold their WebSockets to a fleet of gateways: servers that keep connections open and write frames to them.

TLS, the connection's encryption, ends on the gateways. A Network Load Balancer (NLB), AWS's load balancer for plain connections, spreads phones over them.

The canvases do not include the NLB; the bill beat prices it.

The gateways run on ECS, AWS's container service, using Fargate: AWS runs the containers. Each task is one container with its own vCPUs and memory.

A vCPU is one virtual processor. A gateway task has 1 vCPU and 4 GB.

A held connection costs memory more than CPU: about 40 KB for its encryption state and buffers (our estimate).

The fleet spans 3 availability zones, separate data centres in one Region. If a zone fails, the other two must hold all 4M connections.

At most 50,000 fit on a task, about half its memory: 4M / 50,000 = 80 tasks in two zones. So 120 tasks, 40 a zone, 33,333 connections each normally.

Phones send over HTTPS to a stateless message API, as Slack's and Discord's clients do. They receive over the WebSocket.

To deliver, the API must know which gateway holds each device. A session registry maps each user to their devices, and each device to its gateway.

It lives in Valkey, an in-memory key-value store that AWS runs as ElastiCache.

A gateway writes a device's entry on connect and deletes it on close. Each gateway also refreshes its own liveness key every 10 s.

So a dead gateway's entries are ignored once its key expires, with no write for each heartbeat.

The trade-off:

A fleet sized by memory, paid for all day. In exchange, the server can reach any online device the moment a message is stored.

Check in: Why do phones send over HTTPS rather than over the WebSocket?

A box for every user

Users have several devices and expect history on all of them. A phone back from an hour offline must find exactly what it missed.

Every user gets a box: a per-user inbox, one ordered list of rows numbered 1, 2, 3, for all their one-to-one and group chats.

Fan-out is turning one message into a copy for each recipient. Making the copies when the message is written is fan-out on write.

Here a message writes one row into each member's box, the sender's own included, in one transaction. The sender's other devices see it too.

Each row carries two numbers. The conversation number sets display order, the same for every member.

The box number is the next number in that user's box: the sync cursor. A device remembers its last box number and asks for the rows after it.

Both numbers come from counters in Valkey. The API also gives each message a message id: 64 bits built from the time, the API task and a sequence.

The message id breaks display ties. It is never the box number.

The table, box_rows, has the primary key (user_id, box_number). Each row also holds conversation_id, conversation_number, message_id, sender_id, kind and body.

The kind is message, read, receipt, mention, edit or delete. Each member's row carries its own copy of the body.

Reading writes too. A read marker records how far a user has read a chat. It is a row in the reader's own box, so every device clears the badge.

In a one-to-one chat a read also writes a receipt in the sender's box, so the sender sees "read". A group writes the marker only.

In a group, "read by" is worked out when the sender asks for it.

The mix at the peak: 80% of messages one-to-one, 20% in groups of 8 members on average.

Member rows: 0.8 x 2 + 0.2 x 8 = 3.2 a message.

A user reads about once every 2.5 messages received, 0.4 reads a message. One-to-one: 0.8 x 0.4 = 0.32 reads, 2 rows each, 0.64 rows.

Groups: 0.2 x 7 recipients x 0.4 = 0.56 reads, 1 row each, 0.56 rows. So 1.2 read rows.

In all 4.4 rows a message: 3.2 member rows and 1.2 read rows.

Step through the figure to follow one group message into three boxes.

The trade-off:

A message costs a row a member, plus read rows. In exchange, a device catches up with one question, whatever it missed.

One group message into three boxes

1. Every user has one box: an ordered list of rows numbered 1, 2, 3, for all their chats.
2. A sends to a group of three. The API takes the conversation's next number, n+1, and the next number in each member's box.
3. One transaction writes a row into each member's box, the sender's included. Each row carries its box number and the conversation number.
4. When B reads the chat, a read marker goes into B's own box. A group writes no receipt.
5. B's laptop, back online, asks one question: the rows after b. It gets the message and the marker, and clears its badge.

Check in: Why does each row carry two numbers?

Reading the result tiles

Each stage has a live canvas and four result tiles. Four things hold on every stage.

Load: the buttons count messages sent per second on Stages 1 to 3, and device connects per second on Stages 4 and 5. The tile labels both req/sec.

A card turns red at 90% of its limit. A part saturates past 100%, and only then does the result name it; otherwise the last tile notes the busiest part.

Latency is server time on the slowest path. The tile treats a part's calls as parallel, so a chain of calls reads short; each stage gives the chain.

Only cards that model a queue raise their latency near capacity. The containers, the database, the load balancer and Valkey start to queue over 70% busy.

At full use they reach 4 times their base latency.

The channel table that Stage 3 adds prints a fixed 2 ms at any load. Really, a throttled table call fails at once and is retried with backoff.

An overloaded node queues without bound. Past saturation every tile is a floor, which is why it prints "≥".

Cost: each card bills its peak load all month; the bill beat uses the average month.

The cost tile also bills each stage's replies sent to phones, per GB after 100 GB a month free. Data coming in is free.

Frames pushed down held connections are not replies to a request, so the bill beat prices them by hand.

Where a card reads differently from AWS, the stage that shows it says so.

Check in: A latency tile reads 3ms. Is that what a sender waits to see "sent"?

Small chats, a box for every user

Stage 1 is one-to-one and group chat only, at the evening peak of 6,000 messages sent a second.

The phone posts each message through an ALB, AWS's application load balancer, to the Message API.

Each message carries a client id, a value the phone makes for it, so the API can recognise a retry.

The API checks the client id, takes the numbers from Valkey and inserts the 4.4 rows in one transaction. Then it answers "sent" and sends frames.

The API takes about 1 ms of CPU a message (our estimate). Each 1-vCPU task serves 1,000 a second, 700 at its 70% target: 9 tasks, at 66.7%.

Read actions arrive on the WebSocket, and the gateway passes them to the API. Their CPU is in the 1 ms, and their rows are in the 4.4.

Sessions and counters is Valkey, a primary and a replica. Each message needs 11 commands: 66,000 a second, 41.3% of the card's 160,000.

That is the client id, the conversation number and a group's member list. Then a box number and a device lookup for each of the 4.4 rows.

The card is a cache with every read a hit, standing in for a store.

Its 160,000 commands a second are AWS's measured figure for one core of Graviton2, AWS's own processor.

This node runs the newer Graviton3, so it does at least that. Only Stage 2 passes it.

Message boxes is Aurora, AWS's managed PostgreSQL: a writer and two readers, one in each zone.

Our CPU model is about 0.1 ms a row, so a 4-vCPU instance serves 40,000 a second. 6,000 x 4.4 = 26,400 rows a second puts the writer at 66.0%.

Real write throughput is also bound by commit and log latency, which AWS publishes no figure for.

The readers sit at 0% here: they serve devices catching up on reconnect, which Stages 4 and 5 count. Storage is priced by hand in the bill beat.

Nothing saturates; the note reads "Busiest: Message API at 67%". The cost tile shows $11k/mo, $11,358.21.

Of that, $3,446.54 is replies to senders. A reply is the stored message's numbers, about 0.3 KB (our estimate).

The load balancer card assumes 0.6 new connections a call. Each new one starts with a 4 KB TLS handshake, mostly the server's certificates.

So a reply averages 0.3 + 0.6 x 4 = 2.7 KB. A phone that keeps its connection between sends gets only the 0.3 KB.

Press 3,000, the daily average: the API runs 5 tasks, the writer is at 33.0%, and the cost tile shows $9,099/mo.

Press 10,800: a year on, the evening peak is 1.8 times today's. The result names the Message boxes. That preset is your turn, at the end of the page.

The trade-off:

Every message writes a row for each member, plus read rows. In exchange, every device syncs with one cursor.

Stage 1: Small chats, a box for every user

• Load: 6,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $11k/mo

Check in: Why does a group message write a row in the sender's own box?

What a send costs the gateways, and the sender

On Stage 1 the gateways are the largest line on the cost tile, $5,033.66 of $11,358.21, 44%. Their card reads under 1%.

Each message makes 1.888 frames on average: to the recipients' devices online and to the sender's other devices. The message chain after Stage 3 derives it.

That is 11,328 frames a second. The Gateways card counts CPU only, about 0.05 ms a frame, so it reads under 1%.

Each task holds 33,333 connections, about a third of its memory, and the card cannot see memory. The fleet is bought for connections.

The latency tile shows 3ms: 1.5 ms at the load balancer, 1 ms in the API and 0.2 ms on Valkey. That is 2.7 ms.

A send is really a chain: Valkey, about 0.3 ms, then the insert and its commit, which waits for Aurora's storage. About 5 ms of server time.

A phone adds its mobile round trip, 50 to 150 ms, before it shows "sent".

Delivery is async, so no tile shows it. A frame reaches an online recipient's gateway in about 1 ms inside the Region.

The trade-off:

Delivery waits on nobody, so the sender sees "sent" as soon as the rows commit. The gateways cost the most and do the least CPU work.

Check in: The gateways are the largest line on the cost tile and read under 1%. Are they too big?

Retries, and a gap in the numbers

Phones retry on bad networks, and servers crash between steps. A sync cursor is only useful if a device can trust its numbers.

Before taking any number, the API stores the client id in Valkey only if it is absent. The key holds "pending" and the time, for 10 minutes.

A retry finds the key. If the message is already stored, the API returns it. If not, and the key is over 10 s old, the API takes the send over.

Otherwise it answers "in progress, retry". A retry never takes numbers, and a unique key on the sender and client id backs the check.

The API takes both numbers in one Valkey round trip, then has 10 s, the send deadline, to commit. The deadline is the API's clock plus 10 s.

The rows go in with one statement that checks the deadline by the database's own clock. Its statement timeout is set to the time remaining.

It commits in that same statement, so a paused API task cannot commit at 15 s: its insert inserts nothing.

So box numbers are dense, not gap-free: each box counts up by one, but a number taken for a send that fails is never used.

Commits can also land out of order, and a frame can be lost in a crash. So a device can see 11 before 10.

A device acks, or confirms, each frame by box number. It keeps its cursor at the last unbroken number, here 9.

On a gap it waits 0.5 s, then asks for the rows after 9, from the writer if a reader comes back short.

That is getDifference-style gap fill, after the call Telegram's clients make on a gap. Gaps are asked for, never guessed.

A number still missing after 30 s, three times the send deadline, can no longer commit. Only then does the device move its cursor past it.

Amazon Time Sync, AWS's clock service, keeps instance clocks far closer than the 20 s between the deadline and the device's give-up.

A Valkey failover can hand out a number again. The insert then fails on the box's key, and the API re-reads that box's last number and retries.

Step through the figure to follow one gap.

The trade-off:

A device sometimes waits half a second and asks again. In exchange, it never shows a hole, and never skips a message that is only late.

Filling a gap

1. The device holds rows up to 9, so its cursor is 9.
2. Row 11 arrives first. The cursor stays at 9, the last unbroken number.
3. After 0.5 s the device asks for the rows after 9. If a reader comes back short, it asks the writer.
4. If 10 committed late, it comes back in the answer, and the cursor moves to 11.
5. If 10 is still missing after 30 s, three times the send deadline, it can never commit. The cursor moves past it.

Check in: A device has rows up to 9 and receives 11. What does it do, and when does it give up on 10?

Communities fanned out into every box

The interviewer asks: We add communities: channels with thousands of members.

Stage 2 keeps the 6,000 messages a second. One in ten is now a post in a community: a channel inside a group of thousands.

A post reaches 2,000 channel members on average, and communities average 5,000 members.

The API treats a post as a big group: a row in each of 2,000 boxes, with a number and a device lookup for each.

Members also mark a channel read when a visit finds new posts, about 6 times a day a daily user.

20M x 6 = 120M markers a day, over 25.92M posts a day: 4.63 markers a post.

So a post brings 2,004.63 rows, and the mix makes 0.9 x 4.4 + 0.1 x 2,004.63 = 204.423 rows a message.

At 6,000 that is 1,226,538 rows a second. The Message boxes saturate, and the warning reads "Writer 3066% used: 1,226,538 writes/s against 40,000".

Commands reach 411.126 a message, 2,466,756 a second. Sessions and counters saturates too.

Its warning reads "cache.r7g.large: the busiest primary runs 2,466,756 of its 160,000 ops/s (1542%)".

A second warning reads "Primary sends 10.10 Gbps against cache.r7g.large's 0.937 Gbps baseline".

That counts the card's 0.5 KB value for each command. The baseline is the rate a node can hold all the time; it bursts higher only for minutes.

Both limits are short: the command limit 15.4 times (2,466,756 / 160,000), the network 10.8 times (10.10 / 0.937).

AWS's own figure is over 1M commands a second a node, but only for nodes 8 or more times this size. Either way, 2.47M is too many for one node.

Valkey would need cluster mode, which splits the keys over nodes called shards. It takes 1 to 500 shards.

At about 1M a node, 2.47M needs at least 3 shards. On this node size it needs 16, at 160,000 each.

Nothing is refused. Inserts queue on the writer and time out, and senders retry.

The latency tile shows "≥ 3ms", a floor. The Valkey card has reached 4 x 0.2 = 0.8 ms, slower than the database card's 4 x 0.1 = 0.4 ms.

So the tile is 1.5 + 1 + 0.8 = 3.3 ms.

Each post also costs the API 4,012 commands and 2,005 rows to prepare, which the API card's 1 ms a message does not show.

The cost tile still shows $11k/mo. The bill does not move, and the database is 31 times short.

Press 3,000, the daily average: 613,269 rows and 1,233,378 commands a second. Both stay named.

Every post costs 2,005 rows, and an edit touches 2,000. A new member of a busy community would need its history copied into their box.

The trade-off:

Fan-out on write costs members times posts. For the same messages, rows grow 46 times.

Stage 2: Communities fanned out into every box

• Load: 6,000 req/sec
• First to saturate: Message boxes
• Estimated cost: $11k/mo

• Latency past saturation is a lower bound, not an estimate. Sessions and counters is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Sessions and counters: cache.r7g.large: the busiest primary runs 2,466,756 of its 160,000 ops/s (1542%).
• Sessions and counters: Primary sends 10.10 Gbps against cache.r7g.large's 0.937 Gbps baseline. It can burst to 12.5 Gbps for 5 to 60 minutes on network credits.
• Message boxes: Writer 3066% used: 1,226,538 writes/s against 40,000 on the writer alone (readers take no writes).

Check in: Why not shard the boxes by user over many databases?

Store once, deliver by subscription

A post to a big channel should cost a couple of writes, not a row a member. Every member online must still get it.

A group stops at 200 members. Anything over 200 is a community, the line Telegram draws between basic groups and supergroups.

A community's posts are stored once, in DynamoDB, AWS's key-value database. Each channel also has a head: one small item holding its latest number.

A post writes two items: the head, then the post.

A ring of community servers owns the communities. Each server owns a share, picked by hashing the community id, as Slack's channel servers own channels.

A server owns a community only while it holds a lease: a time-limited claim, renewed in a DynamoDB lease table. A lease lapses in about 10 s.

When a member's device connects, its gateway subscribes at the owner of each of that member's communities. One subscription covers every member on that gateway.

The owner numbers each post in memory, one order a channel, and stores it once. Then it sends one copy to each subscribed gateway.

Each gateway writes the copy to its members' sockets. That is fan-out on read: copies only for members online, made at the gateway.

A post reaches 600 of its 2,000 members online, 30%, on 1.2 devices each: 720 frames.

The 30% is weighted by posts: busy communities post more and have more members online.

The API writes a mention row into each named member's box, about 0.5 a post. "@everyone" is a flag on the post, never a row a member.

Step through the figure to follow one post.

The trade-off:

A ring of servers to run, with leases. In exchange, a post is two writes whatever the channel's size.

One post, stored once, delivered by subscription

1. A post goes through the API to the community's owner: one of 12 servers on the ring, 4 a zone.
2. The owner numbers the post p+1. It updates the channel's head, then stores the post: two items.
3. One copy goes to each gateway that subscribed for a member online. A gateway with no member online gets none.
4. Each gateway writes the copy to its members' sockets. An average post makes 720 frames from two stored items.

Check in: Where is the cut-off between copying into boxes and storing once?

Communities stored once, delivered by subscription

Stage 3 keeps the 6,000 messages a second, one in ten a post. Small chats are unchanged.

Posts: 6,000 x 0.1 = 600 a second to the Community servers. The ring is 12 tasks of 2 vCPUs, 4 a zone, at about 1 ms a post: 2.5%.

Channel messages has provisioned capacity: units bought ahead, by the hour. A write unit is one write a second of an item up to 1 KB. Read units do the same for reads.

600 posts x 2 items = 1,200 writes a second, against 2,000 write units: 60.0%.

Rows fall to 4.473 a message: small chats' 3.96, 0.463 read markers and 0.05 mention rows. That is 26,838 a second, the writer at 67.1%.

Commands are 11.026 a message, 66,156 a second, 41.3%.

Rows fell 46 times, from 1,226,538 to 26,838 a second. Frames did not: about 443,000 a second either way, 18.5% of the gateways' CPU.

The work that grows with members is now one frame a member online, spread over 120 gateways.

Nothing saturates; the note reads "Busiest: Message boxes at 67%". The Message API is at 66.7%.

The latency tile shows 6ms: 1.5 + 1 + 1 + 2 = 5.5 ms, a post through its owner and the channel table.

The owner's head update and the post's put run one after the other, 2 ms each.

So the server time is 1.5 + 1 + 1 + 2 + 2 = 7.5 ms.

The sender waits until the post is stored. Delivery stays async.

The cost tile shows $14k/mo, $13,675.59: the ring and the channel table add about $2,300.

The ring holds each active channel's recent messages and each community's members and roles: about 35 GB, mirrored to a standby in another zone.

That is about 70 GB of the ring's 96 GB (our estimates). Memory and zones size the ring; its CPU reads 2.5%.

Press 3,000, the daily average: the writer at 33.5%, the channel table at 30.0%, and the cost tile $11k/mo.

The trade-off:

Fan out on write while a group is small; store once and fan out on read when it is big. A post's rows no longer grow with its members.

Stage 3: Communities stored once, delivered by subscription

• Load: 6,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $14k/mo

Check in: What if one community has a million members online?

One owner for each community

An owner pauses, its lease lapses, and another server takes over. Then the first wakes, still sure it owns the community.

This is a question about one channel, not about load, so we reason by hand.

Each lease carries an epoch: a number that rises every time ownership moves. The head stores the epoch of the owner that last wrote it.

Before storing a post, the owner updates the head on two conditions. The stored epoch is not newer than its own, and the head holds the previous number.

The head's condition keeps one order: a post's number is accepted only if the head still holds the number before it.

So two owners can never write the same number or skip one.

The epoch adds a second guarantee once the new owner has written a channel's head: from then on the old owner cannot post there.

Until then a stale owner can still post the next number to an untouched channel. The post is stored in order.

It reaches only the gateways still subscribed to the old owner. The new owner finds it at its next head update, which fails, and sends it on.

Nothing is lost or reordered; at worst one post arrives late.

Bumping every head's epoch at takeover would close that window, at about 50 writes a community.

After a zone loss that is about 667,000 writes, 5.6 minutes of the table's 2,000 units: not worth it.

The post itself also carries a condition that its number is new, a backstop against a duplicate.

These are two plain writes of one unit each. A DynamoDB transaction would bill double: 2,400 units a second against the 2,000.

Posts are keyed by channel and bucket, the post's number divided by 50,000. That is about 30 MB of data, 35 MB stored with DynamoDB's 100 bytes an item.

Discord buckets messages by time instead.

On a move, gateways watch the epoch and resubscribe at the new owner. Its reply carries the heads, so an open channel fetches what it missed.

Each server keeps a warm standby of its communities in another zone. A new owner promotes it, and reads the heads in one batch only if it lagged.

The trade-off:

Every post pays a conditional write on its head. In exchange, every channel keeps one order, and a stale owner is fenced once the new owner writes.

Check in: What does the epoch add over the previous-number check?

The message chain on one screen

Skip on a first read; come back before the interview. Every number on Stage 3 comes from the one before it, at the 6,000 peak.

Messages: 6,000 a second, 90% small chats and 10% community posts.

Load balancer: 6,000 calls a second. An LCU is its billing unit; one covers 25 new connections a second.

The ALB card assumes 0.6 new connections a call, a fixed figure in its model.

Phones that keep their HTTPS connection open between sends open fewer, so the card's load-balancer bill is an upper bound.

6,000 x 0.6 / 25 = 144 LCU. At $5.76 an LCU-month ($0.008 an hour): $16.20 + 144 x $5.76 = $845.64.

Message API: 6,000 / 700 = 8.57, so 9 tasks, $319.92.

Commands: 6,000 x 11.026 = 66,156 a second, 41.3% of 160,000. $252.29.

Rows: 6,000 x 4.473 = 26,838 a second, 67.1% of the writer's 40,000. Three instances at $486.72 each.

Posts: 6,000 x 0.1 = 600 a second, 2.5% of the ring's 24,000. 12 tasks, $1,006.73.

Items: 600 x 2 = 1,200 a second, 60.0% of 2,000 write units. $1,310.65 with the 4,000 read units.

Small-chat frames: a recipient is online half the time at the peak, on 1.2 devices, so 0.6 frames each.

A one-to-one message makes 0.6 + 0.2 for the sender's other devices = 0.8 frames. A group makes 7 x 0.6 + 0.2 = 4.4.

Mixed: 0.8 x 0.8 + 0.2 x 4.4 = 1.52 deliveries a message.

A one-to-one read sends 0.8 frames, 0.2 to the reader's other devices and 0.6 to the sender's. A group read sends 0.2.

Read notices: 0.32 x 0.8 + 0.56 x 0.2 = 0.368 a message.

So 1.888 frames a small-chat message, and on Stage 3 small chats give 0.9 x 1.888 = 1.6992.

Read markers add 0.463 x 0.2 = 0.0926, and mention rows 0.05 x 0.6 = 0.03: 1.8218 frames a message.

Frames: 6,000 x 1.8218 + 600 x 720 = 10,931 + 432,000 = 442,931 a second, 18.5% of 2.4M. 120 tasks, $5,033.66.

Sizes in the chains are binary: a KiB is 1,024 bytes, and a GiB is AWS's billed GB.

Replies: 6,000 a second x 2.7 KiB, about 40,045 GiB in a 30-day month.

After the free 100 GiB, 10,240 bill at $0.09 and the rest at $0.085: $3,446.54.

Total: $13,675.59 a month at the peak, the tile's $14k. The rounded lines sum to the same.

Check in: Close the page and rebuild the chain from 6,000 messages to the writer's rows.

Reconnect sync asks every channel

The interviewer asks: A phone was offline for an hour. How does it catch up?

The page turns to connections. From here the entry counts device connects a second, not messages: 8,333 at the peak.

Held connections, heartbeats, presence (who is online) and pushes to offline phones have rates of their own. They are worked by hand after Stage 5.

On connect, the gateway registers the device, reads its box after its cursor if the box moved, and subscribes to its 4 communities.

Then, in this design, it asks each of the 100 channels the user can see for posts after the device's last number there.

That is 4 communities of 25 visible channels each (our estimates), asked whether or not anything changed.

The Gateways card now counts connects, a different unit on the same fleet. Each takes about 1 ms of CPU for TLS, the token check and the catch-up.

8,333 connects a second is 6.9% of the fleet's 120,000.

Valkey takes 3 commands a connect: register, read the box's latest number, and remove on close. That is 24,999 a second, 15.6%.

About 4 connects in 5 find a moved box (our estimate). So the readers take 6,666 reads a second, 8.3% of the two.

The Community servers take 4 subscribes a connect, at 0.1 ms each: 33,332 a second, 13.9%.

Channel queries: 8,333 x 100 = 833,300 a second, against 4,000 read units.

A strongly consistent query, one that sees every committed write, costs at least one unit even when it finds nothing.

The Channel messages table saturates. The warning reads "Read capacity 20833% used (provisioned)".

The latency tile shows "≥ 3ms", a floor: 1 ms at the gateway plus the table's fixed 2 ms.

The cost tile shows $18k/mo. About $8,743 of it is replies to phones: 8,333 connects a second x 5.5 KB.

A connect reply is the 4 KB TLS handshake plus about 1.5 KB of catch-up (our estimates).

Press 4,167, the daily average: 416,700 queries a second, still named.

This wall is a provisioned setting, so money fixes it. 833,300 read units at 70% is about 1,190,429 units, about $111,424 a month.

That is past the card's 40,000 maximum and AWS's default table quota, which AWS raises on request.

And each connect would still wait on 100 queries: about 10 rounds of 10 in parallel, roughly 50 ms (our estimate). Almost all find nothing.

The trade-off:

Asking every channel costs reads that grow with channels times connects, and almost every answer is "nothing new".

Stage 4: Reconnect sync asks every channel

• Load: 8,333 req/sec
• First to saturate: Channel messages
• Estimated cost: $18k/mo

• Latency past saturation is a lower bound, not an estimate. Channel messages is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Channel messages: Read capacity 20833% used (provisioned): 833300 of 4000 RCU/s at 833300 RPS.

Check in: Why not cache the channel table to answer these queries?

Read only what moved

A phone back from an hour offline needs two things: the rows it missed, and which channels have something new.

Small chats need one read: the device sends its last box number and gets every row after it.

Its read markers and mention rows are rows in that same box, so they arrive with that read.

For communities, the subscribe reply carries each visible channel's head, from the owner's memory.

The app compares each head with its read marker for that channel. A head past the marker means unread; a mention row means a mention.

So the app marks unread channels and mentions without reading any channel. A channel's messages load only when the user opens it, from the owner's memory.

The order on connect is the correctness: register, then read the box, then subscribe.

Anything stored after the registration arrives live. Anything stored before it is in the read.

A reader lags the writer by a few ms. So the read asks for rows up to the latest number in Valkey, retries once if short, then reads the writer.

Step through the figure to compare one reconnect before and after.

The trade-off:

The app works out its own unread counts. In exchange, a reconnect reads one box and no channel.

One reconnect, before and after

1. Stage 4: the gateway reads the box, subscribes to 4 communities, then asks each of 100 visible channels for new posts.
2. Almost every query finds nothing, and each still costs a read unit. At 8,333 connects a second that is 833,300 queries.
3. Stage 5: register, read the box after the cursor, then subscribe. The order means no message falls between them.
4. Each subscribe reply carries the channels' heads. A head past the read marker means unread, and no channel is read.

Check in: Why must the gateway register the device before it reads the box?

Sync from the box, community heads from memory

Stage 5 carries the same 8,333 connects a second. The subscribe reply now carries the heads, and no channel is queried.

The box sync is unchanged: one read, only if the box moved. The readers stay at 8.3%.

The Community servers take the same 33,332 subscribes a second, 13.9%, and now answer each with heads from memory.

Channel messages shows 0%. Its edge is marked async and carries no connects: scroll-back and takeovers only, and no connect waits on them.

Valkey takes Stage 4's 24,999 commands a second, 15.6% of 160,000.

Nothing saturates; the note reads "Busiest: Sessions and counters at 16%".

The latency tile shows 1ms: 1 ms at the gateway plus 0.2 ms on Valkey, with the calls treated as parallel.

A connect is really a chain: TLS and the token, Valkey, the box read, then the subscribes, about 0.5 ms a hop. About 3 ms of server time.

A phone sees about 150 to 450 ms. TCP and TLS 1.3 take two mobile round trips of 50 to 150 ms, and the catch-up takes a third.

The cost tile shows $18k/mo, the same as Stage 4. The table keeps its units for scroll-back, so the fix itself costs nothing.

Press 4,167, the daily average: every card is under 10%.

The trade-off:

Unread state lives in the app and the box. In exchange, a reconnect costs one box read and 4 subscribes, whatever the number of channels.

Stage 5: Sync from the box, community heads from memory

• Load: 8,333 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $18k/mo

Check in: What if a device was offline for a month?

Presence by subscription

Users want to see which friends are online. Telling everyone about every change would swamp the gateways.

Presence is whether a user is online, away or offline. Its frames share the gateways with messages, so we size it by hand.

The heartbeat ends at the gateway: 133,333 a second at the peak, about 2.7 vCPUs, written nowhere.

Every 30 s sits under the NLB's 350 s idle timeout and most mobile network timeouts.

A user comes online when their first device connects, and goes offline 30 s after the last one closes. The app reports away after 10 minutes idle.

A presence session lasts about 12 minutes and makes 2 changes: 2 x 3.33M / 720 s = 9,259 changes a second at the peak.

Broadcast, the naive answer, tells every online user who shares a chat or community. Small-chat partners give 24 frames a change: 40 x 0.5 online x 1.2 devices.

Communities give 1,608: 4 x 5,000 members x 6.7% online x 1.2. That is 1,632 frames a change.

There are 200M memberships: 40,000 communities x 5,000 members, counting members who are not daily users. 3.33M online users x 4 / 200M = 6.7%.

That makes three online shares on this page, kept apart:

Recipients in an active small chat: 50%. People mid-conversation are online.

Members a community post reaches: 30%, weighted by posts. Busy communities post more and have more members online.

All community memberships: 6.7%, used here for presence.

1,632 x 9,259 = 15.1M frames a second, about 755 vCPUs: 6.3 times the whole gateway fleet.

Instead, a device subscribes to the users it is showing, about 25. It sends the set when the view settles, at most once every 30 s.

4M x 25 = 100M subscriptions over 3.33M online users, 30 a user. 9,259 x 30 = 277,778 frames a second, 11.6% of the gateways.

The subscriptions themselves add 4M / 30 = 133,333 frames a second. Frames a change fall 54 times, from 1,632 to 30.

Presence servers hold it in memory, hashed by user, as Slack's do: 4 tasks, about $142 a month. Gateways replay each device's set if one dies.

Big communities show an online count and the visible slice of members only.

Step through the figure to compare broadcast with subscription.

The trade-off:

A device must say what it is showing, at most every 30 s. In exchange, presence frames a change fall 54 times.

Presence: broadcast against subscription

1. A user comes online: one presence change. At the peak there are about 9,259 a second.
2. Broadcast tells every online user who shares a chat or community: about 1,632 frames a change.
3. At 9,259 changes a second that is 15.1M frames, 6.3 times what the whole gateway fleet can write.
4. Subscription tells only the devices showing that user: about 30 frames a change, 277,778 a second, 11.6% of the gateways.

Check in: Why not broadcast each change to everyone who shares a chat or community?

The connection chain on one screen

Skip on a first read; come back before the interview. Stage 5 at the 8,333 peak, plus what no card counts.

Connects: 4M devices / 480 s = 8,333 a second.

Gateways: 8,333 x 1 ms = 8.3 vCPUs busy, 6.9% of 120. 120 tasks, $5,033.66.

Commands: 8,333 x 3 = 24,999 a second, 15.6% of 160,000. $252.29.

Box reads: 8,333 x 0.8 = 6,666 a second, 8.3% of the two readers. Three instances at $486.72 each.

Subscribes: 8,333 x 4 = 33,332 a second, 13.9% of the ring's 240,000. 12 tasks, $1,006.73.

Channel queries: none. The table still costs $1,310.65 for its units.

Replies: 8,333 a second x 5.5 KiB, about 113,292 GiB in a 30-day month.

After the free 100 GiB: 10,240 at $0.09, 40,960 at $0.085 and the rest at $0.07, $8,742.64.

Total: $17,806.13 a month, the tile's $18k. The rounded lines sum to the same.

Held connections: 4M at about 40 KB each. A task holds 33,333 normally, about a third of its memory, and at most 50,000 after a zone loss.

Gateway CPU at the peak, every frame counted: message frames 22.1 vCPUs, presence 13.9, connects 8.3, presence subscriptions 6.7.

Heartbeats add 2.7, and the small frames that remain about 1.1.

In all 54.8 of 120 vCPUs, 45.6%.

The fleet is sized by memory and zones, with CPU to spare for a lost zone.

Check in: Close the page and size the gateway fleet from 4M connections.

Pushes to offline phones

A member with no connected device still needs to know. Their phone gets a push through Apple's or Google's service.

A push goes through APNs on iOS or FCM on Android: Apple's and Google's push services. They reach a phone that holds no connection to us.

One push goes to each member with no connected device: 0.5 for a one-to-one message, 7 x 0.5 = 3.5 for an average group.

Small chats average 0.8 x 0.5 + 0.2 x 3.5 = 1.1 a message. Community posts push only mentions, about 0.5 a post.

So 0.9 x 1.1 + 0.1 x 0.5 = 1.04 pushes a message, 6,240 a second at the peak.

The API puts each push on an SQS queue, AWS's message queue. 3 sender tasks, one a zone, read it in batches of 10 and call APNs or FCM.

FCM v1 has no batch send, so at the peak each sender makes about 2,100 calls a second: about 0.2 vCPU each (our estimate).

FCM takes 600,000 a minute a project, 10,000 a second. About half our pushes are Android: half of the peak's 6,240, about 3,100 a second, 31% of FCM's 10,000.

APNs publishes no rate.

Pushes collapse per conversation: the collapse id on iOS, the notification tag on Android. FCM keeps at most four collapse keys a device, so they are not used.

A push is useful for minutes. If APNs or FCM throttles, SQS holds the pushes, and the sender drops old ones.

Average month, by hand: 3,120 pushes a second on average, all platforms, plus 624 batched receives and deletes. That is 3,744 SQS requests a second.

They cost $3,881.38, plus $106.64 of senders: $3,988.02. Sending in batches of 10 from the API too would make it about $1,077.

The trade-off:

In month one, pushes cost more than any line but the data sent to phones and the gateways. Batching the queue writes cuts them about 3.7 times.

Check in: Why collapse pushes per conversation, and why not with FCM's collapse keys?

Lose a zone at the peak

An availability zone fails at the evening peak. A third of the gateways, community servers and database instances go at once.

The final design, Stages 3 and 5, by hand.

Gateways: 80 survive and take all 4M, 50,000 each.

With client IP preservation off, an NLB supports about 55,000 simultaneous connections, or about 55,000 a minute, to each unique target.

Client IP preservation passes the phone's address through to the task; it is off by default for Fargate's targets.

Holding 50,000 a task, the simultaneous limit is the tight one, so each task registers on two ports.

Devices reconnect with jitter, random delays that spread them over 2 minutes: 1.33M / 120 s = 11,111 a second, plus the usual 8,333.

That is 19,444 connects a second. Every frame counted, the survivors run 65.9 of 80 vCPUs, 82.4%, for those 2 minutes.

TLS session resumption reuses a previous session's keys, so a reconnect costs about 0.4 ms (our estimate). That brings the survivors to 67.8%.

Every reconnect lands on a different gateway, so resumption needs one shared, rotated ticket key across the fleet.

It saves the certificate signature, so the 0.4 ms holds only if the 1 ms was mostly that signature.

An ECDSA P-256 certificate, a type with cheap signatures, is the simpler lever. Or 150 tasks hold it at 65.9%, $1,258.42 a month more.

Aurora: losing a reader's zone, the other reader takes the reads. Losing the writer's zone, a reader is promoted, with about 30 s of failed sends.

Either way the writer carries 26,838 rows a second, 67.1%. One reader carries 10,032 reads, 25.1%, with the moved boxes and counter rebuilds.

The 30 s of failed sends leave about 805,000 rows to retry. Phones retry with jitter over 2 minutes, so the writer runs at 83.9% for those 2 minutes.

Retried over 60 s, it would reach 100.6%.

Community servers: 8 of 12 survive and promote the standbys of the lost third, 13,333 communities, with no reload from the database.

Their CPU, with the resubscribes, is 10.3 of 16 vCPUs, 64.1%. Their memory is about 91% full, tight, until the zone returns.

Step through the figure to watch a zone go.

The trade-off:

Spare in every tier, a shared ticket key and jittered retries. In exchange, a lost zone costs reconnects and 30 s of retried sends, not messages.

A zone fails at the peak

1. 120 gateways, 40 in each zone, hold 4M connections at the peak: 33,333 each.
2. A zone fails. Its 40 gateways go, and 1.33M devices lose their connections.
3. Devices reconnect with jitter over 2 minutes: 11,111 a second, plus the usual 8,333.
4. The 80 survivors hold 50,000 each. Their CPU runs at 82.4% for 2 minutes, 67.8% with TLS session resumption.

Check in: A zone fails at the peak. Which tier runs hottest, and what brings it down?

Data to phones, connections and storage are the bill

Skip on a first read; come back before the interview. The canvases bill the peak all month, and leave out storage, the NLB and pushes.

They also leave out the frames sent down held connections.

The bill is today's load held flat, an average month at AWS prices. Storage sizes here are binary: a GiB is AWS's billed GB.

The nodes on Stage 3 come to $9,672.15 at the average. The tile at 3,000 adds $1,744.62 of replies, counted in the data line.

The NLB bills processed bytes: about 179 MB a second on average, $2,607.21. TLS on the NLB instead would cost $2,896.20, a near tie.

Pushes are $3,988.02, and the presence servers $142.19.

Every byte sent to phones and to the push services is billed. On average that is about 172.9 MB a second, outbound only.

Message frames are 221,465 a second x 512 bytes, 113.4 MB. Presence frames add 20.8 MB, and heartbeat replies 5.3 MB.

Connect replies add 23.5 MB, send replies 8.3 MB, and push calls 1.6 MB, at 0.5 KiB a call (our estimate).

Frames coming in, and the phones' share of each handshake, are data in, which is free.

Over a 30-day month the data out is 417,414 GiB, one price-tier walk for the whole account.

After the free 100 GiB: 10,240 at $0.09, 40,960 at $0.085, 102,400 at $0.07 and 263,714 at $0.05. That is $24,756.88.

Community posts are most of it: 300 posts a second x 720 frames x 512 bytes = 110.6 MB a second.

The lever is Discord's: send a post's body only to devices viewing that community, and a head number to the rest.

Box rows: 13,419 a second x 250 bytes, about 8,098 GiB a month. Channel items add 506.9 GiB a month.

Each month bills the storage held at its middle, so month one bills half a month of rows.

Month one, on I/O-Optimized: box storage $911.06 and channel storage $63.37. The month comes to $42,140.88.

Aurora I/O-Optimized storage costs $0.225 a GB-month and nothing per I/O. Standard costs $0.10 a GB-month plus $0.20 a million I/Os.

Our I/O estimate on Standard is $3,947.62 a month. The I/O-Optimized premium is $336.96 on the instances plus $0.125 a GB-month.

So I/O-Optimized pays only under about 28,900 GiB, about month 4, and the boxes switch to Standard. AWS limits how often a cluster switches.

Month twelve averages 11.5 months of rows: 93,131 GiB of boxes. On I/O-Optimized that storage alone would cost $20,954.43.

With the boxes on Standard: storage $9,313.08 and I/O $3,947.62. The instances drop to $374.40 each, $1,123.20 for three.

Channel storage that month is $1,457.45.

Month twelve comes to $55,547.64, against $63,578.33 kept on I/O-Optimized: Standard saves $8,030.69.

Data to phones is then 44.6% of the bill, storage 19.4%, and the gateways and NLB 13.8%.

At flat load, Aurora's 256 TiB volume fills in about 32 months. But the challenge says the peak grows 1.8 times a year.

At that rate, about 5% a month, the boxes hold about 129,000 GiB after a year.

Month twelve then averages 122,100 GiB, about $12,211 a month on Standard.

Aurora's 256 TiB then fills in about 20 months, not 32. Shard by user within the first year and a half.

Older rows can move to S3, AWS's object storage, at $0.023 a GB.

The managed alternative, API Gateway WebSockets, bills every frame both ways and every connection-minute: about $946,665 to $1,177,681 a month.

Its quotas bite first: 10,000 calls a second across the account, against 733,134 frames out at the peak, 73 times.

New connections are capped at 500 a second, against 8,333. Both quotas are adjustable, and connections end at 2 hours.

The trade-off:

Data to phones, held connections and history are the bill, not messages. Send post bodies only where they are read, and switch storage type once growth outweighs the I/O.

Check in: In month twelve, which line costs the most, and what shrinks it?

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a user see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

A gateway dies: its 33,333 devices reconnect elsewhere. Its registry entries are ignored once its liveness key expires, after 10 s.

A community server dies: its lease lapses in about 10 s, and a standby takes over. Gateways resubscribe, and open channels fetch what they missed.

Valkey fails over: numbers may repeat, so box inserts fail on their key and retry.

Gateways re-assert their devices: 4M commands / 160,000 a second = 25 s at most.

They send in batches, and the 160,000 was measured without them, so it goes faster. No phone reconnects.

The Aurora writer fails over: sends fail for about 30 s. The phone keeps the message and retries with its client id.

A deploy drains each gateway over about 10 minutes with jittered reconnects: about 6,700 more connects a second, inside the zone-loss figures.

us-east-1 is lost: everything stops. A second Region with its own boxes and ring takes the reconnects.

Abuse: about 10 sends a second a user at the API, and slow mode, a limit on how often each member may post.

A hot channel: one DynamoDB partition, the storage unit for a key, takes 1,000 writes a second. Nobody reads that fast, so the owner caps a channel at 200 posts a second.

Edits and deletes: small chats write a row of kind edit or delete in each member's box, so devices apply it in order.

Typing indicators are frames to members with the chat open, at most one every 3 s a typist, never stored.

A delivered tick is a frame to the sender's devices when a recipient's first device acks, never stored. Only one-to-one chats send them.

On Stage 3 the one-to-one share is 0.9 x 0.8 = 0.72.

So ticks are 6,000 x 0.72 x 0.5 online x 1.2 devices = 2,592 a second.

End-to-end encryption, as in WhatsApp and Signal, changes the store. The server keeps ciphertext for each device until delivered, then deletes it.

A group message is encrypted once with the sender's group key and fanned out per device.

A new device has no history without a transfer from another device or an encrypted backup. The server can no longer search, moderate or preview.

The box model still fits, as a queue per device.

At 100 times, 600,000 messages a second make 2.68M box rows a second, so the boxes shard by user over dozens of clusters.

400M connections need about 12,000 gateway tasks, past ECS's 5,000 a service: several services, or large EC2 hosts holding a few hundred thousand each.

Presence and community servers scale out by hash, and each user is homed in one Region.

Say it out loud: "Small chats fan out on write into per-user boxes with two numbers. Communities are stored once and delivered by subscription."

And: "A reconnect reads only what moved: one box, and the heads."

Minutes 0 to 5: requirements, and four rates at the peak: 6,000 messages, 8,333 connects, 9,259 presence changes and 133,333 heartbeats a second.

Then 4M connections held.

Minutes 5 to 13: connections and routing. Gateways sized by memory, the registry, and sends over HTTPS.

Minutes 13 to 22: Stages 1 to 3. Boxes, two numbers, gaps and fan-out on write; communities stored once.

Minutes 22 to 30: Stages 4 and 5. The cursor, the heads and the connect order.

Minutes 30 to 37: presence by subscription, pushes and read receipts.

Minutes 37 to 45: a lost zone, storage growth, end-to-end encryption, and what changes at 100 times.

The trade-off:

Each answer has a price: a ring with leases, a fleet sized by memory, and apps that work out their own unread counts. Copying every post into every box costs far more.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.