Design a nearby friends feature

Do the numbers before the boxes

An interviewer gives a feature and its users, not a load. Work out first what each phone sends, and how often.

A 45-minute version of this answer is at the end.

We assume one AWS Region, us-east-1. Every count on this page is our estimate unless sourced.

At the evening peak, 1.2M people have the map open.

Each phone takes a location every 30 s: 1.2M / 30 = 40,000 samples a second.

It sends its first fix, its first location reading, when the map opens. After that, it sends a sample only after a move of 50 m or more.

One sample in four moves that far. So the phones send 40,000 / 4 = 10,000 location updates a second.

Each user sends an update every 1.2M / 10,000 = 120 s on average.

Each user has 200 friends who share a location with them, on average. At the peak, 20 of them are online, and about 4 are near enough to show.

That 200 is a Snap-like friend list. Life360 circles and Find My shares hold a handful of people; a later beat says what changes then.

Every stage runs at this peak: 10,000 updates sent a second.

Check in: Close the page and derive the 10,000 updates a second.

Polling every 30 seconds

Stage 1 is the first idea: each phone asks the server for its friends every 30 s. That is polling.

The poll goes to a Location API on Fargate, AWS's managed containers. Each task is one container, here with 2 vCPUs, virtual processors.

A poll costs about 1 ms of CPU (our estimate). So a task serves 2 x 1,000 / 1 = 2,000 polls a second, and 30 tasks 60,000.

Every phone polls every 30 s: 1.2M / 30 = 40,000 polls a second, 4 for each sent update. The API is at 40,000 / 60,000 = 66.7%.

Positions live in Valkey, an in-memory key-value store forked from Redis, which AWS runs as ElastiCache. It runs in store mode: nothing is evicted.

Each user has one key, loc:{user}: the latest position and its time, with a 1-hour lifetime, or TTL.

A poll writes the phone's own key if it moved. It reads all 200 friends' keys in one pipeline: commands sent together, one round trip.

That is 200 GETs, plus a SET for one poll in four: 200.25 commands. The store gets 40,000 x 200.25 = 8,010,000 commands a second.

Cluster mode splits the keys over 4 shards. Valkey hashes each key name to a hash slot, and each shard owns a set of slots.

Each shard is a primary node and a replica, a copy in another zone that takes over if the primary fails.

The card rates each cache.m6g.xlarge at 348,000 commands a second, so 4 shards carry 1,392,000. Graviton is AWS's ARM processor, and this node has 4 Graviton2 vCPUs.

AWS measured 348,550 a second on a cache.r6g.xlarge, with the same vCPUs. The test ran Redis 7.0.5, 2,000 clients and 80% GETs at 512 B (AWS Database Blog, February 2023).

The result names Latest locations (Valkey): 8,010,000 against 1,392,000 is 575%. A second warning shows each primary sending 1.64 Gbps, past its 1.25 Gbps network baseline.

A poll reads every friend, online or not, to find the few that moved. A store big enough needs 29 shards, at 79.4%: $9,922.18 a month for the store alone.

And polling is slow. A poll every 30 s adds up to 30 s to a position's age. Polling every second, for a 1 s target, is 30 times this load.

The latency tile shows "≥ 2ms": the API's 1 ms, then Valkey. Up to 70% busy Valkey counts 0.2 ms; past that the time rises, to 4 times at full use.

So the tile adds 1 + 0.2 x 4 = 1.8 ms. That is a floor: once a part saturates, the real wait is longer.

The cost tile shows $11k/mo: the API $2,132.78, the store $1,368.58, and data out to the internet, called egress, $7,733.59.

On the canvas, each GET counts as one call. AWS's figure already includes the node batching many clients' commands, so our pipeline saves round trips, not engine work.

Stage 1 has no load buttons. In the editor, past 15,000 updates a second the API saturates too, and passes on only 60,000 polls.

The trade-off:

A poll is simple and holds no connection. It reads every friend to find about 5 that moved, and it adds up to 30 s of age.

Stage 1: Polling every 30 seconds

• Load: 10,000 req/sec
• First to saturate: Latest locations (Valkey)
• Estimated cost: $11k/mo

• Latency past saturation is a lower bound, not an estimate. Latest locations (Valkey) is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Latest locations (Valkey): cache.m6g.xlarge ×4 shards: the busiest primary runs 2,002,500 of its 348,000 ops/s (575%).
• Latest locations (Valkey): Primary sends 1.64 Gbps against cache.m6g.xlarge's 1.25 Gbps baseline. It can burst to 10 Gbps for 5 to 60 minutes on network credits.

Check in: A poll reads 200 keys. About how many changed since the last poll?

Reading the result tiles

Stage 1 showed a canvas and four result tiles. Four rules hold on every stage.

Load: every stage runs at 10,000 sent updates a second, and the tile labels them req/sec. No stage has load buttons; each stage says why.

A card turns red at 90% of its limit, and the Valkey and Fargate cards also warn over 80%. A part that saturates, past 100%, is named in the result.

Otherwise the last tile notes the busiest part.

Latency is server time on the slowest path. Past saturation it is a floor, which is why it prints "≥". A move takes far longer to reach a friend.

Cost: each card bills its stage load all month. A release lasts minutes, so Stage 4 prices what a release adds.

Each stage ends with a paragraph that starts "On the canvas". It says where a card counts differently from AWS, and what the editor shows at another load.

Check in: Stage 1's latency tile shows "≥ 2ms". How long can a friend's move take to show on your map?

Why push, and when polling wins

The interviewer asks: Polling wastes reads. Push the updates instead.

A poll reads every friend to find the few that moved. Push turns this round: a move goes out once, at once, to the friends who need it.

Push sends only moves. A move reaches a near friend within about 1 s of the phone sending it.

The price is a connection held open for every phone with the map open: 1.2M at the peak.

And a fan-out: each move is copied to the servers of every online friend.

Polling wins when views are rare beside changes, or when the client cannot hold a connection.

A "where are my friends" list opened a few times a day is one read of 200 keys, with no connection held.

Here the map stays open and friends move while you watch, so push wins.

Say it like this: "A poll reads 200 keys to find about 5 that changed. Push sends only the moves, within a second."

Step through the figure to compare a poll with a push.

The trade-off:

Push holds 1.2M connections and needs a fan-out. In exchange, no read is wasted, and a move shows at once.

A poll against a push

1. A poll reads all 200 friends' keys. About 5 changed since the last poll.
2. Push sends only the move: one write and one publish, delivered to the 20 online friends' gateways.

Check in: When would you choose polling for this feature?

One update, from phone to friends

Push needs a path from one phone to the servers of its online friends. Walk one sent update through its states.

Connected. Each phone holds a WebSocket, a connection that stays open both ways, to a gateway task on Fargate.

There is no load balancer. A small directory service, not on the canvas, gives each phone the address of a gateway that is accepting phones.

Sent. After a move of 50 m, the phone sends the update over its WebSocket, with a sequence number. It gets no reply.

Stored. The gateway writes the user's key: SET loc:{user} with the position, the time and the gateway, EX 3600, the 1-hour TTL.

Published. Then it sends SPUBLISH loc:{user}, the sharded publish, with the position. Each user has one channel, named after the user.

Sharded pub/sub sends a message only to the shard that owns its channel. Classic PUBLISH in cluster mode goes to every node.

Delivered. A gateway subscribes once to a channel for all its users who follow that user. So a message reaches about 19 distinct gateways, counted as 20.

Filtered. Each receiving gateway keeps the last position of every channel it follows, about 30 a user. It checks the share and the distance for each local follower.

Shown. A frame goes to each friend who is near: about 4 frames from 20 deliveries.

When its own user moves, the gateway filters again from memory. It reads nothing.

A task holds 5,000 connections (our estimate): 1.2M / 5,000 = 240 tasks. At about 40 KB a connection, that is 200 MB of a task's 2 GB.

Step through the figure to follow one update.

The trade-off:

Each update costs one SET and one publish to 20 gateways. In exchange, no gateway reads anything to filter.

One update, from phone to friends

1. The phone sends a move of 50 m or more over its WebSocket. It gets no reply.
2. The gateway writes the user's key, with a 1-hour TTL.
3. SPUBLISH goes only to the shard that owns the user's channel.
4. The shard delivers to about 20 gateways, one for each online friend.
5. Each gateway checks the share and the distance. About 4 friends get a frame.

Check in: Why does the distance filter run on the subscriber's gateway, not the sender's?

Why no load balancer, and when an NLB wins

Most designs put a load balancer in front of the gateways. This one does not, and an interviewer will ask why.

The reason is control of the release. Our deploy script decides which tasks stop, when, and how fast their phones move.

And the directory already places phones on gateways, so nothing else needs to.

Going without has costs. Each task needs a public address: 240 x $0.005 an hour x 720 hours = $864 a month.

Each task also needs its own TLS certificate, the directory is a service to run, and a flood of connections lands on the tasks.

An NLB, AWS's network load balancer, would cost about $450 a month here. The tasks would then need no public addresses, so cost is not the reason.

An NLB, or Global Accelerator in front, wins in most production designs. It gives one stable address, and TLS can end there.

Global Accelerator also puts AWS's edge network in front, so a flood meets the edge first, not the tasks.

And an NLB drains a stopping task for you. ECS waits out its deregistration delay, 300 s by default and up to 3,600 s, before it stops the task.

In that time, short sessions end on their own.

The release stages hold behind an NLB too: pace the phones that are still connected when a task must stop.

The trade-off:

Without a load balancer we run a directory and pay for addresses. In exchange, the release is ours to pace.

Check in: An interviewer asks why there is no load balancer. What do you say?

Push, every sharing friend subscribed

Stage 2 draws push in its simplest form. On connect, a gateway subscribes to all 200 friends' channels, online or not.

Sessions are short: about 5 minutes (our estimate). So 1.2M / 300 s = 4,000 phones connect every second, 0.4 for each sent update.

A connect writes the user's key and reads the 200 friends' keys for the first map. The close writes the key once more: 202 commands a session.

The store gets 1 + 0.4 x 202 = 81.8 commands an update: 10,000 x 81.8 = 818,000 a second, 58.8% of 1,392,000.

The channels are a second Valkey cluster in pub/sub mode, which relays messages and stores none. It has 3 shards of the same node, each with a replica.

Each connect also publishes its first fix, so friends see the user arrive: 1 + 0.4 = 1.4 publishes an update.

On the card, each publish and its 20 deliveries cost 21 ops: 14,000 x 21 = 294,000 ops a second.

Subscribing is the big load. All channels in one SSUBSCRIBE, the sharded subscribe call, must share a hash slot. So 200 friends take about 200 calls.

A session subscribes to 200 channels and unsubscribes from 200: 400 calls. That is subscribe churn: 4,000 x 400 = 1,600,000 calls a second.

The result names Location channels (Valkey pub/sub). Its warning says the churn alone, 533,333 calls a second a shard, passes the shard's 348,000.

With the publishes, a shard would carry 98,000 + 533,333 = 631,333 ops: 181%.

Subscribing costs 5.4 times the fan-out: 1,600,000 against 294,000. And 180 of the 200 subscriptions wait on friends who are offline and send nothing.

Seven shards would carry it, at 77.8%, for $1,368.58 more a month.

The latency tile shows "≥ 2ms": the gateways' 1.37 ms, then the saturated channels at 0.2 x 4 = 0.8 ms. 2.17 ms is a floor.

The cost tile shows $17k/mo: the gateways $8,531.14, the store $1,368.58, the channels $1,026.43 and egress $5,587.96.

On the canvas, the gateways' 1.37 ms is all their CPU for one sent update. That is 0.57 ms for the update, and 0.4 connects at 2 ms each.

Those are our estimates. An update itself takes about 0.3 ms of server time: its frame, then a SET and an SPUBLISH round trip.

The card charges each delivery a full command, but a delivery is a write into a gateway's socket buffer. One vendor measured it at a quarter to a half of a GET.

The churn saturates the channels unless a subscribe call costs under 0.47 of a GET. The 1.6M calls are typed for this many users, so Stage 2 has no load buttons.

The trade-off:

Subscribing to every friend needs no list of who is online, and no messages between gateways. It costs 400 calls a session, and 180 of the 200 wait on offline friends.

Stage 2: Push, every sharing friend subscribed

• Load: 10,000 req/sec
• First to saturate: Location channels (Valkey pub/sub)
• Estimated cost: $17k/mo

• Latency past saturation is a lower bound, not an estimate. Location channels (Valkey pub/sub) is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Location channels (Valkey pub/sub): Subscribe churn of 533,333 calls/s a shard is above cache.m6g.xlarge's 348,000 ops/s, before any publish.

Check in: Each update goes to 20 gateways. What saturates the channels, if not the updates?

Subscribe only to friends who are online

Stage 3 subscribes only to friends who can send. A gateway must first learn which friends are online: that is presence.

The key now also says where the user is connected. Its gateway field is set at connect, and cleared at the close.

On connect, the gateway writes the user's key with the first fix and waits for the reply. Then it reads the 200 friends' keys, on the primaries.

A key with a gateway means online. The gateway subscribes to those 20 channels, and tells each friend's gateway, with the user's position, to subscribe back.

Friends who come online later subscribe both ways in their own connects.

At a close, the gateway unsubscribes its channels. It also sends "offline" to each gateway that follows the user, which marks the user "last seen".

That gateway unsubscribes the user's channel if no local follower remains. Every subscription is removed once: about 40 unsubscribes a session.

So a session costs 20 + 20 + 40 = 80 calls, not 400: 4,000 x 80 = 320,000 calls a second.

Messages go gateway to gateway, not through Valkey: 20 "subscribe to me" a connect, about 40 "offline" a close. That is 240,000 a second in all, small.

The first fix now travels in the connect SET and the gateways' messages, so a sent update is one publish again.

On the card, a publish and its 20 deliveries cost 21 ops, spread over 3 shards: 7 a shard. Subscribe calls spread the same way.

A shard carries 10,000 x 7 = 70,000 ops of publishes and 320,000 / 3 = 106,667 calls: 176,667 of 348,000, 50.8%.

The store is unchanged at 58.8%. Nothing saturates; the note reads "Busiest: Latest locations (Valkey) at 59%".

The fan-out is unchanged: each update still reaches the gateways of the 20 online friends.

Subscriptions fall too. A session holds about 30 channels on average, 1.2M x 30 in all: about 3.6 GB at about 100 B each (our estimate).

The latency tile shows 2ms: the gateways' 1.37 ms, then a Valkey call at 0.2 ms, 1.57 ms.

The cost tile shows $17k/mo, the same $16,514.10 as Stage 2: the change is code.

On the canvas, a delivery costs a full command. Even so, the channels stay under the 80% line while a subscribe call costs up to 1.95 GETs.

The subscribe calls are typed for this load, so they do not grow if you change the traffic, and Stage 3 has no load buttons.

And a pub/sub answer would rest on two costs nobody publishes, a call and a delivery. So you work the exercise by hand, then check it on the canvas.

The trade-off:

A connect now needs presence, a message to each online friend's gateway and a write before the reads. In exchange, churn falls 5 times.

Stage 3: Subscribe only to friends who are online

• Load: 10,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $17k/mo

Check in: A session now costs 80 calls. Where do they come from?

Two friends open the map at once

Online-only has a race. Two friends connect at the same moment, and each reads the other's key before the other writes it.

Then each sees the other offline, neither subscribes, and they never see each other.

The fix is an order: store, then load. Each gateway writes its own key, waits for the reply, and only then reads its friends' keys.

Each key has one primary, so a read of it sees every write that primary has acknowledged.

So at least one of the two reads sees the other's write. That gateway subscribes, and tells the other gateway to subscribe back.

Send the SET alone, first. In one pipeline with the reads, a cluster client would send it in parallel to another shard, and both could miss.

Reads go to the primaries. A replica can lag its primary, and a lagging read breaks the order. So the replicas take no reads; they are failover targets.

One gap is accepted. An update published between the read and the SSUBSCRIBE is lost: about 1 ms, so about 1 friend in 100,000 misses one update.

The next update corrects it.

The share list, who shares with whom, lives in a database behind a cache, not on the canvas. It too is read only after the own key is acknowledged.

Step through the figure to follow both connects.

The trade-off:

A connect waits one more round trip, and the replicas take no reads. In exchange, two friends who connect at once see each other, short of a store failover.

Two friends open the map at once

1. Read first, and both can see the other offline. Neither subscribes.
2. Write, wait for the reply, then read. At least one read sees the other's write.
3. That gateway subscribes, and tells the other gateway to subscribe back.

Check in: Two friends open the map at the same moment. How does each see the other?

Why per-user channels, and when cell channels win

A candidate can also cut the map into cells, with one channel a cell. Which channel fits this feature?

With a cell channel, a phone publishes to its own cell, and subscribes to its cell plus the 8 around it.

That suits "strangers nearby", where distance is the only filter and nobody follows anybody.

For friends, the filter is friendship. A per-user channel reaches only the 20 gateways that follow it.

A city cell holds thousands of strangers for every friend. A gateway would get all their moves, and drop nearly all of them.

Per-user channels pay in subscribe churn instead, which Stage 3 cut to 80 calls a session.

Say it like this: "Per-user channels, because the filter is friendship. Cells win for strangers nearby, where distance is the only filter."

The trade-off:

Per-user channels churn on every connect. In exchange, a gateway gets only the moves of people its users follow.

Check in: The product adds "strangers nearby". Which channels would you use for it?

Fewer friends, or longer sessions: where the design changes

Online-only earns its extra parts on a large friend list with short sessions. Change either, and the simpler designs come back.

Take N friends, 10% of them online, and everything else as on the canvases.

Polling costs the store 40,000 x (N + 0.25) commands a second. It stays under the 80% line up to 27 friends, at 78.3%, and warns at 28.

Polling still adds up to 30 s of age, whatever the list.

Subscribing to every friend costs 4,000 x 2N calls a second. On the 3 channel shards it reads 73.4% at 80 friends, and warns at 90, 82.4%.

So under about 80 friends, subscribe to every friend. It is simpler: no presence messages and no connect race.

At a Life360 circle of 10, every design here fits. The choice is then about freshness and battery, not load.

Session length moves it too. With hour-long sessions, every friend costs 1.2M / 3,600 x 400 = 133,333 calls a second, under the 210,000 ops of updates.

But it still holds 1.2M x 200 = 240M subscriptions. At about 100 B each (our estimate, uncited), that is 24 GB, about 22.4 GiB.

That fits 3 shards, at 77% of their 29.1 GiB usable, once AWS reserves 25% of each node's 12.93 GiB.

It fits with little room: for growth, for the connections' output buffers, or for an error in the 100 B.

With 5-minute sessions and 200 friends, online-only wins 5 times on calls and 6.7 times on memory.

The trade-off:

Online-only buys 5 times fewer calls with presence and a race to handle. On a small list, that trade is not worth it.

Check in: Your interviewer says each user shares with 10 friends. What would you change?

A release restarts a third of the gateways

The interviewer asks: You ship a new gateway build. What happens to the map?

Stage 4 is Stage 3 at the same peak, during a release. A release adds no users, but every phone on a stopped task must connect again.

The gateways run as two ECS services, one on the old build and one on the new. ECS is the AWS service that runs Fargate tasks.

Our deploy script raises the new service by 80 tasks and waits until they run. Then it lowers the old service by 80 at once, a third of the fleet.

The script never changes the old service's task definition. That would start ECS's own rolling update, which at its default settings can replace all 240 at once.

This is our design, not tested. The script sends nothing before SIGTERM, the signal that tells a task to stop.

So each stopping task closes its 5,000 connections at once. The phones retry spread over 30 s (our client), and come back to 240 tasks.

That is 80 x 5,000 = 400,000 phones in 30 s: 13,333 reconnects a second, besides the usual 4,000 connects.

A reconnect is a full open. The gateway cannot know which friends came online while the phone was away, so it reads all 200 keys.

The stopped task wrote no close, so a reconnect is 200 reads and 1 write: 201 commands. 400,000 / 30 x 201 = 2,680,000 more commands a second.

The store gets 818,000 + 2,680,000 = 3,498,000 commands a second: 349.8 an update.

The result names Latest locations (Valkey): 3,498,000 against 1,392,000 is 251%. Its warning shows the busiest primary at 874,500 commands a second.

Friends' gateways keep their subscriptions, so a reconnect makes about 20 calls. The channels get 320,000 + 266,667 = 586,667 calls: 76.3%, no warning.

The gateways reach 16.8%: 4.04 ms of CPU an update, the usual 0.57 ms plus 1.733 connects at 2 ms each.

Carrying the release would take 13 store shards, at 77.3%: $3,079.30 a month more, for 30 seconds of each release, one a week (our estimate).

The latency tile shows "≥ 5ms": 4.04 ms at the gateways and 0.8 ms at the saturated store, 4.84 ms, a floor.

The cost tile shows $26k/mo. Its egress, $14,713.23 at 8.76 KiB an update (a KiB is 1,024 B), bills the release as if it ran all month.

Each reconnect sends 4.5 KiB, mostly the handshake that sets up encryption. So the release really adds 400,000 x 4.5 KiB = 1.72 GiB: about $0.12.

On the canvas, the store saturates unless a GET costs under about 0.39 of the card's, with writes at full cost. The reconnect rate is typed, so Stage 4 has no load buttons.

The trade-off:

A release costs no users, only reconnects. Each reconnect costs the store a full open, so a third of the fleet at once saturates it.

Stage 4: A release restarts a third of the gateways

• Load: 10,000 req/sec
• First to saturate: Latest locations (Valkey)
• Estimated cost: $26k/mo

• Latency past saturation is a lower bound, not an estimate. Latest locations (Valkey) is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Latest locations (Valkey): cache.m6g.xlarge ×4 shards: the busiest primary runs 874,500 of its 348,000 ops/s (251%).

Check in: The release adds no users. Why does the store saturate?

Small batches, each drained over 110 s

Stage 5 keeps the release and paces it. The batch size, not the release, should set the reconnect rate.

The script starts 12 new tasks and waits until they run. Then it lowers the old service by 12: 60,000 phones.

It waits for those 12 to drain before the next batch. 240 / 12 = 20 batches.

The directory keeps two flags a gateway: "accepting" gets it new phones, and "live" means its phones are connected.

On SIGTERM a gateway clears only "accepting", so it gets no new phones. It closes its phones evenly over 110 s, each with a "reconnect now" frame.

Why 110 s? A Fargate task's stop timeout, from SIGTERM to its forced stop, is 30 s by default. The task definition sets it to 120 s, the maximum.

The drain uses 110 s of it, leaving 10 s to close its Valkey connections and exit.

Our script sends nothing before SIGTERM, and with no load balancer nothing else does. So the stop timeout is all the time a task has.

At the 30 s default, the same 12 tasks would drain in under 30 s. That is at least 60,000 / 30 = 2,000 reconnects a second.

Behind an NLB, ECS would first wait out the deregistration delay, 300 s by default and up to 3,600 s. Here nothing does.

So 60,000 / 110 = 545 reconnects a second, not 13,333.

The draining task writes each close, so a reconnect is 202 commands: 60,000 / 110 x 202 = 110,182 more commands a second.

The store gets 818,000 + 110,182 = 928,182 commands a second, 92.82 an update: 66.7%.

The channels get 20 calls for each reconnect: 320,000 + 10,909 = 330,909 calls, 51.8%. The gateways are at 6.2%.

Nothing saturates; the note reads "Busiest: Latest locations (Valkey) at 67%".

The draining task does not unsubscribe a leaving phone's channels. They drop when its Valkey connection closes; until then it discards their deliveries.

The latency tile shows 2ms: 1.48 ms at the gateways and 0.2 ms at Valkey, 1.68 ms.

The cost tile shows $17k/mo: the same nodes as Stage 3, and $6,020.54 of egress at 3.01 KiB an update.

On the canvas, the reconnect rate is typed for this batch size, so Stage 5 has no load buttons.

Step through the figure to compare Stage 4's release with Stage 5's.

The trade-off:

Each batch waits out a drain, so the release is slow. In exchange, it never takes the store past what it can carry.

Stage 4's release against Stage 5's

1. Stage 4: 80 tasks stop at once. 400,000 phones reconnect in 30 s, and the store saturates.
2. Stage 5: 12 tasks drain over 110 s, 545 reconnects a second. The next batch waits.

Stage 5: Small batches, each drained over 110 s

• Load: 10,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $17k/mo

Check in: Why not let each task drain for 10 minutes instead?

Why pace by batch, and when a long drain wins

Pacing by batch is not the only way to release. The other ways let phones leave on their own first.

Here a task has only its stop timeout, so the batch size sets the reconnect rate, whatever the session lengths.

A longer drain wins when sessions are known to be short. There are three ways to get one.

Behind an NLB, ECS waits the deregistration delay before SIGTERM: 300 s by default, up to 3,600 s.

With no load balancer, the script can clear a gateway's "accepting" flag minutes before it lowers the old service.

ECS blue/green keeps the old fleet through a bake time, while new phones go to the new one.

In each case most phones close on their own. Those closes cost nothing extra: they are already in the 4,000 connects a second.

The price is release time, or a second fleet during the bake.

The risk is the tail: the phones still connected when the old task stops reconnect together.

This page has no figure for how long the longest sessions last. Some phones may hold the map open for hours. So it paces by batch.

Say it like this: "Stop new phones early, and size the batch so the store holds even if every phone is still there."

A handover is a fourth idea. A planned stop hands each phone a token with its online friends, so the next gateway skips the 200 reads.

It must still find friends who arrived in the gap, and it does nothing for a crash.

The trade-off:

Batches bound the worst case but make a release slow. A long drain is fast when sessions are short, and fails when the tail is long.

Check in: When would you let sessions end on their own instead of pacing by batch?

The number chain on one screen

Skip on a first read; come back before the interview. Every number on Stages 1 to 5 comes from the one before it.

Units: memory sizes are decimal, so 1 GB is 1,000,000,000 B. Bytes to phones are binary KiB, and AWS bills a GiB as a GB.

A 30-day month has 2,592,000 s and 720 hours.

Valkey: $0.2376 an hour a node. Store: $0.2376 x 720 x 8 nodes = $1,368.58. Channels, 6 nodes: $1,026.43.

Fargate 1vCPU-2GB: $0.04048 + 2 x $0.004445 = $0.04937 an hour. 240 gateways for 720 hours: $8,531.14.

Fargate 2vCPU-4GB: $0.09874 an hour. 30 API tasks for 720 hours: $2,132.78.

Commands: Stage 1, 40,000 x 200.25 = 8,010,000. Stages 2 and 3, 10,000 x 81.8 = 818,000.

Stage 4, 818,000 + 400,000 / 30 x 201 = 3,498,000. Stage 5, 818,000 + 60,000 / 110 x 202 = 928,182.

Bytes to phones for each sent update: 4 frames x 0.1 KiB, and a ping every 30 s, 4 x 0.1 KiB: 0.8 KiB.

A fresh connect sends 4.9 KiB: 4 KiB of handshake to set up encryption, 0.5 KiB for the first map, and 4 arrival frames. A reconnect sends 4.5.

Stages 2 and 3: 0.8 + 0.4 x 4.9 = 2.76 KiB. Stage 4: 2.76 + 4/3 x 4.5 = 8.76. Stage 5: 2.76 + 3/55 x 4.5 = 3.005, 3.01 on the card.

Stage 1's reply: 4 near friends at 0.1 KiB and 0.2 KiB of headers make 0.6 KiB. A tenth of a 4 KiB handshake, for 10 polls a session, makes 1 KiB.

GB a month = KiB a second x 2,592,000 / 1,048,576, which is KiB a second x 2.4719.

Egress: the first 100 GB free, then $0.09, $0.085, $0.07 and $0.05 a GB, with tiers ending at 10,340, 51,300 and 153,700 GB.

Up to 51,300 GB that is $4,403.20. Stage 1: 98,876.95 GB, $7,733.59. Stages 2 and 3: 68,225.10 GB, $5,587.96.

Stage 4: 216,540.53 GB, $14,713.23. Stage 5: 74,404.91 GB, $6,020.54.

Stage 1: $2,132.78 + $1,368.58 + $7,733.59 = $11,234.95. Stage 5: $8,531.14 + $1,368.58 + $1,026.43 + $6,020.54 = $16,946.69.

On Stages 2 to 4 the engine's months are $16,514.10, $16,514.10 and $25,639.37. The rounded lines add to a cent more, because the engine adds unrounded parts.

A real month: with the evening peak at twice the average (our estimate), egress halves and the fixed fleets do not. Stage 3 then costs about $13,870.

The card also leaves out the $864 of public addresses, and gateway data that crosses zones, at $0.01 a GB.

Check in: Close the page and rebuild Stage 2's 818,000 commands from the peak.

When a gateway, a shard or a subscriber fails

A crash has no drain. Each failure here needs a plan, and none changes a canvas, so we reason by hand.

A key reads as live only if its gateway is on the directory's live list, which every gateway holds. Otherwise its pin shows "last seen", with its age.

A gateway crashes: it sends no "offline". Its 5,000 phones reconnect within their retry delay, about 167 a second over 30 s.

The directory clears both the task's flags. Friends' gateways see it leave the live list, and mark its users "last seen".

A channels primary fails: subscriptions are not copied to replicas. The gateways subscribe on the primaries, so the promoted replica has none of them.

Every gateway resubscribes its channels on that shard: about 12M in Stage 3, a third of all subscriptions. All at once, that saturates the shard for about 34 s.

Paced over 2 minutes it is 100,000 calls a second. With the normal 176,667 ops, the shard is at 79.5%.

Gateways could subscribe on the replicas instead, since shard messages reach every node in the shard. The card does not model replica subscribers.

That splits the deliveries, and keeps those subscriptions through a primary failover.

A store primary fails: replication is asynchronous, so a few milliseconds of acknowledged writes may be lost.

Each gateway rewrites its users' keys on that shard: about 300,000 SETs over a minute, 5,000 a second.

Two friends who connected just then may have missed each other. So users who connected in the 5 s before re-read their friends' keys on that shard.

That is about 20,000 users x 50 keys = 1M reads over a minute: about 17,000 a second, 1.2% of the store.

A gateway reads too slowly: at the pub/sub output buffer limit, 32 MiB, or 8 MiB for 60 s, Valkey disconnects it.

It loses all its subscriptions, and must notice and resubscribe, paced like a failover. A parameter group may change the limit.

The trade-off:

Every failure becomes a burst of reconnects or resubscribes. Pacing each one keeps the store and the channels clean.

Check in: A channels primary fails in Stage 3. What must the gateways do, and how fast?

Ghost mode, revocation and order

Privacy is a question about a few keys, not about load. It must take effect within about 1 s, even against a connect at the same moment.

The server keeps no location history. If a product wants it: opt-in, short retention, or kept on the phone.

Ghost mode. The gateway writes SET loc:{user} "hidden" EX 3600, with no position and no gateway. Then it publishes "hidden".

Friends' gateways drop the cached position and send a "remove" frame. A friend who opens the map reads "hidden", shows nothing and does not subscribe.

Why write first? The key holds the last position for up to an hour, and every friend who connects reads it.

A friend's gateway may read the key just before the hidden SET, and subscribe 1 ms later. So the gateway publishes "hidden" again 5 s after.

Revocation, store then load. Write the share list, then its cache entry, then read the friend's key.

A connecting gateway reads the share list only after its own key is acknowledged.

So either the friend's read sees the new list, or the revoker's read finds the friend's gateway and tells it.

That gateway drops the user from the friend's follow list and sends a "remove" frame. It unsubscribes only when no local follower remains.

Each cache entry carries the list's version and a 60 s TTL. A fill writes only over an older version, so a slow fill cannot restore a revoked list.

Every gateway checks the share for each local user before it sends a frame.

Order. Each update carries a per-phone sequence number, and a gateway drops one with a smaller number. A phone clock that steps back does no harm.

A late write from a draining gateway may leave an older point in the key for a moment. The next update corrects it.

The trade-off:

Ghost mode costs a write and two publishes, revocation a write and a read. In exchange, neither can slip past a connect.

Check in: Ghost mode only stops publishing. What does a friend who opens the map see?

Three more choices, and when the other wins

An interviewer may push on three more choices. Each has a case where the other answer wins.

Valkey pub/sub, or a ring of gateway owners. Pub/sub is managed and sharded, with no membership to run.

A ring of owners, like Slack's channel servers, wins when a message needs order, acknowledgement or replay. Pub/sub gives none; here the next update corrects a lost one.

Our gateways, or a managed socket service. API Gateway WebSockets allow 500 new connections a second per account and Region by default.

We need 4,000, and 13,333 in Stage 4. The quota can be raised.

It also bills each message, pings aside. 10,000 updates in and 40,000 frames out a second is 129.6 billion messages a month.

At $1.00 a million, the only rate confirmed, that is up to $129,600 a month; a volume tier, if any, lowers it.

Connection minutes add $12,960 a month.

A managed service wins at small scale, where a fleet's fixed cost and its releases dominate.

Other Regions: home each user in the Region where they are now. A friend who is near is in the same Region.

So traffic crosses Regions only for the rare update of a friend who travels.

The trade-off:

Each choice trades a managed part for control, or one Region for many. Each answer names the scale where it flips.

Check in: Why not use API Gateway WebSockets for the gateways?

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a user see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

A gateway crashes: its 5,000 phones reconnect, and friends see those pins turn "last seen" until they do.

A channels primary fails: gateways resubscribe, paced over 2 minutes. Some friends' pins stop moving until then.

A store primary fails: a few writes are lost. Gateways rewrite their users' keys, and recent connects re-read their friends.

A release: phones reconnect in paced batches, and each sees a short reconnect.

Ghost mode or a revocation: the user's pin goes from a friend's map within about 1 s.

Say it out loud: "Push on per-user channels, to online friends only. Write your key, wait, then read your friends on the primaries."

And: "With 5-minute sessions, connects outweigh updates. Size for connects, and pace every reconnect."

Minutes 0 to 5: requirements. Friends, near, live, privacy.

Minutes 5 to 12: Stage 1, polling, and why it reads every friend.

Minutes 12 to 22: Stages 2 and 3. Channels, fan-out, churn, online-only and the connect race.

Minutes 22 to 30: Stages 4 and 5. Releases, reconnects, pacing and drains.

Minutes 30 to 45: failover, slow subscribers, ghost mode and revocation, cell channels, other Regions.

The trade-off:

Each answer has a price: a held connection, a write before every read, and slow releases. Each price buys a requirement.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.