Design an email service

Do the numbers before the boxes

An interviewer gives a scale, not a load. If we draw before we do the math, we cannot say why any box has its size.

We assume 324,000 mailboxes at launch, and 3.24 million a year later.

Each mailbox receives 40 messages a day, spam folder included: 324,000 x 40 = 12.96 million a day.

12,960,000 / 86,400 = 150 deliveries a second on average. The peak is twice that, our assumption: 300.

A delivery is one message landing in one of our mailboxes. At 3.24 million mailboxes it is 1,500 on average and 3,000 at the peak.

About as much again is offered and refused as spam: 2 attempts per delivery. Spam was 47% of email traffic in 2024, by Kaspersky's count.

Each mailbox sends 4 a day: 0.1 sends per delivery, to one outside recipient on average.

Each send also stores a Sent copy, so we store 1.1 messages per delivery.

Check in: Close the page and derive 300 deliveries a second, then the attempts and stored messages at that peak.

Accept mail only once it is safe

A sending server hands us a message and waits for one reply. Once we say yes, it deletes its copy, so a lost message is gone.

An MTA, a mail transfer agent, is the server that speaks SMTP. Ours run as Fargate tasks, and change IP every time they scale.

Connected: the sender looks up our 2 or 3 MX names, the DNS records for our mail. They point at Network Load Balancers (NLBs) with a static IP each zone.

The NLB passes the sender's real IP only with client IP preservation turned on, or proxy protocol v2. Preservation is off by default for IP targets.

Without it every SPF and reputation check sees the NLB. The NLB is not drawn; it is priced by hand in the bill.

Checked: mail to an address we do not host is refused at RCPT TO. A session with many bad recipients is slowed, so attackers cannot harvest addresses.

Authenticated: SPF asks whether this IP may send for the domain, with up to 10 DNS lookups. DKIM checks a signature, DMARC the policy, ARC forwarded mail.

Filtered: half the attempts are refused in the session at about 2 ms of CPU each. The rest are fully scanned at about 20 ms.

So an attempt averages (2 + 20) / 2 = 11 ms: 1,000 / 11 = 90.9 a second a task. These are our estimates.

At 50 ms of CPU an attempt, the 3,000-delivery peak needs 6,000 attempts x 0.05 / 0.7 = 429 tasks. That is past the MTA card's 300-task guard.

A refusal in the session costs no storage and no bounce. User filters run later and file to the spam folder.

Stored: only when the message is durable does the MTA answer 250 OK. SMTP gives the sender 10 minutes to wait for it.

Step through the figure to follow one session.

The trade-off:

Waiting for durability makes every reply slower. In exchange, a 250 is a promise we can keep.

One SMTP session, from MX to 250

1. The sender looks up our MX names and reaches a load balancer's static IP. With client IP preservation on, the MTA sees the sender's real IP.
2. Mail to an address we do not host is refused at RCPT TO. A session full of bad recipients is slowed, so nobody harvests addresses.
3. SPF asks whether this IP may send for the domain, DKIM checks a signature, DMARC the policy, and ARC mail that was forwarded.
4. Half the attempts are refused in the session at about 2 ms of CPU. The rest are scanned at about 20 ms: 11 ms on average.
5. The MTA hands the message to the store. Only once its row and body are durable does it answer 250 OK.

Check in: Why must the MTA wait for the store before it answers 250?

What a message costs in DynamoDB

The simplest answer stores each message, body included, as one DynamoDB item. Before we run it, we price every write it causes over its life.

DynamoDB bills writes in WCU, write capacity units: 1 WCU writes up to 1 KB, so a 21 KB message is 21 WCU to put.

An update bills the whole item, at the larger of its sizes before and after. Marking a 21 KB message read costs 21 WCU.

A delete bills the deleted size. We assume half of all mail is opened, 20% is moved and 30% deleted.

Over its life: put 21, mark read 21 x 0.5 = 10.5, and move 42 x 0.2 = 8.4.

Then delete 21 x 0.3 = 6.3, and folder counters 2.2.

That is 21 + 10.5 + 8.4 + 6.3 + 2.2 = 48.4 WCU per stored message. A move is 42 because the folder is in the key: a delete and a put.

Reads bill the items read, not what comes back. A 50-message list reads 50 x 21,504 bytes = 1,075,200: two Query calls, 131.5 read units.

An item holds at most 400 KB, so attachments go to S3, one object each, about 300 KB.

The trade-off:

One item per message is the easiest design to write. In exchange, every flag change pays for the whole message again.

Check in: Why is marking a message read expensive here?

Run the whole message in one table

Stage 1 runs it at launch: 300 deliveries a second, 324,000 mailboxes.

The MTAs take 600 attempts: 600 / (90.9 x 0.7) = 9.4, so 10 tasks at a 70% target, 66%.

The table takes 330 stored messages x 48.4 = 15,972 WCU. 15,972 / 0.7 = 22,817, provisioned as 22,900: 69.7%.

This card counts write units, not requests. Half the attempts are accepted, so the MTA edge carries 48.4 x 0.5 = 24.2 per attempt.

Per delivery: 2 attempts x 24.2, plus 0.1 x 48.4 for the Sent copy, is 53.24 WCU.

Read the four tiles. Here the load buttons and the Load tile count deliveries a second. "First to saturate" notes the Mailbox table, at 70%.

A part saturates when it gets more work than it can serve. A card turns red at 90% of its limit; only past 100% does the result name it.

The latency tile shows 161ms. It adds each part's fixed latency once along the slowest arm: MTA 11 + an attachment upload to S3 150.

By hand, after the message arrives: a 20 ms scan and a 2 ms table put is 22 ms. The 7% with an attachment add a 150 ms upload: about 33 ms on average.

The cost tile shows $23k/mo: $22,538.75. The table is $13,206.66, and Amazon SES, which sends our outbound mail, $7,776.

Data out to the internet, called egress, adds $598.35. Mail comes in free, so we pay only for what our servers send back.

Each attempt is a new session that starts encryption with STARTTLS. To set it up, our MTA sends about 4 KB, mostly its certificates, plus 0.5 KB of SMTP replies.

That is 4.5 KB an attempt, our estimate, and each send gets a 1 KB reply. 600 x 4.5 + 30 x 1 = 2,730 KB a second, 6,748 GB a month. The first 100 GB are free, then $0.09 a GB.

Press 150, the average hour. Nothing saturates; the note names the Inbound MTAs, at 66%.

Press 600. The Mailbox table saturates: the table breaks at 22,900 / 53.24 = 430 deliveries a second.

Its warning reads "31944 of 22900 WCU/s at 31944 RPS". The warning counts write units, not requests.

The trade-off:

Storage: 165 items a second on average, 150 deliveries plus 15 Sent copies, x 2,592,000 seconds x 21,604 bytes = 8,605 GiB a month.

Set as 8,610 GiB, that is $2,152.50 more each month at $0.25 a GiB.

Stage 1: The whole message in one table, at launch

• Load: 300 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $23k/mo

Check in: At what delivery rate does the table run out, and why so low?

The same table, a year later

The interviewer asks: A year later we have 10 times the mailboxes. What breaks?

Stage 2 is Stage 1's design at 3,000 deliveries a second: 3,300 stored messages.

The table needs 3,300 x 48.4 = 159,720 WCU. The card stops at 40,000, the default quota for one table.

The Mailbox table saturates. Its warning reads "399% used", and again it counts write units.

The 161ms tile is a simulator artefact: it follows the longest arm, the attachment upload.

In the real system the table throttles three writes in four: it takes 40,000 of 159,720. That mail never gets its 250, and the sender retries later.

The cost tile shows $135k/mo.

AWS raises the quota on request, so the honest wall is the bill. Storage first: DynamoDB is $0.25 a GiB, S3 $0.023, 10.9 times less.

It grows linearly. Each month adds 86,100 GiB, about 60,300 net of the 30% later deleted: about $15,068 more each month.

After 12 months that is about 723,200 GiB, $180,800 a month. That is an upper bound, as the mailbox count grows during the year.

Then writes: 159,720 / 0.7 = 228,172 WCU with the quota raised, $106,784.50 a month.

Compressing the body about 3 times is a constant factor. A message becomes 19.8 write units instead of 48.4, and a list 48.5 read units instead of 131.5.

But the table still pays $0.25 a GiB, 10.9 times S3, and 19.8 is still 2.6 times the split design's 7.5.

Press 300: Stage 1's load on this table. Nothing saturates; the table reads 39.9%, and the cost tile shows $54k/mo.

The trade-off:

Raising the quota buys time, not a design. The bill keeps growing with every month of stored mail.

Stage 2: The same table, a year later

• Load: 3,000 req/sec
• First to saturate: Mailbox table
• Estimated cost: $135k/mo

• Mailbox table: Write capacity 399% used (provisioned): 159720 of 40000 WCU/s at 159720 RPS.

Check in: Why not just raise the quota?

Keep what changes small and apart

Most writes change a flag or a folder, and the body never changes. So the body should not pay for them.

Mailbox writers store each message three ways. An immutable message row, about 1 KB, goes to a messages table.

Its label items, 0.5 KB each, a thread summary and counters go to a mailbox index table.

A move is cheap because the folder lives on a small label item, keyed by mailbox, label and time. A move deletes one 0.5 KB label item and puts another, 1 write unit each.

The 1 KB message row never moves, because it is keyed by message id, not folder.

The body is compressed about 3 times, encrypted, and packed into a segment object in S3.

A writer puts its segment every 200 ms, or at 4 MiB. A writer at 70% stores 700 messages a second, about 3.4 MiB in 200 ms, so 1 PUT per 140 messages.

The MTA's 250 waits for the segment, the row and the index items. That group commit adds up to about 200 ms.

New write budget. Row: put 1 + delete 1 x 0.3 = 1.3. Index: arrival 3, for the label item, thread and counter.

Then mark read 3 x 0.5, move 4 x 0.2 and delete 3 x 0.3: 3 + 1.5 + 0.8 + 0.9 = 6.2. So 7.5 in all, not 48.4.

A list now reads 50 label items: 3.5 read units, not 131.5.

Step through the figure to compare the two models.

The trade-off:

Three stores instead of one, and a write that waits for all three. In exchange, every flag change costs a kilobyte.

One big item, or a small row and small index items

1. In one item, every change pays for the whole 21 KB message. Weighted by how often each happens, that is 48.4 write units over its life.
2. Split it: an immutable 1 KB row, small label, thread and counter items, and the body packed into S3.
3. Changes now touch kilobyte items, never the body. Over a message's life the index takes 6.2 write units and the row 1.3: 7.5 in all.
4. A 50-message list reads 50 whole items for 131.5 read units, or 50 label items for 3.5.
5. A writer puts its segment every 200 ms. At 700 messages a second, one PUT carries 140 messages.

Check in: Where does marking a message read write now, and how much?

What the store holds

Packing changes how many objects we write, not how many bytes we keep. We work the bytes from the inputs.

A message to 3 of our users is one body and 3 rows: 0.7 unique bodies per delivery, our estimate.

Inbound, per delivery: 0.7 x 20 KB / 3 for the body, plus 0.7 x 0.1 x 300 KB of attachments. That is 4.7 + 21 = 25.7 KB.

A Sent copy is its own message: 6.7 + 0.1 x 300 = 36.7 KB. Over 1.1 stored messages, (25.7 + 3.67) / 1.1 = 26.7 KB.

The inputs are estimates to about 20% either way, so we draw 25 KiB. At 26.7 KiB the store would hold 106 TiB a month, and its storage warning would fire.

At 25 KiB, segments are 99.6 TiB a month, about 1.2 PiB a year. Rows add 4,480 GiB a month and index items 4,880 GiB.

Segments: 3,300 / 140 = 23.6 PUTs a second at the peak, $305 a month. One object a message would be 3,300 a second, $42,768.

The trade-off:

A month of new mail adds about $4,686 of storage, against $21,525 gross in the one big table.

Check in: Where does the 25 KiB a stored message come from, and how sure is it?

Run the split design

Stage 3 runs the split at 3,000 deliveries a second.

Writers: 3,300 / 700 = 4.7, so 5 tasks, 66%.

Messages table: 3,300 x 1.3 = 4,290 WCU on 8,100, 53.0%. It is set higher than 70% needs, for bulk deletes such as emptying Trash.

Mailbox index: 3,300 x 6.2 = 20,460 WCU. 20,460 / 0.7 = 29,229, provisioned as 29,300: 69.8%.

"First to saturate" notes the Mailbox index, at 70%.

The latency tile shows 162ms. It adds each part's fixed latency once along the slowest path: MTA 11 + writer 1 + S3 150.

It does not model the group commit's wait for a segment to fill. So a message is really accepted in about a quarter of a second, and under 0.4 s at worst.

By hand: scan 20, plus a segment wait of 100 on average, plus the segment put 150 and the row and index puts 2. That is 272 ms, and 372 at most.

The cost tile shows $110k/mo. Without SES the drawn lines are about $32,148, against about $145,254 for Stage 2 with the quota raised.

Press 1,500, the average hour. Nothing saturates; the note names the Inbound MTAs, at 69%.

Press 4,500, a heavy day. The Mailbox index saturates, and the Messages table reads 79.4%.

The index is the first wall, at 29,300 / 6.82 = 4,296 deliveries a second, where 6.82 is 1.1 stored messages x 6.2.

The trade-off:

The bill is now mostly SES. The price is a quarter of a second per message, waiting for its segment.

Stage 3: Message rows, an index table, packed bodies

• Load: 3,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $110k/mo

Check in: What does one message cost in write units over its life now, and why?

The whole design on one screen

Every number on the stages comes from the one before it. Here is the chain for Stage 3 at the 3,000 peak, with Stage 1 in brackets.

Mailboxes: 3.24 million (324,000), our assumption.

Deliveries a second: mailboxes x 40 / 86,400 on average, x 2 at the peak: 3,000 (300).

Attempts: 2 per delivery, 6,000 a second (600).

Inbound MTA tasks: attempts / (90.9 x 0.7): 95 tasks, $3,376.91 (10 tasks, $355.46).

Accepted: half the attempts, 3,000 (300). Sends: 0.1 per delivery, 300 (30).

Stored messages: accepted plus Sent copies, 3,300 (330). Writer tasks: 3,300 / 700, so 5, $177.73 (none).

Write units: 3,300 x 1.3 = 4,290 on 8,100, and 3,300 x 6.2 = 20,460 on 29,300 (330 x 48.4 = 15,972 on 22,900).

Tables: $5,023.12 + $15,147.68 ($13,206.66).

S3: 3,300 / 140 segments a second, 99.6 TiB a month, $2,651.27 (21 attachment PUTs a second, 7.6 TiB, $451.16).

SES: 300 emails a second, $77,760 (30, $7,776).

Send API, outbound queue and senders: $235.09 ($151.11).

Egress: 6,000 attempts at 4.5 KB and 300 sends at 1 KB, 27,300 KB a second (2,730).

That is 67,483.52 GB a month, $5,536.05 (6,748.35 GB, $598.35).

The first 100 GB are free. Past 10,340 GB a month the rate falls from $0.09 to $0.085 a GB, and past 51,300 GB to $0.07.

Total, with the peak held a month: about $109,908 (about $22,539), the cost tile's $110k/mo ($23k/mo). At the average month, Stage 3 is $66,461.38.

Check in: Close the page and rebuild the chain from 3.24 million mailboxes to the Stage 3 index write units.

Pointers that survive compaction, and keys that erase

The interviewer asks: How do you compact segments, and how do you erase one user from packed files?

A segment holds many users' bodies. If a row stored a byte offset, every compaction would rewrite rows. If bodies were shared, erasing one user would break the others.

A row stores a body_id: a logical segment id and a key, not an offset. Each segment object ends with a footer that maps keys to offsets.

Compaction rewrites a logical segment into a new object and swaps one segment-map item. No row changes: about 0.007 WCU a message.

Each body is encrypted with its own data key. The row carries that key wrapped by the mailbox key.

Crypto-shredding: erasing a mailbox deletes its key and its rows at once. The bytes stay in the segments, but nobody can read them.

Other recipients keep their own wraps, so their copies still open. Dedup stays inside one SMTP transaction, so there is no shared hash to undo.

Mailbox keys are not one KMS key each: 3.24 million x $1 a month would be $3.24 million a month.

They live in a key table, wrapped by a few KMS keys, and are cached while a mailbox is active.

Erasure is immediate for reads and complete once the 35-day recovery window passes. The key table stays out of longer backups.

A generation is the set of segments written in one period, say a week. The sweep marks which keys in a generation still have a row, then compacts the segments whose live share is low.

Dead bytes are found that way, not by reference counts. Trash purges after 30 days, and a legal hold keeps the mailbox key.

Step through the figure to follow a body_id and an erasure.

The trade-off:

A key per body and a wrap per recipient cost a few bytes a row. In exchange, an erasure is a delete, not a rewrite of up to about 1.2 PiB a year of segments.

A pointer that survives compaction, and a key that erases

1. A row holds a logical segment and a key, not a byte offset. The segment's footer maps each key to where its bytes sit.
2. Compaction rewrites the segment into a new object and swaps one map item. No row changes.
3. Each body has its own key. Each recipient's row carries that key, wrapped by their mailbox key.
4. Erase mailbox A: delete its key and rows. A's copy is unreadable at once, and B still reads the same body.
5. A sweep per segment generation finds the dead bytes later. No reference counts to keep in step.

Check in: How do you erase a user without rewriting every segment?

Reading a mailbox

Reads are not drawn on the stages, as every table here is sized for writes. We work them by hand.

A list reads 50 label items: 3.5 read units eventually consistent, or 7 strongly consistent.

A warm open reads the 1 KB row, 0.5 read units, and one S3 byte range. The segment map and footer are already cached.

A cold open also reads the segment-map item, 0.5 read units, and a byte range of the segment footer. Caches keep most opens warm.

The send API writes the Sent copy before it answers. With no secondary index, a strong read of the mailbox sees it at once.

A reply that arrives before its parent starts its own thread id. When the parent arrives and links them, a small alias item maps the newer id to the older.

A mailbox's new label items follow a rising time, so DynamoDB cannot split them: at most 1,000 WCU. A mail-bombed mailbox gets a temporary refusal at the MTA.

Step through the figure to follow a list and two opens.

The trade-off:

Strong reads cost twice the units. They are worth it only where a user must see their own change.

A list, a warm open and a cold open

1. A list reads 50 small label items from the index: 3.5 read units, or 7 when the client must see its own last change.
2. A warm open reads the 1 KB row, 0.5 read units, and one byte range from S3. The map item and footer are already cached.
3. A cold open also reads the segment-map item and the segment's footer, then the body. Caches keep most opens warm.

Check in: How does a user see their sent message in Sent at once?

One message out

Sending looks simple: hand the message to a service that delivers it. The hard part is what we share with every other user.

Submitted: the client calls the send API, or SMTP submission on port 587 with OAuth or an app password.

Stored and queued: the writers store the Sent copy, then one queue message goes out per send. Mail to our own users goes straight to the writers.

Sent: a sender calls SES, which signs with DKIM and delivers. SES counts each recipient as one email against our quota.

Retried: once SES accepts a send, it retries the remote server itself, for up to 14 hours. Our queue retries only SES API errors.

Bounced or complained: SES reports it, and the address goes on our suppression list.

The quota is ours as an account: 200 emails a second at launch, 600 a year later, raised on request after review.

A provider this size would run its own outbound servers on its own IPs. We draw SES to make the shared quota and reputation visible.

The lesson is the same either way: one account's spam damages everyone's delivery.

The trade-off:

SES saves us running mail servers and warming IP addresses. In exchange, its quota and reputation are one thing every user shares.

Check in: A remote server is down. Who retries, and for how long?

One compromised account, no limits

The interviewer asks: One account is compromised and starts spamming. What breaks?

A stolen password, SMTP submission, and 900 spam emails a second to outside addresses, beside the users' 300.

The send API takes 1,200 sends a second, 0.4 sends per delivery at this load only, so this stage has no load buttons.

SES saturates. Its warning reads "Send rate 1,200/s exceeds the 600/s sending-rate quota: 600/s are throttled."

SES throttles every sender alike, so half of all sends fail and retry, our users' mail included.

Worse, the spam hits spamtraps, addresses no person uses, and drives bounces and complaints. SES acts at once on spamtrap hits.

It reviews an account at 5% hard bounces or 0.1% complaints, and may pause sending at 10% or 0.5%. Then every user's outbound mail stops.

The latency tile shows 162ms: SES is behind the queue, off the path. The cost tile shows $188k/mo.

The trade-off:

One shared quota is cheap to run. In exchange, one bad account spends it for everyone.

Stage 4: One compromised account, no limits

• Load: 3,000 req/sec
• First to saturate: SES
• Estimated cost: $188k/mo

• SES: Send rate 1,200/s exceeds the 600/s sending-rate quota: 600/s are throttled.

Check in: Why is one account everyone's problem?

Per-account limits before the queue

The shared thing needs a door that one account cannot walk through.

The send API enforces limits per account: messages and outside recipients a day, and a rate a minute. Tighter for new or low-trust accounts.

Our policy follows Google's published 2,000 messages and 3,000 outside recipients a day.

A per-minute cap, say 60 recipients a minute, stops the bot after its first 60 sends, in under a tenth of a second. The daily caps alone would let 2,000 to 3,000 through first.

Its sends are refused and never queued. The stage draws that as the accepted share, 300 of 1,200, so it has no load buttons.

Nothing saturates; the note names the Mailbox index, at 70%. The latency tile shows 162ms, and the cost tile $110k/mo, the same rounded figure as Stage 3.

A refused send still gets a 1 KB reply, so the bot adds about $156 a month of egress.

Limits are necessary, not sufficient. 1,000 stolen accounts, each under its cap, could still send 3 million a day.

So the senders also score outbound mail and skip suppressed addresses.

Configuration sets and IP pools per trust tier isolate IP reputation at the receiving providers. SES's own review and pause apply to the whole account.

So true isolation needs a separate SES account per tier, or our own outbound servers.

And sending needs OAuth or an app password with 2FA, never a password alone.

The trade-off:

Limits annoy a few honest heavy senders. In exchange, the quota and the reputation survive one bad account.

Stage 5: Per-account limits before the queue

• Load: 3,000 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $110k/mo

Check in: Why not let the queue absorb the blast?

The real monthly bill

The cost tile holds the peak all month. An honest bill uses the average, and adds what the stages do not draw.

Stage 3 at 1,500 deliveries a second, the average, is $66,461.38 a month. For 3.24 million mailboxes that is $0.0205 a mailbox in month one.

SES is $0.0120 of that. The other drawn lines are $0.0085.

After a year of stored mail, gross before deletes and tiering, storage adds $12,320 for rows, $13,420 for the index and $25,803.57 for segments.

That is $51,543.57 more: about $118,005 a month, $0.0364 a mailbox.

The stored mail itself, about $56,229 a month by then, is the largest line after SES.

Not drawn: search index storage is next. It grows about $2,986 a month, to about $35,800 after a year, before the search instances.

Keep cold mailboxes' search indexes in object storage, and load them on demand.

Not drawn: the load balancers, about $1,428 a month, and SES attachments at $0.12 a GB, about $1,335.

The tile includes egress for our replies and the encryption setup: $2,910.75 at the average. Mail coming in is free.

A sender that reuses one session for many messages pays that setup once. Then the egress line falls to about $358.

Not drawn: users reading their mail. By our estimate half of mail is opened, 10.5 KB a delivery, or 3.5 KB if compressed in transit.

Half of the 10% with a 300 KB attachment are downloaded: 0.5 x 0.1 x 300 = 15 KB a delivery.

So about 18 to 26 KB a delivery goes out, about $4,900 to $7,000 a month more. Transfer between zones is excluded.

At this volume our own outbound mail servers on dedicated IP pools cost less than SES. The reputation lesson is the same, owned by us.

Old mail can move to S3 Glacier Instant Retrieval at $0.004 a GiB. Packing avoids its 128 KB minimum billable object.

DynamoDB's Standard-IA class, about $0.10 a GiB, suits the immutable rows. It is still about 4.3 times S3, so it is no answer for bodies.

The trade-off:

The drawn bill is a floor. Stored mail and search decide whether $0.02 a mailbox is the real price.

Check in: What costs most after SES within a year, and what comes next?

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a user see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

An MTA dies mid-session: no 250 was sent, so the sender retries another MX.

A writer dies: its unflushed segment is lost, but no 250 went out. The sender retries, and the conditional row put drops duplicates.

A 250 is lost on the wire: the sender retries a stored message. The message id, a hash of key headers and the body, makes the put a no-op.

SES throttles: senders retry API errors from the queue, and SES retries remote servers itself.

An account is erased: delete its key and rows, and sweep the segments later.

Sync: IMAP holds a session per client; JMAP is stateless JSON with push.

Flags carry a per-mailbox MODSEQ, a counter that rises with every change, so a client asks only for what changed since its last sync.

If 10% of the 3.24M mailboxes have a session open at the peak, our assumption, that is 324,000 held connections; at launch, 32,400. A connection gateway holds them.

Say it out loud: "Accept only once durable. Rows are immutable, changes hit small index items, and bodies are packed and encrypted per key."

Minutes 0 to 5: requirements, including erasure.

Minutes 5 to 10: the numbers. 40 received and 4 sent a mailbox a day, 3,000 deliveries a second at the peak, 2 attempts per delivery.

Minutes 10 to 17: inbound. MX, a load balancer that keeps the client IP, checks, refusing in the session, a durable 250.

Minutes 17 to 25: storage. Whole items or rows, how updates and queries bill, the write budget, packed segments.

Minutes 25 to 31: deletion and reads. body_id, crypto-shredding and the sweep; search per mailbox; threads; read-your-writes.

Minutes 31 to 38: outbound. The SES quota, reputation, a compromised account, limits and their limits.

Minutes 38 to 45: costs drawn and not drawn, tiering, and the cost per mailbox.

The trade-off:

Each answer has a price. Three stores, a group commit and per-body keys are more to run than one table, and they buy a bill that scales.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.