Design a payment system

Do the numbers before the boxes

An interviewer gives a scale, not a load. If we draw before we do the math, we cannot say why any box has its size.

We size for a commerce platform about Shopify's size: $292.3 billion of sales in 2024.

We take $108.56 as the average payment, Shopify's 2024 Black Friday average cart, for the whole year. That is our assumption.

Payments a year: $292.3 billion / $108.56 = 2.69 billion. A day: 2.69 billion / 365 = 7.38 million.

A second on average: 7,380,000 / 86,400 = 85.4.

We take a peak factor of 2.2: the busiest hour carries 2.2 times the average. 85.4 x 2.2 = 188, rounded up to 200 payments a second.

We assume about 1 read per payment: a receipt, an order status, or a merchant checking a balance. That is 200 reads a second.

We measure load in requests per second: 200 + 200 = 400. Payments are half of it.

Black Friday: Shopify's 2024 peak was $4.6 million of sales a minute. $4,600,000 / $108.56 = 42,373 payments a minute, or 706 a second.

We round that to 700 payments and 700 reads: an entry of 1,400, or 700 / 200 = 3.5 times the ordinary peak.

Shopify's sales grew 24% in 2024. So next year's Black Friday is about 700 x 1.24 = 870 payments a second.

Check in: Close the page and derive 400 requests per second from $292.3 billion a year. Then derive the Black Friday 1,400.

Create, call the PSP, record

A payment moves real money. A shopper must never be charged twice, and we must never lose a payment the bank approved.

A checkout can hold several payment attempts: a declined card, then a second card. Each attempt gets its own idempotency key.

An idempotency key is a new ID the client makes for one attempt. It sends the same ID again on every retry of that attempt.

An attempt is three steps. First, create: insert the attempt, keyed by its idempotency key, with the status "processing".

A checkout may have only one attempt that is processing, unknown or succeeded. A partial unique index on the checkout enforces that.

Second, call the PSP, with the attempt ID as the PSP's own idempotency key. The PSP creates and confirms the payment, and the buyer waits.

Third, record: in one transaction, move the status forward and write the ledger entries.

We create before we call on purpose. If we crash after the call, the "processing" row tells a sweeper to ask the PSP what happened.

We assume a PSP call takes about 1 second, because it asks the card network and the bank that issued the card. No PSP publishes a figure.

Each copy of the Payment API is a task with one vCPU, a virtual processor. A request costs about 1.2 ms of CPU, our estimate.

With async I/O, a call waiting on the PSP holds no thread. So one task serves 1,000 / 1.2 = 833 requests a second.

A synchronous stack holds a worker for each open call. At the Black Friday peak, 700 payments a second x 1 second = about 700 workers.

The API uses auto-scaling, keeping each task near 70% busy. 400 / (833 x 0.7) = 0.7, so its floor of 2 tasks runs, at 24%.

80% of payments go to the primary PSP, Stripe, and 20% to a backup PSP. That share keeps the backup warm and tested.

Stripe's defaults are 100 calls a second per account and 25 per endpoint. A platform this size has a contract: we assume 1,500 a second.

On Black Friday the primary's share is 0.8 x 700 = 560 a second, so 1,500 is 2.7 times it.

At 1 second a call, 560 calls are open at once. So the contract also covers 1,500 calls open, against a default of 50.

The backup's contract is 300 a second, about twice its Black Friday share of 0.2 x 700 = 140.

The trade-off:

The buyer waits a second for the bank. We cannot shorten that, so we must never add to it.

Check in: Say why the attempt is created before the PSP call, and not after it.

Three writes an attempt, all on the writer

Every attempt writes, and a write can go to only one machine. So the writer, not the PSP, sets how far this design goes.

The database is Aurora PostgreSQL: one writer and one read replica, or reader. Writes go to the writer, and reads to the reader.

A webhook is the PSP calling us back with the outcome. A receiver checks its signature, puts it on a queue, and answers 200 at once.

A worker reads the queue, skips event IDs it has seen, and moves the status forward if it is still processing or unknown.

So an attempt makes 3 writes: create, record, and the webhook's update. Whichever of the last two moves the status writes the ledger.

We assume about 2 ms of CPU a write: a few index inserts and log records. Halve it, and every database wall doubles.

A db.r6g.xlarge has 4 vCPUs, so the writer takes 4 x 1,000 / 2 = 2,000 writes a second.

At 200 payments a second, 200 x 3 = 600 writes use 30% of the writer. The reader takes the 200 reads: 10%.

Stage 1 runs this design in SysPad at 400 requests per second. Each card is a part of the system, and each arrow carries requests.

Read the four tiles. "First to saturate" reads "Nothing at this load". Its note names the Payments database, at 30%.

A part saturates when it gets more requests than it can serve.

The latency tile shows 1.00s. It shows the slowest path, which here is the PSP call the buyer waits for.

The load balancer takes 1.5 ms: 1 ms of routing and 0.5 ms of TLS, the encryption. Then 1.2 ms in the API and 1,000 ms at the PSP.

The cost tile shows $1,400/mo, and the database is 70% of it.

$80 of it is egress: data sent out of AWS to the internet. Here those are our answers to shoppers and merchants, about 1 KB each.

Every cost tile prices the peak rate as if it ran all month. Load balancer, queue and egress charges follow traffic, and the average rate is the peak / 2.2.

So an average month of Stage 1 costs about $1,287. Comparisons between stages hold either way.

The writer takes 2,000 writes a second, which is 667 payments. With their reads, that is an entry of 667 x 2 = about 1,333.

Press 1,500. The Payments database saturates: 750 payments x 3 = 2,250 writes a second, against 2,000.

Press 3,200. The Backup PSP saturates too: 0.1 x 3,200 = 320 calls a second, against its 300.

The trade-off:

Every attempt writes three times to one machine. A reader adds no write capacity, so the writer is the wall.

PSP fees are a share of each payment, set by contract. They are not on this bill.

Stage 1: Create, call the PSP, record: 400 a second

• Load: 400 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $1,400/mo

Check in: Which part runs out first as load grows, and at what load? Then name the next wall.

Charge once, even when every call repeats

The interviewer asks: The shopper's network drops, and the app sends Pay again. How do you make sure the card is charged once?

Every call in this design can happen twice. The shopper retries, our call to the PSP times out, and the PSP sends a webhook twice.

Delivery exactly once is not possible across a network. Every step here is delivered at least once.

So we aim for exactly-once effects: every step may repeat, and every write is keyed so a repeat changes nothing.

This is a question about keys and order, not about load. So we reason about it by hand, and no stage goes with this beat.

The order of events matters. Step 1: the attempt is created with the status "processing", before the PSP is called.

Step 2: a duplicate arrives while the attempt is in flight, because the shopper pressed Pay again. It finds "processing" and gets 202, processing, with Retry-After 1.

The page then polls the attempt. A duplicate never starts a second PSP call.

Step 3: our own call to the PSP times out. We retry first, with the same attempt ID, at 1, 2 and 4 seconds, inside an 8-second budget.

If the first call ran, the PSP returns its result for that key. Stripe keeps keys at least 24 hours.

Step 4: still no answer after 8 seconds. We mark the attempt "unknown", tell the shopper we are confirming, and a sweeper asks the PSP every minute.

"Unknown" is not "failed". The card may have been charged, so we never charge it again until we know.

Step 5: only once the outcome is recorded does a repeat with the same key get the stored result: 200, succeeded or failed.

The call and the webhook can race to record the outcome. Each moves the status forward only if it is still processing or unknown.

That is a compare-and-set. The path that changes 1 row writes the ledger entries in the same transaction. The other changes 0 rows and writes nothing.

As a backstop, ledger entries are unique on attempt and entry type, so even a bug cannot post them twice.

The webhook can also arrive twice. A processed-events table, keyed by PSP and event ID, rejects the copy.

Events arrive in any order, so a status only moves forward. "Succeeded" never goes back.

A worker that crashes mid-event sees it again. The dedupe and the forward-only update make the rerun change nothing.

After 5 tries, the queue moves the event to a dead-letter queue, a side queue for failures.

Step through the figure to follow one attempt through its repeats.

The trade-off:

Every write carries a key, and an attempt can sit in "unknown" until the sweeper settles it. A reused key with a different body gets 409.

One checkout through a retry

1. The attempt is created as processing, with its own key, before the PSP is called.
2. The PSP charges the card. Its answer is lost on the way back.
3. A duplicate finds the attempt still processing. It gets 202 and polls, and never starts a second PSP call.
4. We retry the PSP with the same attempt ID. It returns the first charge, so the card is charged once.
5. Now a repeat gets the stored result. A second webhook copy changes nothing.

Check in: The shopper presses Pay twice, and our PSP call times out. Say what each repeat gets, in order, and why the card is charged once.

Double entry, and checking it against the PSP

The interviewer asks: How do you know where every dollar went?

An attempt status says what happened to one order. It cannot tell finance who owes whom, or prove that nothing went missing.

This too is a question about records, not about load, so we work it by hand.

We use double entry: every money movement writes rows whose debits equal their credits.

A $108.56 payment pays the PSP a fee of 2.9% x $108.56 + $0.30 = $3.45, at Stripe's US list price. That fee is our expense.

We charge the merchant our own platform fee, an illustrative 3.4% + $0.30 = $3.99.

Debit PSP receivable 10,511 cents, the net the PSP will settle. Debit PSP fee expense 345.

Credit merchant payable 10,457, the gross less our fee. Credit platform fee revenue 399.

Debits: 10,511 + 345 = 10,856. Credits: 10,457 + 399 = 10,856. Our margin is 399 - 345 = 54 cents.

At settlement, the PSP pays us the net: debit cash 10,511, and credit PSP receivable 10,511.

We write these entries at capture. Digital goods capture at checkout, so this page's record step writes them. Shipped goods capture at shipment.

Amounts are whole cents, never floats, and each currency has its own balance. Ledger rows are insert only.

Reconciliation: we match the PSP's settlement report against our ledger, by PSP reference. We match the net amount, not the gross.

Every line is matched or flagged. In the PSP only means a lost record. In the ledger only means an unknown that failed.

The third flag is a different amount or fee. A fee difference is trued up with an adjusting pair of entries.

The most common mismatch is timing, not money. The PSP's day may end at a different hour, and settlement lags capture by days.

So we match across a rolling window of about 3 days, and flag only what is still unmatched after it.

A peak day is about 26.5 million payments: Shopify's 2024 Black Friday weekend was $11.5 billion over 4 days, about $2.9 billion a day.

Reading 100,000 rows a second from each list, the merge takes 26,500,000 / 100,000 = 265 seconds, about 4.5 minutes. It runs on a reader.

Step through the figure to see one payment balance, and the report match it.

The trade-off:

The ledger is more rows to write, and reconciliation is a daily job someone must watch. Without them, a missing dollar is found by an auditor, not by us.

Double entry, and checking it against the PSP

1. Four entries, and the two sides balance at 10,856 cents. Our margin is 399 - 345 = 54 cents.
2. We match the net amount by PSP reference across a 3-day window. Most mismatches are only late; what is left is flagged.

Check in: Write the four entries for a $108.56 payment with a $3.45 PSP fee and a $3.99 platform fee. Then say which amount reconciliation matches.

Black Friday, and the writer

The interviewer asks: It is Black Friday. What breaks first?

The peak is 700 payments and 700 reads a second: an entry of 1,400, or 3.5 times the ordinary peak.

Each payment is 3 writes: 700 x 3 = 2,100 writes a second, against the writer's 2,000.

Why not add readers? A reader takes reads only. Five more readers leave the writer at 2,100 against 2,000.

The reader is at 700 / 2,000 = 35%. Reads are not the problem.

Press 400 on Stage 2. At the ordinary peak, nothing saturates.

Press 1,400. The Payments database saturates.

Read the stage's warnings: "Writer 105% used: 2,100 writes/s against 2,000 on the writer alone (readers take no writes)."

The latency tile still shows 1.00s, with no "≥". The tile shows the slowest path: the PSP call, which the buyer waits for.

The writes are drawn beside the PSP call, not before and after it. So the red writer card is the wall, and the tile does not move.

In a real checkout, the writes queue in front of the PSP call and after it, and checkouts time out.

The cost tile shows $1,952/mo.

The trade-off:

Black Friday is one day, but the writer must carry its peak. A design sized for an ordinary day fails on the day that matters most.

Stage 2: The same design on Black Friday, 1,400 a second

• Load: 1,400 req/sec
• First to saturate: Payments database
• Estimated cost: $1,952/mo

• Payments database: Writer 105% used: 2,100 writes/s against 2,000 on the writer alone (readers take no writes).

Check in: The reader is at 35%. Would a second reader help? Say why.

Scale the writer up

Writes scale up, not out, until we shard. So we give the one writer more processors.

We move the cluster to db.r6g.2xlarge: 8 x 1,000 / 2 = 4,000 writes a second. 2,100 / 4,000 = 52.5%.

The cluster uses one instance class here, so the reader doubles too.

Scale up has a long runway. A db.r6g.16xlarge has 64 vCPUs: 32,000 writes a second, 15 times today's peak.

We size up before November and down after January. A change of instance class is a short failover to a reader of the new size.

Next year's Black Friday, 870 payments a second, fits: 870 x 3 = 2,610 writes, 65%.

Press 1,400 on Stage 3. Nothing saturates. The note on "First to saturate" names the Payment API, at 56%.

The cost tile shows $2,925/mo. Kept all year, an ordinary average month is about $2,260: Stage 1's $1,287, plus $973 for the bigger pair.

With this writer, the entry that breaks it is 4,000 / 1.5 = about 2,667 requests a second.

Press 2,800. The Payments database saturates: 1,400 payments x 3 = 4,200 writes, against 4,000.

The Backup PSP card is red too, at 280 of its 300. That is under its limit, so the result does not name it.

The trade-off:

The bigger writer doubles the database bill, to $1,947 a month for the pair. Sizing down after January saves about $973 a month for nine months.

Stage 3: A bigger writer for the peak

• Load: 1,400 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $2,925/mo

Check in: Which number on the database card did you change, and why not the other one?

A hot merchant without a hot row

The interviewer asks: A creator's product drop sends one merchant a quarter of the peak. What breaks?

One merchant gets 25% of the peak: 0.25 x 700 = 175 payments a second. The writer has room, so the question is one row, not the machine.

We work it by hand.

Say each payment updates the merchant's balance row inside the record transaction. The row stays locked from its UPDATE to the commit.

We assume the commit waits about 2 ms, for 4 of 6 storage copies, and the other statements take about 1 ms.

UPDATE first, then the ledger inserts, then commit: about 3 ms. One row takes about 333 a second, and 175 use 53% of it.

UPDATE last, just before commit: about 2 ms, about 500 a second, 35% used.

Lock, call the PSP, update, commit: about 1,003 ms. The row then takes about 1 a second, and 175 arrive.

Our choice puts no shared row on the payment path. Ledger rows are insert only, and a job folds them into the balance every few seconds.

If a balance must be live, use 8 sub-balance rows per merchant, picked by attempt ID. They take 8 x 333 = about 2,667 payments a second for that merchant.

The trade-off:

A folded balance is a few seconds behind the ledger. That is the price of having no hot row on the payment path.

Check in: One merchant gets 175 payments a second. Where do you write its balance, and why?

The primary PSP goes down

The interviewer asks: The primary PSP goes down at 12:01 on Black Friday. What happens?

The primary answers errors and timeouts. A circuit breaker on it trips and sends every new payment to the backup.

The backup's contract is still 300 a second. It now gets all 700.

The breaker watches three signals over a 5-second window. Errors over 20%, p95 latency over 3 seconds, or more than 1,200 calls open trips it.

The 1,200 is 80% of the 1,500 contracted open calls, and p95 is the time 95% of calls beat.

Its states: closed, calls flow; open, calls go to the backup at once; half-open, after 30 seconds, 1% of calls probe the primary.

Thirty seconds of healthy answers close it again.

Why latency and open calls? A primary slowed to 5 seconds a call gives no errors under an 8-second timeout, and open calls climb to 560 x 5 = 2,800.

At 1,500 open, it completes only 1,500 / 5 = 300 a second. Open calls pass 1,200 within 1 to 2 seconds, so the breaker trips in its window.

While it decides, the primary's share keeps arriving: 560 x 5 = about 2,800 attempts fail or go unknown.

Our target is 99.95% of attempts on the peak day, in one Region. That allows 26.5 million x 0.05% = 13,250 failures, so 2,800 fit.

The backup passes on 300 a second, so 400 of every 700 payments a second fail. A backup sized for its everyday share is not a backup.

A 30-minute outage at the peak fails 400 x 1,800 = 720,000 checkouts, about $78 million of orders.

A timeout at the primary is "unknown". Retry it at the backup, and the card can be charged twice, so we ask the primary by its attempt ID.

A failure we know never reached the primary, such as a refused connection, is safe to retry at the backup at once.

The backup can charge the card at all because cards are held in a third-party vault, not in one PSP.

Step through the figure to see the breaker trip and the backup fill.

Press 1,400 on Stage 4. The Backup PSP saturates. The Primary PSP card shows no numbers: it is down, so the router sends it nothing.

Read the stage's warnings. The Backup PSP line says it is over its configured capacity of 300 a second.

The latency tile shows "≥ 4.00s", a lower limit: SysPad takes the backup's 1,000 ms and multiplies it by 4 at full load.

The writer gets 1,400 + 300 = 1,700 writes. The cost tile shows $2,801/mo, less than Stage 3 only because fewer webhooks arrive.

Press 400. On an ordinary day, the backup alone carries everything: 200 of its 300.

The trade-off:

A backup costs nothing until the day it is needed. Its contract must cover that day, not an ordinary one.

The primary PSP goes down

1. On Black Friday the primary takes 560 a second and the backup 140. Both have room.
2. A slow primary gives no errors, but open calls pass 1,200 within 2 seconds. The circuit breaker opens, and new payments go to the backup.
3. The primary goes down, and the backup gets all 700. Its contract takes 300, so 400 payments a second fail.
4. A payment in flight when the primary died may be charged. We mark it unknown and never retry it at the backup.
5. Contract the backup for the whole peak, 1,200 a second. Now 700 fits, at 58%.

Stage 4: The primary PSP is down, and the backup takes everything

• Load: 1,400 req/sec
• First to saturate: Backup PSP
• Estimated cost: $2,801/mo

• Latency past saturation is a lower bound, not an estimate. Backup PSP is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Backup PSP: External dependency above its configured capacity (300 rps).

Check in: The backup was at 47% before the outage. Why is it red now?

Contract the backup for the whole peak

The backup must carry everything on the day the primary is down. Its limit is a contract, and Stripe asks for 6 weeks' notice for a large increase.

We raise the backup's contract to 1,200 a second. Next year's peak is 870, and the year after that is 870 x 1.24 = 1,079.

The $4.6 million a minute is also an average over a minute, so single seconds run higher. 1,200, with 1,200 calls open, covers both.

700 / 1,200 = 58.3%. Every webhook arrives again, so the writer is back at 2,100 writes, 52.5%.

We still send the backup 20% every day, and we test the failover monthly by moving real traffic.

With this mix, the backup runs out at an entry of 1,200 / 0.5 = 2,400, and the writer at 2,667. The walls are close, so we size them together.

Press 1,400 on Stage 5. Nothing saturates, and the Primary PSP card stays empty: it is still down.

The note on "First to saturate" names the Backup PSP, at 58%. The latency tile shows 1.00s again: the PSP call, now through the backup.

The cost tile shows $2,925/mo, the same as Stage 3. The fix is a contract, not a machine.

Press 2,500. The Backup PSP saturates: 1,250 payments a second, against 1,200.

The Payments database card turns red too, at about 93%. That is under its limit, so the result does not name it.

The trade-off:

The contract adds nothing to this bill. It is the cheapest insurance on the page, and it must be signed before the season.

Stage 5: The backup contracted for the whole peak

• Load: 1,400 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $2,925/mo

Check in: What did you change, and what did it cost?

The API and the data model

The interviewer asks: Walk me through the API and the data model.

None of these parts breaks under load, so no stage is about them. An interviewer asks about each one, so we answer each in a line or two.

Pay: POST /v1/checkouts/{checkout_id}/attempts with the amount, currency and vault token. A new UUID per attempt goes in the Idempotency-Key header.

It returns 200 with the attempt ID and the outcome: succeeded, requires_action or failed. Until the outcome is known, it returns 202, processing.

Errors: 402 for a decline. 409 for the same key with a different body, or for a checkout that already has a live or paid attempt.

Status: GET /v1/attempts/{id} returns the status, amount and PSP reference, or 404.

Refund: POST /v1/payments/{id}/refunds, with its own key. It returns 409 for more than was captured.

Balance: GET /v1/merchants/{id}/balance returns available and pending, per currency. Webhooks: POST /webhooks/{psp} answers 200 within milliseconds.

Data: payment_attempts is keyed by the attempt ID, which is the idempotency key. It holds the checkout, merchant, amount, currency, status and PSP reference.

A partial unique index on the checkout, over processing, unknown and succeeded attempts, allows one live or paid attempt per checkout.

ledger_entries is insert only, unique on attempt and entry type. processed_events is keyed by PSP and event ID, kept 30 days.

merchant_balances holds available and pending per currency, folded from the ledger every few seconds.

Cards: two PSPs can charge one card only if it is held somewhere neutral. This design assumes a third-party vault, with hosted fields and a forwarding proxy.

"We see only a token" means the vault's token, which works with either PSP. Our servers stay out of the cardholder-data environment.

The vault's proxy sits on the payment path. At an assumed 20 to 50 ms it adds 2% to 5% to the PSP call, and it is not drawn.

Our own vault would put us in full PCI scope, the card security rules, with a yearly audit.

Network tokens from the card networks are portable only if we are the token requestor. Through one PSP, they tie us to it.

Currency: we store the rate taken at authorisation with the payment. Rates refresh every minute from a cache.

Fraud: the PSP's risk score runs inside the call. An in-house model would add about 15 ms, 1.5% of the PSP's second.

The trade-off:

Each choice has a price. The vault adds its proxy to every payment, and an in-house fraud model would add its time too.

Check in: A shopper presses Pay and gets 202, processing. Say what the page does next, and what a second press with the same key gets.

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a shopper or a merchant see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and how a lost dollar is found.

Auth and capture: we authorise at checkout, and capture shipped goods at shipment. An online card authorisation usually holds about 7 days.

A full refund, by our policy: debit merchant payable 10,457 and platform fee 399, and credit PSP receivable 10,856. The PSP keeps its 345.

Read-your-writes: a status read just after paying could reach the reader and see "processing". So for 5 seconds, that shopper reads the writer.

Webhooks: answer 200 at once. After a 1-hour receiver outage at the peak, up to 700 x 3,600 = 2.52 million events come back.

Each event is a write, so the writer's spare caps the drain: 2,520,000 / (4,000 - 2,100) = about 22 minutes at the peak.

At the ordinary peak it is about 12 minutes. So we cap the worker at the writer's spare, or it floods the writer and fails live payments.

Payouts: 1 million merchants at the PSP's 15 creates a second take 1,000,000 / 15 = 18.5 hours. So payouts go as bank batch files.

The dead-letter queue: alarm on any message, fix the cause, then send the messages back. Dedupe and forward-only updates make that safe.

A Region outage of 10 minutes at the peak hits 700 x 600 = 420,000 attempts. Aurora Global Database loses about 1 second of data and recovers in about 10 minutes.

Sharding later, by merchant: a payment's three writes stay on one shard. But a merchant with 25% of the peak puts 525 writes a second on its shard.

Say it out loud: "The system costs about $1,300 in an ordinary month. The money is in the PSP fees: $3.45 a payment at list price."

At list price, 2.69 billion payments pay about $9.3 billion a year in PSP fees. A platform this size negotiates lower: at an illustrative 2%, $5.8 billion.

Minutes 0 to 5: requirements. Pay, record, status, webhooks, payouts and reconciliation. Say "never charge twice" first.

Minutes 5 to 10: the numbers. $292.3 billion / $108.56 = 2.69 billion payments a year, 200 a second at an ordinary peak, 700 on Black Friday.

Minutes 10 to 15: the API with an idempotency key per attempt. Attempts, ledger entries with their PSP fee, and processed events.

Minutes 15 to 22: create, call the PSP, record in double entry, and webhooks through a queue.

Minutes 22 to 28: exactly-once effects. The retry order, the webhook race, and the key that stops each repeat.

Minutes 28 to 34: Black Friday. The writer is the wall, readers do not help, scale up, and the hot merchant by hand.

Minutes 34 to 40: the primary PSP fails. The circuit breaker, a backup sized for the whole peak, and unknown is not failed.

Minutes 40 to 45: reconciliation and its 3-day window, payouts, the vault, a Region outage, and sharding later.

The trade-off:

Each answer has a price. The backup contract, the vault and the monthly failover test cost effort every month, for a day that may never come.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.