SysPad › Interview practice › Design a digital wallet

Design a digital wallet

The interviewer says: "Design a digital wallet." People and merchants hold money in wallets, and send it to each other instantly.

A user sees a balance and recent history. Users deposit money from a card or a bank, and withdraw to a bank.

We never lose money and never create it. A transfer applies exactly once, and a user's transfer never takes a balance under zero.

Our side of a transfer takes under 50 ms, because a person is waiting at a till or in a chat.

History may run about a second behind. Every balance must be rebuildable from the journal, for regulators and disputes.

How we work it: Interview ready means we prove every transfer moves money once and never overdraws, and we know which wall a busy wallet hits first.

Do the numbers before the boxes

An interviewer gives a scale, not a load. If we draw before we do the math, we cannot say why any box has its size.

We size for a wallet about PayPal's size: 26.3 billion payments in 2024. We treat each payment as one wallet transfer, an assumption.

A day: 26.3 billion / 365 = 72.05 million. A second on average: 72,050,000 / 86,400 = 834.

We take a peak factor of 2.2: the busiest hour carries 2.2 times the average. 834 x 2.2 = 1,835, rounded up to 2,000 transfers a second.

We assume about 3 views of a balance and history per transfer: the payer before and after, and the payee after. That is 6,000 views a second.

We measure load in requests per second: 2,000 + 6,000 = 8,000. Transfers are 25% of it.

Twist 1, a TV charity appeal: viewers send 600 donations a second to one wallet, each with its 3 views.

That is 2,600 transfers and 7,800 views a second: an entry of 10,400.

Twist 2, a festival day: 2.5 times the ordinary peak, our assumption. That is 5,000 transfers, 15,000 views, and an entry of 20,000.

For storage only: PayPal counts 434 million active accounts.

Check in: Close the page and derive 8,000 requests per second from 26.3 billion payments a year. Then derive the appeal and the festival loads.

Show an answer

26.3B / 365 = 72.05M a day; / 86,400 = 834 a second; x 2.2 = 1,835, rounded up to 2,000 transfers. 3 views each: 6,000. Total 8,000.

Appeal: 2,000 + 600 = 2,600 transfers, x 3 = 7,800 views: 10,400. Festival: 2.5 x 2,000 = 5,000 transfers, 15,000 views: 20,000.

A transfer is one transaction

Money moves from one wallet to another. If the debit lands and the credit does not, money disappears. If the credit lands alone, money is created.

Wallets live in a DynamoDB table, the Balances table. A transfer is one transaction: a set of writes that all apply, or none do.

It has 3 items. Debit the payer: subtract the amount, only if the balance covers it. Credit the payee: add the amount.

Put the transfer record, only if no record with that transfer ID exists yet. The transfer records are the journal.

If any condition fails, nothing applies. A payer with too little money gets 402, and no balance changes.

The overdraft check is part of the transaction, so a user's transfer can never take a balance under zero.

Holds: a card deposit from a new or risky account is held for a few days. The debit condition then checks the balance less held funds.

A chargeback is a system debit without the balance condition, weeks later. The balance can go negative, the wallet is restricted, and its next credits repay it.

DynamoDB sells write capacity units (WCU) per second. One is a write of an item up to 1 KB.

A transaction writes each item twice, once to prepare and once to commit. So a transfer costs 3 x 2 = 6 write units, not 3.

Each item is 1 KB or less, so 2,000 transfers a second need 2,000 x 6 = 12,000 WCU.

We buy provisioned capacity: a fixed number of units, paid for whether used or not. We buy 24,000 WCU, so 12,000 is 50%.

Why that much room: DynamoDB takes minutes to add units on its own, and the appeal later needs 15,600 at once.

Storage: a transfer record is about 500 bytes, and so is a wallet. We count 1 KB as 1,000 bytes.

90 days of records: 72.05 million x 90 x 500 bytes = 3,020 GiB. Wallets: 434 million x 500 bytes = 202 GiB. Together, about 3,220 GiB.

Older records go to S3. Each copy of the Wallet API is a task with one vCPU, a virtual processor.

A request costs about 1 ms of CPU, so a task serves 1,000 a second. The API uses auto-scaling, keeping each task near 70% busy.

8,000 / (1,000 x 0.7) = 11.4, so 12 tasks run, at 67%.

The trade-off:

Atomic costs double. Without a transaction a transfer would be 3 write units, but not atomic. The 6 write units are the bill.

Check in: Why do 2,000 transfers a second need 12,000 write units, and not 6,000?

Show an answer

Each transfer writes 3 items, and a transaction writes each item twice, to prepare and to commit. So 2,000 x 3 x 2 = 12,000.

Views from a read model

Views are three of every four requests. If they read the Balances table, every view competes with the transfers for one table.

So views read a separate History table: a read model with each wallet's balance and last 10 lines.

Splitting the write path from the read path like this is called CQRS, command and query responsibility segregation.

A projector reads the Balances table's change stream. A transfer changes 3 items, so the stream carries 3 change records: payer, payee and transfer record.

The projector writes one item per record: 2 history lines, one per wallet with its balance after, and 1 transfer status.

2,000 transfers make 6,000 records a second. A projector task takes 0.5 ms a record, so 5 tasks run, at 60%.

A view reads 10 lines of 300 bytes, 3 KB, in one query. An eventually consistent read of up to 4 KB costs half a read capacity unit (RCU).

Reads: 6,000 views x 0.5 = 3,000 RCU, against the 5,000 we buy: 60%. Writes: 6,000 items at 1 WCU each, against 12,000 WCU: 50%.

90 days of history: 3 items a transfer x 72.05 million a day x 90 x 300 bytes = 5,440 GiB.

The history runs about a second behind: the stream, then the projector. That is our estimate.

Stream records of one transaction may arrive at different times. So each write is keyed by wallet and transfer, and the projector never assumes they arrive together.

The Send screen shows an exact balance by reading the wallet item, strongly consistent, for 1 RCU.

That read is for the screen only. The transfer's own condition protects the money, so a stale screen can only lead to a declined transfer.

Step through the figure to follow one transfer into the read model.

The trade-off:

The read model is a second table to pay for, and it is about a second behind. The screen that must be exact reads the wallet item instead.

Writes on one table, views on another

  1. A transfer is one transaction on the Balances table. Views never touch it.
  2. A projector turns each transfer's 3 change records into 2 history lines and 1 status. The read model runs about a second behind.
  3. The app reads the read model. The Send screen reads the wallet item, but the transaction's condition is what protects the money.

Check in: A transfer changes 3 items. Say what the projector writes for it, and why a repeat changes nothing.

Show an answer

3 change records arrive: payer, payee and transfer record. The projector writes one item for each: 2 history lines, one per wallet, and 1 transfer status.

Each write is keyed by wallet and transfer, or by transfer, so a record that arrives twice writes the same item again and changes nothing.

Run the design

Stage 1 runs this design in SysPad at 8,000 requests per second. Each card is a part of the system, and each arrow carries requests.

Read the four tiles. "First to saturate" reads "Nothing at this load". Its note names the Wallet API, at 67%.

A part saturates when it gets more requests than it can serve.

A card turns red at 90% of its limit. A part is saturated only past 100%, and only then does the result name it.

The latency tile shows 5ms. It shows the slowest path, which here is the transfer path.

That is 1.5 ms at the load balancer, 1 ms in the API and 2 ms at the table: 4.5 ms.

The cost tile shows $25k/mo, and DynamoDB is 79% of it. The tile holds the peak all month.

About $3,405 of it is egress, data sent out to the internet: 8,000 answers a second, about 2 KB each on average.

AWS bills egress per GB, after 100 free GB a month.

An average month runs at 8,000 / 2.2 = 3,636 requests a second. With the tables' units kept at the peak, it costs about $21,855.

Only the charges that grow with traffic fall: the load balancer, the API and projector tasks, and egress.

With auto-scaling holding the tables near 70% busy, as the API does, an average month would cost about $10,200.

The History table runs out first: 5,000 RCU is 10,000 views a second, at an entry of 10,000 / 0.75 = 13,333.

Press 14,000. The History table saturates: 10,500 views need 5,250 read units, against 5,000.

Press 17,000. The Balances table saturates too: 17,000 x 1.5 = 25,500 write units, against 24,000.

Stage 1: Transfers as transactions, views from a read model: 8,000 a second

  • Load: 8,000 req/sec
  • First to saturate: Nothing at this load
  • Estimated cost: $25k/mo

Check in: Which table runs out first as load grows, and at what load? Then name the next wall.

Show an answer

The History table, on reads: 5,000 RCU is 10,000 views a second, at an entry of 13,333.

Then the Balances table: 24,000 write units is 4,000 transfers a second, at an entry of 16,000.

One transfer through every state

The interviewer asks: The app sends a transfer, the network drops, and the app sends it again. What happens to the money?

Every request can arrive twice, and a crash can happen at any step. Delivery exactly once is not possible across a network.

So we aim for exactly-once effects: every step may repeat, and a repeat changes nothing.

This is a question about keys and states, not about load. So we reason about it by hand, and no stage goes with this beat.

An attempt is one user intent to send money. It gets one key, the transfer ID, made when the user presses Send.

That key is the idempotency key. Retries of the attempt, after a timeout or an app restart, reuse it.

A new intent gets a new key, and so does a second try after a decline.

The answer rule: the API reads why a transaction cancelled. A conflict, or no record yet, means the first is in flight: 202, processing.

A failed "no record yet" condition on the record means it committed. The API reads the record and answers 200 with its stored outcome.

State 1, before the transaction: nothing has happened. The client retries with the same key.

State 2, in flight: a duplicate arrives while the first is still running. It conflicts, answers 202, processing, and the app polls.

State 3, the client times out: the result is unknown to it. It retries with the same key, and the answer rule decides.

State 4, committed but the answer was lost. The retry fails its condition, and gets 200 with the stored outcome: completed or declined.

A decline cancels the whole transaction, record and all. So the API stores the decline with a second, separate conditional put.

That costs the cancelled 6 write units, still billed, plus 1: 7. At an assumed 2% declines, that adds 2,000 x 0.02 = 40 write units a second.

The SDK's own request token lasts only 10 minutes. The transfer record, not the token, is the guard that lasts.

State 5, the projector crashes after 1 of a transfer's 3 writes. The stream gives the records again, and the keyed writes change nothing.

A withdrawal to a bank is a saga: steps that each commit on their own, with an undo step for a failure.

First debit the wallet and put a pending withdrawal, in one transaction. Then call the bank, with the withdrawal ID as its key.

A crash before the bank call leaves the pending record, so a sweeper resumes it. A final bank failure credits the wallet back.

Step through the figure to follow one transfer through each state.

The trade-off:

Every write carries a key, and a retried or declined transfer pays its write units again. The money is correct, and the bill is a little higher.

One transfer, T1, through every crash

  1. All three writes apply, or none do. The record under T1 is the guard.
  2. A duplicate in flight gets 202 and polls. It never debits twice.
  3. The answer was lost after the commit. The retry's condition fails, and the API answers 200 with the stored outcome.
  4. A crash at any step leaves either nothing or the record. The retry finds out which.
  5. A declined attempt cancels everything. A second put stores it as declined, so a retry gets 200 declined.

Check in: The app sends a transfer twice, and the first answer is lost after the commit. Say what each try gets, and why the money moves once.

Show an answer

The first try commits: debit, credit and the transfer record, all or nothing. Its answer is lost.

If the second try arrives while the first is in flight, it conflicts and gets 202 processing. The app polls.

If it arrives after the commit, its condition on the record fails, and the API answers 200 with the stored outcome.

The record keyed by the attempt is the guard, so nothing is debited twice.

The journal, double entry and replay

The interviewer asks: How do you prove every balance is correct?

A balance is one number. An auditor wants to see how it got there, and that no money was made or lost.

This too is a question about records, not about load, so we work it by hand.

The transfer records are the journal, and they never change. This is event sourcing: the history of events is the record of truth.

Each record's lines sum to zero. That is double entry. With no fee, there are two lines: debit the payer, credit the payee.

With a fee, the same record has three lines. Debit the payer the amount plus the fee, credit the payee the amount, and credit fee revenue the fee.

The fee wallet takes a line of every paid transfer, so it is never an item in a transfer. Its balance is summed from the records in batches.

So a paid transfer is still 3 items and 6 write units.

The balance is a stored snapshot on the wallet item, changed in the same transaction as the journal entry.

Why not fold the journal on every read? A wallet with 10,000 transfers would need 10,000 reads for one balance.

Each wallet item carries a version that every transfer increments. To replay, we fold a wallet's journal from zero and compare its balance and version.

After 90 days the journal lives in S3, so replay reads the export. We replay a sample of wallets every night.

The invariant: all balances, plus credits and fees not yet summed in, equal the money held for customers: deposits less withdrawals less fees.

Reconciliation checks that invariant daily, from the export. Any difference stops payouts until it is explained.

We also match deposits against the card processor's settlement and withdrawals against the bank's report, over a window of a few days.

Amounts are whole cents, never floats, and each wallet holds one currency.

The trade-off:

Snapshot and journal must change together, so every transfer is a transaction. Replay and reconciliation are daily jobs someone must watch.

Check in: A $10 transfer carries a 30 cent fee. Write its lines, and say how many items the transaction writes.

Show an answer

One record with three lines that sum to zero: debit the payer 1,030 cents, credit the payee 1,000, credit fee revenue 30.

The fee wallet is summed from the records later, so the transaction still writes 3 items: payer, payee and record. 6 write units.

Two debits at once, and why there is no two-phase commit

The interviewer asks: Two payments leave the same wallet at the same moment, and only one fits. Can it go under zero?

Each debit checks the balance. If both check first and both write after, the wallet goes negative. We reason about this by hand.

The check is inside the transaction, so it cannot be raced. DynamoDB does not queue the two debits like a SQL row lock.

One transaction succeeds, and the other is cancelled with a conflict. The client retries it with backoff, because the SDK does not retry a cancelled transaction.

On the retry, the condition sees the new balance. If the balance is now too low, the answer is 402.

Why not one Postgres primary? A conditional UPDATE holds the row lock to commit, about 2 to 5 ms, so one row takes 200 to 500 updates a second.

At our assumed 4 ms of CPU a transfer, 64 vCPUs do 64 x 1,000 / 4 = 16,000 transfers a second. The festival fits, so one primary is a fine answer here.

But its failover stops every transfer. Aurora recovers in under 60 seconds, often under 30: at 2,000 a second, 30 seconds is 60,000 transfers.

DynamoDB removes the single writer and its failover. It costs 6 write units a transfer, and no SQL.

Across SQL shards, a transfer needs a protocol. Two-phase commit holds locks on both shards until a coordinator decides, and a coordinator crash blocks them.

A saga runs local steps in order: debit A, then credit B, with a credit back to A if B fails. Money is visibly in flight between steps.

Order a saga debit first, never credit first, so a crash can never create money.

TC/C, try then confirm or cancel, first reserves on both sides: a hold on A and a credit waiting on B. Then it confirms or cancels both.

DynamoDB splits a table into partitions, and runs its own prepare and commit across them in one Region. So inside a Region we need none of these.

Sharding SQL by wallet, a cross-shard transfer is 3 local transactions instead of 1. With 4 shards that averages 3 - 2/4 = 2.5 a transfer.

So 4 writers give only 4 / 2.5 = 1.6 times one writer. 16 shards give 16 / 2.875 = 5.6 times.

Transactions are atomic only in the Region that ran them. So each wallet has one home Region that takes its writes.

Step through the figure to compare the four answers.

The trade-off:

A cancelled debit costs its write units and a retry. On one very busy wallet, those conflicts become the wall, as the appeal shows.

Two debits on one wallet, and across shards

  1. Two debits race on one wallet. One commits, the other cancels, and its retry sees the new balance.
  2. Two-phase commit locks both shards until the coordinator decides. If it crashes, both stay locked.
  3. A saga debits first, then credits, with an undo step. Money is in flight between them.
  4. With 4 shards, 3 of 4 transfers cross, so 4 writers give only 1.6 times one.

Check in: Two debits of one wallet arrive at once, and only one fits. Say what each gets, on DynamoDB and on SQL.

Show an answer

DynamoDB: one transaction commits; the other is cancelled with a conflict. Its retry sees the new balance and gets 402.

SQL: the second waits for the row lock, about 2 to 5 ms, then its condition fails and it gets 402. Either way, the balance never goes negative.

One wallet, one key, one ceiling

The interviewer asks: A TV charity appeal sends 600 donations a second to one wallet. What breaks?

The appeal runs for an hour. With their views, the load is 2,600 transfers and 7,800 views a second: an entry of 10,400.

The Balances table gets 2,600 x 6 = 15,600 write units a second, 65% of its 24,000. The table has room.

But every donation's transaction writes the appeal's wallet: 600 x 2 = 1,200 write units a second, on one item.

DynamoDB keeps one item in one partition, and a partition takes at most 1,000 WCU a second.

So one item takes at most 500 transactional updates a second. Buying more units for the table does not move that.

Overlapping transactions on one item also cancel each other, so the real limit is lower. AWS publishes no figure for it.

Stage 2 shows the appeal at its own load. "First to saturate" names the Balances table.

Read the stage's warnings. The hottest key draws 1,200 WCU a second, past the 1,000 one partition takes.

No write-capacity warning appears: the table is at 65%. One key is the wall, not the table.

The latency tile now starts with "≥", a lower limit, because the table on the transfer path is saturated.

Fewer change records reach the projector while the Balances table throttles. Throttled and cancelled donations fail, and donors retry, which adds load.

The cost tile shows $26k/mo.

The trade-off:

The table has 8,400 write units spare, and one wallet has none. Money cannot fix a single key.

Stage 2: A charity appeal inside every donation's transaction

  • Load: 10,400 req/sec
  • First to saturate: Balances table
  • Estimated cost: $26k/mo
  • Latency past saturation is a lower bound, not an estimate. Balances table is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
  • Balances table: Hot-partition write throttling: hottest key draws 1200 WCU/s, above the 1000 WCU/s per-partition ceiling.

Check in: The table is at 65%. Why is its card red?

Show an answer

One item, the appeal's wallet, takes 1,200 write units a second against the 1,000 one key allows. More units for the table do not help that key.

Pending credits, folded in batches

The appeal's wallet must leave every donation's transaction, and its credits must still arrive.

A donation now debits the donor, puts a new pending-credit item, and puts the transfer record. Still 3 items and 6 write units.

Pending credits are spread over 16 keys: the appeal wallet and a bucket number. Each key takes 37.5 puts a second, 75 write units.

A folder queries the 16 keys and adds the credits up. Each fold is one transaction: up to 99 conditional deletes, plus 1 update adding their sum.

That is 100 actions, the limit for one transaction. A crash leaves either nothing or the whole fold, so no credit is ever added twice.

At 600 donations a second, that is 600 / 99 = about 7 folds a second, run one after another so they never conflict.

The appeal's balance is up to about a second behind, and credits can never overdraw it.

Why not split the appeal into 10 sub-wallets? Each would still take 60 donations a second, inside transactions on one item.

Those overlap and cancel each other. Pending credits put each donation on a new item, so there is nothing to conflict on.

The folds add 600 deletes x 2 + 7 updates x 2 = about 1,214 write units a second, which this stage does not draw.

With them, the Balances table runs near 70% at this load.

Step through the figure to see the credits fold.

Press 10,400 on Stage 3. Nothing saturates. The note on "First to saturate" names the History table, at 78%.

The cost tile shows $26k/mo, about the same as Stage 2. The fix is a design, not units.

The next wall is the History table's reads again, at an entry of 13,333. Press 14,000: it saturates, and the appeal does not.

With the folds, the Balances table is near 94% there, close to its own wall.

The trade-off:

The fold costs write units of its own, and the charity sees its total rise in steps, about a second behind each donation.

The appeal wallet, and the fix

  1. Every donation's transaction writes the appeal wallet: 1,200 write units on one item. The table is at 65%.
  2. Ten sub-wallets still put 60 transactions a second on each item. Those overlap and cancel.
  3. Each donation writes its own pending-credit item, over 16 keys. About 7 folds a second each delete up to 99 and add their sum, in one transaction.

Stage 3: Donations as pending credits, folded in batches

  • Load: 10,400 req/sec
  • First to saturate: Nothing at this load
  • Estimated cost: $26k/mo

Check in: Why not split the appeal wallet into 10 sub-wallets? And why is the fold one transaction?

Show an answer

Each sub-wallet would still get 60 donations a second inside transactions on one item, and those overlap and cancel. Pending credits share no item.

The fold deletes its pending credits and adds their sum in one transaction. A crash leaves nothing or the whole fold, never a credit counted twice.

Festival day on the ordinary units

The interviewer asks: Festival day: transfers run at 2.5 times the ordinary peak. What breaks?

The appeal is over. 5,000 transfers and 15,000 views a second: an entry of 20,000.

The Balances table needs 5,000 x 6 = 30,000 write units, against 24,000. The 15,000 views need 7,500 read units, against 5,000.

No key is hot this time. Both tables are simply short of units we bought.

That is money, and the date is known. Auto-scaling reacts in minutes, so a planned spike is provisioned ahead.

Press 8,000 on Stage 4. At the ordinary peak, nothing saturates.

Press 20,000. Both tables saturate. Read the stage's warnings: the Balances table is at 125% of its write units.

The History table is at 150% of its read units, and at 100% of its write units.

The latency tile starts with "≥" again. The cost tile shows $31k/mo.

The trade-off:

Units bought for a festival sit idle the rest of the year, unless we lower them after.

Stage 4: Festival day on the ordinary units, 20,000 a second

  • Load: 20,000 req/sec
  • First to saturate: Balances table
  • Estimated cost: $31k/mo
  • Latency past saturation is a lower bound, not an estimate. Balances table is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
  • Balances table: Write capacity 125% used (provisioned): 30000 of 24000 WCU/s at 30000 RPS.
  • History table: Read capacity 150% used (provisioned): 7500 of 5000 RCU/s at 27000 RPS.
  • History table: Write capacity 100% used (provisioned): 12000 of 12000 WCU/s at 27000 RPS.

Check in: Which red card is design, and which is money?

Show an answer

Both are money this time: each table is short of provisioned units, and no single key is over its wall.

Provision both tables for the day

We raise both tables the day before, and lower them after.

Balances: 40,000 write units, the AWS default quota for one table. 30,000 / 40,000 = 75%.

Past 40,000, we ask AWS for a higher table quota, weeks ahead.

History: 11,000 read units and 22,000 write units. Reads use 7,500 / 11,000 = 68%, and writes 15,000 / 22,000 = 68%.

DynamoDB limits how often provisioned units can be lowered in a day, so we plan the change down too.

Press 20,000 on Stage 5. Nothing saturates. The note on "First to saturate" names the Balances table, at 75%.

The cost tile shows $44k/mo, if the units were held all month. The day is what they are for.

The Balances table's 40,000 write units take 6,667 transfers a second, at an entry of 26,667.

Press 28,000. The Balances table saturates: 7,000 transfers x 6 = 42,000 write units, against 40,000.

The History card turns red too, at about 93%. It is under its limit, so the result does not name it.

The trade-off:

The 6 write units a transfer set the bill: 5,000 transfers a second use three quarters of a table's default quota.

Stage 5: Both tables provisioned for the day

  • Load: 20,000 req/sec
  • First to saturate: Nothing at this load
  • Estimated cost: $44k/mo

Check in: What is the next wall after the festival fix?

Show an answer

The Balances table, at an entry of about 26,667: 6,667 transfers a second x 6 = 40,000 write units, the default table quota.

The API and the data model

The interviewer asks: Walk me through the API and the data model.

None of these parts breaks under load, so no stage is about them. An interviewer asks about each one, so we answer each in a line or two.

Transfer: POST /v1/transfers with the payer, payee, amount in cents, currency and a note. The transfer ID goes in the Idempotency-Key header.

It returns 200 with the outcome and the payer's new balance, or 202, processing, while the attempt is in flight.

Errors: 402 for too little money, with the decline stored under its key. 409 for the same key with a different body. 423 for a frozen wallet.

Status: GET /v1/transfers/{id} returns the stored record, strongly consistent, or 404.

Wallet: GET /v1/wallets/{id} returns the balance and last 10 lines from the read model. With exact=true, it reads the wallet item.

Deposits and withdrawals: POST /v1/topups and /v1/withdrawals return 202 with a saga ID.

Data: the Balances table holds three kinds of item, each under its own key prefix.

A wallet item, W# and the wallet ID, holds the balance, held funds, currency, version and status, about 500 bytes.

A transfer record, T# and the transfer ID, holds its lines and times, about 500 bytes. It is kept 90 days, then exported to S3.

A pending credit is keyed by P#, the wallet ID and a bucket, with the transfer ID inside. The fold that adds it deletes it.

The History table is keyed by wallet, then time and transfer ID. A line holds the amount and the balance after it, about 300 bytes.

Deposit: charge the card through a card processor, then credit the wallet only on its "succeeded", never on "processing".

Reads come in three kinds. The app's balance and history read the read model: about a second behind, at half a read unit.

The Send screen's balance reads the wallet item, strongly consistent, for 1 RCU. A transfer's status reads the record, also exact.

The trade-off:

Each choice has a price. Three item kinds share one table, so the table must be sized for all of them.

Check in: A user retries a transfer after it was declined. Same key or a new one? Say what each one gets.

Show an answer

A retry of the same attempt, with the same key, gets 200 with the stored "declined".

A new try after adding funds is a new intent with a new key, so it runs as a new transfer.

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does a user see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and how stale the data can get.

The 50 ms budget: a 3-item transaction takes about 25 ms at p50 and 92 ms at p99, by an independent benchmark. A real transfer is about 28 ms, or 43 ms with fraud checks.

The projector stops for 10 minutes: history goes 10 minutes old. 6,000 x 600 = 3.6 million records wait in the stream, which keeps them 24 hours.

Exact reads still work. On restart, 20 tasks take 40,000 records a second, so the backlog clears in 3,600,000 / 34,000 = about 106 seconds.

A transfer between wallets homed in two Regions is a saga. First a debit and an outbound record at home, then a queue, then a deduped credit there.

If that credit fails for good, a compensation credits the payer back. The money is in flight for seconds.

Deposit holds: a bank deposit is held until it settles, 1 to 3 business days. A card deposit from a new account is held a few days.

FX: a conversion is a transfer from the user's USD wallet to their EUR wallet, at a quoted rate stored in the record. Our FX accounts are lines, summed like fees.

A Region fails: writes for wallets homed there stop. We fail over to the other Region's replica and check the last minutes for half-applied transfers.

Crash points in one breath: before, retry; during, all or nothing; after commit, the record answers. Projector, keyed writes; folder, atomic; withdrawal, the pending record.

Say it out loud: "Every transfer is 6 write units, so the transaction is the bill. Without it a transfer is 3, and not atomic."

Minutes 0 to 5: requirements. Transfer, balance, history, deposit and withdraw. Never create or lose money.

Minutes 5 to 10: the numbers. 26.3 billion a year, 2,000 transfers a second, 3 views each, 8,000 a second.

Minutes 10 to 15: the transfer transaction, its 6 write units, and the key per attempt.

Minutes 15 to 22: the read model, and the exact read on the Send screen.

Minutes 22 to 28: overdraft under concurrency. One Postgres primary, two-phase commit, saga and TC/C, and why DynamoDB needs none inside a Region.

Minutes 28 to 34: the appeal. A hot wallet, and pending credits folded in batches.

Minutes 34 to 40: the festival. Units are money, provision ahead, and the 40,000 quota.

Minutes 40 to 45: replay, reconciliation, a crash at each step, and multi-Region.

The trade-off:

Each answer has a price. Pending credits delay a hot balance by about a second, and festival units cost money every day they are held.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.

Say it out loud

How do you get to 8,000 requests per second?
26.3B / 365 / 86,400 = 834 transfers a second. x 2.2 = 1,835, rounded up to 2,000. 3 views each: 6,000. Total 8,000.
Why does a transfer cost 6 write units?
It is one transaction of 3 items, and a transaction writes each item twice, to prepare and to commit. A fee adds a line, not an item.
What is in the transfer transaction?
Debit the payer only if the balance covers it, credit the payee, and put the transfer record only if none exists. All or nothing.
What is an attempt, and what is its key?
One user intent to send money, with one key made when the user presses Send. Retries reuse it. A new intent, even after a decline, gets a new key.
A retry of a transfer arrives. What does it get?
In flight, a conflict or no record yet: 202 processing. After commit, its record condition fails: 200 with the stored outcome.
How is a declined transfer stored?
The decline cancels the whole transaction. A second conditional put stores the record as declined: 6 cancelled write units plus 1, so 7.
What does the projector write for one transfer?
3 change records arrive. It writes 2 history lines, one per wallet, and 1 transfer status, each keyed so a repeat changes nothing.
What protects the money if the Send screen shows a stale balance?
The debit's condition inside the transaction. The exact read is for the screen. A stale screen can only lead to a declined transfer.
Why does the appeal wallet saturate while the table is at 65%?
One item takes 1,000 WCU a second, and 600 donations x 2 = 1,200. Transactions on one item also cancel each other.
How do you fix a hot wallet?
Credits become pending-credit items over 16 keys. About 7 folds a second each delete 99 and add their sum, in one transaction.
Why not one Postgres primary?
It fits: 64 vCPUs do 16,000 transfers a second. But one hot row takes 200 to 500 updates, and a 30-second failover stops 60,000 transfers.
Festival day: both tables are red. What do you do?
Both are money: provision the day before. Balances at 40,000 write units, the default table quota. Past that, ask AWS weeks ahead.

Check yourself

Each transfer is one DynamoDB transaction of 3 items, each under 1 KB. 2,000 transfers a second arrive.

How many write units a second does the Balances table need?

The History table has 5,000 read units. A view costs half a read unit, and views are 75% of the entry rate.

At what entry rate do its reads run out?

In the appeal, the Balances table is at 65% of its write units, but its card is red.

Why?

The appeal wallet gets 600 donations a second inside transactions.

Which fix removes the hot item?

A transaction holds at most 100 actions. Each fold deletes pending credits and adds their sum to the wallet in one transaction. 600 donations a second arrive.

About how many folds a second does the appeal need?

A transfer is still running when its duplicate, with the same key, arrives.

What does the duplicate get?

A transfer committed, but its answer was lost. The client retries with the same key.

What happens?

The Send screen shows a balance read a moment before the user presses Send. Another payment leaves the wallet in between.

What stops an overdraft?

A cross-shard transfer runs as a saga: one step on the payer's shard, one on the payee's.

Which step goes first?

A SQL wallet store is split into 4 shards by wallet. A cross-shard transfer is 3 local transactions, and a same-shard one is 1.

How much more write capacity do 4 writers give?

On festival day both tables are red. No key is hot, and both are short of the units provisioned.

Which red card is design, and which is money?

Your turn

On Stage 1, press 14,000 and choose "Open in SysPad", then press Simulate. The result names the History table.

Two settings on that card look like capacity: "Read capacity units" and "Write capacity units". Only one of them is short.

Change it until the result names no bottleneck and no card is red. Then say why the History table cannot go under 87.5%, however much you add.

Open the last stage in the editor and change it. Or start from an empty canvas and build the design from memory.

Lessons for this problem

The finished architecture

A complete design of this problem in the example gallery. It may size some parts differently.

Previous: Design a payment system · Next: Design a web crawler

All interview practice · Lessons · Open the SysPad canvas