SysPad › Lessons › security lessons › AWS KMS

How AWS KMS works

Managed encryption keys. The trap: a REGIONAL request quota (5,500-100,000 req/s) shared account-wide - services that call KMS per request throttle at scale. Cache data keys (envelope encryption) so you call it rarely. $1/key/month + $0.03 per 10k requests.

There is a service in your AWS account with a hard requests-per-second ceiling that your API gateway, your workers, your CI pipeline, and every teammate's experiments all share. Most teams discover it exists on the day production throttles. It is KMS, the encryption key service, and the fix is one caching pattern.

What you will learn

Why does a fast service suddenly throttle on encryption at scale?
What does "the quota is regional and account-wide" actually mean for your design?
How does envelope encryption with data key caching make KMS calls almost free?
When is calling KMS per request acceptable?

One service holds all the keys

KMS stores and guards encryption keys. The keys never leave it: you send data (or a data key) to KMS and it encrypts or decrypts using the key it holds. S3, EBS, RDS and friends all lean on it under the hood.

The shared regional ceiling

KMS cryptographic operations have a regional request quota: 5,500 req/s for symmetric keys in most regions, 50,000 to 100,000 req/s in the biggest ones. It is shared by the ENTIRE account in that region, every service, every pipeline, every person.

Envelope encryption: the fix

Envelope encryption means KMS gives you a data key once. You cache it in memory and encrypt/decrypt locally at CPU speed, no network, no quota. KMS is only called when the cached key expires or rotates.

The AWS Encryption SDK ships data key caching out of the box. A 99% cache hit rate turns 10,000 KMS calls/s into 100.

⚓ In the SysPad simulation, drag the "Data key cache hit rate" slider on a KMS node and watch the utilization collapse. That slider is this whole lesson.

If the data key is cached in my process memory, is that safe?

That is the accepted trade. The data key in memory is scoped, rotated, and never persisted; compromise of a running process was already game-over for the data it can read. You bound the blast radius with short cache TTLs and per-context keys, not by calling KMS per request.

What it costs

Two meters: $1 per key per month, and about $0.03 per 10,000 requests. Neither hurts at sane call rates, which is the point: the quota bites long before the bill does, and both point at the same fix.

Flashcards

Your checkout service throttles on KMS during a sale. The data pipeline team says their exports started failing at the same time. Coincidence?: No. The KMS request quota is regional and account-wide, so one hot caller starves every other KMS user in the account. The fix is envelope encryption with data key caching on the hot path.
A teammate proposes encrypting every API payload by calling KMS Encrypt on each request, at 5,000 RPS. What do you point at?: The regional quota (5,500/s default) and the envelope pattern: get a data key once, cache it, do crypto locally. KMS calls should scale with key rotations, not with traffic.
KMS spend looks tiny on the bill. Does that mean the design is fine?: Not necessarily: at $0.03 per 10k requests the bill stays small even at call rates that exhaust the quota. On KMS, throttling arrives long before cost does.

Sources

Open the SysPad canvas · Official AWS KMS documentation