SysPad › Lessons › security lessons › Secrets Manager

How Secrets Manager works

Managed secrets with rotation (DB credentials, API keys). GetSecretValue has a ~10,000 req/s regional quota - fetch once and cache in-process (AWS caching clients default to 1h TTL), never per request. $0.40/secret/month + $0.05 per 10k calls.

Where does your database password live? If the answer is "in an environment variable someone pasted in 2023," you have met the problem Secrets Manager solves. And once you adopt it there is a second lesson waiting: services that ASK for the secret on every request turn a config store into a bottleneck.

What you will learn

What does Secrets Manager give you over environment variables?
How does automatic rotation work without downtime?
Why is fetching a secret per request an anti-pattern?
When is Parameter Store the cheaper right answer?

A vault with an API

Secrets Manager stores credentials (DB passwords, API keys, tokens) encrypted with KMS, gates access with IAM, audits every read, and can rotate secrets on a schedule by invoking a Lambda that updates both the secret and the target database.

Fetch once, cache always

GetSecretValue has a regional quota around 10,000 req/s, and a network round trip of ~25ms. Secrets change rarely, so the correct shape is: fetch at startup, cache in-process, refresh on a TTL. AWS ships caching clients that default to a 1 hour TTL.

Rotation without downtime

Managed rotation flips credentials in stages: create a new password, update the database, verify, then mark it current. Done right (two valid users, or the overlap window) nothing breaks mid-rotation, and a leaked credential has a shelf life measured in days instead of years.

What about Parameter Store? It also holds strings.

SSM Parameter Store (standard tier) is free and fine for plain config and even SecureString secrets IF you do not need managed rotation or cross-account access. The rule of thumb: rotating credentials → Secrets Manager ($0.40/secret/mo); static config → Parameter Store ($0).

What it costs

$0.40 per secret per month plus $0.05 per 10,000 API calls. With sane caching the API meter rounds to zero; without caching, a 3,000 RPS service spends ~$39/day asking the same question.

⚓ On the SysPad node, the "Client cache hit rate" slider is the lesson: drag it from 99% to 0% under load and watch both cost and utilization spike.

Flashcards

Code review: a handler calls GetSecretValue for the DB password at the top of every request, "to always have the freshest value." Your comment?: Secrets change monthly, not per request: cache in-process with the AWS caching client (1h TTL default). Per-request fetches add ~25ms, burn the 10k/s shared quota, and cost real money at scale.
The night after enabling rotation, half your fleet starts failing DB auth. What is the likely mechanic?: Cached clients still hold the pre-rotation password and the old credential was revoked immediately. Fix: the alternating-users rotation strategy (both valid during overlap) or a cache TTL shorter than the revocation window.
You need to store: (a) a rotating RDS password, (b) a static feature-flag string. Where does each go?: (a) Secrets Manager: managed rotation earns the $0.40/mo. (b) Parameter Store standard tier: free, versioned, IAM-gated. Paying Secrets Manager prices for plain config is a common small waste.

Sources

Open the SysPad canvas · Official Secrets Manager documentation