SysPad › Lessons › monitoring lessons › Amazon CloudWatch
How Amazon CloudWatch works
Metrics, logs, alarms and dashboards. Never a throughput bottleneck - but log ingestion ($0.50/GB) is the classic AWS bill shock: a chatty service can spend more on logging than on compute. Watch bytes-per-request.
There is a famous genre of AWS billing horror story, and it is not EC2 or a forgotten database. It is logging. CloudWatch charges for every gigabyte your services write, and a chatty app at scale can spend more on describing its work than on doing it.
What you will learn
- What are the three things CloudWatch actually does?
- Why does log INGESTION, not storage, dominate the bill?
- How do bytes-per-request and RPS multiply into bill shock?
- What are the standard moves when logging costs explode?
Metrics, logs, alarms
CloudWatch is three tools sharing a name: metrics (numbers over time: CPU, latency, queue depth), logs (text your apps emit), and alarms (thresholds on metrics that page someone or trigger autoscaling).
It never slows a request down
CloudWatch sits OFF the request path: agents batch and ship logs asynchronously, so it adds no user-facing latency and can never be your throughput bottleneck. In a SysPad simulation a CloudWatch node shows zero latency and zero utilization by design.
Its entire simulation story is the cost line. Which brings us to the fun part.
The bill-shock multiplication
Ingestion costs $0.50 per GB. The bill is therefore bytes-per-request x RPS x time, and it scales with your success: 5KB of logs at 1,000 RPS is roughly 13 TB and $6,500 a month.
The standard cost moves
In order of impact: log LESS per request (drop debug logs in prod, sample verbose paths), set retention on every log group (never-expire is the default nobody means to choose), and route bulk logs to S3 via Firehose for about a tenth of the price, querying them with Athena when needed.
⚓ On a SysPad CloudWatch node, the "Log volume per request" slider is the whole story: nudge it from 2KB to 10KB under real traffic and watch the monthly cost line.
Why not just log everything and filter later?
Because you pay at the door, not at the search. Ingestion is billed per GB whether anyone ever reads the line. "Log everything to CloudWatch, grep later" is the exact pattern S3 + Athena exists to replace: cheap to store, pay only when you query.
Flashcards
- Finance flags a $9k/month CloudWatch line. Your teammate starts tightening log RETENTION from 90 to 30 days. Will that fix it?
- Barely: storage is $0.03/GB-mo while ingestion is $0.50/GB, so almost all of the $9k is the write path. Cut bytes-per-request (sampling, less debug logging) or reroute bulk logs to S3.
- During an incident someone suggests turning off CloudWatch "to reduce request latency." Sensible?
- No: log shipping is asynchronous and off the request path, so it adds no user-facing latency. Turning off observability during an incident only makes you blind.
- A startup logs 8KB per request "for debuggability" at 50 RPS and pays almost nothing. What changes when they hit 5,000 RPS?
- The bill scales linearly with traffic: ~100x the RPS is ~100x the ingestion, roughly $5,000+/month for logs alone. Bytes-per-request is a decision you re-visit at every scale jump.
Sources
Open the SysPad canvas · Official Amazon CloudWatch documentation