SysPad › Lessons › monitoring lessons › Datadog
How Datadog works
SaaS observability (metrics, APM, logs). Never a throughput bottleneck - but it bills per host AND per GB AND per indexed event simultaneously, which is how observability ends up costing more than the infra it watches.
There is a running joke in infrastructure circles that the most expensive part of a system is watching it. Datadog is the best-in-class observability suite AND the punchline of that joke, because it bills on several independent meters at once: hosts, traces, log volume, log indexing, custom metrics. Each one reasonable. The sum, famous.
What you will learn
- What do metrics, APM traces, and logs each answer?
- Which meters make up a Datadog bill, and which one explodes?
- What is the index-vs-archive decision for logs?
- How do you keep observability under ~10% of the infra it observes?
One agent, three signals
A Datadog agent on each host ships three signals: metrics (is it healthy?), APM traces (where did this request spend its 900ms, across services?), and logs (what exactly happened?). The product's magic is correlation: click from a slow trace straight to its logs.
The multi-axis bill
Per-host infrastructure (~$15-23/host/mo), per-host APM (~$31-40), log ingestion (~$0.10/GB), log indexing (~$1.70 per million events kept searchable), and custom metrics each bill separately. Autoscaling that doubles your hosts doubles two meters without anyone deploying anything.
Ingest everything, index little
Datadog splits logs into ingest (receive + archive, cheap) and index (searchable, 15x the story). Mature setups ingest 100% but index 10-25%: errors, slow requests, sampled success traffic, with exclusion filters dropping health checks and debug noise. Archives can be rehydrated when an investigation needs the rest.
⚓ On the SysPad node, the "Logs indexed" slider against log volume is exactly this decision, and it dwarfs the host sliders at scale.
What are custom metrics and why do they ambush bills?
Any metric you emit with tags becomes one BILLABLE series per unique tag combination. Tag a metric with user_id and you have created a metric per user: cardinality explosion, priced per hundred series. The rule: tag with bounded sets (service, region, status), never unbounded ones (user, request id).
Flashcards
- Nobody changed observability config, but the Datadog bill rose 60% this month. The platform team did enable aggressive autoscaling. Connect the dots.
- Per-host meters (infrastructure + APM) scale with instance count: more autoscaled hosts, more billed hosts. Watching the watcher's per-host axes is part of capacity planning.
- Log costs dominate the bill; 40% of indexed events turn out to be load-balancer health checks. What is the two-minute fix?
- An exclusion filter: keep ingesting (cheap, archived) but stop INDEXING health checks. Indexing is the expensive meter and filters apply instantly.
- A developer adds `user_id` as a tag on a request-count metric "for debugging." Why does billing notice within days?
- Each unique tag value creates a billable custom-metric series: 200k users = 200k series. Bounded tags only; per-user questions belong in traces/logs.
Sources
- Datadog pricing (all products)
- Log management: ingest vs indexing (Logging without Limits)
- Custom metrics and cardinality billing
Open the SysPad canvas · Official Datadog documentation