SysPad › Examples › Metrics monitoring & alerting
Metrics monitoring & alerting
Agents push metrics through a collection stream into a time-series DB; a Query Service feeds dashboards while an Alert Manager evaluates rules against the same store and pages out.
One TSDB serving both a dashboard read path and an independent alert-evaluation path.
- Throughput: 5,000 req/sec
- First to saturate: Alert Topic
- Estimated cost: $69k/mo
How it works
- Metric Agents
- Every host and service runs a collection agent that pushes metrics on a fixed interval - this is a push pipeline, so the agents are the front door, not a client behind a load balancer.
- Collection Stream
- A durable buffer between a huge fan-in of agents and the aggregator, so a burst (a deploy, a fleet-wide GC pause) is absorbed as stream backlog instead of dropped datapoints.
- TSDB
- The superposition centre: the ingest flow writes it, the dashboard flow scans it for charts, and the alert flow scans it to evaluate rules - three journeys stacking load on one store.
- Alert Manager
- Re-evaluates every alert rule against the same TSDB on a short cycle, independent of the dashboard path - most evaluations find nothing, so only a small fraction go on to page out.
Request flows
Collect metrics
- Metric Agents
- Collection Stream
- Aggregator
- TSDB
View a dashboard
- TSDB
- Query Service
- Dashboards
Evaluate alert rules
- TSDB
- Alert Manager
Fire an alert
- Alert Manager
- Alert Topic
- PagerDuty