SysPad › Examples › Industrial IoT + analytics
Industrial IoT + analytics
Device telemetry through IoT Core’s Rules Engine, dual-written to a hot path (MSK → Lambda → Timestream → CloudWatch) and a cold path (Firehose → S3 lake → Glue + EMR → Redshift → QuickSight).
IoT Core’s account-wide publish quota near saturation, and one ingested message feeding both a real-time and a historical analytics path.
- Load: 16,000 req/sec
- First to saturate: Delivery Stream
- Estimated cost: $386k/mo
How it works
- IoT Core
- Fifty thousand devices publishing into one account-wide quota. This is deliberately run close to that ceiling, because that ceiling, not compute, is what stops a fleet from growing.
- Telemetry Stream
- The hot path. Kafka keeps the messages from each device in order and can replay them. The Stream Processor is its one consumer group. The cold path reads IoT Core directly. The 6 brokers are a multiple of the 3 zones. At 168 h retention the disks hold 55,371 of 98,304 GiB (56.33%). Storage costs $5,537.11 a month against $1,814.40 for the brokers.
- Live Metrics
- Recent readings stay in memory for twelve hours and then age down to cheap magnetic storage automatically. That tiering is the reason to use a time-series store instead of a general database.
- Delivery Stream
- The cold path gets the same messages, buffered for a minute before landing. Nobody is waiting on this, so batching wins over latency here.
- Telemetry Lake
- Full-fidelity history, kept because the question you will want to ask in a year has not been thought of yet.
Lessons for this architecture
- How Amazon MSK works
- How AWS Lambda works
- How Timestream works
- How Amazon CloudWatch works
- How Kinesis Data Firehose works
- How S3 works
- How AWS Glue works
- How Amazon EMR works
- How Redshift works
- How Amazon QuickSight works