SysPad › Examples › Distributed message queue
Distributed message queue
Producers write to a partitioned, replicated broker cluster (MSK). Two consumer groups each drain the full log independently, one into durable storage and one into real-time analytics.
The headline message-queue property: one durable log written once and read in full by every independent consumer group, so the broker’s read load is a multiple of its write load.
- Throughput: 20,000 req/sec
- First to saturate: Real-time Analytics
- Estimated cost: $71k/mo
How it works
- Broker Cluster
- The partitioned, replicated log and the superposition centre: producers write it once, and every consumer group reads the whole stream at its own tracked offset. One write flow plus two read flows stack here, which is why its ops rate is roughly triple the publish rate.
- Order Consumer
- One consumer group, scaled to the partition count - each task owns a slice of the partitions and commits offsets so a restart resumes where it left off, not from zero.
- Analytics Consumer
- A second, independent consumer group reading the SAME partitions at its own offset. It gets the full stream again without touching the order group’s throughput - add it, or remove it, and the producers never change.
Request flows
Publish a message
- Producer Service
- Broker Cluster
Drain (order group)
- Broker Cluster
- Order Consumer
- Processed Data
Drain (analytics group)
- Broker Cluster
- Analytics Consumer
- Real-time Analytics