SysPad › Examples › Distributed message queue

Distributed message queue

Producers write to a partitioned, replicated broker cluster (MSK). Two consumer groups each drain the full log independently, one into durable storage and one into real-time analytics.

The headline message-queue property: one durable log written once and read in full by every independent consumer group, so the broker’s read load is a multiple of its write load.

Load: 20,000 req/sec
First to saturate: Real-time Analytics
Estimated cost: $51k/mo

How it works

Broker Cluster: The partitioned, replicated log. Producers write each message once, and every consumer group reads the whole stream at its own offset. In flow mode the broker load is the publish rate of 20,000 messages a second. The drain flows do not add to it. Each message leaves a leader 4 times (2 groups + RF 3 - 1), so the network is at 8.53%. The disk is the tighter limit: 168 h x RF 3 is 69,214 GiB on 6 x 16,384 GiB (70.41%). Storage costs $6,921.39 a month against $3,628.80 for the brokers.
Order Consumer: One consumer group, scaled to the partition count - each task owns a slice of the partitions and commits offsets so a restart resumes where it left off, not from zero.
Analytics Consumer: A second consumer group that reads the same partitions at its own offset. Its offsets are independent, and the producers never change. But each message is a full extra read on the network of the leaders. This group cuts the network ceiling of the brokers from 312,500 to 234,375 messages a second (6 x m5.2xlarge, 2 KB).

Request flows

Publish a message

Producer Service
Broker Cluster

Drain (order group)

Broker Cluster
Order Consumer
Processed Data

Drain (analytics group)

Broker Cluster
Analytics Consumer
Real-time Analytics

Lessons for this architecture

Worked step by step

The same interview question, built up one twist at a time, with a simulation at each stage.

Design a distributed message queue

Other interview practice

Open this architecture in SysPad · All examples