SysPad › Lessons › messaging lessons › Amazon MSK

How Amazon MSK works

Fully managed Apache Kafka. Standard brokers are sized on AWS's Equation 1 (storage, EBS and network baselines; each consumer group re-reads the stream), Express brokers on their throttle table. Retention fills the broker disks and is billed.

Apache Kafka is the de facto standard for event streaming at scale, but running a Kafka cluster (brokers, ZooKeeper/KRaft, rebalancing, patching) is a job in itself. MSK runs it for you while keeping real Kafka, so your existing Kafka apps, tools, and connectors work unchanged. The mental model is a stream, but with Kafka’s vocabulary.

What you will learn

Topics, partitions, and offsets

Producers write records to a topic. Each topic is split into partitions, append-only logs. A record’s key chooses its partition, and records are strictly ordered within a partition (not across them).

Every record has a position called an offset. Consumers remember their offset rather than deleting records, so reading is just "advance my offset."

Consumer groups: share work, or read independently

Consumers join a consumer group. Kafka assigns each partition to one member of the group, so the group shares the load and scales by adding members (up to the partition count).

A different group reading the same topic gets its own offsets and its own full copy of the data stream, so billing, analytics, and search can all consume the same topic without taking messages from each other.

That copy is not free. The brokers send the whole stream once to every group, on top of the copies they send to follower replicas. AWS sizes broker network for exactly this: each message leaves a leader (groups + replication factor - 1) times. Adding a group can saturate the brokers and slow every group and every producer.

⚓ High-volume event backbones: log/metrics pipelines, change-data-capture, stream processing with Kafka Streams/Flink.

Retention and replay

Kafka keeps records for a configured retention (by time or size), regardless of whether they’ve been consumed. Because consumers track offsets and data persists, a consumer can rewind and reprocess history, after a bug fix, or to feed a brand-new consumer from the beginning.

On Standard brokers every replica keeps the whole retention on its broker disk. A week of 20,000 two-KB messages a second at three copies is about 69,000 GiB, more than four brokers can hold at 16,384 GiB each. Tiered storage keeps one remote copy of older data instead; Express brokers have no disk limit and bill per GB stored.

MSK vs Kinesis vs SQS/SNS

Pick MSK when you specifically want Apache Kafka, existing Kafka code/connectors, the Kafka ecosystem, or multi-cloud portability. Pick Kinesis for a similar streaming model that’s fully AWS-managed with less operational surface. Pick SQS/SNS for simple queueing or fan-out without a replayable log.

Flashcards

Where does Kafka guarantee ordering, and what decides a record’s partition?
Order is guaranteed within a partition; the record key (hashed) chooses the partition.
What is an offset, and why does it let consumers replay?
An offset is a consumer’s position in a partition. Records persist, so rewinding the offset re-reads past data.
How do you scale consumption within one group vs read the same topic in two ways?
Add members to a consumer group (partitions split across them). Use a separate group to read the whole topic independently with its own offsets.
A consumer group has 10 members but a topic has 4 partitions. How many actively consume?
At most 4, one partition per member, so 6 members sit idle. Parallelism is capped by the partition count.
Why choose MSK over Kinesis?
You want real Apache Kafka: existing Kafka apps/connectors, the Kafka ecosystem, or portability. Kinesis is the more hands-off AWS-native alternative.

Sources

Open the SysPad canvas · Official Amazon MSK documentation

Other messaging lessons