SysPad › Lessons › messaging lessons › Amazon MSK

How Amazon MSK works

Fully managed Apache Kafka. Producers write to partitioned topics replicated across brokers. Throughput scales with broker count and instance size; replication factor divides usable write capacity.

Apache Kafka is the de facto standard for event streaming at scale, but running a Kafka cluster (brokers, ZooKeeper/KRaft, rebalancing, patching) is a job in itself. MSK runs it for you while keeping real Kafka, so your existing Kafka apps, tools, and connectors work unchanged. The mental model is a stream, but with Kafka’s vocabulary.

What you will learn

What are topics, partitions, and offsets, and why is order only per-partition?
How do consumer groups share work, and how do separate groups read the same data?
Why can Kafka consumers re-read (replay) while a queue can’t?
When do you pick MSK (Kafka) over Kinesis or SQS/SNS?

Topics, partitions, and offsets

Producers write records to a topic. Each topic is split into partitions, append-only logs. A record’s key chooses its partition, and records are strictly ordered within a partition (not across them).

Every record has a position called an offset. Consumers remember their offset rather than deleting records, so reading is just "advance my offset."

Consumer groups: share work, or read independently

Consumers join a consumer group. Kafka assigns each partition to one member of the group, so the group shares the load and scales by adding members (up to the partition count).

A different group reading the same topic gets its own offsets and its own full copy of the data stream, so billing, analytics, and search can all consume the same topic without stepping on each other.

⚓ High-volume event backbones: log/metrics pipelines, change-data-capture, stream processing with Kafka Streams/Flink.

Retention and replay

Kafka keeps records for a configured retention (by time or size), regardless of whether they’ve been consumed. Because consumers track offsets and data persists, a consumer can rewind and reprocess history, after a bug fix, or to feed a brand-new consumer from the beginning.

MSK vs Kinesis vs SQS/SNS

Pick MSK when you specifically want Apache Kafka, existing Kafka code/connectors, the Kafka ecosystem, or multi-cloud portability. Pick Kinesis for a similar streaming model that’s fully AWS-managed with less operational surface. Pick SQS/SNS for simple queueing or fan-out without a replayable log.

Flashcards

Where does Kafka guarantee ordering, and what decides a record’s partition?: Order is guaranteed within a partition; the record key (hashed) chooses the partition.
What is an offset, and why does it let consumers replay?: An offset is a consumer’s position in a partition. Records persist, so rewinding the offset re-reads past data.
How do you scale consumption within one group vs read the same topic in two ways?: Add members to a consumer group (partitions split across them). Use a separate group to read the whole topic independently with its own offsets.
A consumer group has 10 members but a topic has 4 partitions. How many actively consume?: At most 4, one partition per member, so 6 members sit idle. Parallelism is capped by the partition count.
Why choose MSK over Kinesis?: You want real Apache Kafka: existing Kafka apps/connectors, the Kafka ecosystem, or portability. Kinesis is the more hands-off AWS-native alternative.

Sources

Open the SysPad canvas · Official Amazon MSK documentation