SysPad › Lessons › messaging lessons › Kinesis Data Streams

How Kinesis Data Streams works

Real-time data streaming for logs, events, and clickstreams. Each shard handles 1,000 records/sec write (1 MB/s) and 2 MB/s read shared across standard consumers - enhanced fan-out gives each consumer its own pipe.

A queue forgets a message the moment it’s read. A stream remembers. Kinesis Data Streams is an append-only log of records that sticks around for hours or days, so many consumers can each read it independently, and you can rewind and replay the past. That memory is the whole difference between a queue and a stream.

What you will learn

How is a "stream" different from a "queue", and why does that matter?
What are shards, and why are records only ordered within one?
How do multiple consumers read the same stream without stepping on each other?
When should you reach for a stream instead of SQS?

A stream remembers; a queue forgets

SQS is consume-and-delete: once a worker handles a message, it’s gone. A stream is an append-only log: records stay for a retention window(24 hours by default, extendable to 7 days at base price and up to 365 days as paid long-term retention), and reading them doesn’t remove them.

That persistence enables two things a queue can’t: multiple independent consumers, and replay from an earlier point in time.

Shards and per-shard ordering

A stream is divided into shards. Each record carries a partition key that’s hashed to pick its shard. Records are strictly ordered within a shard, but not across shards.

Each shard has a fixed throughput budget (about 1 MB/sec or 1,000 records/sec in, ~2 MB/sec out). You scale by adding shards, so pick a partition key with good spread, or one hot key overloads a single shard.

Many consumers, each at their own position

Because records persist, several consumers can read the same stream independently, each tracking its own position. One can do real-time analytics while another archives, from the same data.

With enhanced fan-out, each registered consumer gets its own ~2 MB/sec per shard read pipe (instead of sharing the shard’s read budget), for low-latency parallel consumption.

⚓ Clickstreams, IoT telemetry, log/metric pipelines, and real-time analytics.

Stream or queue?

Choose a stream (Kinesis) when you need ordered records, multiple independent consumers, replay, or high-volume real-time analytics. Choose a queue (SQS) for simple decoupled task processing where each message is handled once and then discarded.

Flashcards

What happens to a Kinesis record after a consumer reads it?: It stays in the stream for the retention window (24h default, up to 365 days). Reading doesn’t delete it, enabling multiple consumers and replay.
Across a 4-shard Kinesis stream, is global ordering guaranteed?: No, ordering is guaranteed only within a shard. Records sharing a partition key land on the same shard and stay ordered.
Throughput is maxed and one shard is far hotter than the others. Likely cause?: A low-cardinality partition key sending most records to one shard. Each shard caps at ~1 MB/s or 1,000 rec/s in; spread keys and add shards.
How do analytics and archival both process the same stream?: As independent consumers, each tracking its own position. Enhanced fan-out gives each its own per-shard read throughput.
One-line: when a stream over a queue?: When you need ordering, multiple consumers, or replay. For handle-once-then-discard tasks, a queue (SQS) is simpler.

Sources

Open the SysPad canvas · Official Kinesis Data Streams documentation