SysPad › Lessons › messaging lessons › Redis Streams
How Redis Streams works
Append-only log with consumer groups (XADD / XREADGROUP) - a lightweight, in-memory Kafka-style stream inside Redis. Sub-millisecond latency; throughput is bounded by the node's single-threaded ops/sec capacity.
You don’t always need a whole Kafka cluster to get a stream. Redis has a built-in stream data type: an append-only log with consumer groups, living right inside the Redis you may already run. It’s the lightweight, in-memory way to do streaming, fast and simple, as long as you respect what "in Redis" means for durability.
What you will learn
- How is a Redis Stream different from a Redis list or pub/sub?
- How do consumer groups split work and make sure nothing is lost on a crash?
- Why does "it lives in Redis" decide whether you can trust it with data?
- When is a Redis Stream enough, and when do you need Kafka/Kinesis?
An append-only log inside Redis
A Redis Stream is a log data structure. XADD appends an entry and returns a time-ordered ID; XREAD reads entries from any ID onward. Unlike Redis pub/sub (fire-and-forget, no history), stream entries persist in the structure and can be re-read.
Consumer groups: shared work with acknowledgements
With consumer groups (XREADGROUP), members of a group split the entries so each is processed once by the group. A delivered entry enters that consumer’s pending list until it’s acknowledged with XACK.
If a worker dies mid-processing, its un-acked entries stay pending and can be reclaimed by another worker, so work isn’t silently lost. It’s the same at-least-once idea you saw with SQS, expressed in Redis commands.
⚓ Task queues, lightweight event processing, and real-time pipelines where Redis is already in the stack.
It lives in Redis (so durability depends on the engine)
A Redis Stream is a Redis data structure, so its durability is the engine’s durability. On a cache-style ElastiCache deployment, recent entries can be lost on failover or restart. On MemoryDB (durable, multi-AZ transaction log), the stream is persisted safely.
Redis Streams vs Kafka/Kinesis
Reach for a Redis Stream when you want simple, very low-latency streaming and Redis is already in your stack, without standing up a streaming platform. Reach for Kafka (MSK) or Kinesis when you need very high sustained throughput, long retention, large-scale partitioning, an ecosystem of connectors, or strong durability guarantees out of the box.
Flashcards
- How does a Redis Stream differ from Redis pub/sub?
- Pub/sub is fire-and-forget with no history; a stream is an append-only log whose entries persist and can be re-read/replayed by ID.
- How does a consumer group ensure a crashed worker’s entries aren’t lost?
- Delivered entries sit in a pending list until
XACK. Unacked entries can be reclaimed by another consumer, at-least-once processing. - Same Redis Stream, run on ElastiCache vs MemoryDB, what differs?
- Durability. ElastiCache can lose recent entries on failover; MemoryDB persists them via its multi-AZ transaction log.
- Why isn’t a Redis Stream an unbounded archive?
- It lives in memory and is trimmed (MAXLEN or by age) to bound size, so old entries are dropped, retention is limited.
- When is Kafka/Kinesis the better choice over a Redis Stream?
- For very high sustained throughput, long retention, large-scale partitioning, connectors, or built-in strong durability, Redis Streams are lighter but bounded by memory/engine.
Sources
- Redis, Streams introduction (XADD, XREAD, IDs)
- Redis, Consumer groups (XREADGROUP, pending entries, XACK)
- AWS, MemoryDB (durable Redis-compatible engine)
- AWS, ElastiCache data durability considerations
Open the SysPad canvas · Official Redis Streams documentation