SysPad › Lessons › messaging lessons › Kinesis Data Firehose

How Kinesis Data Firehose works

Serverless delivery stream that buffers records and loads them into S3, Redshift, or OpenSearch. Auto-scales to ingest throughput; latency is governed by the buffer size/interval, not compute.

Firehose answers a narrow but constant need: "I have a stream of data, just reliably dump it into S3 (or Redshift, or OpenSearch) for me." No shards, no consumers to write, no scaling knobs. You point it at a destination and it batches and delivers, the managed "last mile" of a data pipeline.

What you will learn

What problem does Firehose solve that Kinesis Data Streams makes you do yourself?
Why does Firehose add a little latency, and how is that controlled?
What can it do to your data in transit (transform, convert, compress)?
When do you need Data Streams instead of Firehose?

A managed delivery stream

Firehose (now branded Amazon Data Firehose, formerly Kinesis Data Firehose) is a delivery stream: you send it records, it loads them into a destination, S3, Redshift, OpenSearch, Splunk, or an HTTP endpoint. There are no shards to provision and no consumer code to maintain; it scales automatically.

Buffering: near real-time, by design

Firehose buffers records and flushes when it hits a size threshold (1 to 128 MB) or a time limit (0 to 900 seconds), whichever comes first.

Transforming on the way through

Firehose can process records in flight: invoke a Lambda transform to clean or enrich them, convert formats (e.g. JSON to Parquet/ORC for analytics), and compress or encrypt before delivery.

⚓ The standard way to land logs/events in an S3 data lake or an OpenSearch index without writing a delivery service.

What happens to records that fail to deliver or transform?

Firehose retries, and records that still can’t be delivered (or that a transform marks as failed) are written to an error/backup S3 location you configure, so nothing is silently dropped and you can reprocess later.

Firehose vs Data Streams

Use Firehose when the goal is simply to load streaming data into a store with minimal effort. Use Data Streams when you need custom real-time processing, multiple independent consumers, sub-second latency, strict per-shard ordering, or replay of historical records.

Flashcards

What does Firehose manage that Kinesis Data Streams makes you handle?: Shard sizing, consumer code, and scaling. Firehose is a fully managed delivery stream, you just pick a destination.
What controls how quickly Firehose delivers records?: Its buffering hints: size (1 to 128 MB) and time (0 to 900s), flushing on whichever hits first. Low interval = near real-time; high = fewer, larger, cheaper writes.
How do you land incoming JSON as compressed Parquet in S3 with Firehose?: Enable format conversion (JSON→Parquet) and compression in the delivery stream; optionally a Lambda transform to clean records first.
Name typical Firehose destinations.: S3, Redshift, OpenSearch, Splunk, and generic HTTP endpoints.
You need replay and multiple independent consumers. Firehose or Data Streams?: Data Streams, Firehose has no replayable log or arbitrary consumers; it only delivers to a destination.

Sources

Open the SysPad canvas · Official Kinesis Data Firehose documentation