Design a metrics monitoring and alerting system

Do the numbers before the boxes

An interviewer gives a scale, not a load. If we draw before we do the math, we cannot say why any box has its size.

We assume a large company: 5,000 Kubernetes nodes running 100,000 pods.

We assume about 1,000 series a node and 100 a pod. 5,000 x 1,000 + 100,000 x 100 = 15 million active series, the ones still receiving samples.

Each is scraped every 15 seconds: 15,000,000 / 15 = 1 million samples a second.

There is no peak factor for ingest. Scrapes run on a timer, not with user traffic.

Agents send samples by remote write, in batches of 2,000: 1,000,000 / 2,000 = 500 write requests a second.

Dashboards: 225 engineers with 20 panels each, refreshed every 30 seconds. 225 x 20 / 30 = 150 queries a second.

Rules: 9,000 alert and recording rules every 60 seconds, so 9,000 / 60 = 150 evaluations a second. 30 teams share the system.

The simulation has one entry, in write requests a second. Queries and rule evaluations ride on it at 0.3 each, so a load button scales all three.

Check in: Close the page and derive 500 write requests a second from 5,000 nodes and 100,000 pods. Then derive the 150 queries a second.

One sample through every state

Before we size anything, we follow one sample from the app that made it to the file that keeps it for 13 months. Time is not in the simulation, so this is by hand.

Collected: an agent on each node scrapes its own node and pods every 15 seconds. That is pull, close to the source.

Sent: the agent pushes batches of 2,000 samples out to the store. Pull is local and push is remote.

Buffered: if the store is unreachable, the agent keeps samples in its write-ahead log (WAL) for about 2 hours, then replays them in order.

Ingested: a distributor, a stateless front door, checks the team's rate and label limits and hashes the series. It writes the sample to 3 ingesters, which hold recent data.

The distributor answers once 2 of the 3 have it, a quorum.

Admitted: each ingester checks the team's series limit before it creates a new series in memory.

Stored in memory: each ingester keeps the newest samples in memory, called the head, in chunks of up to 120 samples. Its WAL is on disk.

After 2 hours the head is cut into a block and uploaded to S3. A compactor then merges the 3 copies into 1, and merges blocks.

Queried: the last 13 hours come from ingesters, and older data from blocks in S3. Expired: blocks past 13 months are deleted whole.

Step through the figure to follow one sample.

The trade-off:

Three copies triple the memory and the writes. In exchange, losing one ingester loses no sample.

One sample, from the agent to S3

1. An agent on each node scrapes every 15 seconds. Its write-ahead log holds about 2 hours if the store is unreachable.
2. The agent pushes samples to a distributor by remote write, 2,000 samples in each request.
3. The distributor writes each sample to 3 ingesters. It answers once 2 of the 3 have it.
4. Each ingester checks the team's series limit, then keeps the sample in memory, in its head, and in a write-ahead log on disk.
5. Every 2 hours, each ingester cuts its head into a block and uploads it to S3. S3 now holds 3 copies.
6. The compactor merges the 3 copies into 1. It then merges 2-hour blocks into 12-hour and 24-hour blocks.
7. A query reads the last 13 hours from the ingesters and older data from S3. Blocks are deleted after 13 months.

Check in: Follow one sample from the app to S3. Say where it lives at each step, and why the distributor answers after 2 of 3 ingesters.

Size the stateless fleets from published ratios

Grafana Mimir publishes how much each part handles per core. We run its parts as Fargate fleets, and size each one from those ratios.

We do not use Amazon Timestream for LiveAnalytics here: it stopped taking new customers on 2025-06-20.

Distributors take 25,000 samples a second per core: 0.04 ms a sample. A request of 2,000 samples is 2,000 x 0.04 = 80 ms of CPU.

A distributor task has one vCPU, a virtual processor, so it takes 1,000 / 80 = 12.5 requests a second.

With auto-scaling at a 70% target, 500 / (12.5 x 0.7) = 57.1, so 58 tasks run, at 69%.

Queriers take 10 queries a second per core at about 100 ms: 150 / (10 x 0.7) = 21.4, so 22 tasks, at 68%.

The ruler evaluates rules the same way, with its own 22 tasks. It reads recent data straight from the ingesters, not from S3.

The task caps are cost guards we chose, not AWS limits. 100 distributors take 1,250 requests a second, 2.5 times today.

40 queriers take 400 queries a second, 2.7 times today, and so do 40 ruler tasks.

The trade-off:

A cap is a budget. It stops a runaway bill, and in exchange the fleet stops growing at the cap.

Check in: Size the distributors for 500 write requests a second, at a 70% target.

Ingesters are a planned ring

Ingesters hold state, so we cannot add and remove them with load the way we do distributors.

Ingesters hold 300,000 series per core. At one sample per 15 seconds that is 300,000 / 15 = 20,000 sample-writes a second, 0.05 ms each.

Each sample goes to 3 ingesters, so a request becomes 2,000 x 3 = 6,000 sample-writes, and 500 requests become 3 million a second.

An ingester task has 4 vCPUs: 4 x 20,000 = 80,000 sample-writes a second.

Ingesters form a ring: each series hashes to a place on it. A new ingester takes only new series, so moving load takes hours.

So the ring is planned, not auto-scaled: 60 tasks, 20 in each of 3 zones. 3,000,000 / 4,800,000 = 62.5% of the ring's CPU.

Mimir's minimum would be 45 million series / 300,000 a core = 150 cores, 37.5 tasks at 100%. 60 tasks leave 28% to grow before 80%.

Memory, by hand: Mimir gives 2.5 GB per 300,000 series, so 2.5 x 2^30 / 300,000 = 8,948 bytes a series.

15 million x 3 = 45 million series in memory. 45 million / 60 = 750,000 a task x 8,948 bytes = 6.25 GiB of a task's 16.

Mimir asks for 50% extra memory to absorb peaks, so 9.4 GiB. In production, ingesters run as a StatefulSet with a disk each; Fargate stands in for their CPU and price.

The trade-off:

Ingesters hold every series 3 times in memory, so they are most of the bill. A planned ring also runs ahead of its load.

Check in: Why is the ring 60 tasks, not auto-scaled, and how much memory does each task use?

Run the first design

Stage 1 runs this design at 500 write requests a second. Each card is a part of the system, and each arrow names its own unit.

Read the four tiles. "First to saturate" reads "Nothing at this load"; its note names the Distributors, at 69%.

A part saturates when it gets more requests than it can serve. A card turns red at 90% of its limit; only past 100% does the result name it.

The latency tile shows 100ms. It shows the slowest path, which here is a dashboard query.

The write path adds to 1.5 + 80 + 0.05 = 81.55 ms. That is a CPU budget, not a measured time.

Its parts are the load balancer, a distributor's work on one request, and the ingester card's cost of one sample-write.

The cost tile shows $17k/mo: $16,619.58 / 15 = $1,107.97 per million active series a month. Ingesters are 61% of it.

Data sent out to the internet, called egress, is $1,324.39 of it. AWS charges it by the GB, after the first 100 GB a month.

Almost all of it is dashboard answers: 150 a second, each about 40 KiB of JSON, our estimate. Writes come in free and get back an empty reply.

Ingest and rules run flat, so the tile is nearly the real month. Only dashboards are quieter off hours, so fewer answers go out: an average month is about $15,369.

The ingesters run out first. The ring of 60 tasks x 80,000 = 4.8 million sample-writes a second, which is 4,800,000 / 6,000 = 800 write requests.

Press 850. The Ingesters saturate: 850 write requests x 6,000 = 5.1 million sample-writes, against 4.8 million.

Press 1,300. The Distributors saturate too, 1,300 requests against 100 tasks x 12.5 = 1,250. The Queriers and Ruler cards turn red, at 97.5%.

Both take 0.3 x 1,300 = 390 of their 400, so the result names neither. Dashboard queries queue in the busy Queriers: the latency tile reads 375ms.

The trade-off:

Every series costs memory 3 times over. The bill follows series, not traffic.

Stage 1: 15 million series, 1 million samples a second

• Load: 500 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $17k/mo

Check in: Which part runs out first, and at what load? Give it in write requests and in sample-writes.

How many bytes 13 months take

Retention sets the disk. We need the bytes a sample takes, how many times each byte is written, and where it sits as it ages.

Compressed, a sample takes about 1.37 bytes in Gorilla and 1 to 2 bytes in Prometheus. We use 1.5 bytes.

A sample's bytes are written about 12 times: 3 WALs, 3 head chunk files, 3 uploaded blocks, then 3 rewrites by the compactor. That is our estimate.

Disk = samples a second x seconds kept x bytes a sample. Our sizes are binary: 1 GiB is 2^30 bytes, and 1 TiB is 2^40.

A 30-day month: 1,000,000 x 2,592,000 x 1.5 bytes = 3.54 TiB. 13 months: 46.0 TiB, one copy after compaction.

The last 2 hours sit in ingester memory: 45 million series x 8,948 bytes = 375 GiB across the ring.

Up to 13 hours, the WAL and recent blocks sit on ingester disks. Mimir plans 5 GB per 300,000 series: 45 million / 300,000 x 5 GB = 750 GB.

From 13 hours to 13 months, compacted blocks live in S3 and are read from there.

Mimir does not downsample. It keeps raw samples, and makes long queries fast by splitting and caching them.

Thanos does downsample, but it adds the coarser copies beside the raw ones. It saves space only with a shorter raw retention, such as raw for 30 days.

The trade-off:

S3 is the cheap part: 46 TiB is about $1,084 a month. Memory for recent data is the expensive part.

Check in: Derive 13 months of disk from 1 million samples a second.

One label, three times the series

The interviewer asks: A team adds user_id to a request counter. What breaks?

A series is one combination of label values, so series multiply. Take http_requests_total with service, endpoint, status and pod.

50 services x 20 endpoints x 5 statuses x 20 pods = 100,000 series. That count of distinct series is cardinality.

Add user_id with 10,000 active users: 100,000 x 10,000 = 1 billion series. At 3 copies of 8,948 bytes, that is about 24 TiB of memory.

That is the worst case, with every user on every combination.

In our stage each of the counter's 100,000 series sees about 300 users in the 2 hours it stays in memory. 100,000 x 300 = 30 million series.

Active series go from 15 to about 45 million. Samples triple to 3 million a second, so write requests triple to 1,500.

Dashboards and rules stay at 150 each, now a share of 0.1.

Press 500 on Stage 2. The whole stage runs at a third: Stage 1's 15 million series, with dashboards and rules at 50 a second.

Without limits, the system is healthy at that series count.

Press 1,500. The Distributors and the Ingesters saturate. The distributors pass on 1,250 requests: 1,250 x 6,000 = 7.5 million sample-writes, against 4.8 million.

The latency tile now shows "≥ 322ms", a lower limit. The write path, through saturated distributors, is now the slowest path.

Memory is worse, and we work it by hand. 45 million x 3 = 135 million series in memory x 8,948 bytes = 1.10 TiB.

The ingesters have 60 x 16 GiB = 0.94 TiB. They run out of memory and crash, which is worse than slow: every team loses ingest.

The cost tile shows $19k/mo. Step through the figure to watch one label multiply the series.

The trade-off:

One label multiplied one counter by 300 and the whole system by 3. Cardinality, not traffic, is the bill and the risk.

One label multiplies the series

1. Series multiply across labels. 50 services x 20 endpoints x 5 statuses x 20 pods = 100,000 series.
2. Add user_id, with 10,000 active users. At worst, with every user on every combination, the counter is 1 billion series.
3. Each series is held 3 times, at 8,948 bytes a copy. At worst that is about 24 TiB of memory.
4. In our stage, each of the counter's 100,000 series sees about 300 users in its 2 hours in memory. That is 30 million series, and all series go from 15 million to 45 million.
5. The ingesters need 1.10 TiB of memory and have 0.94 TiB. They run out of memory and crash, and every team loses ingest.

Stage 2: One label triples the series, with no limits

• Load: 1,500 req/sec
• First to saturate: Distributors
• Estimated cost: $19k/mo

• Latency past saturation is a lower bound, not an estimate. Distributors, Ingesters are saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Distributors: Scaled to the 100-task ceiling and 100% used at 1500 RPS.
• Ingesters: Fargate 4vCPU-16GB ×60 at 100% of a fixed 60-task ceiling.

Check in: Why did one team's label take ingest down for every team?

Per-team limits, in two places

The flood must stop before it reaches ingester memory, and only the team that caused it should notice.

Each team is a tenant, named by a header. It has two limits, checked in two places.

Distributors check its samples a second and burst first. Past them, a request gets a 429 before any ingester sees it.

The agent backs off and buffers, or drops, depending on its settings. Distributors also refuse series with too many labels, with a 400.

Ingesters check the tenant's active series, here up to 1 million, because only they know the count.

A sample for a new series past that limit gets a 400 before the series is created. It costs a lookup, but no memory.

A limit holds whatever the flood. Stage 3 draws it at the flood's own load, where a third of the samples pass, so it has no load buttons.

The ingesters see the same 15 million series as before: 3 million sample-writes a second.

Distributors hold no state, so we let them scale to 200 tasks: 1,500 / (12.5 x 0.7) = 171.4, so 172 tasks, at 70%.

Refusing a sample costs about as much CPU as accepting it, our assumption: the request is decompressed and parsed before it is counted.

Shuffle sharding gives each tenant a few of the ingesters, so one tenant's flood reaches a few, not all.

Over its rate, the team's existing series lose samples too. So it drops the label at its agent with a relabel rule, and the load returns to Stage 1's.

At 1,500 nothing saturates; "First to saturate" notes the Distributors, at 70%. The latency tile is back to 100ms, a dashboard query.

The cost tile shows $22k/mo. Most of the rise from Stage 1 is the extra distributors.

The trade-off:

Absorb at the stateless front door, and protect the stateful ingesters with limits. The noisy team pays with refused samples, and nobody else does.

Stage 3: Per-team limits, and a front door that absorbs

• Load: 1,500 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $22k/mo

Check in: What does the team see when its limits are hit, and what do the other teams see?

An incident: everyone opens the same dashboards

The interviewer asks: An incident starts, and every engineer opens the same dashboards. What slows down, and are alerts late?

The label is fixed, so writes are back to 500 requests a second. In the incident, dashboards run at 5 times their load: 750 queries a second.

That is a share of 750 / 500 = 1.5 on the entry. Rules stay at 150 evaluations a second.

Press 250 on Stage 4: half the incident, 375 queries a second. Nothing saturates, but the Queriers card turns red at 93.8%, under its 400.

Queries queue in the busy Queriers, and the latency tile reads 338ms.

Press 500. The Queriers saturate: 750 queries against 40 tasks x 10 = 400. The latency tile shows "≥ 400ms", a lower limit.

At 500 the Ruler card reads 68%. It is a separate fleet, so rule evaluations never queue behind dashboards.

It does share the ingesters: the ruler and the queriers both read the last 13 hours there.

The capped queriers pass on at most 400 queries a second. With the ruler's 150, that is 550 reads against the usual 300.

The ingester card counts writes only, so those reads are worked by hand. Writes use 62.5% of ingester CPU, and reads share the rest.

Alerting is at risk only if the ingesters run out of CPU. The querier cap, per-tenant query limits and a cache keep it off.

The cost tile shows $22k/mo. Most of the rise from Stage 1 is egress: dashboards ask 5 times as many queries.

The trade-off:

A separate ruler fleet is 22 more tasks on an ordinary day. It buys alerts that never wait behind a dashboard rush.

Stage 4: An incident: five times the dashboard queries

• Load: 500 req/sec
• First to saturate: Queriers
• Estimated cost: $22k/mo

• Latency past saturation is a lower bound, not an estimate. Queriers is saturated on the critical path, where the queueing model stops tracking a real queue - the true figure is higher by an amount this model cannot know. Relieve the bottleneck to get a latency number worth reading.
• Queriers: Scaled to the 40-task ceiling and 100% used at 750 RPS.

Check in: Dashboards are slow. Are alerts late? Say why or why not.

A results cache in front of the queriers

In an incident, most queries ask for the same few panels again and again.

A query-frontend splits each query by day and aligns its step. A refresh then reuses the cached days and recomputes only the newest interval.

Results go to Memcached, on 2 cache.r7g.large nodes. We plan for 90% of query work from the cache, our estimate.

So 750 x 0.1 = 75 queries a second reach the queriers: 75 / (10 x 0.7) = 10.7, so 11 tasks, at 68%.

Reads reaching the ingesters fall to 75 queries plus 150 evaluations, 225 a second, under the usual 300. The cache protects alerting too.

Press 500 on Stage 5. Nothing saturates; the note on "First to saturate" names the Distributors, at 69%.

The latency tile shows 100ms. It is now a rule evaluation, 100 ms on the ruler.

The tile shows the slowest path, weighted by how often each path runs. A dashboard query reaches a querier only for the 10% of work the cache leaves.

So a query weighs 0.2 + 0.1 x 100 = 10.2 ms on the tile, 0.2 ms of it at the cache.

By hand, a query that misses the cache takes 100.2 ms, and a hit takes 0.2 ms.

The cost tile shows $21k/mo. The cache costs 2 nodes x $0.219 an hour x 720 hours = $315.36 a month.

It saves 40 - 11 = 29 querier tasks, about $1,031 a month, so the parts cost less than in Stage 4.

All 750 answers a second still go out to browsers, $6,020.54 of egress. A cache saves querier work, not bytes.

Press 850. With dashboards cached, the Ingesters are the next wall again.

The trade-off:

A cache is one more part to run, and a cached panel can be one refresh behind. In an incident that trade is worth it.

Stage 5: A results cache in front of the queriers

• Load: 500 req/sec
• First to saturate: Nothing at this load
• Estimated cost: $21k/mo

Check in: Why does the cache help more in an incident than on an ordinary day?

One alert through every state

The interviewer asks: A service starts failing at 10:00. When is someone paged, and how many pages do they get?

An alert that pages on one bad scrape wakes people for nothing. One that waits too long misses the outage. Time is not in the simulation, so we work it by hand.

Evaluated: the ruler runs the rule every 60 seconds, on recent data from the ingesters.

Pending: the condition is true, but it has not held for the rule's for: time, 5 minutes here. One bad scrape cannot page.

Firing: the condition held for 5 minutes in a row, so the ruler sends the alert to Alertmanager.

Worst time to fire: up to 15 seconds to the scrape, up to 60 to the next evaluation, plus 5 minutes. That is 15 + 60 + 300 = 375 seconds, 6 minutes 15.

Notified: Alertmanager waits 30 seconds to group alerts, then pages. New alerts in that group wait 5 minutes; a firing alert repeats every 4 hours. These are its defaults.

Grouping by alert name and cluster turns 200 failing pods into one page, not 200.

An inhibition rule mutes a warning while its matching critical alert fires. A silence is different: a person mutes alerts for a set time, such as during maintenance.

Alertmanager runs 3 replicas that gossip. The ruler sends each alert to all 3, and the cluster sends each page once.

If a ruler replica dies, another takes its rule groups. An alert may be evaluated twice, and Alertmanager still pages once.

Resolved: the condition turns false, and a resolved notice goes out. keep_firing_for stops an alert flapping between firing and resolved.

If the whole alerting path stops, nothing pages at all. So a dead man's switch alert always fires, and an outside service pages when it stops.

Step through the figure to follow the alert.

The trade-off:

The for: time trades speed for quiet. A real outage can go unpaged for 6 minutes 45 seconds, and in exchange one bad scrape never wakes anyone.

One alert, from 10:00 to the page

1. A service starts failing at 10:00.
2. The next scrape can come up to 15 seconds later.
3. The ruler evaluates up to 60 seconds after that. The condition is true, so the alert is pending.
4. The condition holds for 5 minutes, so the alert fires at 10:06:15. At worst that is 15 + 60 + 300 = 375 seconds.
5. Alertmanager waits 30 seconds to group alerts, then pages at 10:06:45. Grouping turns 200 failing pods into 1 page.
6. New alerts in the group wait 5 minutes. A firing alert pages again every 4 hours.
7. When the condition turns false, the alert resolves. A dead man's switch always fires, so an outside service pages if alerting stops.

Check in: A service fails at 10:00, and its rule has a for: of 5 minutes. At the latest, when does the alert fire, and when is someone paged?

The API, the ring and the wire

The interviewer asks: Walk me through the API, and what happens when you add an ingester.

None of these parts breaks under load, so no stage is about them. An interviewer asks about each one, so we answer each in a line or two.

Write: remote write, a compressed protobuf of 2,000 samples, with the tenant in a header (X-Scope-OrgID in Mimir).

Read: PromQL over GET /api/v1/query_range, with a query, a start, an end and a step. Rules: YAML rule groups per tenant.

Series identity: the metric name plus its sorted label pairs, hashed onto the ingesters' hash ring to choose 3.

Adding an ingester: it registers its tokens on the ring, and new series that hash to them go to it at once.

Existing series stay put until they go idle and leave the head, up to about 4 hours. Removing an ingester is slower, one zone at a time.

On the wire, remote write sends each series' labels with its samples, so a sample is about 20 bytes. A batch is about 40 KiB, our estimate.

That is 500 x 40 KiB = 19.5 MiB a second into the distributors, and about 3 times that out to the ingesters.

Late data: an agent that was cut off replays its WAL in order. Mimir also accepts out-of-order samples within a window, say 30 minutes.

A retried sample has the same time and value, so it is ignored.

A stream in front of storage is the alternative: Kafka on Amazon MSK, partitioned by series hash. Partition by tenant, and one big tenant overloads a partition.

Deleting a tenant marks its blocks for deletion. Deleting one series inside a block needs a rewrite: a tombstone, then compaction.

The trade-off:

A ring keeps each series on the same 3 ingesters, which makes writes cheap. In exchange, changing the ring takes hours to settle.

Check in: You add 20 ingesters to the ring. When do they start taking load?

Build the store yourself

The interviewer asks: Design the storage engine without Mimir. And why pull at all?

An interviewer may take away the off-the-shelf store, or push on the collection model. Both answers are short.

The engine: an append-only WAL for crashes, a head in memory with each series' open chunk, and every 2 hours a block of chunk files.

Each block has an inverted index: each label pair maps to a sorted list of series ids, so a query intersects lists.

Chunks store timestamps as the change in their gaps, and values XOR the one before. That is how a sample gets to 1.5 bytes.

Pull: the agent finds targets by service discovery and keeps an up series for each, so a dead target shows.

Push: the agent sends batches out over HTTPS, so the store never reaches through firewalls into 5,000 nodes.

A short-lived batch job can end before a scrape. It pushes its last numbers to a Pushgateway, which is scraped; use one only for such jobs.

A 13-month panel over the counter's 100,000 series reads 100,000 x 390 days x 5,760 samples a day = 2.25 x 10^11 samples, 314 GiB.

A recording rule that sums it by service every 60 seconds keeps 50 series: 28.1 million samples, 40 MiB.

The query-frontend splits the range into days and runs them in parallel. Finished days never change, so they come from the cache.

The trade-off:

A recording rule costs one evaluation a minute forever. In exchange, a 13-month panel reads megabytes, not hundreds of gigabytes.

Check in: A dashboard shows 13 months of the request counter. Why can the raw query not answer in 1 second, and what makes it fast?

Say this in 45 minutes

The interviewer asks: What fails in this design, and what does an engineer see?

A design that works on a good day is half an answer. The interviewer asks what fails, who notices, and what it costs.

One ingester down: each of its series has 2 other copies, and a write needs 2 of 3. In production it restarts from its own disk and replays its WAL.

On Fargate, as drawn, it starts empty, because its disk is deleted. Its 2 other copies cover it, and restarting one zone at a time keeps 2.

S3 or the block readers down: old-data queries fail, but alerts and recent dashboards keep working.

A distributor crash: it holds no state, and the agent retries the 5xx. An Alertmanager replica down: the other 2 keep the cluster.

Per-user latency is not a user_id label. Sample traces instead: 1% of 10,000 requests a second is 100 traces a second, and no new series.

Exemplars attach a trace id to a histogram sample, so a slow bucket on a dashboard links to one real request.

Cost per million active series: $1,107.97 a month on the tile, and $15,368.62 / 15 = $1,024.57 in an average month.

Amazon Managed Service for Prometheus would charge at least $47,783 a month before query charges. It runs no ingesters for you.

Dashboard answers leave AWS either way, so add the same $1,324.39 of egress to it. That is about 3 times Stage 1's $16,619.58.

Say it out loud: "15 million series every 15 seconds is 1 million samples a second, and every series sits in memory 3 times. Cardinality is the risk."

Minutes 0 to 5: requirements. Series, retention, alerts and tenants.

Minutes 5 to 10: the numbers. 15 million series / 15 seconds = 1 million samples a second, 500 batches, 150 queries, 150 rules.

Minutes 10 to 17: a sample's states. Push and pull, the head and 2-hour blocks, 1.5 bytes a sample.

Minutes 17 to 24: the first design. Distributors, a planned ring of 60 ingesters, Mimir's ratios. Ingesters are the bill.

Minutes 24 to 31: cardinality. One label, 3 times the series, out of memory. Limits in two places, and shuffle sharding.

Minutes 31 to 37: queries. Fan-out, the results cache, an incident, and the separate ruler.

Minutes 37 to 45: alerts. The for: time, grouping, inhibition, dedup and a dead man's switch. Long queries, raw retention, crashes.

The trade-off:

Each answer has a price. Three copies triple memory, and a separate ruler and a results cache are more parts to run.

Check in: Set a timer for 45 minutes. Give the whole answer out loud, and check it against the pacing plan.