Cardinality¶

03:14 AM. Page: Prometheus OOM-killed, restarting, WAL replay taking 40 minutes. Nothing about traffic changed this week. The only recent change: Friday's deploy added one new label, user_id, to an existing HTTP metric, "for better debugging."

What happened? A) a memory leak in Prometheus itself, B) the scrape interval got misconfigured, C) one label turned the metric's series count into the billions, or D) a network partition caused duplicate scrapes. Pick one before reading on.

It's (C): user_id looks like observability, but it is a new time series per user per other-label combination — an index that grows until the TSDB cannot compact, scrape, or query — because cardinality is not "how many rows," it is how many distinct series identities the engine keeps hot.

Start with the situation¶

SaaS observability. You already scrape:

http_requests_total{service="api", region="us-east", status="200", endpoint="/checkout"}

A PM wants “latency per user” on the same Grafana. The patch is:

http_requests_total{..., user_id="u123456"}

Dashboards over hundreds of millions of events/day can store per-user facts in ClickHouse. They cannot index per-user Prom series.

IoT analogue: {device_id, sensor} is the series — 10 M devices × 5 sensors = 50 M series. Painful but bounded. Add firmware_build + session_id and you are gone.

Explore numbers: cardinality calculator.

Why the obvious approach breaks¶

Every unique (name, label set) is a series. Each series has:

• an inverted index entry (label value → series ids);
• a head block in RAM;
• WAL / chunk writes even if you scrape once.

Cost is #series, not #samples, until samples pile up. A label with 10⁷ values multiplies every other dimension.

series ≈ ∏ |values(label_i)|   (if independent)

They are never fully independent, but the product is the planning bound. Engineering reviews use the product, not hope.

Build the mental picture¶

A series is a named line on a graph. Grafana can draw 20 lines. Prometheus can hold millions of lines in the index. It cannot hold a line per customer per endpoint per status per user.

Columns in ClickHouse are not lines. GROUP BY user_id is a query-time hash table. You pay it when you query, not at every scrape.

flowchart TB
    subgraph prom["Prometheus identity"]
        S1["svc, region, status, endpoint, user=1"]
        S2["svc, region, status, endpoint, user=2"]
        S3["... user=N"]
    end
    subgraph ch["ClickHouse row"]
        R["one row: columns include user_id"]
    end

If you can name a bound (“we have ≤ 200 endpoints”), it can be a label. If the bound is “our user table,” it is a column in an event store.

Under the hood¶

What Prometheus stores¶

• Head: in-memory samples for active series.
• Postings: inverted index, label pair → series.
• Chunks: compressed samples on disk, per series.

Memory scales roughly linearly with active series (plus label string length). Order of magnitude: hundreds of bytes to a few KB per series of overhead, not counting samples. 10 M series is a dedicated, well-run Prom/VM. 10⁹ series is not a Prom problem.

Churn (series that appear once — trace_id) is worse: index grows, compaction storms, scrape payloads huge, then the series goes idle but not immediately forgotten.

Explosion, stepwise¶

Endpoint at 500 is already a conversation (path templating vs raw URLs). User_id is not a conversation; it is a page.

Raw URLs: /orders/8891 as endpoint is user-level cardinality. Template to /orders/:id before it becomes a label.

Why ClickHouse survives the same points¶

CREATE TABLE http_events
(
    ts         DateTime64(3),
    service    LowCardinality(String),
    region     LowCardinality(String),
    status     UInt16,
    endpoint   LowCardinality(String),
    user_id    String,
    latency_ms UInt32
)
ENGINE = MergeTree
ORDER BY (service, endpoint, ts);

user_id is a column. The primary index does not allocate a granule list per user unless you sort by user. 5×10⁸ rows/day is parts and compression, not 5×10¹² series.

GROUP BY user_id on a day can still OOM — that is query cardinality, not ingest identity. Different knob (max_memory_usage, two-level agg, pre-agg).

Ingest cardinality vs query cardinality¶

ClickHouse lets you ingest 2 M users/day as a column for cheap, then refuse the query GROUP BY user_id without customer_id =. That is an application limit, not a TSDB series limit. Prometheus does not give you that split: ingest is identity.

Pinot sits in the middle: ingest of a high-card column as a raw dimension can be fine; promoting it to inverted/star-tree makes it Prom-shaped again.

IoT: device_id is bounded — until it is not¶

10 M devices × 5 sensors = 50 M series if {device_id, sensor} is the Prom identity. Some VM clusters hold that. Add session_id (new UUID each Wi‑Fi join) and you get churn: series that live for 20 minutes and pollute the index forever-ish.

Better IoT split:

fleet Prom/VM:  temperature{model, site, customer}     # bounded
drill-down CH:  (ts, device_id, sensor, value)         # 10 M ids as column
                                                ORDER BY (device_id, sensor, ts)

device_id in ClickHouse with that ORDER BY is the skip key for “chart this thermostat.” In Prom it is 10 M lines on every global query unless you always {device_id="..."}.

VictoriaMetrics / Mimir / Thanos¶

They raise the ceiling (better compression, sharding). They do not make user_id a good label. 100× more series is still the wrong model for events.

Put it to work¶

Bound labels (Prom):

http_requests_total{service, region, env, status, endpoint}   # endpoint templated
http_request_duration_seconds_bucket{..., le}

Per-user / per-request (ClickHouse / Pinot):

INSERT INTO http_events VALUES
    (now64(), 'api', 'us-east', 200, '/checkout', 'u123456', 42);

IoT Prom-style (if you must): device_id only if 10 M series is funded (VM cluster, not one pod). Prefer: fleet metrics with {model, site} + drill-down SQL on {device_id, ts, value}.

Recording rules do not fix cardinality. A rule that does sum by (user_id) creates more series if you already exploded, or fails if you did not store user_id. Rules are for reducing dimensions (sum by (service)).

Relabel to drop:

# Prometheus scrape relabel — drop user_id if a client added it
metric_relabel_configs:
  - regex: user_id
    action: labeldrop

Measure:

prometheus_tsdb_head_series
prometheus_tsdb_symbol_table_size_bytes
count({__name__=~".+"})
topk(10, count by (__name__)({__name__=~".+"}))

If one metric name owns 80% of series, you found the label.

Python bound (same as the calculator):

services, regions, statuses, endpoints, users = 50, 10, 20, 500, 1_000_000
print("without users", services * regions * statuses * endpoints)
print("with users", services * regions * statuses * endpoints * users)
# 5e6 vs 5e12

Where teams get caught¶

How it fails¶

Practice the idea¶

Use the cardinality calculator to predict the multiplier from adding user_id. Then run the time-series lab and observe prometheus_tsdb_head_series before and after the label is enabled.

What happened next¶

C. Friday's user_id label multiplied the metric's series count by the number of distinct users, and each series is an identity the TSDB keeps in an in-memory index. Traffic was unchanged all week — the number of things being counted was not.

That is why the restart took 40 minutes. WAL replay rebuilds the index, and the index is now proportional to users rather than to endpoints, so recovery time grew with the same multiplier that caused the OOM.

The label was added "for better debugging," and per-user latency is a reasonable thing to want. It is a request for an events store with high-cardinality columns, not for another dimension on a metric — the same distinction the TSDB page sorts engines by.

Check your understanding¶

http_request_duration_seconds_bucket has labels service (80), le (12 buckets), endpoint (templated 40), method (8). Traffic: 200 M requests/day, 2 M DAU.

Engineer A adds user_id. Engineer B adds customer_id (2,000 tenants). Engineer C writes requests to ClickHouse with those as columns and keeps Prom without them.

Estimate series for A and B (product bound). Who is right for a “p95 per tenant” product dashboard vs SLO per service?

Reference¶

Behaviour at the next orders of magnitude, the trade-offs, the alternatives, and what to check when inheriting someone else's version of this — kept here rather than in the walkthrough above.

How to investigate¶

1. topk(20, count by (__name__, user_id)(http_requests_total)) — if this even parses slowly, stop.
2. Cardinality dashboard: series per metric name, per label name.
3. count(count by (user_id)(http_requests_total)) → unique users as series dimension.
4. App: dump /metrics size. If it is 80 MB, the incident started in the process, not in Prom.
5. ClickHouse: SELECT uniqExact(user_id) FROM http_events WHERE ts > now() - 3600 — this is allowed. SELECT count() FROM system.metrics analogue in Prom is head_series.

Marks/parts: if you tried to “fix” Prom by dumping into CH and ORDER BY (user_id, ts) with 50 M users, you get a wide index and tiny granules per user — different problem (too many marks), still better than Prom.

Scale 10× / 100× / 1000׶

Traffic growth is a sample problem. Dimension growth is a series problem. Only the second is this page.

Trade-offs¶

Hybrid is normal: Prom for {service, endpoint} SLOs; ClickHouse for user/session drill-down.

Alternatives¶

Apply¶

Label review in PR: bound? if not, reject. Path templating in HTTP metrics is a platform standard, not an app choice.

If a product metric needs user_id, it is an event — Kafka topic, OLAP table, ORDER BY (customer_id, ts).

Platform checklist before merge:

1. Product of label bounds written in the PR (not “should be small”).
2. HTTP paths templated in middleware.
3. Histogram le buckets counted in the product.
4. A cardinality dashboard exists (head_series by metric name).
5. If someone says “we’ll add remote_write,” that scales samples, not a 10¹² series bound.

Incident smell: /metrics grew from 2 MB to 80 MB after a “debug label” shipped on Friday. Roll back the label before you scale the TSDB. The TSDB is working as designed.

For a walk-through of the product-of-bounds arithmetic, use the cardinality calculator. Change one label at a time. The cliff is never “10× traffic”; it is the unbounded dimension.

Compare engines once the identity is a column: TSDBs, ClickHouse, TSDB vs OLAP.