Cardinality¶

03:14 AM. Page: Prometheus OOM-killed, restarting, WAL replay taking 40 minutes. Nothing about traffic changed this week. The only recent change: Friday's deploy added one new label, user_id, to an existing HTTP metric, "for better debugging."

What happened? A) a memory leak in Prometheus itself, B) the scrape interval got misconfigured, C) one label turned the metric's series count into the billions, or D) a network partition caused duplicate scrapes. Pick one before reading on.

It's (C): user_id looks like observability, but it is a new time series per user per other-label combination — an index that grows until the TSDB cannot compact, scrape, or query — because cardinality is not "how many rows," it is how many distinct series identities the engine keeps hot.

Start with the situation¶

SaaS observability. You already scrape:

http_requests_total{service="api", region="us-east", status="200", endpoint="/checkout"}

A PM wants “latency per user” on the same Grafana. The patch is:

http_requests_total{..., user_id="u123456"}

Dashboards over hundreds of millions of events/day can store per-user facts in ClickHouse. They cannot index per-user Prom series.

IoT analogue: {device_id, sensor} is the series — 10 M devices × 5 sensors = 50 M series. Painful but bounded. Add firmware_build + session_id and you are gone.

Explore numbers: cardinality calculator.

Why the obvious approach breaks¶

Every unique (name, label set) is a series. Each series has:

• an inverted index entry (label value → series ids);
• a head block in RAM;
• WAL / chunk writes even if you scrape once.

Cost is #series, not #samples, until samples pile up. A label with 10⁷ values multiplies every other dimension.

series ≈ ∏ |values(label_i)|   (if independent)

They are never fully independent, but the product is the planning bound. Engineering reviews use the product, not hope.

Build the mental picture¶

A series is a named line on a graph. Grafana can draw 20 lines. Prometheus can hold millions of lines in the index. It cannot hold a line per customer per endpoint per status per user.

Columns in ClickHouse are not lines. GROUP BY user_id is a query-time hash table. You pay it when you query, not at every scrape.

flowchart TB
    subgraph prom["Prometheus identity"]
        S1["svc, region, status, endpoint, user=1"]
        S2["svc, region, status, endpoint, user=2"]
        S3["... user=N"]
    end
    subgraph ch["ClickHouse row"]
        R["one row: columns include user_id"]
    end

If you can name a bound (“we have ≤ 200 endpoints”), it can be a label. If the bound is “our user table,” it is a column in an event store.

Under the hood¶

What Prometheus stores¶

• Head: in-memory samples for active series.
• Postings: inverted index, label pair → series.
• Chunks: compressed samples on disk, per series.

Memory scales roughly linearly with active series (plus label string length). Order of magnitude: hundreds of bytes to a few KB per series of overhead, not counting samples. 10 M series is a dedicated, well-run Prom/VM. 10⁹ series is not a Prom problem.

Churn (series that appear once — trace_id) is worse: index grows, compaction storms, scrape payloads huge, then the series goes idle but not immediately forgotten.

Explosion, stepwise¶

Endpoint at 500 is already a conversation (path templating vs raw URLs). User_id is not a conversation; it is a page.

Raw URLs: /orders/8891 as endpoint is user-level cardinality. Template to /orders/:id before it becomes a label.

Why ClickHouse survives the same points¶

CREATE TABLE http_events
(
    ts         DateTime64(3),
    service    LowCardinality(String),
    region     LowCardinality(String),
    status     UInt16,
    endpoint   LowCardinality(String),
    user_id    String,
    latency_ms UInt32
)
ENGINE = MergeTree
ORDER BY (service, endpoint, ts);

user_id is a column. The primary index does not allocate a granule list per user unless you sort by user. 5×10⁸ rows/day is parts and compression, not 5×10¹² series.

GROUP BY user_id on a day can still OOM — that is query cardinality, not ingest identity. Different knob (max_memory_usage, two-level agg, pre-agg).

Ingest cardinality vs query cardinality¶

ClickHouse lets you ingest 2 M users/day as a column for cheap, then refuse the query GROUP BY user_id without customer_id =. That is an application limit, not a TSDB series limit. Prometheus does not give you that split: ingest is identity.

Pinot sits in the middle: ingest of a high-card column as a raw dimension can be fine; promoting it to inverted/star-tree makes it Prom-shaped again.

IoT: device_id is bounded — until it is not¶

10 M devices × 5 sensors = 50 M series if {device_id, sensor} is the Prom identity. Some VM clusters hold that. Add session_id (new UUID each Wi‑Fi join) and you get churn: series that live for 20 minutes and pollute the index forever-ish.

Better IoT split:

fleet Prom/VM:  temperature{model, site, customer}     # bounded
drill-down CH:  (ts, device_id, sensor, value)         # 10 M ids as column
                                                ORDER BY (device_id, sensor, ts)

device_id in ClickHouse with that ORDER BY is the skip key for “chart this thermostat.” In Prom it is 10 M lines on every global query unless you always {device_id="..."}.

VictoriaMetrics / Mimir / Thanos¶

They raise the ceiling (better compression, sharding). They do not make user_id a good label. 100× more series is still the wrong model for events.

Put it to work¶

Bound labels (Prom):

http_requests_total{service, region, env, status, endpoint}   # endpoint templated
http_request_duration_seconds_bucket{..., le}

Per-user / per-request (ClickHouse / Pinot):

INSERT INTO http_events VALUES
    (now64(), 'api', 'us-east', 200, '/checkout', 'u123456', 42);

IoT Prom-style (if you must): device_id only if 10 M series is funded (VM cluster, not one pod). Prefer: fleet metrics with {model, site} + drill-down SQL on {device_id, ts, value}.

Recording rules do not fix cardinality. A rule that does sum by (user_id) creates more series if you already exploded, or fails if you did not store user_id. Rules are for reducing dimensions (sum by (service)).

Relabel to drop:

# Prometheus scrape relabel — drop user_id if a client added it
metric_relabel_configs:
  - regex: user_id
    action: labeldrop

Measure:

prometheus_tsdb_head_series
prometheus_tsdb_symbol_table_size_bytes
count({__name__=~".+"})
topk(10, count by (__name__)({__name__=~".+"}))

If one metric name owns 80% of series, you found the label.

Python bound (same as the calculator):

services, regions, statuses, endpoints, users = 50, 10, 20, 500, 1_000_000
print("without users", services * regions * statuses * endpoints)
print("with users", services * regions * statuses * endpoints * users)
# 5e6 vs 5e12

Where teams get caught¶

How it fails¶

How to investigate¶

1. topk(20, count by (__name__, user_id)(http_requests_total)) — if this even parses slowly, stop.
2. Cardinality dashboard: series per metric name, per label name.
3. count(count by (user_id)(http_requests_total)) → unique users as series dimension.
4. App: dump /metrics size. If it is 80 MB, the incident started in the process, not in Prom.
5. ClickHouse: SELECT uniqExact(user_id) FROM http_events WHERE ts > now() - 3600 — this is allowed. SELECT count() FROM system.metrics analogue in Prom is head_series.

Marks/parts: if you tried to “fix” Prom by dumping into CH and ORDER BY (user_id, ts) with 50 M users, you get a wide index and tiny granules per user — different problem (too many marks), still better than Prom.

Scale 10× / 100× / 1000׶

Traffic growth is a sample problem. Dimension growth is a series problem. Only the second is this page.

Trade-offs¶

Hybrid is normal: Prom for {service, endpoint} SLOs; ClickHouse for user/session drill-down.

Alternatives¶

Apply¶

Label review in PR: bound? if not, reject. Path templating in HTTP metrics is a platform standard, not an app choice.

If a product metric needs user_id, it is an event — Kafka topic, OLAP table, ORDER BY (customer_id, ts).

Platform checklist before merge:

1. Product of label bounds written in the PR (not “should be small”).
2. HTTP paths templated in middleware.
3. Histogram le buckets counted in the product.
4. A cardinality dashboard exists (head_series by metric name).
5. If someone says “we’ll add remote_write,” that scales samples, not a 10¹² series bound.

Incident smell: /metrics grew from 2 MB to 80 MB after a “debug label” shipped on Friday. Roll back the label before you scale the TSDB. The TSDB is working as designed.

For a walk-through of the product-of-bounds arithmetic, use the cardinality calculator. Change one label at a time. The cliff is never “10× traffic”; it is the unbounded dimension.

Compare engines once the identity is a column: TSDBs, ClickHouse, TSDB vs OLAP.

Practice the idea¶

Use the cardinality calculator to predict the multiplier from adding user_id. Then run the time-series lab and observe prometheus_tsdb_head_series before and after the label is enabled.

Check your understanding¶

http_request_duration_seconds_bucket has labels service (80), le (12 buckets), endpoint (templated 40), method (8). Traffic: 200 M requests/day, 2 M DAU.

Engineer A adds user_id. Engineer B adds customer_id (2,000 tenants). Engineer C writes requests to ClickHouse with those as columns and keeps Prom without them.

Estimate series for A and B (product bound). Who is right for a “p95 per tenant” product dashboard vs SLO per service?