Observability Platform Architecture¶

02:47 AM. Grafana is empty for checkout-service. Every other service still shows traffic. Someone on the bridge says "just add a Prometheus label for user_id so we can see which customer is affected" — and someone else objects. A. Add the label; Prometheus can take it. B. The label would work but would kill the TSDB at scale — reach for ClickHouse instead. C. The dashboard is empty because of an ingest problem that has nothing to do with labels. D. Restart Flink and see if it comes back. Pick one before reading on.

The dashboard being empty is a retention/routing question, and the user_id-as-label instinct is the cardinality trap this page exists to prevent. Logs, metrics, traces, and security events share one ingest path here, but the hard problem is not "collect everything" — it is answering an on-call query in a second while not paying NVMe prices for a year of raw JSON.

This page walks V1 → bottleneck → V2, then 10×. Related: ClickHouse, TSDB vs OLAP, ClickHouse vs Trino, cardinality simulation.

Requirements (write these down first)¶

Out of scope for V1: full-text log search like Elasticsearch, customer-facing product analytics, and a second "security lake" cluster.

Capacity sketch (do this before buying nodes)¶

Uncompressed ingest:

Kafka retention is a replay buffer, not the observability store. If you set Kafka to 7 days at this volume you are paying for a second, worse ClickHouse.

Produce/consume parallelism: a Flink job at 5M/s with 1k events/s/core (enrich + JSON parse is not free) wants thousands of cores or cheaper parse (binary proto, vectorised decode). V1 will not be 5M/s. V1 exists to prove the query shape.

V1 — fewest parts (≈ 50k events/s, one team)¶

apps  →  Kafka (3 brokers, 24–48 partitions, RF=3)
      →  Flink (parse, drop debug if needed, add service metadata)
      →  ClickHouse (single replica or 1 shard × 2 replicas)
      →  Grafana

Keep Prometheus (or VictoriaMetrics) beside this path for scrape metrics and Alertmanager. Do not ingest per-request logs into Prometheus.

What V1 includes¶

• One events topic, keyed by service (or service + host if one service is huge).
• ClickHouse ORDER BY (service, timestamp) — matches "this service, last 15 minutes."
• TTL 7 days on the MergeTree table.
• Kafka retention 24–72 h for replay only.
• Structured logs (JSON/proto). Unstructured text search is not V1.

What you would not add yet¶

• Iceberg, Spark, Trino
• Pinot
• A second Kafka cluster "for logs vs traces"
• Elasticsearch / OpenSearch
• Per-team Kafka clusters
• A metadata catalogue as a blocker to shipping Grafana

V1 ClickHouse table¶

CREATE TABLE events
(
    timestamp   DateTime64(3),
    service     LowCardinality(String),
    host        LowCardinality(String),
    level       LowCardinality(String),
    trace_id    String,
    message     String,
    attributes  Map(String, String)
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (service, timestamp, host)
TTL timestamp + INTERVAL 7 DAY
SETTINGS index_granularity = 8192;

trace_id is not in ORDER BY. Point lookups of a random trace will scan the service+time range — that is acceptable if the UI always has a time window. A bloom skip index on trace_id is a later, cheap add. Do not rebuild the primary key around traces unless that is the #1 query.

This architecture works while… / breaks when…¶

Works while:

ingest stays roughly ≤ 50k events/s and one team owns the whole path
queries filter by service and a time window — the ORDER BY prefix
7 days of hot retention fits one ClickHouse shard's disk after ~10:1 compression
dropping debug logs under overload is an acceptable relief valve
Kafka is a 24–72 h replay buffer, not the store of record
enrichment is stateless (parse, cast, attach service metadata)

Breaks when:

one service is a large fraction of traffic — a single-key partition pins ingest
a required query does NOT lead with service + time (trace_id lookups, "all errors
  across the fleet by user_id") — the ORDER BY stops pruning and reads go linear
insert batches get small — thousands of parts, and query time collapses on merges,
  not on CPU
hot retention outgrows one shard, so 7 days no longer fits without sharding
forensics over months arrives as a requirement — it does not belong on hot storage
enrichment becomes stateful (dedup, trace stitching, tenant-aware field dropping)

The first four are the V2 triggers, and each has a metric you can alert on today: per-partition lag skew, query scan bytes, active parts per table, disk headroom.

Bottleneck at the end of V1¶

You will hit one of these first. Name it; do not "scale the diagram."

The ClickHouse cliff is usually parts (tiny inserts) or wrong ORDER BY, not CPU. Tiny Kafka Engine flushes create thousands of parts; queries read too many files. See incident: ClickHouse 10× slower.

Hot service: if api-gateway is 40% of traffic, keying only by service pins 40% of the topic to one partition. Compound the key (service + host or a hash suffix on the hot service only).

V2 — remove the measured bottleneck (≈ 500k–5M events/s)¶

Split hot path (incident response) from cold path (forensics, compliance).

graph TD
    A["Applications"] --> K["Kafka\nRF=3, 1–3 day retention\nmain topic 400–800 partitions"]
    K --> F["Flink\nenrich, redact, route, downsample"]
    F --> CH["ClickHouse cluster\nsharded by service hash\n7 day TTL, NVMe"]
    K --> SP["Spark hourly/nightly"]
    SP --> ICE["Iceberg on object storage\n1-min and hourly rollups\n90d / 1y"]
    CH --> G["Grafana hot dashboards"]
    ICE --> TRI["Trino\ninvestigations, joins to CMDB"]
    TRI --> G
    P["Prometheus / VM"] --> AM["Alertmanager"]
    A --> P

Hot path¶

Kafka → Flink → ClickHouse.

• Latency: seconds from log line to Grafana.
• Retention: 7 days.
• Cost: NVMe, lots of cores, worth it for on-call.
• Shard ClickHouse by cityHash64(service) (or service + time) so one noisy service does not fill every node.

Cold path¶

Kafka → Spark (or Flink batch) → Iceberg.

• Latency: minutes to hours.
• Retention: 90 days 1-min, 1 year hourly.
• Cost: object storage.
• Trino for "what did this service do last quarter?" and joins to asset inventory. Not for the 15-minute error graph.

What V2 still does not add¶

• Pinot, unless you have thousands of QPS of fixed-dimension, sub-second-fresh dashboards. Grafana on-call is not that. See ClickHouse vs Pinot.
• Elasticsearch as the system of record. If you need inverted-index search on message, add it as a derived 7-day index, not a second ingest silo.
• Dual writes from every app (one to Kafka, one to ES, one to S3). One producer; fan-out in the cluster.

Table design details that actually matter¶

Metrics in ClickHouse (when cardinality is too high for a TSDB)¶

CREATE TABLE metrics
(
    timestamp DateTime64(3),
    metric    LowCardinality(String),
    labels    Map(LowCardinality(String), String),
    value     Float64
)
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(timestamp)
ORDER BY (metric, timestamp)
TTL timestamp + INTERVAL 7 DAY;

Maps are convenient and easy to abuse. A label key with a million values still hurts compression and GROUP BY. Prefer known columns for dimensions you filter on (service, status_code).

Rollups (warm/cold)¶

Flink or ClickHouse materialized views:

CREATE MATERIALIZED VIEW metrics_1m
ENGINE = SummingMergeTree
PARTITION BY toYYYYMMDD(ts_minute)
ORDER BY (metric, service, ts_minute)
AS
SELECT
    toStartOfMinute(timestamp) AS ts_minute,
    metric,
    labels['service']          AS service,
    count()                    AS n,
    sum(value)                 AS sum_value
FROM metrics
GROUP BY ts_minute, metric, service;

Do not keep raw 5M/s in Iceberg "just in case" without a compaction and expiry plan. That is how you get the Iceberg snapshot incident.

How it fails¶

Cost (order of magnitude, 5M/s)¶

Tiered retention is a 10–50× storage win versus "hot forever." Compute (Flink parse) can exceed storage at this rate — binary encodings and dropping debug are cost features.

Evolution at 10× (50M events/s)¶

You probably do not replace ClickHouse. You:

1. Pre-aggregate in Flink (per service per second) so ClickHouse sees fewer rows for dashboards. Keep a sampled raw table for trace drill-down.
2. Split topics by telemetry type (logs vs spans) so retention and ORDER BY can differ.
3. Shard more — but by the query key (service), not randomly, or every query becomes a scatter-gather.
4. Pull Prometheus out of the event path entirely if it is still mixed in.
5. Regional Kafka — produce locally, aggregate centrally, or query federated ClickHouse. Cross-AZ traffic at 50M/s is a WAN bill.
6. Refuse new high-card dimensions in the hot table; they go to cold/sampled.

If 10× is 10× query QPS (many tenants staring at live graphs), that is the moment to evaluate Pinot or aggressive ClickHouse caching — a different bottleneck than ingest.

Decision table¶

Apply this at work¶

1. Write the three query SLAs (15 min / 24 h / 90 d) with example SQL.
2. Compute GB/day uncompressed and compressed.
3. Choose ORDER BY from the first Grafana panel, not from "all possible filters."
4. Set Kafka retention as replay (days), ClickHouse TTL as hot (days), Iceberg as archive (months).
5. Add a cardinality budget: which fields may never become labels.
6. Draw V1 with three boxes. Put V2 on a second slide. Ship V1.

Work the ClickHouse ORDER BY lab and the cardinality calculator before you freeze the schema.

Query catalogue (write these in the design doc)¶

The stack follows from these queries, not from "we need observability."

If Q2 becomes "full-text across 30 days of unstructured logs," you have a search product. That is a new V2 component (OpenSearch) with 7-day retention, not a reason to abandon CH.

Partition and Flink parallelism (Stage 3)¶

At 5M events/s, 800 partitions:

• Flink Kafka source: source parallelism ≤ 800. Typical: 200–800 depending on parse cost.
• Checkpoint interval 30–60 s. At this rate, checkpoint size is state, not the events (events are not stored in Flink if you only map). If you sessionize or buffer 1-min windows, state = keys × window. 100k services × 1 min of raw is huge — aggregate in the window, do not buffer raw.
• ClickHouse insert: 100k–1M rows/block, every 1–5 s. Tiny blocks → too many parts.

Kafka consumer group for Flink: one group for the hot path. A second group for Spark batch. That is why Kafka exists — fan-out. Do not share a group between Flink and Spark.

Sampling and overload¶

When ingest exceeds what CH can merge:

1. Drop debug/info first (keep error/fatal + traces).
2. Sample info at 1% per service, not globally (or a quiet service vanishes).
3. Never sample traces that are in an error path if the product is debugging.
4. Emit an ingest_dropped_total metric to Prometheus so the drop is visible.

Overload is a requirement ("what may we drop"). If you cannot answer, you will drop randomly when disks fill.

On-call 15 minutes¶

1. Grafana empty? Check CH max(timestamp) and Kafka lag per partition.
2. One service missing? Hot key or parser fail for that service — quality parse-fail counter.
3. All services slow to query? system.parts and query_log marks — incident 4.
4. Alerts firing on CPU but not on errors? You are looking at Prometheus; the event path can still be dead.

Do not restart Flink as step 1. Restarts rewind checkpoints and duplicate inserts unless sinks are idempotent.

Stage table (resume of the ladder)¶

Jumping to Stage 3 on day one is how you spend two quarters on Keeper and never ship Grafana. Ship Stage 1 with the same ORDER BY you will need at Stage 3. Changing ORDER BY later is a migration, not a config flag.