Fraud Detection Architecture¶

14:02:11.900 — a $4,200 transaction is authorized. 14:02:12.300 — the fraud model finally returns a high-risk score, 400 ms after the authorization already fired. The chargeback lands six weeks later. A design review asks: what should have been on the 200 ms path that wasn't? A. A faster model. B. A 3-hop graph query to check known fraud rings before scoring. C. Keyed feature lookups only — no graph hop — with rings caught downstream instead. D. More Kafka partitions. Pick one before reading on.

C is the shape of this architecture: transactions, devices, cards, IPs, merchants, users. Four products share a bus and must not share a latency budget:

1. Score this payment — p99 < 200 ms (authorisation path).
2. Find rings — multi-hop graph, minutes to hours, batch is fine.
3. Investigate a case — ad-hoc SQL over months.
4. Report — fraud rate by MCC/region, dashboards.

No single database is good at all four. The architecture is a deliberate split. Related: graph vs relational, Spark vs Flink, Flink state.

Requirements¶

Capacity sketch (2,000 tx/s V1 → 20,000 at 10×)¶

Partitions: 24–48 at 2k/s if scoring is heavier than the broker. Key for scoring state is user_id / card_id so velocity counts are local to a Flink key. If you key only by transaction_id, every velocity check is a distributed mess.

V1 — fewest parts (ship a score)¶

Payment API → Kafka (or sync call into a scorer that also logs to Kafka)
           → Flink (rules + keyed velocity + in-process model)
           → response (sync sidecar) or async with a tight timeout
           → Kafka `scores` + Iceberg (async)
           → ClickHouse (features and analyst tables, async)

Synchronous pattern that stays honest: the payment service calls a scorer process that shares the same Flink/feature logic or a request/reply Kafka with a 150 ms timeout. Many teams run a gRPC feature service + a model sidecar, and use Flink only to maintain features. V1 can be:

• Flink job maintains user_id → velocity in state, writes snapshots to ClickHouse/Redis.
• Scorer reads Redis/CH + local rules in < 50 ms.

Do not wait for Iceberg commits on the charge.

What V1 includes¶

• Rules: amount vs user average, country change, velocity (>5 tx / 1 min).
• Keyed Flink or Redis sliding counters.
• ClickHouse user_features updated async (seconds late is OK for some features; mark them as_of).
• Iceberg append of every tx + score for later labels (chargeback).
• A fail policy: timeout → default score.

What you would not add yet¶

• Neo4j
• Online graph embeddings
• Trino on the hot path
• Pinot
• Per-feature microservices
• Exactly-once business side effects (emails) without an outbox

This architecture works while… / breaks when…¶

Works while:

the decision needs single-entity features: this user, this card, this device
the 200 ms budget is met by rules + keyed counters read from Redis/ClickHouse
a timeout can safely fall back to a default score (the fail policy is written down
   and someone owns it)
feature staleness of seconds is acceptable for the async-maintained features
labels (chargebacks) arrive slowly enough that a batch retrain cadence is fine

Breaks when:

the fraud signal is relational — rings of accounts sharing devices, cards, addresses.
   No amount of per-user velocity state finds a pattern that only exists BETWEEN
   entities. This, not volume, is what buys a graph store.
the feature budget stops closing: each added feature is a lookup, and 200 ms is a
   hard external constraint you cannot scale your way out of
fail-open becomes financially unacceptable, so the fallback path itself needs an SLO
investigation load grows — analysts need ad-hoc history, which is a different store
   and a different latency class from scoring
model iteration outpaces the label pipeline, and "is this model better" cannot be
   answered from what you retained

The first is the important one: it is a shape change, not a scale change. The V1 architecture cannot be tuned into answering it.

Bottleneck at the end of V1¶

The 200 ms budget dies on network hops, not on a logistic regression.

V2 — add graph and investigation the slow way¶

graph TD
    TX["Auth / tx events"] --> K["Kafka"]
    K --> FL["Flink\nvelocity, rules, model features"]
    FL --> API["Score response <200ms"]
    FL --> CH["ClickHouse\nuser/device/merchant features"]
    FL --> K2["graph-edge events"]
    K --> ICE["Iceberg years"]
    K2 --> BATCH["Spark daily/hourly"]
    BATCH --> NEO["Neo4j derived graph"]
    BATCH --> CH
    ICE --> TRI["Trino investigation"]
    TRI --> CH
    NEO --> INV["Analyst ring UI"]
    PG["OLTP accounts"] --> CDC["Debezium"]
    CDC --> K

Hot path (< 200 ms) — still no graph¶

tx in → Flink / scorer
        lookup user_features (CH or KV)
        lookup device_risk
        velocity in local state
        model in-process
      → score + reasons

ClickHouse for analytical features (30-day aggregates). Redis/KV for hot counters if CH p99 is unsafe. Total hops: one or two, in parallel.

CREATE TABLE user_features
(
    user_id              String,
    last_updated         DateTime,
    avg_txn_amount_30d   Float32,
    txn_count_30d        UInt32,
    unique_devices_30d   UInt16,
    unique_merchants_30d UInt16,
    fraud_flags_30d      UInt8,
    risk_score           Float32
)
ENGINE = ReplacingMergeTree(last_updated)
ORDER BY user_id;

Ring detection (batch)¶

hourly/daily:
1. Edges from Kafka/Iceberg: user–device, device–ip, user–card, user–merchant
2. MERGE into Neo4j (or rebuild)
3. GDS connected components / WCC
4. Components over threshold → ClickHouse + case queue

Neo4j is derived. If it burns down, scoring continues. Rebuild from Iceberg.

Investigation¶

Trino over Iceberg + CH. Time travel when a model version is disputed. Do not give investigators production Redis.

Why four stores¶

How it fails¶

What V2 still does not add¶

• Online multi-hop during authorisation.
• A second stream processor "for ML."
• Storing PCI PAN in ClickHouse/Iceberg (tokenise; security).
• Analysts querying Kafka with Trino as the investigation UI (possible, wrong SLA).

Evolution at 10× (20k tx/s)¶

1. More partitions keyed by user/card — velocity state shards with keys.
2. CH cluster for features; or move the hottest keys to KV entirely.
3. Regional scoring — do not RTT to another continent in the 200 ms.
4. Graph job may need graph compute (Spark GraphX / GDS on a bigger box), not 10× Cypher writes.
5. Feature count will try to explode; budget dimensions like cardinality in observability.
6. Still no Neo4j on the hot path at 10×. Especially not at 10×.

Decision table¶

Apply this at work¶

1. Write the four use cases on a whiteboard. Assign an SLA and a store to each. Anything that sits on two SLAs is a bug.
2. Budget the 200 ms: parse, features, model, margin. Cut hops until it fits.
3. Define fail-open vs fail-closed with product/risk, not with engineering taste.
4. Put transaction_id on every score, feature snapshot, and alert.
5. Rebuild the graph from the lake on purpose once, so you know you can.

Pair with Flink labs (state + watermarks) and graph modelling.

Feature budget (200 ms)¶

Write a table in the design review. Numbers are illustrative; measure.

If CH p99 is 80 ms, it does not fit. Move that feature to KV or accept staleness. Adding "one more microservice" is adding a tail.

Labels and chargebacks¶

Fraud models need labels that arrive days later. Architecture:

• Score time: write transaction_id, score, model_version, feature_snapshot_id to Iceberg.
• Chargeback time: join labels in Spark (minutes, skew-prone — incident 2 if one merchant).
• Never train on CH TTL data that already expired.

If you cannot join scores to labels, you do not have ML; you have rules with extra steps.

Graph edge model (batch)¶

(:User)-[:USED]->(:Device)
(:User)-[:USED]->(:IP)
(:User)-[:HAS]->(:Card)
(:User)-[:PAYS]->(:Merchant)

Connected component size > N → queue. False positives: families, offices, CGNAT IPs. IP edges need recency TTL or the graph becomes one component.

Do not emit an edge per tick of a web session. Sample or debounce in Flink before the graph topic.

Fail-open vs fail-closed¶

This is not an engineering preference. Write it with risk. Timeouts must implement the policy, not hang until the processor dies.

PCI / PII¶

• PAN → token before Kafka.
• CVV never stored.
• Scores, devices, emails still PII-ish; mask in notebooks (security).
• Analysts query Iceberg curated, not Redis prod.

On-call 15 minutes¶

1. p99 score > 200 ms: trace which hop (CH vs model vs Kafka reply). Not "scale Flink" first.
2. Alert storm after a replay: missing idempotency on transaction_id.
3. Graph UI empty: batch job failed; scoring should still work.
4. One user hot: velocity state for a bot — isolate; do not salt before velocity (you would split the count).

Salt after you no longer need a single key for state, or use a dedicated key-group for bots.

Why Redis appears and when to remove it¶

Redis is a p99 tool for hot features. It is a bad 30-day aggregate store and a bad SoR. Populate from Flink; rebuild from CH/Kafka on flush. If CH p99 is already 8 ms, skip Redis (operational cost). V1 in many shops is CH-only until the SLO misses.