Testing Strategy for Distributed Systems¶

Prerequisites: Observability Overview, Production Reliability Practices

← Observability

Why This Exists¶

The test pyramid — many unit tests, fewer integration tests, few end-to-end tests — was designed for a single deployable. In a distributed system, "integration test" stops meaning "test how my modules interact" and starts meaning "test how my service interacts with someone else's service, which they deploy independently, on their own schedule, without asking me." That change breaks the pyramid's usual assumptions in a specific way: you can't spin up every dependency's real implementation in a test environment, and a mock you wrote yourself will faithfully test against whatever you believe the dependency does — which silently diverges from what it actually does the moment either side changes.

The interview signal here isn't "do you write tests" — it's whether the testing strategy accounts for the fact that the biggest risk in a microservices system is the boundary between services, and that's exactly the boundary unit tests can't see and full end-to-end tests can't afford to cover exhaustively.

The Pyramid, Redrawn for Services¶

flowchart TB
    E2E["End-to-end<br/>(few — slow, flaky, expensive)"]
    Contract["Contract tests<br/>(the layer unit/integration alone can't cover)"]
    Integration["Integration tests<br/>(your service + real infra it owns:<br/>its DB, its cache)"]
    Unit["Unit tests<br/>(many — fast, isolated)"]
    Unit --> Integration --> Contract --> E2E

• Unit tests stay exactly as they were — fast, isolated, testing one function or class's logic. Nothing about distribution changes this layer.
• Integration tests now mean "my service plus the infrastructure I own" — my database, my cache, my message broker client — not another team's service. Spinning up a real Postgres in a test container is integration testing; spinning up another team's entire service (with its dependencies, recursively) is not integration testing, it's a fragile, slow imitation of production.
• Contract tests are the layer that doesn't exist in the classic pyramid and is the actual fix for the cross-service risk — covered in detail below.
• End-to-end tests run against a real (or close-to-real) deployed environment, exercising a full user journey across real services. Necessary for catching what nothing else can (a genuinely wrong end-to-end flow), but slow and prone to flakiness from causes that have nothing to do with the code under test (a flaky dependency, network noise, test-data pollution from another test) — which is exactly why they should be few and reserved for the highest-value journeys, not the primary safety net.

Contract Testing: The Fix for "My Mock Lied to Me"¶

The failure this solves is specific and common: Team A calls Team B's API. Team A writes tests against a mock of Team B's API, built from documentation or from reading Team B's code at the time. Team B changes their API — removes a field, changes a status code's meaning — and Team A's tests keep passing, because the mock never changed, only reality did. The tests were green and the integration was broken; nobody found out until it broke in production.

A contract test flips who's responsible for keeping the mock honest:

sequenceDiagram
    participant Consumer as Consumer (Team A)
    participant Broker as Contract broker
    participant Provider as Provider (Team B)

    Consumer->>Broker: Publish contract: "I expect GET /orders/{id}<br/>to return {id, status, total}"
    Provider->>Broker: Fetch latest contract(s) from consumers
    Provider->>Provider: Run contract as a test against<br/>MY REAL implementation
    Provider-->>Broker: Pass/fail result
    Note over Consumer,Provider: If Team B's CI fails this check,<br/>they know BEFORE deploying —<br/>not after Team A's prod breaks

The consumer (Team A) writes down what it actually expects from the provider — as a machine-checkable contract, not documentation that can silently drift. The provider (Team B) runs that contract against its real implementation as part of its own CI pipeline. If Team B's change breaks what Team A actually depends on, Team B's build fails, on Team B's side, before the breaking change ships — not weeks later when Team A's production traffic hits it. This is the consumer-driven contract testing pattern (Pact is the best-known open-source implementation), and its value is precisely that it moves the failure from "silent production incident" to "loud CI failure owned by the party that caused it."

Feature Flags as a Testing Tool, Not Just a Release Tool¶

Feature flags are usually introduced for progressive delivery (see Deployment Strategies), but they're also a testing strategy in their own right: a flag lets you deploy code to production disabled, verify it behaves correctly with real traffic patterns flowing past it (not just enabled for the user making the request), and enable it gradually — which catches a category of bug that no pre-production test environment reliably reproduces, because pre-production traffic never has production's actual shape (scale, data skew, weird edge-case inputs real users generate). The trade-off: flags accumulate, and a codebase with stale flags left in "temporarily" months ago is its own form of technical debt — a flag needs an owner and a removal date the same way a TODO does.

Testing Failure Modes Deliberately¶

Chaos engineering (see Production Reliability Practices) is the production-facing half of this; the same discipline belongs earlier, in the test environment, before the failure mode is discovered live:

• Does your service handle a dependency timing out, not just a dependency returning an error? These are different code paths, and timeout handling is the one that's usually undertested (see Distributed Systems Fundamentals on why timeouts, not clean errors, are the default failure mode of a remote call).
• Does a retry actually behave safely — is the retried operation idempotent, or does the test suite only ever exercise the happy path where nothing needed retrying? (See the idempotency pattern in Distributed Systems.)
• Fault injection in a test environment (a proxy that deliberately delays or drops a percentage of calls to a dependency) surfaces these gaps before a chaos experiment does it in production, at a fraction of the blast radius.

Realistic Example¶

A checkout service calls a separate inventory service to reserve stock. Team Checkout's integration tests use a hand-written mock of the inventory API that returns a successful reservation instantly. Team Inventory ships a breaking change — they rename sku on the reservation response (or add a required request field warehouse_id that checkout does not send). Additive response fields should not break consumers. Inventory's own tests pass; checkout starts failing in production within minutes, because the hand-written mock never saw the real contract. Postmortem action: replace the mock with a consumer-driven contract — checkout publishes exactly what fields it reads from the response and what it sends on the request; inventory's CI runs that contract against its real implementation on every change. Additive response fields stay green; removing/renaming a field checkout depends on, or adding a required request field, fails inventory's build before merge.

Trade-offs¶

Interview Questions¶

Key Takeaways¶

Back to: Observability