gRPC vs HTTP vs HTTP/2 in Production¶

Model: One request per connection (or connection pooling)

Timeline:
Client connection 1:  [Request 1] ──→ [Response 1] ──→ CLOSE or IDLE
Client connection 2:  [Request 2] ──→ [Response 2] ──→ CLOSE or IDLE
Client connection 3:  [Request 3] ──→ [Response 3] ──→ CLOSE or IDLE

With connection pooling (realistic):
Client pool (size=10):
  Conn 1: [Req A] ──→ [Resp A] ──→ [Req D] ──→ [Resp D]  (backend-1)
  Conn 2: [Req B] ──→ [Resp B] ──→ [Req E] ──→ [Resp E]  (backend-2)
  Conn 3: [Req C] ──→ [Resp C] ──→ [Req F] ──→ [Resp F]  (backend-3)
  Conn 4-10: IDLE (waiting for work)

Distribution: If each connection is round-robin'd to a backend,
              then Req A,D → backend-1, Req B,E → backend-2, etc.
              Traffic distributed across backends.

Model: One connection, many concurrent streams

Timeline:
Single client connection:
  ├─ Stream 1: [Request A] ──→ [Response A]
  ├─ Stream 2: [Request B] ──→ [Response B]  (concurrent!)
  ├─ Stream 3: [Request C] ──→ [Response C]
  └─ Stream N: [Request N] ──→ [Response N]

All streams multiplexed on one TCP connection.

Distribution: All streams on one connection → routed to ONE backend
              → That backend handles all traffic from this client
              → UNEVEN DISTRIBUTION (hotspot)

gRPC over HTTP/2:
  Client creates ONE connection (or small pool)
  Each RPC call = new stream on that connection
  → All calls to same backend → hotspot

Example (Python gRPC):
import grpc

# One channel per target
channel = grpc.aio.secure_channel("payment:50051", ...)
stub = PaymentServiceStub(channel)

for i in range(1000):
    response = await stub.Charge(ChargeRequest(...))
    # All 1000 calls on ONE connection to ONE backend

Client code:
  hosts = ["backend-1:50051", "backend-2:50051", "backend-3:50051"]
  for host in hosts:
    channel = grpc.Channel(host)
    channels.append(channel)  # Pool of channels

  for i in range(1000):
    channel = channels[i % len(channels)]  # Round-robin
    response = stub.Call(channel)

Result: Traffic distributed round-robin across backends
        Each backend gets ~333 requests

Service: payment (ClusterIP 10.0.0.10)
  Endpoints: 10.0.1.5:50051 (pod-1)
            10.0.1.6:50051 (pod-2)
            10.0.1.7:50051 (pod-3)

DNS resolution (payment:50051 → ???):
  Every query returns the SAME answer: 10.0.0.10 (the Service ClusterIP)
  There is no "round-robin per query" at the DNS layer for ClusterIP —
  pod selection happens LATER, at the packet/connection level, via
  kube-proxy's iptables/IPVS rules (see below) — not via DNS.

Query 1: payment → 10.0.0.10 → client connects to 10.0.0.10:50051
Query 2: payment → 10.0.0.10 → same answer, always
Query 3: payment → 10.0.0.10 → same answer, always

The client's ONE TCP connection to 10.0.0.10 gets NAT'd by kube-proxy
to exactly one pod IP at connection-setup time (the SYN packet) —
and then every packet on that connection stays pinned to that pod
for the connection's lifetime. That's where the hotspot actually
comes from: not DNS variance, but a single long-lived gRPC connection
getting NAT'd to one pod once and reused forever.

Client pod:
  ├─ Application container
  └─ Envoy sidecar (intercepts all traffic)
     └─ Knows about all endpoints (via control plane)
     └─ Load-balances across endpoints
     └─ Health checks failing endpoints
     └─ Retries on failure

Traffic flow:
  App → localhost:50051 (local Envoy)
     → Envoy round-robins to backend pod Envoys
     → 10.0.1.5, 10.0.1.6, 10.0.1.7 equally

Result: Even distribution
        Each backend gets ~333 requests

Service: payment (ClusterIP 10.0.0.10)

iptables rules:
  -A KUBE-SERVICES -d 10.0.0.10/32 -p tcp -m tcp --dport 50051
    -j KUBE-SVC-XYZ123

  -A KUBE-SVC-XYZ123
    -m statistic --mode random --probability 0.33
      -j KUBE-SEP-POD1
    -m statistic --mode random --probability 0.50
      -j KUBE-SEP-POD2
    -j KUBE-SEP-POD3

  -A KUBE-SEP-POD1 -j DNAT --to-destination 10.0.1.5:50051
  -A KUBE-SEP-POD2 -j DNAT --to-destination 10.0.1.6:50051
  -A KUBE-SEP-POD3 -j DNAT --to-destination 10.0.1.7:50051

gRPC client creates 1 channel (1 TCP connection)
That connection established to random backend at SYN time
All 1000 gRPC calls on that connection → same backend
Result: UNEVEN if client created connection early (high traffic on one backend)
        EVEN only if connections are short-lived (unlikely for gRPC)

IPVS (IP Virtual Server):
  More intelligent than iptables
  Per-connection or per-packet load balancing available
  Sticky session support

Configuration:
  ipvs.k8s.io/lb-algorithm: rr  (round-robin, per-connection)
                           wrr  (weighted round-robin)
                           lc   (least connections)
                           sh   (source hash)

Setup:
  3 backend pods
  10 client pods, each talking to a plain ClusterIP Service "payment"
  Each client creates 1 gRPC channel (1 TCP connection)

Expected: 10 * N calls / 3 backends = 3.33N calls per backend
Actual: could be anywhere from perfectly even to wildly skewed —
        it depends entirely on kube-proxy's per-CONNECTION selection,
        not on DNS (every client resolves the exact same ClusterIP).

Why:
1. All 10 clients resolve "payment" to the SAME ClusterIP (10.0.0.10) —
   there is no DNS-level variance to reason about here at all.
2. Each client opens ONE TCP connection to 10.0.0.10. kube-proxy's
   iptables rules pick a backend pod for that connection at SYN time,
   using weighted random selection (see the iptables section below) —
   this is where the actual randomness comes from.
3. With only 10 connections spread randomly across 3 backends,
   small-sample variance is entirely plausible without anything being
   "wrong" — 10 independent coin-weighted picks across 3 buckets does
   not reliably land near 3.33/3.33/3.33.

If each client sends 1000 calls on its one connection:
   A plausible unlucky draw: pod-1 gets 4 connections, pod-2 gets 3,
   pod-3 gets 3 → pod-1: 4000 calls, pod-2: 3000, pod-3: 3000
   (1.33x imbalance — small-sample variance in kube-proxy's random
   pick, not a DNS effect)

The lesson is the same either way: gRPC's one-connection-per-channel
model means the unit of load-balancing granularity is "which pod did
THIS connection land on," not "which pod did this request land on" —
and with few, long-lived connections, random per-connection selection
doesn't average out the way per-request selection would.

Setup:
  1 client pod (makes many requests)
  3 backend pods
  Client uses single gRPC channel (1 connection)

Timeline:
1. Client pod starts
2. DNS query for payment → resolves to the ClusterIP (same answer every
   time — no per-query variance; this is not where the randomness is)
3. Client creates single channel → TCP connects to the ClusterIP;
   kube-proxy's iptables rule picks ONE backend pod at SYN time
   (random selection happens here, not at DNS)
4. Client sends 100k requests → all multiplexed as HTTP/2 streams on
   that one already-established connection → all land on that one pod

Result:
   pod-1 (whichever pod kube-proxy happened to NAT the SYN to): 100k requests
   pod-2: 0 requests
   pod-3: 0 requests

Why:
   - gRPC connection pooling (single channel, reused for every RPC)
   - kube-proxy makes its pod selection ONCE, at connection setup, and
     that choice is then fixed for the connection's entire lifetime
   - No failover (pod-1 responsive, so connection stays)

Setup:
  10 clients, each makes requests
  3 backends

Scenario 1 (even clients):
  Each client's connection lands on a backend via kube-proxy's random
  per-connection pick (not DNS — DNS gives every client the same
  ClusterIP)
  Each client makes steady requests
  → Roughly even distribution IF the sample of connections is large
    enough for kube-proxy's randomness to average out, and IF load
    per client is balanced

Scenario 2 (bursty traffic):
  Client A starts (connects to backend-1)
  Client B starts (connects to backend-2)
  Client C starts (connects to backend-3)
  Clients D-J start (connect to backends 1-3)

  Traffic suddenly: All 10 clients make requests simultaneously
  Distributed across 3 backends (10/3 ≈ 3.33 clients per backend)

  BUT: If backend-1 is slower (GC pause, CPU spike):
    Requests queue up on backend-1
    Latency increases
    Clients retransmit
    More requests pile up
    Cascading failure

Result: Backend-1 becomes hotspot due to performance variance

Setup:
  Backend service with gRPC connection pool
  Database query (20 ms latency)

Scenario:
1. 100 clients connect (100 connections to backend)
2. Each client makes 100 requests per second
3. Each request → database call (20 ms)
4. Backend goroutines: 1 per request = 100 * 100 = 10,000 goroutines
5. Each goroutine blocked on database call (20 ms) = 20ms wait

Normal case:
   Throughput: 100 * 1000 / 0.020 = 5,000,000 requests/sec (ideal)
   Actual: ~5,000 requests/sec (100 requests/sec/connection × 100 connections)

If connection limit is hit (OS default ~1024):
   New connections rejected (ECONNREFUSED)
   Clients see errors
   Retries overwhelm system

Setup:
  10 clients, 1 backend connection each (gRPC)
  Backend A: handling all 10 clients
  Backend B, C: idle

Failure:
  Backend A: GC pause (500 ms)
  → All 10 clients' requests timeout or queue
  → Clients retry
  → More requests pile up
  → Backend A CPU spikes
  → Cascading failure

Recovery depends on:
  ✓ Timeout on client (requests fail fast, retry to different backend)
  ✗ No timeout (requests wait, no retry, cascade)

Better: Multiple backend connections per client
        If one connection slow, other connections still work

Setup:
  Client makes N requests per day
  Each request creates gRPC channel (connection)
  Channels never closed

Day 1: 1000 connections open
Day 2: 2000 connections open
Day 3: 3000 connections open
...
Day 10: 10,000 connections open → OS limit reached → ECONNREFUSED

Production impact:
  Gradual degradation
  No obvious failure
  Only noticed after weeks

Setup:
  Backend pod created at 9:00 AM (IP 10.0.1.5)
  DNS TTL: 30 seconds
  Client cached DNS result at 9:00:01 AM

9:00:31 AM:
  Backend pod killed (IP 10.0.1.5 no longer exists)
  New pod created (IP 10.0.1.99)
  Service updated to point to new pod

Client still holds connection to 10.0.1.5 (stale)
Requests timeout (connection reset)
After timeout, client retries
DNS query now returns 10.0.1.99
New connection to new pod
Requests succeed

Duration of outage: Request timeout (default 30s) before retry

Setup:
  AWS ALB with target group, HTTP/2 / gRPC listener configured
  Health checks per backend

ALB routing — this is the one place in this page where the
"connection-pinned" assumption does NOT automatically hold:
  ALB terminates the client's HTTP/2 connection itself, then parses
  individual gRPC calls (it understands HTTP/2 framing and can
  distinguish separate streams within one connection) and can forward
  each one to a different backend target — i.e. ALB can load-balance
  PER REQUEST, not just per connection, when configured for gRPC.
  This is a genuinely different capability from a plain L4 NLB or from
  kube-proxy's iptables NAT, both of which only see opaque TCP streams
  and must pin at the connection level because they have no visibility
  into what's inside it.

Result: Even distribution per-request IS achievable with ALB + gRPC
        target group configuration — this is not automatically true
        out of the box, but it's a real, supported ALB feature, unlike
        NLB/kube-proxy where per-request balancing isn't possible at all.
        Automatic failover (unhealthy backend removed)

Naive (wrong for gRPC) formula: pool_size = target_rps × per_request_latency × headroom

Example:
  Target RPS: 10,000
  Per-request latency: 50 ms
  Headroom: 2

  "pool_size" = 10,000 × 0.050 × 2 = 1,000
  This is the number of CONCURRENT IN-FLIGHT RPCs you need capacity for
  — NOT the number of TCP connections. A single HTTP/2 connection can
  multiplex hundreds to low-thousands of concurrent streams (bounded by
  MAX_CONCURRENT_STREAMS, commonly 100–1000 depending on server config)
  before you need a second connection at all.

Real sizing question: how many backend pods should share this load?
  If MAX_CONCURRENT_STREAMS per connection = 100, and you need 1,000
  concurrent in-flight RPCs of headroom, you need at least
  1,000 / 100 = 10 connections to avoid stream-limit backpressure —
  and separately, at least that many (or a multiple, for even spread)
  distinct backend pods to avoid concentrating all 10 connections on
  too few pods.

Implementation:
  Option 1: One channel per backend pod you want in the spread (via a
            headless Service + client-side round-robin resolver)
  Option 2: A service mesh sidecar, which handles per-request spread
            without the app managing channel count at all

# Measure traffic per backend — grpc_server_handled_total is a COUNTER,
# not a histogram, so histogram_quantile doesn't apply here at all;
# histogram_quantile only makes sense against a _bucket metric emitted
# by a histogram (e.g. grpc_server_handling_seconds_bucket).

# Traffic volume per backend — just the request rate, no quantile:
sum(rate(grpc_server_handled_total[5m])) by (backend)

# Expected: ~equal (within 10% variance)
# Bad: One backend 3-4x higher than others

# If you actually want LATENCY per backend (a real use for
# histogram_quantile), use the histogram's _bucket series instead:
histogram_quantile(0.99,
  sum(rate(grpc_server_handling_seconds_bucket[5m])) by (le, backend)
)

# Check why traffic is uneven:
#   1. kube-proxy's per-connection pod selection (not DNS — see Part 2)
#   2. Connection reuse (client creating a single long-lived connection)
#   3. Backend performance (one backend slower, connections queue)

# grpc_server_started_total - grpc_server_handled_total gives you
# IN-FLIGHT RPCs (started but not yet finished) — a proxy for load,
# but NOT the same thing as active TCP connections. One connection can
# have hundreds of in-flight streams; this metric can't tell you how
# many distinct connections are open.

sum(grpc_server_started_total - grpc_server_handled_total) by (instance)

# Expected: stable (within normal variance), tracks concurrent RPC load
# Bad: Continuously increasing (RPCs not completing — hung streams,
#      deadlocked handlers, or a genuine backlog building)
# Bad: Sudden spikes (thundering herd / cascade)

# To actually count active TCP CONNECTIONS, you need a connection-level
# metric, not an RPC-level one — e.g. Envoy's
# envoy_cluster_upstream_cx_active, or OS-level `ss -s` / conntrack
# counts against the target pods.

**Q: What's the difference between HTTP/2 and HTTP/1.1 in terms of connection pooling?**

"HTTP/1.1 creates a pool of connections (size N), each connection handles one request at a time, connections are reused. HTTP/2 creates one connection and multiplexes many concurrent requests as streams on that connection. Result: HTTP/1.1 needs larger pool (more memory, more TCP overhead) but distributes traffic across backends naturally. HTTP/2 uses minimal connections but all traffic on one connection goes to one backend (hotspot)."