Lab: Retry Storm¶
Pairs with: Circuit Breakers
Reproduces that page's opening scenario for real: Toxiproxy injects a downstream timeout in front of a trivial, otherwise-healthy backend that logs every request it actually receives. Measure retry amplification directly — 10 client requests become roughly 40 real backend hits, not an assumed number.
docker-compose.yml¶
name: labs-retry-storm
services:
backend:
image: python:3.12-slim
command: ["python", "/app/backend.py"]
volumes:
- ./backend.py:/app/backend.py:ro
healthcheck:
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 3s
timeout: 3s
retries: 20
toxiproxy:
image: ghcr.io/shopify/toxiproxy:2.9.0
depends_on:
backend:
condition: service_healthy
ports:
- "8474:8474" # toxiproxy control API
- "20001:20001" # proxied listener -> backend:8000
command: ["-host=0.0.0.0"]
backend.py¶
The backend — logs every request it actually receives (excluding its own /health endpoint, so container healthchecks don't contaminate the measurement):
"""Trivial backend: logs every request it actually receives, responds
instantly. Stands in for "Fraud" from circuit-breakers.md's Why This
Exists section — a dependency that is otherwise healthy and fast, whose
only problem is that responses aren't reaching the caller in time
(simulated by Toxiproxy, not by this server)."""
import http.server
import sys
class Handler(http.server.BaseHTTPRequestHandler):
def do_GET(self):
# /health is the container healthcheck's path — deliberately not
# logged, so it never contaminates the request counts the retry-storm
# exercises measure via `docker logs | grep -c`.
if self.path == "/health":
self.send_response(200)
self.end_headers()
return
print(f"backend received request from {self.client_address}", flush=True)
self.send_response(200)
self.end_headers()
self.wfile.write(b"ok")
def log_message(self, fmt, *args):
pass
if __name__ == "__main__":
http.server.HTTPServer(("0.0.0.0", 8000), Handler).serve_forever()
Exercises¶
The full walkthrough (healthy baseline, inject the fault, measure amplification, recover) lives in the lab's README: