Cascading Timeout & Latency Injection

Mental Model

Cascading timeout propagation occurs when an upstream service blocks waiting for a slow downstream dependency. In distributed microservice architectures, this blocking behavior can deplete thread pools across multiple service boundaries, propagating a localized failure all the way back to the API gateway and clients.

The core mental model is a side-by-side comparison of two identical service chains: a Vulnerable Lane (with no circuit breaker protection) and a Protected Lane (with automatic circuit breaker protection). Injecting latency or errors into the deep downstream dependency shows how thread pool depletion spreads upstream, how retries amplify traffic under failure (retry storms), and how circuit breakers dynamically open, fail-fast, and safely probe for recovery.

Components and Why They Exist

Browser Clients

The browser clients represent end users or upstream callers generating requests. They dispatch requests at a configurable rate (RPS) and track metrics like success rate and p95 latency. Comparing the client metrics across the two lanes highlights how fast-fail protection preserves p95 latency and availability under fault injection.

Gateway

The gateway serves as the single entry point for all client requests in each microservice lane. It routes incoming requests to Service A while operating within a finite worker thread pool. When Service A slows down in the Vulnerable lane, the Gateway thread pool saturates completely and begins dropping incoming requests.

Service A

Service A is an intermediate upstream microservice in the call chain. It handles traffic from the Gateway and makes downstream network calls to Service B; in the Resilient lane, it wraps these calls with a circuit breaker. The Resilient lane’s Service A card displays the circuit breaker state badge (CLOSED, OPEN, HALF-OPEN) alongside its preserved worker thread capacity.

Service B

Service B is the downstream dependency that interacts directly with the Database. It processes requests from Service A and executes queries against the backing store. Fault-injection controls apply latency spikes (such as 2500ms) or elevated error rates directly to the Service B node.

Database

The database is the persistent storage layer queried by Service B. It executes SQL queries and data fetches for incoming service requests. During a retry storm, the dbRps counter on the Database card surges in the Vulnerable lane while staying bounded in the Resilient lane.

Edge Cases and Misconceptions

• Upstream Thread Pool Exhaustion: When a downstream service slows down, every upstream server calling it must hold its connection open. In thread-per-request architectures, this holds worker threads hostage. Once all threads are used, the server cannot accept any new traffic, propagating the outage upstream.
• Retry Storm Amplification: Retries are designed to handle transient network hiccups. However, if a downstream dependency suffers a persistent outage or overload, retries amplify the failure. If every client retries 3 times, traffic volume increases by up to 300%, turning a minor slowdown into a complete outage.
• Circuit Breaker Flapping: A circuit breaker that continually trips open, enters half-open, and immediately trips open again is “flapping.” This indicates that the downstream service has not recovered, or the probe traffic is too high. Production systems use exponential backoff and jitter for half-open probes to avoid overloading recovering nodes.
• Timeouts Alone Do Not Prevent Exhaustion: Setting a 1-second timeout does not protect thread pools under heavy traffic. If requests arrive faster than 1 per second per thread, the threads will still saturate and exhaust the pool. Circuit breakers are required to fail fast immediately.