Load Balancer Crash Simulator

Load balancers are critical components in modern architecture, serving as the single point of entry for user traffic.

Mental Model

When a load balancer crashes, the entire system is at risk of an outage unless resilient architectures are in place. In a High Availability (HA) active-passive setup, the standby load balancer monitors the active primary node via heartbeat keepalives. If heartbeats stop, the standby node promotes itself to active, taking over the virtual IP. Clients query a DNS server to resolve the load balancer’s IP; if the IP changes during failover, clients may continue targeting the dead node due to stale DNS caches (ignoring DNS TTLs). The load balancer routes traffic to a pool of backend servers, while a health checker monitors backend responsiveness, dynamically adding or removing nodes to prevent routing traffic to broken application instances.

Components and Why They Exist

Clients

Clients are the source of the request traffic consuming the application. Each client sends HTTP requests, caches DNS results locally, and handles connection failures by retrying or surfacing error counters on the client panel when packets hit a dead ingress IP.

Active Load Balancer (LB A / LB B)

The active load balancer is the primary entry point responsible for distributing traffic across healthy backend servers. It receives incoming client requests and routes particles to healthy backends while displaying an active status badge on its node card.

Standby Load Balancer (LB A / LB B)

The standby load balancer provides failover redundancy to prevent a single point of failure at the ingress layer. It monitors the primary node via heartbeat pings and promotes its own status badge from standby to active when the primary load balancer crashes.

DNS Server / Cache

The DNS server translates user-facing hostnames into the IP address of the active load balancer. It serves IP records with a configured Time-to-Live (TTL) and updates mappings during failover, though clients continue using cached IPs until their local TTL counter counts down.

Backend Nodes / Servers

The backend servers form the application pool that executes request logic and queries databases. Each server accepts connections from the load balancer, queues incoming request dots, and lights up red when queue depth and CPU utilization cross the overload threshold.

Health Checker

The health checker ensures the load balancer maintains an accurate view of backend node viability. It sends periodic probes to backend endpoints, flipping health status badges to isolate unresponsive servers and slowly ramping traffic back as recovered nodes warm up.

Edge Cases and Misconceptions

• Misconception: High Availability (HA) failover is instantaneous and zero-loss: HA failover always introduces a small detection window (e.g., keepalive timeout) where the standby node determines the primary is dead. During this transition, client requests will fail.
• Misconception: Reducing DNS TTL to 0 solves client cache issues: Many clients (browsers, JVM runtimes, proxy servers) ignore DNS TTL settings or enforce a minimum cache duration (e.g., 60 seconds). Even if DNS is updated, traffic will still flow to the dead node’s IP.
• Edge Case: Fail-Open Routing vs. Cascading Failures: When health checks mark all backends unhealthy (e.g., due to a temporary network partition between the LB and backends), a “fail-open” policy routes traffic anyway to prevent a total outage. If the backends were actually overloaded, failing open might permanently crash them.