What Does an API Gateway Do?

An API Gateway is a core architectural pattern in system design. Instead of client applications directly calling multiple backend microservices—which increases client complexity and exposes internal network topologies—clients make all requests through a single entry point: the API Gateway. The gateway acts as a reverse proxy, orchestrating routing, security policy, traffic control, and protocol translation at the edge of the private network. It should enforce cross-cutting policy without becoming the owner of downstream business transactions.

Mental Model

Think of an API Gateway like the secure lobby receptionist of a large corporate headquarters. Instead of visitors (clients) wandering directly into private offices (microservices) to find specific staff members, they must enter through the lobby. The security guard validates their badge ID (Auth Proxy), checks if their briefcase contains any banned items (Request Validator / WAF), and limits how many visitors can enter at once (Rate Limiter). If a visitor asks for general information (Response Cache), the receptionist answers them immediately without sending them upstairs. Otherwise, they check the directory (Service Discovery) to find the correct room and direct the visitor (Route Load Balancer). If a particular floor is temporarily closed due to an emergency (failing microservice), the receptionist immediately tells visitors to turn back rather than letting them get stuck in the elevator (Circuit Breaker).

Components and Why They Exist

Client Devices

Client devices represent external users and applications sending API requests to interact with backend services. Outbound HTTPS request packets originate from the client node and traverse the gateway’s perimeter security and traffic-shaping checks before reaching internal services.

Request Validator

The request validator rejects malformed or schema-violating requests early to protect internal microservice compute resources. It inspects incoming headers, query parameters, and JSON payloads against predefined API schemas, blocking invalid payloads right at the edge.

Allow/Deny Firewall (WAF)

The Web Application Firewall (WAF) serves as a perimeter filter that blocks disallowed IP subnets and common web exploit payloads. Requests arriving from blocklisted IPs or matching known attack signatures are rejected immediately before routing proceeds.

Auth Proxy

The authentication proxy centralizes identity verification so individual microservices do not have to duplicate token-checking logic. It extracts the bearer JWT header from incoming requests, verifies the cryptographic signature against cached JWKS public keys, and checks expiration claims locally.

Rate Limiter

The rate limiter prevents service overload by protecting backend microservices from traffic bursts and abuse. Each admitted request drains a token from the caller’s token bucket counter, while requests arriving against an empty bucket are throttled with HTTP 429 Too Many Requests until tokens refill.

Response Cache

The response cache accelerates read paths for idempotent, frequently requested endpoints. Repeat GET calls that match a warm cache entry return directly from gateway memory in under 2ms, bypassing downstream microservice and database lookups entirely.

Route Load Balancer (Router)

The route load balancer distributes admitted requests across healthy microservice instances. By querying service discovery registries to resolve active backend service IPs, the router balances incoming traffic across target instances using policies like round-robin or least connections.

Circuit Breaker

The circuit breaker prevents cascading failures when a downstream microservice becomes slow or unresponsive. Once downstream error rates cross the configured threshold, the breaker trips to an OPEN state and short-circuits incoming requests to a fast fallback response instead of tying up gateway threads.

Edge Cases and Misconceptions

API Gateway vs. Load Balancer

A common misconception is that an API Gateway is simply a Load Balancer (like Nginx or HAProxy). While both act as reverse proxies, a traditional load balancer routes traffic at layer 4 (TCP) or layer 7 (HTTP) based on basic routing rules. An API Gateway is application-aware, running custom security code (JWT validation), rate-limiting algorithms, caching policies, protocol translation, and request composition.

Cascading Thread Starvation

Without a Circuit Breaker, if a downstream service (such as the database) becomes slow, the gateway’s routing threads will wait for TCP timeouts. If traffic is high, all gateway worker threads will become blocked waiting for responses. This starves the gateway of threads, causing it to reject requests for completely unrelated, healthy services.

Caching Stale Data

Gateway-level caching dramatically decreases latency, but it introduces the problem of data inconsistency. If a user updates their profile, a cached GET request from the gateway might still serve the old profile for the duration of the Cache TTL. Gateways must coordinate eviction events or utilize low TTL values for dynamic resources.

Distributed Rate Limiting Sync

In a clustered setup where multiple gateway nodes run behind a global load balancer, rate limits must be shared. If rate limit states are kept purely in-memory on each gateway node, a client could bypass their quota by distributing requests across different nodes. Production gateways sync rate limit token buckets using a centralized, high-speed cache like Redis.