URL Shortener System Design
Walk through a URL shortener from long URL submission and Base62 code generation through collision checks, sharded storage, redirect resolution, browser and Redis caching, shard failure
Continue with the guided workspace
Use the paired workspace to connect this guide's mental model with live component state, visible signals, and the outcome each step or scenario is meant to prove.
A URL shortener turns a long link into a short code, stores the mapping, and redirects anyone who hits the short code back to the original URL. The design tension is between the write path (generating unique codes at high throughput) and the read path (resolving redirects in milliseconds under massive traffic), solved with Base62 encoding, sharding, and cache-aside Redis.
For parameter experimentation, the URL Shortener Simulator is also available.
Mental Model
Think of a URL shortener like a coat check at a theater. The write path is the drop-off: you hand over a large coat (long URL), the clerk prints a tiny ticket index (Base62 code), and hangs the coat on a specific numbered rack (database shard) after verifying the ticket number isn’t already taken (collision check). The read path is the pick-up: you show your ticket (short URL) and get your coat immediately. To save steps, popular coats are kept on a hanger right next to the counter (Redis cache), or you might even keep a permanent duplicate at home (browser cache) so you don’t have to visit the theater at all.
Components and Why They Exist
Client
The client browser submits long links to be shortened and requests short URLs when users click links. It sends POST payloads for URL creation and follows HTTP 301 or 302 redirect headers when resolving a shortened link.
API Gateway
The API gateway regulates incoming traffic and enforces edge routing and rate-limiting rules. It validates input parameters, checks per-caller rate limits, and routes writes to the Write Service while directing read lookups to the Redirect Service.
Write Service
The Write Service orchestrates short URL creation, collision verification, and database persistence. On every link-creation request, it invokes the ID generator or hash engine, coordinates collision verification checks, and commits the validated mapping to the target database shard.
ID Generator / Hash Engine
The ID generator or hash engine produces compact alphanumeric keys to represent long destination URLs. It either encodes sequential Snowflake-style numeric IDs into Base62 strings (such as gXyZ7) or extracts a prefix from an MD5/MurmurHash digest.
Collision Checker
The collision checker ensures no two distinct long URLs are assigned the same short code when hash truncation is used. By querying storage before committing, it flags any candidate hash collision and triggers a salted retry until a unique code is verified.
Shard Router
The shard router determines which database shard owns each short URL so storage and I/O scale horizontally. Hashing the short code selects the destination database shard deterministically for both writes and cache-miss reads.
URL DB Shards
The URL database shards provide horizontally partitioned persistent storage for billions of URL mappings. Each shard node stores (short_code, long_url) rows for its assigned hash range and isolates the blast radius if an individual shard goes offline.
Redirect Service
The Redirect Service resolves short codes back to their original destination URLs on the read path. It checks the Redis cache first, falls back to the appropriate database shard on a cache miss, and returns an HTTP 301 Moved Permanently (or 302) redirect response carrying the target long URL.
Redis Cache
The Redis cache accelerates lookups for popular short URLs and shields the database shards from read spikes. Hot (short_code -> long_url) mappings stay in memory so read lookups hit the Redis cache and bypass database shards completely.
Browser Cache
The browser cache eliminates repeat network trips by caching HTTP 301 permanent redirects locally on the client device. Once a client caches a 301 destination URL, subsequent clicks on the same short link resolve locally in the browser and bypass the server network entirely.
Edge Cases and Misconceptions
- Collisions in Hashing: If using MD5/MurmurHash and taking the first 6 characters, collisions will eventually occur. The system must query the database to verify if the code exists. If it does and maps to a different long URL, the system appends a salt (random suffix) to the long URL and hashes again.
- 301 vs. 302 Analytics Impact: While a cacheable HTTP 301 can reduce server traffic, it can hide later user behavior from the shortener. Advertisers or analytics-heavy services prefer HTTP 302 (Temporary Redirect) because it forces the browser to request the server every time, enabling full logging of click analytics (IPs, referrers, and user agents).
- ID Generation Coordination: Using a central database for auto-incrementing IDs creates a single point of failure. Autocomplete and URL systems use distributed ID generators (like Twitter Snowflake) to generate non-overlapping unique IDs lock-free.
Common Questions
Why include both browser cache and Redis cache?
Browser 301 cache can bypass the service entirely for repeated clicks, while Redis cache accelerates service-side lookups before falling back to the database.
Why does collision handling matter?
Short code collisions can point users to the wrong destination unless every generated code is checked and reserved before storage.
What is the difference between Base62 encoding and hashing for URL shortening?
Base62 encoding converts a unique auto-incrementing ID (from a generator like Snowflake) into a 6-to-8 character alphanumeric string, ensuring no collisions. Hashing (like MD5 or MurmurHash) generates a hash of the long URL and takes the first few characters. Hashing can cause collisions where two different URLs produce the same short code, requiring collision detection and resolution.
Why is 301 Permanent Redirect preferred over 302 Temporary Redirect, and what is the trade-off?
A 301 redirect is cacheable by the client and intermediaries according to cache rules; browsers may retain it for a long time. Subsequent clicks on the short URL bypass the shortener server entirely, reducing server load. The trade-off is that a cached redirect can bypass the service for later clicks, reducing server-side analytics for those requests. A 302 redirect forces the client to query the server every time, allowing 100% accurate analytics but increasing server load.
What happens during a database shard failure?
When a storage shard goes down, only the short URLs routed to that specific shard become unavailable. The rest of the system continues to write and resolve URLs normally, demonstrating the fault tolerance of sharding.
Why is Base62 used instead of Base64?
Base62 uses [A-Z, a-z, 0-9], which are all alphanumeric characters. Base64 adds + and /, which have special meanings in URLs (e.g., / is a path separator and + represents a space). Using Base62 ensures that short codes can be safely appended to URL paths without URL encoding.
What is cache-aside caching in URL shorteners?
When resolving a short code, the Redirect Service first queries Redis. If there is a cache hit, the URL is returned. If there is a cache miss, the service reads the mapping from the database, writes it to Redis (warming the cache), and returns it, accelerating future requests.