Pastebin System Design

Mental Model

A pastebin service allows users to store plain text (typically code snippets) and share them via a short URL. Designing a scale-out pastebin requires segregating raw document storage from search metadata, pre-allocating short-URL keys to avoid write-time locks, and accelerating read paths using CDN caches.

For interactive exploration of these behaviors, you can run the live visualization: Pastebin System Design Walkthrough.

Components and Why They Exist

Client Browser

The client browser represents end users who either compose new pastes (writes) or retrieve saved pastes (reads). Outbound HTTP requests flow from the client browser toward the CDN edge or API gateway depending on whether the operation is a read or write.

CDN Edge Cache

The CDN edge cache reduces read latency and limits origin server load by caching popular pastes at the network edge. Incoming read requests hit this edge node first, returning a cached payload immediately on a cache hit or forwarding to the load balancer on a cache miss.

Load Balancer & API GW

The load balancer and API gateway tier distributes traffic evenly and protects backend services from abuse. Sliding-window rate limiting runs at this gateway, returning 429 Too Many Requests directly to the client browser whenever a caller exceeds its quota.

API Application Server

The API application server executes the core paste creation and retrieval logic in a stateless manner. On writes, the server claims a unique key from the Key Generation Service, uploads raw paste text bodies to S3, commits search metadata to the SQL database, and invalidates stale CDN cache entries.

Key Generation Service (KGS)

The Key Generation Service (KGS) prevents database locks and key collision checks at write time. It pre-generates unique Base62 strings offline, buffers them in memory, and hands over pre-allocated keys to the API server in $O(1)$ time.

Amazon S3 Object Store

The Amazon S3 object store provides horizontally scalable blob storage for large raw text files, keeping relational database indexes compact. Raw paste payloads upload directly into S3 buckets as immutable text objects addressed by their short key.

Metadata DB (SQL)

The SQL metadata database serves as the indexed relational store mapping short paste keys to their metadata. Write operations insert compact rows containing the short URL key, S3 object path, expiration timestamp, and author ID, while read misses execute fast indexed lookups.

Expiration Cleaner

The expiration cleaner garbage-collects expired paste data in the background to free database index pages and S3 storage. Running asynchronously on a schedule, it queries the SQL database for expired timestamps, issues bulk SQL row deletions, and purges the corresponding S3 blobs.

Edge Cases and Misconceptions

Key Exhaustion and Recovery

A common misconception is that the Key Generation Service (KGS) never runs out of keys. If KGS crashes or its buffer runs dry due to a massive spike in write traffic, the API Application Server must fall back to dynamic key generation. Generating keys dynamically requires expensive database unique-key checks, leading to high write latencies and write lock escalations.

CDN Cache Invalidation vs. Natural Expiration

When a paste is updated or deleted, it is crucial to invalidate the CDN edge cache immediately. If the CDN cache is not invalidated, users will continue to retrieve the expired or old paste version from local edge caches until the Cache TTL expires naturally, leading to cache inconsistency.

Soft Delete vs. Hard Delete in Garbage Collection

Simply removing a metadata row from the SQL DB does not immediately free space on S3. Production systems mark the paste as inactive (soft delete) in SQL first. The Expiration Cleaner daemon subsequently processes this queue in batches, executing hard deletes on the Metadata DB and S3 asynchronously to avoid overloading database transaction logs.