Message Queue Patterns

Message queues are foundational components in distributed systems, enabling asynchronous communication, decoupling services, and absorbing traffic spikes.

Mental Model

By separating message production from consumption, queues allow applications to remain responsive and scale independently. Producers publish messages to a central Broker. In point-to-point (Lists), messages are consumed by a single competing worker via destructive pop (BRPOP). In publish-subscribe (Pub/Sub), messages are fanned out in real-time to all active subscribers; this channel is ephemeral, so offline subscribers do not receive old messages unless a durable stream is used. In streaming logs (Streams), messages are appended to a persistent log and can be read by Consumer Groups, where each message is delivered to a single group consumer. Streams delivery records a pending reference in the Pending Entries List (PEL) while the stream entry remains, until explicitly acknowledged with XACK (which removes the handled message from the PEL to prevent reclaims). Because Redis Streams has no native partitions, ordered IDs are sequenced chronologically in a single log, but concurrent consumer processing and completion can occur out of order as XREADGROUP ... > delivers newer entries while earlier ones remain pending in the PEL. Persistently failing poison messages are quarantined via an application-managed policy: after reaching 3 failed delivery attempts (observed via PEL delivery counts), the worker explicitly writes to XADD stream:orders:dlq and clears the original PEL entry with XACK. Because at-least-once delivery can redeliver unacknowledged messages after crashes or timeouts, consumers must be designed to safely handle potential duplicate processing.

Components and Why They Exist

Producers

Producers are the client applications that generate and send event payloads into the messaging pipeline. Each producer serializes message data and publishes request envelopes toward the broker via LPUSH, PUBLISH, or XADD without waiting for downstream workers to finish processing.

Message Broker (Redis)

The Redis message broker serves as the central middleware managing message routing and storage buffers. Depending on the active pattern, it routes and stores messages in lists (LPUSH/BRPOP), ephemeral channels (PUBLISH), or append-only stream logs (XADD), while tracking queue depth and the Pending Entries List (PEL).

Consumers

Consumers are the worker applications that retrieve and process message records asynchronously. Depending on the delivery mode, each consumer polls a list, receives broadcast copies over pub/sub, or reads assigned stream records concurrently within a consumer group to execute business logic.

Consumer Groups

Consumer groups enable parallel message processing of a single Redis Stream across multiple worker nodes. A consumer group divides stream consumption among registered group consumers without native partitions. Monotonically ordered stream IDs are distributed across group members via XREADGROUP ... >, which can deliver newer entries while earlier entries remain pending in the PEL, meaning consumer processing and completion can occur out of order and workers must remain idempotent.

Pending List (PEL) / Acknowledgment (ACK)

The Pending Entries List (PEL) tracks in-flight messages delivered to consumers within a group. Streams delivery records a pending reference in the PEL while the stream entry remains in the log, holding unacknowledged messages in the pending state until a worker finishes. Calling XACK removes a successfully handled message from the PEL so it is not reclaimed by peers; if a message goes unacknowledged, it stays in the PEL until claimed via XCLAIM or XAUTOCLAIM for redelivery.

Dead Letter Queue (DLQ)

The Dead Letter Queue (DLQ) isolates poison messages that cause repeated consumer processing failures. Redis Streams does not natively count retries or perform automated DLQ routing; instead, an application-managed policy inspects PEL delivery counters via XPENDING or XAUTOCLAIM/XCLAIM. When a message reaches 3 failed delivery attempts, the consumer application explicitly redirects the malformed payload to a dead-letter stream using XADD stream:orders:dlq and issues XACK on the source stream to clear the stalled entry from the PEL.

Edge Cases and Misconceptions

• Misconception: Redis Streams have native topic partitions like Kafka: Redis Streams have no native partitions. A stream is a single, append-only chronological log of monotonically increasing IDs (timestamp-sequence). Consumer groups divide message distribution among workers reading from that single log. Because workers process concurrently and at varying speeds, message completion can occur out of order, and XREADGROUP ... > delivers newer unread entries even while older entries remain unacknowledged in the PEL.
• Misconception: Redis Streams automatically manages retry limits and dead letter routing: Redis Streams does not automatically count retries or send failed messages to a Dead Letter Queue (DLQ). The broker only maintains the delivery counter inside the Pending Entries List (PEL) metadata. Enforcing a retry threshold (such as 3 failed delivery attempts), redirecting poison messages to a secondary stream via XADD stream:orders:dlq, and removing the failed message from the primary stream’s PEL via XACK is entirely an application-managed policy implemented in worker code.
• Misconception: Destructive list pop guarantees reliable delivery: Redis Lists (LPUSH/BRPOP) remove the message immediately upon retrieval. If the consumer crashes before processing completes, the message is lost permanently, creating an at-most-once loss risk. Reliable at-least-once delivery requires Redis Streams with consumer groups and explicit acknowledgment (XACK).
• Misconception: XACK guarantees delivery: XACK does not guarantee delivery; it only confirms that a message was successfully processed and tells Redis to remove it from the Pending Entries List (PEL). If a worker crashes before sending XACK, the message remains in the PEL and will be reclaimed for redelivery, which can cause duplicate processing.
• Misconception: Consumer groups prevent duplicate processing: Consumer groups deliver each new message to one consumer under normal conditions, but failures or timeouts trigger reclaims from the PEL, meaning at-least-once delivery can cause duplicate processing if a consumer fails after processing but before XACK.
• Misconception: Pub/Sub is a durable queuing mechanism: Redis Pub/Sub is fire-and-forget. If a subscriber is offline when a message is published, the message is permanently lost. Durability requires a persistent queue (like Redis Lists or Redis Streams).
• Misconception: Point-to-point queues automatically fan-out messages: Point-to-point (using LPUSH/BRPOP) operates on a competing consumers model where only one consumer grabs and processes a message. Fan-out behavior is achieved using Pub/Sub or Streams with separate consumer groups.
• Edge Case: Unacknowledged Message Leaks (OOM): In Redis Streams, delivered messages record pending references in the Pending Entries List (PEL) while stream entries remain. Until they are explicitly acknowledged with XACK, the PEL entries persist. If consumers crash or fail to acknowledge processed messages, the PEL will grow indefinitely, consuming Redis memory and potentially causing an Out-Of-Memory (OOM) crash.