Dead-letter queues

3 min read
On trails:EDARELstep 6/9

A dead-letter queue (DLQ) holds messages that could not be processed, set aside so the rest of the traffic keeps moving. The usual occupant is a poison message: one that fails every time it is tried, such as a payload that cannot be parsed or a record that breaks a rule. Without a DLQ, a poison message is either retried forever, blocking what is behind it, or dropped without a trace.

What the record should hold

Keep the original message unchanged and put the failure details beside it. A person will read this later, to decide whether to fix and replay it. It needs:

  • the original body and headers, including messageId and correlationId
  • the error: exception type, message and, where size allows, the stack trace
  • the number of attempts made
  • where it came from: queue or topic, and for a log, the partition and offset
  • when it failed, and which consumer failed it

Who provides it

BrokerBuilt in?What moves a message there
Azure Service BusYes: a sub-queue on every queue and subscriptionMax delivery count (default 10), expiry when dead-lettering on expiry is on, or the app calling DeadLetterMessageAsync with a reason and description
RabbitMQYes: a dead letter exchange, set by policy or queue argumentbasic.reject or basic.nack with requeue off, per-message TTL, a queue length limit, or the delivery-limit on a quorum queue. The x-death header records the history
Kafka consumersNoThe application publishes the failed record to a dead-letter topic itself
Kafka Connect, Kafka StreamsYes, as configurationSink connectors: errors.tolerance=all with errors.deadletterqueue.topic.name. Streams: errors.dead.letter.queue.topic.name (KIP-1034, release not confirmed here)
MassTransitYes: an _error queue per receive endpointA consumer exception after retries. Exception details go in the headers and a Fault is published. Messages with no consumer go to _skipped
Table 1: dead-lettering by broker

An ordinary Kafka consumer has no built-in dead-letter queue. The consumer catches the failure and produces the record to another topic, and it should commit the offset only after that produce succeeds. In RabbitMQ the default for classic queues is weak: the message leaves the original queue as soon as it is published to the dead letter exchange, so a failed hand-over loses it. Quorum queues offer an at-least-once dead-letter strategy for that case.

In code

A Kafka consumer writing a dead letter with Confluent.Kafka. The original headers should be copied too, which the snippet leaves out.

DeadLetter.csC#
static async Task DeadLetterAsync(    IProducer<string, string> producer,    ConsumeResult<string, string> result,    Exception error, int attempts, CancellationToken ct){    var record = new Message<string, string>    {        Key = result.Message.Key,        Value = result.Message.Value,        Headers = new Headers        {            { "dlq-error", Encoding.UTF8.GetBytes($"{error.GetType().Name}: {error.Message}") },            { "dlq-attempts", Encoding.UTF8.GetBytes(attempts.ToString()) },            { "dlq-source", Encoding.UTF8.GetBytes($"{result.Topic}/{result.Partition.Value}/{result.Offset.Value}") },        },    };     await producer.ProduceAsync("payments.dlq", record, ct);}

Looking after it

A DLQ that nobody reads is a slow data loss. Alert on its depth (see observing event flowsObserving event flowsFollowing one order through several services, and the three queue numbers that warn first.Opens in a new tab), decide who triages it, and keep a replay path for after the fix. A replay is a redelivery, so the consumer must be idempotentIdempotent consumersA consumer that gives the same result whether a message arrives once or five times.Opens in a new tab. On Service Bus nothing cleans the DLQ automatically, and messages stay until someone completes them. What a dead letter does to the messages around it is covered in dead letters and ordered messagesDead letters and ordered messagesA dead-lettered message leaves the messages after it running on state that never happened.Opens in a new tab.

Sources