Delivery guarantees

A delivery guarantee states what can happen to a message between the broker accepting it and a consumer finishing with it: it can be lost, it can arrive more than once, or neither. There are three named levels.

Guarantee	Can be lost	Can repeat	How a consumer gets it
At-most-once	Yes	No	Settle before processing: Service Bus ReceiveAndDelete, RabbitMQ without acknowledgements, a Kafka offset committed before the work
At-least-once	No	Yes	Settle after processing: Service Bus PeekLock then Complete, RabbitMQ manual acknowledgement, a Kafka offset committed after the work
Exactly-once	No	No	Only where one system controls both the read and the write, as inside Kafka

Why a repeat cannot be ruled out

The broker learns that a message is done only from the consumer’s acknowledgement. Suppose the consumer updates the database and crashes before it acknowledges. The broker cannot tell whether the work happened, so it has two choices. If it treats the message as done, the work may be lost: at-most-once. If it sends the message again, the work may run twice: at-least-once. Most brokers choose the second, and RabbitMQ’s guide tells consumers to be idempotent for that reason.

What Kafka’s exactly-once covers

Kafka has two pieces. The idempotent producer gets an ID from the broker and a sequence number per message, so a resend does not add a second copy to the log. Transactions let a producer write to several partitions atomically, together with the consumer offset. A consumer sees only committed transactional records when isolation.level is read_committed, and the default is read_uncommitted.

That covers reading from Kafka, processing and writing back to Kafka, which is what Kafka Streams does. When the output goes to an external system, Kafka’s documentation says that exactly-once needs cooperation from that system, for example storing the offset in the same place as the output. An email, an HTTP call or an arbitrary database write is outside what a Kafka transaction can undo.

What to do

Take at-least-once delivery and make the consumer idempotent. The Azure Architecture Center calls the result effectively-once processing: the broker may deliver twice, and the second delivery changes nothing. The next note shows how, in idempotent consumersIdempotent consumersA consumer that gives the same result whether a message arrives once or five times.Opens in a new tab.