Retries and backoff

A retry runs a failed message again after a delay, on the assumption that the failure is temporary. It suits a database timeout or a service that is briefly down. It does nothing for a payload that cannot be parsed or a bug in the handler, which fail the same way every time.

Transient or permanent

Sort the error first. A timeout, a throttling response or a lost connection is transient: retry. A deserialization failure or a rule violation is permanent: send the message to a dead-letter queueDead-letter queuesWhere a message goes when it cannot be processed, so the rest of the traffic keeps moving.Opens in a new tab at once, since retrying only delays the same result and holds up the messages behind it.

Backoff and jitter

Retrying immediately adds load to a dependency that is already struggling. Wait longer after each failure: 1 second, 2, 4, 8, up to a cap. If many consumers failed together, they also retry together, in synchronised waves. The AWS Architecture Blog compares adding jitter, a random part of the delay, and finds that it cuts the work clients do substantially. Full jitter picks a random delay between zero and the current ceiling.

public static class Backoff
{
    private static readonly double CapSeconds = TimeSpan.FromMinutes(1).TotalSeconds;

    // Full jitter: a random delay between zero and min(cap, 2^attempt seconds).
    public static TimeSpan Delay(int attempt)
    {
        var ceiling = Math.Min(CapSeconds, Math.Pow(2, attempt));
        return TimeSpan.FromSeconds(Random.Shared.NextDouble() * ceiling);
    }
}

public async Task HandleWithRetryAsync(PaymentReceived msg, CancellationToken ct){    const int maxAttempts = 5;    for (var attempt = 1; ; attempt++)    {        try        {            await handler.HandleAsync(msg, ct);            return;        }        catch (Exception ex) when (IsTransient(ex) && attempt < maxAttempts)        {            logger.LogWarning(ex, "Attempt {Attempt} failed for {MessageId}", attempt, msg.MessageId);            await Task.Delay(Backoff.Delay(attempt), ct);        }        // Not transient, or out of attempts: the exception escapes and the message is dead-lettered.    }}

The when filter on line 11 is what separates the two error kinds: only transient errors, and only below the attempt limit, are caught. Everything else leaves the method.

Where the retries run

A loop like this holds the message, and the consumer, for the whole wait. That suits a few short delays. For long delays, the broker or framework redelivers the message later instead. MassTransit calls these two retry (UseMessageRetry) and redelivery (UseDelayedRedelivery), and moves the message to an error queue when they are used up.

The catch

Every retry is a repeat delivery, so the handler has to be idempotent (see idempotent consumersIdempotent consumersA consumer that gives the same result whether a message arrives once or five times.Opens in a new tab). A retried message also arrives after the ones that came behind it, which matters when order does (see message orderingMessage orderingOrder exists only within one key, and only while one consumer handles that key at a time.Opens in a new tab). And when several layers each retry, the delays multiply: the Azure Architecture Center advises letting the lower layer fail fast and keeping one retry policy at the level that knows the whole operation.