Skip to content

Add configurable retry backoff before dead-lettering - #356

Open
nabisobhi wants to merge 4 commits into
masterfrom
nabisobhi-dlq-retry-backoff
Open

Add configurable retry backoff before dead-lettering#356
nabisobhi wants to merge 4 commits into
masterfrom
nabisobhi-dlq-retry-backoff

Conversation

@nabisobhi

@nabisobhi nabisobhi commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Problem

When a dead letter queue is configured with WithMaxRetries(n), retries currently happen in a tight loop with no delay. A transient failure (DB timeout, downstream 503) gets hammered n times within microseconds, which neither gives the dependency time to recover nor relieves any pressure on it.

Change

Two new options on DeadLetterQueueOptions:

options.WithDeadLetterQueue("orders.dead-letter")
       .WithMaxRetries(3)
       .WithExponentialRetryBackoff(TimeSpan.FromSeconds(1), factor: 2.0,
                                    maxDelay: TimeSpan.FromSeconds(30));

// or a fixed delay
options.WithDeadLetterQueue().WithMaxRetries(3)
       .WithRetryBackoff(TimeSpan.FromSeconds(5));

The delay before retry attempt n (first retry = 1) is initialDelay * factor^(n-1), optionally capped by maxDelay. When neither option is configured the behavior is unchanged — no delay — so this is opt-in.

Internally the policy resolves to a Func<int, TimeSpan> (null = no delay), threaded through ConsumerConfigurationBuilderConsumerConfigurationConsumer, following the same path as the existing MaxRetries. Both AddConsumer overloads are wired.

Cancellation

The delay is awaited as Task.Delay(delay, cancellationToken) from inside the catch block, so cancelling during a backoff propagates OperationCanceledException rather than being re-caught and dead-lettered. The message is neither dead-lettered nor committed on shutdown, so it is redelivered. Covered by a test.

Timer limit

Worth a look during review: the first commit clamped exponential overflow to TimeSpan.MaxValue, but Task.Delay throws ArgumentOutOfRangeException beyond the platform timer limit. Since the delay is awaited inside the catch block, that would have crashed the consumer with a confusing argument error instead of backing off — reachable from plain WithExponentialRetryBackoff(1s, 2.0) with a high maxRetries, or directly via WithRetryBackoff(TimeSpan.FromDays(60)).

Delays are now validated and clamped to MaxSupportedRetryDelay (int.MaxValue ms ≈ 24.8 days), with an explicit maxDelay still winning when smaller. A double.IsNaN guard also prevents a negative TimeSpan when a zero initial delay is combined with an overflowing factor (0 * ∞).

No longer stacked

This PR was originally stacked on #353 (dead letter queue bypass). That has since merged, so the diff is now against master and shows only the backoff change.

Because #353 landed as a squashed commit whose contents had evolved during its own review, merging master in conflicted across every file. The resolution keeps master's version of the bypass feature — including the snapshotted BypassPredicate and the expanded XML docs — and re-applies the backoff on top. The merged diff against master is additive apart from the lines the backoff change legitimately modifies.

Notes

  • No new NuGet dependencies.
  • Public API is additive only — no signature or behavior changes to existing members, so this stays on the 2.1.0 line.
  • Bypass and diagnostics code untouched.

Tests

TestDeadLetterQueueOptions gains coverage for the delay sequences, factor, maxDelay precedence, validation errors, the overflow clamp and the NaN/zero case; plus consumer-level tests for delay ordering, attempt numbering and cancellation. No wall-clock assertions — the backoff function is faked and its attempt arguments recorded, so the tests stay fast and non-flaky.

Full suite: 181 passed, 0 failed. Build clean, 0 warnings.

nabisobhi and others added 3 commits August 30, 2026 20:34
Adds a configurable bypass predicate so fatal/systemic exceptions propagate
and crash the consumer instead of being retried or dead-lettered. This
prevents a systemic outage (e.g. database down) from silently draining an
entire topic into the dead letter queue.

New fluent API on DeadLetterQueueOptions:
  .BypassFor<TException>()
  .BypassWhen(ex => ...)

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retries previously happened in a tight loop with no delay, hammering a
failing downstream within microseconds. DeadLetterQueueOptions now exposes
WithRetryBackoff(TimeSpan) for a fixed delay and WithExponentialRetryBackoff
(TimeSpan, double, TimeSpan?) for an exponentially increasing delay, capped
by an optional maxDelay. The resolved policy is threaded to Consumer as a
Func<int, TimeSpan> mapping attempt number (first retry = 1) to delay; null
means no delay, preserving the existing default behavior.

The delay honors the cancellation token, so cancelling during a backoff
propagates OperationCanceledException instead of dead-lettering.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Task.Delay throws ArgumentOutOfRangeException past the platform timer limit,
and because the delay is awaited inside the catch block in Consumer.Dispatch
that would crash the consumer with a confusing argument error instead of
backing off. Exponential backoff previously clamped overflow to
TimeSpan.MaxValue, which is well beyond that limit.

Introduce DeadLetterQueueOptions.MaxSupportedRetryDelay (int.MaxValue ms),
clamp the computed exponential delay to it after applying an explicit
maxDelay cap, and reject configured delays above it with
InvalidConfigurationException. Also guard against NaN, which a zero initial
delay combined with an overflowing factor produced and which previously fell
through to a large negative TimeSpan.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nabisobhi
nabisobhi marked this pull request as draft August 30, 2026 18:57
Base automatically changed from nabisobhi-bookish-waddle to master September 10, 2026 12:22
The dead letter queue bypass work this branch was stacked on landed on
master as a squashed commit, so every file conflicted. Resolutions keep
master's version of the bypass feature, including the snapshotted
BypassPredicate and the expanded XML docs, and re-apply the retry backoff
on top of it.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@nabisobhi
nabisobhi marked this pull request as ready for review September 11, 2026 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant