Add configurable retry backoff before dead-lettering - #356
Open
nabisobhi wants to merge 4 commits into
Open
Conversation
Adds a configurable bypass predicate so fatal/systemic exceptions propagate and crash the consumer instead of being retried or dead-lettered. This prevents a systemic outage (e.g. database down) from silently draining an entire topic into the dead letter queue. New fluent API on DeadLetterQueueOptions: .BypassFor<TException>() .BypassWhen(ex => ...) Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Retries previously happened in a tight loop with no delay, hammering a failing downstream within microseconds. DeadLetterQueueOptions now exposes WithRetryBackoff(TimeSpan) for a fixed delay and WithExponentialRetryBackoff (TimeSpan, double, TimeSpan?) for an exponentially increasing delay, capped by an optional maxDelay. The resolved policy is threaded to Consumer as a Func<int, TimeSpan> mapping attempt number (first retry = 1) to delay; null means no delay, preserving the existing default behavior. The delay honors the cancellation token, so cancelling during a backoff propagates OperationCanceledException instead of dead-lettering. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Task.Delay throws ArgumentOutOfRangeException past the platform timer limit, and because the delay is awaited inside the catch block in Consumer.Dispatch that would crash the consumer with a confusing argument error instead of backing off. Exponential backoff previously clamped overflow to TimeSpan.MaxValue, which is well beyond that limit. Introduce DeadLetterQueueOptions.MaxSupportedRetryDelay (int.MaxValue ms), clamp the computed exponential delay to it after applying an explicit maxDelay cap, and reject configured delays above it with InvalidConfigurationException. Also guard against NaN, which a zero initial delay combined with an overflowing factor produced and which previously fell through to a large negative TimeSpan. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The dead letter queue bypass work this branch was stacked on landed on master as a squashed commit, so every file conflicted. Resolutions keep master's version of the bypass feature, including the snapshotted BypassPredicate and the expanded XML docs, and re-apply the retry backoff on top of it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
nabisobhi
marked this pull request as ready for review
September 11, 2026 12:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When a dead letter queue is configured with
WithMaxRetries(n), retries currently happen in a tight loop with no delay. A transient failure (DB timeout, downstream 503) gets hammeredntimes within microseconds, which neither gives the dependency time to recover nor relieves any pressure on it.Change
Two new options on
DeadLetterQueueOptions:The delay before retry attempt
n(first retry = 1) isinitialDelay * factor^(n-1), optionally capped bymaxDelay. When neither option is configured the behavior is unchanged — no delay — so this is opt-in.Internally the policy resolves to a
Func<int, TimeSpan>(null = no delay), threaded throughConsumerConfigurationBuilder→ConsumerConfiguration→Consumer, following the same path as the existingMaxRetries. BothAddConsumeroverloads are wired.Cancellation
The delay is awaited as
Task.Delay(delay, cancellationToken)from inside the catch block, so cancelling during a backoff propagatesOperationCanceledExceptionrather than being re-caught and dead-lettered. The message is neither dead-lettered nor committed on shutdown, so it is redelivered. Covered by a test.Timer limit
Worth a look during review: the first commit clamped exponential overflow to
TimeSpan.MaxValue, butTask.DelaythrowsArgumentOutOfRangeExceptionbeyond the platform timer limit. Since the delay is awaited inside the catch block, that would have crashed the consumer with a confusing argument error instead of backing off — reachable from plainWithExponentialRetryBackoff(1s, 2.0)with a highmaxRetries, or directly viaWithRetryBackoff(TimeSpan.FromDays(60)).Delays are now validated and clamped to
MaxSupportedRetryDelay(int.MaxValuems ≈ 24.8 days), with an explicitmaxDelaystill winning when smaller. Adouble.IsNaNguard also prevents a negativeTimeSpanwhen a zero initial delay is combined with an overflowing factor (0 * ∞).No longer stacked
This PR was originally stacked on #353 (dead letter queue bypass). That has since merged, so the diff is now against
masterand shows only the backoff change.Because #353 landed as a squashed commit whose contents had evolved during its own review, merging master in conflicted across every file. The resolution keeps master's version of the bypass feature — including the snapshotted
BypassPredicateand the expanded XML docs — and re-applies the backoff on top. The merged diff against master is additive apart from the lines the backoff change legitimately modifies.Notes
Tests
TestDeadLetterQueueOptionsgains coverage for the delay sequences, factor,maxDelayprecedence, validation errors, the overflow clamp and the NaN/zero case; plus consumer-level tests for delay ordering, attempt numbering and cancellation. No wall-clock assertions — the backoff function is faked and its attempt arguments recorded, so the tests stay fast and non-flaky.Full suite: 181 passed, 0 failed. Build clean, 0 warnings.