Skip to content

Add indexer retry/backoff + concurrency cap; ease SendFundsTest load - #974

Merged
dangershony merged 1 commit into
mainfrom
fix/indexer-resilience-and-sendfunds-load
Sep 23, 2026
Merged

dangershony merged 1 commit into
mainfrom
fix/indexer-resilience-and-sendfunds-load

Conversation

@dangershony

Copy link
Copy Markdown
Member

Problem

Chasing intermittent SendFundsTest and MultiFundClaimAndRecoverTest UAT failures against the shared Angornet test indexer (test.indexer.angor.io), which has no healthy fallback (signet.angor.online/signet2.angor.online are both currently unreachable) and returns transient Internal Server Error responses under concurrent load.

Root cause

Wallet balance/UTXO refresh (WalletOperations.UpdateDataForExistingAddressesAsync) fans out one indexer request per address via Task.WhenAll with no bound. For a wallet with many derived addresses — and especially with multiple wallets refreshing concurrently (3 processes in SendFundsTest, 4+ in MultiFundClaimAndRecoverTest) — this fires a burst of simultaneous requests the indexer can't reliably handle.

Unlike PublishTransactionAsync (which already retries broadcasts 3x with backoff), the read paths (FetchUtxoAsync, GetTransactionInfoByIdAsync) had no retry at all, so a single flaky response aborted the whole operation — surfacing as SendAmount failed ... Indexer test.indexer.angor.io returned an error: Internal Server Error or We couldn't load the claimable funds for this project: Internal Server Error.

(This builds on #973, which fixed a related but distinct bug where the same failure mode crashed with a confusing JSON parse exception instead of a clear error message.)

Changes

  • MempoolSpaceIndexerApi: add 3-attempt retry with backoff around FetchUtxoAsync and GetTransactionInfoByIdAsync (both pure reads, safe to retry), matching the existing broadcast-retry pattern in PublishTransactionAsync.
  • WalletOperations: add a SemaphoreSlim-based throttle (max 8 concurrent) around FetchUtxoForAddressAsync so gap-limit scans and multi-wallet activity don't hammer the indexer with unbounded concurrent requests.
  • SendFundsTest:
    • Stagger address-fetch/send calls across the 3 test wallets instead of firing them in a tight simultaneous burst.
    • Add a bounded retry (3 attempts, 5s backoff) to the test's own GetAddress helper — a pure read, safe to retry.
    • Reduce TotalRounds from 10 to 5. The test was firing 43 send transactions total (10×3 main rounds + 4×3 burst rounds + 1 sweep) against a single shared, fragile test indexer — excessive load for a UAT test. 5 rounds still exercises the same code paths (multi-round send/receive, burst/unconfirmed spends, final sweep) with roughly half the indexer load.

Testing

  • dotnet test src/shared/Angor.Shared.Tests/Angor.Shared.Tests.csproj — 160/160 passing throughout.
  • Re-ran SendFundsTest and MultiFundClaimAndRecoverTest multiple times after each incremental change — both now pass reliably (previously failed intermittently with indexer errors/timeouts at various points: receive-address timeout, send failure, claimable-funds load failure).

Chasing intermittent SendFundsTest and MultiFundClaimAndRecoverTest UAT
failures against the shared Angornet test indexer (test.indexer.angor.io),
which has no healthy fallback and returns transient "Internal Server
Error" responses under concurrent load.

Root cause: wallet balance/UTXO refresh (WalletOperations.
UpdateDataForExistingAddressesAsync) fans out one indexer request per
address via Task.WhenAll with no bound — for a wallet with many
addresses, and especially across multiple wallets refreshing at once,
this fires a burst of simultaneous requests that the indexer can't
reliably handle. Unlike PublishTransactionAsync (which already retries
broadcasts 3x with backoff), the read paths (FetchUtxoAsync,
GetTransactionInfoByIdAsync) had no retry at all, so a single flaky
response aborted the whole operation.

Changes:
- MempoolSpaceIndexerApi: add 3-attempt retry with backoff around
  FetchUtxoAsync and GetTransactionInfoByIdAsync (both pure reads,
  safe to retry), matching the existing broadcast-retry pattern.
- WalletOperations: add a SemaphoreSlim-based throttle (max 8
  concurrent) around FetchUtxoForAddressAsync so gap-limit scans and
  multi-wallet activity don't hammer the indexer with unbounded
  concurrent requests.
- SendFundsTest: stagger address-fetch/send calls across the 3 test
  wallets instead of firing them in a tight simultaneous burst, add a
  bounded retry to the test's own GetAddress helper (a pure read), and
  reduce TotalRounds from 10 to 5 (43 send transactions was excessive
  load for a UAT test; this still exercises the same code paths —
  multi-round send/receive, burst/unconfirmed spends, final sweep).

Verified: Angor.Shared.Tests 160/160 passing. Re-ran SendFundsTest and
MultiFundClaimAndRecoverTest multiple times after each change — both
now pass reliably (previously failed intermittently with indexer
errors/timeouts).
@dangershony
dangershony merged commit e655526 into main Sep 23, 2026
1 check passed
@dangershony
dangershony deleted the fix/indexer-resilience-and-sendfunds-load branch September 23, 2026 23:39
dangershony added a commit that referenced this pull request Sep 24, 2026
#975)

Found while re-running the full UAT suite after merging #973/#974:
SendFundsTest.ThreeUsersSendToEachOther reproducibly failed (2/2 runs)
at the final sweep-all verification with:

  Expected sweptA to be less than 4.0 because A receives C's balance
  minus the network fee (fee is subtracted from the swept amount),
  but found 4.0.

Root cause: FundsViewModel.TotalBalance formats the displayed balance
with "F4" (4 decimal places). At the 2 sat/vB fee rate used by this
test, a single-input sweep transaction costs roughly 200-300 sats
(~0.000002-0.000003 BTC) — three orders of magnitude below what F4
can represent. The fee is genuinely subtracted on-chain; it just
rounds away completely at this display precision, so the strict
BeLessThan assertion was never reliable at this fee rate. This wasn't
caught before because earlier runs failed on unrelated indexer/faucet
issues before ever reaching this check; now that #973/#974 fixed
those, the test reaches this point reliably and exposed the latent
assertion bug.

Fix: relax to BeLessThanOrEqualTo, since "fee not visible at 4-decimal
display precision" is expected, correct behavior, not a bug.

Verified: re-ran SendFundsTest after the fix — passes reliably.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant