Skip to content

ci(anvil): limit mutation concurrency - #765

Draft
martin-kolinek wants to merge 9 commits into
users/ralfbiedert/rallocator-v1-integration-20260915from
diagnose/anvil-mutants-jobs-1
Draft

martin-kolinek wants to merge 9 commits into
users/ralfbiedert/rallocator-v1-integration-20260915from
diagnose/anvil-mutants-jobs-1

Conversation

@martin-kolinek

Copy link
Copy Markdown
Collaborator

Purpose

Diagnostic change for the Linux and Linux-ARM mutation runner shutdowns observed on #764.

Both jobs successfully found 1,025 mutants and passed their unmutated baseline, then timed out on the same allocator mutations before the runner service received SIGTERM (exit 143). The generated Anvil recipe currently passes --jobs 0, which enables automatic CPU-based parallelism; the former Oxidizer mutation runner used one mutation job at a time.

Change

Run PR diff mutation testing with --jobs 1 instead of --jobs 0.

This intentionally modifies a generated Anvil recipe only for diagnosis. If the mutation matrix completes reliably, the equivalent change should be made in the upstream Cargo Anvil catalog and regenerated here rather than merging this generated-file override as the permanent fix.

Evidence

Validation

  • just --dry-run anvil-mutants-diff
  • git diff --check

martin-kolinek and others added 2 commits September 16, 2026 20:09
Run one diff mutant at a time to test whether automatic parallelism causes hosted runner shutdowns on allocator-heavy changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Point only the diagnostic mutation group at PR 764's original base so it runs the same allocator mutants with single-job concurrency.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.9%. Comparing base (aa06840) to head (c12b303).
⚠️ Report is 7 commits behind head on users/ralfbiedert/rallocator-v1-integration-20260915.

❌ Your project check has failed because the head coverage (99.9%) is below the target coverage (100.0%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@                                  Coverage Diff                                   @@
##           users/ralfbiedert/rallocator-v1-integration-20260915    #765     +/-   ##
======================================================================================
  Coverage                                                  99.9%   99.9%             
======================================================================================
  Files                                                       648     648             
  Lines                                                     89603   90644   +1041     
======================================================================================
+ Hits                                                      89600   90643   +1043     
+ Misses                                                        3       1      -2     
Flag Coverage Δ
linux 99.9% <ø> (-0.1%) ⬇️
linux-arm 99.9% <ø> (+<0.1%) ⬆️
windows 99.9% <ø> (+<0.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

martin-kolinek and others added 6 commits September 17, 2026 10:43
Run only the next deterministic Linux allocator mutant and stream cgroup and process memory telemetry so evidence survives runner termination.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Advance the isolated Linux diagnostic to the next deterministic allocate_medium mutation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Run the next eight deterministic allocate_medium mutants as a bounded telemetry batch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Advance the Linux shutdown diagnostic to the next deterministic mutation block.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Narrow the reproduced OOM batch to its first deterministic mutant.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Test the mutation that turns the finite telemetry bitmap iterator into an unbounded stream.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@martin-kolinek

Copy link
Copy Markdown
Collaborator Author

🤖 Diagnostic result: the runner shutdown is a genuine OOM caused by one deterministic mutant, not mutation parallelism.

Exact mutant:

crates/rallocator/src/allocator.rs:1955:23:
replace &= with |= in telemetry_used_slice_indices

This changes remaining &= remaining - 1, which clears one bitmap bit per iteration, into remaining |= remaining - 1. The iterator therefore stops making progress and supplies an unbounded stream to snapshot collection.

Linux x64 and ARM64 both reproduced the failure with --jobs 1 and only this single mutant selected. Streamed telemetry showed system memory rising to roughly 15.8-15.9 GiB of 15.9 GiB, available memory falling to 0-167 MiB, and rallocator-* processes reaching roughly 8.4-8.6 GiB plus 6.8-7.1 GiB RSS before the runner received SIGTERM/exit 143.

Earlier isolated mutations at allocator.rs:1259:57 and the following eight allocate_medium arithmetic mutants were all caught normally and stayed near 2.4 GiB used memory.

This means --jobs 1 should not be upstreamed as the fix. The repository should skip or otherwise contain this runaway iterator mutant; a general Anvil improvement would be per-mutant memory containment rather than lower concurrency.

Exclude the finite telemetry bitmap iterator from mutation generation because bit-clearing mutants create unbounded snapshot streams that OOM Linux runners.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant