[HouMinXi]

What

When the sandbox child failed before exec, it wrote b"error" to the status pipe and exited 127. On a placement failure the parent never read the pipe. The #31 qualification stop therefore recorded only payload process was not placed in the run cgroup, and the child errno was lost.

This PR covers the #31 acceptance item "preserve a bounded child startup diagnostic, placement/gate status ... on every failure". It does not cover the timing half.

Startup diagnostic:

• The child records the step it was in when it stopped: setup, place (writing its pid to cgroup.procs), gate, stdio or exec. It writes error stage=<step> type=<exception> errno=<n|none>, capped at 200 bytes.
• If building that message raises (for example MemoryError), the child writes error stage=unknown instead, then _exit(127) as before.
• The child reads the gate once. Before, it looped on empty reads, so a gate closed without the go byte (the parent failed or died) left it spinning at full CPU. EOF now raises EOFError. The child reports error stage=gate type=EOFError errno=none and exits 127.
• The parent records the child (_process, payload_pid) right after fork(), before it closes its copies of the child's fds. If one of those closes fails, the failed start still kills and reaps the child.
• Only bytes 0x20-0x7e are kept. Anything else becomes ?, both when the child encodes and when the parent decodes, so control characters and non-ASCII bytes never reach the error text.
• On a placement failure the message includes the child's exit status (exit N, signal N for a negative return code, or still running at deadline) plus the diagnostic. The pipe is read without blocking. When nothing is buffered, the message says no startup diagnostic.
• An exec failure reports the same diagnostic: payload failed to exec bubblewrap (error stage=exec ...). The read asks for 200 - len(status) bytes each time, so it never reads past the cap.

File descriptors:

• A failed start closes every pipe and cgroup.procs descriptor the parent still holds, each exactly once. On main the placement-failure path leaked gate_w and status_r. Opening cgroup.procs failing, or fork() failing, leaked 4-5 descriptors (the two pipe pairs and, for a fork failure, the cgroup.procs fd). All three paths are fixed and tested.

Cgroup cleanup:

• _remove_cgroup used to return silently when rmdir still failed at its 10 s deadline (EBUSY with live tasks, ENOTEMPTY, EACCES), leaving the run cgroup behind with no record. It now sets Supervisor.cleanup_error (printable, at most 200 characters, with the last errno) and raises CgroupCleanupError, a subclass of IsolationError. The exception message is <cleanup_error>: <cgroup path>. It is printable and capped at 400 characters, and its __cause__ is the last rmdir error. The error type and errno come before the path, so a long path cannot push them past the cap.
• cleanup_error is reset at the start of each removal, so an old failure does not survive a later successful removal.
• A cgroup that is already gone still counts as removed, with no error and no retry (fix(isolate): treat an already-removed run cgroup as a successful removal #320). rmdir is tried at least once before the deadline is checked.
• teardown() forces removal (rewrites cgroup.kill between retries). If an earlier removal already ran out its deadline (a failed start), teardown makes one more attempt instead of waiting another 10 s. It is marked done only after the cgroup is gone, so a second teardown retries instead of returning silently.
• On a failed start, the start error stays the raised exception. A cleanup failure is attached to it as a note. Notes go through isolate.attach_note(), which drops any failure while adding the note, the same as ledger.py. It also skips a note the exception already carries. Without that, a start failure whose cgroup could not be removed got the same cleanup note twice: once from Supervisor.start() and again from _run_sandboxed's teardown.
• verify_isolation_support() used to ignore a probe cgroup it could not remove. It now raises IsolationUnavailable(reason="probe_cleanup_failed") with a bounded, printable detail (probe cgroup was not removed (type=... errno=...): <path>, the path last for the same reason), or adds that detail as a note when another probe error is already on its way out. A probe cgroup that is already gone (FileNotFoundError) counts as removed and reports nothing.
• MutmutAdapter.probe() reports probe_cleanup_failed as probe-cleanup-failed in phase cleanup with retryable=True. Missing isolation stays isolation-unavailable in phase probe with retryable=False.

The mutmut engine (python_mutmut._run_sandboxed, also used by the gremlins, cargo-mutants, Stryker and patch-corpus adapters):

• If the command fails to start or is interrupted, that error propagates, including KeyboardInterrupt, SystemExit and CancelledError. A CgroupCleanupError from teardown is attached as a note instead of replacing it. Before this change the cleanup error replaced it, and the original was only __context__.
• The supervisor thread is joined in a finally on both paths. memory.peak is read inside the guarded block, so an interrupt during that read still tears down and joins.
• If the command finished but its cgroup was left behind, the run fails closed: CgroupCleanupError is raised with receipt, exit_code and timed_out set, so the result of the command is not lost. I chose this over returning the exit code, because a cgroup left behind means the host is not in the state the next run assumes.
• MutmutAdapter.run() turns that error into a TargetResult with run_state=INCOMPLETE, reason_code="cgroup-cleanup-failed" and Cleanup(state=INCOMPLETE, owned_group_empty=False, errors=(<cleanup error>,)). The command's receipt is kept, and so is the baseline if it had already run. A cleanup-failure.json artifact records the message, exit code and timeout flag. Before this change run() always reported owned_group_empty=True.

Other adapters:

• Every sandbox failure result is now built by base.sandbox_failure_result(). It takes no earlier infrastructure errors. The only caller that passed any (mutmut) always passed an empty list at that point, so that parameter is gone. The mutmut, gremlins, Stryker, cargo-mutants and patch-corpus adapters return it from run() for both CgroupCleanupError and IsolationUnavailable, instead of letting either escape.
• Gremlins, Stryker and cargo-mutants collect receipts, artifacts and the baseline in a RunEvidence object owned by run(). A failure on the second or third command keeps every earlier receipt and the passed baseline. Patch-corpus and mutmut already kept theirs.
• IsolationUnavailable from Supervisor.start() gives run_state=INCOMPLETE with the isolation error's code and an isolation-failure.json artifact (message, reason, cleanup failure). Supervisor.start() re-checks isolation before each command, so the error is reported in phase baseline if no baseline was recorded yet and mutation after that, not probe. A probe cgroup left behind stays a cleanup error. If the run cgroup was also left behind after the start failure, _run_sandboxed records that on the exception, and the result reports it as a retryable cgroup-cleanup-failed error with owned_group_empty=False. Otherwise cleanup is COMPLETE.
• The gremlins probe runs gremlins --version in the sandbox. A cleanup failure there returns UNAVAILABLE_ISOLATION with a retryable cgroup-cleanup-failed error. Missing isolation there now returns UNAVAILABLE_ISOLATION too, instead of raising out of mutation_dispatch. The probe workspace is a TemporaryDirectory; before, each call left a forge-gremlins-probe-* directory in the temp dir. The Stryker and cargo-mutants probes do not use the sandbox.

The existing message prefixes are unchanged. The placement check, gate order and limits are also unchanged.

Refs #31

Tests

tests/test_isolate_startup_diagnostic.py has 51 test functions (67 cases). The integration tests drive the real Supervisor.start() fork, gate and placement path on a host without cgroups or bwrap. The cgroup root is a plain directory and cgroup.procs is a regular file, so the real child writes its pid there and the placement check finds it. Faults are injected through a proxy installed as isolate.os. It forwards to the real os, and the overrides act only in the child unless a test says otherwise.

• Each test has a 15-second limit, enforced by SIGALRM in the main thread (CI has no pytest-timeout). A regression that blocks a read or loops forever fails that test and does not hang the CI step. Tests that start a real child stop it through a helper: it joins the supervisor thread, kills a child still stuck before exec, and reaps it, so no thread or process outlives a failed test.
• One test per child stage, each asserting the exact message: place, exec, setup, gate, stdio.
• A gate closed without the go byte: the child reports stage=gate type=EOFError errno=none and exits 127.
• The parent's first os.close after fork() fails: the start error is raised, the child is killed (-SIGKILL) and reaped, and every fd is closed.
• The stage=unknown fallback when the diagnostic formatter raises MemoryError or KeyboardInterrupt.
• On the exec path: non-ASCII bytes become ?. With each parent read capped at 150 bytes, a 300-byte diagnostic makes the parent ask for 200 and then 50 bytes, and it keeps exactly 200. No sleep.
• The placement deadline with the child still blocked on the gate.
• cgroup.procs open failure, fork() failure, and every parent fd closed exactly once on the placement and exec failure paths.
• A parent os.close that fails during start cleanup: the placement error is still raised and every fd is still closed.
• A non-blocking read that fails with EBADF raises instead of reading as "no startup diagnostic". After a diagnostic read the pipe is non-blocking, checked with os.get_blocking.
• _error_detail prints errno=none for an errno that is not an int.
• Start failure with the real _remove_cgroup, a fake rmdir (ENOTEMPTY) and a fake clock: the placement error is raised with the cleanup note, one attempt is made, teardown makes one more and raises, then succeeds once the cgroup goes, and a third teardown does nothing.
• Removal unit tests with a fake clock: EBUSY, ENOTEMPTY and EACCES at the deadline; the 400-character printable message and its __cause__; with a 600-character path, the message still starts with the full detail, including errno=16; a stale cleanup_error cleared by a later success; forced and unforced cgroup.kill rewrites; teardown arguments (force=True, timeout=10, then timeout=0).
• attach_note() when __notes__ is not a list, when reading it raises a BaseException that is not an Exception (the call returns normally, reads __notes__ once and does not try to add the note), and when the same note is attached twice (kept once). Two notes that differ only in the errno at the end are both kept, and a note already in the list is skipped wherever it sits (X, Y, X, Y gives [X, Y]). Only an equal note is skipped: a note that contains another, sits inside one, or shares its end with one is still added, in either order (cleanup failed: /x and cleanup failed, errno=16 and rmdir errno=16). Notes are compared in full: two 401-character notes that differ only in the first or the last character are both kept, as are notes that differ only in case or in a trailing space, and first, second, first gives [first, second], so a repeated note keeps its first place. attach_note does not cut notes, so the long notes are stored whole. Notes are compared by value, not identity: a second first built at runtime ("".join(["fir", "st"]), checked to be a different object) is skipped.
• Probe cleanup: a probe cgroup left behind raises probe_cleanup_failed with a 200-character bound (a 250-character root loses only the end of its path), is a note on a "lacks memory.max" error, and a clean probe stays silent. A probe cgroup already gone is silent, and adds no note to a "lacks memory.max" error.

tests/test_python_mutmut_cleanup.py (new, 19 cases) uses a fake Supervisor and SupervisorThread:

• an in-flight RuntimeError, KeyboardInterrupt, SystemExit or CancelledError with a cleanup failure: the same error object is raised, with the cleanup message as its note, and the thread is joined;
• the note cannot be added: the original is still raised;
• a start failure that already carries the cleanup note: the note appears once;
• teardown raising some other error: the thread is still joined;
• a finished command with a cleanup failure: CgroupCleanupError with the receipt, exit code 7 and timed_out=False;
• run() with a cleanup failure after the baseline and after mutmut: INCOMPLETE, owned_group_empty=False, the cleanup message in cleanup.errors and infrastructure_errors with retryable=True, the exact cleanup-failure.json contents (message, exit code, timed_out), that artifact listed in native_artifacts, the receipts kept, and the passed baseline kept in the second case;
• KeyboardInterrupt while reading memory.peak: teardown runs once and the thread is joined;
• the probe: probe_cleanup_failed is a retryable cleanup error, other isolation failures are not retryable;
• IsolationUnavailable on the baseline command: phase baseline, baseline UNKNOWN, no receipts;
• IsolationUnavailable on the mutmut command after a passed baseline: phase mutation, INCOMPLETE, the baseline receipt and the baseline kept, cleanup COMPLETE; and the same with the run cgroup also left behind: owned_group_empty=False and both errors reported.

tests/test_mutation_adapter_cleanup.py (new, 28 cases) runs each adapter's real run() against a fake sandbox:

• gremlins (baseline, inventory, outcomes), Stryker (baseline, stryker) and cargo-mutants (baseline, mutants), failing at each command with a cgroup left behind: INCOMPLETE, every earlier receipt plus the failing one, the PASSED baseline once it has run, its artifact, and cleanup-failure.json;
• each adapter with IsolationUnavailable at each command, in phase baseline on the first command and mutation after it, and with the run cgroup also left behind;
• the gremlins probe: a cleanup failure, missing isolation (unavailable and probe_cleanup_failed), a run cgroup left behind after a start failure, and no forge-gremlins-probe-* directory left after any of four outcomes;
• patch-corpus on a corpus entry with a cleanup failure or missing isolation (phase mutation): the passed baseline, the receipts and both artifacts kept;
• patch-corpus with a sandbox failure on the baseline command: baseline UNKNOWN, only the failure artifact;
• patch-corpus failing on the second of two entries: the first entry's receipt and event artifact are kept;
• in every failing run() test above: the result's target_id, adapter_version, tool_version and extractor version, owned_mounts_removed, the target_id of every error, the probe-cleanup and isolation error messages, and the failure artifact's digest and byte count checked against the published file.

tests/test_mutation_isolate.py::test_remove_cgroup_force_rewrites_kill expects CgroupCleanupError, because its cgroup never empties.

Mutation check: the previous description said 33 of 33 mutants were killed. That held for my set, but review ran 48 mutants against 672db92 and 4 survived: retryable=False on the cleanup error, exit_code or timed_out dropped from cleanup-failure.json, and the artifact not added to native_artifacts. One more (attach_note catching IsolationError) was equivalent. The test changes above kill those 4.

For 926132a I cleared __pycache__, checked that a deliberate break fails first, then applied 91 hand-written mutants one at a time to isolate.py, python_mutmut.py, base.py and the four other adapters, running the four test files above against each. All 91 were killed. Besides the earlier set (updated for the renamed helper), they cover: dropping the IsolationUnavailable catch in each run() and in the gremlins probe; not recording the cleanup failure on the exception or ignoring it in the result; not recording a baseline, receipt or artifact in each adapter, or not passing them to the result; and leaking the probe temp dir. For ec5189f I ran 8 more mutants on the phase change, after a deliberate break failed first: dropping it, swapping baseline and mutation, inverting the baseline check, always baseline, always mutation, applying it to the cleanup phase too (two forms), and never recording the isolation error as a cleanup error. All 8 were killed. Review then found 11 surviving mutants at ec5189f in base.py and patch_corpus.py. For 8419350 I ran 19 mutants plus a deliberate break that failed first, from a separate copy of the tree with __pycache__ cleared and PYTHONDONTWRITEBYTECODE=1. They were: the evidence digest and byte count; the result's target_id, adapter_version, tool_version and extractor version; owned_mounts_removed=False; target_id and message in each of the three error helpers; deleting baseline = None in patch-corpus run(); dropping the event artifact append; and four on the new attach_note check (removed, inverted, return made pass, default None). All 19 were killed. An earlier form of the check (isinstance(notes, list) and note in notes) had an equivalent mutant, so the check is now note in getattr(exc, "__notes__", ()). A non-list __notes__ makes add_note fail anyway, and that failure is dropped. Cross-check then found two mutants of that check that survived at 8419350: comparing only the first 8 characters, and comparing only the last note. For 12065ea I added tests for both and reran 7 mutants of the check after a deliberate break failed first: those two, first note only, check removed, inverted, return made pass, and default None. All 7 were killed. Review then found a third survivor at 12065ea: matching any note that contains the new one (note in n). For 991651b I added a test that attaches two notes where one contains the other or shares its end, in both orders, and reran 11 mutants of the check after a deliberate break failed first: contains, contained in (n in note), last 3 characters only, last 8 characters only, first 8 characters only, last note only, first note only, check removed, inverted, return made pass, and default None. All 11 were killed. Cross-check then found 7 more mutants that survived at 991651b: comparing only the last 16 or 400 characters, only the first 111 or 399 characters, ignoring case, ignoring surrounding space, and moving a repeated note to the end. For 3340a58 I added a test for those cases. Before the commit, the 7 survived against 991651b's tests. With the new test, I reran them with the earlier 11 after a deliberate break failed first, and all 18 were killed. Cross-check then found one more survivor at 3340a58: comparing by identity (n is note), because every test note was a literal or the same object reused. For ac73acf I added a test with an equal note built at runtime. Before the commit, the mutant survived against 3340a58's tests. With the new test, I reran it with the earlier 18 after a deliberate break failed first, and all 19 were killed.

Review of ac73acf found that two mutants were caught only by hanging: removing os.set_blocking from the diagnostic read, and >= to > on the _remove_cgroup deadline. For 5d5f164 I ran 10 mutants from a separate git archive copy with __pycache__ cleared, PYTHONDONTWRITEBYTECODE=1, Python 3.12 and pytest 9.1.1, no -x and nothing deselected. A deliberate break (stage = "gat") failed first. All 10 were killed:

Mutant	Result
procs_fd left out of parent_fds	killed, 19 failed
attach_note catching Exception instead of BaseException	killed, 1 failed
os.set_blocking removed	killed, 3 failed (two by the 15 s limit, one by the new get_blocking check)
_remove_cgroup deadline >= to >	killed, 6 failed, no hang (1.5 s)
isinstance(code, int) to code is not None	killed, 1 failed
gate EOF back to the spin loop	killed by the 15 s limit
gate EOF ignored	killed, 1 failed
_process recorded after the closes again	killed, 1 failed
probe message with the path first	killed, 3 failed
run cgroup message with the path first	killed, 6 failed

No mutant left a process behind. The infra_errors mutant from review no longer applies, because the parameter is gone.

The previous rounds' 52 mutants on the diagnostic, fd and decode code are unchanged. The one equivalent mutant there (the exec read loop as while True) still applies.

Commands run at 5d5f164:

• pytest tests/test_isolate_startup_diagnostic.py tests/test_mutation_isolate.py tests/test_python_mutmut_cleanup.py tests/test_mutation_adapter_cleanup.py: 133 passed, 8 skipped (those need a real cgroup/bwrap host), on Python 3.14 and on Python 3.12. By file: 67, 19 (+8 skipped), 19 and 28.

• pytest tests/test_mutation*.py tests/test_isolate*.py tests/test_python_mutmut_cleanup.py: 2458 passed, 25 skipped, 1 failed. The failure is test_mutation_ledger_ci_controls.py::test_ci_null_baseline_deadline_is_a_config_error_before_any_baseline, which fails the same way with main's isolate.py on this box.

• pytest tests/ -k "isolate or sandbox or mutation_engine or mutmut": 183 passed, 16 skipped, 1 failed. The failure is test_cli_integration.py::TestSandboxWarning::test_sandbox_warning_emitted (no toolchain detected on this box), which also fails on main.

• ruff check, ruff format --check (ruff 0.17.3) on every touched file, and scripts/lint_changed.py --base origin/main: clean.

• Full suite: CI is the reference. The Linux tests run for 5d5f164 (run 40917406244) ran from 13:02 to 13:44 EDT and passed: real ownership and cancellation regressions 568 passed; cgroup-backed mutation sandbox 32 passed; full service-free selection 28982 passed, 81 skipped, 68 deselected; local process integration 11 passed, 7 deselected. CodeQL and the changed-file lint also passed. The run for ac73acf (run 40907382613) passed: the full service-free selection had 28977 passed, 81 skipped, 68 deselected; the cgroup-backed mutation sandbox step had 32 passed; the local process integration step had 11 passed, 7 deselected; the real ownership and cancellation regressions had 568 passed. The run for 3340a58 (run 40906340594) was cancelled at 12:01 EDT. I do not quote box counts for the full suite: on this box 61 tests fail the same way at 9c1cc62 and at this branch (no toolchain detected, and the reporting_product_findings frozen-hash checks), which CI does not show.

Not run by me: anything on a real cgroup v2 + bwrap host. The CI step above is the only run on one.

Residual risk

• The exec-path read waits for EOF. If another thread forks while the status write end is open, that fork holds a copy and the read blocks until the copy closes. The placement path reads without blocking.
• If the child is killed outright (not an exception), it writes nothing. On the placement path that gives no startup diagnostic. On the exec path, an empty pipe reads as a successful exec, and the failure shows up later as the payload's exit status.
• A placement failure in Supervisor.start() raises a plain IsolationError, which no adapter's run() catches. It still escapes run(), and the receipts, baseline and artifacts gathered before it are lost. This predates the PR and is not fixed here: reporting it needs its own error code and phase.
• The diagnostic is formatted in the child after fork() from a threaded process, as the existing child code already did. The child only builds a short string, writes and calls os._exit.
• Review reproduced about 15k inodes under .planning/ (15,229 in its worktree), written by test_invocation_owner and test_invocation_audit. That predates this PR and is not addressed here. My subset runs above do not include those tests, and this clone has no .planning/ afterwards.

#31 work left after this PR

• A pre-launch topology and migration-permission check: that a normal user can move a process into the run cgroup.
• Separate startup and cleanup timings on failure (the other half of the diagnostic acceptance item).
• Live qualification on a delegated cgroup v2 + bwrap host.
• Reporting NOT RUN separately from PASS.

This PR does not qualify Linux execution.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[HouMinXi]

Fixed in e4deb22 (ordinary integration of current main; no force push).

Mixed isolation-verification + probe-directory cleanup failures now retain the original cause and a separate typed, bounded cleanup error in capability reports, adapter results, and failure artifacts. Payload-group ownership remains distinct from probe ownership; exception notes are not parsed. Legacy artifact shape is unchanged when no typed probe failure exists.

Validation: three sequential five-lens local review cycles with no unresolved findings (manual requirements lens; no external-model qualification claim). 164 focused tests passed / 8 native-host skips; 826 adjacent passed / 3 skips / 11 process-cancellation failures reproduced on unchanged main. Six legacy envelope/artifact parity cases passed; omission control produced 13 expected failures. No test weakening or containment changes.

Exact-head hosted CI is running: Linux tests 41288675833, CodeQL 41288675837, lint 41288675825. Native/ownership qualification remains pending those results.