[HouMinXi]

Current-head acceptance update (2026-11-01)

Reviewed full range 0ef23cd942edb60bfbe0693f19ac93f21a1ba9f9..e4032a90d1e736aa199d7922f79a66f103ccabab (8 files, +1419/-19).

This remains the narrow option-c slice of #28: reverted pytest exit 1/2 requires a failed-test witness (excluding unexpected passes), or a supported collection/setup import error tied to a binding/module the fix adds. Other errors remain inconclusive. It does not close #28 or settle its broader baseline, GREEN-evidence and three-run requirements. Non-pytest runners, require_witness, GREEN policy and the overfit guard retain their prior behavior.

Current implementation supersedes the historical hunk parser described below:

• Added-name evidence comes from complete fixed/reverted Python images owned by the existing transaction, checked using descriptors, file identity and byte digests, then parsed as ASTs. It is not inferred from isolated added/deleted lines.
• Only direct module bindings introduce candidates; all syntactically possible prior bindings conservatively exclude them. Prior wildcard imports and ambiguous shared aliases fail closed. Invalid, missing, symlinked, non-regular or oversized images cannot establish import evidence. The transaction's mutation/restoration lifecycle is unchanged.
• JUnit errors require a recognized terminal exception type and anchored diagnostic. Earlier chained errors, source text, ambiguous multiline messages and teardown errors cannot earn RED. Genuine supported import witnesses remain covered.
• The historical hunk-start/multiline-import limitation no longer applies. Dotted-path/optional src/ root mapping remains bounded, and unknown exception renderings fail closed. JUnit is not authenticated against a malicious plugin forging its report.

Current validation includes real-pytest spoofing/positive controls, full-image binding boundaries and transaction restoration checks. Independent review reran 179 witness/boundary tests successfully; a separate five-file run had 510 passes and one environment-related failure reproduced at the exact base. Historical counts and mutant tables below apply only to their stated earlier revisions, not this head. The sixth requirements-acceptance lens found no blocking defect within this partial scope.

Verified at 2026-11-01 09:27 UTC: all three exact-head workflows completed successfully — Linux tests 41289226254, Lint 41289226267, and CodeQL 41289226215. Linux ownership, cgroup, full service-free selection and process-integration steps all succeeded. This records CI results, not PR approval or a merge action.

---

Historical description and evidence — earlier revisions

The original text is preserved below for provenance. Implementation details, line counts, test counts and mutation results refer to earlier revisions unless independently confirmed in the current-head update above; superseded descriptions are not current completion claims.

Refs #28 (option c)

What changes

FIXVAL used to count pytest exit 1 or 2 on the reverted code as "the test fails without the fix", whatever the reason. A test file with a broken fixture, a syntax error or an unrelated import therefore passed.

The reverted pytest run now writes --junitxml to junit.xml inside a fresh tempfile.mkdtemp directory (mode 0700), and removes the directory with rmtree afterwards. The run is RED only when the report has:

• a <failure> (a test call failed) whose message is not an unexpected pass, or
• a collection or setup <error> that is an import of a name the fix adds.

An unexpected pass is a strict XPASS (message [XPASS(strict)] ...) or a unittest expectedFailure that succeeded (message Failed: Unexpected success). pytest reports both as <failure>, and #28 counts neither as a RED witness.

"A name the fix adds" (src/code_forge/_fixval_junit.py) means a module-level def/class/assignment/import bound on a + line and not on a - line of the same production file, keyed by its dotted module (with and without src.), or a module the patch creates, renames or copies into place. A name inside a multi-line from x import (...) counts too, and so does a name on a line that continues a module-level import ending in \. A \ at the end of a comment (import os # see \) continues nothing. A path that is not an importable module (for example pkg/foo-bar.py) adds nothing. The matched error texts are cannot import name 'X' from 'M', module 'M' has no attribute 'X', and No module named 'M' / bad magic number in 'M' for created modules only. cannot import name 'X' from 'M' and module 'M' has no attribute 'X' also count when the fix creates module M.X, so from pkg import newmod is RED. Teardown errors never count.

Everything else is FIXVAL_INCONCLUSIVE: pytest exit N without a failed test or an import of a name the fix adds. That includes a missing or unparseable report. When the command already has a bare --, --junitxml goes right before it; after it, pytest would read the flag as a path and exit 4. If the project's pytest addopts has -p no:junitxml, pytest writes no report and the run is inconclusive. That is the conservative outcome: FIXVAL gives no verdict rather than a wrong RED.

Unchanged: non-pytest runners, the require_witness path (it passes no --junitxml), the overfit guard.

Known limits

Each of these fails closed (inconclusive, never a wrong RED) and is noted in a code comment:

• Only src/ is stripped from a module path. A module under another root (lib/pkg/mod.py imported as pkg.mod) matches only by its full dotted path, so an error that names pkg.mod stays inconclusive. Guessing roots would widen RED.
• No module named 'pkg.new' is not RED when the fix creates pkg/new/mod.py as a namespace package, because the patch alone does not show whether pkg/new/ existed before.
• A hunk that starts inside a multi-line import (parenthesised or backslash-continued) does not see the opening line, so names on its lines are not read.
• ElementTree resolves no external entities, but entity expansion is capped only by expat 2.4.0 and later. With an older expat a crafted report could use a lot of memory. Writing one takes code that already runs in the test process.

What remains on #28

This PR covers option c for exit 1 and 2 only. Still open on #28:

• handling a baseline that is not PASS, config errors and timeouts;
• evidence rules for the GREEN runs (fix applied);
• the three-run policy;
• controls that use real processes rather than mocks.

Tests

New file tests/test_fixval_junit_witness.py (92 tests):

• added_names: module-level bindings only, a name on both - and + lines is not new, path to module mapping, a non-importable path (pkg/foo-bar.py) adds nothing, unreadable patch, multi-line parenthesised imports (added inside, opening line as context, a comment with ), separate pre- and post-image state, a comment after the opening parenthesis, an import inside a function, an import left open at the end of one hunk), backslash-continued imports (with and without a space before \, a comment on the last line, import a, \, an assignment or an indented import that ends in \, a comment that ends in \ on the opening line and on a continued line, a plain import followed by a binding, a \ inside parentheses), renamed and copied files add their module, a rename to a name the module already had adds none, a plain modification adds none.
• red_witness reads every <failure> in a testcase: an unexpected pass before a real failure, and a real failure before an unexpected pass, are both RED.
• red_witness for a submodule the fix creates: cannot import name 'newmod' from 'pkg' and module 'pkg' has no attribute 'newmod' are RED; another name from pkg is not, and neither is pkg.newmod when the fix creates pkg.newmod2 or other.newmod. No module named 'pkg.newer' is not RED when the fix creates pkg.new, No module named 'pkg' is not RED when it creates pkg.newmod, and No module named 'pkg.new' is not RED when it creates pkg.new.mod (the namespace parent limit).
• red_witness counterexamples: teardown error, same name from another module, a different name, has no attribute for another name, SyntaxError, setup RuntimeError, ModuleNotFoundError for a module the fix did not add, empty added set, skipped-only, empty and malformed XML.
• Reports from real pytest runs: a strict XPASS (with and without a reason) and a unittest unexpected success are not RED; a real failed test next to them is RED; a reverted run whose only failures are unexpected passes is inconclusive.
• Real pytest runs: collection or setup import of an added name is PASS; unrelated import, non-import collection error, fixture RuntimeError and teardown error are inconclusive; the report is deleted; from pkg import newmod with pkg/newmod.py missing is PASS; a command with -- before the test path still gets its report.
• Where --junitxml goes: before pytest's -- for pytest, python -m pytest and coverage run -m pytest; appended when the only -- belongs to uv run.
• The report directory: a fresh 0700 directory under TMPDIR, empty when pytest starts, removed afterwards. If the directory cannot be removed, the verdict is still returned. A require_witness run gets no --junitxml.
• run_fixval end to end: the reverted module raises at import, and the run is now inconclusive (main: PASS).

Updated tests:

• test_fixval.py: the mock writes a report with a failed test.
• test_fixval_exit_classification.py: fake pytest exit 1/2 with no report, and a real import error with no added names, are now inconclusive.
• test_fixval_test_support.py: the collection-error case was renamed; it still passes, now through the added-name rule.

Mutation testing, by hand: I worked on a copy of the tree with __pycache__ cleared and PYTHONDONTWRITEBYTECODE=1. The test set was test_fixval_junit_witness.py, test_fixval.py, test_fixval_exit_classification.py and test_fixval_test_support.py (424 pass unmutated at the seventh commit).

At the seventh commit I ran every mutant against all four files in full, without -x and with nothing deselected. From a copy of the tree with __pycache__ removed before each mutant:

PYTHONDONTWRITEBYTECODE=1 TMPDIR=<private dir> python -m pytest -q -p no:cacheprovider \
  tests/test_fixval_junit_witness.py tests/test_fixval.py \
  tests/test_fixval_exit_classification.py tests/test_fixval_test_support.py

The unmutated run gave 424 passed with no flaky test. The deliberate mutant (red_witness always returns True) ran first. All 22 mutants were killed:

Mutant	Failed
red_witness always returns True (sanity)	32
No module named: a created module equal to or under the missing one (M17c)	2
comment-backslash guard reads the raw line, both sites	3
the same, continued-line site only	1
the same, opening-line site only	2
backslash condition dropped from the opening-line check	3
elif to if after the parenthesised opening	1
a comment after from x import ( not allowed	1
one import state shared by all hunks of a file	1
the opening line searched for (.search) instead of matched	1
cannot import name: M.X as a substring of a created module	1
has no attribute: the same substring match	1
cannot import name: any created module ending in .X	1
has no attribute: the same .X suffix match	1
No module named: a prefix match on created modules	1
backslash continuation never started	7
backslash continuation never ended	2
any line ending in \ starting a continuation	2
an indented import starting one (.search for .match)	1
a continued line not read	7
a comment kept on a continued line	2
a comment kept inside parentheses	1

At the sixth commit, the same set without the first seven real mutants above was killed, but that run used -x, so it showed only the first failing test for each mutant. Against the code at the fifth commit (bb90558), 7 tests of the file fail: the backslash cases that bind a name on a continued line. Against the sixth commit (f39df7d), 3 fail: the two comment-backslash cases and the comment-backslash RED case.

At the fifth commit, a deliberate failing mutant (red_witness always returns True) was killed first. All 19 real mutants were killed too:

• only the last <failure> read (findall(...)[-1:]);
• only the first <failure> read (findall(...)[:1]);
• ignore_errors=True dropped from the cleanup rmtree;
• the created-submodule check dropped from cannot import name;
• the same check dropped from has no attribute;
• M.X built as X.M;
• the inside-an-import state never set;
• context lines not fed to the import state;
• comments not stripped inside an import;
• an import that never closes;
• one shared state for both sides of a hunk;
• the opening-line pattern never matching;
• a rename not counted as creation;
• a rename keeping the old path's dotted names;
• --junitxml always appended;
• the -- offset taken from argv instead of cmd;
• the flag put after --;
• -- searched in the whole command, wrapper included;
• _with_junitxml bypassed.

Earlier rounds: 13 mutants at the third commit (M14, M09, the two unexpected-pass prefixes, the skip check both ways, exact match for startswith, continue for return, rmtree dropped, its guard inverted, a fixed report directory, --junitxml dropped, the verdict ignored) and find in place of findall at the fourth were all killed. I did not rerun those 14 at the fifth commit.

The new tests were also run against the code at the fourth commit (838be08): 11 of them fail there, including the from pkg import newmod and -- cases.

Local run (Linux, Python 3.14): every test file that mentions fixval gave 1695 passed and 12 skipped, with no failures (python -m pytest -q -p no:cacheprovider $(rg -l -i fixval tests -g '*.py'), 24 files). test_fixval_junit_witness.py also passes under Python 3.12.15 (92 passed), the version CI pins. scripts/lint_changed.py --base origin/main with ruff 0.17.3 is clean.

Commits:

1. f8a9bcc: the JUnit RED rule.
2. 590054a: only adds # noqa: S102 to an existing exec in tests/test_fixval.py. Touching that file makes the changed-files lint check all of it.
3. e127032: unexpected passes are not RED, and the report goes in a private directory.
4. 838be08: tests only. An unexpected pass before a real <failure> in one testcase, and a report directory that os.rmdir cannot remove (this test kills the ignore_errors mutant).
5. bb90558: submodule imports, multi-line imports, renames and copies, --junitxml before --, and the known-limit comments.
6. f39df7d: backslash-continued imports, and tests for the opening-line comment, function-level imports, imports left open across hunks and the M.X / No module named boundaries.
7. bdcdba7: a \ at the end of a comment continues no import, and tests for the comment-backslash case, a plain import, a \ inside parentheses and the No module named parent cases.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →