template: the verifier's diff shows every edited line, unsafe diffs are refused, and skipped tests count as regressions

[ArjunS07]

Purpose

The audit of 2026-10-08 found three holes in the separate verifier (datasmith#45):

• A .gitattributes file in the agent's diff can hide benchmark edits from the tamper check.
• A diff can add files that run code outside the patched sources.
• Test files come from the agent's diff.

This PR closes the three holes. It also counts a skipped or uncollected test as a regression.

Changes

1. The verifier's diff shows every edited line. fc_diff in test.sh uses --text --no-ext-diff --no-textconv with GIT_NO_REPLACE_OBJECTS=1. The collect command also uses --no-ext-diff --no-textconv.
   • Effect: benchmarks/** -diff in the agent's .gitattributes does not turn a benchmark edit into "Binary files differ", so the tamper check sees the edited lines.
   • Before: the verifier's patch.diff followed the agent's .gitattributes.
2. Unsafe diffs are refused. prepare.sh refuses a tree.diff that creates a symlink or adds *.pth, sitecustomize.py or usercustomize.py. The verifier then uses the starting tree, so the trial has no patch, and it writes agent_tree.txt (tree_diff_refused) and tree_diff_refused.txt.
   • Effect: the agent cannot make Python run its code at startup in the verifier.
   • Before: git apply applied such files.
3. Test files are restored before pytest, and the base side runs on a true base tree. Merged from fix/restore-tests-before-pytest (ad45ffc):
   • Test files the agent changed, added or deleted go back to fc-baseline before the agent-side run, and the list is in test_edits.json.
   • The base side checks out the base version of every changed file, including committed and untracked ones, instead of using git stash. Afterwards it checks that the agent tree is back byte for byte.
   • Effect: the agent cannot weaken the tests that judge it, and a committed change no longer skips the base run.
   • Before: pytest ran the agent's tests, and git stash missed committed changes.
4. Testing helpers are restored, and pytest uses the base configuration. Files under testing/ and _testing/ (for example numpy/testing, pandas/_testing, xarray/testing) count as test code and are restored. Both pytest runs get -c with a copy of the base commit's pytest config file, picked in pytest's order (pytest.ini, .pytest.ini, pyproject.toml, tox.ini, setup.cfg), and --rootdir at the repository. The agent's config edits are listed in test_edits.json as config_changed.
   • Effect: the agent cannot weaken an assertion helper or change the pytest configuration. That includes dotted TOML keys and an added .pytest.ini, because pytest never reads the agent's config files. If the base config cannot be written, the check fails (config:unavailable in regressed).
   • Before: only tests/, conftest.py and pytest.ini were restored.
5. A pytest plugin that only the agent run loads is a regression. Each run records its loaded plugins. A plugin that loads in the agent run but not in the base run (for example a pytest11 entry point from the agent's package metadata) adds plugin:<name> to regressed. prepare.sh also refuses *.dist-info and *.egg-info files in tree.diff.
   • Effect: the agent cannot load a plugin that rewrites failed reports.
   • Before: no check.
6. A refused or broken diff is recorded. parser.py copies agent_tree.txt into reward.json as patch.submission_rejected. Such a trial already ends with the fail penalty, because the patch is empty and failure_reason returns no_patch. The new field gives the specific reason.
7. More outcomes count as a regression. A test that passes at base is now a regression unless it also passes with the agent: failed, error, skipped, xfailed or not collected.
   • Effect: code that calls pytest.skip() or breaks import cannot hide a failing test.
   • Before: only failed and error counted.

Verification

• pytest tests/docker/test_separate_verifier_tree.py tests/docker/test_pytest_runner_regression.py: 24 passed. New tests:
  • .gitattributes with benchmarks/** -diff: the verifier's fc_diff still shows +def time_x(): return 0. With the old plain git diff the same test fails.
  • A diff with evil.pth and a symlink is refused, and the verifier keeps the starting tree.
  • Agent code that calls pytest.skip(), and agent code that breaks import, each report the base-passing test as regressed.
  • A weakened pkg/testing/helpers.py is restored, and the broken change is reported as a regression.
  • With the base config in setup.cfg, an agent pyproject.toml with the dotted key pytest.ini_options.addopts, or an added .pytest.ini, is not read. Both tests fail without -c.
  • A pytest11 entry point from a dist-info folder that the agent adds is reported as plugin:fcevil.
  • A tree.diff with a dist-info file is refused.
  • A sitecustomize.py under a non-ASCII folder (é/) is refused. With the old parsing (without -z), the same test fails.
• pytest tests -k "template or adapter or parser or pytest_runner or separate or overrides": 120 passed.

Notes

• Base: survey/phase1-template-shadow-conftest, which has template: run test.sh in Harbor's separate verifier, with the agent's tree diff as its only input #45. That template is not in use yet; all sessions switch together after the parity runs.
• Still open, not in this PR:
  • agent code runs as root in the verifier (the audit session writes the specification);
  • the agent's package is imported inside the pytest process, so its __init__ can change pytest hooks without any configuration. This is the same kind of hole as root in the verifier;
  • the snapshot gateway port is inside the allowlist range.