diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index 4aaaac66..3dff4035 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -22,6 +22,8 @@ jobs: run: pip install pyyaml --quiet - name: Verify automation quarantine run: bash tests/automation-quarantine-test.sh + - name: Verify fix-outcome verdicts + run: bash tests/outcome-verdict-test.sh - name: Run E2E tests run: bash tests/e2e.sh - name: Verify canonical descriptile policy repair diff --git a/docs/experiments/E1-fix-outcome-replay-2026-10-20.adoc b/docs/experiments/E1-fix-outcome-replay-2026-10-20.adoc new file mode 100644 index 00000000..55a1ca37 --- /dev/null +++ b/docs/experiments/E1-fix-outcome-replay-2026-10-20.adoc @@ -0,0 +1,129 @@ +// SPDX-License-Identifier: CC-BY-SA-4.0 += E1 — Fix-outcome replay: how many "success" rows does a rescan confirm? +:revdate: 2026-10-20 +:toc: preamble + +Experiment E1 of the Kautz-ladder ultraplan (`standards/docs/ULTRAPLAN-KAUTZ-LADDER-2026-10-19.adoc`). +Question: of the rows that `gitbot-fleet/shared-context/learning/fix-outcomes.jsonl` labels +`outcome: success`, how many would a rescan with the fixer's *own* detection predicate confirm? + +== Verdict + +*The `success` label carries no evidence that anything was fixed.* + +* Of the 214 rows that can be replayed here, *0 are confirmed*: +** 21 are refuted: the target was still present when the success was claimed, and still present seven days later. +** 109 are vacuous: there was nothing to fix at claim time. +** 84 cannot be checked, because the repo's history begins after them. +* Across the whole log, the 21,498 success rows collapse to *136 distinct (repo, file, pattern) keys*. 20,169 of the rows carry an empty `file`. +* Root cause: both shell writers set `outcome: "success"` when the fix script exits 0. The scripts exit 0 even when they changed nothing. + +== Data + +[cols="2,1",options="header"] +|=== +| Fact | Value +| Rows / success / failure | 21,585 / 21,498 / 87 +| Writers (`bot`) | hypatia 21,303 · dispatch-runner 270 · robot-repo-automaton 12 +| Distinct success keys (repo, file, pattern) | *136* (hypatia: 21,216 rows → 136 keys) +| Success rows with empty `file` | *20,169* (93.8 %) +| `recipe-shell-quote-vars` success rows on `.yml` | 113 — the fixer never edits YAML (see below) +| Date span | 2026-02-10 → 2026-04-05 +|=== + +The file is a Git LFS object (3.2 MB). It needs `git lfs pull` to read. + +== Method + +* *Replay set.* Every row whose repo is checked out in this session: `standards` (70 rows) and `verisimdb` (144 rows). All 214 have an empty `file`, so the replay is repo-level. +* *Predicate.* Each recipe's own detection rule, taken from the script that `fix-script-registry.json` maps it to, and applied with the script's own third-party exclusions: +** `recipe-shell-quote-vars` (`fix-shell-quoting.sh`): `\b(cd|cp|mv|rm|mkdir|rmdir|chmod|chown|echo) \$[A-Z_][A-Z0-9_]*` in `*.sh` and `*.bash`. +** `recipe-remove-believe-me` (`fix-believe-me.sh`): a non-comment `believe_me` in `*.idr`. This is the script's own selection test, and it is the effect the recipe name claims (removal). Its actual edit is an annotation; see "Recipe name ≠ recipe effect" below. +* *Time points.* The predicate is evaluated with `git grep` on the tree at three commits: the one current at `fixed_at`, the one at `fixed_at + 7 days` (time for a fix PR to land), and `origin/main` HEAD. +* *Script.* `scripts/audit-fix-outcomes.sh` (bash + jq + git). Output: `E1-replay-2026-10-20.tsv`, alongside this report. +* *Sanity checks.* +** The predicate fires on a synthetic `cd $FOO`. +** The verisimdb root commit shows one real hit (`verisimdb-data/scripts/scan-all.sh`), which disappears by 2026-03-22. + +== Results + +[cols="1,2,1,1,1,1",options="header"] +|=== +| Repo | Recipe | Rows | Refuted (still present at claim and +7 d) | Vacuous (nothing to fix at claim) | Not checkable (before history) +| standards | remove-believe-me | 70 | *21* (2026-02-12/13) | 49 | 0 +| verisimdb | remove-believe-me | 70 | 0 | 28 | 42 +| verisimdb | shell-quote-vars | 74 | 0 | 32 | 42 +| *Total* | | *214* | *21* | *109* | *84* +|=== + +Confirmed fixes attributable to the bot: *0 of 130* checkable rows. + +=== What actually removed the standards `believe_me` calls + +They went in commit `8225e55f` (2026-02-27, "Auto-commit: Sync changes", owner). That diff deletes lines such as `(believe_me cb)` together with the bot's `-- PROOF_TODO: Replace believe_me with actual proof` annotation. A human/sync commit did the removal, two weeks after the bot logged success. The 49 standards rows logged after it re-claim success on a repo where the predicate was already clear. + +=== verisimdb + +* `believe_me` never appears in verisimdb's history: `git log -S believe_me -- '*.idr'` returns 0 commits. Every checkable `remove-believe-me` row there is vacuous. +* The repo's history starts on 2026-03-21. It was extracted from `nextgen-databases`, so the 84 earlier rows refer to a tree that no longer exists here. + +== Root causes (structural, independent of the sample) + +. *Exit code taken as outcome.* +** `run-fleet.sh:459-475` writes `success` when `"$fix_path" …` exits 0. +** `fleet-coordinator.sh:309-315` writes `success` when `fixed_count > 0`, with `pattern: "auto-fix"` and no file. +** Neither one re-scans afterwards. +** `fix-shell-quoting.sh` exits 0 after "No fixable patterns found". +** `fix-believe-me.sh` exits 0 after only annotating. +. *Recipe name ≠ recipe effect.* `recipe-remove-believe-me` never removes anything. It inserts a `PROOF_TODO` comment above each call, so a rescan for `believe_me` is unchanged by design. +. *Collected ≠ edited.* `fix-shell-quoting.sh` collects `.github/workflows/*.yml`, but its edit loop only handles `*.sh` and `*.bash`. That accounts for the 113 `.yml` success rows. +. *Repeat logging.* The same repo-level claim is re-appended on every pass: about 156 rows per key on average. +. *Schema drift.* +** The log rows (`bot, file, fixed_at, outcome, pattern, repo`) match neither shell writer's schema (`timestamp, fix_script, severity` and `count` respectively), so a third writer (Hypatia-side) produced 98.7 % of them. +** `learning-monitor.sh:77` counts `select(.success == true)`, a field no row has, so its success count is always 0. + +== Consequences + +* Anything that learns confidence from this log learns from a label that is "the script didn't crash". The safety triangle's `auto_execute ≥ 0.95` gate is exactly where that would bite. +* This is consistent with the automation quarantine (exit 78) staying in place, and it supports the ruling that the policy interpreter comes first. + +== What changed (gitbot-fleet, same PR as this report) + +* *Verdicts, not exit codes.* `scripts/lib/outcome-verdict.sh` computes a verdict from the recipe's predicate before and after the fix: +** `verified`: work found before, none after. +** `no_op`: nothing to fix before, and nothing after. +** `incomplete`: work remains after the run, including targets that appear during it (0 → n). Run summaries count it with the failures. +** `unverified`: no predicate. +** `failed`: non-zero exit. +* *Writers.* +** `dispatch-runner.sh` and `run-fleet.sh` record `verdict`, `exit_code`, `before_hits` and `after_hits`. +** `outcome` stays, for older readers (Hypatia reads these rows), and reports the exit status only. +** `fleet-coordinator.sh`'s batch generation is recorded as `generated` / `unverified`: nothing has run at that point. +* *Hypatia closed loop.* `verified` maps to `success`; `failed` and `incomplete` map to `failure`; `no_op` and `unverified` are not reported, so they cannot move a recipe's confidence. Only `verified` repos trigger the post-run rescan. +* *Predicates live in the fixers.* +** ` --check ` prints the count of remaining targets and changes nothing. Shell quoting and believe-me have it now. +** A script opts in with the line `# outcome-predicate: --check`. Unmarked fixers are never run with `--check`, and their verdict is `unverified`. +** A predicate that cannot run (a `find` or `grep` error, no PCRE) exits non-zero rather than printing 0, so the verdict is `unverified`, not `verified`. +** Under `--parallel`, `dispatch-runner.sh` holds a per-repo lock around check, fix and check. Two jobs on one repo otherwise both count the targets before either fix and both record `verified`. +* *Rename.* +** `recipe-annotate-believe-me` is the canonical id. `recipe-remove-believe-me` stays as a registry alias, listed under `deprecated_recipe_ids`, because Hypatia still emits it. `dispatch-runner.sh` reports outcomes under the canonical id, so a verified annotation never raises the removal recipe's confidence. +** The script's predicate measures what it actually does: un-annotated `believe_me` calls. E1 above measured the claimed effect (removal). +* *Reporting.* `learning-monitor.sh` counted a `.success` field that no row has. It now reports per verdict, with pre-verdict rows shown as legacy. +* *Replay.* `scripts/audit-fix-outcomes.sh` reruns this experiment. It reads the rows of all three writers. It excludes the fixers' own third-party directories (`lib/third-party-excludes.sh`), and reports NA, not 0, when `git grep` cannot run. Rerun with those changes, it reproduces `E1-replay-2026-10-20.tsv` byte for byte. It measures `recipe-remove-believe-me` by removal and `recipe-annotate-believe-me` by annotation, as each claims. `E1-replay-2026-10-20.tsv` is its output for the rows above. The HEAD column moves as the repos do. + +All of this sits behind the automation quarantine (exit 78), which this change does not lift. It is a precondition for lifting it. + +== Still open + +* *Hypatia.* Its rows (98.7 % of the log) come from Hypatia's own writer, which is outside this repo. It should read `verdict` and stop writing `success` for unverified fixes. +* *Remaining predicates.* `recipe-heredoc-to-install`, `recipe-todo-to-fill` and the other fixers have no `--check` yet, so their verdict is `unverified`. +* *Row keys.* Rows should be appended once per (repo, file, pattern, rev), with a non-empty `file` or an explicit `scope: repo`. +* *History.* Historical rows stay as written. The log is evidence; read them as unverified. + +Type/proof angle: the verdict is a function of `(predicate, tree_before, tree_after)`, not of the fixer. Make it a total function in the policy interpreter, with no input the fixer controls. Then "success" can't be asserted, only computed. This is the same shape as the HYP-01 `sound` level. + +== Limits + +* Only 214 of 21,585 rows (1 %) could be replayed. Only 2 of 78 repos are in this session's scope, and both are repo-level rows. The structural root causes apply to the whole log regardless. +* The predicates are the fixers' own. A row that a stronger checker would call fixed but the fixer's predicate would not is out of scope. The opposite case (fixer's predicate clear, real bug remains) is also not measured. +* `heredoc-to-install` and `todo-to-fill` (4,608 rows) have no row in a checked-out repo, so they were not replayed. diff --git a/docs/experiments/E1-replay-2026-10-20.tsv b/docs/experiments/E1-replay-2026-10-20.tsv new file mode 100644 index 00000000..9d5c5a4e --- /dev/null +++ b/docs/experiments/E1-replay-2026-10-20.tsv @@ -0,0 +1,215 @@ +repo pattern outcome fixed_at rev_at hits_at hits_plus7d hits_head +standards recipe-remove-believe-me success 2026-02-12T23:14:48Z f6a8cfff8 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T13:22:33.573423Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:00:44.080831Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:02:45.476032Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:04:17.740959Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:05:09.709017Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:06:40.391612Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:07:02.072825Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:07:20.702698Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:07:48.993689Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:08:28.024000Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:08:52.699902Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:42:48.668144Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T14:45:16.647618Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T15:15:51.030098Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T15:16:35.995678Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T15:17:22.982466Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T15:18:00.426933Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T15:19:08.026339Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T15:20:17.415955Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-02-13T15:25:38.969125Z 414c53e29 3 3 0 +standards recipe-remove-believe-me success 2026-03-08T01:31:47.968291Z acb1a4221 0 0 0 +standards recipe-remove-believe-me success 2026-03-08T01:33:56.087592Z acb1a4221 0 0 0 +standards recipe-remove-believe-me success 2026-03-12T01:12:10.147484Z 2db2759c8 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:06:48.286430Z 9c783ddb6 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:09:13.326291Z 9c783ddb6 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:13:29.508872Z 9c783ddb6 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:16:12.598814Z 9c783ddb6 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:19:23.565139Z 9c783ddb6 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:21:53.438296Z e6c6c831b 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:34:03.597425Z 13214fe9e 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:36:57.503956Z 13214fe9e 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:40:18.451014Z 13214fe9e 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:43:11.677515Z 13214fe9e 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:46:11.544352Z 84c102265 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:49:14.985985Z 9da19bf7b 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:52:07.740311Z 9da19bf7b 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:55:00.065961Z 9da19bf7b 0 0 0 +standards recipe-remove-believe-me success 2026-03-16T00:57:15.690676Z 6f96c8d2f 0 0 0 +standards recipe-remove-believe-me success 2026-03-20T23:16:47.870923Z 65da2018a 0 0 0 +standards recipe-remove-believe-me success 2026-03-20T23:19:02.351712Z 65da2018a 0 0 0 +standards recipe-remove-believe-me success 2026-03-20T23:22:34.130533Z 65da2018a 0 0 0 +standards recipe-remove-believe-me success 2026-03-22T13:49:32.519362Z 1af824499 0 0 0 +standards recipe-remove-believe-me success 2026-03-22T13:57:12.129440Z 1af824499 0 0 0 +standards recipe-remove-believe-me success 2026-03-22T14:01:40.437048Z 1af824499 0 0 0 +standards recipe-remove-believe-me success 2026-03-22T22:27:03.615303Z 1af824499 0 0 0 +standards recipe-remove-believe-me success 2026-03-22T22:28:57.654105Z 1af824499 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T11:19:08.850609Z 4e1aed49e 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T11:24:43.345019Z 4e1aed49e 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T11:28:01.129262Z 4e1aed49e 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T11:32:45.254277Z 4e1aed49e 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T20:57:19.875976Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T20:59:16.067014Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T21:05:19.051634Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T21:08:06.563270Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T21:11:01.738789Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T21:15:31.938988Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T21:24:42.007110Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T21:30:01.892450Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-24T21:35:30.439588Z 509acf311 0 0 0 +standards recipe-remove-believe-me success 2026-03-28T07:50:13.392365Z 080397d28 0 0 0 +standards recipe-remove-believe-me success 2026-03-28T07:52:41.174647Z 080397d28 0 0 0 +standards recipe-remove-believe-me success 2026-03-30T03:12:22.447529Z 75c1f9077 0 0 0 +standards recipe-remove-believe-me success 2026-03-30T03:15:58.731543Z 75c1f9077 0 0 0 +standards recipe-remove-believe-me success 2026-03-30T03:22:45.435670Z 75c1f9077 0 0 0 +standards recipe-remove-believe-me success 2026-03-30T03:28:25.783365Z 75c1f9077 0 0 0 +standards recipe-remove-believe-me success 2026-04-03T14:16:23.420489Z 888a30486 0 0 0 +standards recipe-remove-believe-me success 2026-04-03T14:20:24.616145Z 888a30486 0 0 0 +standards recipe-remove-believe-me success 2026-04-03T14:24:04.177899Z 888a30486 0 0 0 +standards recipe-remove-believe-me success 2026-04-05T03:06:41.677942Z dc184309e 0 0 0 +verisimdb recipe-remove-believe-me success 2026-02-12T23:14:48Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T13:22:33.585044Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:00:44.102576Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:02:45.561363Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:04:17.902703Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:05:09.884520Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:06:40.413536Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:07:02.105653Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:07:20.741550Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:07:49.118160Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:08:28.086859Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:08:52.771373Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:42:48.886164Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T14:45:16.738439Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T15:15:51.083856Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T15:16:36.057938Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T15:17:23.044773Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T15:18:00.519911Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T15:19:08.104144Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T15:20:17.489471Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-02-13T15:25:39.065242Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-08T01:31:48.360403Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-08T01:33:56.484791Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-12T01:12:10.909077Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:06:48.791175Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:09:13.797857Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:13:30.143194Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:16:13.237123Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:19:24.230848Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:21:53.999500Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:34:04.097828Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:36:58.018241Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:40:19.402845Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:43:12.256821Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:46:12.214353Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:49:15.564986Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:52:08.523765Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:55:00.769459Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-16T00:57:16.381934Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-20T23:16:48.368015Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-20T23:19:02.845978Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-20T23:22:34.604644Z - pre-history - 0 +verisimdb recipe-remove-believe-me success 2026-03-22T13:49:33.100972Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-22T13:57:12.523454Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-22T14:01:40.803081Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-22T22:27:04.279480Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-22T22:28:58.382684Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T11:19:09.434004Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T11:24:44.185713Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T11:28:01.890394Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T11:32:46.123494Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T20:57:20.293570Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T20:59:16.791631Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T21:05:19.453493Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T21:08:06.909599Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T21:11:02.112680Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T21:15:32.308605Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T21:24:42.374698Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T21:30:02.292537Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-24T21:35:30.910557Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-28T07:50:13.890490Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-28T07:52:41.726322Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-30T03:12:23.040238Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-30T03:15:59.135302Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-30T03:22:45.814258Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-03-30T03:28:26.190776Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-04-03T14:16:24.043316Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-04-03T14:20:25.303969Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-04-03T14:24:04.713626Z df526526a 0 0 0 +verisimdb recipe-remove-believe-me success 2026-04-05T03:06:41.949254Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-02-12T23:09:29Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T13:22:33.148733Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:00:38.980724Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:02:40.709154Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:04:12.651838Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:05:05.496246Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:06:39.555763Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:07:00.905784Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:07:19.696438Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:07:44.856178Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:08:25.998136Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:08:50.690971Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:42:39.963503Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T14:45:14.170131Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T15:15:49.320702Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T15:16:34.370523Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T15:17:21.207361Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T15:17:57.791136Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T15:19:05.547737Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T15:20:14.679164Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-02-13T15:25:36.123310Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-08T01:31:32.769821Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-08T01:33:43.645126Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-12T01:11:44.008470Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:06:33.599934Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:08:56.603898Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:13:12.523280Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:15:53.592116Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:19:06.341698Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:21:37.620703Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:33:44.941833Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:36:36.836008Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:39:54.933499Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:42:53.114948Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:45:52.189128Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:48:53.939978Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:51:46.698519Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:54:41.405390Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-16T00:56:54.872860Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-20T23:16:30.302393Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-20T23:18:43.110632Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-20T23:22:16.571300Z - pre-history - 0 +verisimdb recipe-shell-quote-vars success 2026-03-22T13:49:15.166837Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-22T13:56:58.775607Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-22T14:01:26.754660Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-22T22:26:43.678604Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-22T22:28:32.448779Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T11:06:36.319279Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T11:11:37.628221Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T11:18:50.322830Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T11:24:16.656722Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T11:27:33.821597Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T11:32:11.371900Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T11:35:00.147560Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T20:57:05.928645Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T20:58:59.205122Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T21:05:05.312961Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T21:07:53.663513Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T21:10:48.259000Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T21:15:18.953955Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T21:24:28.123716Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T21:29:48.692311Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-24T21:35:13.524760Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-28T07:49:55.052951Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-28T07:52:23.342273Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-30T03:12:07.134910Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-30T03:15:44.248316Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-30T03:22:31.497129Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-03-30T03:28:12.395939Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-04-03T14:15:58.277659Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-04-03T14:20:03.262249Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-04-03T14:23:44.851294Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-04-03T23:44:03.953921Z df526526a 0 0 0 +verisimdb recipe-shell-quote-vars success 2026-04-05T03:06:38.618528Z df526526a 0 0 0 diff --git a/fleet-coordinator.sh b/fleet-coordinator.sh index e186faf3..c46742ec 100755 --- a/fleet-coordinator.sh +++ b/fleet-coordinator.sh @@ -305,13 +305,15 @@ process_findings() { local fixed_count=$(execute_auto_fixes "$repo_name" "$latest_findings") total_auto_fixed=$((total_auto_fixed + fixed_count)) - # Record outcomes for completed fixes + # execute_auto_fixes only *generates* fix-batch scripts; nothing + # has run yet, so this is not a fix outcome. Record it as + # generated/unverified rather than "success" (experiment E1). if [[ $fixed_count -gt 0 ]]; then - log_bot "hypatia" "Recording $fixed_count fix outcomes to learning pipeline" + log_bot "hypatia" "Recording $fixed_count generated fix batches (unverified) to learning pipeline" # Use jq to safely construct JSON (prevents injection via repo_name) - jq -n --arg repo "$repo_name" --argjson count "$fixed_count" \ + jq -cn --arg repo "$repo_name" --argjson count "$fixed_count" \ --arg fixed_at "$(date -Iseconds)" \ - '{pattern: "auto-fix", repo: $repo, outcome: "success", count: $count, fixed_at: $fixed_at, bot: "robot-repo-automaton"}' \ + '{pattern: "auto-fix", repo: $repo, outcome: "generated", verdict: "unverified", count: $count, fixed_at: $fixed_at, bot: "robot-repo-automaton"}' \ >> "$SHARED_CONTEXT/learning/fix-outcomes.jsonl" fi fi diff --git a/learning-monitor.sh b/learning-monitor.sh index d4d7dde9..e292cdc9 100755 --- a/learning-monitor.sh +++ b/learning-monitor.sh @@ -73,15 +73,26 @@ fi echo "" echo -e "${CYAN}🔧 Auto-Fix Outcomes${NC}" if [[ -f "$LEARNING_DIR/fix-outcomes.jsonl" ]]; then + # Only a predicate-confirmed fix counts as successful. Rows written + # before verdicts existed record an exit status only: they are reported + # separately as legacy, never as successes (experiment E1). total_fixes=$(wc -l < "$LEARNING_DIR/fix-outcomes.jsonl") - successful=$(jq -r 'select(.success == true)' "$LEARNING_DIR/fix-outcomes.jsonl" | wc -l) - failed=$((total_fixes - successful)) + count_verdict() { + { jq -r --arg v "$1" 'select(.verdict == $v) | 1' "$LEARNING_DIR/fix-outcomes.jsonl" 2>/dev/null || true; } | wc -l + } + verified=$(count_verdict verified) + incomplete=$(count_verdict incomplete) + failed=$(count_verdict failed) + no_op=$(count_verdict no_op) + unverified=$(count_verdict unverified) + legacy=$( { jq -r 'select(has("verdict") | not) | 1' "$LEARNING_DIR/fix-outcomes.jsonl" 2>/dev/null || true; } | wc -l) if [[ $total_fixes -gt 0 ]]; then - success_rate=$(awk "BEGIN {printf \"%.1f\", ($successful/$total_fixes)*100}") - echo -e " Total fixes attempted: $total_fixes" - echo -e " Successful: ${GREEN}$successful${NC} (${success_rate}%)" - echo -e " Failed: ${RED}$failed${NC}" + echo -e " Total outcome rows: $total_fixes" + echo -e " Verified (predicate confirms): ${GREEN}$verified${NC}" + echo -e " Incomplete / failed: ${RED}$incomplete${NC} / ${RED}$failed${NC}" + echo -e " No-op / unverified: $no_op / ${YELLOW}$unverified${NC}" + echo -e " Legacy (exit status only, no evidence): ${YELLOW}$legacy${NC}" fi else echo -e " ${YELLOW}No fix attempts yet${NC}" diff --git a/run-fleet.sh b/run-fleet.sh index a5972854..de146364 100755 --- a/run-fleet.sh +++ b/run-fleet.sh @@ -294,6 +294,11 @@ cmd_fix() { return 78 fi + # Fix outcomes are verdicts from each recipe's predicate, not exit codes. + # Sourced after the quarantine gate so the gate never depends on it. + # shellcheck source=scripts/lib/outcome-verdict.sh + source "$SCRIPTS_DIR/lib/outcome-verdict.sh" + preflight log_head "Auto-Fix Pipeline" @@ -453,27 +458,34 @@ cmd_fix() { echo "$finding" > "$tmp_finding" printf " [FIX] %-40s " "$fpattern" - if "$fix_path" "$repo_path" "$tmp_finding" >/dev/null 2>&1; then - echo -e "${GREEN}OK${NC}" - ((total_applied++)) || true - - # Log outcome - jq -n --arg repo "$repo_name" --arg pattern "$fpattern" \ - --arg script "$fix_script" --arg ts "$(timestamp)" \ - --arg severity "$fseverity" \ - '{timestamp: $ts, repo: $repo, pattern: $pattern, fix_script: $script, severity: $severity, outcome: "success"}' \ - >> "$LEARNING_DIR/fix-outcomes.jsonl" - else - echo -e "${RED}FAILED${NC}" + local before after rc verdict + before=$(recipe_check_count "$fix_path" "$repo_path") + rc=0 + "$fix_path" "$repo_path" "$tmp_finding" >/dev/null 2>&1 || rc=$? + after=$(recipe_check_count "$fix_path" "$repo_path") + verdict=$(outcome_verdict "$rc" "$before" "$after") + + if [[ "$verdict" == failed || "$verdict" == incomplete ]]; then + echo -e "${RED}${verdict^^}${NC} ($before -> $after)" ((total_failed++)) || true - - jq -n --arg repo "$repo_name" --arg pattern "$fpattern" \ - --arg script "$fix_script" --arg ts "$(timestamp)" \ - --arg severity "$fseverity" \ - '{timestamp: $ts, repo: $repo, pattern: $pattern, fix_script: $script, severity: $severity, outcome: "failure"}' \ - >> "$LEARNING_DIR/fix-outcomes.jsonl" + else + echo -e "${GREEN}${verdict}${NC} ($before -> $after)" + ((total_applied++)) || true fi + # `outcome` is the exit status, kept for older readers; + # `verdict` is whether the predicate confirms the fix. + jq -cn --arg repo "$repo_name" --arg pattern "$fpattern" \ + --arg script "$fix_script" --arg ts "$(timestamp)" \ + --arg severity "$fseverity" --arg verdict "$verdict" \ + --arg outcome "$(legacy_outcome "$verdict")" \ + --argjson exit_code "$rc" --arg before "$before" --arg after "$after" \ + '{timestamp: $ts, repo: $repo, pattern: $pattern, fix_script: $script, severity: $severity, + outcome: $outcome, verdict: $verdict, exit_code: $exit_code, + before_hits: (if $before == "NA" then null else ($before | tonumber) end), + after_hits: (if $after == "NA" then null else ($after | tonumber) end)}' \ + >> "$LEARNING_DIR/fix-outcomes.jsonl" + rm -f "$tmp_finding" fi done <<< "$auto_findings" diff --git a/scripts/audit-fix-outcomes.sh b/scripts/audit-fix-outcomes.sh new file mode 100755 index 00000000..32b954a5 --- /dev/null +++ b/scripts/audit-fix-outcomes.sh @@ -0,0 +1,69 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# +# audit-fix-outcomes.sh — replay fix-outcomes rows against git history (E1). +# +# For every row of a fix-outcomes log whose repo is checked out locally, this +# evaluates the recipe's own detection predicate on the tree at three commits: +# the one current at fixed_at, the one at fixed_at + 7 days (time for a fix PR +# to land), and origin/main HEAD. It writes TSV to stdout and changes nothing. +# Report: docs/experiments/E1-fix-outcome-replay-2026-10-20.adoc. +# +# Usage: audit-fix-outcomes.sh ... +# +# The log in shared-context/learning/ is a Git LFS pointer; fetch the object +# into a scratch path first rather than over the committed pointer. +# The checkouts need full history (git fetch --unshallow). +# +# Predicates mirror the fixers' own (their --check modes, applied with +# `git grep` to a revision instead of a working tree). Recipes without one +# print NA. +set -euo pipefail +LOG=$1; shift +# The fixers' own exclusion list (lib/third-party-excludes.sh), as glob +# pathspecs: with `glob` magic `**/` also matches no directory, so a +# root-level vendor/ is excluded exactly as the fixers' find excludes it. +# shellcheck source=lib/third-party-excludes.sh +source "$(dirname "${BASH_SOURCE[0]}")/lib/third-party-excludes.sh" +EX=() +for d in "${THIRD_PARTY_SKIP_DIRS[@]}"; do EX+=(":(exclude,glob)**/$d/**"); done +IDR_EX=("${EX[@]}" ':(exclude,glob)**/.pack/**') # fix-believe-me.sh also skips .pack +# grep_hits : the git grep output, or exit 1 when +# git grep itself failed (status > 1: no PCRE, unreadable revision). Status 1 +# is "no match", which is a count of 0, not an error. +grep_hits() { + local repo=$1 rc=0 out; shift + out=$(git -C "$repo" grep "$@" 2>/dev/null) || rc=$? + ((rc <= 1)) || return 1 + printf '%s\n' "$out" +} +sum_counts() { awk -F: 'NF { s += $NF } END { print s + 0 }'; } +pred() { # repo rev pattern -> count of remaining hits, or NA + local repo=$1 rev=$2 pat=$3 out + case $pat in + recipe-shell-quote-vars) + out=$(grep_hits "$repo" -P -h -c '\b(cd|cp|mv|rm|mkdir|rmdir|chmod|chown|echo) \$[A-Z_][A-Z0-9_]*' "$rev" -- '*.sh' '*.bash' "${EX[@]}") || { echo NA; return; } + sum_counts <<< "$out";; + # The removal recipe claimed to remove believe_me, so an annotated call + # still counts against it. The annotate recipe is measured as its fixer's + # --check measures it: a call is discharged by a PROOF_TODO line above. + recipe-remove-believe-me) + out=$(grep_hits "$repo" -P -h -c '^(?!\s*--).*believe_me' "$rev" -- '*.idr' "${IDR_EX[@]}") || { echo NA; return; } + sum_counts <<< "$out";; + recipe-annotate-believe-me) + out=$(grep_hits "$repo" -h -B1 -e believe_me "$rev" -- '*.idr' "${IDR_EX[@]}") || { echo NA; return; } + awk '/^--$/ { prev = ""; next } !/^[[:space:]]*--/ && /believe_me/ && prev !~ /PROOF_TODO/ { n++ } { prev = $0 } END { print n + 0 }' <<< "$out";; + *) echo NA;; + esac +} +printf 'repo\tpattern\toutcome\tfixed_at\trev_at\thits_at\thits_plus7d\thits_head\n' +for repo in "$@"; do + name=$(basename "$repo"); head=$(git -C "$repo" rev-parse origin/main) + jq -r --arg r "$name" 'select(.repo==$r) | [.pattern // .recipe_id // .pattern_id, .outcome, .fixed_at // .timestamp] | select(.[2] != null) | @tsv' "$LOG" | sort -u | + while IFS=$'\t' read -r pat out ts; do + at=$(git -C "$repo" rev-list -1 --before="$ts" origin/main || true) + p7=$(git -C "$repo" rev-list -1 --before="$(date -u -d "$ts + 7 days" +%FT%TZ)" origin/main || true) + [ -z "$at" ] && { printf '%s\t%s\t%s\t%s\t-\tpre-history\t-\t%s\n' "$name" "$pat" "$out" "$ts" "$(pred "$repo" "$head" "$pat")"; continue; } + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$name" "$pat" "$out" "$ts" "${at:0:9}" "$(pred "$repo" "$at" "$pat")" "$(pred "$repo" "$p7" "$pat")" "$(pred "$repo" "$head" "$pat")" + done +done diff --git a/scripts/dispatch-runner.sh b/scripts/dispatch-runner.sh index 82b1762b..ef7ff6ae 100755 --- a/scripts/dispatch-runner.sh +++ b/scripts/dispatch-runner.sh @@ -155,6 +155,17 @@ mkdir -p "$(dirname "$OUTCOME_FILE")" # waiting for the nightly sweep. RESCAN_REPOS_FILE="$(mktemp /tmp/dispatch-rescan-repos.XXXXXX)" +# Verdicts come from the recipe's predicate, not the fix script's exit code. +# shellcheck source=lib/outcome-verdict.sh +source "$(dirname "${BASH_SOURCE[0]}")/lib/outcome-verdict.sh" + +# Parallel jobs on one repo take a per-repo lock around check/fix/check +# (repo_lock), so --parallel needs flock. Refuse rather than run unlocked. +if [[ "$PARALLEL" -gt 1 ]] && ! command -v flock &>/dev/null; then + echo "ERROR: --parallel $PARALLEL needs flock (util-linux) to serialise fixes per repo" >&2 + exit 1 +fi + # --- Hypatia closed-loop contract --- # # Per hypatia/docs/operations/dispatch-runner-contract.adoc, after each @@ -237,14 +248,28 @@ hypatia_record_outcome() { return "$rc" } +# record_outcome [exit before after] +# +# is one of lib/outcome-verdict.sh's verdicts, or refused_* for a +# policy refusal. `outcome` is kept for older readers and reports the exit +# status only; `verdict` says whether anything was fixed. record_outcome() { local pattern_id="$1" local recipe_id="$2" local repo="$3" local file="$4" - local outcome="$5" + local verdict="$5" + local exit_code="${6:-}" + local before="${7:-NA}" + local after="${8:-NA}" local bot="dispatch-runner" + local outcome + case "$verdict" in + refused_*) outcome="$verdict" ;; + *) outcome=$(legacy_outcome "$verdict") ;; + esac + local ts ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ") @@ -255,9 +280,18 @@ record_outcome() { --arg repo "$repo" \ --arg file "$file" \ --arg outcome "$outcome" \ + --arg verdict "$verdict" \ + --arg exit_code "$exit_code" \ + --arg before "$before" \ + --arg after "$after" \ --arg ts "$ts" \ --arg bot "$bot" \ - '{pattern_id: $pid, recipe_id: $rid, repo: $repo, file: $file, outcome: $outcome, timestamp: $ts, bot: $bot}') + '{pattern_id: $pid, recipe_id: $rid, repo: $repo, file: $file, outcome: $outcome, + verdict: $verdict, + exit_code: (if $exit_code == "" then null else ($exit_code | tonumber) end), + before_hits: (if $before == "NA" then null else ($before | tonumber) end), + after_hits: (if $after == "NA" then null else ($after | tonumber) end), + timestamp: $ts, bot: $bot}') echo "$json" >> "$OUTCOME_FILE" # Avoid duplicate JSONL records when the two configured stores are the same. @@ -270,18 +304,20 @@ record_outcome() { fi # Hypatia closed-loop contract — best-effort, never fails the dispatch - # loop. Map shell outcomes to the contract's outcome vocabulary: - # success/failure → pass through - # refused_* → false_positive (the recipe is misfiring, not - # the code; surfaces this to auto-quarantine) - # anything else → skip (don't poison the metric). + # loop. Map verdicts to the contract's outcome vocabulary: + # verified → success + # failed / incomplete → failure (the fix ran but the work remains) + # refused_* → false_positive (the recipe is misfiring, not + # the code; surfaces this to auto-quarantine) + # no_op / unverified → skip: no evidence either way, so it must not + # move the recipe's confidence. local hypatia_outcome - case "$outcome" in - success) + case "$verdict" in + verified) hypatia_outcome="success" echo "$repo" >> "$RESCAN_REPOS_FILE" ;; - failure) + failed|incomplete) hypatia_outcome="failure" ;; refused_license_policy|refused_*) @@ -304,6 +340,9 @@ execute_entry() { repo=$(echo "$entry" | jq -r '.repo') pattern_id=$(echo "$entry" | jq -r '.pattern_id') recipe_id=$(echo "$entry" | jq -r '.recipe_id // "none"') + # Report under the canonical id: a deprecated alias must not earn credit + # for the recipe it was renamed from. + recipe_id=$(canonical_recipe_id "$FLEET_SCRIPTS/fix-script-registry.json" "$recipe_id") confidence=$(echo "$entry" | jq -r '.confidence // 0') auto_fixable=$(echo "$entry" | jq -r '.auto_fixable // false') fix_script=$(echo "$entry" | jq -r '.fix_script // "none"') @@ -424,31 +463,45 @@ execute_entry() { tmp_finding=$(mktemp /tmp/dispatch-finding-XXXXXX.json) echo "$entry" > "$tmp_finding" + local before after rc verdict + if [[ "$PARALLEL" -gt 1 ]]; then repo_lock "$PARALLEL_TMPDIR" "$repo_path"; fi + before=$(recipe_check_count "$FLEET_SCRIPTS/$fix_script" "$repo_path") + echo " Running: $fix_script $repo_path $tmp_finding" - if "$FLEET_SCRIPTS/$fix_script" "$repo_path" "$tmp_finding" 2>&1 | sed 's/^/ /'; then - echo " OK" - ((SUCCEEDED++)) || true - record_outcome "$pattern_id" "$recipe_id" "$repo" "" "success" - else - echo " FAILED (exit $?)" + rc=0 + "$FLEET_SCRIPTS/$fix_script" "$repo_path" "$tmp_finding" 2>&1 | sed 's/^/ /' \ + || rc=${PIPESTATUS[0]} + + after=$(recipe_check_count "$FLEET_SCRIPTS/$fix_script" "$repo_path") + if [[ "$PARALLEL" -gt 1 ]]; then repo_unlock; fi + verdict=$(outcome_verdict "$rc" "$before" "$after") + echo " $verdict (exit $rc, targets $before -> $after)" + if [[ "$verdict" == failed || "$verdict" == incomplete ]]; then ((FAILED++)) || true - record_outcome "$pattern_id" "$recipe_id" "$repo" "" "failure" + else + ((SUCCEEDED++)) || true fi + record_outcome "$pattern_id" "$recipe_id" "$repo" "" "$verdict" "$rc" "$before" "$after" rm -f "$tmp_finding" elif command -v "$RRA_BIN" &>/dev/null; then # Use robot-repo-automaton fix command echo " Running: $RRA_BIN fix --repo $repo_path --commit" - if "$RRA_BIN" fix --repo "$repo_path" --commit 2>&1 | sed 's/^/ /'; then - echo " OK" - ((SUCCEEDED++)) || true - record_outcome "$pattern_id" "$recipe_id" "$repo" "" "success" - else - echo " FAILED (exit $?)" + # robot-repo-automaton exposes no predicate, so a clean exit + # is recorded as unverified, never as a confirmed fix. + local rc=0 + "$RRA_BIN" fix --repo "$repo_path" --commit 2>&1 | sed 's/^/ /' \ + || rc=${PIPESTATUS[0]} + local verdict + verdict=$(outcome_verdict "$rc" NA NA) + echo " $verdict (exit $rc)" + if [[ "$verdict" == failed ]]; then ((FAILED++)) || true - record_outcome "$pattern_id" "$recipe_id" "$repo" "" "failure" + else + ((SUCCEEDED++)) || true fi + record_outcome "$pattern_id" "$recipe_id" "$repo" "" "$verdict" "$rc" else echo " SKIP: No fix mechanism available (no fix_script, no $RRA_BIN)" ((SKIPPED++)) || true diff --git a/scripts/fix-believe-me.sh b/scripts/fix-believe-me.sh index 425de2a3..193f7c92 100755 --- a/scripts/fix-believe-me.sh +++ b/scripts/fix-believe-me.sh @@ -7,16 +7,36 @@ # believe_me bypasses the type checker — it should be replaced with proper proofs. # # Usage: fix-believe-me.sh +# outcome-predicate: --check set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" source "$SCRIPT_DIR/lib/third-party-excludes.sh" 2>/dev/null || true +# --check : print how many believe_me calls still lack a PROOF_TODO +# on the line above, change nothing. This script annotates; it never removes a +# believe_me, so annotation is what its predicate measures (recipe id +# recipe-annotate-believe-me; see lib/outcome-verdict.sh). +if [[ "${1:-}" == "--check" ]]; then + [[ -d "${2:-}" ]] || { echo "--check: not a directory: ${2:-}" >&2; exit 2; } + # One count per xargs batch. A find or awk failure exits non-zero here, + # so the caller records NA instead of a zero that would read as "fixed". + counts=$(find "$2" -type f -name "*.idr" -not -path "*/.git/*" \ + "${FIND_THIRD_PARTY_EXCLUDES[@]}" -not -path "*/.pack/*" -print0 \ + | xargs -0 -r awk 'FNR == 1 { prev = "" } + $0 !~ /^[[:space:]]*--/ && /believe_me/ && prev !~ /PROOF_TODO/ { n++ } + { prev = $0 } + END { print n + 0 }') \ + || { echo "--check: predicate failed" >&2; exit 2; } + printf '%s\n' "$counts" | awk '{ s += $1 } END { print s + 0 }' + exit 0 +fi + REPO_PATH="${1:?Usage: $0 }" FINDING_JSON="${2:?Missing finding JSON file}" -echo "=== believe_me Removal ===" +echo "=== believe_me Annotation ===" echo " Repo: $REPO_PATH" FIXED_COUNT=0 diff --git a/scripts/fix-script-registry.json b/scripts/fix-script-registry.json index d9caf09d..473baa6d 100644 --- a/scripts/fix-script-registry.json +++ b/scripts/fix-script-registry.json @@ -1,6 +1,9 @@ { "license": "MPL-2.0", "description": "Maps recipe IDs and categories to their fix scripts", + "deprecated_recipe_ids": { + "recipe-remove-believe-me": "recipe-annotate-believe-me" + }, "registry": { "by_category": { "CommandInjection": "fix-command-injection.sh", @@ -79,6 +82,7 @@ "recipe-scorecard-token-permissions": "fix-workflow-permissions.sh", "recipe-scorecard-license": "fix-license-file.sh", "recipe-deno-least-privilege": "fix-deno-permissions.sh", + "recipe-annotate-believe-me": "fix-believe-me.sh", "recipe-remove-believe-me": "fix-believe-me.sh", "recipe-pin-github-actions": "fix-unpinned-actions.sh", "recipe-fix-spdx-license": "fix-missing-spdx.sh", diff --git a/scripts/fix-shell-quoting.sh b/scripts/fix-shell-quoting.sh index 16c0a5e7..1837eeb7 100755 --- a/scripts/fix-shell-quoting.sh +++ b/scripts/fix-shell-quoting.sh @@ -7,12 +7,29 @@ # Targets PA009-shell-unquoted-var findings. # # Usage: fix-shell-quoting.sh +# outcome-predicate: --check set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" source "$SCRIPT_DIR/lib/third-party-excludes.sh" 2>/dev/null || true +# --check : print how many expansions the edits below would quote, +# change nothing. This is the recipe's predicate (see lib/outcome-verdict.sh). +if [[ "${1:-}" == "--check" ]]; then + [[ -d "${2:-}" ]] || { echo "--check: not a directory: ${2:-}" >&2; exit 2; } + # grep exits 1 for a batch with no match, which is a count, not an error. + # Any other failure (find, grep 2, no PCRE) exits non-zero here, so the + # caller records NA instead of a zero that would read as "fixed". + hits=$(find "$2" -type f \( -name "*.sh" -o -name "*.bash" \) \ + -not -path "*/.git/*" "${FIND_THIRD_PARTY_EXCLUDES[@]}" -print0 \ + | xargs -0 -r sh -c 'grep -ohP -- "$0" "$@"; [ $? -le 1 ]' \ + '\b(cd|cp|mv|rm|mkdir|rmdir|chmod|chown|echo) \$[A-Z_][A-Z0-9_]*') \ + || { echo "--check: predicate failed" >&2; exit 2; } + if [[ -n "$hits" ]]; then printf '%s\n' "$hits" | wc -l; else echo 0; fi + exit 0 +fi + REPO_PATH="${1:?Usage: $0 }" FINDING_JSON="${2:?Missing finding JSON file}" diff --git a/scripts/lib/outcome-verdict.sh b/scripts/lib/outcome-verdict.sh new file mode 100644 index 00000000..a29e831c --- /dev/null +++ b/scripts/lib/outcome-verdict.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# +# outcome-verdict.sh — decide a fix outcome from evidence, not from exit codes. +# +# A fix script that exits 0 has only shown that it did not crash: experiment E1 +# (docs/experiments/E1-fix-outcome-replay-2026-10-20.adoc) found 0 of 130 +# checkable "success" rows confirmed by a rescan. The verdict instead compares +# the recipe's own detection predicate before and after the fix. +# +# Predicate contract: a fix script that supports verification answers +# --check +# by printing one non-negative integer (targets it would still fix) and +# changing nothing. The fixer and its checker are the same file, so they +# cannot drift apart. A script opts in with the header line +# # outcome-predicate: --check +# and only such scripts are ever called with --check: a fixer without the +# mode would read "--check" as its repo path and might run its edits. +# +# Verdicts: +# verified work found before, none after +# no_op nothing to fix before the run, nothing after +# incomplete work remains after the run (even if none was found before) +# unverified the script has no predicate (no evidence either way) +# failed the fix script exited non-zero +# +# Source this file; it defines functions only. + +# Count remaining targets for one recipe. Prints the count, or "NA" when the +# script has no --check mode or its answer is not a plain integer. +recipe_check_count() { + local fix_path="$1" repo_path="$2" out + if grep -qx '# outcome-predicate: --check' "$fix_path" 2>/dev/null && + out=$("$fix_path" --check "$repo_path" 2>/dev/null) && [[ "$out" =~ ^[0-9]+$ ]]; then + printf '%s\n' "$out" + else + printf 'NA\n' + fi +} + +# outcome_verdict -> verdict on stdout +outcome_verdict() { + local rc="$1" before="$2" after="$3" + if [[ "$rc" != 0 ]]; then + echo failed + elif [[ "$before" == NA || "$after" == NA ]]; then + echo unverified + elif ((after > 0)); then + # Work left after the run, including 0 -> n: a fixer that creates + # its own targets, or a tree changed under it, is not a no-op. + echo incomplete + elif ((before == 0)); then + echo no_op + else + echo verified + fi +} + +# Legacy `outcome` value for consumers that predate `verdict`. It reports the +# exit status only; read `verdict` for whether anything was fixed. +legacy_outcome() { + if [[ "$1" == failed ]]; then echo failure; else echo success; fi +} + +# canonical_recipe_id -> the id outcomes are +# reported under. A deprecated id (registry `deprecated_recipe_ids`) maps to +# its replacement, so a verified annotation is never credited to the removal +# recipe it was renamed from. Unknown ids, or no registry, pass through. +canonical_recipe_id() { + local registry="$1" rid="$2" out + if [[ -f "$registry" ]] && + out=$(jq -r --arg r "$rid" '.deprecated_recipe_ids[$r] // $r' "$registry" 2>/dev/null) && + [[ -n "$out" ]]; then + printf '%s\n' "$out" + else + printf '%s\n' "$rid" + fi +} + +# repo_lock / repo_unlock: serialise check/fix/check on +# one repo. Two parallel jobs for the same repo would otherwise both count +# targets before either fix, both count zero after, and both record a +# verified fix that happened once. The lock is released by closing its fd. +repo_lock() { + local key + key=$(printf '%s' "$2" | sha256sum | cut -c1-16) + exec {REPO_LOCK_FD}> "$1/repo-$key.lock" + flock "$REPO_LOCK_FD" +} + +repo_unlock() { + exec {REPO_LOCK_FD}>&- +} diff --git a/scripts/tests/dispatch-paths.sh b/scripts/tests/dispatch-paths.sh index b41c277c..f5adc9f0 100644 --- a/scripts/tests/dispatch-paths.sh +++ b/scripts/tests/dispatch-paths.sh @@ -7,6 +7,9 @@ fixture="$(mktemp -d)" trap 'rm -rf "$fixture"' EXIT mkdir -p "$fixture/repos/sample" "$fixture/checkout/scripts" "$fixture/primary/dispatch" cp "$fleet_test_root/scripts/repo-path-overrides.json" "$fixture/checkout/scripts/" +# dispatch-runner sources its outcome-verdict library from beside itself. +mkdir -p "$fixture/checkout/scripts/lib" +cp "$fleet_test_root/scripts/lib/outcome-verdict.sh" "$fixture/checkout/scripts/lib/" # While dispatch is quarantined (docs/AUTOMATION-QUARANTINE.adoc), the # production runner must refuse every call. The path/outcome contracts diff --git a/tests/outcome-verdict-test.sh b/tests/outcome-verdict-test.sh new file mode 100755 index 00000000..8ce803f7 --- /dev/null +++ b/tests/outcome-verdict-test.sh @@ -0,0 +1,209 @@ +#!/usr/bin/env bash +# SPDX-License-Identifier: MPL-2.0 +# +# Fix outcomes are verdicts from each recipe's own predicate, not exit codes +# (experiment E1, docs/experiments/E1-fix-outcome-replay-2026-10-20.adoc). +# +# Covers: +# 1. outcome_verdict truth table. +# 2. recipe_check_count: integer from a --check script, NA without one, +# and NA when the predicate itself fails (never a false 0). +# 3. The two predicate-carrying fixers, end to end on a fixture repo: +# targets > 0 before, 0 after, --check itself changes nothing, and a +# second run is a no_op. +# 4. dispatch-runner's record_outcome row and rescan trigger (extracted, +# like hypatia-record-outcome-smoke.sh, so the quarantined main loop +# never runs). +# 5. Deprecated recipe ids report under their replacement. +# 6. Two parallel jobs on one repo record one fix, not two (repo_lock). +# 7. The E1 replay (audit-fix-outcomes.sh) counts as the fixers do, and +# reads NA when git grep cannot run. + +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +TMP="$(mktemp -d)" +trap 'rm -rf "$TMP"' EXIT + +# shellcheck source=../scripts/lib/outcome-verdict.sh +source "$ROOT/scripts/lib/outcome-verdict.sh" + +pass=0 +fail=0 +check() { + local label="$1" expected="$2" actual="$3" + if [[ "$expected" == "$actual" ]]; then + pass=$((pass + 1)) + echo "PASS: $label" + else + fail=$((fail + 1)) + echo "FAIL: $label — expected '$expected', got '$actual'" + fi +} + +# --- 1. truth table ------------------------------------------------------- +check "exit 0, work found, none left" verified "$(outcome_verdict 0 3 0)" +check "exit 0, nothing to fix" no_op "$(outcome_verdict 0 0 0)" +check "exit 0, work remains" incomplete "$(outcome_verdict 0 3 1)" +check "exit 0, targets appear (0 -> n)" incomplete "$(outcome_verdict 0 0 2)" +check "exit 0, no predicate" unverified "$(outcome_verdict 0 NA NA)" +check "exit 0, predicate lost after fix" unverified "$(outcome_verdict 0 3 NA)" +check "non-zero exit wins over counts" failed "$(outcome_verdict 1 3 0)" +check "legacy outcome of a failure" failure "$(legacy_outcome failed)" +check "legacy outcome of a no-op" success "$(legacy_outcome no_op)" + +# --- 2. recipe_check_count ------------------------------------------------ +cat > "$TMP/no-check.sh" <<'EOF' +#!/usr/bin/env bash +echo "=== fixer without a predicate ===" +EOF +chmod +x "$TMP/no-check.sh" +mkdir -p "$TMP/empty" +check "script without --check is NA" NA "$(recipe_check_count "$TMP/no-check.sh" "$TMP/empty")" + +# A fixer that has not opted in must never be run with --check: it would take +# "--check" as its repo path and could run its edits. +cat > "$TMP/unmarked.sh" < "$TMP/shim-find/find" +printf '#!/bin/sh\nexit 2\n' > "$TMP/shim-grep/grep" +chmod +x "$TMP/shim-find/find" "$TMP/shim-grep/grep" +printf 'cd $HOME\n' > "$TMP/hits/a.sh" +printf 'f = believe_me 1\n' > "$TMP/hits/A.idr" +for script in fix-shell-quoting.sh fix-believe-me.sh; do + check "$script: find failure is NA" NA \ + "$(PATH="$TMP/shim-find:$PATH" recipe_check_count "$ROOT/scripts/$script" "$TMP/hits")" +done +rc=0 +out=$(PATH="$TMP/shim-grep:$PATH" "$ROOT/scripts/fix-shell-quoting.sh" --check "$TMP/hits" 2>/dev/null) || rc=$? +check "grep error fails --check, prints no count" "2:" "$rc:$out" + +# --- 3. fixers end to end ------------------------------------------------- +repo="$TMP/repo" +mkdir -p "$repo/src" "$repo/node_modules/dep" +echo '{}' > "$TMP/finding.json" +printf '#!/bin/sh\ncd $HOME\nrm $TMPDIR && echo $X\nmv "$A" b\n' > "$repo/run.sh" +printf 'cd $VENDORED\n' > "$repo/node_modules/dep/vendored.sh" +printf 'module M\nf : Int\nf = believe_me 1\n-- believe_me in a comment\ng = believe_me 2\n' \ + > "$repo/src/A.idr" + +for spec in "fix-shell-quoting.sh 3" "fix-believe-me.sh 2"; do + read -r script expected_before <<< "$spec" + fixer="$ROOT/scripts/$script" + snapshot=$(find "$repo" -type f -exec sha256sum {} + | sort) + before=$(recipe_check_count "$fixer" "$repo") + check "$script: --check changes nothing" "$snapshot" "$(find "$repo" -type f -exec sha256sum {} + | sort)" + check "$script: targets before" "$expected_before" "$before" + + rc=0 + "$fixer" "$repo" "$TMP/finding.json" > /dev/null 2>&1 || rc=$? + after=$(recipe_check_count "$fixer" "$repo") + check "$script: first run verified" verified "$(outcome_verdict "$rc" "$before" "$after")" + + rc=0 + "$fixer" "$repo" "$TMP/finding.json" > /dev/null 2>&1 || rc=$? + check "$script: second run is a no-op" no_op \ + "$(outcome_verdict "$rc" "$after" "$(recipe_check_count "$fixer" "$repo")")" +done +check "vendored file left alone" 'cd $VENDORED' "$(cat "$repo/node_modules/dep/vendored.sh")" + +# --- 4. dispatch-runner record_outcome ------------------------------------ +awk ' + /^# --- Hypatia closed-loop contract ---/ { capture = 1 } + capture { print } + /^record_outcome\(\)/ && capture { seen = 1 } + seen && /^}/ { exit } +' "$ROOT/scripts/dispatch-runner.sh" > "$TMP/extracted.sh" +export REPOS_BASE="$TMP/repos" HYPATIA_OUTCOME_REPORT=off +OUTCOME_FILE="$TMP/outcomes.jsonl" +export OUTCOME_FILE_CENTRAL="$OUTCOME_FILE" # read by the sourced record_outcome +RESCAN_REPOS_FILE="$TMP/rescan" +: > "$RESCAN_REPOS_FILE" +# shellcheck disable=SC1091 +source "$TMP/extracted.sh" + +record_outcome "PA009" "recipe-shell-quote-vars" "repo-a" "" verified 0 3 0 +record_outcome "PA009" "recipe-shell-quote-vars" "repo-b" "" no_op 0 0 0 +record_outcome "PA009" "recipe-x" "repo-c" "" unverified 0 +record_outcome "PA009" "recipe-license-hygiene" "repo-d" "" refused_license_policy + +check "verified row" '{"verdict":"verified","outcome":"success","exit_code":0,"before_hits":3,"after_hits":0}' \ + "$(sed -n 1p "$OUTCOME_FILE" | jq -c '{verdict, outcome, exit_code, before_hits, after_hits}')" +check "unverified row has no counts" '{"verdict":"unverified","before_hits":null,"after_hits":null}' \ + "$(sed -n 3p "$OUTCOME_FILE" | jq -c '{verdict, before_hits, after_hits}')" +check "refusal keeps its own outcome" '{"verdict":"refused_license_policy","outcome":"refused_license_policy","exit_code":null}' \ + "$(sed -n 4p "$OUTCOME_FILE" | jq -c '{verdict, outcome, exit_code}')" +check "only verified repos trigger a rescan" "repo-a" "$(cat "$RESCAN_REPOS_FILE")" + +# --- 5. deprecated recipe ids --------------------------------------------- +registry="$ROOT/scripts/fix-script-registry.json" +check "deprecated id reports as its replacement" recipe-annotate-believe-me \ + "$(canonical_recipe_id "$registry" recipe-remove-believe-me)" +check "canonical id unchanged" recipe-annotate-believe-me \ + "$(canonical_recipe_id "$registry" recipe-annotate-believe-me)" +check "unknown id unchanged" recipe-x "$(canonical_recipe_id "$registry" recipe-x)" +check "no registry, id unchanged" recipe-x "$(canonical_recipe_id "$TMP/none.json" recipe-x)" + +# --- 6. parallel jobs on one repo ----------------------------------------- +# Without repo_lock both jobs count 3 before either fix and both record +# verified for one fix (measured: "verified verified"). With it, the second +# job counts what the first left. +par="$TMP/par" +mkdir -p "$par" "$TMP/locks" +printf 'cd $A\nrm $B\necho $C\n' > "$par/x.sh" +fixer="$ROOT/scripts/fix-shell-quoting.sh" +job() { + repo_lock "$TMP/locks" "$par" + local b a + b=$(recipe_check_count "$fixer" "$par") + sleep 0.3 + "$fixer" "$par" "$TMP/finding.json" > /dev/null 2>&1 + a=$(recipe_check_count "$fixer" "$par") + repo_unlock + outcome_verdict 0 "$b" "$a" > "$TMP/job$1" +} +job 1 & job 2 & wait +check "two jobs on one repo record one fix" "no_op verified" \ + "$(sort "$TMP/job1" "$TMP/job2" | paste -sd' ')" + +# --- 7. the E1 replay agrees with the fixers ------------------------------ +# A root-level vendor/ (and .pack/ for Idris) is excluded as the fixers' +# find excludes it, and a git grep that cannot run reads NA, never 0. +ar="$TMP/audit/ar" +mkdir -p "$ar/vendor" "$ar/.pack" "$TMP/shim-git" +printf 'cd $A\n' > "$ar/vendor/a.sh" +printf 'f = believe_me 1\n' > "$ar/A.idr" +printf 'g = believe_me 2\n' > "$ar/.pack/B.idr" +git -C "$ar" init -q -b main +git -C "$ar" add -A +GIT_COMMITTER_DATE=2026-01-01T00:00:00Z git -C "$ar" -c user.email=t@t -c user.name=t \ + commit -q --date=2026-01-01T00:00:00Z -m fixture +git -C "$ar" update-ref refs/remotes/origin/main HEAD +printf '%s\n' \ + '{"repo":"ar","pattern":"recipe-shell-quote-vars","outcome":"success","fixed_at":"2026-02-01T00:00:00Z"}' \ + '{"repo":"ar","pattern_id":"PA004","recipe_id":"recipe-annotate-believe-me","outcome":"success","timestamp":"2026-02-01T00:00:00Z"}' \ + > "$TMP/audit/log.jsonl" +audit() { "$ROOT/scripts/audit-fix-outcomes.sh" "$TMP/audit/log.jsonl" "$ar" | awk -F'\t' 'NR > 1 { print $2 "=" $6 }' | sort | paste -sd' '; } +check "replay counts as the fixers do" \ + "recipe-annotate-believe-me=$(recipe_check_count "$ROOT/scripts/fix-believe-me.sh" "$ar") recipe-shell-quote-vars=$(recipe_check_count "$fixer" "$ar")" \ + "$(audit)" +printf '#!/bin/sh\nfor a; do [ "$a" = grep ] && exit 128; done\nexec %s "$@"\n' "$(command -v git)" > "$TMP/shim-git/git" +chmod +x "$TMP/shim-git/git" +check "replay predicate failure is NA" "recipe-annotate-believe-me=NA recipe-shell-quote-vars=NA" \ + "$(PATH="$TMP/shim-git:$PATH" audit)" + +echo "" +echo "Results: $pass passed, $fail failed" +[[ "$fail" -eq 0 ]]