diff --git a/.audit-harness/CHANGELOG.md b/.audit-harness/CHANGELOG.md index f0d37abeb5c..6ad26bd0342 100644 --- a/.audit-harness/CHANGELOG.md +++ b/.audit-harness/CHANGELOG.md @@ -6,6 +6,45 @@ project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +## [1.4.0] - 2026-09-01 + +### Added + +- `worktree-run` — pre-push gate runner that checks the exact ref being pushed + in a disposable `git worktree`: `verify` + `escape-scan --range` on the push + range fail-closed, `conform` + `audit --fast` advisory, gate-result/v1 rows + written outside the repo. No push authority, no LLM stage, no repo writes. + Wired as a lefthook `pre-push` job in this repo (reference recipe for + consumers). Extracted from the no-mistakes discovery (2026-08-30): the + isolated-worktree pre-push loop was the one idea worth keeping; the LLM + review stage, auto-fix authority, and proxy-push were deliberately not + adopted. Honors `AUDIT_HARNESS_DISABLE=1`. +- `bias-count` now counts source-introspection tests — a test whose only + evidence is reading the implementation source (`inspect.getsource`, reading + files under `src/`) instead of asserting observable behaviour of a public + interface. Deterministic counterpart of the prose rule in the + implement-tests auto-remediation reference; advisory like every bias + pattern (promotion to blocking goes through `fp-rate`). + +### Fixed + +- `bias-count` no longer dies mid-report on a clean directory: under + `pipefail`, a zero-match grep exited 1 and aborted the scan (truncated + output, exit 1) before the summary — the "gate that looks like it ran" + failure class from 1.3.1. Zero-match patterns now count as 0. +- Expanded the default `harness-hash` denominator to include package test + scripts, `tests/TESTING.md`, common JavaScript and Python coverage/mutation + configs, and primary CI workflow files. Previously an edit could weaken the + command or workflow that enforced a pinned threshold without changing one of + the few files protected by the manifest. +- Added an offline contract suite proving those files are pinned, their edits + or removal produce `HARNESS_TAMPERED` (exit 2), and ordinary source changes do + not require a policy re-pin. +- Accessibility detection now recognises dedicated a11y/accessibility test + files and directories for native, QML, CLI/TUI, and other non-web stacks. + Previously it recognised only four web dependencies and falsely advised + projects to add irrelevant tooling despite executable accessibility tests. + > **Riding a future v2.1 routine release (descoped from 1.2.0):** OTel event-name > polish (iah-E07b/c). The `agent.rollout.gate.evaluated` and `gate.decision.emitted` > event names are already locked + tested on main (PRs #78, #81 per NORMATIVE @@ -21,6 +60,10 @@ now detects escapes it previously let through. Adopters on 1.3.0 should upgrade: two of the five were **fail-open**, meaning a real threshold-lowering escape merged clean. +**First required dogfood (2026-07-23):** CCPI (`claude-code-plugins-plus-skills`) +made `audit-harness conform --strict` a **required** branch-protection context +(`skill-conform`, own workflow — never inside `ci-required`). See CCPI#1118. + ### Fixed - **`set -e` killed the script mid-load, producing a silent total bypass @@ -414,7 +457,8 @@ Initial release. Extracted from the `audit-tests` Claude Code skill v7.0.0 to en - **`audit-harness gherkin-lint`** — advisory Gherkin quality check. - **`audit-harness crap`** — CRAP (Complexity × Coverage) scorer for Python, JS/TS, Go, Rust. -[Unreleased]: https://github.com/jeremylongshore/intent-audit-harness/compare/v1.2.2...HEAD +[Unreleased]: https://github.com/jeremylongshore/intent-audit-harness/compare/v1.4.0...HEAD +[1.4.0]: https://github.com/jeremylongshore/intent-audit-harness/compare/v1.3.1...v1.4.0 [1.2.2]: https://github.com/jeremylongshore/intent-audit-harness/compare/v1.2.1...v1.2.2 [1.2.1]: https://github.com/jeremylongshore/intent-audit-harness/compare/v1.2.0...v1.2.1 [1.2.0]: https://github.com/jeremylongshore/intent-audit-harness/compare/v1.1.8...v1.2.0 diff --git a/.audit-harness/PROVENANCE b/.audit-harness/PROVENANCE index 93d9f316b1c..8744ea7930a 100644 --- a/.audit-harness/PROVENANCE +++ b/.audit-harness/PROVENANCE @@ -1,4 +1,4 @@ source-repo: jeremylongshore/intent-audit-harness -source-version: v1.3.1 -source-tarball: https://github.com/jeremylongshore/intent-audit-harness/archive/refs/tags/v1.3.1.tar.gz -installed-at: 2026-07-29T06:10:55Z +source-version: v1.4.0 +source-tarball: https://github.com/jeremylongshore/intent-audit-harness/archive/refs/tags/v1.4.0.tar.gz +installed-at: 2026-09-05T02:13:18Z diff --git a/.audit-harness/README.md b/.audit-harness/README.md index eb230ea7e22..590af226fe2 100644 --- a/.audit-harness/README.md +++ b/.audit-harness/README.md @@ -4,6 +4,8 @@ [![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE) [![Provenance](https://img.shields.io/badge/sigstore-provenance-066da5)](https://www.npmjs.com/package/@intentsolutions/audit-harness) +[![ko-fi](https://ko-fi.com/img/githubbutton_sm.svg)](https://ko-fi.com/U5S225PTME) + Part of the **[Intent Eval Platform](https://github.com/intent-solutions-io/intent-eval-platform)** — the umbrella grouping the platform's six repos: five converge via a shared Evidence Bundle schema (`intent-eval-core`, `intent-eval-lab`, `audit-harness`, `j-rig-skill-binary-eval`, `intent-rollout-gate`), plus `intent-eval-dashboard` as a satellite consumer (not part of the convergence taxonomy). Deterministic test-enforcement toolkit. Companion to the `audit-tests` and `implement-tests` Claude Code skills — but usable standalone in any repo that wants hash-pinned, escape-scanned, AI-resistant quality gates. @@ -106,10 +108,22 @@ git commit -m "chore(test): lower coverage floor to 75" The harness enforces this rule: **policy changes must be conscious, not silent.** -Engineer-owned files (`tests/TESTING.md`, `features/*.feature`, `.dependency-cruiser.cjs`, `stryker.conf.json`, etc.) are hashed into a manifest. Any diff that changes their content without a fresh `audit-harness init` is caught by pre-commit / CI and **REFUSED**. +Engineer-owned files (`tests/TESTING.md`, package test scripts, CI workflows, +coverage and mutation configs, acceptance features, and architecture rules) are +hashed into a manifest. Any diff that changes their content without a fresh +`audit-harness init` is caught by pre-commit / CI and **REFUSED**. Ordinary +application source is deliberately outside this denominator. AI agents remain useful (they can read policy, they can implement within constraints). What they can't do is silently weaken the constraints. That's the entire design. +The same walls can close the loop **before a push leaves the machine**: +`audit-harness worktree-run --pre-push` checks the exact ref being pushed in a +disposable `git worktree` — `verify` + `escape-scan` on the push range stay +fail-closed, `conform` + `audit` contribute advisory gate-result/v1 rows — then +removes the worktree. It has no push authority, no LLM stage, and never writes +to the repo. Wire it as a lefthook `pre-push` job (this repo's `lefthook.yml` +is the reference) so CI becomes confirmation instead of discovery. + See `audit-tests/references/philosophy.md` in the companion skill for the full rationale. ## The 7-layer testing taxonomy diff --git a/.audit-harness/VERSION b/.audit-harness/VERSION index 7574079828d..0d0c52f84db 100644 --- a/.audit-harness/VERSION +++ b/.audit-harness/VERSION @@ -1 +1 @@ -v1.3.1 +v1.4.0 diff --git a/.audit-harness/bin/audit-harness.js b/.audit-harness/bin/audit-harness.js index eb8b337a155..e72324bae75 100755 --- a/.audit-harness/bin/audit-harness.js +++ b/.audit-harness/bin/audit-harness.js @@ -31,6 +31,7 @@ const COMMANDS = { 'currency': { script: 'currency.py', args: [] }, 'migration-notes': { script: 'migration-notes.py', args: [] }, 'gen-layer-applicability': { script: 'gen-layer-applicability.py', args: [] }, + 'worktree-run': { script: 'worktree-run.sh', args: [] }, }; // Gate commands that may be no-op'd by the AUDIT_HARNESS_DISABLE kill-switch. @@ -38,6 +39,7 @@ const COMMANDS = { // itself (every gate enforcement=disabled). verify/init/list always run. const KILLABLE_GATES = new Set([ 'escape-scan', 'cred-gate', 'arch', 'bias', 'gherkin-lint', 'crap', 'emit-evidence', + 'worktree-run', ]); function usage() { @@ -118,6 +120,12 @@ Commands: emit-evidence Wrap a gate-result JSON envelope in an in-toto Statement v1 (predicate https://evals.intentsolutions.io/gate-result/v1) Read JSON on stdin: --json | audit-harness emit-evidence + worktree-run Pre-push gate runner: checks the ref being pushed in + a disposable git worktree (verify + escape-scan on + the push range fail-closed; conform + audit advisory) + and emits gate-result/v1 rows. No push authority, no + repo writes. --pre-push | --ref REF | --range A..B | + --out FILE. Wire via lefthook pre-push (see README). Evidence Bundle (v0.3.0+): All gates support --json to emit machine-readable gate-result envelopes diff --git a/.audit-harness/package.json b/.audit-harness/package.json index 96721de67ae..0cc5247b882 100644 --- a/.audit-harness/package.json +++ b/.audit-harness/package.json @@ -1,6 +1,6 @@ { "name": "@intentsolutions/audit-harness", - "version": "1.3.1", + "version": "1.4.0", "description": "Deterministic test-enforcement harness — escape-scan, hash-pinning, CRAP, architecture checks, bias detection, Gherkin lint. Companion to the audit-tests and implement-tests Claude Code skills.", "license": "Apache-2.0", "author": "Jeremy Longshore ", diff --git a/.audit-harness/scripts/audit.py b/.audit-harness/scripts/audit.py index 592e2870b83..46ea1d4a9ba 100755 --- a/.audit-harness/scripts/audit.py +++ b/.audit-harness/scripts/audit.py @@ -183,6 +183,21 @@ def d_perf(repo, deps): def d_a11y(repo, deps): if any(x in deps for x in ("axe-core", "@axe-core/playwright", "jest-axe", "pa11y")): return True, "a11y tooling dep" + # Accessibility is not a web-only layer. QML, native, CLI/TUI, mobile, and + # desktop projects commonly assert roles, names, focus, keyboard routing, + # contrast, or screen-reader contracts in ordinary language-native tests. + # Keep the filename signal test-specific so an accessibility.md design + # document alone does not masquerade as executable evidence. + if has_dir(repo, "a11y", "accessibility"): + return True, "a11y/accessibility test dir" + test_exts = (".js", ".ts", ".py", ".go", ".rs", ".java", ".kt", ".cs", ".rb") + if has_file_matching( + repo, + lambda f: ("a11y" in f.lower() or "accessibility" in f.lower()) + and ("test" in f.lower() or "spec" in f.lower()) + and f.lower().endswith(test_exts), + ): + return True, "a11y/accessibility test file" return False, "no accessibility test infrastructure detected" diff --git a/.audit-harness/scripts/bias-count.sh b/.audit-harness/scripts/bias-count.sh index ea599420aad..4df86807909 100755 --- a/.audit-harness/scripts/bias-count.sh +++ b/.audit-harness/scripts/bias-count.sh @@ -71,7 +71,11 @@ count_pattern() { local label="$1" local pattern="$2" local count - count=$(grep -rn "$pattern" "$TEST_DIR" 2>/dev/null | wc -l) + # `|| true`: under pipefail a zero-match grep exits 1, which killed the whole + # scan mid-report (truncated output, exit 1) on any clean directory — the + # "gate that looks like it ran" failure class from 1.3.1. wc's count is + # captured either way. + count=$(grep -rn "$pattern" "$TEST_DIR" 2>/dev/null | wc -l) || true TOTAL_BIAS=$((TOTAL_BIAS + count)) printf " %-30s %d\n" "$label" "$count" } @@ -88,13 +92,20 @@ count_pattern "Symmetric input (1,1)" "(1, 1)" count_pattern "Symmetric input (100,100)" "(100, 100)" count_pattern "Range-only assertion" "assert.*<=.*<=" count_pattern 'Substring check (in str)' '" in ' +# Source-introspection tests: a test whose evidence is reading the +# implementation source (inspect.getsource, reading files under src/) instead +# of asserting observable behaviour of a public interface. Deterministic +# counterpart of the prose rule in auto-remediation.md; advisory like every +# other bias pattern here (promotion to blocking goes through fp-rate). +count_pattern "Source-introspection (getsource)" "inspect\.getsource\|getsource(" +count_pattern "Source-read test (src/ file)" "readFileSync([^)]*src/\|open([^)]*['\"]src/" echo -# Count test functions -TEST_COUNT=$(grep -rn "def test_\|it('\|it(\"\\|test('\|test(\"" "$TEST_DIR" 2>/dev/null | wc -l) +# Count test functions (`|| true` — see count_pattern) +TEST_COUNT=$(grep -rn "def test_\|it('\|it(\"\\|test('\|test(\"" "$TEST_DIR" 2>/dev/null | wc -l) || true -# Count total assertions -ASSERT_COUNT=$(grep -rn "assert\b\|assertEqual\|expect(" "$TEST_DIR" 2>/dev/null | wc -l) +# Count total assertions (`|| true` — see count_pattern) +ASSERT_COUNT=$(grep -rn "assert\b\|assertEqual\|expect(" "$TEST_DIR" 2>/dev/null | wc -l) || true # Assertion density if [ "$TEST_COUNT" -gt 0 ]; then diff --git a/.audit-harness/scripts/harness-hash.sh b/.audit-harness/scripts/harness-hash.sh index e37e8ac88ab..b35fc9fa10a 100755 --- a/.audit-harness/scripts/harness-hash.sh +++ b/.audit-harness/scripts/harness-hash.sh @@ -55,6 +55,12 @@ done set -- "${_filtered_args[@]+"${_filtered_args[@]}"}" PATTERNS=( + # Wall 1: the declared test contract and package-level command surface. A + # coverage floor is not protected if an edit can simply replace `npm test` + # with a weaker command or remove the CI invocation that enforces it. + "TESTING.md" + "tests/TESTING.md" + "package.json" # Wall 1: acceptance "features/**/*.feature" "features/*.feature" @@ -71,9 +77,39 @@ PATTERNS=( "test/**/*ArchTests.cs" "tests/**/*ArchTests.cs" # Coverage thresholds (edits to these are escape attempts — hash them) + ".c8rc" ".c8rc.json" + ".nycrc" + ".nycrc.json" + ".nycrc.yml" + ".nycrc.yaml" + ".coveragerc" + "coverage.toml" + "jest.config.js" + "jest.config.cjs" + "jest.config.mjs" + "jest.config.ts" + "vitest.config.js" + "vitest.config.cjs" + "vitest.config.mjs" + "vitest.config.ts" + "pytest.ini" + "pyproject.toml" + "setup.cfg" + "tox.ini" "stryker.conf.json" "stryker.config.js" + "stryker.config.cjs" + "stryker.config.mjs" + # Wall 1: primary CI enforcement. Workflow changes are legitimate, but they + # must be accompanied by an explicit re-pin so removing a test, race, + # mutation, or audit step cannot masquerade as an ordinary source edit. + ".github/workflows/*.yml" + ".github/workflows/*.yaml" + ".gitlab-ci.yml" + ".circleci/config.yml" + "azure-pipelines.yml" + "Jenkinsfile" ) # Optional per-repo extra patterns appended from .harness-hash-extra-patterns diff --git a/.audit-harness/scripts/kernel-shadow-check.sh b/.audit-harness/scripts/kernel-shadow-check.sh index ffccc43a01c..7440969ed36 100755 --- a/.audit-harness/scripts/kernel-shadow-check.sh +++ b/.audit-harness/scripts/kernel-shadow-check.sh @@ -32,6 +32,22 @@ # This is a separate, shallower contract from the # kernel authoring/v1 validity SSoT — intentionally # different, not a re-declaration. +# * **/dist/**, **/build/** +# — compiler output. Whatever a bundle or .d.ts contains +# is a mechanical restatement of source that this +# detector already checks; flagging both reports one +# authoring decision twice and cannot be fixed in the +# generated file. +# +# NOT a shadow (by construction — the anchors below): +# * `export { type EvidenceBundle, ... } from "@intentsolutions/core/..."` +# A re-export entry names the kernel's type in order to FORWARD it. That is +# the single-source-of-truth pattern this detector exists to encourage, and +# matching it was a false positive that flagged three j-rig files for doing +# exactly the right thing. The identifier in a re-export or import list is +# followed by `,`, `}`, or a newline — never by declaration syntax — so the +# anchors require `=`, `{`, `<`, `(`, `:`, `extends`, or `implements` after +# the name. # # Background: iah-E02 (the architecture question — peerDep-only vs full TS port # vs second-emitter — that historically blocked a standing kernel-shadow check) @@ -78,36 +94,128 @@ is_allowlisted() { schemas/conform/*) return 0 ;; node_modules/*) return 0 ;; .git/*) return 0 ;; + dist/*|*/dist/*) return 0 ;; + build/*|*/build/*) return 0 ;; *) return 1 ;; esac } +# The class-2 declaration anchor (see the block comment above the class-2 loop). +CLASS2_PATTERN='(^|[[:space:]])((export|declare|abstract)[[:space:]]+)*(interface|class|type)[[:space:]]+(EvidenceBundlePayload|EvidenceBundle|GateResultV1)([[:space:]]*[{<(:=]|[[:space:]]+(extends|implements)[[:space:]])' + +# is_kernel_imported FILE SYMBOL — true when SYMBOL appears inside an import or +# re-export statement that resolves to @intentsolutions/core in FILE. Statements +# are reconstructed by joining the file and splitting on ';', so a multi-line +# `import {\n A,\n B,\n} from "@intentsolutions/core/..."` is handled. +is_kernel_imported() { + local file="$1" sym="$2" + awk -v sym="$sym" ' + { buf = buf " " $0 } + END { + n = split(buf, stmts, /;/) + for (i = 1; i <= n; i++) { + if (stmts[i] ~ /@intentsolutions\/core/ && + stmts[i] ~ ("(^|[^A-Za-z0-9_])" sym "([^A-Za-z0-9_]|$)")) { found = 1 } + } + exit(found ? 0 : 1) + } + ' "$file" +} + +# is_kernel_derivation FILE LINE — true when LINE is a type alias whose right-hand +# side is a pure derivation of a kernel-imported symbol. Anything else (an +# interface, a class, a structural type literal, a union, or a derivation from a +# locally-declared schema) returns false and is treated as a real declaration. +is_kernel_derivation() { + local file="$1" line="$2" rhs sym + # Only `type X = ...` can derive; interface/class always declare a shape. + [[ "$line" =~ (^|[[:space:]])((export|declare|abstract)[[:space:]]+)*type[[:space:]] ]] || return 1 + [[ "$line" == *=* ]] || return 1 + + rhs="${line#*=}" + # Strip comments, trailing semicolon, and surrounding whitespace. + rhs="${rhs%%//*}" + rhs="$(printf '%s' "$rhs" | sed -E 's/[[:space:]]*;?[[:space:]]*$//; s/^[[:space:]]+//')" + + if [[ "$rhs" =~ ^z\.(infer|input|output)\$ ]]; then + sym="${BASH_REMATCH[2]}" + elif [[ "$rhs" =~ ^typeof[[:space:]]+([A-Za-z_][A-Za-z0-9_]*)$ ]]; then + sym="${BASH_REMATCH[1]}" + elif [[ "$rhs" =~ ^([A-Za-z_][A-Za-z0-9_]*)$ ]]; then + sym="${BASH_REMATCH[1]}" + else + return 1 # structural / union / intersection → a real declaration + fi + + is_kernel_imported "$file" "$sym" +} + shadows=() # 1. JSON Schema documents claiming a kernel-owned canonical $id. # The kernel owns gate-result/ and evidence-bundle/ ids under # evals.intentsolutions.io. conform/v1 ids are the harness's own (allowlisted # structurally by the schemas/conform/ path skip below). +# +# EXEMPT: a redirect stub. A document that carries an `x-redirect` marker is +# the ratified discoverability pattern (Blueprint B § 7.0 "Lab specs/ MAY host +# redirect stubs"; ISEDC Session 5 DR-018 § 6.4 Option α-minus) — it claims the +# id in order to $ref the kernel's schema, which is referencing, not +# re-declaring. The lab's own schema-drift.yml already allowlists exactly this +# marker; flagging it here would contradict a gate the platform ratified. # shellcheck disable=SC2016 # the grep pattern's $id is a literal, not a shell var while IFS= read -r f; do [[ -z "$f" ]] && continue rel="${f#./}" is_allowlisted "$rel" && continue + grep -qE '"x-redirect"[[:space:]]*:' "$f" && continue shadows+=("$rel (re-declares a kernel-owned JSON Schema \$id)") done < <(grep -rIlE '"\$id"[[:space:]]*:[[:space:]]*"https://evals\.intentsolutions\.io/(gate-result|evidence-bundle)/' \ --include='*.json' --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null || true) -# 2. TS/Python source DEFINING (not importing) a kernel-owned type/class. -# Definitions look like `interface GateResultV1`, `class EvidenceBundle`, -# `type EvidenceBundlePayload = ...`. Imports (`import { GateResultV1Schema } -# from '@intentsolutions/core/...'`) are NOT matched by these anchors. +# 2. TS/Python source DEFINING (not importing, not re-exporting) a kernel-owned +# type/class. The keyword alone is not enough to tell a definition from a +# re-export — `export { type EvidenceBundle } from "@intentsolutions/core"` +# carries `type EvidenceBundle` too. What separates them is what FOLLOWS the +# identifier, so the anchor requires actual declaration syntax: +# +# interface X { interface X interface X extends Y +# class X { class X class X extends Y +# class X(Base): class X: class X implements Y (py/ts) +# type X = type X = +# +# A re-export or import entry is followed by `,`, `}`, `;`, or end-of-line and +# therefore cannot match. `declare`/`abstract` prefixes are tolerated. +# +# EXEMPT: a pure DERIVATION of a kernel-imported symbol, e.g. +# export type EvidenceBundle = z.infer; +# where `EvidenceBundlePayloadSchema` is imported/re-exported from +# @intentsolutions/core in the same file. Such an alias has no independent +# shape — it is defined BY the kernel schema and changes when the kernel +# changes, so it cannot drift, which is the entire harm this detector guards +# against. Only three RHS forms qualify (`z.infer|input|output`, +# `typeof S`, bare `S`); anything structural (`{`, `|`, `&`) is a real +# declaration and still flags, as does a derivation from a NON-kernel symbol. while IFS= read -r f; do [[ -z "$f" ]] && continue rel="${f#./}" is_allowlisted "$rel" && continue - shadows+=("$rel (defines a kernel-owned type — should import from @intentsolutions/core)") -done < <(grep -rIlE \ - '(^|[[:space:]])(export[[:space:]]+)?(interface|class|type)[[:space:]]+(GateResultV1|EvidenceBundle|EvidenceBundlePayload)\b' \ + + # Per-LINE triage: a file is reported only if it holds at least one match that + # is not an exempt kernel derivation. + real_hits=() + while IFS= read -r hit; do + [[ -z "$hit" ]] && continue + lineno="${hit%%:*}" + text="${hit#*:}" + if is_kernel_derivation "$f" "$text"; then continue; fi + real_hits+=("$lineno") + done < <(grep -nE "$CLASS2_PATTERN" "$f" 2>/dev/null || true) + + [[ ${#real_hits[@]} -eq 0 ]] && continue + lines="$(IFS=,; echo "${real_hits[*]}")" + shadows+=("$rel (defines a kernel-owned type at line(s) ${lines} — should import from @intentsolutions/core)") +done < <(grep -rIlE "$CLASS2_PATTERN" \ --include='*.ts' --include='*.py' --exclude-dir=node_modules --exclude-dir=.git . 2>/dev/null || true) if [[ ${#shadows[@]} -eq 0 ]]; then diff --git a/.audit-harness/scripts/worktree-run.sh b/.audit-harness/scripts/worktree-run.sh new file mode 100755 index 00000000000..97694ff79e4 --- /dev/null +++ b/.audit-harness/scripts/worktree-run.sh @@ -0,0 +1,165 @@ +#!/usr/bin/env bash +# worktree-run.sh — run the deterministic gate set in a disposable git worktree +# BEFORE a push leaves the machine, so CI becomes confirmation, not discovery. +# +# Extracted from the no-mistakes discovery (2026-08-30): the one idea worth +# keeping from that tool is closing the gate loop pre-push in an isolated +# worktree. Everything else about it (LLM review stage, auto-fix authority, +# proxy push) is deliberately NOT here. This runner has: +# - NO push authority (it never touches a remote) +# - NO LLM stage (every gate is an exit code) +# - NO writes to the repo (rows go to a temp file or --out; the worktree is +# disposable and removed on exit) +# +# Gate set (same checks CI runs, resolved per repo): +# verify harness-hash --verify in the worktree fail-closed (exit 2) +# escape-scan --range .. fail-closed (exit 1/2) +# conform conform.py (advisory rows) never raises exit +# audit audit.py --fast (advisory rows) never raises exit +# +# The advisory gates emit gate-result/v1 rows for evidence; only the two +# already-promoted fail-closed gates (verify, escape-scan) can block a push, +# per docs/gate-promotion.md. Promotion of the advisory rows to blocking goes +# through the measured FP ≤5% + engineer-pin path, never through this script. +# +# Usage: +# bash worktree-run.sh --pre-push # as a git/lefthook pre-push hook +# bash worktree-run.sh --ref # gate an arbitrary ref (default HEAD) +# bash worktree-run.sh --range A..B # explicit escape-scan range +# bash worktree-run.sh ... --out FILE # write gate-result rows to FILE +# +# In --pre-push mode the git pre-push stdin protocol +# ( per line) is consumed if +# present; when stdin is empty (lefthook may drain it) the range falls back to +# merge-base with the upstream or origin/HEAD. +# +# Exit codes (mirrors escape-scan/verify): +# 0 — clean (advisory findings do not change the exit code) +# 1 — CHALLENGE (escape-scan) +# 2 — REFUSE (escape-scan) or HARNESS_TAMPERED (verify) + +set -euo pipefail + +[ "${BASH_VERSINFO:-0}" -ge 4 ] || { echo 'audit-harness requires bash >= 4' >&2; exit 3; } + +# Kill-switch: same lever every gate honors. +if [[ "${AUDIT_HARNESS_DISABLE:-0}" == "1" ]]; then + echo "worktree-run: AUDIT_HARNESS_DISABLE=1 — skipping (no gates run)" >&2 + exit 0 +fi + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +REPO_ROOT="$(git rev-parse --show-toplevel)" + +REF="HEAD" +RANGE="" +OUT="" +PRE_PUSH=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --pre-push) PRE_PUSH=1; shift ;; + --ref) REF="$2"; shift 2 ;; + --range) RANGE="$2"; shift 2 ;; + --out) OUT="$2"; shift 2 ;; + *) echo "worktree-run: unknown arg: $1" >&2; exit 3 ;; + esac +done + +ZEROS="0000000000000000000000000000000000000000" + +if [[ "$PRE_PUSH" == "1" ]]; then + # git pre-push stdin: + # Take the first non-delete line. `read -t` so a drained/absent stdin + # (lefthook, manual invocation) falls through instead of hanging. + while read -r -t 1 _lref lsha _rref rsha; do + [[ "$lsha" == "$ZEROS" ]] && continue # branch deletion — nothing to gate + REF="$lsha" + if [[ "$rsha" != "$ZEROS" ]]; then RANGE="${rsha}..${lsha}"; fi + break + done || true +fi + +SHA="$(git rev-parse --verify "${REF}^{commit}")" + +if [[ -z "$RANGE" ]]; then + # New branch or no stdin: scan everything since the fork point. + BASE="$(git merge-base "$SHA" '@{upstream}' 2>/dev/null \ + || git merge-base "$SHA" origin/HEAD 2>/dev/null \ + || true)" + if [[ -n "$BASE" && "$BASE" != "$SHA" ]]; then RANGE="${BASE}..${SHA}"; fi +fi + +[[ -n "$OUT" ]] || OUT="$(mktemp -t worktree-run-rows.XXXXXX)" + +# ---- disposable worktree ---------------------------------------------------- +WT_PARENT="$(mktemp -d -t worktree-run.XXXXXX)" +WT="$WT_PARENT/wt" +cleanup() { + # shellcheck disable=SC2317 # invoked via the EXIT trap + git -C "$REPO_ROOT" worktree remove --force "$WT" >/dev/null 2>&1 || true + # shellcheck disable=SC2317 + rm -rf "$WT_PARENT" +} +trap cleanup EXIT +git -C "$REPO_ROOT" worktree add --detach --quiet "$WT" "$SHA" + +WORST=0 +raise() { local ec=$1; if (( ec > WORST )); then WORST=$ec; fi; } + +TMP_ROWS="$WT_PARENT/rows" +mkdir -p "$TMP_ROWS" + +# ---- gate 1: verify (fail-closed) ------------------------------------------- +vec=0 +(cd "$WT" && bash "$SCRIPT_DIR/harness-hash.sh" --verify) >&2 || vec=$? +case "$vec" in + 0) echo "worktree-run: verify PASS" >&2 ;; + 3) echo "worktree-run: verify SKIP (no .harness-hash manifest at $SHA)" >&2 ;; + *) echo "worktree-run: verify FAIL — HARNESS_TAMPERED at $SHA" >&2; raise 2 ;; +esac + +# ---- gate 2: escape-scan on the push range (fail-closed) -------------------- +eec=0 +if [[ -n "$RANGE" ]]; then + # Run inside the worktree so the hash verification and diff both see the + # tree being pushed; the worktree shares the object database, so range + # refs resolve identically. + (cd "$WT" && bash "$SCRIPT_DIR/escape-scan.sh" --range "$RANGE" --json) \ + > "$TMP_ROWS/escape-scan.json" || eec=$? + case "$eec" in + 0) echo "worktree-run: escape-scan PASS ($RANGE)" >&2 ;; + 1) echo "worktree-run: escape-scan CHALLENGE ($RANGE)" >&2; raise 1 ;; + *) echo "worktree-run: escape-scan REFUSE ($RANGE)" >&2; raise 2 ;; + esac +else + echo "worktree-run: escape-scan SKIP (no range resolvable for $SHA)" >&2 +fi + +# ---- gates 3+4: conform + audit (advisory — never raise the exit code) ------ +python3 "$SCRIPT_DIR/conform.py" "$WT" > "$TMP_ROWS/conform.json" 2>/dev/null \ + || echo "worktree-run: conform INDETERMINATE (advisory)" >&2 +python3 "$SCRIPT_DIR/audit.py" --fast "$WT" > "$TMP_ROWS/audit.json" 2>/dev/null \ + || echo "worktree-run: audit INDETERMINATE (advisory)" >&2 + +# ---- assemble gate-result/v1 rows ------------------------------------------- +python3 - "$TMP_ROWS" "$OUT" <<'PY' +import json, pathlib, sys +rows_dir, out = pathlib.Path(sys.argv[1]), sys.argv[2] +rows = [] +for f in sorted(rows_dir.glob("*.json")): + try: + data = json.loads(f.read_text()) + except (json.JSONDecodeError, OSError): + continue + rows.extend(data if isinstance(data, list) else [data]) +pathlib.Path(out).write_text(json.dumps(rows, indent=2) + "\n") +print(f"worktree-run: {len(rows)} gate-result rows -> {out}", file=sys.stderr) +PY + +if (( WORST == 0 )); then + echo "worktree-run: CLEAN — safe to push $SHA" >&2 +else + echo "worktree-run: BLOCKED (exit $WORST) — fix or get an engineer-approved reason; do not weaken the gate" >&2 +fi +exit "$WORST" diff --git a/.harness-hash b/.harness-hash index faa69aea083..a981be889cd 100644 --- a/.harness-hash +++ b/.harness-hash @@ -1,3 +1,31 @@ +93548d4da0669738f0aef91d85918c57a53f7b215fb16f7d2732597345262ac2 .github/workflows/_ci-clients.yml +d929b9a7d2df25b3c5af44027ba54fbc8ed0ce68112cb73a31be3be7ea68832e .github/workflows/_ci-desktop-macos.yml +9be2006bd3944b72353abef829f636eb8cf024f1bf9304292071ba8e6afd17c1 .github/workflows/_ci-desktop.yml +650fd2cb0cac3e82d590ff754bf296ca5463ab3b8cd8948ff48a0dfd377f52b8 .github/workflows/_ci-relay.yml +455d3280326a3b199faf57c4106efd3fe13a2ecceeb0c9dec20a2417536e5205 .github/workflows/_ci-rust.yml +ec0aa6d0195b5d29947067f19c2c8dccbd2846a4ba869d2f7f925be9ad0e6a28 .github/workflows/_ci-security.yml +a1d06430fcbcd402c6cc1d44f414816a6c7dd03627fcc6516f157baf4826e77e .github/workflows/auto-tag-on-release-pr-merge.yml +204967c844449e80a92fa95ccc1ed52211dc6f929563f741c16a75be107814cc .github/workflows/benchmark-harbor.yml +310300d5be2da6e91b2014eb3fc054a9a486c233685572998cc7940adb2fa98f .github/workflows/ci.yml +3c45d230ece229c5791036f210b8744eaef091b11fa3f9c7f0876359faf26c49 .github/workflows/codex-security-review.yml +777c37a608112f13cf1910b635e61a033e1350a07ac41b8feec0ffe4669f4189 .github/workflows/desktop-release-cache-proof.yml +52e3bd2c27e49e178b30f7211cfe582d6cda97757f925517993713b4e9400751 .github/workflows/desktop-release-candidate.yml +0de9d5be3626590c7dc5ae1c26847fcc001783ca7b34cbb578cd095f3dcddbcf .github/workflows/docker.yml +fed235dab5439493114cc9e158b07f0effeeebba35951ea1421fa84be8ab676c .github/workflows/fork-gates.yml +3aa2b028031b68e88f761512014d286492756536a2290c0b97fa0a956475866a .github/workflows/helm-chart.yml +1d1a472534c2d784cc89a248b359e3b2aabf5951dce7187dc19b2eae9e718f87 .github/workflows/linux-canary.yml +aeb769f5e09b66d37e71d5f8fa0f68344323f7ac9136d05009af1ad5e6a02021 .github/workflows/macos-intel-canary.yml +719bd017ae76399ac49968de1ec88bcf0a86e73a22877ce9386837b920854603 .github/workflows/mesh-lifecycle.yml +f6a8a658d0a1ddc05c44d769f0699797d7f45bd933cadb0fe16ad69e9ae723a8 .github/workflows/minio-image.yml +8107fd090f7058fdcbf9d34fbf8ec0169fdb137779730f7df8dbcf028d7e787c .github/workflows/mobile-release-candidate.yml +957a92722f5a071f3c4b6c2fb3d941d2768d9a12e9f0375c11c00e1c8e3b0f00 .github/workflows/promote-oss-desktop-release.yml +c90d07dc6606043daa1112579a566a5dc1b6fdbfbcc46215fbb83b59cf1c7705 .github/workflows/push-gateway-helm-chart.yml +3adb1a138d577ce96d158f223ca38e7a025332fbb0a101292c34954dbcbe607e .github/workflows/release.yml +799671acd31d2b68bb74a95afad81711c1d30b9f6a68b85eadce251674a9b64c .github/workflows/signed-macos-canary.yml +b2e03e59b4204cb930c4d9564208d68a3ebc4320cea9fa94a3ef78d3d20a8616 .github/workflows/sprig-image.yml +5f0acd1d87facf33f5a832d51b3153e8fe094eb38e377a2fc72a85bf06425b88 .github/workflows/sprig.yml +4de45ecd84e5cc5d4299e0b26948d2bbab5886998ff6c3043efcf39326559129 .github/workflows/staging-dev-relay-image.yml +d16ebc5cdfbd8e3121ed34ce152aa9c42285f6a9cd3ebc3a985b13307a6cffa0 .github/workflows/windows-canary.yml 693b40b91b58f0bbb14850f6ade06543e06711e2a630cfb0921ac81831e78475 000-docs/000-INDEX.md 44c9f7341fbf1503988685c53377495bd16b5f25de46683f39080fbcf0b698d0 000-docs/001-PP-PLAN-buzz-adoption-master-blueprint.md def765bd4f637f1b2cfe5abdb508837e3247b771c8530d0ef394a587c6adc250 000-docs/002-DR-DECR-buzz-adoption-decision-record.md @@ -10,8 +38,10 @@ fbeaab57a0f6fe0ab1cb876154b1b356e69e88f2803440eeeeb26689723eafdf 000-docs/008-A 6f0ded9cf97ef7be290d20ae7f691379ebe2d523dca7c93be87e5a6c213c8201 000-docs/009-AA-AUDR-fork-contract-breach-2026-08-16.md 9283c9e1ea61318c453ce0f2bc4e2ef944ee06b565d8e210c98e524bc9a37029 FORK.md e27febff4e5caeaebda952139681dff83f582c793fef8b4773c810a1e571fb3f REVIEW.md +bc2d49e33dcf272aef4340ebe6903d50eb892e1eb5daeb1131d249666c56c7d6 TESTING.md 43f3750a570e82cfa0c0cdef6ff51d962f7dbd98f892b590c0a30955043c0ab4 TEST_AUDIT.md cc43f7e468b229f00bbabe4c91661c10d3a52080ba5455b13aa356fed4c0a396 lefthook-local.yml +b657ada55607de606b5a49d233ff6f3c2243e1f117a65f523e36084caadae957 package.json 1930fe68667c7c7b5e32ebc52df208edf866658f00c9948cd75b12d5728de66b scripts/fork-gates/check-additive-only.sh ccadeaf1be721528e9aa9955725dae2303abfb0b350ab4a23b451494ce04e47f scripts/fork-gates/check-must-survive.sh 4fddc2945e53ca49c40dd067b0f6bff2e86cc1e111ce4743b24f4968d14e9d0b scripts/fork-gates/test-additive-only.sh