diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 1650b09..c71051c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,7 +7,8 @@ hypothesis with inspectable evidence. 1. Read `AGENTS.md`, `PROJECT_CHARTER.md`, `reference/ARCHITECTURE_AND_ACCEPTANCE_BASELINE.md`, and the applicable phase context. -2. Run `git status --short --branch` and confirm the assigned branch and exclusive worktree. +2. Run `git status --short --branch` and confirm the assigned branch and exclusive checkout ownership. + Work sequentially in the saved checkout; add a worktree only for a concrete parallel need. 3. Create or update an ExecPlan under `plans/` for multi-step work. 4. Record the observed problem, writable-path allowlist, minimum acceptance condition, and explicit non-goals. diff --git a/PROJECT_CHARTER.md b/PROJECT_CHARTER.md index 58cd403..f8db490 100644 --- a/PROJECT_CHARTER.md +++ b/PROJECT_CHARTER.md @@ -17,6 +17,11 @@ It should: ## Primary user +The authorized 2026-09-05 experiment first accepts a maintainer-reviewed candidate test and two +local revisions, to measure the value of independent verification. This is a bounded entry point +into steps 4, 7 and 8 above; the original exploration/generation thesis remains unproven. Its +engineering acceptance does not change `SPIKE_CONDITIONAL` or the production gates below. + A maintainer of a web application who has: - a reproducible functional regression; diff --git a/README.md b/README.md index 091c6d7..e2937ae 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,15 @@ ReproLock investigates turning a user-supplied functional bug description or recorded browser workflow into an independently checked, maintainable, standalone Playwright regression test. -## Status — 2026-09-04 +## Status — Experimental -The current implementation is one private TypeScript root application with deterministic evidence -utilities, a loopback fixture, one case-specific local functional-regression Spike, and an -owner-authorized local Demo for that supplied case. +The current implementation is one TypeScript root application with deterministic evidence +utilities, a local candidate-test verifier, a loopback fixture and a verifiable case Demo. + +The new experimental entry point accepts a reviewed, self-contained Playwright test and two exact +local Git worktrees. It runs the same test on both versions, checks observed reset and assertion +steps, and exports evidence whose differential gate can be recomputed. See +[verify a candidate test](docs/local-verification.md) for the supported contract and commands. Safe Unfollow #163 has an unchanged standalone test with **20/20 expected pre-fix failures and 20/20 post-fix passes**. The honest manual/recorder baseline achieves the same differential and @@ -26,8 +30,6 @@ The decision remains **SPIKE_CONDITIONAL**. Evidence structure and checkpoint di but this unusually detailed case has not demonstrated enough incremental effort or maintenance value to authorize production implementation. Source hosting and passing CI do not grant product GO. There is no published package, general explorer/compiler, or released GitHub Action. -The recovered source and current engineering checks are available in -[private PR #1](https://github.com/AlbertXXuu/ReproLock/pull/1). ## Verifiable local Demo @@ -56,10 +58,9 @@ The screenshot shows the saved 2026-09-04 run in the current English interface; ## Development -Use the saved D-drive ReproLock project as the fixed entry point. Work sequentially on a task -branch in that checkout, then return it to the verified `main` after an authorized merge. -Add a worktree only for a concrete parallel need. See the -[ownership and cleanup policy](docs/foundation/branch-and-worktree-policy.md). +Clone the repository into your development directory, then install the pinned dependencies. +Contribute sequentially on a task branch in your checkout. The maintainer's saved D-drive checkout +follows the same [ownership and cleanup policy](docs/foundation/branch-and-worktree-policy.md). Node.js 24.20.0 is the pinned primary runtime; 22.23.2 is the minimum. pnpm is pinned to 11.19.0. The workspace fixes `enableGlobalVirtualStore: false` for both CI and ordinary terminals. @@ -68,6 +69,8 @@ replacing `node_modules`. Run the explicit frozen install below when dependencie no machine-specific store path or per-session environment override is required. ```bash +git clone https://github.com/AlbertXXuu/ReproLock.git +cd ReproLock corepack enable pnpm install --frozen-lockfile pnpm exec playwright install chromium @@ -75,6 +78,18 @@ pnpm check pnpm package:smoke ``` +If PowerShell cannot find `pnpm`, or `corepack enable` cannot write into the Node installation, +use `corepack pnpm` in place of `pnpm`. For example: + +```powershell +corepack pnpm install --frozen-lockfile +corepack pnpm exec playwright install chromium +corepack pnpm demo --config demo.local.json +``` + +This invokes the pinned package manager directly without requiring a global pnpm shim. A Node +installation with Corepack is required; reopen your terminal after installing the runtime. + `pnpm check` runs format, lint, brand/README validation, root and both standalone typechecks, unit/process tests, Chromium fixture and Demo browser tests, and verification of the checked-in Spike bundle. CI repeats those checks on @@ -119,6 +134,8 @@ internal consistency, and do not authenticate who executed the experiment. - `src/domain/` and `src/evidence/`: terminal contracts, canonical JSON and atomic evidence writing. - `src/demo/`: the bounded local Safe Unfollow Demo, runner, reporter and current-run verifier. +- `src/verify/`: explicit local candidate execution, native assertion observations and a separate + versioned differential-evidence contract. - `fixtures/loopback/` and `tests/`: executable foundation and bounded Spike regression checks. - `spikes/local-functional-regression/`: frozen standalone output, case-specific evidence tools, historical records and separate dated revalidation. @@ -133,8 +150,8 @@ Historical Wave 1 branches and stash objects remain preserved and are not curren ## Next gate -The owner has separately authorized the branded local single-case Demo. Removing the conditional status -requires a user-supplied, less-structured local functional case and an explicit scope/value decision. -That case must retain a model-free stable differential and show measurable effort or maintenance -benefit over an honest manual baseline. No second target, provider credentials, production packages, -public release or website work is admitted by this Demo task. +The current increment tests whether independent candidate verification is useful across supplied +local cases. Removing the conditional status requires a less-structured real case, measured effort +or maintenance benefit against an honest baseline, and an explicit scope/value decision. Fixture +checks establish engineering behavior; real maintainer adoption and saved human time remain open +questions. See the [current plan](plans/04-local-differential-verifier.md). diff --git a/docs/demo.md b/docs/demo.md index b026c61..25e05ec 100644 --- a/docs/demo.md +++ b/docs/demo.md @@ -14,6 +14,21 @@ pnpm install --frozen-lockfile pnpm exec playwright install chromium ``` +On Windows, if `pnpm` is not found or enabling shims needs administrator access, use +`corepack pnpm install --frozen-lockfile` and `corepack pnpm exec playwright install chromium`. +The start command also works as `corepack pnpm demo --config demo.local.json`. + +To prepare fresh targets, run the following from ReproLock in a directory where these sibling +names are unused. These are disposable local worktrees of the already registered example: + +```sh +git clone https://github.com/ignromanov/safe-unfollow.git ../safe-unfollow +git -C ../safe-unfollow worktree add --detach ../safe-unfollow-pre 64c8a1d0f4c1a9a4ffbab2ea319d89bcab21ad47 +git -C ../safe-unfollow worktree add --detach ../safe-unfollow-post ab55329e354dfb121486d7ff1f7daa2fa2e2e5fa +npm --prefix ../safe-unfollow-pre ci +npm --prefix ../safe-unfollow-post ci +``` + Copy `demo.example.json` to ignored `demo.local.json` and set both paths to the **supplied local** Safe Unfollow worktrees. Paths resolve relative to the shell's working directory. The required origin is `https://github.com/ignromanov/safe-unfollow.git`; exact clean revisions are: @@ -23,7 +38,10 @@ origin is `https://github.com/ignromanov/safe-unfollow.git`; exact clean revisio | pre-fix | `64c8a1d0f4c1a9a4ffbab2ea319d89bcab21ad47` | | post-fix | `ab55329e354dfb121486d7ff1f7daa2fa2e2e5fa` | -Run `npm ci` in each supplied target before starting. The Demo checks the committed lockfile, +For the fresh setup above, set `targets.pre-fix` to `../safe-unfollow-pre` and `targets.post-fix` +to `../safe-unfollow-post`. Existing prepared targets can be reused after prerequisite checks pass. + +Each target needs `npm ci` once, as shown in the fresh setup above. The Demo checks the committed lockfile, clean Git state, Vite installation and frozen test/config hashes. It starts the target's existing `vite` development script through its installed Node entry point with the fixed arguments `--host 127.0.0.1 --port 4173 --strictPort`. Keep 4173 free; it never kills another port owner. diff --git a/docs/local-verification.md b/docs/local-verification.md new file mode 100644 index 0000000..6b6585b --- /dev/null +++ b/docs/local-verification.md @@ -0,0 +1,109 @@ +# Verify a local candidate test + +This experimental CLI runs one reviewed Playwright test unchanged against two exact clean local +Git worktrees, then checks whether the observed business assertion fails before the fix and passes +after it. It uses the pinned Node and Playwright installation from ReproLock. Start with the +[installation instructions](../README.md#development). + +## Prepare the inputs + +Use a repository supplied or selected by its maintainer. Create disposable pre-fix and post-fix +worktrees from the same Git repository, install their locked dependencies, and inspect the existing +start command. V1 supports an explicit Node entry point inside each target, including an installed +Vite entry point. The application must serve only the configured loopback address. The coordinator +checks clean full revisions, tracked package/lock files and unchanged start-entry fingerprints. + +Review one self-contained TypeScript test importing only `@playwright/test`. Use the standard +`page`/`context` fixtures. The test must contain: + +- One top-level `test.step('reset', ...)`, including assertions of the actual initial state. A fresh + browser context alone does not reset server-side data; configure your disposable application and + explicitly assert its reset postcondition. +- One later top-level `test.step('outcome', ...)` with a single direct `expect(actual).toBe(expected)` + comparison of an observed scalar. Read the actual UI state before comparing it. Use the same + assertion callsite on both versions. An example is `expect(await page.title()).toBe('Editor')`. + +Actions and readiness waits belong between reset and outcome. Assertions should reflect the +maintainer's intended behavior. ReproLock checks execution evidence; human review establishes that +the reset and oracle mean what the issue requires. An assertion such as `expect(true).toBe(true)` +does not independently verify application state. + +Use one test with zero retries. Expected failures, skips, extra tests, incomplete steps, locator/API +errors, unknown matcher diagnostics and unobserved outcomes cannot confirm a differential. Native +value comparison diagnostics are parsed conservatively; V1 recognizes `toBe` scalar differences. +The test remains an ordinary Playwright file when used without ReproLock. + +## Configuration + +Save the following as `output/my-case.local.json`, replacing the paths and full revisions with +your supplied local inputs. Paths resolve relative to this configuration file: + +```json +{ + "schemaVersion": 1, + "candidate": "../my-case.spec.ts", + "targets": [ + { "path": "../../app-pre", "revision": "FULL_PRE_FIX_40_CHARACTER_SHA" }, + { "path": "../../app-post", "revision": "FULL_POST_FIX_40_CHARACTER_SHA" } + ], + "start": { + "nodeScript": "node_modules/vite/bin/vite.js", + "args": ["--host", "127.0.0.1", "--port", "4173", "--strictPort"] + }, + "origin": "http://127.0.0.1:4173", + "readyPath": "/", + "resetDescription": "Describe the actual reset and the UI postcondition checked by the test", + "repetitions": 3, + "timeoutMs": 180000, + "testTimeoutMs": 15000 +} +``` + +The array order is pre-fix, then post-fix. Repetitions are 1–20; the total execution deadline is +at most 25 minutes, with a maximum 60-second test timeout. The supported start entry uses Node, +an argument array and no shell. Configure the same entry point and arguments for both versions. + +```sh +pnpm regression check output/my-case.local.json +pnpm regression run output/my-case.local.json +pnpm regression verify output/verify/run-REPLACE_WITH_PRINTED_ID/export.json +``` + +`check` reads inputs and tests port availability without starting the application. `run` prints +the actual run directory, terminal status, per-attempt classifications and `differential`. +Exit 0 means the requested differential was confirmed; 2 means it was not confirmed, 124 is a +deadline and 130 is cancellation. `verify` independently recomputes the gate and exits 2 for a +partial, contradictory or non-differential bundle. Three repetitions do not satisfy the historical +Spike's separate 20+20 gate or establish broad reliability. + +Ctrl+C cancels the current run and stops its owned processes. It never kills an unrelated process +by port. Unknown cleanup prevents confirmation. On Windows this requires permission to inspect +and terminate your own processes; a restricted execution sandbox may leave cleanup unverified. + +## Evidence and trust + +Each `run` invocation creates a new `output/verify/run-*` directory. Frozen candidate and guard sources, +effective Playwright configuration, source/target fingerprints, partial observations and final +reports remain separate from historical Spike records. Completed attempts are atomically saved +as they occur, so interruption does not turn missing attempts into successes. + +The portable `export.json` contains the candidate, declared revisions, settings hashes, runtime +source hashes, target fingerprints and minimized observations. It includes native step categories, +assertion callsites, order and parent links, error digests and hashed expected/received scalar +representations. It omits error messages, screenshots, page bodies and target directory names. +Inspect the candidate itself before sharing; it is source code provided by the caller. The +`local-*` files and application logs stay local and can contain paths or application output. + +The verifier recomputes hashes, classifications, repetition identity and the pre/post gate from +the pinned reporter's observations. These records establish internal consistency, not authenticated +proof of who executed the test. The reporter and candidate are trusted executable code. The browser +guard restricts the standard fixtures to the configured origin and the runner passes a small OS +environment allowlist; this is a local execution boundary, not an untrusted-code sandbox. Custom +browser launches, hidden model/network calls and dynamic loading are outside the reviewed contract. + +## Value experiment + +The engineering fixture verifies the CLI contract. A second real case should measure end-to-end +setup, authoring, review, corrections, execution and later maintenance against an equally strong +manual baseline. Agent elapsed time is recorded as agent operation time; it is not a measurement +of saved developer hours. External adoption and human effort benefit remain separate product gates. diff --git a/harness/context/04-local-differential-verifier.md b/harness/context/04-local-differential-verifier.md new file mode 100644 index 0000000..ab1d998 --- /dev/null +++ b/harness/context/04-local-differential-verifier.md @@ -0,0 +1,65 @@ +# Local candidate differential verification + +## Scope and baseline + +Owner-approved follow-up to the 2026-09-05 public-readiness review. Base main: +`5b13e23bfda5ffc954c77c9bfbe7a4f35f7d75e2`, initially clean. Sole source writer on +`codex/local-differential-verifier`; independent reviewers read source and evidence. The active +AlvenX website task owns another project. Historical Spike files and recorded Demo exports remain +untouched. Product status remains `SPIKE_CONDITIONAL`. + +## Actual commands so far + +| Command | Observed result | +| --- | --- | +| Preserved early-cancel reproducer on the audited base | Direct cancellation lost; first HTTP trial returned 202 with no abort controller | +| `node --test tests/demo/run.test.ts`, before patch | Failed: actual `startup-error`, expected `cancelled` | +| Same test after patch | Passed; immediate/repeated cancel and shutdown preserve zero-attempt cancellation | +| `node --test tests/verify/cli.test.ts`, first implementation | Passed, real local Git/Chromium differential and operational negative controls, approximately 61 seconds | +| Same integration test after evidence tightening | Passed, approximately 56 seconds | +| `node node_modules/typescript/bin/tsc --project tsconfig.base.json --noEmit` | Passed after replacing unavailable TypeScript compiler API with Node's parse-only module API | + +Additional actual checks: + +- Ordinary-user PATH `pnpm --version`: 11.19.0. `pnpm check`: **82/82 unit/process and 4/4 browser + tests passed**, with formatter, lint, brand, types, historical evidence and recorded exports + passing. Local log: ignored `output/verify-check-20260905.log`. +- Expanded `node --test tests/verify/cli.test.ts`: passed in approximately 65 seconds, including + cancellation after a real completed attempt, retained partial observations, verified cleanup + and dirty-target rejection. +- Parent `python foundation/brand/validate_brand.py` and + `python operations/tools/validate_workspace.py`: passed. +- `pnpm package:smoke`: passed, 116 files / 391,169 bytes at that snapshot; final archive contents + additionally require the new verifier modules and usage document. +- Independent final source reviewer found no remaining submission-blocking issue. Documentation + reviewer verified ordinary-user `corepack pnpm --version` = 11.19.0 and corrected copy/setup order. + +Independent-export installation, second-case observations and GitHub delivery are pending. +None of these engineering checks is a product GO or an assertion of human time saved. + +## Stable current contract + +`pnpm regression check ` validates explicitly supplied local worktrees and a reviewed +self-contained Playwright candidate. `run` executes it unchanged on both clean expected commits. +`verify ` recomputes the new export's gate independently from any stored success flag. + +V1 supports a trusted Node start entry, loopback readiness, 1–20 repetitions, one test/worker, +zero retries, verified reset and one native scalar `toBe` outcome assertion at the same callsite. +No model is called. Original candidate, startup/reset configuration and oracle are reviewed inputs. +Outcome meaning cannot be established from labels or trivial assertions alone. + +Evidence uses a new schema independent of historical Spike/demo formats. Reporter observations +record bounded native step structure, callsites and hashed comparison/error data. Configuration, +runtime source and target fingerprints are bound to executions; completed attempts are retained +on interruption. Local diagnostic files are distinct from the portable candidate/observations. +Hashes establish consistency, not authenticated execution or an untrusted-code sandbox. + +## Value protocol + +Use an owner-selected second case with less structure than Safe Unfollow. Record provenance and +prior knowledge, establish reset/oracle on pre only, freeze candidate/config before post, and +retain every failed or inconclusive attempt. Compare an ordinary Playwright workflow with the +verifier using the same business coverage. Separate setup, authoring, corrections, review and +execution; report agent-operation measurements as such. Human time saved, independent usability, +maintainer retention and later maintenance require actual participants and remain unmeasured until +observed. Do not turn successful fixture or second-case execution into those missing claims. diff --git a/package.json b/package.json index 9192e1b..706b158 100644 --- a/package.json +++ b/package.json @@ -22,6 +22,7 @@ "check": "pnpm format:check && pnpm lint && pnpm brand:verify && pnpm typecheck && pnpm typecheck:standalone && pnpm test && pnpm evidence:verify && pnpm demo:verify:recorded", "brand:verify": "node scripts/verify-brand.mjs", "demo": "node src/demo/server.ts", + "regression": "node src/verify/cli.ts", "demo:verify": "node src/demo/evidence.ts", "demo:verify:recorded": "node scripts/verify-demo-recordings.mjs", "typecheck:standalone": "tsc --project spikes/local-functional-regression/generated/tsconfig.json --noEmit && tsc --project spikes/local-functional-regression/baseline/tsconfig.json --noEmit", diff --git a/plans/04-local-differential-verifier.md b/plans/04-local-differential-verifier.md new file mode 100644 index 0000000..32c3742 --- /dev/null +++ b/plans/04-local-differential-verifier.md @@ -0,0 +1,75 @@ +# Local differential verification — execution plan + +Owner authorization: 2026-09-05 conversation, following the public-readiness audit and the +proposal to accept a supplied local repository, two revisions and a candidate Playwright test. +Product decision remains `SPIKE_CONDITIONAL`; this is a bounded value-validation increment. + +## Baseline and ownership + +- Saved D-drive checkout, sole writer in this task, branch `codex/local-differential-verifier`. +- Base `5b13e23bfda5ffc954c77c9bfbe7a4f35f7d75e2`; initial status clean. The other active + AlvenX task is working on AlvenXWebsite. Read-only reviewers own no source paths. +- Prior independent archive checks: 80/80 unit/process, 4/4 browser, evidence and package pass; + latest base Node 22/24 CI passed. The base differs from the audited core only by header CSS. +- Preserved cancellation reproducer run on 2026-09-05: direct cancel not applied; first HTTP + trial returned start/cancel 202 while controller absent. Reproduce against current source with + a regression test before fixing; preserve old reproducer and all historical runs. + +## Writable scope + +`src/demo/run.ts`, focused lifecycle tests; new `src/verify/` and `tests/verify/` only for the +approved narrow caller; one small disposable fixture/example; package scripts; setup/usage docs, +README, CONTRIBUTING and the charter's current-experiment clarification, this plan and +`harness/context/04-local-differential-verifier.md` and integration build log. `src/demo/process.ts` +may accept an explicit non-inheriting environment for the new caller, preserving its Demo default. +No historical Spike bytes, recorded exports, brand assets, other projects, old worktrees, or +unrelated user files. Local fresh evidence goes under ignored output; no overwrites. + +## Ordered work and acceptance + +1. Add a failing early-cancel regression. Latch cancellation synchronously, including shutdown; + verify no unstarted attempts, truthful terminal state and repeated cancellation safety. +2. Implement the smallest explicit-config local candidate verifier. Reuse process ownership and + canonical writers. Bind clean exact target revisions and frozen candidate bytes; separate + reset, business assertion failure, infrastructure error and inconclusive outcomes. Use trusted + configured commands, loopback readiness, zero retries, bounded time/output and owned cleanup. + Preserve inspectable per-run evidence and independently recompute the differential gate. +3. Exercise the real CLI against disposable revisioned fixtures and negative controls. A fixture + is engineering acceptance, never the second real product case or a measured human baseline. +4. Improve ordinary-terminal setup, complete target preparation and external-reader quickstart. + Keep English product documentation/UI. No visual changes planned. +5. Locate already supplied material for the second, less structured real case. If absent, request + the repository, immutable issue and pre/post revisions while finishing all unblocked work. + Pre-register total human setup/review/edit time and later maintenance comparison; do not invent + timings, independent participants or adoption. No external messages without authorization. +6. Independent review, targeted regressions, complete checks, parent brand/workspace checks, + package archive and independent install/CLI smoke. Commit, push, PR, follow CI, merge after + all required checks, return D checkout to verified main as previously authorized. + +## Completion and limits + +Report exact checks, run artifacts, commit/PR/CI and remaining product inputs. Source integration +does not grant product GO or change GitHub visibility. The public switch is a separate reviewed +decision after the public-readiness items are complete. No npm publication or new provider work. + +## Progress and decisions + +- Current-source `node --test tests/demo/run.test.ts` first failed with actual `startup-error` + versus expected `cancelled`; it passed after synchronously creating the abort controller. +- Two successive real `node --test tests/verify/cli.test.ts` executions passed against disposable + committed Node fixtures, using actual Chromium. Negative controls include ordinary thrown marker + text, browser closure, refreshed report hashes with contradictory records, and a deadline. +- Read-only review found missing partial-attempt retention and source/target bindings; both were + added before broad acceptance. Native `toBe` scalar comparison, one outcome assertion/callsite, + and reviewed reset keep the first caller bounded; locator failures remain inconclusive. +- The pinned TypeScript 7 package exposes no former compiler parser API. Candidate static imports + are parsed without evaluation using Node's VM module parser and native TypeScript stripping; + no parser dependency was added. Trusted candidate review remains required. +- Second-case input: owner conditionally selected the existing DrawDB candidate on 2026-09-05, + subject to provenance and fit checks. A read-only/source reviewer owns a separate fresh local + experiment directory for pre-only investigation; no ReproLock source or historical bytes there + are modified. Record public-source selection and prior title hints honestly. +- Current implementation passed 82/82 unit/process checks and 4/4 browser checks; the expanded + real CLI regression separately passed cancellation after a completed attempt, preserved partial + observations, verified cleanup and dirty-target rejection. Parent brand/workspace checks passed. + Independent source and documentation review found no remaining submission blocker. diff --git a/scripts/package-smoke.mjs b/scripts/package-smoke.mjs index 4f4d3ef..d0b5feb 100644 --- a/scripts/package-smoke.mjs +++ b/scripts/package-smoke.mjs @@ -33,6 +33,10 @@ try { assert.ok(entries.includes("package/package.json")); assert.ok(entries.includes("package/src/domain/verdict.ts")); for (const path of [ + "src/verify/cli.ts", + "src/verify/evidence.ts", + "src/verify/reporter.ts", + "docs/local-verification.md", "src/demo/server.ts", "src/demo/public/index.html", "src/demo/public/app.js", @@ -42,7 +46,10 @@ try { "docs/assets/InstrumentSans-wdth-wght.woff2", "docs/assets/InstrumentSans-OFL.txt", ]) - assert.ok(entries.includes(`package/${path}`), `standalone Demo archive must contain ${path}`); + assert.ok( + entries.includes(`package/${path}`), + `standalone source archive must contain ${path}`, + ); assert.ok( entries.includes( "package/spikes/local-functional-regression/generated/safe-unfollow-163.spec.ts", diff --git a/src/demo/process.ts b/src/demo/process.ts index ad1d329..de5563b 100644 --- a/src/demo/process.ts +++ b/src/demo/process.ts @@ -6,8 +6,8 @@ const execute = promisify(execFile); type ProcessIdentity = { pid: number; parent: number; started: string }; /** Windows environment names are case insensitive, including in nested package-manager shells. */ -export function childEnvironment(extra: NodeJS.ProcessEnv = {}): NodeJS.ProcessEnv { - const entries = Object.entries({ ...process.env, ...extra }); +export function childEnvironment(extra: NodeJS.ProcessEnv = {}, inherit = true): NodeJS.ProcessEnv { + const entries = Object.entries({ ...(inherit ? process.env : {}), ...extra }); return Object.fromEntries( entries.map(([key, value]) => [process.platform === "win32" ? key.toUpperCase() : key, value]), ); @@ -54,10 +54,16 @@ export class OwnedProcess { private observing: Promise | null = null; private readonly timer: NodeJS.Timeout; - constructor(command: string, args: string[], cwd: string, extra: NodeJS.ProcessEnv = {}) { + constructor( + command: string, + args: string[], + cwd: string, + extra: NodeJS.ProcessEnv = {}, + inherit = true, + ) { this.child = spawn(command, args, { cwd, - env: childEnvironment(extra), + env: childEnvironment(extra, inherit), shell: false, windowsHide: true, detached: process.platform !== "win32", diff --git a/src/demo/run.ts b/src/demo/run.ts index 3dae39a..a933a79 100644 --- a/src/demo/run.ts +++ b/src/demo/run.ts @@ -169,16 +169,18 @@ export class DemoRunner { this.cancellation?.abort(); } async shutdown(): Promise { - await this.startup?.catch(() => {}); this.cancel(); + await this.startup?.catch(() => {}); await this.completion; } start(): Promise { if (this.active) return Promise.reject(new Error("One run is already active")); + this.cancellation = new AbortController(); this.startup = this.initialize(); return this.startup.finally(() => { this.startup = null; + if (!this.completion) this.cancellation = null; }); } @@ -193,7 +195,6 @@ export class DemoRunner { this.directory = join(resolved, id); await mkdir(this.directory); this.reports.clear(); - this.cancellation = new AbortController(); this.live = { run: { schemaVersion: 1, @@ -294,6 +295,7 @@ export class DemoRunner { live.run.sourceHashes[`demo/${name}`] = digest( await readFile(join(this.root, "src/demo", name)), ); + stopped(); const prerequisites = await checkPrerequisites(this.root, this.config); stopped(); if (!prerequisites.ok) { diff --git a/src/verify/cli.ts b/src/verify/cli.ts new file mode 100644 index 0000000..5569674 --- /dev/null +++ b/src/verify/cli.ts @@ -0,0 +1,538 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { + access, + mkdir, + mkdtemp, + readFile, + readdir, + realpath, + stat, + writeFile, +} from "node:fs/promises"; +import { createServer } from "node:net"; +import { dirname, isAbsolute, join, relative, resolve, sep } from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { promisify } from "node:util"; +import { chromium } from "@playwright/test"; +import { OwnedProcess } from "../demo/process.ts"; +import { serializeCanonicalJson } from "../evidence/canonical-json.ts"; +import { writeCanonicalJsonAtomically } from "../evidence/writer.ts"; +import { + type Bundle, + configurationDigest, + type Execution, + hash, + parseReport, + verifyBundle, +} from "./evidence.ts"; + +const root = resolve(dirname(fileURLToPath(import.meta.url)), "../.."); +const exec = promisify(execFile); +type Target = { path: string; revision: string }; +export type Config = { + schemaVersion: 1; + candidate: string; + targets: [Target, Target]; + start: { nodeScript: string; args: string[] }; + origin: string; + readyPath: string; + resetDescription: string; + repetitions: number; + timeoutMs: number; + testTimeoutMs: number; +}; +const inside = (parent: string, child: string): boolean => { + const distance = relative(parent, child); + return distance !== ".." && !distance.startsWith(`..${sep}`) && !isAbsolute(distance); +}; +const object = (value: unknown): Record => { + assert.ok( + value && typeof value === "object" && !Array.isArray(value), + "Configuration must contain objects", + ); + return value as Record; +}; +const bounded = (value: unknown, fallback: number, max: number): number => { + const result = value ?? fallback; + assert.ok( + typeof result === "number" && Number.isInteger(result) && result >= 1 && result <= max, + "Numeric limit out of range", + ); + return result; +}; + +/** Paths resolve against the configuration file, not the invoking shell. Commands are explicit data. */ +export async function readConfig(path: string): Promise { + const base = dirname(resolve(path)); + const c = object(JSON.parse(await readFile(path, "utf8"))); + assert.equal(c.schemaVersion, 1, "Unsupported configuration version"); + assert.ok(typeof c.candidate === "string", "A candidate file is required"); + assert.ok( + Array.isArray(c.targets) && c.targets.length === 2, + "Supply pre-fix then post-fix worktrees", + ); + const targets = c.targets.map((value) => { + const t = object(value); + assert.ok( + typeof t.path === "string" && + typeof t.revision === "string" && + /^[a-f0-9]{40}$/u.test(t.revision), + "Full target path and commit are required", + ); + return { path: resolve(base, t.path), revision: t.revision }; + }) as [Target, Target]; + assert.notEqual(targets[0].revision, targets[1].revision, "Choose two different revisions"); + const start = object(c.start); + assert.ok( + typeof start.nodeScript === "string" && + /^[a-zA-Z0-9_./-]+\.(?:[cm]?js|ts)$/u.test(start.nodeScript) && + !isAbsolute(start.nodeScript) && + !start.nodeScript.split("/").includes(".."), + "Use a relative trusted Node entry point", + ); + assert.ok( + Array.isArray(start.args) && + start.args.length <= 32 && + start.args.every( + (a) => typeof a === "string" && a.length <= 512 && !a.includes("\0") && !/[\r\n]/u.test(a), + ), + "Supply a bounded argument array", + ); + assert.ok(typeof c.origin === "string"); + const url = new URL(c.origin); + assert.ok( + url.protocol === "http:" && + url.hostname === "127.0.0.1" && + url.port && + !url.username && + !url.password && + url.href === `${url.origin}/`, + "Use an explicit http://127.0.0.1:port origin", + ); + assert.ok( + typeof c.readyPath === "string" && + c.readyPath.startsWith("/") && + new URL(c.readyPath, url).origin === url.origin && + !c.readyPath.includes("?"), + "Use a local readiness path", + ); + assert.ok( + typeof c.resetDescription === "string" && + c.resetDescription.trim().length >= 10 && + c.resetDescription.length <= 1000, + "Describe the reviewed reset and its observable postcondition", + ); + return { + schemaVersion: 1, + candidate: resolve(base, c.candidate), + targets, + start: { nodeScript: start.nodeScript, args: start.args as string[] }, + origin: url.origin, + readyPath: c.readyPath, + resetDescription: c.resetDescription, + repetitions: bounded(c.repetitions, 3, 20), + timeoutMs: bounded(c.timeoutMs, 180_000, 1_500_000), + testTimeoutMs: bounded(c.testTimeoutMs, 15_000, 60_000), + }; +} + +const environment = (): NodeJS.ProcessEnv => + Object.fromEntries( + Object.entries(process.env).filter(([key]) => + /^(?:PATH|PATHEXT|SYSTEMROOT|WINDIR|COMSPEC|TEMP|TMP|HOME|USERPROFILE|LOCALAPPDATA|APPDATA|PLAYWRIGHT_BROWSERS_PATH)$/iu.test( + key, + ), + ), + ); +async function git(path: string, args: string[]): Promise { + return ( + await exec("git", ["-C", path, ...args], { + timeout: 10_000, + maxBuffer: 1_048_576, + windowsHide: true, + env: environment(), + }) + ).stdout.trim(); +} +async function targetIdentity( + target: Target, + script: string, +): Promise<{ common: string; fingerprint: string }> { + const path = await realpath(target.path); + assert.equal( + await realpath(await git(path, ["rev-parse", "--show-toplevel"])), + path, + "Target must be a Git root", + ); + assert.equal(await git(path, ["rev-parse", "HEAD"]), target.revision, "Target revision mismatch"); + assert.equal(await git(path, ["status", "--porcelain"]), "", "Target must be clean"); + const entry = await realpath(join(path, script)); + assert.ok(inside(path, entry), "Start entry must stay inside its target"); + const common = await realpath(resolve(path, await git(path, ["rev-parse", "--git-common-dir"]))); + await git(path, ["ls-files", "--error-unmatch", "package.json"]); + const sources: Record = { + entry: hash(await readFile(entry)), + package: hash(await readFile(join(path, "package.json"))), + }; + let locks = 0; + for (const name of ["package-lock.json", "pnpm-lock.yaml", "yarn.lock"]) { + try { + sources[name] = hash(await readFile(join(path, name))); + await git(path, ["ls-files", "--error-unmatch", name]); + locks++; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + } + } + assert.ok(locks > 0, "A committed target lockfile is required"); + return { common, fingerprint: hash(serializeCanonicalJson(sources)) }; +} +async function candidateSource(path: string): Promise { + assert.ok((await stat(path)).size <= 131_072, "Candidate exceeds 128 KiB"); + const source = await readFile(path, "utf8"); + // Parse without evaluating candidate code, using the pinned Node runtime's module parser. + const parsed = await exec( + process.execPath, + [ + "--experimental-vm-modules", + "--input-type=module", + "-e", + "import {SourceTextModule} from 'node:vm'; import {stripTypeScriptTypes} from 'node:module'; import {readFileSync} from 'node:fs'; console.log(JSON.stringify(new SourceTextModule(stripTypeScriptTypes(readFileSync(process.argv[1],'utf8'))).dependencySpecifiers));", + path, + ], + { timeout: 10_000, maxBuffer: 1_048_576, windowsHide: true, env: environment() }, + ); + const imports: unknown = JSON.parse(parsed.stdout); + assert.ok( + Array.isArray(imports) && + imports.length > 0 && + imports.every((value) => value === "@playwright/test"), + "V1 candidates import only @playwright/test", + ); + assert.ok( + !/\bimport\s*\(/u.test(source), + "Dynamic imports are outside the self-contained candidate contract", + ); + return source; +} +async function portFree(origin: string): Promise { + const server = createServer(); + await new Promise((accept, reject) => { + server.once("error", () => + reject(new Error("Target port is occupied; its owner was left running")), + ); + server.listen(Number(new URL(origin).port), "127.0.0.1", () => server.close(() => accept())); + }); +} +export async function check(config: Config): Promise { + await candidateSource(config.candidate); + await access(chromium.executablePath()); + await portFree(config.origin); + const pre = await targetIdentity(config.targets[0], config.start.nodeScript); + const post = await targetIdentity(config.targets[1], config.start.nodeScript); + assert.equal( + pre.common, + post.common, + "Targets must be worktrees of the same supplied repository", + ); + assert.notEqual(await realpath(config.targets[0].path), await realpath(config.targets[1].path)); +} + +/** Execute a reviewed local candidate unchanged on two exact clean worktrees; never infer commands. */ +export async function run(config: Config, cancellation: AbortSignal): Promise { + const candidate = await candidateSource(config.candidate); + const parent = join(root, "output/verify"); + await mkdir(parent, { recursive: true }); + assert.ok(inside(await realpath(root), await realpath(parent)), "Output escaped the coordinator"); + const directory = await mkdtemp(join(parent, "run-")); + const guard = `import { test } from '@playwright/test';\nimport './candidate.spec.ts';\ntest.beforeEach(async ({ context, baseURL }) => {\n await context.route('**/*', route => new URL(route.request().url()).origin === baseURL ? route.continue() : route.abort());\n await context.routeWebSocket('**/*', socket => { const url = new URL(socket.url()); url.protocol = url.protocol === 'ws:' ? 'http:' : 'https:'; if (url.origin === baseURL) socket.connectToServer(); else socket.close(); });\n});\n`; + const runtimeFiles = { + cli: join(root, "src/verify/cli.ts"), + reporter: join(root, "src/verify/reporter.ts"), + evidence: join(root, "src/verify/evidence.ts"), + process: join(root, "src/demo/process.ts"), + }; + const sourceHashes: Record = { candidate: hash(candidate), guard: hash(guard) }; + for (const [name, path] of Object.entries(runtimeFiles)) + sourceHashes[name] = hash(await readFile(path)); + const bundle: Bundle = { + schemaVersion: 1, + candidate, + candidateSha256: hash(candidate), + repetitions: config.repetitions, + status: "error", + executions: [], + revisions: [config.targets[0].revision, config.targets[1].revision], + fingerprints: [], + sourceHashes, + settings: { + origin: config.origin, + readyPath: config.readyPath, + startScript: config.start.nodeScript, + startArgsSha256: hash(serializeCanonicalJson(config.start.args)), + timeoutMs: config.timeoutMs, + testTimeoutMs: config.testTimeoutMs, + resetDescriptionSha256: hash(config.resetDescription), + }, + }; + const signal = AbortSignal.any([cancellation, AbortSignal.timeout(config.timeoutMs)]); + const stopped = (): void => { + signal.throwIfAborted(); + }; + await writeFile(join(directory, "candidate.spec.ts"), candidate, { flag: "wx" }); + await writeFile(join(directory, "guard.spec.ts"), guard, { flag: "wx" }); + const assertSources = async (): Promise => { + for (const [name, path] of Object.entries(runtimeFiles)) + assert.equal(hash(await readFile(path)), sourceHashes[name], "Runtime source changed"); + assert.equal( + hash(await readFile(config.candidate)), + bundle.candidateSha256, + "Candidate changed", + ); + assert.equal( + hash(await readFile(join(directory, "candidate.spec.ts"))), + bundle.candidateSha256, + "Frozen candidate changed", + ); + assert.equal( + hash(await readFile(join(directory, "guard.spec.ts"))), + sourceHashes.guard, + "Guard changed", + ); + }; + // Configuration is local-only: it may contain paths and reviewed startup arguments. + await writeCanonicalJsonAtomically({ + outputRoot: directory, + relativePath: "local-config.json", + value: config, + }); + await writeCanonicalJsonAtomically({ + outputRoot: directory, + relativePath: "started.json", + value: { + schemaVersion: 1, + startedAt: new Date().toISOString(), + candidateSha256: hash(candidate), + }, + }); + try { + stopped(); + await check(config); + stopped(); + for (const target of config.targets) + bundle.fingerprints.push((await targetIdentity(target, config.start.nodeScript)).fingerprint); + for (const [index, target] of config.targets.entries()) { + stopped(); + await portFree(config.origin); + const before = await targetIdentity(target, config.start.nodeScript); + assert.equal(before.fingerprint, bundle.fingerprints[index]); + await assertSources(); + const execution: Execution = { + revision: target.revision, + cleanBefore: true, + cleanAfter: false, + exitCode: null, + cleanup: false, + report: null, + reportSha256: null, + fingerprintBefore: before.fingerprint, + fingerprintAfter: null, + configurationSha256: configurationDigest(bundle), + }; + bundle.executions.push(execution); + const processes: OwnedProcess[] = []; + const reportFile = join(directory, `report-${index}.json`); + try { + const app = new OwnedProcess( + process.execPath, + [join(target.path, config.start.nodeScript), ...config.start.args], + target.path, + environment(), + false, + ); + processes.push(app); + await app.observe(); + const until = Date.now() + Math.min(30_000, config.timeoutMs); + let ready = false; + while (Date.now() < until && !app.exited) { + stopped(); + try { + const response = await fetch(new URL(config.readyPath, config.origin), { + redirect: "error", + signal: AbortSignal.any([signal, AbortSignal.timeout(1_000)]), + }); + ready = response.ok; + await response.body?.cancel(); + if (ready) break; + } catch { + stopped(); + } + await delay(100, undefined, { signal }); + } + assert.ok(ready && !app.exited, "Target readiness failed"); + stopped(); + const pwConfig = { + testDir: directory, + testMatch: "guard.spec.ts", + workers: 1, + retries: 0, + repeatEach: config.repetitions, + forbidOnly: true, + timeout: config.testTimeoutMs, + globalTimeout: config.timeoutMs, + outputDir: join(directory, `local-artifacts-${index}`), + use: { + baseURL: config.origin, + trace: "off", + screenshot: "off", + video: "off", + serviceWorkers: "block", + }, + reporter: [[join(root, "src/verify/reporter.ts"), { outputFile: reportFile }]], + }; + const configPath = join(directory, `playwright-${index}.config.mjs`); + const configSource = `export default ${JSON.stringify(pwConfig)};\n`; + await writeFile(configPath, configSource, { flag: "wx" }); + await assertSources(); + assert.equal( + hash(await readFile(configPath)), + hash(configSource), + "Effective configuration changed", + ); + const browser = new OwnedProcess( + process.execPath, + [join(root, "node_modules/@playwright/test/cli.js"), "test", "--config", configPath], + root, + environment(), + false, + ); + processes.push(browser); + await browser.observe(); + execution.exitCode = await new Promise((accept) => { + const abort = () => accept(cancellation.aborted ? 130 : 124); + signal.addEventListener("abort", abort, { once: true }); + void browser.closed.then((code) => { + signal.removeEventListener("abort", abort); + accept(code); + }); + if (signal.aborted) abort(); + }); + stopped(); + assert.ok( + !app.exited && !processes.some((p) => p.outputExceeded), + "Execution environment failed", + ); + assert.equal( + hash(await readFile(configPath)), + hash(configSource), + "Effective configuration changed", + ); + assert.ok((await stat(reportFile)).size <= 4_194_304, "Report exceeded 4 MiB"); + execution.report = parseReport(JSON.parse(await readFile(reportFile, "utf8"))); + execution.reportSha256 = hash(serializeCanonicalJson(execution.report)); + } finally { + const cleanups = []; + for (const owned of [...processes].reverse()) cleanups.push(await owned.stop()); + execution.cleanup = cleanups.length > 0 && cleanups.every((c) => c.verified); + execution.fingerprintAfter = await targetIdentity(target, config.start.nodeScript).then( + (after) => after.fingerprint, + () => null, + ); + execution.cleanAfter = execution.fingerprintAfter === before.fingerprint; + if (!execution.report) { + // Preserve completed attempts after termination without claiming an unfinished report is complete. + const partials = (await readdir(directory)) + .filter( + (name) => + name.startsWith(`report-${index}.json.partial-`) && /partial-\d+$/u.test(name), + ) + .sort((a, b) => Number(b.split("-").at(-1)) - Number(a.split("-").at(-1))); + for (const name of partials) { + try { + assert.ok((await stat(join(directory, name))).size <= 4_194_304); + execution.report = parseReport( + JSON.parse(await readFile(join(directory, name), "utf8")), + ); + execution.reportSha256 = hash(serializeCanonicalJson(execution.report)); + break; + } catch { + /* An interrupted atomic publication cannot become a completed observation. */ + } + } + } + for (const [n, owned] of processes.entries()) + await writeFile(join(directory, `local-process-${index}-${n}.log`), owned.output, { + flag: "wx", + }); + } + assert.ok(execution.cleanup && execution.cleanAfter, "Target or cleanup verification failed"); + await assertSources(); + } + bundle.status = "completed"; + } catch { + bundle.status = signal.aborted ? (cancellation.aborted ? "cancelled" : "timeout") : "error"; + } finally { + await writeCanonicalJsonAtomically({ + outputRoot: directory, + relativePath: "export.json", + value: bundle, + }); + await writeCanonicalJsonAtomically({ + outputRoot: directory, + relativePath: "verification.json", + value: verifyBundle(bundle), + }); + } + return directory; +} + +async function main(): Promise { + const [command, file, ...extra] = process.argv.slice(2); + assert.ok( + file && extra.length === 0 && ["check", "run", "verify"].includes(command ?? ""), + "Usage: pnpm regression config.json | pnpm regression verify export.json", + ); + if (command === "verify") { + assert.ok((await stat(file)).size <= 8_388_608, "Export exceeds 8 MiB"); + const result = verifyBundle(JSON.parse(await readFile(file, "utf8"))); + console.log(JSON.stringify(result)); + process.exitCode = result.differential ? 0 : 2; + return; + } + const config = await readConfig(file); + if (command === "check") { + await check(config); + console.log("Local configuration, revisions, dependencies and port are ready"); + return; + } + const controller = new AbortController(); + const cancel = (): void => controller.abort(); + process.once("SIGINT", cancel); + process.once("SIGTERM", cancel); + try { + const directory = await run(config, controller.signal); + const bundle = JSON.parse(await readFile(join(directory, "export.json"), "utf8")); + const result = verifyBundle(bundle); + console.log(JSON.stringify({ directory, status: bundle.status, ...result })); + process.exitCode = result.differential + ? 0 + : bundle.status === "cancelled" + ? 130 + : bundle.status === "timeout" + ? 124 + : 2; + } finally { + process.removeListener("SIGINT", cancel); + process.removeListener("SIGTERM", cancel); + } +} +if (process.argv[1] && pathToFileURL(resolve(process.argv[1])).href === import.meta.url) { + await main().catch(() => { + console.error( + "Configuration or preflight failed. Check the documented schema, clean revisions, dependencies and loopback port.", + ); + process.exitCode = 2; + }); +} diff --git a/src/verify/evidence.ts b/src/verify/evidence.ts new file mode 100644 index 0000000..147ba0b --- /dev/null +++ b/src/verify/evidence.ts @@ -0,0 +1,336 @@ +import assert from "node:assert/strict"; +import { createHash } from "node:crypto"; +import { serializeCanonicalJson } from "../evidence/canonical-json.ts"; + +export const hash = (value: string | Buffer): string => + createHash("sha256").update(value).digest("hex"); +export type Step = { + id: number; + parent: number | null; + category: "expect" | "test.step" | "other"; + label: "reset" | "outcome" | "other"; + begin: number; + end: number; + error: string | null; + line: number | null; + column: number | null; + comparison: { matcher: "toBe"; expected: string; received: string } | null; +}; +export type Observation = { + repetition: number; + retry: number; + expectedStatus: string; + status: string; + errors: string[]; + steps: Step[]; +}; +export type Report = { + schemaVersion: 1; + planned: number; + status: string; + errors: string[]; + observations: Observation[]; +}; +export type Verdict = "pass" | "functional-failure" | "inconclusive"; +const sha = /^[a-f0-9]{64}$/u; +const record = (value: unknown): Record => { + assert.ok(value && typeof value === "object" && !Array.isArray(value), "Expected an object"); + return value as Record; +}; +const integer = (value: unknown, min: number, max: number): void => { + assert.ok( + Number.isInteger(value) && Number(value) >= min && Number(value) <= max, + "Invalid integer", + ); +}; +const hashes = (value: unknown): void => { + assert.ok( + Array.isArray(value) && + value.length <= 200 && + value.every((v) => typeof v === "string" && sha.test(v)), + "Invalid error hashes", + ); +}; + +/** Validate untrusted persisted observations before deriving any outcome. */ +export function parseReport(value: unknown): Report { + const r = record(value); + assert.equal(r.schemaVersion, 1); + integer(r.planned, 0, 20); + assert.ok(["passed", "failed", "timedout", "interrupted"].includes(String(r.status))); + hashes(r.errors); + assert.ok(Array.isArray(r.observations) && r.observations.length <= 20); + for (const item of r.observations) { + const o = record(item); + integer(o.repetition, 0, 19); + integer(o.retry, 0, 20); + assert.ok( + ["passed", "failed", "timedOut", "skipped", "interrupted"].includes(String(o.status)), + ); + assert.ok(["passed", "failed", "skipped"].includes(String(o.expectedStatus))); + hashes(o.errors); + assert.ok(Array.isArray(o.steps) && o.steps.length <= 200); + const ids = new Set(); + const events = new Set(); + for (const item of o.steps) { + const s = record(item); + integer(s.id, 0, 199); + integer(s.begin, 0, 399); + integer(s.end, -1, 399); + assert.ok(!ids.has(Number(s.id))); + ids.add(Number(s.id)); + assert.ok(!events.has(Number(s.begin))); + events.add(Number(s.begin)); + if (s.end !== -1) { + assert.ok(Number(s.end) > Number(s.begin) && !events.has(Number(s.end))); + events.add(Number(s.end)); + } + if (s.parent !== null) { + integer(s.parent, 0, 199); + assert.ok(Number(s.parent) < Number(s.id)); + } + assert.ok(["expect", "test.step", "other"].includes(String(s.category))); + assert.ok(["reset", "outcome", "other"].includes(String(s.label))); + assert.ok(s.error === null || (typeof s.error === "string" && sha.test(s.error))); + if (s.line !== null) integer(s.line, 1, 100_000); + if (s.column !== null) integer(s.column, 1, 100_000); + assert.equal(s.line === null, s.column === null); + if (s.comparison !== null) { + const comparison = record(s.comparison); + assert.equal(comparison.matcher, "toBe"); + hashes([comparison.expected, comparison.received]); + assert.ok(s.category === "expect" && s.error !== null); + } + } + for (const item of o.steps) { + const s = item as Step; + if (s.parent !== null) { + const parent = (o.steps as Step[]).find((p) => p.id === s.parent); + assert.ok( + parent && + parent.begin < s.begin && + (parent.end === -1 || (s.end !== -1 && parent.end > s.end)), + ); + } + } + } + return value as Report; +} + +/** Only observed native assertion differences after a verified reset can be business failures. */ +export function classify(o: Observation): Verdict { + if ( + o.retry !== 0 || + o.expectedStatus !== "passed" || + !["passed", "failed"].includes(o.status) || + o.steps.some((s) => s.end < 0) + ) + return "inconclusive"; + const resets = o.steps.filter((s) => s.category === "test.step" && s.label === "reset"); + const outcomes = o.steps.filter((s) => s.category === "test.step" && s.label === "outcome"); + const reset = resets[0], + outcome = outcomes[0]; + if ( + resets.length !== 1 || + outcomes.length !== 1 || + !reset || + !outcome || + reset.parent !== null || + outcome.parent !== null || + reset.error || + reset.end >= outcome.begin + ) + return "inconclusive"; + const inside = (s: Step, parent: Step): boolean => { + let id = s.parent; + while (id !== null) { + if (id === parent.id) return true; + id = o.steps.find((p) => p.id === id)?.parent ?? null; + } + return false; + }; + const resetChecks = o.steps.filter((s) => s.category === "expect" && inside(s, reset)); + const checks = o.steps.filter((s) => s.category === "expect" && inside(s, outcome)); + if ( + !resetChecks.length || + resetChecks.some((s) => s.error) || + checks.length !== 1 || + checks[0]?.line === null + ) + return "inconclusive"; + const failed = checks.filter((s) => s.error); + if (o.status === "passed" && !o.errors.length && !o.steps.some((s) => s.error)) return "pass"; + const failure = failed[0]; + if ( + o.status !== "failed" || + failed.length !== 1 || + !failure?.comparison || + failure.comparison.expected === failure.comparison.received || + o.errors.length !== 1 || + o.errors[0] !== failure.error || + outcome.error !== failure.error + ) + return "inconclusive"; + // Any unrelated hook/API/fixture error prevents a business classification, even if a marker matches. + if ( + o.steps.some( + (s) => + s.error && + (s.error !== failure.error || + (s.id !== failure.id && s.id !== outcome.id && !inside(failure, s))), + ) + ) + return "inconclusive"; + return "functional-failure"; +} + +export type Execution = { + revision: string; + cleanBefore: boolean; + cleanAfter: boolean; + exitCode: number | null; + cleanup: boolean; + report: Report | null; + reportSha256: string | null; + fingerprintBefore: string; + fingerprintAfter: string | null; + configurationSha256: string; +}; +export type Bundle = { + schemaVersion: 1; + candidate: string; + candidateSha256: string; + repetitions: number; + status: "completed" | "cancelled" | "timeout" | "error"; + revisions: [string, string]; + fingerprints: string[]; + sourceHashes: Record; + settings: { + origin: string; + readyPath: string; + startScript: string; + startArgsSha256: string; + timeoutMs: number; + testTimeoutMs: number; + resetDescriptionSha256: string; + }; + executions: Execution[]; +}; +export function configurationDigest( + bundle: Pick, +): string { + return hash( + serializeCanonicalJson({ + settings: bundle.settings, + sourceHashes: bundle.sourceHashes, + repetitions: bundle.repetitions, + }), + ); +} + +/** This gate is recomputed; no stored success flag or process exit code alone is authoritative. */ +export function verifyBundle(value: unknown): { + differential: boolean; + outcomes: Verdict[][]; + issues: string[]; +} { + try { + const b = record(value); + assert.equal(b.schemaVersion, 1); + assert.ok(typeof b.candidate === "string" && Buffer.byteLength(b.candidate) <= 131_072); + assert.equal(b.candidateSha256, hash(b.candidate)); + integer(b.repetitions, 1, 20); + assert.ok(["completed", "cancelled", "timeout", "error"].includes(String(b.status))); + assert.ok(Array.isArray(b.executions) && b.executions.length <= 2); + assert.ok( + Array.isArray(b.revisions) && + b.revisions.length === 2 && + b.revisions.every((r) => typeof r === "string" && /^[a-f0-9]{40}$/u.test(r)) && + b.revisions[0] !== b.revisions[1], + ); + hashes(b.fingerprints); + const sources = record(b.sourceHashes); + for (const name of ["candidate", "guard", "cli", "reporter", "evidence", "process"]) + assert.ok(typeof sources[name] === "string" && sha.test(sources[name])); + assert.equal(sources.candidate, b.candidateSha256); + const settings = record(b.settings); + assert.ok( + typeof settings.origin === "string" && /^http:\/\/127\.0\.0\.1:\d+$/u.test(settings.origin), + ); + assert.ok( + typeof settings.readyPath === "string" && + new URL(settings.readyPath, settings.origin).origin === settings.origin, + ); + assert.ok(typeof settings.startScript === "string"); + hashes([settings.startArgsSha256, settings.resetDescriptionSha256]); + integer(settings.timeoutMs, 1, 1_500_000); + integer(settings.testTimeoutMs, 1, 60_000); + const outcomes: Verdict[][] = []; + let eligible = b.status === "completed" && b.executions.length === 2; + const revisions: string[] = []; + for (const item of b.executions) { + const e = record(item); + assert.ok(typeof e.revision === "string" && /^[a-f0-9]{40}$/u.test(e.revision)); + revisions.push(e.revision); + for (const key of ["cleanBefore", "cleanAfter", "cleanup"]) + assert.equal(typeof e[key], "boolean"); + assert.ok(e.exitCode === null || (Number.isInteger(e.exitCode) && Number(e.exitCode) >= 0)); + assert.equal(e.revision, b.revisions[outcomes.length]); + hashes([e.fingerprintBefore, e.configurationSha256]); + assert.equal(e.configurationSha256, configurationDigest(value as Bundle)); + assert.equal(e.fingerprintBefore, (b.fingerprints as string[])[outcomes.length]); + if (e.fingerprintAfter !== null) hashes([e.fingerprintAfter]); + if (!e.report) { + assert.equal(e.reportSha256, null); + eligible = false; + outcomes.push([]); + continue; + } + const report = parseReport(e.report); + assert.equal(e.reportSha256, hash(serializeCanonicalJson(report))); + const observations = [...report.observations].sort((a, b) => a.repetition - b.repetition); + const verdicts = observations.map(classify); + outcomes.push(verdicts); + const expected = outcomes.length === 1 ? "functional-failure" : "pass"; + eligible &&= + e.cleanBefore === true && + e.cleanAfter === true && + e.cleanup === true && + e.exitCode === (expected === "pass" ? 0 : 1) && + e.fingerprintAfter === e.fingerprintBefore && + report.status === (expected === "pass" ? "passed" : "failed") && + report.errors.length === 0 && + report.planned === b.repetitions && + observations.length === b.repetitions && + observations.every((o, i) => o.repetition === i && verdicts[i] === expected); + } + eligible &&= revisions[0] !== revisions[1]; + const observed = (b.executions as Execution[]).flatMap((e) => e.report?.observations ?? []); + const callsites = observed + .flatMap((o) => + o.steps.filter( + (s) => + s.category === "expect" && + s.line !== null && + o.steps.some((p) => p.id === s.parent && p.label === "outcome"), + ), + ) + .map((s) => `${s.line}:${s.column}`); + eligible &&= callsites.length === Number(b.repetitions) * 2 && new Set(callsites).size === 1; + return { + differential: eligible, + outcomes, + issues: eligible + ? [] + : [ + "Complete clean pre-fix assertion failures and post-fix passes with verified cleanup are required", + ], + }; + } catch { + return { + differential: false, + outcomes: [], + issues: ["Evidence schema, trace or hash validation failed"], + }; + } +} diff --git a/src/verify/reporter.ts b/src/verify/reporter.ts new file mode 100644 index 0000000..6c3992f --- /dev/null +++ b/src/verify/reporter.ts @@ -0,0 +1,119 @@ +import { linkSync, unlinkSync, writeFileSync } from "node:fs"; +import { basename } from "node:path"; +import { stripVTControlCharacters } from "node:util"; +import type { + FullResult, + Reporter, + Suite, + TestCase, + TestError, + TestResult, + TestStep, +} from "@playwright/test/reporter"; +import { hash, type Observation, type Report, type Step } from "./evidence.ts"; + +const errorHash = (error: TestError): string => + hash(error.message ?? error.value ?? "unknown error"); +/** V1 deliberately accepts native value comparisons only; locator/API errors are inconclusive. */ +export function nativeComparison(message: string): Step["comparison"] { + const text = stripVTControlCharacters(message); + const expected = /^Expected: (.+)$/mu.exec(text)?.[1]; + const received = /^Received: (.+)$/mu.exec(text)?.[1]; + return /expect\(received\)\.toBe\(expected\)/u.test(text) && expected && received + ? { matcher: "toBe", expected: hash(expected), received: hash(received) } + : null; +} +function publish(path: string, value: unknown): void { + const temporary = `${path}.tmp`; + writeFileSync(temporary, JSON.stringify(value), { flag: "wx" }); + try { + linkSync(temporary, path); + } finally { + unlinkSync(temporary); + } +} + +export default class EvidenceReporter implements Reporter { + private readonly path: string; + private readonly report: Report = { + schemaVersion: 1, + planned: 0, + status: "interrupted", + errors: [], + observations: [], + }; + private sequence = 0; + private current: Step[] = []; + private readonly steps = new Map(); + private exceeded = false; + constructor(options: { outputFile: string }) { + this.path = options.outputFile; + } + onBegin(_config: unknown, suite: Suite): void { + this.report.planned = suite.allTests().length; + } + onTestBegin(): void { + this.sequence = 0; + this.current = []; + this.steps.clear(); + } + onStepBegin(_test: TestCase, _result: TestResult, step: TestStep): void { + if (this.current.length >= 200) { + if (!this.exceeded) { + this.exceeded = true; + this.report.errors.push(hash("Step budget exceeded")); + } + return; + } + const entry: Step = { + id: this.current.length, + parent: step.parent ? (this.steps.get(step.parent)?.id ?? null) : null, + category: + step.category === "expect" || step.category === "test.step" ? step.category : "other", + label: step.title === "reset" || step.title === "outcome" ? step.title : "other", + begin: this.sequence++, + end: -1, + error: null, + comparison: null, + line: + step.location && basename(step.location.file) === "candidate.spec.ts" + ? step.location.line + : null, + column: + step.location && basename(step.location.file) === "candidate.spec.ts" + ? step.location.column + : null, + }; + this.current.push(entry); + this.steps.set(step, entry); + } + onStepEnd(_test: TestCase, _result: TestResult, step: TestStep): void { + const entry = this.steps.get(step); + if (!entry) return; + entry.end = this.sequence++; + entry.error = step.error ? errorHash(step.error) : null; + entry.comparison = + step.category === "expect" && step.error ? nativeComparison(step.error.message ?? "") : null; + } + onTestEnd(test: TestCase, result: TestResult): void { + const observation: Observation = { + repetition: test.repeatEachIndex, + retry: result.retry, + expectedStatus: test.expectedStatus, + status: result.status, + errors: result.errors.map(errorHash), + steps: this.current, + }; + if (this.report.observations.length < 20) { + this.report.observations.push(observation); + publish(`${this.path}.partial-${this.report.observations.length}`, this.report); + } else this.onError({ message: "Observation budget exceeded" }); + } + onError(error: TestError): void { + if (this.report.errors.length < 200) this.report.errors.push(errorHash(error)); + } + onEnd(result: FullResult): void { + this.report.status = result.status; + publish(this.path, this.report); + } +} diff --git a/tests/demo/run.test.ts b/tests/demo/run.test.ts new file mode 100644 index 0000000..3d95261 --- /dev/null +++ b/tests/demo/run.test.ts @@ -0,0 +1,53 @@ +import assert from "node:assert/strict"; +import { cp, mkdir, mkdtemp, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import test from "node:test"; +import { setTimeout as delay } from "node:timers/promises"; +import { DemoRunner } from "../../src/demo/run.ts"; + +test("cancel during initialization is retained, with no attempted target execution", { + timeout: 15_000, +}, async () => { + const root = await mkdtemp(join(tmpdir(), "reprolock-early-cancel-")); + const runner = new DemoRunner(root, { + targets: { "pre-fix": join(root, "missing-pre"), "post-fix": join(root, "missing-post") }, + timeoutMs: 10_000, + }); + try { + await cp( + "spikes/local-functional-regression/generated", + join(root, "spikes/local-functional-regression/generated"), + { recursive: true }, + ); + await mkdir(join(root, "src/demo"), { recursive: true }); + for (const name of ["run.ts", "reporter.ts", "evidence.ts", "process.ts"]) + await cp(join("src/demo", name), join(root, "src/demo", name)); + for (const stop of [ + () => { + runner.cancel(); + runner.cancel(); + }, + () => runner.shutdown(), + ]) { + const pending = runner.start(); + assert.equal(runner.active, true); + const stopping = stop(); + const id = await pending; + await stopping; + const until = Date.now() + 10_000; + while (runner.active && Date.now() < until) await delay(10); + assert.equal(runner.active, false); + const final = JSON.parse(await readFile(join(root, "output/demo", id, "final.json"), "utf8")); + assert.equal(final.run.status, "cancelled"); + assert.deepEqual(final.run.executions, []); + assert.deepEqual(final.attempts, []); + assert.equal(final.verification.integrity, true); + assert.equal(final.verification.consistent, true); + assert.equal(final.verification.differential, false); + } + } finally { + await runner.shutdown(); + await rm(root, { recursive: true, force: true }); + } +}); diff --git a/tests/verify/cli.test.ts b/tests/verify/cli.test.ts new file mode 100644 index 0000000..16f5674 --- /dev/null +++ b/tests/verify/cli.test.ts @@ -0,0 +1,287 @@ +import assert from "node:assert/strict"; +import { execFile } from "node:child_process"; +import { mkdir, mkdtemp, readFile, readdir, realpath, rm, writeFile } from "node:fs/promises"; +import { createServer } from "node:net"; +import { tmpdir } from "node:os"; +import { join, relative, resolve, sep } from "node:path"; +import test from "node:test"; +import { setTimeout as delay } from "node:timers/promises"; +import { promisify } from "node:util"; +import { childEnvironment } from "../../src/demo/process.ts"; +import { serializeCanonicalJson } from "../../src/evidence/canonical-json.ts"; +import { hash, verifyBundle } from "../../src/verify/evidence.ts"; +import { readConfig, run } from "../../src/verify/cli.ts"; + +const execute = promisify(execFile); +const sourceRoot = process.cwd(); +const cli = join(sourceRoot, "src/verify/cli.ts"); +const candidate = `import { test, expect } from '@playwright/test'; +test('a cleared counter can be incremented', async ({ page }) => { + await test.step('reset', async () => { + await page.goto('/'); + expect(await page.getByRole('status').textContent()).toBe('0'); + }); + await page.getByRole('button', {name:'Increment'}).click(); + await test.step('outcome', async () => { + expect(await page.getByRole('status').textContent()).toBe('1'); + }); +});`; +async function freePort(): Promise { + const s = createServer(); + await new Promise((done) => s.listen(0, "127.0.0.1", done)); + const address = s.address(); + assert.ok(address && typeof address !== "string"); + await new Promise((done) => s.close(() => done())); + return address.port; +} +test("real CLI proves a local differential and rejects inconsistent or non-business evidence", { + timeout: 180_000, +}, async () => { + const base = await realpath(tmpdir()); + const directory = await mkdtemp(join(base, "reprolock-verifier-")); + const repo = join(directory, "repo"), + post = join(directory, "post"); + const git = async (...args: string[]): Promise => + ( + await execute("git", ["-C", repo, ...args], { + timeout: 10_000, + windowsHide: true, + env: childEnvironment(), + }) + ).stdout.trim(); + const runs: string[] = []; + try { + await mkdir(repo); + await git("init"); + await git("config", "user.name", "ReproLock fixture"); + await git("config", "user.email", "fixture@example.invalid"); + await writeFile( + join(repo, "package.json"), + JSON.stringify({ private: true, type: "module", scripts: { start: "node server.mjs" } }), + ); + await writeFile( + join(repo, "package-lock.json"), + JSON.stringify({ name: "disposable-fixture", lockfileVersion: 3, packages: {} }), + ); + const server = (value: number): string => + `import {createServer} from 'node:http'; createServer((_,r)=>{r.setHeader('content-type','text/html');r.end(${JSON.stringify(`

0

`)});}).listen(Number(process.argv[2]),'127.0.0.1');`; + await writeFile(join(repo, "server.mjs"), server(0)); + await git("add", "."); + await git("commit", "-m", "fixture: broken increment"); + const preSha = await git("rev-parse", "HEAD"); + await writeFile(join(repo, "server.mjs"), server(1)); + await git("add", "."); + await git("commit", "-m", "fixture: restore increment"); + const postSha = await git("rev-parse", "HEAD"); + await git("worktree", "add", "--detach", post, postSha); + await git("checkout", "--detach", preSha); + const spec = join(directory, "candidate.spec.ts"); + await writeFile(spec, candidate); + const port = await freePort(); + const config = { + schemaVersion: 1, + candidate: spec, + targets: [ + { path: repo, revision: preSha }, + { path: post, revision: postSha }, + ], + start: { nodeScript: "server.mjs", args: [String(port)] }, + origin: `http://127.0.0.1:${port}`, + readyPath: "/", + resetDescription: + "Fresh browser context and navigation; independently assert initial counter zero", + repetitions: 2, + timeoutMs: 90_000, + testTimeoutMs: 10_000, + }; + const configPath = join(directory, "config.json"); + await writeFile(configPath, JSON.stringify(config)); + await execute(process.execPath, [cli, "check", configPath], { + timeout: 30_000, + windowsHide: true, + env: childEnvironment(), + }); + const invoke = async (): Promise<{ + directory: string; + status: string; + differential: boolean; + }> => { + const result = await execute(process.execPath, [cli, "run", configPath], { + timeout: 120_000, + windowsHide: true, + env: childEnvironment(), + }).catch((error) => { + if (typeof error.stdout === "string" && error.stdout.trim().startsWith("{")) + return error as { stdout: string }; + throw error; + }); + const resultValue = JSON.parse(result.stdout.trim()); + runs.push(resultValue.directory); + return resultValue; + }; + const successful = await invoke(); + assert.equal(successful.differential, true, JSON.stringify(successful)); + const exportPath = join(successful.directory, "export.json"); + const bundle = JSON.parse(await readFile(exportPath, "utf8")); + assert.deepEqual(verifyBundle(bundle).outcomes, [ + ["functional-failure", "functional-failure"], + ["pass", "pass"], + ]); + await execute(process.execPath, [cli, "verify", exportPath], { + timeout: 10_000, + windowsHide: true, + }); + const mutations: ((b: typeof bundle) => void)[] = [ + (b) => { + b.executions[1].exitCode = 1; + }, + (b) => { + b.executions[0].cleanup = false; + }, + (b) => { + b.executions[0].cleanAfter = false; + }, + (b) => { + b.executions[1].report.observations[0].expectedStatus = "failed"; + }, + (b) => { + b.executions[1].report.observations[0].repetition = 1; + }, + (b) => { + b.executions[0].report.observations[0].steps.find( + (s: { label: string }) => s.label === "reset", + ).label = "other"; + }, + (b) => { + b.executions[0].report.observations[0].steps.find( + (s: { comparison: unknown }) => s.comparison, + ).comparison = null; + }, + (b) => { + b.executions[0].revision = b.executions[1].revision; + }, + (b) => { + b.executions[1].fingerprintAfter = "0".repeat(64); + }, + (b) => { + b.settings.origin = "http://127.0.0.1:9999"; + }, + (b) => { + b.executions[1].report.observations[0].steps = []; + }, + (b) => { + b.candidate += "\n// changed"; + }, + ]; + for (const mutate of mutations) { + const changed = structuredClone(bundle); + mutate(changed); + for (const e of changed.executions) e.reportSha256 = hash(serializeCanonicalJson(e.report)); + assert.equal(verifyBundle(changed).differential, false); + } + // Same error wording in ordinary user code is not evidence of a native assertion failure. + await writeFile( + spec, + candidate.replace( + "expect(await page.getByRole('status').textContent()).toBe('1');", + "throw new Error('expect(received).toBe(expected) Expected: 1 Received: 0');", + ), + ); + const misleading = await invoke(); + assert.equal(misleading.differential, false); + const negative = JSON.parse(await readFile(join(misleading.directory, "export.json"), "utf8")); + assert.deepEqual(verifyBundle(negative).outcomes, [ + ["inconclusive", "inconclusive"], + ["inconclusive", "inconclusive"], + ]); + // Test timeout is operational; even the pre-fix side cannot count as a functional failure. + await writeFile( + spec, + candidate.replace( + "expect(await page.getByRole('status').textContent()).toBe('1');", + "await page.close(); await expect(page.getByRole('status')).toHaveText('1');", + ), + ); + const closed = await invoke(); + assert.equal(closed.differential, false); + const closedBundle = JSON.parse(await readFile(join(closed.directory, "export.json"), "utf8")); + assert.ok( + verifyBundle(closedBundle) + .outcomes.flat() + .every((v) => v === "inconclusive"), + ); + // Cancel after an observed attempt; completed observations must survive terminating the worker. + const slow = candidate.replace( + "async ({ page }) => {", + "async ({ page }, info) => { if (info.repeatEachIndex > 0) await new Promise(resolve => setTimeout(resolve, 15000));", + ); + await writeFile(spec, slow); + config.repetitions = 20; + await writeFile(configPath, JSON.stringify(config)); + const previous = new Set(await readdir(join(sourceRoot, "output/verify"))); + const abort = new AbortController(); + const running = run(await readConfig(configPath), abort.signal); + let observed = false; + try { + const until = Date.now() + 40_000; + while (!observed && Date.now() < until) { + for (const name of (await readdir(join(sourceRoot, "output/verify"))).filter( + (name) => !previous.has(name), + )) { + try { + const report = JSON.parse( + await readFile( + join(sourceRoot, "output/verify", name, "report-0.json.partial-1"), + "utf8", + ), + ); + observed = report.observations.length === 1; + } catch { + /* Wait for the first immutable observation. */ + } + } + if (!observed) await delay(50); + } + } finally { + abort.abort(); + } + const cancelledDirectory = await running; + runs.push(cancelledDirectory); + assert.equal(observed, true, "First attempt was not persisted within the bounded wait"); + const cancelled = JSON.parse(await readFile(join(cancelledDirectory, "export.json"), "utf8")); + assert.equal(cancelled.status, "cancelled"); + assert.ok( + cancelled.executions[0].report.observations.length >= 1 && + cancelled.executions[0].report.observations.length < 20, + ); + assert.equal(cancelled.executions[0].cleanup, true); + assert.equal(verifyBundle(cancelled).differential, false); + // Dirty targets are rejected by preflight before any application starts. + await writeFile(join(repo, "untracked.txt"), "temporary fixture change"); + await assert.rejects( + execute(process.execPath, [cli, "check", configPath], { + timeout: 30_000, + windowsHide: true, + env: childEnvironment(), + }), + ); + await rm(join(repo, "untracked.txt")); + await writeFile(spec, candidate); + config.timeoutMs = 1; + await writeFile(configPath, JSON.stringify(config)); + const timed = await invoke(); + assert.equal(timed.status, "timeout"); + assert.equal(timed.differential, false); + assert.equal(await git("status", "--porcelain"), ""); + } finally { + await git("worktree", "remove", post).catch(() => {}); + for (const run of runs) { + const resolved = resolve(run), + output = resolve(sourceRoot, "output/verify"); + assert.ok(relative(output, resolved).startsWith("run-") && resolved.startsWith(output + sep)); + await rm(resolved, { recursive: true, force: true }); + } + assert.ok(directory.startsWith(base + sep)); + await rm(directory, { recursive: true, force: true }); + } +});