diff --git a/docs/gate/fresh-baseline.md b/docs/gate/fresh-baseline.md index cf70061..3a70e14 100644 --- a/docs/gate/fresh-baseline.md +++ b/docs/gate/fresh-baseline.md @@ -4,7 +4,7 @@ doc_type: spec status: draft owner: B4 created: 2026-08-14 -updated: 2026-08-14 +updated: 2026-09-06 confidence: MED supersedes: null sources_verified: true @@ -186,7 +186,49 @@ ADR-0010 split one overloaded field into two: This harness follows the code at `HEAD`, not the issue's original point 4: `cost_fresh` feeds `repairCostVsFresh()` only. `amortizedTokensOverN()` stays `no_data` until a `cost_program_build` is separately measured and attached via `ReplayRunnerOptions.costProgramBuild` / -`programBuildId` — a different measurement than this one, not yet built. +`programBuildId` — a different measurement than this one. + +### `--cost-program-build`: the wiring for the other half + +The driver can now carry that payment. **The wiring exists; the measurement does not** — those are +different claims and this section is careful to keep them apart. + +Before this, `ReplayRunner` accepted `costProgramBuild`/`programBuildId` and +`amortizedTokensOverN()` consumed them, but no path in between could pass one: `gate:matrix` had +`--cost-fresh` and nothing else. The §12 curve was therefore unreachable from the harness *even if +a number existed*. `gate:matrix --cost-program-build ` closes that, reading: + +```json +{ + "usable": true, + "program_build_id": "", + "cost_program_build": { + "tokens_in": 0, "tokens_out": 0, "wall_clock_ms": 0, "model_id": "" + } +} +``` + +It refuses (exit 2) on a missing file, invalid JSON, `usable != true`, a missing +`program_build_id`, or a **zero-token cost**. That last rule is stricter than `--cost-fresh`'s and +deliberately so: a zero `cost_fresh` makes §9 report `no_data`, but a zero `cost_program_build` +makes §12 report a *curve* — one declining to nothing, which publishes the strongest possible form +of the claim on a number nobody measured. + +The payment is attached to **exactly one** run in the whole matrix — the first live run to start — +through a latch (`programBuildPaymentLatch`) rather than an index check, because the driver has +several places a run can begin and "is this the first one?" would have to be right in all of them. +Attaching it to every run is precisely the arithmetic ADR-0010 exists to prevent: §12's numerator +sums the payment, so repeating it grows the numerator linearly with N and flattens the curve. +`out/matrix-run.json` records `cost_program_build_source`, the payment, the build id, and +`program_build_paid` — whether a run actually took it, since a matrix that skipped every version +leaves it unclaimed. + +**Nothing in this repo writes that document.** `cost_program_build` is what it cost to *produce* +the compiled program, and today that is a developer typing `src/recorder/cli.ts` by hand — a +developer-day and zero tokens ([#127](https://github.com/DevToolie/Paragent/issues/127)). Until an +agent records the trajectory, there is no token count to put in the file, and the loader will +refuse every attempt to fabricate one. The §12 curve stays `no_data`; what changed is that it will +compute the moment a real measurement exists, with no further code change. ## Status: harness only @@ -194,7 +236,9 @@ This PR ships the measurement **mechanism** — the client, the runner, the entr wiring into `gate:matrix`, and this definition. It does **not** ship a measured number, because: - No live model call has been made against this code. `cost_fresh` stays zeros; `repair cost vs - fresh` and `amortized tokens/task` stay `no_data` in `gate:report`'s output. + fresh` and `amortized tokens/task` stay `no_data` in `gate:report`'s output. The + `--cost-program-build` wiring added later for #39 step 4 does not change that: it transports a + measurement, it does not make one, and no producer for its input exists (#127). - A live baseline run costs real money and needs `ANTHROPIC_API_KEY`. Per CONTRIBUTING rule 3, *never invent a metric* — there is no number here to invent, and none is. - The issue's own checklist wants **at least 3 fresh runs, mean and spread both reported**, and @@ -248,6 +292,15 @@ table below is what a human reads out of it once a live run has actually happene ## Open questions / what I could not verify +- **The `--cost-program-build` document has no producer and therefore no validated example.** + Every test writes the file by hand. The shape is asserted against the loader, not against + anything that emits it, so the first real producer (#127) may find the fields named wrong. +- **"First live run pays" is a choice, not a derivation.** ADR-0010 says one run carries the + payment and that a recompile is a visible second payment; it does not say *which* run. The first + is the only one that makes the curve start at full price, but a matrix whose first version is + skipped mid-run could leave the payment on a run that is not the earliest recorded — the latch + hands it to the first run that *starts*, and `program_build_paid` is the only signal a reader + gets. - **No live measurement exists.** Every number this harness could produce is `[PENDING TRACK-1]` — see "Status" above. This doc defines the measurement; it does not report one. - **Whether the model's self-reported `success` is trustworthy enough to publish.** See "What diff --git a/experiments/gate-v1/live-run.ts b/experiments/gate-v1/live-run.ts index 4c61437..170bc89 100644 --- a/experiments/gate-v1/live-run.ts +++ b/experiments/gate-v1/live-run.ts @@ -193,6 +193,17 @@ export interface LiveRunOptions { * `ReplayRunnerOptions.costFresh` and ADR-0010. */ costFresh?: Cost; + /** + * Claim the matrix's ONE-TIME program-build payment (#39 step 4, ADR-0010). + * + * Called once per run. It answers with the payment for the **first** run that + * asks and `undefined` for every run after, across the whole matrix — which + * is the entire point: §12's numerator sums `cost_program_build`, so a + * payment repeated per run grows it linearly with N, flattens the curve, and + * shows nothing. A callback rather than a `Cost` because "was this the first + * run?" is not a question this function can answer — it sees one version. + */ + claimProgramBuildPayment?: () => { cost: Cost; program_build_id: string } | undefined; /** Repeats of the program against this version. Defaults to 1. */ runs?: number; /** @@ -422,6 +433,11 @@ export async function runVersionLive( } } + // Claimed here, immediately before the runner that will carry it, so a + // run that never starts cannot consume the payment and leave the curve + // with no first point. + const payment = opts.claimProgramBuildPayment?.(); + const runner = new ReplayRunner({ dryRun: false, page, @@ -433,6 +449,13 @@ export async function runVersionLive( : {}), ...(opts.repairClient ? { repairClient: opts.repairClient } : {}), ...(opts.costFresh ? { costFresh: opts.costFresh } : {}), + // Both or neither: ReplayRunner throws on a cost without an id. + ...(payment + ? { + costProgramBuild: payment.cost, + programBuildId: payment.program_build_id, + } + : {}), }); const params: ParamBindings = { diff --git a/experiments/gate-v1/run-matrix.ts b/experiments/gate-v1/run-matrix.ts index 9805b0a..8b42b97 100644 --- a/experiments/gate-v1/run-matrix.ts +++ b/experiments/gate-v1/run-matrix.ts @@ -142,6 +142,18 @@ export interface Args { * Absent means the pre-#39 default: `zeroCost()`, unchanged. */ costFresh?: string; + /** + * Path to a measured **one-time** program-build cost (#39 step 4, ADR-0010). + * Attached to exactly ONE run row in the whole matrix as + * `cost_program_build` + `program_build_id`, which is what moves + * `amortizedTokensOverN()` — the PRD §12 demo curve — off `no_data`. + * + * Not the same quantity as `--cost-fresh` and not interchangeable with it: + * `cost_fresh` is a per-run comparison baseline that belongs on every row, + * this is a capital cost that belongs on one. ADR-0010 exists because they + * were one field. + */ + costProgramBuild?: string; } /** @@ -202,6 +214,9 @@ const VALUED_FLAGS = { "cost-fresh": (args, value) => { args.costFresh = value; }, + "cost-program-build": (args, value) => { + args.costProgramBuild = value; + }, } satisfies Record void>; /** Flag names without the leading `--`, in declaration order. */ @@ -290,6 +305,15 @@ function usage(): void { off no_data. Refuses (exit 2) if the file is missing or not usable=true — a dry-run baseline or a zero-measured one is never wired in silently. Ignored under --dry-run. + --cost-program-build + Measured ONE-TIME cost of producing the compiled program + (#39 step 4, ADR-0010). Attached to exactly ONE run row as + cost_program_build + program_build_id, which is what moves + the §12 amortization curve off no_data. NOT a substitute + for --cost-fresh: that is a per-run baseline on every row, + this is a capital cost on one. Refuses (exit 2) if the file + is missing, not usable=true, has no program_build_id, or + measures zero tokens. Ignored under --dry-run. --headed Show the browser (live runs only). --keep-up Leave each container running after its run, for inspection. --no-preamble Skip the login preamble, for programs that log in as part @@ -419,6 +443,142 @@ export async function loadCostFreshBaseline(filePath: string): Promise { return doc.mean_cost_fresh; } +/** + * A measured one-time program-build payment, and the build it paid for. + * + * The two travel together because `ReplayRunner` refuses the cost without the + * id at construction (ADR-0010): an unattributed payment cannot be told apart + * from a double payment when a recompile shows up as a second step in the + * curve. + */ +export interface ProgramBuildPayment { + cost: Cost; + program_build_id: string; +} + +/** + * Load a measured one-time program-build cost (#39 step 4, ADR-0010). + * + * ## Nothing writes this file yet, and that is the point + * + * `cost_program_build` is what it cost to *produce* the compiled program. Today + * that is a developer typing `src/recorder/cli.ts` by hand, which costs a + * developer-day and zero tokens + * ([#127](https://github.com/DevToolie/Paragent/issues/127)). So there is no + * producer for this document in the repo, and this loader will refuse every + * attempt to fake one. + * + * That is deliberate. Until #39 step 4 was wired, the §12 curve PRD calls *the + * demo* could not be produced by anything in the tree even if a number existed + * — `ReplayRunner` accepted `costProgramBuild`, `amortizedTokensOverN()` + * consumed it, and no path in between passed one. This closes that gap so the + * curve computes the moment a measurement exists, rather than needing a code + * change at the same time. + * + * ## Expected document + * + * ```json + * { + * "usable": true, + * "program_build_id": "", + * "cost_program_build": { + * "tokens_in": 0, "tokens_out": 0, "wall_clock_ms": 0, "model_id": "" + * } + * } + * ``` + * + * ## Refuses rather than degrades + * + * Same posture as `loadCostFreshBaseline`, and one rule beyond it: **a + * zero-token build cost is rejected.** For `cost_fresh` a zero is caught by + * `usable`; here it is worth its own check, because a zero build cost does not + * produce `no_data` — it produces a *curve*, one that declines to nothing and + * reads as the strongest possible version of the claim. That is the exact + * failure #123 was filed about, arriving from the other direction. + */ +export async function loadCostProgramBuild( + filePath: string, +): Promise { + let text: string; + try { + text = await readFile(filePath, "utf8"); + } catch (err) { + const e = err as NodeJS.ErrnoException; + throw new Error( + `--cost-program-build ${filePath}: ` + + `${e.code === "ENOENT" ? "file not found" : errMessage(err)}. ` + + "Nothing in the repo writes this document yet — the compiled program is " + + "still hand-written (#127), so run 1 costs zero tokens. See " + + "docs/gate/fresh-baseline.md.", + { cause: err }, + ); + } + let parsed: unknown; + try { + parsed = JSON.parse(text); + } catch { + throw new Error(`--cost-program-build ${filePath}: not valid JSON`); + } + const doc = parsed as { + usable?: boolean; + not_a_measurement?: string; + program_build_id?: string; + cost_program_build?: Cost; + }; + if (doc.usable !== true) { + throw new Error( + `--cost-program-build ${filePath}: not usable ` + + `(${doc.not_a_measurement ?? "usable is not true"}). ` + + "A dry-run or unmeasured build cost cannot be wired into a gate run.", + ); + } + const cost = doc.cost_program_build; + if (!cost) { + throw new Error(`--cost-program-build ${filePath}: missing cost_program_build`); + } + if (typeof doc.program_build_id !== "string" || doc.program_build_id.length === 0) { + throw new Error( + `--cost-program-build ${filePath}: missing program_build_id. ` + + "ADR-0010 requires the id alongside the payment — an unattributed " + + "payment is indistinguishable from a double payment in the curve.", + ); + } + if ((cost.tokens_in ?? 0) + (cost.tokens_out ?? 0) <= 0) { + throw new Error( + `--cost-program-build ${filePath}: cost_program_build measures zero tokens. ` + + "Unlike cost_fresh, a zero here does not report no_data — it plots a " + + "curve declining to nothing, which publishes the strongest form of the " + + "claim on a number nobody measured (#123). Omit the flag instead.", + ); + } + return { cost, program_build_id: doc.program_build_id }; +} + +/** + * Hand out the one-time payment to the first run that asks, and to no other. + * + * §12's curve is `(sum(cost_program_build where present + repair + replay)) / N`. + * Attach the payment to every run and the numerator grows linearly with N, the + * mean goes flat, and the plot shows nothing — the same arithmetic ADR-0010 + * separated the two fields to prevent, reintroduced at the driver instead of in + * the schema. + * + * Made a latch rather than an index check because the driver has three places a + * run can start (dry, live, per-version repeats) and "is this the first one?" + * would have to be right in all of them. Here, correctness is a property of the + * closure: it can only answer once. + */ +export function programBuildPaymentLatch( + payment?: ProgramBuildPayment, +): () => ProgramBuildPayment | undefined { + let remaining = payment; + return () => { + const claim = remaining; + remaining = undefined; + return claim; + }; +} + function errMessage(err: unknown): string { return err instanceof Error ? err.message : String(err); } @@ -439,6 +599,12 @@ interface WalkOptions { shouldStop: () => boolean; /** Measured fresh-baseline (#39), attached to every LIVE run row's cost_fresh. */ costFresh?: Cost; + /** + * Claims the matrix's one-time program-build payment (#39 step 4, ADR-0010). + * Threaded through rather than resolved here so "exactly one run pays" holds + * across versions, not just within one. + */ + claimProgramBuildPayment?: () => ProgramBuildPayment | undefined; /** * Real repair client (#27), built from `--repair-model`. Absent means the * stub — `ReplayRunner`'s default, and what every run got before #165. @@ -507,6 +673,9 @@ async function walkVersions( ...(opts.repairClient ? { repairClient: opts.repairClient } : {}), ...(baseline ? { baseline } : {}), ...(opts.costFresh ? { costFresh: opts.costFresh } : {}), + ...(opts.claimProgramBuildPayment + ? { claimProgramBuildPayment: opts.claimProgramBuildPayment } + : {}), }); // A skip can now arrive *with* completed runs (interrupted partway, or the @@ -749,6 +918,38 @@ async function main(): Promise { } } + // #39 step 4, second half: the ONE-TIME program-build payment. Loaded beside + // --cost-fresh and validated the same way, but attached to exactly one run + // rather than broadcast — ADR-0010 splits the two because summing a per-run + // baseline over N flattens the §12 curve. + let programBuildPayment: ProgramBuildPayment | undefined; + if (args.costProgramBuild !== undefined) { + if (args.dryRun) { + console.log( + ` note: --cost-program-build is ignored under --dry-run — dry-run rows stay all-zero.`, + ); + } else { + try { + programBuildPayment = await loadCostProgramBuild(args.costProgramBuild); + const c = programBuildPayment.cost; + console.log( + ` cost-program-build: ${args.costProgramBuild} — ` + + `build=${programBuildPayment.program_build_id} tokens_in=${c.tokens_in} ` + + `tokens_out=${c.tokens_out} wall_clock_ms=${c.wall_clock_ms}` + + `${c.model_id ? ` model_id=${c.model_id}` : ""}`, + ); + console.log( + " (attached to the FIRST completed live run only — §12 amortizes one payment)", + ); + } catch (err) { + console.error(`gate:matrix: ${errMessage(err)}`); + process.exit(2); + return; + } + } + } + const claimProgramBuildPayment = programBuildPaymentLatch(programBuildPayment); + // #165: `--repair-model` reached `Args` and stopped there — nothing built a // client from it, so `ReplayRunner` fell back to `StubRepairModelClient` and // the run reported a self-heal rate of 0 and zero repair cost that both look @@ -829,6 +1030,12 @@ async function main(): Promise { process.on("SIGTERM", onSignal); const port = args.port ?? DEFAULT_HOST_PORT; + // Claiming after the walk answers "did a run take it?": the latch is empty + // iff some run already claimed it. Consuming it here is safe — the matrix is + // over — and it is the only way to know without threading a flag back up. + const buildPaymentClaimedByRun = (): boolean => + programBuildPayment !== undefined && claimProgramBuildPayment() === undefined; + const { runs, baseline } = await walkVersions({ walked, program, @@ -842,6 +1049,7 @@ async function main(): Promise { persist, shouldStop: () => stopRequested, ...(costFresh ? { costFresh } : {}), + ...(programBuildPayment ? { claimProgramBuildPayment } : {}), ...(repairClient ? { repairClient } : {}), }); @@ -889,6 +1097,19 @@ async function main(): Promise { // must not have to diff run rows against a source file to tell the // two apart. ...(costFresh ? { cost_fresh_source: args.costFresh, cost_fresh: costFresh } : {}), + // #39 step 4: the one-time payment, and — separately — whether a run + // actually took it. A matrix that skipped every version leaves the + // payment unclaimed, and a reader must be able to tell "the curve has + // no first point" from "the flag was never passed" without diffing + // NDJSON rows against this file. + ...(programBuildPayment + ? { + cost_program_build_source: args.costProgramBuild, + cost_program_build: programBuildPayment.cost, + program_build_id: programBuildPayment.program_build_id, + program_build_paid: buildPaymentClaimedByRun(), + } + : {}), runs, }, null, diff --git a/tests/unit/gate-matrix.test.ts b/tests/unit/gate-matrix.test.ts index 3aa61d6..acd4e46 100644 --- a/tests/unit/gate-matrix.test.ts +++ b/tests/unit/gate-matrix.test.ts @@ -30,6 +30,8 @@ import { assignValue, buildSection9Floor, loadCostFreshBaseline, + loadCostProgramBuild, + programBuildPaymentLatch, VALUED_FLAG_NAMES, type Args as MatrixArgs, } from "../../experiments/gate-v1/run-matrix.js"; @@ -577,6 +579,11 @@ describe("gate:matrix valued flags (#165)", () => { "task-key": { value: "open-dashboards-list", expect: (a) => a.taskKey, want: "open-dashboards-list" }, "repair-model": { value: "claude-opus-5", expect: (a) => a.repairModel, want: "claude-opus-5" }, "cost-fresh": { value: "/tmp/baseline.json", expect: (a) => a.costFresh, want: "/tmp/baseline.json" }, + "cost-program-build": { + value: "/tmp/build.json", + expect: (a) => a.costProgramBuild, + want: "/tmp/build.json", + }, }; const emptyArgs = (): MatrixArgs => ({ @@ -631,3 +638,120 @@ describe("gate:matrix valued flags (#165)", () => { expect(() => assignValue(emptyArgs(), "param", "novalue")).toThrow(/expects key=value/); }); }); + +// --------------------------------------------------------------------------- +// #39 step 4 (second half) — the ONE-TIME program-build payment. +// +// ADR-0010 split `cost_program_build` from `cost_fresh` because the two were +// one field read as two quantities. The driver then wired only the per-run +// half: `--cost-fresh` broadcast a baseline onto every row, and nothing could +// attach a build payment at all — so `amortizedTokensOverN()`, the PRD §12 +// demo curve, returned `no_data` no matter what was measured. +// +// Two things have to hold, and the second is the one that silently breaks: +// the loader must refuse anything unmeasured, and exactly ONE run in the whole +// matrix may carry the payment. +// --------------------------------------------------------------------------- + +describe("loadCostProgramBuild (#39 step 4 / ADR-0010)", () => { + let dir: string; + + beforeEach(() => { + dir = mkdtempSync(path.join(tmpdir(), "paragent-cost-build-")); + }); + + afterEach(() => { + rmSync(dir, { recursive: true, force: true }); + }); + + function writeDoc(overrides: Record): string { + const file = path.join(dir, "build.json"); + writeFileSync( + file, + JSON.stringify({ + usable: true, + program_build_id: "grafana-create-stat-dashboard@2026-08-14", + cost_program_build: { + tokens_in: 41_200, + tokens_out: 3_100, + wall_clock_ms: 612_000, + model_id: "claude-opus-5", + }, + ...overrides, + }), + "utf8", + ); + return file; + } + + it("returns the measured payment and the build it paid for", async () => { + const payment = await loadCostProgramBuild(writeDoc({})); + expect(payment.cost.tokens_in).toBe(41_200); + expect(payment.cost.model_id).toBe("claude-opus-5"); + expect(payment.program_build_id).toBe("grafana-create-stat-dashboard@2026-08-14"); + }); + + it("names the missing producer instead of falling through", async () => { + // Nothing in the repo writes this document — the program is still + // hand-written (#127). The error says so rather than reading as a typo. + await expect(loadCostProgramBuild(path.join(dir, "absent.json"))).rejects.toThrow( + /file not found/, + ); + }); + + it("refuses invalid JSON", async () => { + const file = path.join(dir, "bad.json"); + writeFileSync(file, "{not json", "utf8"); + await expect(loadCostProgramBuild(file)).rejects.toThrow(/not valid JSON/); + }); + + it("refuses a document that does not claim to be a measurement", async () => { + await expect( + loadCostProgramBuild(writeDoc({ usable: false, not_a_measurement: "dry run" })), + ).rejects.toThrow(/not usable \(dry run\)/); + }); + + it("refuses a payment with no build id", async () => { + // ReplayRunner throws on the pair anyway; failing here names why. + await expect( + loadCostProgramBuild(writeDoc({ program_build_id: "" })), + ).rejects.toThrow(/missing program_build_id/); + }); + + it("refuses a zero-token build cost — it plots a curve, not no_data", async () => { + // The dangerous case, and the reason this check exists beyond `usable`. A + // zero `cost_fresh` reports no_data; a zero `cost_program_build` reports a + // curve declining to nothing, which publishes the strongest form of the + // claim on a number nobody measured (#123). + await expect( + loadCostProgramBuild( + writeDoc({ + cost_program_build: { tokens_in: 0, tokens_out: 0, wall_clock_ms: 900 }, + }), + ), + ).rejects.toThrow(/measures zero tokens/); + }); +}); + +describe("programBuildPaymentLatch (#39 step 4 / ADR-0010)", () => { + const payment = { + cost: { tokens_in: 41_200, tokens_out: 3_100, wall_clock_ms: 612_000 }, + program_build_id: "build-1", + }; + + it("hands the payment to the first caller and to nobody else", () => { + // The whole §12 curve depends on this. Attach the payment to every run and + // the numerator grows linearly with N, the mean goes flat, and the demo + // plot shows nothing — ADR-0010's arithmetic, reintroduced at the driver. + const claim = programBuildPaymentLatch(payment); + expect(claim()).toEqual(payment); + expect(claim()).toBeUndefined(); + expect(claim()).toBeUndefined(); + }); + + it("answers undefined forever when no payment was passed", () => { + const claim = programBuildPaymentLatch(); + expect(claim()).toBeUndefined(); + expect(claim()).toBeUndefined(); + }); +});