Skip to content

feat(gate): wire the one-time program-build cost into the matrix (#39 step 4) - #193

Open
myselfsiddharth wants to merge 1 commit into
mainfrom
feat/39-cost-program-build-wiring
Open

feat(gate): wire the one-time program-build cost into the matrix (#39 step 4)#193
myselfsiddharth wants to merge 1 commit into
mainfrom
feat/39-cost-program-build-wiring

Conversation

@myselfsiddharth

Copy link
Copy Markdown
Contributor

Part of #39 — its step 4, as ADR-0010 redefined it. Does not close #39: the live measurement is separate work that costs real money.

The gap

ReplayRunner has accepted costProgramBuild/programBuildId since ADR-0010, and amortizedTokensOverN() has consumed them. Nothing in between could pass one — gate:matrix had --cost-fresh and no equivalent.

So the PRD §12 amortization curve, the plot the PRD calls the demo, was unreachable from the harness even if a number existed. docs/gate/fresh-baseline.md said as much in its own words: "a different measurement than this one, not yet built."

#39's step 4 originally said to attach a baseline "so repairCostVsFresh() and amortizedTokensOverN() compute". The issue's own follow-up comment records that ADR-0010 turned that into two jobs; #191/#143 landed the first. This is the second.

What

gate:matrix --cost-program-build <path>, reading:

{
  "usable": true,
  "program_build_id": "<id>",
  "cost_program_build": { "tokens_in": 0, "tokens_out": 0, "wall_clock_ms": 0, "model_id": "<m>" }
}

Attached to exactly ONE run in the whole matrix. That is the entire point: §12's numerator sums cost_program_build, so a payment repeated per run grows it linearly with N, flattens the mean, and plots nothing — the same arithmetic ADR-0010 split the two fields to prevent, reintroduced at the driver instead of in the schema.

Implemented as a latch (programBuildPaymentLatch) rather than an index check: the driver has several places a run can start (dry, live, per-version repeats) and "is this the first one?" would have to be right in all of them. As a closure it can only answer once.

matrix-run.json records cost_program_build_source, the payment, the build id, and program_build_paid — whether a run actually took it, since a matrix that skipped every version leaves it unclaimed.

Refuses rather than degrades

Same posture as --cost-fresh (missing file, bad JSON, usable != true), plus a missing program_build_id — ADR-0010 requires the id alongside the payment, and ReplayRunner throws on the pair anyway; failing at load names why.

And one rule stricter than --cost-fresh: a zero-token build cost is rejected. A zero cost_fresh makes §9 report no_data. A zero cost_program_build makes §12 report a curve — one declining to nothing, which publishes the strongest possible form of the claim on a number nobody measured. That is #123's failure mode arriving from the other direction, and it is the one case here where the honest-looking default is the dangerous one.

The wiring exists; the measurement does not

Kept deliberately apart, because they are different claims.

Nothing in this repo writes that document. cost_program_build is what it cost to produce the compiled program, and today that is a developer typing src/recorder/cli.ts by hand — a developer-day and zero tokens (#127). The loader refuses every attempt to fabricate one, and says so by name in the not-found error rather than reading as a typo.

The §12 curve stays no_data. What changed is that it will compute the moment a real measurement exists, with no further code change.

Testing

npm run ci           # 578 unit + 36 integration, green
npm run test:canary  # 52, green
npm run gate:matrix -- --dry-run --cost-program-build <file>   # correctly ignored, rows stay all-zero

11 new tests: 6 on the loader (happy path, absent producer, bad JSON, not-usable, missing id, zero tokens), 2 on the latch's exactly-once behaviour, and the #165 flag-table walk now covers --cost-program-build — a flag accepted but never assigned fails there.

Docs

docs/gate/fresh-baseline.md gains the document format, the "exactly one run pays" rule and why, the zero-cost refusal and why it is stricter, and two open questions:

🤖 Generated with Claude Code

…step 4)

`ReplayRunner` has accepted `costProgramBuild`/`programBuildId` since ADR-0010
and `amortizedTokensOverN()` has consumed them, but nothing in between could
pass one: `gate:matrix` had `--cost-fresh` and no equivalent. So the PRD §12
amortization curve — the plot the PRD calls the demo — was unreachable from
the harness *even if a number existed*.

`gate:matrix --cost-program-build <path>` closes that. It reads a measured
one-time payment plus the build it paid for, and attaches it to exactly ONE
run in the whole matrix.

Exactly one is the entire point. §12's numerator sums `cost_program_build`, so
a payment repeated per run grows it linearly with N, flattens the mean, and
plots nothing — the same arithmetic ADR-0010 split the two fields to prevent,
reintroduced at the driver instead of in the schema. It is a latch rather than
an index check because the driver has several places a run can start, and "is
this the first one?" would have to be right in all of them; here it is a
property of the closure.

The loader refuses rather than degrades, matching `--cost-fresh`, with one
rule beyond it: a zero-token build cost is rejected. A zero `cost_fresh` makes
§9 report `no_data`; a zero `cost_program_build` makes §12 report a *curve*,
one declining to nothing, which publishes the strongest possible form of the
claim on a number nobody measured (#123 from the other direction).

`matrix-run.json` records the source, the payment, the build id, and
`program_build_paid` — whether a run actually took it, since a matrix that
skipped every version leaves it unclaimed.

**The wiring exists; the measurement does not.** Nothing in this repo writes
that document: `cost_program_build` is what it cost to *produce* the compiled
program, and today that is a developer typing `src/recorder/cli.ts` by hand —
a developer-day and zero tokens (#127). The loader refuses every attempt to
fabricate one. The §12 curve stays `no_data`; what changed is that it will
compute the moment a real measurement exists, with no further code change.

`docs/gate/fresh-baseline.md` gets the document format, the "exactly one run
pays" rule, and two open questions — the shape has no producer to validate it
against, and "first live run pays" is a choice ADR-0010 does not make.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@myselfsiddharth
myselfsiddharth requested a review from a team as a code owner September 7, 2026 05:39
@github-actions
github-actions Bot requested a review from OM152002 September 7, 2026 05:39
@github-actions github-actions Bot added size/L <= 600 changed lines documentation Improvements or additions to documentation gate PRD section 9 gate measurement area: experiments Touches experiments labels Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: experiments Touches experiments documentation Improvements or additions to documentation gate PRD section 9 gate measurement size/L <= 600 changed lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Measure the fresh-reasoning baseline cost (the §9 denominator)

1 participant