feat(gate): wire the one-time program-build cost into the matrix (#39 step 4) - #193
Open
myselfsiddharth wants to merge 1 commit into
Open
feat(gate): wire the one-time program-build cost into the matrix (#39 step 4)#193myselfsiddharth wants to merge 1 commit into
myselfsiddharth wants to merge 1 commit into
Conversation
…step 4) `ReplayRunner` has accepted `costProgramBuild`/`programBuildId` since ADR-0010 and `amortizedTokensOverN()` has consumed them, but nothing in between could pass one: `gate:matrix` had `--cost-fresh` and no equivalent. So the PRD §12 amortization curve — the plot the PRD calls the demo — was unreachable from the harness *even if a number existed*. `gate:matrix --cost-program-build <path>` closes that. It reads a measured one-time payment plus the build it paid for, and attaches it to exactly ONE run in the whole matrix. Exactly one is the entire point. §12's numerator sums `cost_program_build`, so a payment repeated per run grows it linearly with N, flattens the mean, and plots nothing — the same arithmetic ADR-0010 split the two fields to prevent, reintroduced at the driver instead of in the schema. It is a latch rather than an index check because the driver has several places a run can start, and "is this the first one?" would have to be right in all of them; here it is a property of the closure. The loader refuses rather than degrades, matching `--cost-fresh`, with one rule beyond it: a zero-token build cost is rejected. A zero `cost_fresh` makes §9 report `no_data`; a zero `cost_program_build` makes §12 report a *curve*, one declining to nothing, which publishes the strongest possible form of the claim on a number nobody measured (#123 from the other direction). `matrix-run.json` records the source, the payment, the build id, and `program_build_paid` — whether a run actually took it, since a matrix that skipped every version leaves it unclaimed. **The wiring exists; the measurement does not.** Nothing in this repo writes that document: `cost_program_build` is what it cost to *produce* the compiled program, and today that is a developer typing `src/recorder/cli.ts` by hand — a developer-day and zero tokens (#127). The loader refuses every attempt to fabricate one. The §12 curve stays `no_data`; what changed is that it will compute the moment a real measurement exists, with no further code change. `docs/gate/fresh-baseline.md` gets the document format, the "exactly one run pays" rule, and two open questions — the shape has no producer to validate it against, and "first live run pays" is a choice ADR-0010 does not make. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #39 — its step 4, as ADR-0010 redefined it. Does not close #39: the live measurement is separate work that costs real money.
The gap
ReplayRunnerhas acceptedcostProgramBuild/programBuildIdsince ADR-0010, andamortizedTokensOverN()has consumed them. Nothing in between could pass one —gate:matrixhad--cost-freshand no equivalent.So the PRD §12 amortization curve, the plot the PRD calls the demo, was unreachable from the harness even if a number existed.
docs/gate/fresh-baseline.mdsaid as much in its own words: "a different measurement than this one, not yet built."#39's step 4 originally said to attach a baseline "so
repairCostVsFresh()andamortizedTokensOverN()compute". The issue's own follow-up comment records that ADR-0010 turned that into two jobs; #191/#143 landed the first. This is the second.What
gate:matrix --cost-program-build <path>, reading:{ "usable": true, "program_build_id": "<id>", "cost_program_build": { "tokens_in": 0, "tokens_out": 0, "wall_clock_ms": 0, "model_id": "<m>" } }Attached to exactly ONE run in the whole matrix. That is the entire point: §12's numerator sums
cost_program_build, so a payment repeated per run grows it linearly with N, flattens the mean, and plots nothing — the same arithmetic ADR-0010 split the two fields to prevent, reintroduced at the driver instead of in the schema.Implemented as a latch (
programBuildPaymentLatch) rather than an index check: the driver has several places a run can start (dry, live, per-version repeats) and "is this the first one?" would have to be right in all of them. As a closure it can only answer once.matrix-run.jsonrecordscost_program_build_source, the payment, the build id, andprogram_build_paid— whether a run actually took it, since a matrix that skipped every version leaves it unclaimed.Refuses rather than degrades
Same posture as
--cost-fresh(missing file, bad JSON,usable != true), plus a missingprogram_build_id— ADR-0010 requires the id alongside the payment, andReplayRunnerthrows on the pair anyway; failing at load names why.And one rule stricter than
--cost-fresh: a zero-token build cost is rejected. A zerocost_freshmakes §9 reportno_data. A zerocost_program_buildmakes §12 report a curve — one declining to nothing, which publishes the strongest possible form of the claim on a number nobody measured. That is #123's failure mode arriving from the other direction, and it is the one case here where the honest-looking default is the dangerous one.The wiring exists; the measurement does not
Kept deliberately apart, because they are different claims.
Nothing in this repo writes that document.
cost_program_buildis what it cost to produce the compiled program, and today that is a developer typingsrc/recorder/cli.tsby hand — a developer-day and zero tokens (#127). The loader refuses every attempt to fabricate one, and says so by name in the not-found error rather than reading as a typo.The §12 curve stays
no_data. What changed is that it will compute the moment a real measurement exists, with no further code change.Testing
11 new tests: 6 on the loader (happy path, absent producer, bad JSON, not-usable, missing id, zero tokens), 2 on the latch's exactly-once behaviour, and the #165 flag-table walk now covers
--cost-program-build— a flag accepted but never assigned fails there.Docs
docs/gate/fresh-baseline.mdgains the document format, the "exactly one run pays" rule and why, the zero-cost refusal and why it is stricter, and two open questions:🤖 Generated with Claude Code