feat(runner): delegated repair client — the host agent repairs, no API key (#189) - #192
Open
myselfsiddharth wants to merge 1 commit into
Open
feat(runner): delegated repair client — the host agent repairs, no API key (#189)#192myselfsiddharth wants to merge 1 commit into
myselfsiddharth wants to merge 1 commit into
Conversation
…I key (#189) `AnthropicRepairModelClient` makes Paragent buy its own inference. An agent calling Paragent already has model access, so requiring a second credential means paying twice for a key the user did not need — and with `npx paragent` as the advertised Quick Start, that is the difference between "works out of the box" and "works after you provision a key". `DelegatedRepairModelClient` calls no model. `propose()` hands the caller the authorized repair request and takes back a proposal; the host reasons on its own budget. A callback, not a protocol — it is the primitive an MCP tool would wrap rather than replace, so no wire format is frozen. The dangerous part is the accounting, not the ergonomics. The host pays and does not report usage back to a library, so a delegated repair records zero tokens — arithmetically identical to the stub's honest zero, and meaning the opposite. Folded into §9's `mean(cost_repair)` those zeros move the kill-line ratio *down*, which reads as "the thesis passed". A fabricated pass is worse than a fabricated fail: nobody goes looking for the bug behind good news. So (ADR-0020): - `RepairProposal.cost_measured` and `RunMetric.repair_cost_measured` mark a cost nobody observed. Absent means measured; the flag is only ever emitted false, and the runner makes it sticky for the run. - `repairCostVsFresh()` computes over measured rows only — all-excluded is `no_data`, never `0`. `amortizedTokensOverN()` returns `no_data` if any run in the window is unmeasured, because its numerator is a sum. - Outcome aggregates keep those rows: whether a repair worked is measured regardless of who paid. - `buildGateReport` publishes `cost_provenance`, so an exclusion is visible in the report instead of looking identical to a silent drop. This does **not** unblock #39. A measured denominator needs a real `usage` block, which is exactly what a delegated client cannot produce. Egress is unchanged: the handler receives `serializeRepairContext()`'s output and nothing else, and the merge-blocking canary now asserts that on the delegated path too — a host agent is a third party the same way an API is, and one a reader is likelier to think of as "inside". `sanitizeProposedAction` moved to `repair.ts` so both clients share the frozen-assertion guard and the keyless client does not load the Anthropic SDK. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #189.
Why
AnthropicRepairModelClient(#27) makes Paragent buy its own inference. That imposes a shape the thesis does not require: the pitch is giving an agent that already exists the ability to drive a browser without paying full reasoning cost every run — and such an agent already has model access. Requiring it to provisionANTHROPIC_API_KEYmeans paying twice for a credential it did not need. Withnpx paragentas the advertised Quick Start (#179), that is the difference between "works out of the box" and "works after you provision a key".What
DelegatedRepairModelClientcalls no model.propose()hands the caller the authorized repair request and takes back a proposal; the host does the reasoning on its own budget.An injected async callback, per the issue's design question 1 — it is the primitive an MCP tool would wrap rather than replace, so no wire format is frozen here.
Decline (
null), throw, timeout, and garbage all land oncorrected_action: null→REPAIR_EXHAUSTED. Never a retry: a silent retry loop is a hidden cost and an unbounded one.The part that actually needed deciding
Not the ergonomics — the accounting. The host pays and does not report usage back to a library, so a delegated repair records zero tokens. In a
Costthat is arithmetically identical to the stub's zero, which is honest (no model ran) and means the opposite.§9's kill line is
mean(cost_repair) >= 70% * mean(cost_fresh). Fold unmeasured zeros in and the ratio moves down — the direction that reads as "repair is cheap, the thesis passed". A fabricated pass is worse than a fabricated fail, because nobody goes looking for the bug behind good news. The same zeros enteramortizedTokensOverN's numerator, where a sum understates every point after the first delegated run — #123's failure mode again.ADR-0020 decides it:
RepairProposal.cost_measured/RunMetric.repair_cost_measuredmark a cost nobody observed. Absent means measured; the flag is only ever emittedfalse, andReplayRunnermakes it sticky for the run — one unmeasured proposal makes the whole run'scost_repairan undercount.repairCostVsFresh()computes over measured rows only. All rows excluded →no_data, never0.amortizedTokensOverN()returnsno_dataif any run in the window is unmeasured. Window is ordered first, filtered second, so exclusion cannot slide a later run in.buildGateReportpublishescost_provenance(runs_total/runs_cost_measured/runs_cost_unmeasured), so an honest exclusion doesn't look identical to a silent drop.This does not unblock #39. A measured denominator needs a real
usageblock, which is exactly what a delegated client cannot produce. A gate measurement run still needsAnthropicRepairModelClient.Egress
Unchanged. The handler receives
serializeRepairContext()'s output and nothing else (ADR-0012) — no second serializer.tests/canary/repair-egress.test.ts(merge-blocking) now asserts it on the delegated path too: a host agent is a third party the same way an API is, and one a reader is likelier to think of as "inside".Also
sanitizeProposedActionmoved fromrepair-anthropic.tstorepair.tsso both clients share one frozen-assertion guard and the keyless client doesn't load the Anthropic SDK to get it. Same function, same behaviour; still exported from the package index.Testing
No live model, no key, no spend — that is the point of the issue. 23 new unit tests (propose / decline / throw / timeout / late-rejection / assertion-mutation / payload), 7 aggregate tests including a guard-the-guard case proving the excluded rows would have moved the number, and 3 runner tests pinning the flag's trip from proposal to emitted row.
Known gaps (also in the ADR)
RepairEgressPayloadproposes a useful action, not just a well-formed one, is unmeasured.binand nomain/exports, so library consumers deep-import. The README says so rather than showing an import that doesn't resolve. Adding a root export freezes a public API surface and is deliberately not decided here.🤖 Generated with Claude Code