Skip to content

feat(route): measured-promotion gate + outcome-calibrated routing#64

Merged
CodeWithJuber merged 2 commits into
masterfrom
claude/gated-promotions
Jul 15, 2026
Merged

feat(route): measured-promotion gate + outcome-calibrated routing#64
CodeWithJuber merged 2 commits into
masterfrom
claude/gated-promotions

Conversation

@CodeWithJuber

Copy link
Copy Markdown
Owner

What & why

Implements the ROADMAP "Next" item "Advisory → gated promotions". The risk predictor already had this discipline (predictor.evaluate = kill-criteria: learned weights promote only if they beat the heuristic on held-out data). This generalizes that into a reusable primitive and applies it to the first of the three named advisory signals.

src/promote.js — the measured-promotion gate. promotionGate(samples, spec) fits a candidate on the train split, scores it and the baseline on the held-out test split, and promotes the candidate only if it beats the baseline by a margin under a metric — the honesty register (overview §4), never an assertion. Cold-start or a candidate that fails to fit falls back to the baseline (never throws).

First application: outcome-calibrated routing (forge route calibrate). Fits an affine correction of the k-NN rubric's score toward a held-out labeled fixture (CALIBRATION_SAMPLES, distinct from EXEMPLARS so it measures generalization, not memorization) and promotes it only if it lowers held-out MAE past the margin. Advisory by default — recommend() keeps the rubric; calibratedComplexity() applies the calibration only when the gate blessed it (mirrors predictor.riskFor).

On the shipped fixture the gate correctly refuses the calibration (it would raise held-out error) — which is exactly the point:

$ forge route calibrate
  samples: 24 labeled task(s)
  held-out MAE: rubric 0.152 · calibrated 0.226
  → keep the rubric — baseline retained — candidate did not beat it by the margin
  advisory — routing stays on the rubric until a promoted calibration is adopted
  • New src/promote.jspromotionGate, mae (pure, zero-dep).
  • src/route.jsCALIBRATION_SAMPLES, fitComplexityCalibration, applyCalibration, calibrateRouting, calibratedComplexity.
  • src/cli.jsforge route calibrate.
  • New test/promote.test.js — 8 cases.
  • Docs — GUIDE route section, CHANGELOG.md, ROADMAP (consolidation ʿilm→fahm and M6 hazard remain as further applications of the now-shipped gate).

Checklist

  • npm test passes — 674 tests, 672 pass, 0 fail (2 pre-existing skips); 8 new cases
  • npm run check passes (Biome) — exit 0
  • New public functions have a test
  • Conventional commit message (feat:)
  • CHANGELOG.md updated under ## [Unreleased]
  • No new runtime dependency
  • Substrate/docs updated — routing/gate reconciled in GUIDE + ROADMAP (forge docs check green)

Risk & rollback

  • Risk level: low — purely additive; recommend() and the rubric are unchanged (the calibration is advisory and opt-in via calibratedComplexity). Fully unit-tested, no state/schema/config changes.
  • Rollback plan: revert the commit; nothing persisted, no behavior change to existing routing.

Extra checks

  • npm run typecheck passes
  • Input validated at boundaries; errors handled — cold-start / fit-failure fall back to baseline, never throw
  • Logs contain no secrets/PII
  • If AI-assisted: the gate is the honest kill-criteria pattern from predictor.js; the held-out fixture is deliberately distinct from the k-NN bank, and the measured outcome (refusal) is shown as-is

🤖 Generated with Claude Code


Generated by Claude Code

claude and others added 2 commits July 12, 2026 22:21
Implements the ROADMAP "Next" item "Advisory → gated promotions". A new
src/promote.js generalizes the risk predictor's kill-criteria (predictor.evaluate)
into a reusable gate: an advisory signal may become active ONLY if a candidate beats
the current baseline on a held-out split under a metric + margin — the honesty
register (overview §4), never an assertion.

First application: outcome-calibrated routing. `forge route calibrate` fits an affine
correction of the k-NN rubric's score toward a held-out labeled fixture (distinct from
the exemplar bank, so it measures generalization) and promotes it only if it lowers
held-out MAE past the margin. Advisory by default — recommend() keeps the rubric;
calibratedComplexity() applies the calibration only when the gate blessed it (mirrors
predictor.riskFor). On the shipped fixture the gate correctly REFUSES the calibration
(it would raise held-out error), which is the point.

- src/promote.js: promotionGate + mae (pure, zero-dep).
- src/route.js: CALIBRATION_SAMPLES fixture, fitComplexityCalibration,
  applyCalibration, calibrateRouting, calibratedComplexity.
- src/cli.js: `forge route calibrate`.
- test/promote.test.js: 8 cases (gate promote/keep/cold-start/fit-failure + routing).
- Docs: GUIDE route section, CHANGELOG, ROADMAP (consolidation/hazard remain as
  applications of the now-shipped gate).

Verified: npm test (672 pass), npm run check, npm run typecheck, forge docs check.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WTc41NYip9GZLu4n9a5vLs
@CodeWithJuber CodeWithJuber marked this pull request as ready for review July 15, 2026 11:13
@CodeWithJuber CodeWithJuber merged commit 91fdbd1 into master Jul 15, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants