feat(route): measured-promotion gate + outcome-calibrated routing#64
Merged
Conversation
Implements the ROADMAP "Next" item "Advisory → gated promotions". A new src/promote.js generalizes the risk predictor's kill-criteria (predictor.evaluate) into a reusable gate: an advisory signal may become active ONLY if a candidate beats the current baseline on a held-out split under a metric + margin — the honesty register (overview §4), never an assertion. First application: outcome-calibrated routing. `forge route calibrate` fits an affine correction of the k-NN rubric's score toward a held-out labeled fixture (distinct from the exemplar bank, so it measures generalization) and promotes it only if it lowers held-out MAE past the margin. Advisory by default — recommend() keeps the rubric; calibratedComplexity() applies the calibration only when the gate blessed it (mirrors predictor.riskFor). On the shipped fixture the gate correctly REFUSES the calibration (it would raise held-out error), which is the point. - src/promote.js: promotionGate + mae (pure, zero-dep). - src/route.js: CALIBRATION_SAMPLES fixture, fitComplexityCalibration, applyCalibration, calibrateRouting, calibratedComplexity. - src/cli.js: `forge route calibrate`. - test/promote.test.js: 8 cases (gate promote/keep/cold-start/fit-failure + routing). - Docs: GUIDE route section, CHANGELOG, ROADMAP (consolidation/hazard remain as applications of the now-shipped gate). Verified: npm test (672 pass), npm run check, npm run typecheck, forge docs check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WTc41NYip9GZLu4n9a5vLs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Implements the ROADMAP "Next" item "Advisory → gated promotions". The risk predictor already had this discipline (
predictor.evaluate= kill-criteria: learned weights promote only if they beat the heuristic on held-out data). This generalizes that into a reusable primitive and applies it to the first of the three named advisory signals.src/promote.js— the measured-promotion gate.promotionGate(samples, spec)fits a candidate on the train split, scores it and the baseline on the held-out test split, and promotes the candidate only if it beats the baseline by amarginunder ametric— the honesty register (overview §4), never an assertion. Cold-start or a candidate that fails to fit falls back to the baseline (never throws).First application: outcome-calibrated routing (
forge route calibrate). Fits an affine correction of the k-NN rubric's score toward a held-out labeled fixture (CALIBRATION_SAMPLES, distinct fromEXEMPLARSso it measures generalization, not memorization) and promotes it only if it lowers held-out MAE past the margin. Advisory by default —recommend()keeps the rubric;calibratedComplexity()applies the calibration only when the gate blessed it (mirrorspredictor.riskFor).On the shipped fixture the gate correctly refuses the calibration (it would raise held-out error) — which is exactly the point:
src/promote.js—promotionGate,mae(pure, zero-dep).src/route.js—CALIBRATION_SAMPLES,fitComplexityCalibration,applyCalibration,calibrateRouting,calibratedComplexity.src/cli.js—forge route calibrate.test/promote.test.js— 8 cases.CHANGELOG.md, ROADMAP (consolidation ʿilm→fahm and M6 hazard remain as further applications of the now-shipped gate).Checklist
npm testpasses — 674 tests, 672 pass, 0 fail (2 pre-existing skips); 8 new casesnpm run checkpasses (Biome) — exit 0feat:)CHANGELOG.mdupdated under## [Unreleased]forge docs checkgreen)Risk & rollback
recommend()and the rubric are unchanged (the calibration is advisory and opt-in viacalibratedComplexity). Fully unit-tested, no state/schema/config changes.Extra checks
npm run typecheckpassespredictor.js; the held-out fixture is deliberately distinct from the k-NN bank, and the measured outcome (refusal) is shown as-is🤖 Generated with Claude Code
Generated by Claude Code