feat(providers): remap tiers onto a custom gateway's real model IDs#65
Merged
Merged
Conversation
model_tiers.json pins public Anthropic IDs (claude-haiku-4-5-…, claude-sonnet-5, …). A self-hosted LiteLLM/proxy gateway usually serves its own model names, so a stock ID sent verbatim 404s. New src/gateway_model_map.js asks the gateway what it actually serves (GET /v1/models, once per process) and scores each advertised model against every tier's family — the family word (haiku/sonnet/opus/fable) gates the match, the setOverlap name-token coefficient picks the best id, ties break toward the id closest to the canonical name — then remaps each tier onto a real gateway model. - Zero breaking change: the MODELS export shape is untouched; this is a resolution-time layer. resolveModel (providers) and buildRunner (adjudicate) consult it only when the resolved id is a stock Anthropic ID, so an explicit .forge/providers.json alias or ANTHROPIC_MODEL override always wins. - Fail-safe: no gateway, unreachable /v1/models, or no family match returns the stock id unchanged. Direct api.anthropic.com sessions never probe and are byte-identical. - forge doctor surfaces the resolved tier→model mapping under a "gateway models" row so users can verify it and pin explicit IDs if a family scored wrong. - The key travels via the child's env, never argv (llm.js pattern); zero runtime deps. Adds 9 tests and documents the behavior in GUIDE, ARCHITECTURE, and CHANGELOG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019PXuKmJp92Gdo2FudNjSx2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
src/model_tiers.jsonpins public Anthropic IDs (claude-haiku-4-5-…,claude-sonnet-5,claude-opus-4-8,claude-fable-5). A self-hosted LiteLLM/proxy gateway almost always serves those models under its own names (bedrock-claude-haiku,prod-sonnet-5,claude-3-5-sonnet-v2), so a stock ID sent verbatim to such a gateway 404s.New
src/gateway_model_map.jsasks the gateway what it actually serves —GET /v1/models, once per process — and scores each advertised model against every tier's family:haiku/sonnet/opus/fable) is a hard gate — absent it, a model is never a candidate for that tier, so nothing unrelated is mis-assigned;setOverlapcoefficient of the tier's name tokens ({haiku,4,5}) with the id's tokens — a version match (claude-sonnet-5) scores 1.0, a bare match (prod-sonnet) scores lower;This is the repo's own DATA-is-a-table / DECISION-is-a-formula rule: the tier families are data, the pick is a graded, inspectable, testable score.
Zero breaking change. The
MODELSexport shape is untouched — this is a resolution-time layer.resolveModel(providers) andbuildRunner(adjudicate) consult it only when the resolved id is a stock Anthropic ID, so an explicit.forge/providers.jsonalias or anANTHROPIC_MODELoverride always wins. It fails safe to the stock id on no gateway / unreachable/v1/models/ no family match, and directapi.anthropic.comsessions never probe — byte-identical to before.forge doctorprints the resolvedtier→modelmapping under a new gateway models row so users can verify it and pin explicit IDs if a family scored wrong. The key travels via the child process's env (never argv — thellm.jspattern); no new runtime dependency.Checklist
npm testpasses (Node 18/20/22) — 676 tests, 0 fail locallynpm run checkpasses (Biome lint + format)test/gateway_model_map.test.js(+ doctor silence test)feat:)CHANGELOG.mdupdated under## [Unreleased]Risk & rollback
gatewayBase()is null).resolveModel/buildRunnerfall back to the stockMODELS[tier].idexactly as before.Extra checks (tick if applicable)
npm run typecheckpasses/v1/models, non-string ids all → silent fallback)forge doctor+ a simulated renaming catalog)🤖 Generated with Claude Code
https://claude.ai/code/session_019PXuKmJp92Gdo2FudNjSx2
Generated by Claude Code