Skip to content

feat(providers): remap tiers onto a custom gateway's real model IDs#65

Merged
CodeWithJuber merged 1 commit into
masterfrom
claude/forge-work-system-setup-p26ka5
Jul 15, 2026
Merged

feat(providers): remap tiers onto a custom gateway's real model IDs#65
CodeWithJuber merged 1 commit into
masterfrom
claude/forge-work-system-setup-p26ka5

Conversation

@CodeWithJuber

Copy link
Copy Markdown
Owner

What & why

src/model_tiers.json pins public Anthropic IDs (claude-haiku-4-5-…, claude-sonnet-5, claude-opus-4-8, claude-fable-5). A self-hosted LiteLLM/proxy gateway almost always serves those models under its own names (bedrock-claude-haiku, prod-sonnet-5, claude-3-5-sonnet-v2), so a stock ID sent verbatim to such a gateway 404s.

New src/gateway_model_map.js asks the gateway what it actually serves — GET /v1/models, once per process — and scores each advertised model against every tier's family:

  • the family word (haiku/sonnet/opus/fable) is a hard gate — absent it, a model is never a candidate for that tier, so nothing unrelated is mis-assigned;
  • among candidates, the pick is the setOverlap coefficient of the tier's name tokens ({haiku,4,5}) with the id's tokens — a version match (claude-sonnet-5) scores 1.0, a bare match (prod-sonnet) scores lower;
  • ties break toward the id closest to the canonical name (fewest tokens), then lexicographically for determinism.

This is the repo's own DATA-is-a-table / DECISION-is-a-formula rule: the tier families are data, the pick is a graded, inspectable, testable score.

Zero breaking change. The MODELS export shape is untouched — this is a resolution-time layer. resolveModel (providers) and buildRunner (adjudicate) consult it only when the resolved id is a stock Anthropic ID, so an explicit .forge/providers.json alias or an ANTHROPIC_MODEL override always wins. It fails safe to the stock id on no gateway / unreachable /v1/models / no family match, and direct api.anthropic.com sessions never probe — byte-identical to before. forge doctor prints the resolved tier→model mapping under a new gateway models row so users can verify it and pin explicit IDs if a family scored wrong. The key travels via the child process's env (never argv — the llm.js pattern); no new runtime dependency.

Checklist

  • npm test passes (Node 18/20/22) — 676 tests, 0 fail locally
  • npm run check passes (Biome lint + format)
  • New public functions have a test — test/gateway_model_map.test.js (+ doctor silence test)
  • Conventional commit message (feat:)
  • CHANGELOG.md updated under ## [Unreleased]
  • No new runtime dependency (dev deps ok)
  • Substrate/docs updated if this changes router/gate — GUIDE + ARCHITECTURE updated

Risk & rollback

  • Risk level: low — additive, fail-safe, and inert for direct-API users (no code path changes when gatewayBase() is null).
  • Rollback plan: revert this commit; resolveModel/buildRunner fall back to the stock MODELS[tier].id exactly as before.

Extra checks (tick if applicable)

  • npm run typecheck passes
  • Input validated at boundaries; errors handled (fetch failure, junk /v1/models, non-string ids all → silent fallback)
  • Logs contain no secrets/PII — auth key passed via child env, never argv or stdout
  • If AI-assisted: verified the resolution path end-to-end (forge doctor + a simulated renaming catalog)

🤖 Generated with Claude Code

https://claude.ai/code/session_019PXuKmJp92Gdo2FudNjSx2


Generated by Claude Code

model_tiers.json pins public Anthropic IDs (claude-haiku-4-5-…, claude-sonnet-5,
…). A self-hosted LiteLLM/proxy gateway usually serves its own model names, so a
stock ID sent verbatim 404s. New src/gateway_model_map.js asks the gateway what it
actually serves (GET /v1/models, once per process) and scores each advertised model
against every tier's family — the family word (haiku/sonnet/opus/fable) gates the
match, the setOverlap name-token coefficient picks the best id, ties break toward the
id closest to the canonical name — then remaps each tier onto a real gateway model.

- Zero breaking change: the MODELS export shape is untouched; this is a
  resolution-time layer. resolveModel (providers) and buildRunner (adjudicate)
  consult it only when the resolved id is a stock Anthropic ID, so an explicit
  .forge/providers.json alias or ANTHROPIC_MODEL override always wins.
- Fail-safe: no gateway, unreachable /v1/models, or no family match returns the stock
  id unchanged. Direct api.anthropic.com sessions never probe and are byte-identical.
- forge doctor surfaces the resolved tier→model mapping under a "gateway models" row
  so users can verify it and pin explicit IDs if a family scored wrong.
- The key travels via the child's env, never argv (llm.js pattern); zero runtime deps.

Adds 9 tests and documents the behavior in GUIDE, ARCHITECTURE, and CHANGELOG.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019PXuKmJp92Gdo2FudNjSx2
@CodeWithJuber CodeWithJuber marked this pull request as ready for review July 15, 2026 11:11
@CodeWithJuber CodeWithJuber merged commit 742a89b into master Jul 15, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants