Repository navigation
Conversation
Move NodeRoom's default model ids off retired or deprecated ones, per model-defaults SPEC Part A section A.2.9: - one DEFAULT_AGENT_MODEL (gemini-3.8-flash) in convex/modelDefaults.ts replaces the gemini-3.5-flash literals (Google deprecated it 2026-10-08); env chains unchanged - top_paid default anthropic/claude-sonnet-4 -> anthropic/claude-sonnet-5.5; Haiku defaults -> claude-haiku-5-5; pi-ai default gemini-2.5-flash -> DEFAULT_AGENT_MODEL - TTS gemini-3.1-flash-tts-preview -> gemini-3.8-flash-tts (shutdown 2026-11-17) - catalog: 5.5 and gemini pricing rows, tier allowlists, fallback chains pruned; the dotted claude-id regex becomes an explicit table (claude-opus-4.7 now maps to claude-opus-4-7) - Claude 5.5 request shape: forced tool_choice downgraded to auto for Sonnet/Opus 5.5, explicit effort on the AI-SDK path, 2048 max_tokens floor, @ai-sdk/anthropic >= 3.0.120 - judge scripts pinned to gemini-3.6-flash with temperature removed - ladder: --real needs a dollar cap, per-dispatch cost reservation, receipt carries the tested commit, budget and dispatch policy Merge is gated on the L1-L4 ladder receipt (not run here: paid). The adaptiveRouter fast lane stays on gemini-3.5-flash until that receipt exists. Spec: model-defaults SPEC Part A §A.2 (NodeRoom) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Scaffold Handoff — For Your Coding Agent
Immutability CheckMode: advisory ✅ No immutable files were modified in this branch. Changed Files
Needs Adversarial Review — Do NOT Apply YetThese proposals passed the reject check but have not been approved by
Safety Boundary
Immutable files (never modify):
Scaffold files (safe to modify):
|
HomenShum
added
the
needs-live-eval
HomenShum
marked this pull request as draft
October 14, 2026 20:09
- merge gate: PR is a draft with the needs-live-eval label; the ladder receipt, the gemini-3.8-flash-tts listen check and the top_paid Environment smoke are listed as explicit blockers (PR body, CHANGELOG lane) - ladder --json-out is now reviewable under A.0.3: command, candidateIds, echoedModels, per-dispatch rows (provider-echoed model from the raw body, usage, cost, fallbackUsed), environment, budget.requestCeiling - a refused reservation or unpriced id is terminal for the meter: no fallback model, retry or candidate dispatches after it; the provider-neutral Google fallback honours crossModelFallback: false - claude-haiku-5-5 is priced tier-aware (>100K prompt tokens: $0.50/$2.50) in every cost and reservation path; claude-opus-4-7 gets pricingAsOf and an agreement row - Sonnet/Opus 5.5 send strict tool definitions where a forced tool_choice becomes auto; capture reasoner and provider parser send the explicit effort - .env.example copies of the changed defaults now match the code (and a test pins them) - request-shape tests for the TTS route, private stream route, provider parser default, judge scripts; ladder budget tests (b) and (e) follow the specified scenarios Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- long-context price tier is applied per request only (priceRequest / convexPriceRequest in reserveDispatch, meter settlement, runtime priceStep, calculateRequestCost, exactConvexUsageCost); summed usage is priced at base-tier rates (priceAggregateTokens) and the ladder sums settled dispatch costs; haiku-5-5 long-context tier gets cachedInputPer1M 0.05 - tests that fail on revert for the five DEFAULT_AGENT_MODEL call sites, every remaining judge, benchmark and smoke script through a fake transport, and both AGENT_MODEL fallbacks in qaMatrix - providerParserLive env-precedence test uses a non-default env id again - LangChain bridge passes strict: true on the forced tool_choice downgrade - probe.ts and providerParseSmoke.ts send the explicit claude effort; providerParseSmoke compares the provider to "gemini" so the Gemini smoke reaches Google - ladder pre-flight request ceiling and estimate multiply free-auto by OPENROUTER_FREE_AUTO_LIMIT - pricing-source inventory lists run.ts (OpenRouter snapshot first) and the proxy sweep Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- SPEC A.0.10: resolveModelAlias throws UnknownModelError for an id that is not priced, not declared by the alias table and not free-looking, so adapter model() and convexModel() (and the fallback lists they read) refuse it before any request; an unknown -free / :free id resolves to openrouter/free-auto. Pricing and telemetry readers use lookupModelAlias - a free primary no longer falls back to a paid model: adapter fallbackModelFor and convexModel drop the paid candidate unless allowPaidFallback is set and a reservation exists (SpendMeter / finite maxCostUsd) - ladder runRung reports the sum of its settled dispatch costs when a rung fails after charged dispatches (was hardcoded 0) - ladder request ceiling and estimate resolve the route alias before the free-auto fan-out test - probe.ts gpt-6.1-sol is asserted on the wire (URL, model, no sampling) - benchmark/run.ts pricing moved to pricingAtRun.ts; the pricing agreement test runs it and the --dry-run sweep on a dated OpenRouter /models fixture - tests that dispatched never-heard-of ids now assert the refusal; their invariants stay on the alias-declared unpriced id claude-sonnet-4.5 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Refreshes NodeRoom's stale default model ids (Google deprecated
gemini-3.5-flash2026-10-08; OpenRouter dropsgoogle/gemini-2.5-*2026-10-20;claude-sonnet-4retired 2026-06-15;gemini-3.1-flash-tts-previewshuts down 2026-11-17) and fixes the request shapes the new ids reject. Env chains and caller arguments are unchanged: if prod setsAGENT_MODEL,AGENT_ORCHESTRATOR_MODEL,AGENT_TOP_PAID_MODEL,VOICE_GEMINI_TTS_MODELetc., prod does not change - this PR moves code fallbacks only.This PR is a draft and carries the
needs-live-evallabel. Both stay until the three merge blockers below are closed. The label is a block, not a note (SPEC A.0.3). Nothing in this PR made a paid provider call.MERGE BLOCKERS (all three need a provider key; none was run here)
L1-L4 ladder receipt gates the
gemini-3.8-flashdefault (and theadaptiveRouter.ts:136fast-lane literal).tests/qaMatrix.test.ts("keeps the Convex interactive fallback on a ladder-safe route") fails on purpose until it exists:gemini-3.8-flashhas no PASS row indocs/qa/production-matrix.json. The check is not weakened.Pre-flight (re-run on this head with
fetchreplaced by a function that throws; 0 fetch calls): worst case $8.0552 for 2 routes x 4 rungs, request ceiling 144, so the cap must be at least that or the run refuses to start. Actual spend is expected to be a small fraction of it. Commit the receipt in a receipt-only commit on top of the tested commit (A.0.3 form a) with a clean tree.The
--json-outreport now carries everything A.0.3 reviews:command,candidateIds,echoedModels,rows[](one per provider request:requestedModel,resolvedModel,echoedModel,usage,costUsd,fallbackUsed,rung),environment,testedCommit(40-hex),gitDirty,constantSha256,budget(maxUsd,estimateUsd,spentUsd,requests,requestCeiling,stoppedBy) anddispatchPolicy. Reject the receipt unless:requestedRoutes == candidateIds == ["gemini-3.6-flash","gemini-3.8-flash"]; rungs are exactly L1-L4 with 8 passing result rows; every row hasresolvedModel == requestedModel,fallbackUsed == falseand anechoedModelequal to the requested id (agemini-3.8-flashrequest answered as another model is the failure this field exists to expose; 3.5 is already served as 3.6);budget.stoppedBy == null;budget.requests <= budget.requestCeiling;gitDirty == false;dispatchPolicy == {maxRetries:0, crossModelFallback:false}.echoedModelis read from the raw response body (modelVersion/model): the AI SDK'sresponse.modelIdis the requested id for Google, so it could never show a re-route.Manual listen check of
gemini-3.8-flash-tts(convex/http.ts:36, SPEC A.0.7: a frozen-class id with an announced shutdown inside 60 days moves to its vendor-named replacement, and a listen test is the gate). Shape-verified here by a fake-transport test throughPOST /voice/synthesize; the audio itself is unverified. To close it: on a deployment withGOOGLE_GENERATIVE_AI_API_KEYandVOICE_TTS_PROVIDER=gemini, POST one short line to/voice/synthesize, confirmX-Voice-TTS-Model: gemini-3.8-flash-ttsand a playable WAV in theKorevoice, and record the result on this PR. The oldgemini-3.1-flash-tts-previewshuts down 2026-11-17.Observed Environment smoke for
top_paid(convex/agentJobs.ts:2101,anthropic/claude-sonnet-5.5over OpenRouter). Whether OpenRouter rewrites a forcedtool_choice, and whether it passesstrict: truetool definitions through, is unobserved, so the request is sent astool_choice: autoplusstrict: trueand one live request has to show a 200 with a tool call (SPEC 1.3, A.2.9).Also not run:
prod:gate(network, audit, design and proof gates). CI'sverifyjob fails at thenpm auditstep on rootundici 8.10.0, the same version asorigin/main, so that failure is not from this PR (PR #256 bumps it); failing checks are not required for merge, which is why the block is the draft state plus the label.Runtime sites (44; migrate 26, freeze 16, defer 2 per spec)
convex/agent.ts:346gemini-3.5-flashconvex/agent.ts:345minimax/minimax-m3convex/agent.ts:783gemini-3.5-flashconvex/agent.ts:65z-ai/glm-4.7-flashconvex/agentJobs.ts:2103gemini-3.5-flashconvex/agentJobs.ts:2102minimax/minimax-m3convex/agentJobs.ts:2101anthropic/claude-sonnet-4AGENT_TOP_PAID_MODEL ?? AGENT_MODEL ?? constunchanged.convexModelnow sendstool_choice: autoplusstrict: truetool definitions for this id (SPEC 1.3) instead of a forced choice. Stays a draft until the observed Environment smoke (merge blocker 3). Done in code; stays a draft until an observed Environment smoke shows OpenRouter accepts the request (tool_choice downgraded to auto meanwhile).convex/agentJobRunner.ts:512gemini-3.5-flashconvex/agentJobRunner.ts:514openrouter/free-autoconvex/http.ts:175gemini-3.5-flashconvex/streaming.ts:68gemini-3.5-flashconvex/http.ts:35gemini-3.1-flash-tts-previewconvex/http.ts:36) - Google shutdown 2026-11-17, vendor-named replacement, frozen-class voice exception (A.0.7).VOICE_GEMINI_TTS_MODELprecedence unchanged. The manual listen check is merge blocker 2. A fake-transport test drivesPOST /voice/synthesizeand asserts the wire id.convex/http.ts:283gpt-4o-mini-transcribeconvex/http.ts:352gpt-4o-mini-ttsconvex/okfEmbeddingProvider.ts:43gemini-embedding-2src/landing/modelRegistry.ts:37z-ai/glm-5.2src/landing/modelRegistry.ts:48anthropic/claude-fable-5src/ui/Chat.tsx:1518moonshotai/kimi-k3src/ui/Chat.tsx:1690gemini-3.5-flashsrc/nodeagent/models/modelCatalog.ts:305moonshotai/kimi-k3src/nodeagent/models/modelCatalog.ts:232gpt-5.4src/nodeagent/models/modelCatalog.ts:272claude-haiku-4.5src/nodeagent/models/modelCatalog.ts:288gemini-3-flash-previewsrc/nodeagent/models/modelCatalog.ts:342gpt-5.4-minisrc/nodeagent/models/modelCatalog.ts:187claude-haiku-4.5src/nodeagent/models/modelCatalog.ts:612claude-sonnet-4.6claude-opus-4.7now maps toclaude-opus-4-7(it was silently rewritten toclaude-opus-4-8); theopusandclaude-opusshortcuts are pinned toclaude-opus-4-8, which is what they effectively resolved to before, so their behaviour is unchanged;claude-*-5.5dotted ids map toclaude-*-5-5; every other dotted 4.x id that can sit in a persisted row is listed.src/nodeagent/models/modelCatalog.ts:641z-ai/glm-5.2src/nodeagent/models/modelCatalog.ts:806z-ai/glm-5.2AGENT_FALLBACK_MODELSprecedence unchanged.src/nodeagent/models/adapter.ts:132claude-haiku-4-5sdkToolChoiceForModeldowngradesrequiredtoauto(withstrict: truetools) for forcedToolChoice-forbidden ids, and the ladder meter'sreserveDispatch+maxOutputTokenssit inside the threewithRetryclosures with the dispatch-policy option (A.2.9 Ladder budget).src/nodeagent/models/adapter.ts:152gemini-3.5-flashsrc/nodeagent/models/adapter.ts:216(env-provided)src/nodeagent/models/piAiAdapter.ts:9gemini-2.5-flashmodel()), so the test asserts only the id's name, provider and pricing, NOT a pi-ai catalog lookup.src/nodeagent/capture/reasoning.ts:23claude-haiku-4-5CAPTURE_REASONING_MODELprecedence unchanged; now sends the explicit SPEC 1.3 effort (claudeEffortProviderOptions).src/nodeagent/models/providerParserLive.ts:110gemini-3.5-flashsrc/nodeagent/models/providerParserLive.ts:111gpt-5.5src/nodeagent/models/providerParserLive.ts:112claude-haiku-4-5src/nodeagent/models/providerParserLive.ts:113z-ai/glm-5.2src/nodeagent/core/adaptiveRouter.ts:82minimax/minimax-m3gemini-3.5-flashuntil the ladder receipt exists (spec: moves only after the receipt) - defer, condition = receipt file present.src/lib/models/modelRouter.ts:47nebius/zai-org/GLM-5.2src/nodeagent/models/openRouterFreeModels.ts:59cohere/north-mini-code:freebackend/app/config.py:78nvidia/nemotron-3-nano-omniGET {GMI_MAAS_BASE_URL}/models+ the backend's pytest; not covered by npm gatesbtb_noderoom_agent/harbor_adapter.py:83z-ai/glm-5.2scripts/gemini-demo-media-judge.ts:21gemini-3.5-flashscripts/design-qa/rubric-panel.ts:31gemini-3.5-flashExecutable (E) and runtime-adjacent (R) sites (20; migrate 7, freeze 12, defer 1)
src/nodeagent/models/modelCatalog.ts:63claude-opus-4.7longContext, applied by every cost and reservation path), claude-sonnet-5-5 ($2/$10), claude-opus-5-5 ($4/$20), gemini-3.8-flash ($0.75/$3.75; $1.50/$7.50 from 2027-01-01), gemini-3.6-flash ($0.75/$3.75), gemini-3.5-flash-lite ($0.30/$2.50); the claude-opus-4-7 row ($5/$25) the alias table now needs carriespricingAsOf; retired 4.x rows kept for persisted rooms; all rows are in the agreement test (A.0.6).src/nodeagent/core/creditModel.ts:57z-ai/glm-5.2src/nodeagent/models/convexModel.ts:968(any claude-*)anthropicStepalready sent no temperature/thinking/tool_choice/prefill; added the 2048max_tokensfloor for 5.5 ids (thinking counts), reading blocks by type (test), andstrict: truetool definitions where a forced choice is downgraded toauto;tool_choiceforwarding at :842/:892 sendsauto(plus strict tools) for forcedToolChoice-forbidden ids.scripts/benchmark/modelEvalConfig.ts:126deepseek/deepseek-v4-flashscripts/benchmark/run.ts:63gemini-3.1-flash-litescripts/benchmark/probe.ts:13claude-opus-4-8evals/ladder.ts:670gpt-5.4-nanoevals/professionalLiveCatalog.ts:51deepseek/deepseek-v4-flashdeepseek-v4-flashretirement is a different transport - per-transport fact, B.1); research_champion by evidence; also financeModelLive.ts:983, halo-live-path-calibration.ts:22, package.json:241,245scripts/finch-official-judge.py:20gpt-5-miniscripts/finauditing-official-judge.py:26gpt-5-miniscripts/mbabench-official-sweep.py:26google/gemini-3-flash-previewscripts/judge-demo-gif.ts:24gemini-3.5-flashscripts/walkthroughs/judge-video.ts:53gemini-3.5-flashscripts/eval/providerParseSmoke.ts:21claude-haiku-4-5scripts/startup-diligence-provider-eval.ts:150gemini-3.5-flashscripts/bankertoolbench-nodeagent-full-sweep.ps1:7z-ai/glm-5.2scripts/bankertoolbench-nodeagent-clean-sequential.mjs:8gpt-4.1-miniscripts/inference-nodetrace/run.mjs:13gpt-5.4scripts/walkthroughs/voiceover.ts:60gpt-4o-mini-ttspackage.json:82gpt-5-miniClass N (24) dispositions - docs loop, test assertions, examples, fixtures
doc (5)
src/nodeagent/models/modelCatalog.ts:1header comment ("Primary NodeBench default: z-ai/glm-5.2"): freeze - glm-5.2 is untouched by this PR.README.md:1604(npx convex env set AGENT_MODEL gemini-3.5-flash): defer - an operator command that names the ladder-recorded route. It moves togemini-3.8-flashin the same commit as the ladder receipt (merge blocker 1), exactly like theadaptiveRouter.ts:136literal; moving it first would recommend a route with no PASS row.production-agent.json:108(modelMigration.pinnedModelz-ai/glm-5.2): freeze - pins the glm-5.2 default, unchanged.docs/eval/FIRST_USER_E2E.md:209(z-ai/glm-5.2 env example): freeze - recorded run instructions, glm-5.2 unchanged.docs/historical mentions (841 docs, 20 evidence, 30 .proofloop, 8 episodes files): freeze - hash-bound evidence and archived prototypes are never rewritten.eval-fixture (1)
public/eval/proofloop-prod-proxy-benchmark-matrix.json: freeze - generated benchmark artifact derived fromsrc/eval/proofloopProdProxyBenchmarkMatrix.ts; glm-5.2 unchanged.example (7)
.env.example:65NODEAGENT_FREE_AUTO_GOOGLE_MODEL: migrate ->gemini-3.8-flash(the code default)..env.example:116GEMINI_SEO_VIDEO_JUDGE_MODEL: migrate ->gemini-3.6-flash(the judge script default)..env.example:165PROVIDER_PARSER_GEMINI_MODEL: migrate ->gemini-3.8-flash,gemini-3.6-flash; the neighbouring:167PROVIDER_PARSER_ANTHROPIC_MODEL->claude-haiku-5-5,claude-sonnet-5-5and:168PROVIDER_PARSER_OPENROUTER_MODELwithout the unlistednex-agi/nex-n2-proare part of the same copy (the inventory records them under :165).:84VOICE_GEMINI_TTS_MODEL->gemini-3.8-flash-tts(the shutdown-deadline fix). Env vars beat code defaults, so a.envbuilt from the old file would have silently restored every one of these ids..env.example:66AGENT_MODEL=gpt-5.4-nano,:103AGENT_FILE_EGRESS_MODEL,:147FREE_AUTO_JOB_MODEL: freeze - values this PR does not change.src/nodeagent/core/subagentDispatcher.ts:32('minimax/minimax-m3'in a zod.describe): freeze - research lane champion.tests/modelRefreshServerRoles.test.ts, last describe) now fails if any of these.env.examplevalues drifts from what the code defaults to.test-assertion (11)
modelAliasClaude.test.ts:11(claude-opus-4.7 -> claude-opus-4-7),phaseModel.test.ts:73(the top_paid fallback fixture is nowanthropic/claude-sonnet-5.5; claude-sonnet-4 retired 2026-06-15),agentJobsRuntime.test.ts(new: default model per route policy, asserted as literals),openRouterFreeModels.test.ts(autofollows the constant),openAiTokenLimit.test.ts(gpt-6 case),providerEgressPolicy.test.ts(re-run, passes unchanged), and in this roundproviderParserLive.test.ts(env id is the non-defaultgemini-3.5-flash-lite, expecting[gemini-3.8-flash, gemini-3.5-flash-lite, gemini-3.6-flash], so env precedence is proven; a second case with a default env id proves de-duplication).providerParseSmoke.tsalso compared the provider to"google"whilegetProviderForModelreturns"gemini", so the Gemini smoke never reached Google; fixed (pre-existing on main).kimiK3Default.test.ts:22,convexModelFreeAutoMode.test.ts,chatModelPickerPolish.test.tsx:168(claude-sonnet-4-6 is still a live catalog row),openRouterConvexBenchmark.test.ts,finchOfficialJudgeCostGuard.test.ts(frozen official judge),modelEvalMatrix.test.ts(promotion tags).Hash-bound evidence files untouched.
Review round 3 (Codex round 2 defects and the two round-1 findings that stayed open)
adapter.fallbackModelForandconvexModeldrop a paid fallback for a free primary (isFreeModelId) unless the caller setsallowPaidFallback: trueand a reservation exists: adapter = aSpendMeter(meteredDispatchreserves the paid model's worst case before the request), Convex = a finitemaxCostUsdonnext()(the failover controller estimates each candidate against it and skips what does not fit). Tests: a concrete:freemodel that is down under the default policy,allowPaidFallbackwithout a meter, a meter without the flag, both together, a paid fallback the meter cannot afford,fallbackModelIds, the deploymentAGENT_FALLBACK_MODELSlist, and paid -> paid / paid -> free unchanged (tests/modelCostClass.test.ts). No production caller setsallowPaidFallback: default is deny.resolveModelAliasthrowsUnknownModelErrorfor an id that is not priced, not declared by the alias table and not free-looking; an unknown-free/:freeid resolves toopenrouter/free-auto.model(),convexModel(),judge(), the adaptive router and the fallback lists therefore refusegemini-9-flash-unknown,acme/paid-unknown-model,claude-opus-4.9before any request (fake transport: 0 fetch calls), metered or not. Readers that must tolerate historical ids (priceRun,reserveDispatch,creditModel,proofloopModelTracking, the egress policy,benchmark/run.ts, the interop normalizer, the LangChain strict check) call the newlookupModelAlias, which is the old behaviour.runRungreports the sum of its dispatch costs. Test drivesmain()with a charged answer thenECONNRESET: row cost equals the dispatch sum andbudget.spentUsd(reverting tocost: 0fails it).ladderRequestCeilingandestimateLadderUsdtest/free-auto$/onresolveModelAlias(route):free,free-auto,openrouter-auto,openrouter-free-auto,nodeagent-autonow countOPENROUTER_FREE_AUTO_LIMITper step;openrouter/freeand concrete aliases count 1.gpt-6.1-solrequest shapeExisting tests that dispatched never-heard-of ids were retargeted, not loosened:
evals/ladderBudget.test.ts(g),tests/convexModelFreeAutoMode.test.ts(two tests) andtests/modelAliasClaude.test.ts. The premise of each (an id the catalog does not know reaches a provider) is the defect, so each now asserts the refusal, and their original invariants (zero requests, meter untouched, terminal stop, conservative reservation, hard-budget block) are kept on the one kind of id the adapter can still route without a price row: an id the alias table declares (claude-sonnet-4.5->claude-sonnet-4-5). The old ids stay in comments in the diff.Spec discrepancies for a human decision (SPEC section C) - not decided in this PR
nodeagent/free-auto-> paidgemini-3.8-flash. The spec'sadapter.ts:152row sanctions it (NODEAGENT_FREE_AUTO_GOOGLE_MODELprecedence unchanged) but A.0.10 says a free lane never becomes a paid request. Left as the spec has it: it already honourscrossModelFallback: false(the ladder receipt policy) and a refused reservation. Decision: keep it as a named exception (then it needsallowPaidFallbackplus an awaited reservation ingenerateProviderNeutralAgentText), or remove the Google leg.claude-sonnet-4.5,claude-opus-4.5,claude-opus-4.6and what they resolve to). A.0.5/A.0.6 keep old strings valid for persisted rows, A.0.10 says unknown throws. They stay routable; a metered run refuses them (UnpricedModelError), an unmetered caller prices them at the guessed $1/$5. Decision: add verified price rows (no vendor price is in the spec) or drop the aliases.agentJobs.createOrReusestill stores anymodelPolicystring (Codex'sconvex-testprobe). The id is refused when the job runs (convexModel), not when it is created, because live clients persist strings such asscriptedandfast; validating at creation needs a decision on those.getProviderForModelis still a classifier (any slash id classifies as OpenRouter) because free-auto candidates discovered live (for exampleopenrouter/owl-alpha) must classify. The gate is the resolver and the two boundaries, not the classifier.:freeid now resolves toopenrouter/free-auto, as A.0.10 asks. NodeRoom has no price row for the spec's measured free picknvidia/nemotron-3-super-120b-a12b:free, so a request naming it is served by the free-auto route rather than that model.Pricing sources (A.0.6) - every source is enumerated and in
tests/modelPricingAgreement.test.tsmodelCatalog.modelPricing- the only hand-typed table. Every row this PR adds or relies on now carriespricingAsOf(2026-10-10), includingclaude-opus-4-7($5/$25, whichclaude-opus-4.7now resolves to) and is in the agreement cases.claude-haiku-5-5andanthropic/claude-haiku-5.5also carry the >100K-prompt tier ($0.50/$2.50) aslongContext.getModelPricing- accessor over (1).adapter.priceRequest,priceRequestStrict,reserveDispatch- tier-aware throughmodelCatalog.priceTokens, and ONLY for one request's prompt (the ladder meter uses the strict price and the reservation).adapter.priceRunprices usage summed over requests at the row's base tier (priceAggregateTokens): a sum of 60K + 60K prompts is not a 120K prompt.convexModel.convexPriceRequest,exactConvexUsageCost,convexAttemptCostReservationUsd- per-request, tier-aware;convexPriceRunis the base-tier aggregate.calculateRequestCost(useCachedInput)is per-request and reads the tier's own cache rate.creditModelprivatepriceRunandproofloopModelTrackingcost estimate - aggregate (base tier,priceAggregateTokens).scripts/benchmark/pricingAtRun.ts(moved out ofrun.tsin round 3, which runs the benchmark on import) -pricingAtRunForPREFERS the OpenRouter/modelssnapshot (priceFromOpenRouterInfo, dated by that run'sfetchedAt+routeSnapshotId, recorded on every row'spricingAtRun) and falls back to (1) throughcatalogPricingAtRun, then to a flaggedfallback_estimate($1/$5). It is not purely a catalog reader.scripts/proofloop-proxy-model-sweep.ts(:131/:238) - prices from the live OpenRouter catalog fetched when the sweep runs (datedfetchedAtper row); no fallback table.Chat.tsxprice chips - read (1), show the base tier.Round 3 (review defect 5): sources 6 and 7 are no longer excluded. Their per-token to per-1M conversion is code that can disagree with (1), and A.0.6 allows no exclusion.
tests/modelPricingAgreement.test.tsnow runs source 6 (pricingAtRunFor/catalogPricingAtRun) and source 7 (the sweep script itself in--dry-run, fakefetch) on a dated OpenRouter/modelsfixture (2026-10-10, literal per-token strings) and asserts{inPerM, outPerM}(and the cache-read rate) equal the catalog row for every OpenRouter-transport refresh row. Reverted to check:prompt * 1_000_000->* 100_000inpricingAtRun.ts, andparsed * 1_000_000->* 100_000in the sweep: each fails its agreement test (Codex's 10x undercharge now fails). The no-hand-typed-table check stays, now pointed atpricingAtRun.tsand assertingrun.tsimports it.The tier fix is behaviour, not just data: 120K input + the 8192-token cap now reserves $0.08048 (was $0.016096), and the pre-flight estimate prices each step by its prompt size.
Round 2 (review defect 2): the tier is chosen per REQUEST only (
reserveDispatch, meter settlement, runtimepriceStep=priceRequest/convexPriceRequest,calculateRequestCost,exactConvexUsageCost). Aggregated usage is priced at base-tier rates (priceRun,convexPriceRun,creditModel,proofloopModelTracking), and the ladder sumsmeter.dispatches[].costUsd(metered) or the runtime's per-stepusage.costUsd(scripted) instead of repricing summed usage.priceRun('claude-haiku-5-5', 120000, 2000)was $0.065 while two settled 60K/1K requests cost $0.013; a run of those two now reports $0.013.HAIKU_5_5_LONG_CONTEXTalso carries the vendor'scachedInputPer1M: 0.05, so cached tokens above 100K no longer bill at $0.50/M.Breaking-parameter fixes (spec section 1.3)
sdkToolChoiceForModel(adapter.ts),convexModel.openAiCompatibleToolChoice(OpenRouteranthropic/claude-*-5.5) and the LangChain bridge sendautoinstead of a forcedrequired/anyfor Sonnet/Opus 5.5 (and Fable 5.1); Haiku 5.5 keeps a forced choice. Keyed by exact wire id inmodelCatalog.forbidsForcedToolChoice. New this round: wherever that downgrade happens the tool definitions carrystrict: true(needsStrictTools: AI-SDK adapter, direct AnthropicanthropicStep, OpenRouteropenAiTool), which is SPEC 1.3's replacement for the forced choice. It is applied only on the downgrade, so no other path changes. Round 2: the LangChain bridge now passes{ strict: true }throughbindToolson the downgrade too (asserted in the LangChain request-shape test).modelMaxOutputTokens(id)(sharedsrc/nodeagent/models/outputTokens.ts): 2048 floor for 5.5 ids because thinking counts towardmax_tokens;anthropicStepreads content blocks by type (test).effort(haiku medium / sonnet high / opus medium) on every AI-SDK Anthropic path that moved to a 5.5 id: the adapter,capture/reasoning.ts,providerParserLive, and, new in round 2,scripts/benchmark/probe.tsandscripts/eval/providerParseSmoke.tsthrough one helper (claudeEffortProviderOptions);@ai-sdk/anthropic^3.0.84->^3.0.120.openAiTokenLimitregex also matchesgpt-6(precondition only; no gpt-6 id lands here).gemini-3.6-flashwithtemperatureremoved (deprecated on 3.6+);GEMINI_JUDGE_MODELenv chains unchanged.resolveModelAlias: the dotted-id regex rewrite is deleted for an explicit table;claude-opus-4.7now maps toclaude-opus-4-7(was silently-4-8). Theopusandclaude-opusshortcuts are pinned toclaude-opus-4-8(their previous effective target, behaviour preserved). Old 4.x strings stay valid for persisted rows.--realrefuses to start withoutLADDER_MAX_COST_USD/--max-cost-usd; job-wide meter;reserveDispatchinside everywithRetrydispatch (maxRetries: 0on the SDK, one reservation per physical request);priceRunStrictthrowsUnpricedModelErrorinstead of a guessed $1/$5;dispatch: { maxRetries, crossModelFallback }option. New this round: a refused reservation or unpriced id is terminal for the meter (the failed dispatch can no longer fall through toAGENT_FALLBACK_MODEL, a retry, or a free-auto candidate), and the provider-neutralnodeagent/free-autoGoogle fallback honourscrossModelFallback: false(it previously always went to paid Gemini).ladder:freegets--max-cost-usd=1because a cap is now mandatory. Round 2: the pre-flight request ceiling and USD estimate multiply a free-auto route byopenRouterFreeAutoLimit()(OPENROUTER_FREE_AUTO_LIMIT, default 8, max 20), the fan-out the adapter enforces, not by--free-auto-top.Test proof
New request-shape tests build a real request through the repo's own builders with a fake
fetch, asserting the literal wire id (never imported from the code under test) and that forbidden keys are absent:tests/modelRefreshRequestShape.test.ts-convexModel(direct Anthropic, OpenRouter),adapter.model,aiSdkReasoner, the LangChain bridge, pi-ai default; now alsostricttools and effort.tests/modelRefreshServerRoles.test.ts(new) - the roles that had no fake-transport test: Gemini TTS (drivesPOST /voice/synthesize), private stream (drivesPOST /stream-private-reply, nothing injected: the route's ownAGENT_MODEL ?? DEFAULT_AGENT_MODELreachesgeminiStream), provider parser (runLiveProviderParserwith no model argument, so the default candidate reaches the wire; effort and wire id for haiku and sonnet), judge scripts (rubric-panel, visual-polish, mp4-workflow-review, SEO judge and walkthroughs/judge-video are executed with fixtures and a fakefetch; round 2 also drivesjudge-demo-gif,judge-state-captures,spreadsheetbench-chart-visual-grade,gemini-demo-media-judge,gemini-ui-review,painJudge, the chart-visual probe, walkthrough-review (CLI default and shipped config),scripts/benchmark/probe.ts,providerParseSmokeand the startup-diligence provider eval through a fake transport; the ASTtemperaturechecks stay), and.env.exampleagainst the code defaults.tests/modelPricingAgreement.test.ts(A.0.6 agreement, tier-aware),tests/modelCatalogRefresh.test.ts.evals/ladderBudget.test.tsdrives the actual adapter path. Round 1 tightened (b) to the specified sequence (charged response, then ECONNRESET) and (e) to usage that stays within every reservation: the cap is the pre-flight estimate itself, the context outgrows the scripted dry run, and the test asserts the grid stops before the cap (spentUsd < maxUsd, a refused reservation,budget.requestsequals the fetch count and is below the ceiling). Also covered: terminal-stop behaviour, per-dispatch receipt rows (echoedModel,fallbackUsed) and the report fields.tests/modelRefreshCallSites.test.ts(new, round 2) - the fiveDEFAULT_AGENT_MODELcall sites with no unit-reachable transport (convex/agent.tsx2,agentJobRunner.ts,streaming.ts,Chat.tsx) are pinned at the expression that owns the default;tests/qaMatrix.test.tsnow checks EVERYAGENT_MODEL ?? ...fallback inconvex/agent.ts, not the first. Re-inlininggemini-3.5-flashat each of the five fails a named test.tests/modelAdapterProviderNeutralRoute.test.ts-crossModelFallback: falsenever reaches paid Gemini.Each new assertion was checked by reverting the line it protects (TTS constant,
DEFAULT_AGENT_MODELand each of its five call sites, a judge script'stemperature/model, the parser default and effort, strict tools, the tier, the terminal stop, the fallback guard): the matching test fails every time. Round 3 correction: that claim did not hold for thescripts/benchmark/probe.tsgpt-6.1-solrow (the test answered OpenAI with a 404 and asserted nothing about the request; Codex swapped the id and it stayed green). It is now asserted on the wire (POST https://api.openai.com/v1/responses,body.model === "gpt-6.1-sol", no sampling keys, no tools) and reverting the id togpt-6.1-solxfails it. The round-3 reversions (cost-class guards in the adapter and inconvexModel, the resolver throw, the failed-rung cost, the alias fan-out, both pricing conversions, the probe id) were each run and each fails its named test.Check output (
npm run check= typecheck + convex tsc + vitest)Round 3, head
6dabf77f:tests/dockerSandboxProbe.test.ts, which timed out under the parallel run in round 1, passed in this one.tests/qaMatrix.test.tsstill checks everyAGENT_MODEL ?? ...fallback inconvex/agent.tsagainst the ladder; its failure is the unrecordedgemini-3.8-flashladder row (merge blocker 1); nothing was weakened. The ladder was not run live and no paid provider call was made.The files typechecked for the first time because the new tests import them needed two type-only fixes (
scripts/design-qa/rubric-panel.tscontent type, an unused import in the SEO judge); both lines are unchanged fromorigin/main.Not in this PR / for the human
backend/app/config.py:78-80,llm.py:37on GMI MaaS): defer - listGET {GMI_MAAS_BASE_URL}/models, confirm the ids, run the backend pytest.src/landing/modelRegistry.ts:48(anthropic/claude-fable-5): defer until the forced-tool-choice downgrade is observed live (merge blocker 3).README.md:1604andadaptiveRouter.ts:136: move togemini-3.8-flashwith the ladder receipt.gpt-5-mini(retires 2026-12-11, URGENT decision),google/gemini-3-flash-preview, benchmark judgegemini-3.1-flash-lite(2027-05-07).undiciinpackage.json/package-lock.jsonand PR fix(evals): reject unconfigured real-mode runs #260 ownsevals/runEval.ts; this PR touches neither. The lockfile change here adds a nested@ai-sdk/anthropic/node_modules/undici6.29.0 entry; re-runnpm ciafter whichever PR merges second.Spec: model-defaults SPEC Part A section A.2 (NodeRoom)