Skip to content

[Fix] Delegated work uses the wrong model when users request or confirm a model - #3439

Merged
daniel-lxs merged 10 commits into
developfrom
fix/model-confirmation-routing-082kw9yzxjcl5
Oct 9, 2026
Merged

daniel-lxs merged 10 commits into
developfrom
fix/model-confirmation-routing-082kw9yzxjcl5

Conversation

@roomote-roomote

@roomote-roomote roomote-roomote Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

​Opened on behalf of @daniel-lxs. View the task or mention @roomote-roomote for follow-up asks.

Related issue

Internal maintainer-requested model-routing repair; no linked GitHub issue.

Why this PR exists

  • A maintainer explicitly invited this PR in the linked issue or discussion
  • I am a maintainer / this is internal Roomote work

Delegated work missed genuine human model requests and confirmations. A phrase-based eligibility gate then rejected freely worded requests despite high classifier intent. This revision replaces that gate while retaining canonical provenance and durable replay boundaries.

What changed

  • Read authorization evidence from canonical human events and verified surface identities. Assistant, tool and foreign-source content cannot become human messages through pasted sender/state fields or compatibility wrappers.
  • Interpret requests semantically with every enabled named-model option. Remove the directive/affirmation/default grammar and eligible-ID veto; interpret default polarity, redirects, continuations and capability requests through the classifier.
  • Recognize an explicit named deployment-default choice before the non-default intent gate, so saved routing cannot override it.
  • Classify an assistant's final owned model-approval question separately from human requests. Unrelated questions do not promote nearby recommendations into approval. Same-turn tool identities cannot supply a model referent through adjacent assistant context; earlier-turn tools do not taint later owned proposals.
  • Preserve sanitized received human boundaries before native steer delivery, retain them through compaction and replay, and atomically preserve the latest server-owned snapshot during prompt upserts.
  • Keep intent 0.5, named-model 0.6, routing 0.8 and native per-call timeout 5000ms unchanged. Agent claims are excluded from classifier evidence; only an authorized capability choice may use a hint to resolve identity. Saved administrator routing remains independent.

Both task launches and PR reviews use this resolver. Confirmation paths add one semantic proposal call; their maximum combined decision wait can therefore reach ten seconds. This is a classifier-backed interpretation boundary, not a universal authorization guarantee.

How it was tested

Exact pushed head: ee718f27766b9a69ac1f3b6b5f062b0c928df707.

  • Full cloud-agents suite: 2571 passed, one existing skip, 159 files. Real database upserts, canonical provenance, original/replayed instruction boundaries, tool-heavy compaction, no-claim selection and proposal-failure fallback are covered. Removed deterministic grammar assertions were replaced with provenance contracts and real semantic evaluations. The named-default regression fails on the preceding head and passes with the correction.
  • Pre-push monorepo lint, fast types and knip passed; formatting completed without bypassing hooks.
  • New natural-phrasing set: 36 freely worded cases, frozen before scoring, expanded across small/23-model catalogs and claim present/absent to 144 cases. Unchanged two-pass comparison: develop 688ff092 132/144 twice, prior 22346b2c 144/144 twice, current 144/144 twice. Zero current failures or paired regressions in this set. One automatically redacted phrase from case authoring was preserved unchanged.
  • Original thirteen supplied safety regressions, separate from that set: 52/52 twice across both catalogs and claim conditions.
  • Older 464-case template set, also retained unchanged: fresh develop 341/464 twice, prior 410/464 twice, current 460/464 twice. Every safety group passes. Four earlier-choice continuation cases fall below the unchanged intent threshold in each pass: eight failed outcomes, six worse than at least one comparison pin. Exact inputs, scores, all paired comparisons and every intermediate failure are retained in the task evidence; no case was removed or relabeled.
  • Native OpenRouter decision requests use typesafe/jev-1.13, observed typesafe/jev-1.13-20260917, with every raw request/response and both stages preserved. No final transport errors or primary-result replacement retries.
  • Main final-head CI checks passed, including Test, Lint, Type Check, Knip, app Docker build and all CodeQL analyses. Automated re-review is clean on this exact head, with all nine historical findings resolved. The unaffected worker-image job is intentionally skipped.
  • Judgment coverage caveat: the full local attempt and both remote wrapper runs report incomplete evidence/inference coverage, despite successful wrapper checks. The bounded final correction's commit judgment passed six judgments. Full final-head remote Judgment coverage is incomplete; it is not counted as a complete semantic check.

Browser proof is not applicable to this server-side change. No production rollout was performed. The PR remains draft.

Checklist

  • The PR title follows the repo convention: [Fix], [Feat], [Improve], [Refactor], [Docs], or [Chore] followed by a user-facing description
  • This PR is small and scoped to one change
  • pnpm lint and pnpm check-types pass locally
  • I added tests or included a clear manual validation note above
  • I removed secrets, tokens, private keys, and customer data from code, logs, and screenshots
  • If this change should appear in the changelog, I ran pnpm changeset

This is a larger single-boundary repair. Normal fast static gates passed; formatting-inclusive legacy commands and an interactive changeset were not run. Residual classifier misses and incomplete Judgment coverage remain explicit review caveats.

@roomote-community

roomote-community Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

No new code issues found. All nine prior findings remain addressed. See task

Reviewed the latest intent-classifier wording change in context with chronological human evidence, scoped proposal acceptance, capability selection, named-default precedence and both launch/review callers. It retains earlier human model references while explicitly including capability, cost and speed requests. Confidence thresholds, provenance boundaries, replay persistence and routing fallback are unchanged.

Current-head Test, Lint, Type Check and Knip passed. App Docker build and JavaScript/TypeScript analysis were still running when checked. Repository validation suites were left to CI; this review did not independently rerun live-classifier evaluations. The PR description's semantic-evaluation results remain pinned to ee718f2, not this wording update; its incomplete Judgment-coverage caveat remains applicable.

Reviewed 2b55567

Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts Outdated
@roomote-roomote

roomote-roomote Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Fixed both accepted findings in 22346b2c: replay consent is recovered at the original persisted human boundary, and the catalog test includes the capability choice. Both threads are resolved; automated re-review reports no new issues.

The original catalog failure was introduced by this PR: 1 failed/4 passed on 9ee75c3 versus 5/5 on develop 688ff09 with the same dependencies/environment. Replay regressions fail against the original PR service source and pass with the boundary fix. Full package validation: 2523 passed, one existing skip.

Remote results on final head 22346b2: all 20 executed checks succeeded, including Test and Roomote review. No pending/failing checks. One unaffected worker-image build is intentionally skipped, not an executed pass.

Fresh unchanged frozen real-classifier sets: main 132/136, supplement 46/46; one separately preserved transport-timeout recovery passed 1/1. Three below-threshold rule-only misses remain documented; thresholds were not changed. Original and new raw evidence are preserved in the originating task.

The PR remains draft. No merge or auto-merge action.

@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review October 8, 2026 16:21
@roomote-roomote
roomote-roomote Bot marked this pull request as draft October 8, 2026 16:24
@roomote-roomote

roomote-roomote Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor Author

Pushed the semantic request repair and named-default correction in ee718f27766b9a69ac1f3b6b5f062b0c928df707. Canonical human provenance defines authority; the classifier interprets freely worded requests over all enabled model options, with no phrase-based eligible-ID veto. Owned proposal interpretation is separate, and replay/steer snapshots retain their durable boundaries. A confident named deployment-default choice now overrides routing without requiring non-default intent. The preceding grammar-based completion claim is superseded.

Exact-head evidence: full cloud-agents suite2571passed,oneexisting skip. New144-case natural-phrasing set, frozen before scoring and rerun twice on this immutable head: must-switch72/72twice and negatives72/72twice. Develop688ff092:60/72must-switch and72/72negatives twice; prior22346b2c:72/72for both twice. Zero paired regressions in this set. Original13 safety regressions expanded across catalogs/claims:52/52twice.

The separate older464 template set remains intact: develop341/464twice,prior410/464twice,current460/464twice. Every safety group passes; four earlier-choice continuation cases miss the unchanged0.5intent gate in each pass. All8failed outcomes and6paired outcomes worse than at least one pin are retained with exact inputs/scores, along with all intermediate failures. No threshold tuning, relabeling, removed cases or overwritten primary results. No final transport errors.

Final-head Test, Lint, Type Check, Knip, app Docker build and CodeQL analyses passed. Automated re-review is clean on ee718f2, and all nine historical findings are resolved. Worker-image build intentionally skipped. Judgment caveat: successful remote wrapper jobs have incomplete evidence/inference coverage; the full local attempt was also incomplete. The bounded final-correction commit judgment passed6judgments, which does not replace full coverage.

PR metadata refreshed and draft restored after automatic promotion. No merge/auto-merge, Slack post or production rollout. Browser proof is not applicable. Full evidence remains in the originating task. Finite classifier evaluation does not establish universal authorization correctness.

Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-model-authorization.ts Outdated
Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts
Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-model-authorization.ts Outdated
Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-model-authorization.ts Outdated
Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-model-authorization.ts Outdated
Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-service.ts Outdated
@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review October 8, 2026 19:51
@roomote-roomote
roomote-roomote Bot marked this pull request as draft October 8, 2026 19:56
Comment thread packages/cloud-agents/src/server/fast-agent/fast-agent-launch-model.ts Outdated
@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review October 8, 2026 21:37
@roomote-roomote
roomote-roomote Bot marked this pull request as draft October 8, 2026 21:37
@roomote-roomote roomote-roomote Bot changed the title [Fix] Delegated work uses the wrong model when users confirm a model proposal [Fix] Delegated work uses the wrong model when users request or confirm a model Oct 8, 2026
@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review October 8, 2026 23:41
@roomote-roomote
roomote-roomote Bot marked this pull request as draft October 8, 2026 23:47
@roomote-roomote
roomote-roomote Bot marked this pull request as ready for review October 8, 2026 23:48
@roomote-roomote
roomote-roomote Bot marked this pull request as draft October 8, 2026 23:48
@daniel-lxs
daniel-lxs marked this pull request as ready for review October 9, 2026 00:21

@daniel-lxs daniel-lxs left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated this wording-only change against develop and the previous head with live classifier comparisons covering explicit model requests, confirmation and continuation, conflicting saved routing, and negative provenance controls. No observed develop regression in the independent cases; original-choice fixtures now pass repeatedly. Model handoff was also verified through real delegated launches on the preceding head, whose launch logic is unchanged here. Confidence-based fallback limitations remain documented.

@daniel-lxs
daniel-lxs merged commit 071f2ec into develop Oct 9, 2026
29 checks passed
@daniel-lxs
daniel-lxs deleted the fix/model-confirmation-routing-082kw9yzxjcl5 branch October 9, 2026 00:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant