Repository navigation
Fix requested-model context windows and validate the 1M smoke flow (#5274) - #5280
Merged
Merged
Conversation
Trecek
force-pushed
the
impl-context-window-5274-20261014
branch
from
October 15, 2026 04:53
5be500f to
a1203e4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude gateway sessions can request one model while serving another. Parent-turn context-window telemetry now uses the requested model's reported budget, preserves served model names, and leaves ambiguous metadata unknown. The E2E configuration requests sonnet[1m]; canary and fleet gates check the observed 1M budget.
This PR also repairs the live-smoke blockers encountered during the explicitly authorized CI-fix iterations. The scope expansion, original audit findings, corrections, and evidence are documented below. No permission guard or expected-failure rule was relaxed.
Audit response for #5274 / PR #5280
Implementation worktree: /home/talon/projects/worktrees/impl-context-window-5274-20261014
PR: #5280
Implementation HEAD: 493a6e9
Plan: /home/talon/projects/generic_automation_mcp/.autoskillit/temp/rectify/rectify_context_window_requested_model_5274_2026-10-13_214401_combined.md
Audit remediation source: /home/talon/projects/generic_automation_mcp/.autoskillit/temp/audit-impl/remediation_context_window_5274_2026-10-14_163623.md
Disposition
The original COVERED code, test and documentation findings remain satisfied. No production source or test changes were needed for this remediation; the PR remains at its previously green implementation SHA.
Use Approach B: retain and justify the CI-blocker fixes. The original plan did not authorize those file changes by itself. Later user instructions explicitly required fixing the failed live smoke validation and authorized unlimited iterations. The scope expansion is therefore deliberate and documented, rather than denied or silently classified as #5274 code.
The code remains at the already-tested PR SHA. Mutation experiments occur only in a detached disposable worktree and are never pushed to the PR. The experiment was restored in detached commit 0ae809ae9e577345d76bfb1f969c79de97f4112d; both changed files are byte-identical to the PR implementation, the worktree is clean, and restoration pre-commit checks passed.
Corrections to the supplied audit
Scope findings
The machine-readable manifest is /home/talon/projects/worktrees/impl-context-window-5274-20261014/.autoskillit/temp/audit-impl/deviations_5274.json. It contains one entry for each of the nine ODD identifiers, source symbols, plan sections, actual changes, falsifiable reasons, files, commits, and failure evidence.
ODD-001
Symbol: _prepare_owned_write_scope
The mandatory live smoke failed because the default artifact watch directory was converted into an allowed write prefix. The worker was not read-only. User explicitly required us to fix the red smoke CI and authorized repeated iterations. This deliberately extends the original file scope; it preserves read-only, bounded, inherited, and explicit output-dir behavior.
Change: Honor a skill's unrestricted writes when it has no explicit output directory; continue watching artifacts.
Commit(s): 9552861
Evidence: 38620637033; .autoskillit/temp/diagnose-ci/diagnosis_2026-10-14_191803.md
ODD-002
Symbol: push_to_remote / _resolve_push_url
After the worker succeeded, live run 38635748210 failed because push_to_remote ignored the explicit sandbox URL and hardcoded an absent upstream remote. Direct URL push loses tracking refs needed for cleanup; reuse of _add_or_set_upstream fixes the actual destination while preserving tracking. Authorized CI repair, not a context-window algorithm substitution.
Change: Configure the clone's upstream from the validated remote URL before push; extract existing URL lookup to stay within the complexity limit.
Commit(s): 7194c38
Evidence: 38635748210; .autoskillit/temp/resolve-failures/fix_log_smoke_scope_4.md
ODD-003
Symbol: _EXPECTED_GIT_REF_SITES
The authorized push fix shifts existing sites. Local gate on 7194c38 reported exactly two inventory failures. Updating locations maintains the existing inventory; it introduces no new exemption or acceptance-policy relaxation.
Change: Refresh three source line locations, retaining every ref classification.
Commit(s): be6f451
Evidence: .autoskillit/temp/test_check/raw_output_b3e53171.log
ODD-004
Symbol: test_allows_rendered_smoke_pull_request_command
Run 38644799697 showed successful worker and push, then a guard denial for the shell-variable redirect in PR creation. This tests the fixture correction against the real guard without weakening it. Correct attribution is be6f451; 493a6e9 changes only the cascade map.
Change: Run the rendered PR-creation fixture through git_ops_guard in a real linked worktree.
Commit(s): be6f451
Evidence: .autoskillit/temp/diagnose-ci/diagnosis_38644799697.md
ODD-005
Symbol: test_run_skill_skips_auto_fill_when_output_dir_has_template
The old assertion encoded the artifact-watch/write-permission conflation fixed by ODD-001. It correctly continues requiring the artifact watcher and verifies no implicit permission prefix. Correct attribution is e7dbad4; that commit changes only this test file.
Change: Assert artifact watching independently of unrestricted write permission.
Commit(s): e7dbad4
Evidence: .autoskillit/temp/resolve-failures/fix_log_smoke_scope_2_2026-10-14_202853.md
ODD-006
Symbol: test_unrestricted_skill_does_not_inherit_an_implicit_temp_write_limit
Regression coverage is required for ODD-001 and verifies that the narrowly changed unrestricted case does not relax the explicit or read-only boundaries. No blanket guard bypass was added.
Change: Cover unrestricted/no-output, read-only, and explicit-output behavior through the real run_skill path.
Commit(s): 9552861
Evidence: .autoskillit/temp/resolve-failures/fix_log_smoke_scope_1_2026-10-14_201210.md
ODD-007
Symbol: TestPushToRemoteMocked / explicit remote URL integration test
These exercise ODD-002's new behavior and real CI failure. The integration test proves that validation and actual push use the same destination. Existing force/protection behavior remains covered.
Change: Mock remote preparation separately; test configuration failures plus real missing and incorrect upstream remotes, destination commits, and tracking refs.
Commit(s): 7194c38
Evidence: .autoskillit/temp/resolve-failures/fix_log_smoke_scope_4.md
ODD-sandbox-smoke
Symbol: sandbox-smoke implement and create_pull_request steps
Run 38632632184 committed but returned only the marker, intentionally rejected by the existing content validator. Run 38644799697 later rejected an unresolved $body_file redirect. These minimal fixture changes satisfy existing validation and guards rather than masking failures. The fixture still has no model override.
Change: Request commit hash and changed files before the completion marker; use the resolved temp path for PR body output and --body-file.
Commit(s): 5f11ba3, be6f451
Evidence: .autoskillit/temp/resolve-failures/fix_log_smoke_scope_3_2026-10-14_205118.md; .autoskillit/temp/diagnose-ci/diagnosis_38644799697.md
ODD-test_git_ops_guard
Symbol: LAYER_CASCADE_CONSERVATIVE['recipe']
The architectural cascade guard required this entry after ODD-004 added a direct recipe import. Local run on be6f451 passed 37,501 tests and failed only that drift check. The audit's report-cascade attribution is incorrect; the entry is in recipe.
Change: Add infra/test_git_ops_guard.py to the recipe cascade for the regression's new recipe-loader import.
Commit(s): 493a6e9
Evidence: .autoskillit/temp/test_check/raw_output_375dcf9c.log
Procedural findings
Paths below are relative to /home/talon/projects/worktrees/impl-context-window-5274-20261014/.autoskillit/temp/rectify unless a different artifact is named. Audit-response artifacts are in /home/talon/projects/worktrees/impl-context-window-5274-20261014/.autoskillit/temp/audit-impl.
Default-model consumer audit
Bounded scan: /home/talon/projects/worktrees/impl-context-window-5274-20261014/.autoskillit/temp/audit-impl/default-model-consumer-scan.txt.
Config/translation check: /home/talon/projects/worktrees/impl-context-window-5274-20261014/.autoskillit/temp/audit-impl/default-model-translation.txt.
CoreRunConfig accepts the nonempty sonnet[1m] string. ClaudeCodeBackend translates it to claude-sonnet-5[1m]. Additional scan hits in coercion, launch resolution, managed-batch validation and configuration setters do not add an unsuffixed-only restriction. The default may still be superseded by the repository's existing explicit model/provider-profile precedence; that is preserved.
Live evidence
Probe deviations and decision rules
Credential values were read from the existing user MiniMax profile and Claude OAuth store and passed to the same environment variables expected by the prescribed container calls. They are never written to the evidence index or PR. The runner adds stdin=DEVNULL because the child otherwise consumes the container shell heredoc; timeout=180 bounds each call. These are execution corrections, not waived prerequisites.
On 2026-10-14, Claude Code 2.1.280 returned all five successful results. The explicit suffixed init model is the exact result key, the alias resolves to a single matching key, unsuffixed MiniMax reproduces 200K, and suffixed JSON/stream both report 1M. Both phases were allowed by the decision rules. These are CLI-reported budgets, not a measurement of provider maximum accepted input.
Re-audit handoff and limits
This is the implementer's finding-by-finding remediation and evidence index, not an independent 88-requirement audit or a replacement GO verdict. The audit-impl skill is unavailable on this Codex backend because required fixed-set join semantics cannot be realized here. A supported audit runner should use this report and deviation_manifest_path=/home/talon/projects/worktrees/impl-context-window-5274-20261014/.autoskillit/temp/audit-impl/deviations_5274.json.
The original standalone audit failed to persist; its complete assessment ledger was not included in the remediation file. Reconstructing that ledger or claiming its verdict changed would fabricate evidence. This report preserves the supplied NO GO and supplies typed artifact references for a supported follow-up. No code change was made to the actively tracked cook/order guard issue merely to publish this response.
Typed follow-up audit references (validated integer schema_version and byte_size): /home/talon/projects/worktrees/impl-context-window-5274-20261014/.autoskillit/temp/audit-impl/typed-audit-artifact-refs.json
Step 0 — Live Discriminating Probe
Probe date: 2026-10-14. All five calls exited 0 with a parseable result and is_error=False. Phase A and Phase B decision rules passed.
Machine-readable justifications for all nine ODD findings
{ "schema_version": 1, "generated_by": "audit-remediation", "plan_path": "/home/talon/projects/generic_automation_mcp/.autoskillit/temp/rectify/rectify_context_window_requested_model_5274_2026-10-13_214401_combined.md", "base_branch": "develop", "implementation_head": "493a6e99038e2ed086574ed479db38cfee487525", "authorization": { "source": "User instructions in this conversation", "instructions": [ "We can't do anything with this PR if you can't get the tests to pass.", "Yeah, of course. You can go as many iterations as you need.", "If you deviated from something or decided to skip something, that is okay. I just need your justification for why." ] }, "deviations": [ { "finding_id": "ODD-001", "original_symbol_name": "_prepare_owned_write_scope", "plan_section": "Original scope: Phase B verification item 4; no planned permission-code edit.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Honor a skill's unrestricted writes when it has no explicit output directory; continue watching artifacts.", "justification": "The mandatory live smoke failed because the default artifact watch directory was converted into an allowed write prefix. The worker was not read-only. User explicitly required us to fix the red smoke CI and authorized repeated iterations. This deliberately extends the original file scope; it preserves read-only, bounded, inherited, and explicit output-dir behavior.", "files_affected": [ "src/autoskillit/server/tools/tools_execution/_run_skill_session.py" ], "commits": [ "955286153" ], "evidence": "38620637033; .autoskillit/temp/diagnose-ci/diagnosis_2026-10-14_191803.md", "what_the_plan_said": "Original scope: Phase B verification item 4; no planned permission-code edit.", "what_i_did_instead": "Honor a skill's unrestricted writes when it has no explicit output directory; continue watching artifacts.", "why": "The mandatory live smoke failed because the default artifact watch directory was converted into an allowed write prefix. The worker was not read-only. User explicitly required us to fix the red smoke CI and authorized repeated iterations. This deliberately extends the original file scope; it preserves read-only, bounded, inherited, and explicit output-dir behavior." }, { "finding_id": "ODD-002", "original_symbol_name": "push_to_remote / _resolve_push_url", "plan_section": "Original scope: Phase B verification item 4; no planned clone-module edit.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Configure the clone's upstream from the validated remote URL before push; extract existing URL lookup to stay within the complexity limit.", "justification": "After the worker succeeded, live run 38635748210 failed because push_to_remote ignored the explicit sandbox URL and hardcoded an absent upstream remote. Direct URL push loses tracking refs needed for cleanup; reuse of _add_or_set_upstream fixes the actual destination while preserving tracking. Authorized CI repair, not a context-window algorithm substitution.", "files_affected": [ "src/autoskillit/workspace/clone/__init__.py" ], "commits": [ "7194c3859" ], "evidence": "38635748210; .autoskillit/temp/resolve-failures/fix_log_smoke_scope_4.md", "what_the_plan_said": "Original scope: Phase B verification item 4; no planned clone-module edit.", "what_i_did_instead": "Configure the clone's upstream from the validated remote URL before push; extract existing URL lookup to stay within the complexity limit.", "why": "After the worker succeeded, live run 38635748210 failed because push_to_remote ignored the explicit sandbox URL and hardcoded an absent upstream remote. Direct URL push loses tracking refs needed for cleanup; reuse of _add_or_set_upstream fixes the actual destination while preserving tracking. Authorized CI repair, not a context-window algorithm substitution." }, { "finding_id": "ODD-003", "original_symbol_name": "_EXPECTED_GIT_REF_SITES", "plan_section": "Original scope excludes clone edits and their inventory updates.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Refresh three source line locations, retaining every ref classification.", "justification": "The authorized push fix shifts existing sites. Local gate on 7194c3859 reported exactly two inventory failures. Updating locations maintains the existing inventory; it introduces no new exemption or acceptance-policy relaxation.", "files_affected": [ "tests/arch/test_git_ref_qualification.py" ], "commits": [ "be6f45119" ], "evidence": ".autoskillit/temp/test_check/raw_output_b3e53171.log", "what_the_plan_said": "Original scope excludes clone edits and their inventory updates.", "what_i_did_instead": "Refresh three source line locations, retaining every ref classification.", "why": "The authorized push fix shifts existing sites. Local gate on 7194c3859 reported exactly two inventory failures. Updating locations maintains the existing inventory; it introduces no new exemption or acceptance-policy relaxation." }, { "finding_id": "ODD-004", "original_symbol_name": "test_allows_rendered_smoke_pull_request_command", "plan_section": "Original scope: Phase B live verification; no planned guard-test edit.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Run the rendered PR-creation fixture through git_ops_guard in a real linked worktree.", "justification": "Run 38644799697 showed successful worker and push, then a guard denial for the shell-variable redirect in PR creation. This tests the fixture correction against the real guard without weakening it. Correct attribution is be6f45119; 493a6e990 changes only the cascade map.", "files_affected": [ "tests/infra/test_git_ops_guard.py" ], "commits": [ "be6f45119" ], "evidence": ".autoskillit/temp/diagnose-ci/diagnosis_38644799697.md", "what_the_plan_said": "Original scope: Phase B live verification; no planned guard-test edit.", "what_i_did_instead": "Run the rendered PR-creation fixture through git_ops_guard in a real linked worktree.", "why": "Run 38644799697 showed successful worker and push, then a guard denial for the shell-variable redirect in PR creation. This tests the fixture correction against the real guard without weakening it. Correct attribution is be6f45119; 493a6e990 changes only the cascade map." }, { "finding_id": "ODD-005", "original_symbol_name": "test_run_skill_skips_auto_fill_when_output_dir_has_template", "plan_section": "Original scope: no planned write-scope test edit.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Assert artifact watching independently of unrestricted write permission.", "justification": "The old assertion encoded the artifact-watch/write-permission conflation fixed by ODD-001. It correctly continues requiring the artifact watcher and verifies no implicit permission prefix. Correct attribution is e7dbad426; that commit changes only this test file.", "files_affected": [ "tests/server/test_tools_execution_step_resolution.py" ], "commits": [ "e7dbad426" ], "evidence": ".autoskillit/temp/resolve-failures/fix_log_smoke_scope_2_2026-10-14_202853.md", "what_the_plan_said": "Original scope: no planned write-scope test edit.", "what_i_did_instead": "Assert artifact watching independently of unrestricted write permission.", "why": "The old assertion encoded the artifact-watch/write-permission conflation fixed by ODD-001. It correctly continues requiring the artifact watcher and verifies no implicit permission prefix. Correct attribution is e7dbad426; that commit changes only this test file." }, { "finding_id": "ODD-006", "original_symbol_name": "test_unrestricted_skill_does_not_inherit_an_implicit_temp_write_limit", "plan_section": "Original scope: no planned write-scope regression test.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Cover unrestricted/no-output, read-only, and explicit-output behavior through the real run_skill path.", "justification": "Regression coverage is required for ODD-001 and verifies that the narrowly changed unrestricted case does not relax the explicit or read-only boundaries. No blanket guard bypass was added.", "files_affected": [ "tests/server/test_tools_execution_write_prefix.py" ], "commits": [ "955286153" ], "evidence": ".autoskillit/temp/resolve-failures/fix_log_smoke_scope_1_2026-10-14_201210.md", "what_the_plan_said": "Original scope: no planned write-scope regression test.", "what_i_did_instead": "Cover unrestricted/no-output, read-only, and explicit-output behavior through the real run_skill path.", "why": "Regression coverage is required for ODD-001 and verifies that the narrowly changed unrestricted case does not relax the explicit or read-only boundaries. No blanket guard bypass was added." }, { "finding_id": "ODD-007", "original_symbol_name": "TestPushToRemoteMocked / explicit remote URL integration test", "plan_section": "Original scope: no planned clone-push tests.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Mock remote preparation separately; test configuration failures plus real missing and incorrect upstream remotes, destination commits, and tracking refs.", "justification": "These exercise ODD-002's new behavior and real CI failure. The integration test proves that validation and actual push use the same destination. Existing force/protection behavior remains covered.", "files_affected": [ "tests/workspace/test_clone_push.py" ], "commits": [ "7194c3859" ], "evidence": ".autoskillit/temp/resolve-failures/fix_log_smoke_scope_4.md", "what_the_plan_said": "Original scope: no planned clone-push tests.", "what_i_did_instead": "Mock remote preparation separately; test configuration failures plus real missing and incorrect upstream remotes, destination commits, and tracking refs.", "why": "These exercise ODD-002's new behavior and real CI failure. The integration test proves that validation and actual push use the same destination. Existing force/protection behavior remains covered." }, { "finding_id": "ODD-sandbox-smoke", "original_symbol_name": "sandbox-smoke implement and create_pull_request steps", "plan_section": "Original scope: fixture carried no model override; no planned prose/PR-command edit.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Request commit hash and changed files before the completion marker; use the resolved temp path for PR body output and --body-file.", "justification": "Run 38632632184 committed but returned only the marker, intentionally rejected by the existing content validator. Run 38644799697 later rejected an unresolved $body_file redirect. These minimal fixture changes satisfy existing validation and guards rather than masking failures. The fixture still has no model override.", "files_affected": [ "scripts/e2e/recipes/sandbox-smoke.yaml", "tests/infra/test_e2e_harness.py" ], "commits": [ "5f11ba315", "be6f45119" ], "evidence": ".autoskillit/temp/resolve-failures/fix_log_smoke_scope_3_2026-10-14_205118.md; .autoskillit/temp/diagnose-ci/diagnosis_38644799697.md", "what_the_plan_said": "Original scope: fixture carried no model override; no planned prose/PR-command edit.", "what_i_did_instead": "Request commit hash and changed files before the completion marker; use the resolved temp path for PR body output and --body-file.", "why": "Run 38632632184 committed but returned only the marker, intentionally rejected by the existing content validator. Run 38644799697 later rejected an unresolved $body_file redirect. These minimal fixture changes satisfy existing validation and guards rather than masking failures. The fixture still has no model override." }, { "finding_id": "ODD-test_git_ops_guard", "original_symbol_name": "LAYER_CASCADE_CONSERVATIVE['recipe']", "plan_section": "Original scope: add the execution-to-E2E-harness cascade entry only.", "intended_change": "Implement #5274's requested-model telemetry and 1M fleet checks, then pass the prescribed live verification.", "implemented_change": "Add infra/test_git_ops_guard.py to the recipe cascade for the regression's new recipe-loader import.", "justification": "The architectural cascade guard required this entry after ODD-004 added a direct recipe import. Local run on be6f45119 passed 37,501 tests and failed only that drift check. The audit's report-cascade attribution is incorrect; the entry is in recipe.", "files_affected": [ "tests/_test_filter.py" ], "commits": [ "493a6e990" ], "evidence": ".autoskillit/temp/test_check/raw_output_375dcf9c.log", "what_the_plan_said": "Original scope: add the execution-to-E2E-harness cascade entry only.", "what_i_did_instead": "Add infra/test_git_ops_guard.py to the recipe cascade for the regression's new recipe-loader import.", "why": "The architectural cascade guard required this entry after ODD-004 added a direct recipe import. Local run on be6f45119 passed 37,501 tests and failed only that drift check. The audit's report-cascade attribution is incorrect; the entry is in recipe." } ], "clarifications": [ { "topic": "ea0448d9b", "finding": "Misclassified as unrelated smoke work", "justification": "Corrects the exception spy introduced by Phase B's unexpected-fleet-error test. The original subclass changed RuntimeError's class name and broke the test's own expected output. The fix preserves RuntimeError identity and spies on traceback recording. It is plan-required test completion, not a new smoke feature." }, { "topic": "Probe runner", "justification": "Existing MiniMax profile and Claude OAuth supplied the same credentials the prescribed environment needs; MINIMAX_API_KEY was not initially exported. stdin=DEVNULL prevents the child CLI from consuming the enclosing shell heredoc; timeout=180 bounds the process. All five calls actually ran on pinned Claude Code 2.1.280 and their original stdout/stderr are retained. No provider/model prerequisite was skipped." }, { "topic": "Evidence outside Git diff", "justification": "The plan requests artifacts beneath .autoskillit/temp; repository policy and the Reject staged AutoSkillit temp artifacts hook prohibit committing that directory. Retain originals with checksums, index their absolute paths, and publish verification summaries and manifest in the PR body. Do not force-add logs or disable the hook." } ], "generated_at": "2026-10-15T00:43:12.170907+00:00" }Verification artifact locations and SHA-256 checksums
Evidence root: /home/talon/projects/worktrees/impl-context-window-5274-20261014
Closes #5274.