Fix requested-model context windows and validate the 1M smoke flow (#5274)

docs/developer/diagnostics.md

@@ -255,11 +255,15 @@ inclusive. Historical numeric zero values load as `unknown` because their
 observation status cannot be recovered.
 
 `context_fraction` is the normalized cache-read proxy
-`cache_read_tokens / context_window_tokens` for that row's observed model. It
+`cache_read_tokens / context_window_tokens` for the session's requested model. It
 does not include uncached or newly cached input and is not total context
 occupancy. `context_window_tokens` is positive capacity metadata, not an
 accounting observation; a missing capacity or unobserved cache read leaves the
-fraction `null`. Claude snapshots deduplicate by
+fraction `null`. For Claude, the CLI-reported window comes from
+`modelUsage[<the session's requested model>].contextWindow`, identified by the
+init record or the sole `modelUsage` key. The turn row's `model` remains the
+served name. This does not measure the provider's maximum accepted input or
+repair upstream reporting errors. Claude snapshots deduplicate by
 `(backend, provider_used, message.id)`, preserving first-seen order and the
 first timestamp. Identical message IDs from different source pairs remain
 separate rows. Codex rows

docs/operations/observability.md

@@ -65,6 +65,13 @@ observation. `context_window_tokens` is positive capacity metadata, not a
 measure; `context_fraction` exists only with observed cache-read usage and a
 known capacity.
 
+For Claude turns, the window is the CLI-reported context budget from
+`modelUsage[<the session's requested model>].contextWindow`. The init record
+identifies that request; otherwise the sole `modelUsage` key is used. Ambiguous,
+invalid or conflicting evidence leaves the window unknown. The turn's `model`
+is still the served name. This reports the CLI's budget, without independently
+verifying the provider's maximum accepted input or correcting upstream reports.
+
 ## Mid-run accessors
 
 The orchestrator can read accumulators mid-run via the status MCP tools:

scripts/e2e/README.md

@@ -48,8 +48,9 @@ must have:
 
 - a pull-request CI workflow, so `wait_for_ci` has something to wait for;
 - a `Taskfile.yml` with `install-worktree` and `test-check` tasks;
-- a `.autoskillit/config.yaml` that sets none of `features`, `quota_guard` or
-  `agent_backend` — as the project layer it outranks the harness's user-layer config;
+- a `.autoskillit/config.yaml` that sets none of `features`, `quota_guard`,
+  `agent_backend`, `model` or `linux_tracing` — the project layer outranks the harness's
+  user-layer config, and a tracing log directory would move logs away from the gate;
 - no committed `.claude/settings.json` or `.claude/settings.local.json` with `env` provider
   or model keys — project and local settings outrank user settings.
 
@@ -69,18 +70,40 @@ The harness writes that block (mode `0600`):
 | `CLAUDE_CODE_AUTO_COMPACT_WINDOW` | `1000000` |
 | `CLAUDE_CODE_MAX_CONCURRENT_SUBAGENTS` | `6` |
 
-The values follow MiniMax's Claude Code guide
-(<https://platform.minimax.io/docs/token-plan/claude-code>) and the repository's own
-`providers.profiles.minimax`. The `[1m]` suffix makes Claude Code assume a 1M-token window
-for the unrecognized model id; the compact window is capped at it. Six concurrent subagents
+The env model values carry `[1m]` so sessions resolved through them are sized at 1M;
+they differ from [MiniMax's current Claude Code guide](https://platform.minimax.io/docs/m-plan/claude-code),
+whose manual example uses `MiniMax-M3.1-Flash-Preview[1m]` and a 524,288 compact window
+(checked 2026-10-14). Six concurrent subagents
 covers `implement` (four children) and `resolve-review` (three to six); a spawn past the
 limit fails without retry.
 
+| Setting | Sessions it governs in E2E |
+|---|---|
+| `--model claude-sonnet-5[1m]` (from `model.default: sonnet[1m]`) | Fleet food truck and `run_skill` workers, fresh and resumed; also the canary and in-session `sonnet` subagents, which inherit the parent's id. A recipe step's own `model:` replaces this default. Subagent windows are not in turn rows, so the gate does not observe them |
+| `ANTHROPIC_MODEL` | Contract-nudge resumes, which pass no `--model` and resolve to `MiniMax-M3[1m]`. Their single-envelope output records no turn row, so the gate does not observe it |
+| `ANTHROPIC_DEFAULT_HAIKU_MODEL` | A `haiku`-class step (`--model haiku`) and in-session `haiku` subagents |
+| `ANTHROPIC_DEFAULT_OPUS_MODEL` | In-session `opus` subagents. AutoSkillit's `opus` class passes `claude-opus-5-5`, not the alias |
+| `ANTHROPIC_DEFAULT_SONNET_MODEL` | Provider fallback: AutoSkillit passes no bare `sonnet`, and `sonnet` subagents inherit the parent's id |
+| `CLAUDE_CODE_AUTO_COMPACT_WINDOW` | Every session, capped at that session's model window |
+
+The live probe confirms the explicit `[1m]` declaration yields 1M on the pinned
+2.1.280/custom-endpoint path. [Claude Code 2.1.285](https://github.com/anthropics/claude-code/releases/tag/v2.1.285)
+changed custom-endpoint sessions to use Sonnet 5's native 1M window; the
+[current model-config docs](https://code.claude.com/docs/en/model-config) describe
+`[1m]` as having no effect when Sonnet already has a native 1M window. The harness
+retains the suffix while supporting the pinned path.
+
+[Claude Code 2.1.295](https://github.com/anthropics/claude-code/releases/tag/v2.1.295)
+documents retrying without a rejected context-1m beta. Do not assume that fallback
+exists at the pinned floor or that every older release had the same defect;
+attribute failures from the probe's actual responses. The floor change belongs to #5276.
+
 The harness relies on this provider contract: the Anthropic Messages API at
 `ANTHROPIC_BASE_URL`, Bearer authentication from `ANTHROPIC_AUTH_TOKEN`, tool use, and
-acceptance of the explicit `claude-*` model ids AutoSkillit passes with `--model`. The
-canary passes no `--model`, so `ANTHROPIC_MODEL` covers it; `ANTHROPIC_DEFAULT_HAIKU_MODEL`
-covers the bare `haiku` alias.
+successful calls using the exact `claude-sonnet-5[1m]` id that fleet sessions and the
+canary pass with `--model`, including any beta-header behavior of the pinned client.
+The canary passes `--model` `FLEET_MODEL` and fails unless
+`modelUsage[FLEET_MODEL].contextWindow` is 1,000,000.
 
 Every child process the harness starts gets an environment with `MINIMAX_API_KEY`,
 `E2E_SANDBOX_TOKEN`, `CLAUDE_CODE_OAUTH_TOKEN` and every `ANTHROPIC_*` variable removed, so
@@ -137,6 +160,8 @@ executes it explicitly.
 
 For model tests, `autoskillit-config.yaml` becomes `~/.autoskillit/config.yaml` in the container:
 
+- `model.default: sonnet[1m]` — puts the window declaration on the id AutoSkillit
+  passes with `--model`; `tests/infra/test_e2e_harness.py` derives `FLEET_MODEL` from it;
 - `quota_guard.enabled: false` — the guard measures Anthropic account quota through Claude
   credentials (`~/.claude/.credentials.json`) the container does not have; `fleet run` also
   gets `--disable-quota-guard`;
@@ -196,7 +221,8 @@ cleanup and redaction never depend on GitHub cancelling the job.
 ## Artifacts and redaction
 
 For model tests, the container's AutoSkillit data directory (`~/.local/share/autoskillit`) is bind-mounted
-from the runner, so session logs survive a killed container. Those files belong to uid 1000
+from the runner, so session logs survive a killed container; the harness also reads this
+log tree after fleet runs. Those files belong to uid 1000
 and are often `0600`, so a second container run copies `out/` and `data/logs/` into
 `upload/` world-readable with both secrets replaced by `[REDACTED]`, skips symlinks, and
 fails if any secret survives. Only `upload/` is uploaded, and only when redaction
@@ -233,6 +259,14 @@ one sandbox pull request numbered above the pre-run baseline exists in the catal
 `expected_pull_request_state` with a non-empty diff. `gh` reports states in upper case; the
 harness lower-cases them. Its existing cleanup closes newly opened PRs with their branches.
 
+After `fleet run`, the harness reads `sessions.jsonl` from the directory it gave the
+fleet through `AUTOSKILLIT_LOG_DIR`. Every session new in this run must have
+`turn_usage.jsonl` rows at 1,000,000, and a successful envelope must have at least
+one new session. Missing windows fail closed, and window failures survive an
+expected-failure match. This checks the CLI-reported budget: failures retain the
+session `dir_name` and observed value rather than inferring a cause or substituting
+the expected value.
+
 `headless-smoke` exercises one orchestrator and one recipe worker (`peak_sessions: 2`). The
 harness stages `sandbox-smoke.yaml` from the read-only `/opt/e2e` mount into the disposable
 sandbox clone's `.autoskillit/recipes/` directory before invoking fleet by recipe name. This

scripts/e2e/autoskillit-config.yaml

@@ -3,6 +3,9 @@
 quota_guard:
   enabled: false
 
+model:
+  default: sonnet[1m]
+
 features:
   fleet: true
   fleet_headless_run: true

scripts/e2e/e2e_harness.py

@@ -25,6 +25,9 @@
 MINIMAX_BASE_URL = "https://api.minimax.io/anthropic"
 MINIMAX_MODEL = "MiniMax-M3[1m]"
 MINIMAX_COMPACT_WINDOW_TOKENS = "1000000"
+# tests/infra/test_e2e_harness.py derives this --model value from the shipped config.
+FLEET_MODEL = "claude-sonnet-5[1m]"
+REQUIRED_CONTEXT_WINDOW_TOKENS = 1_000_000
 # implement runs four concurrent children and resolve-review three to six; a spawn past the
 # limit fails without retry.
 MAX_CONCURRENT_SUBAGENTS = "6"
@@ -169,6 +172,22 @@ def check_canary_output(stdout: str) -> list[str]:
     reply = payload.get("result")
     if not isinstance(reply, str) or CANARY_TOKEN not in reply:
         return [f"canary: reply does not contain {CANARY_TOKEN}"]
+    return _check_reported_window(payload)
+
+
+def _check_reported_window(payload: dict[str, Any]) -> list[str]:
+    model_usage = payload.get("modelUsage")
+    metadata = model_usage.get(FLEET_MODEL) if isinstance(model_usage, dict) else None
+    if not isinstance(metadata, dict):
+        return [f"canary: claude output has no modelUsage entry for {FLEET_MODEL}"]
+    window = metadata.get("contextWindow")
+    if type(window) is not int:
+        return [f"canary: {FLEET_MODEL} reported no integer context window"]
+    if window != REQUIRED_CONTEXT_WINDOW_TOKENS:
+        return [
+            f"canary: {FLEET_MODEL} reported a {window}-token context window; "
+            f"the harness requires {REQUIRED_CONTEXT_WINDOW_TOKENS}"
+        ]
     return []
 
 
@@ -178,7 +197,7 @@ def run_canary(
     workdir = Path(tempfile.mkdtemp(prefix="e2e-canary-"))
     result, failure = _call(
         runner,
-        ["claude", "-p", CANARY_PROMPT, "--output-format", "json"],
+        ["claude", "-p", CANARY_PROMPT, "--output-format", "json", "--model", FLEET_MODEL],
         what="claude -p",
         cwd=workdir,
         env=env,
@@ -307,28 +326,96 @@ def check_envelope(returncode: int, stdout: str) -> tuple[list[str], dict[str, A
     return failures, envelope
 
 
+def session_index_dir_names(log_dir: Path) -> frozenset[str]:
+    index = log_dir / "sessions.jsonl"
+    if not index.exists():
+        return frozenset()
+    names = set()
+    for line_number, line in enumerate(index.read_text().splitlines(), start=1):
+        if not line.strip():
+            continue
+        try:
+            row = json.loads(line)
+        except json.JSONDecodeError as exc:
+            raise ValueError(f"{index}:{line_number}: invalid session index JSON: {exc}") from exc
+        if isinstance(row.get("dir_name"), str):
+            names.add(row["dir_name"])
+    return frozenset(names)
+
+
+def _check_session_turn_windows(log_dir: Path, dir_name: str) -> list[str]:
+    path = log_dir / "sessions" / dir_name / "turn_usage.jsonl"
+    prefix = f"session windows: {dir_name}: "
+    if not path.exists():
+        return [prefix + "no turn usage"]
+    rows = []
+    for line_number, line in enumerate(path.read_text().splitlines(), start=1):
+        if not line.strip():
+            continue
+        try:
+            rows.append(json.loads(line))
+        except json.JSONDecodeError as exc:
+            return [prefix + f"{path}:{line_number}: invalid turn usage JSON: {exc}"]
+    if not rows:
+        return [prefix + "no turn usage"]
+    if any(type(row.get("context_window_tokens")) is not int for row in rows):
+        return [prefix + "no observed context window"]
+    windows = sorted({row["context_window_tokens"] for row in rows})
+    if windows != [REQUIRED_CONTEXT_WINDOW_TOKENS]:
+        return [prefix + f"observed {windows} != required {REQUIRED_CONTEXT_WINDOW_TOKENS}"]
+    return []
+
+
+def check_session_windows(
+    log_dir: Path, baseline: frozenset[str], *, sessions_required: bool
+) -> list[str]:
+    new_sessions = session_index_dir_names(log_dir) - baseline
+    if not new_sessions and sessions_required:
+        return [f"session windows: the fleet run recorded no sessions in {log_dir}"]
+    return [
+        failure
+        for dir_name in sorted(new_sessions)
+        for failure in _check_session_turn_windows(log_dir, dir_name)
+    ]
+
+
 def run_fleet(
     test: e2e_catalog.CatalogTest,
     out: Path,
     env: Mapping[str, str],
     runner: Runner,
     runtime_ingredients: Mapping[str, str] | None = None,
 ) -> tuple[list[str], dict[str, Any] | None]:
-    result, failure = _call(
-        runner,
-        fleet_run_argv(test, runtime_ingredients),
-        what="fleet run",
-        cwd=SANDBOX_CLONE,
-        env=env,
-        timeout=test.timeout_sec + e2e_catalog.HARNESS_GRACE_SEC,
-        evidence=out / "fleet-run",
-    )
-    if result is None:
-        return [str(failure)], None
-    (out / "fleet-run.stdout.log").write_text(result.stdout, encoding="utf-8")
-    (out / "fleet-run.stderr.log").write_text(result.stderr, encoding="utf-8")
-    (out / "envelope.json").write_text(last_line(result.stdout) + "\n", encoding="utf-8")
-    return check_envelope(result.returncode, result.stdout)
+    log_dir = Path(env["AUTOSKILLIT_LOG_DIR"])
+    baseline = session_index_dir_names(log_dir)
+    failures: list[str] = []
+    envelope: dict[str, Any] | None = None
+    try:
+        result, failure = _call(
+            runner,
+            fleet_run_argv(test, runtime_ingredients),
+            what="fleet run",
+            cwd=SANDBOX_CLONE,
+            env=env,
+            timeout=test.timeout_sec + e2e_catalog.HARNESS_GRACE_SEC,
+            evidence=out / "fleet-run",
+        )
+        if result is None:
+            failures.append(str(failure))
+        else:
+            (out / "fleet-run.stdout.log").write_text(result.stdout, encoding="utf-8")
+            (out / "fleet-run.stderr.log").write_text(result.stderr, encoding="utf-8")
+            (out / "envelope.json").write_text(last_line(result.stdout) + "\n", encoding="utf-8")
+            failures, envelope = check_envelope(result.returncode, result.stdout)
+    finally:
+        failures.extend(
+            check_session_windows(
+                log_dir,
+                baseline,
+                sessions_required=envelope is not None and envelope.get("success") is True,
+            )
+        )
+    return failures, envelope
 
 
 def match_recipe_failure(
@@ -1016,7 +1103,11 @@ def _run_test(
     if test.kind == "canary":
         return run_canary(test, out, scrubbed, runner), []
     token = env["E2E_SANDBOX_TOKEN"]
-    return run_recipe(test, catalog.sandbox_repository, token, out, scrubbed, runner)
+    recipe_env = {
+        **scrubbed,
+        "AUTOSKILLIT_LOG_DIR": str(home / ".local" / "share" / "autoskillit" / "logs"),
+    }
+    return run_recipe(test, catalog.sandbox_repository, token, out, recipe_env, runner)
 
 
 def _copy_failure_files(

scripts/e2e/recipes/sandbox-smoke.yaml

@@ -85,7 +85,8 @@ steps:
         only those two files, then create one new commit containing only
         sandbox/text.py and tests/test_smoke_canary.py with message
         `smoke: add sandbox canary`. Do not run a test command or call tools
-        that dispatch work to another worker.
+        that dispatch work to another worker. Report the commit hash and
+        changed file paths in your final response before the completion marker.
       cwd: "${{ context.worktree_path }}"
       output_dir: "${{ context.worktree_path }}"
       step_name: implement