fix(e2e): recover ownership evidence and reconcile sandbox branches

.autoskillit/.gitignore

@@ -1,4 +1,5 @@
 temp/
+.state.yaml
 .secrets.yaml
 .onboarded
 sync_manifest.json

.autoskillit/test-filter-manifest.yaml

@@ -243,6 +243,9 @@ tests/execution/backends/fixtures/codex_ndjson/*.json:
 tests/execution/fixtures/*.json:
   - execution/
 
+tests/infra/fixtures/e2e_sessions/**:
+  - infra/
+
 tests/config/golden_configs/*.yaml:
   - config/
 

.github/workflows/e2e.yml

@@ -108,7 +108,7 @@ jobs:
           TEST: ${{ matrix.test }}
         run: |
           set -euo pipefail
-          docker_args=()
+          docker_args=(--shm-size=3g)
           if [[ '${{ matrix.kind }}' != 'clean-install' ]]; then
             docker_args+=(--env MINIMAX_API_KEY --env E2E_SANDBOX_TOKEN --env E2E_IMPLEMENTATION_ISSUE_URL)
             docker_args+=(--volume "$RUNNER_TEMP/e2e/data:/home/autoskillit/.local/share/autoskillit")

docs/developer/diagnostics.md

@@ -67,6 +67,68 @@ Logs are stored in a **global** directory (not per-project), so they persist acr
 
 `summary.json` contains: `session_id`, `dir_name`, `pid`, `cwd`, `evidence_worktree_path`, `evidence_worktree_source`, `evidence_worktree_detail`, `skill_command`, `success`, `subtype`, `exit_code`, `start_ts`, `snapshot_count`, `anomaly_count`, `peak_rss_kb`, `peak_oom_score`, `peak_fd_ratio`, `session_type`, `session_errors`, `child_outcomes` (see [Child Terminal Reasons](#child-terminal-reasons)).
 
+### Rollout declarations
+
+`summary.json.rollout` and `sessions.jsonl.rollout` store the same optional
+declaration. Configure it in `.autoskillit/config.yaml`:
+
+```yaml
+rollout:
+  active:
+    change_id: "prompt-cost-study"
+    effective_at: "2026-11-01T18:00:00Z"
+    scope:
+      skills: ["make-plan"]
+      backends: ["claude-code", "codex"]
+      roles: ["planner"]
+    expected_effect: "Reduce planning input tokens while preserving requirement coverage."
+```
+
+The stored object has these same fields, with `effective_at` normalized to UTC
+ISO-8601 (`2026-11-01T18:00:00+00:00` here). An active declaration must be complete;
+`change_id` and `expected_effect` must be nonblank strings. Scope has exactly the
+`skills`, `backends`, and `roles` lists of nonblank strings; empty lists are valid.
+Scope describes the change and does not filter launches. All admitted Claude Code
+and Codex headless runs carry the active marker, including launched failures.
+Missing `rollout`, absent/null `active`, or clearing `active` means inactive.
+New inactive records store null; older rows without the field remain unassigned.
+
+Capture occurs after final launch admission, immediately before the first runner
+call. The marker and logical start remain fixed through the same-run nudge and
+terminal logging even if configuration changes. Summary `start_ts`, index
+`timestamp`, and trace enrollment `logical_run_start_ts` share that instant;
+operation timing, nudge-turn timing, and enrollment `enrolled_at` remain separate.
+Enrollment schema 3 preserves the marker/start for crash recovery. Older sidecars
+recover without a marker and retain the historical trace-mtime fallback.
+
+Declare the prediction and stable change id **before** cutover, launch marked
+baseline work, apply the change at the stated UTC boundary, and retain the same
+declaration for after runs. Clear `active` when observation ends. A revised
+prediction, scope, or cutover requires a new `change_id`; stability across
+independent runs is the operator's responsibility. A pre-cutover marked row is
+evidence that its prediction was recorded beforehand. A current config or first
+after row alone cannot prove that timing; file mtimes do not provide declaration
+history. Keeping the prediction allows comparison with observations without
+replacing it with an explanation developed after seeing outcomes.
+
+Use the stored run start and cutover as parsed timezone-aware instants, preserving
+fractional seconds. The canonical comparison is:
+
+```python
+started_at = datetime.fromisoformat(row["timestamp"])
+cutover_at = datetime.fromisoformat(rollout["effective_at"])
+if started_at < cutover_at:
+    cohort = "before"
+elif started_at >= cutover_at:
+    cohort = "after"
+```
+
+A before/after delta is a prompt to investigate, not proof that the change caused it.
+Work difficulty, repository state, and model versions can differ across cohorts.
+See the [streaming retained-index query](../operations/observability.md#rollout-cohorts)
+for validation and bounded aggregation. Never infer marker membership from hashes,
+repository revisions, mtimes, or guessed dates.
+
 ### Execution-candidate manifests
 
 Before evaluating a primary binding or configured alternative, `run_skill`
@@ -178,7 +240,7 @@ Completed sessions with per-request evidence publish `turn_usage.jsonl` before
 version-4 token descriptor carries `turn_usage_file`, `turn_usage_count`, and
 `turn_usage_schema_version`. The file reference is `null` with count zero when
 no series was observed. The JSONL row schema is version 2; the retained
-`sessions.jsonl` index uses schema version 16.
+`sessions.jsonl` index uses schema version 17.
 
 Each token-bearing row carries non-empty `backend` and `provider_used`, plus
 nullable source `message_id`, `request_id`, `timestamp`, and observed `model`.

docs/operations/observability.md

@@ -58,7 +58,7 @@ a named normalization contract. A ratio with an `unknown` required measure
 renders `unknown`; `unavailable` and `not_applicable` scopes do not contribute
 to its denominator, and an observed zero denominator renders `—`.
 
-Token descriptor schema v4, turn sidecar schema v2, session index schema v14,
+Token descriptor schema v4, turn sidecar schema v2, session index schema v17,
 fleet campaign state v13, and campaign summary v2 retain these records. Legacy
 numeric zero is decoded as `unknown` because older formats did not prove an
 observation. `context_window_tokens` is positive capacity metadata, not a
@@ -140,6 +140,106 @@ rows by `dir_name`. Archive-only history excludes the current live survivor
 window. The archive retains index metadata, not pruned session artifact
 directories, and operational consumers continue to read only `sessions.jsonl`.
 
+## Rollout cohorts
+
+Declare an evaluated change in `.autoskillit/config.yaml` before applying it:
+
+```yaml
+rollout:
+  active:
+    change_id: "prompt-cost-study"
+    effective_at: "2026-11-01T18:00:00Z"
+    scope:
+      skills: ["make-plan"]
+      backends: ["claude-code", "codex"]
+      roles: ["planner"]
+    expected_effect: "Reduce planning input tokens while preserving requirement coverage."
+```
+
+`change_id` identifies an unchanged declaration across the observation window.
+`effective_at` is the intentional UTC cutover, not a date inferred from code or
+configuration history. `scope` names affected skills, backends, and roles in a
+queryable form; its lists describe the change and never govern admission or marker
+attachment. `expected_effect` preserves the required, non-null, nonblank prediction
+with every run so readers can compare intended effects with subsequent observations.
+Empty scope lists are valid. Missing/null `active` is inactive; incomplete or
+otherwise invalid active declarations are rejected before generic string coercion.
+
+Declare the prediction and stable id before cutover, launch marked baseline work,
+apply the change at the stated UTC boundary, and keep the same declaration active
+while observing after runs. Clear `active` when the observation period ends. Use a
+new `change_id` for a revised prediction, scope, or cutover. Code validates each
+declaration and preserves each admitted run's snapshot; operators maintain stability
+across independent runs. Recording the prediction beforehand distinguishes intended
+effects from explanations developed after seeing outcomes. A pre-cutover marked
+row supplies that evidence; a current configuration or first after row alone does
+not establish preregistration.
+
+Summary and index records store the same nested `rollout` object with normalized
+UTC `effective_at`. Capture happens immediately before the first admitted launch;
+all headless runs then carry it, including failures and runs outside the descriptive
+scope. Same-run nudges and recovery keep that snapshot and logical start despite
+later configuration changes. The index `timestamp` and summary `start_ts` identify
+the admitted logical run. Operation, nudge-turn, and trace-enrollment times have
+different purposes. A subsequent independent run captures the current declaration.
+New unmarked rows contain null; older absent fields never imply cohort membership.
+
+This query counts one explicit change id while streaming the live retained index.
+Set `index_path` to `linux_tracing.log_dir / "sessions.jsonl"` if overriding the log
+root. It skips reader non-row entries and absent/null markers, and raises a visible
+error for malformed selected declarations or timestamps. It covers retained rows
+and excludes archived history; memory and printed output do not grow with row count.
+
+```python
+from datetime import datetime, timedelta
+
+from autoskillit.core import default_log_dir
+from autoskillit.execution.session_log.session_index import iter_tolerant_session_index_lines
+
+index_path = default_log_dir() / "sessions.jsonl"
+change_id = "prompt-cost-study"
+counts = {"before": 0, "after": 0}
+for end_offset, row in iter_tolerant_session_index_lines(index_path):
+    if row is None:
+        continue
+    rollout = row.get("rollout")
+    if rollout is None:
+        continue
+    if not isinstance(rollout, dict):
+        raise ValueError(f"Malformed rollout at byte {end_offset}")
+    if rollout["change_id"] != change_id:
+        continue
+    if set(rollout) != {"change_id", "effective_at", "scope", "expected_effect"}:
+        raise ValueError(f"Incomplete rollout at byte {end_offset}")
+    prediction = rollout["expected_effect"]
+    scope = rollout["scope"]
+    if not isinstance(prediction, str) or not prediction.strip():
+        raise ValueError(f"Invalid prediction at byte {end_offset}")
+    if not isinstance(scope, dict) or set(scope) != {"skills", "backends", "roles"}:
+        raise ValueError(f"Invalid scope at byte {end_offset}")
+    if any(
+        not isinstance(values, list)
+        or any(not isinstance(value, str) or not value.strip() for value in values)
+        for values in scope.values()
+    ):
+        raise ValueError(f"Invalid scope values at byte {end_offset}")
+    started_at = datetime.fromisoformat(row["timestamp"])
+    cutover_at = datetime.fromisoformat(rollout["effective_at"])
+    if started_at.utcoffset() is None or cutover_at.utcoffset() != timedelta(0):
+        raise ValueError(f"Expected aware run start and UTC cutover at byte {end_offset}")
+    if started_at < cutover_at:
+        counts["before"] += 1
+    elif started_at >= cutover_at:
+        counts["after"] += 1
+print(counts)
+```
+
+A before/after delta is a prompt to investigate, not proof that the change caused it.
+Work difficulty, repository state, and model versions can differ between cohorts.
+Compare parsed instants, including fractional seconds: equality is after cutover.
+Do not manufacture cohorts from content hashes, repository revisions, file mtimes,
+or guessed dates. Existing aggregate trends do not consume rollout markers.
+
 ## Native headless OTLP capture
 
 Every headless executor invocation starts one best-effort OTLP/HTTP-JSON sink

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "autoskillit"
-version = "0.10.1266"
+version = "0.10.1271"
 description = "MCP server for orchestrating automated skill-driven workflows with Claude Code"
 readme = "README.md"
 license = {text = "MIT"}

scripts/e2e/README.md

@@ -202,9 +202,19 @@ The implementation pipeline installs synchronous command hooks for `SessionStart
 settings. These hooks follow the [Claude lifecycle schema](https://code.claude.com/docs/en/hooks).
 Each identity spans its first start through its final end, so compaction and resume yield
 a conservative upper bound. Half-open intervals end before simultaneous starts. The
-summary reconciles the dispatched root, managed lineage and enumerable native transcripts;
-missing endpoints, identities or owned transcripts fail coverage. Transcript activity is
-never used to infer a lifetime. A peak exceeding capacity fails validation.
+summary reconciles the dispatched root, managed lineage and enumerable native transcripts.
+AutoSkillit writes `logs/sessions.jsonl` and each session's `summary.json`; their dispatch
+and parent/caller fields are AutoSkillit provenance, separate from Claude Code transcript
+and lifecycle hook fields. Empty links do not imply a Claude Code schema guarantee.
+Managed sessions are adopted through declared parent fields or receipted `run_skill`,
+`complete_run_skill_result` or `recover_run_skill_result` replies in an owned transcript.
+Each must have exactly one index row with the run's `dispatch_id`. Parentless sessions
+without receipts, missing or ambiguous index rows, and receipt-named sessions absent
+from both trace and index fail coverage. Shared dispatch IDs or overlapping lifetimes
+never adopt a session. Missing endpoints, identities or owned transcripts fail coverage;
+transcript activity never infers a lifetime. An auto-compacted root can leave an unmatched
+`SubagentStop`, which still fails coverage and is tracked separately. A peak exceeding
+capacity fails validation.
 
 Commission with a successful MiniMax CI run and repeat against the same seed. Inspect
 planning, dry-walkthrough, implementation, test, integration, audit GO and open-PR evidence,
@@ -226,9 +236,19 @@ and are often `0600`, so a second container run copies `out/` and `data/logs/` i
 fails if any secret survives. Only `upload/` is uploaded, and only when redaction
 succeeded.
 
+When a test does not pass, the harness preserves plans from each pipeline clone under
+`out/pipeline-artifacts/<clone>/<make-plan|rectify>/` and Claude Code JSONL under
+`out/transcripts/`, retaining relative paths. The transcripts include the orchestrator,
+skill sessions and their subagents; the orchestrator's transcript holds each raw
+`bind_plan_set` response and its `plan_parts` argument. These files go through the same
+redaction and upload steps as the other output artifacts. Collection also runs for an
+expected failure or a harness exception. A container killed before or during collection
+may leave no evidence or only partial evidence.
+
 Pipeline evidence includes `session-events.jsonl`, `session-concurrency.json`, compact
 `ownership.json`, the seed, effective configuration, fleet stdout/stderr/envelope, captured
-PR metadata/files, detached-head test commands and associated CI runs/jobs. Collection
+PR metadata/files, detached-head test commands, associated CI runs/jobs, and sandbox head-ref
+listings before the run and after cleanup (`head-refs-before.*`, `head-refs-after.*`). Collection
 occurs inside the container before transcript paths disappear. Incomplete evidence is
 retained and fails the scenario; redaction and upload still run after failure.
 
@@ -302,7 +322,19 @@ must be `main`. No caller `base_branch` ingredient is sent. Sandbox setup and ba
 must succeed before launch.
 
 Paired root transcript tool calls/results prove the created branch; paired root/managed
-claim results prove any added claim label. Discovery paginates through the PR baseline
+claim results prove any added claim label. The collector accepts the direct
+`mcp__autoskillit__` and plugin `mcp__plugin_autoskillit_autoskillit__` namespaces;
+a tracked tool under any other namespace fails the proof. Claude Code persisted
+transcripts carry replies in row-level `mcpMeta.structuredContent` when `mcpMeta`
+is present, otherwise in JSON block content. Malformed metadata or metadata on a
+row with several results fails the paired resource call. Hook display text never
+replaces unusable metadata. The pretty-output hook emits `updatedMCPToolOutput`;
+`_result_object` remains the shared envelope decoder, and metadata presence alone
+does not identify whether the hook ran. These shapes are pinned by
+`tests/infra/fixtures/e2e_sessions/` from Claude Code 2.1.280; re-capture them when
+an otherwise successful call reports a failed branch or claim result.
+
+Discovery paginates through the PR baseline
 and leaves unrelated PRs untouched. Require exactly one owned open, unmerged sandbox PR
 targeting `main`, without auto-merge, with the canonical seed URL and non-removed changes
 under both `sandbox/` and `tests/`. Fetch and detach at the captured head before setup and
@@ -316,6 +348,17 @@ removes only the effective claim label proven added by this run. The seed remain
 and other labels are preserved. Missing ownership proof preserves uncertain resources and
 fails cleanup. Cleanup errors remain fatal alongside the original failure.
 
+Branch reconciliation lists the sandbox's branches before the run and after cleanup.
+A branch created during the run that `ownership.json` does not explain is preserved
+and fails cleanup; baseline branches are excluded from unexplained-branch reports.
+Branches named by the ownership proof retain the existing cleanup rule. The original
+API stdout saved as `head-refs-before.stdout.txt` and `head-refs-after.stdout.txt` supplies
+the in-memory snapshots; accompanying stderr and command records diagnose failed requests.
+This check only detects unexplained branches: the creating run records the failure once,
+and the branch and any PR on it must be removed by hand. The next run includes it in
+its baseline. A manual run overlapping CI can trigger the check because CI serialization
+does not cover local runs.
+
 ## Catalog schema
 
 ```json
@@ -411,6 +454,25 @@ docker run --rm \
   python3 /opt/e2e/e2e_harness.py run --test clean-install --out /artifacts
 ```
 
+### Check a real plan against the plan-set gate before dispatching a pipeline scenario
+
+After the plan-set changes have merged, perform this check before CI commissioning:
+
+1. Check out the #5255 branch revision containing both the `implementation` E2E scenario
+   and the evidence collector. Build and run that revision with the Docker
+   invocation above, selecting `--test implementation`. Use the same revision for
+   commissioning and record the model.
+2. If the run reaches implementation, the first bind succeeded. If the test does not
+   pass, inspect the orchestrator's `bind_plan_set` results in `out/transcripts/`.
+   An observed `success: true` on the first or second bind is the pass condition.
+   A `coverage_failed` result explains the rejection in `coverage_gaps`, and
+   `out/pipeline-artifacts/` retains the rejected plan. Missing evidence is inconclusive
+   and does not satisfy this gate: require an observed successful bind. When a run
+   replans, also confirm the second bind's `plan_parts` names only the replacement plan.
+3. Record the model, plan path and bind result in issue #5259's body using
+   `gh issue edit --body-file`, never as a comment. Only an observed successful bind
+   unblocks `gh workflow run e2e.yml -f tests=implementation`.
+
 ### Sandbox smoke with teardown
 
 Set `MINIMAX_API_KEY` and `E2E_SANDBOX_TOKEN` in the local environment first. Build the same