Add seeded sandbox implementation E2E validation

.autoskillit/test-filter-manifest.yaml

@@ -243,6 +243,9 @@ tests/execution/backends/fixtures/codex_ndjson/*.json:
 tests/execution/fixtures/*.json:
   - execution/
 
+tests/infra/fixtures/e2e_sessions/**:
+  - infra/
+
 tests/execution/fixtures/*.jsonl:
   - execution/
 

.github/workflows/e2e.yml

@@ -104,12 +104,13 @@ jobs:
         env:
           MINIMAX_API_KEY: ${{ matrix.kind != 'clean-install' && secrets.MINIMAX_API_KEY || '' }}
           E2E_SANDBOX_TOKEN: ${{ matrix.kind != 'clean-install' && secrets.E2E_SANDBOX_TOKEN || '' }}
+          E2E_IMPLEMENTATION_ISSUE_URL: ${{ vars.E2E_IMPLEMENTATION_ISSUE_URL }}
           TEST: ${{ matrix.test }}
         run: |
           set -euo pipefail
           docker_args=(--shm-size=3g)
           if [[ '${{ matrix.kind }}' != 'clean-install' ]]; then
-            docker_args+=(--env MINIMAX_API_KEY --env E2E_SANDBOX_TOKEN)
+            docker_args+=(--env MINIMAX_API_KEY --env E2E_SANDBOX_TOKEN --env E2E_IMPLEMENTATION_ISSUE_URL)
             docker_args+=(--volume "$RUNNER_TEMP/e2e/data:/home/autoskillit/.local/share/autoskillit")
           fi
           if [[ "$TEST" == 'headless-smoke' ]]; then

scripts/e2e/README.md

@@ -11,6 +11,7 @@ This directory holds everything the workflow runs:
 | `e2e_catalog.py` | Stdlib loader and validator for the catalog |
 | `e2e_select.py` | Picks the tests an event runs (`select`) and judges the required check (`gate`) |
 | `e2e_harness.py` | Runs one test (`run`), performs exact sandbox teardown (`cleanup`) and redacts artifacts (`redact`) |
+| `e2e_sessions.py` | Captures synchronous lifecycle events, measures concurrency and pairs ownership evidence |
 | `autoskillit-config.yaml` | AutoSkillit user-layer config the harness installs |
 | `post-e2e-failure.sh` | Appends a failure to the test's tracking issue |
 
@@ -40,6 +41,11 @@ model tests:
    because MiniMax serves the Anthropic Messages API.
 3. Create the `e2e` label in this repository.
 4. In branch protection, mark only `e2e-gate` as required — never `select` or `run (…)`.
+5. Before publishing a branch with the selectable `implementation` entry, obtain a passing
+   #5232 sandbox smoke CI run. A maintainer must seed a fixed OPEN sandbox issue describing
+   a small feature and its test, then set repository variable `E2E_IMPLEMENTATION_ISSUE_URL`
+   to its canonical `https://github.com/OWNER/REPO/issues/NUMBER` URL. Reuse that issue on
+   every run. Missing configuration fails preflight; it is not an expected product defect.
 
 ## Sandbox repository requirements
 
@@ -192,7 +198,11 @@ matrix is always parseable (`{"include":[]}` when nothing is selected).
   bypasses sampling because a labelled pull request whose sample fell out would otherwise
   never run for that commit.
 - **Picking:** tests whose `trigger_paths` match a changed path, up to `MAX_TESTS` (2); when
-  none match, exactly one test. Ties are ordered by the SHA-256 hex digest of
+  none match, one eligible ordinary test. A pipeline requires actual size at or above a
+  threshold and a matching trigger path, even with the `e2e` label. This filter also applies
+  to fallback; a pipeline-only ineligible pool selects nothing. At most one pipeline is
+  selected within the total cap. Named workflow dispatch bypasses PR eligibility.
+  Ties are ordered by the SHA-256 hex digest of
   `"{seed}:{name}"`, so the decision is a pure function of the payload and the changed paths.
   Patterns use `fnmatch.fnmatchcase`, where `*` also matches `/` (`scripts/e2e/*` matches
   every file below `scripts/e2e/`).
@@ -208,11 +218,52 @@ succeeded or nothing was selected and `run` was skipped. Every other combination
 The `run` job uses one repository-wide concurrency group (`group: e2e`, `queue: max`,
 `cancel-in-progress: false`) and `max-parallel: 1`, so every selected test of every run
 waits its turn and never cancels another. `select` and `e2e-gate` are not queued. A
-catalog entry's `peak_sessions` is capped at `MAX_PEAK_SESSIONS` (8): one step's fan-out of
-up to six subagents plus the orchestrator and step session. With one test running at a
-time, CI stays at or below eight of the roughly twelve concurrent MiniMax sessions.
+catalog entry's `peak_sessions` is capped at `MAX_PEAK_SESSIONS` (8). For an uncommissioned
+pipeline this declares capacity, not a measured fan-out. The child limit stays six.
 `clean-install` requires exactly zero `peak_sessions`.
 
+The implementation pipeline installs synchronous command hooks for `SessionStart`,
+`SessionEnd`, `SubagentStart` and `SubagentStop`, preserving existing hooks and MiniMax
+settings. These hooks follow the [Claude lifecycle schema](https://code.claude.com/docs/en/hooks).
+The agent hooks use `.+` to require a non-empty agent type: under the default main
+agent, Claude Code also fires `SubagentStop` for internal agents with an empty type.
+An omitted or empty matcher admits those callbacks; `.+` accepts every non-empty
+built-in, custom or namespaced plugin agent name without an allowlist or product-registry
+imports in these stdlib-only scripts. Session hooks have no matcher.
+Sessions must run with the default main agent. The harness fails preflight if the
+user-level settings file has an `agent` key; the fleet command builders pass neither
+`--agent` nor `--settings`, and the bundled plugin sets no main agent. Project/local
+settings in the sandbox, managed settings and other enabled plugins' main-agent defaults
+are not checked. A main agent from those sources makes internal stops carry its name:
+they fail coverage as a named stop without a start, without the blank-type `capture:` code.
+Claude Code 2.1.275 fixed specific matchers admitting blank-type stops; the image's
+2.1.280 pin includes that fix.
+Each identity spans its first start through its final end, so compaction and resume yield
+a conservative upper bound. Half-open intervals end before simultaneous starts. The
+summary reconciles the dispatched root, managed lineage and enumerable native transcripts.
+AutoSkillit writes `logs/sessions.jsonl` and each session's `summary.json`; their dispatch
+and parent/caller fields are AutoSkillit provenance, separate from Claude Code transcript
+and lifecycle hook fields. Empty links do not imply a Claude Code schema guarantee.
+Managed sessions are adopted through declared parent fields or receipted `run_skill`,
+`complete_run_skill_result` or `recover_run_skill_result` replies in an owned transcript.
+Each must have exactly one index row with the run's `dispatch_id`. Parentless sessions
+without receipts, missing or ambiguous index rows, and receipt-named sessions absent
+from both trace and index fail coverage. Shared dispatch IDs or overlapping lifetimes
+never adopt a session. Missing endpoints, identities or owned transcripts fail coverage;
+transcript activity never infers a lifetime. The captured pre-fix trace contains a blank-type
+stop immediately before a compact-source session start; its producing feature is unproven.
+An owned agent record without a named type fails coverage with a `capture:` violation.
+A named stop without its start or transcript still fails coverage, as do traces captured
+before the matcher was installed. The collector retains delivered callbacks and adds
+violations; it does not filter them out. Individual codes are in `out/session-concurrency.json`
+under `violations`. A peak exceeding capacity fails validation.
+
+Commission with a successful MiniMax CI run and repeat against the same seed. Inspect
+planning, dry-walkthrough, implementation, test, integration, audit GO and open-PR evidence,
+plus cleanup and the sandbox CI association. Replace provisional `peak_sessions` with
+the observed conservative peak and set `peak_sessions_evidence` to the CI run URL, tested
+commit and first-start/final-end method. Expected failures do not satisfy commissioning.
+
 Every harness subprocess has a timeout that fires before the job's: `fleet run` gets the
 test's `timeout_sec` plus `HARNESS_GRACE_SEC`, and the job gets that budget in minutes plus
 `JOB_OVERHEAD_MINUTES` for building the image, redaction and upload. So `result.json`, PR
@@ -237,6 +288,13 @@ redaction and upload steps as the other output artifacts. Collection also runs f
 expected failure or a harness exception. A container killed before or during collection
 may leave no evidence or only partial evidence.
 
+Pipeline evidence includes `session-events.jsonl`, `session-concurrency.json`, compact
+`ownership.json`, the seed, effective configuration, fleet stdout/stderr/envelope, captured
+PR metadata/files, detached-head test commands, associated CI runs/jobs, and sandbox head-ref
+listings before the run and after cleanup (`head-refs-before.*`, `head-refs-after.*`). Collection
+occurs inside the container before transcript paths disappear. Incomplete evidence is
+retained and fails the scenario; redaction and upload still run after failure.
+
 `clean-install` mounts no home directories and retains evidence under `out/`. Its redactor
 receives the same required `--secret-env` names with variables unset and skips the absent
 home-log tree. Redaction and upload still run after test failures.
@@ -301,6 +359,57 @@ row and close the bug. Fleet translates the L3 result's
 `reason` into the failed command envelope's `error` and `user_visible_message` fields;
 the harness matches that envelope rather than the recipe payload's `failed_step`/`reason`.
 
+For the implementation pipeline, the project-owned
+[`DispatchCompleted.to_envelope`](../../src/autoskillit/fleet/campaign_state/state_outcomes.py)
+must report success, `dispatch_status="success"` and root/dispatch identities. The nested
+[`implementation` terminal](../../src/autoskillit/recipes/implementation.yaml) must have
+`reason="implementation_complete"`. Successful no-change and already-done terminals remain
+valid recipe behavior but fail this scenario. No PR URL is required in the envelope.
+
+The harness resolves the configured seed once and dispatches its exact URL and title/body
+task once. Effective project-aware config comes from `autoskillit config show`, whose
+implementation calls `load_config(Path.cwd())`; both configured and sandbox default branches
+must be `main`. No caller `base_branch` ingredient is sent. Sandbox setup and baseline tests
+must succeed before launch.
+
+Paired root transcript tool calls/results prove the created branch; paired root/managed
+claim results prove any added claim label. The collector accepts the direct
+`mcp__autoskillit__` and plugin `mcp__plugin_autoskillit_autoskillit__` namespaces;
+a tracked tool under any other namespace fails the proof. Claude Code persisted
+transcripts carry replies in row-level `mcpMeta.structuredContent` when `mcpMeta`
+is present, otherwise in JSON block content. Malformed metadata or metadata on a
+row with several results fails the paired resource call. Hook display text never
+replaces unusable metadata. The pretty-output hook emits `updatedMCPToolOutput`;
+`_result_object` remains the shared envelope decoder, and metadata presence alone
+does not identify whether the hook ran. These shapes are pinned by
+`tests/infra/fixtures/e2e_sessions/` from Claude Code 2.1.280; re-capture them when
+an otherwise successful call reports a failed branch or claim result.
+
+Discovery paginates through the PR baseline
+and leaves unrelated PRs untouched. Require exactly one owned open, unmerged sandbox PR
+targeting `main`, without auto-merge, with the canonical seed URL and non-removed changes
+under both `sandbox/` and `tests/`. Fetch and detach at the captured head before setup and
+tests. Require a successful associated sandbox `CI` run and `test` job whose nested PR
+association has the exact branch revision; an outer merge SHA is not the branch SHA.
+Finally re-read the head and state to reject movement during validation.
+
+Teardown runs after validation, failure or exception. It independently collects metrics,
+closes owned open PRs, deletes evidenced branches (including branches without a PR), and
+removes only the effective claim label proven added by this run. The seed remains OPEN
+and other labels are preserved. Missing ownership proof preserves uncertain resources and
+fails cleanup. Cleanup errors remain fatal alongside the original failure.
+
+Branch reconciliation lists the sandbox's branches before the run and after cleanup.
+A branch created during the run that `ownership.json` does not explain is preserved
+and fails cleanup; baseline branches are excluded from unexplained-branch reports.
+Branches named by the ownership proof retain the existing cleanup rule. The original
+API stdout saved as `head-refs-before.stdout.txt` and `head-refs-after.stdout.txt` supplies
+the in-memory snapshots; accompanying stderr and command records diagnose failed requests.
+This check only detects unexplained branches: the creating run records the failure once,
+and the branch and any PR on it must be removed by hand. The next run includes it in
+its baseline. A manual run overlapping CI can trigger the check because CI serialization
+does not cover local runs.
+
 ## Catalog schema
 
 ```json
@@ -329,15 +438,38 @@ the harness matches that envelope rather than the recipe payload's `failed_step`
 ```
 
 `recipe`, `ingredients` and `expected_pull_request_state` are required for `recipe` tests
-and forbidden for `canary` and `clean-install` tests. `expected_failures` is optional for
-`clean-install` and recipe tests. Clean-install rows use the exact doctor `severity`, `check`
-and `message`; recipe rows require `severity: error`, `check` and `message` matching the failed
-envelope. Every row includes an open bug issue URL
-(`https://github.com/TalonT-Org/AutoSkillit/issues/<number>`). Duplicate diagnostic identities
-and unknown keys are rejected. `peak_sessions` must be
+and forbidden for `canary` and `clean-install` tests. Optional recipe-only fields are
+`pipeline` (boolean, default false), `issue_url_env` (seed variable name),
+`required_changed_paths` (nonempty relative prefixes), `test_command` (nonempty argv), and
+`peak_sessions_evidence` (nonempty provenance string). Environment-owned seeds cannot also
+specify literal `task` or `issue_url` ingredients. Path prefixes cannot be absolute or
+traverse parents. For example, the implementation entry uses:
+
+```json
+{"pipeline": true, "issue_url_env": "E2E_IMPLEMENTATION_ISSUE_URL",
+ "required_changed_paths": ["sandbox/", "tests/"], "test_command": ["task", "test-check"]}
+```
+
+The recipe_fixture field names a staged YAML fixture for the headless smoke scenario.
+
+The expected_failures field is optional for recipes and clean-install. Pipeline rows
+contain exactly check, message, issue; ordinary recipe and clean-install rows also
+contain severity. Ordinary recipes require severity: error and match the failed
+envelope's error and user_visible_message; clean-install allows error or warning
+and matches doctor diagnostics. Pipeline rows match structured implementation findings.
+Every row links to an open AutoSkillit bug issue. Duplicate diagnostic identities and
+unknown keys are rejected. The peak_sessions value must be
 1–8 for model tests and exactly zero for clean-install. `timeout_sec` must keep the
 derived job timeout within 360 minutes.
 
+Recipe expectations match exact stable `(check, message)` product findings linked to real
+bug issues. Setup, discovery, malformed evidence and cleanup failures cannot be admitted.
+Unknown findings and absent expected findings fail. Every recipe records `outcome`:
+`passed`, `expected_failure` (with `passed=false`), or `failed`. A linked known defect may
+use the accepted CLI exit convention, but only `passed` with no expected findings counts
+as successful commissioning. File or deduplicate observed product bugs separately; do not
+predeclare a provider defect or replace the MiniMax settings-file route.
+
 ### Adding a test
 
 1. Add the entry to `catalog.json` with a realistic `peak_sessions` and `timeout_sec`.

scripts/e2e/autoskillit-config.yaml

@@ -3,6 +3,9 @@
 quota_guard:
   enabled: false
 
+branching:
+  default_base_branch: main
+
 model:
   default: sonnet[1m]
 

scripts/e2e/catalog.json

@@ -70,6 +70,35 @@
       "recipe_fixture": "sandbox-smoke.yaml",
       "ingredients": {},
       "expected_pull_request_state": "closed"
+    },
+    {
+      "name": "implementation",
+      "kind": "recipe",
+      "pipeline": true,
+      "peak_sessions": 8,
+      "timeout_sec": 7200,
+      "trigger_paths": [
+        "src/autoskillit/recipes/*",
+        "src/autoskillit/skills/*",
+        "src/autoskillit/skills_extended/*",
+        "src/autoskillit/server/*",
+        "src/autoskillit/execution/*"
+      ],
+      "recipe": "implementation",
+      "ingredients": {
+        "source_dir": "/workspace/sandbox",
+        "open_pr": "true",
+        "auto_merge": "false",
+        "arch_lenses": "false",
+        "audit_impl": "true"
+      },
+      "expected_pull_request_state": "open",
+      "issue_url_env": "E2E_IMPLEMENTATION_ISSUE_URL",
+      "required_changed_paths": [
+        "sandbox/",
+        "tests/"
+      ],
+      "test_command": ["task", "test-check"]
     }
   ]
 }