Skip to content

Add seeded sandbox implementation E2E validation - #5255

Open
Trecek wants to merge 26 commits into
developfrom
impl-issue-5233-20261008-implementation
Open

Trecek wants to merge 26 commits into
developfrom
impl-issue-5233-20261008-implementation

Conversation

@Trecek

@Trecek Trecek commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a seeded sandbox implementation scenario to the MiniMax E2E harness. It resolves a fixed sandbox issue, runs the bundled implementation recipe, validates the owned PR and source/test changes at its captured head, and independently cleans up the PR, branch and proven issue claim after success or failure.

Pipeline selection requires a large PR touching the configured production paths and admits at most one pipeline within the existing test cap. Synchronous lifecycle hooks and transcript ownership evidence provide conservative concurrency measurements; missing evidence fails validation. Recipe findings support narrowly linked expected failures, which do not qualify as commissioning success.

Refs #5233.

Current commissioning status — 2026-10-18

Not ready to merge: two accepted complete MiniMax runs are still missing. Published head e529c48a8d6576ca7bbff29a51c0d7ee12caa167, clean merge-forward of develop a93c20037bddef0c534af3f7da44ca4310733b33 (0.10.1281), includes #5290's fix for #5289. All-files pre-commit and 502 focused tests passed. No tracked source edits beyond that sync.

Verification Result
Tests39177289447 Passed, including Channel B
Automatic E2E39177289446 Gate passed; scenario selection skipped, no fresh smoke claim
Implementation39177297225 Failed: audit semantic handback missing
Implementation39177304001 Failed: semantic rejection followed by invalid audit retry

Both runs produced the requested source/test changes and passed sandbox tests/integration. Both audit workers altered integer values in the supplied ArtifactRef into strings, so strict submission validation rejected them. Their prose claims that valid integers were rejected contradict the actual tool arguments. The subsequent L2 retries omitted the recipe-declared retry_after_audit_attempt_id; that is recorded on #45/#4482, not a newly invented wiring defect. Neither run reached prepare-pr, review or PR creation, so the current #5289 fix remains unverified by this live path and is not reopened.

Confirmed source contract gap: #5291, missing nested ArtifactRef types/required fields in the registered MCP schema despite strict handler validation. Two adversarial reviewers accepted the narrowed issue; the primary investigator independently checked raw records, actual FastMCP metadata and source. Fix base is develop. This does not prove a valid-input validator bug or guarantee a schema change alone makes MiniMax comply. The separate planning/audit ownership gap is tracked at #5292: run2's worker-authored NO GO demanded later PR publication, but it was not an admitted audit verdict.

No sandbox PR or accepted PR/head CI association was produced. Pushed sandbox commits were 0321b91ffca6fa6f1f09ecc69f0fd707be8966d2 and 0117990ac78c19a96e7ece8a4ca5939cd141cd53. Both lifecycle traces have complete coverage/no violations, observed peaks of 5 and 6, and retained root/managed context windows of 1M (165/114 rows). peak_sessions=8 remains provisional: these failed/incomplete runs supply no accepted capacity evidence.

Final sandbox verification: generated /1-3 refs absent, no open PRs, seed1 OPEN/unclaimed with a fail label, main unchanged at d6c143d2494488deeb6680d7d5cf2e55e3e01732; unrelated /1 and /1-2 refs retained. Existing reporter bug #5249 recurred downstream. All workflows finished; no third implementation attempt is running. Local Docker updated to 0.10.1281, stopped, persistent auth/workspace retained.

Next: remediate the confirmed develop-owned contracts, integrate develop, and obtain two fresh complete accepted MiniMax runs against the same seed. Require valid attested audit GO, full routed PR progression, canonical issue metadata, exact PR/head CI association, strict owned cleanup and complete lifecycle evidence before publishing measured capacity or merging.

@Trecek
Trecek force-pushed the impl-issue-5233-20261008-implementation branch from f0f09ee to 017b3f1 Compare October 8, 2026 15:02
@Trecek
Trecek marked this pull request as ready for review October 8, 2026 15:23
@Trecek
Trecek added this pull request to the merge queue Oct 8, 2026
@Trecek
Trecek removed this pull request from the merge queue due to a manual request Oct 8, 2026
Trecek added a commit that referenced this pull request Oct 13, 2026
Managed headless launches currently leave lineage records visible to
Git, so the clone gate can treat AutoSkillit's runtime state as user
changes and refuse a clean checkout. This change gives the existing
lineage namespace its own ignore marker, validates that marker before
creating record directories, and repairs missing markers when a store
operation reads an older namespace.

The temp directory and lineage store share the marker writer. Doctor and
the root guards share the marker predicate; committed project
directories are classified explicitly. Real-Git tests cover launch
cleanliness, legacy repair, dispatch state, and continued detection of
user edits. The static guard rejects unclassified `.autoskillit`
children in production source.

Related issue: #5258. This PR remains draft pending the maintainer-owned
policy registration.

## Validation

- `pre-commit run --all-files` passed before each commit.
- Manual real-Git probe: `before=[]`, `after=[]`; tracked ignore file
unchanged; doctor OK.
- Mutation checks rejected both literal-root and root-variable path
spellings; the temporary mutations were removed.
- Configured `test_check` (`task test-local-gate`) passed on
`068f570b6`: 47,307 passed, 610 skipped, 29 xfailed.
- The independent catalog row-count oracle is synchronized with the
added guard row.
- `core/io/io.py` measures 749 non-import lines against its 750-line
limit.

## Required before completion

A maintainer must add this prescribed entry to `POLICY_SURFACES` in
`tests/arch/_acceptance_policy_surfaces.py`, then rerun the configured
`test_check` gate after that policy change:

```python
PolicySurface(
    "tests/arch/test_autoskillit_root_children_registered.py",
    "_CLASSIFIED_ELSEWHERE",
    "int_map",
    default=0,
),
```

The verified plan explicitly reserves that file for human edits, so it
is unchanged in this PR. The post-merge E2E replay and PR #5255
commissioning remain operator-run checks. Separate follow-up trackers
should be linked before closing #5258; this PR does not close the issue
automatically.


## Audit response

The audit's blocking REQ-006 finding used physical lines (`wc -l`)
instead of the repository's defined budget count.
`tests/arch/_line_budget.py::count_budget_lines` excludes
import-statement spans, and `scripts/check_file_lengths.py` delegates to
that function.

| Revision | Physical lines | Excluded import lines | Enforced budget
lines |
|---|---:|---:|---:|
| Plan base `265f1e00e` | 766 | 23 | 743 |
| PR implementation | 772 | 23 | 749 |

The unchanged cap is 750 non-import lines. The plan's starting and final
budget counts were correct, and its stop-and-report condition was not
triggered. Direct `check_file(io.py)` returns no violation. No
exemption, counter/cap change, unrelated trim, or module split was
needed.

The three ODD test identifiers now exactly match the verified plan in
`6194da647`; their assertions and production behavior are unchanged. The
gated rerun found one additional integration omission: the independent
catalog-count oracle still expected the old table. `068f570b6`
synchronizes that oracle while retaining its strict equality assertion,
and the next full configured gate passed.

E-3 remains deferred for the reason stated in both the plan and audit:
it requires the post-merge integration state and operator-run E2E
commissioning. The human-owned policy registration above remains a
separate prerequisite; no official re-audit GO verdict or merge
completion is claimed.
Trecek added a commit that referenced this pull request Oct 13, 2026
Runtime launches could create or backfill `.autoskillit/.gitignore`,
turning AutoSkillit's own work into Git-visible changes. Init's
`.state.yaml` was also missing from the canonical ignore rules, and the
fleet diagnostic filter could erase a useful clone-gate failure merely
because it named the managed lineage directory.

This change separates runtime temp creation from explicit project setup.
Init and direct install maintain the canonical ignore file; temp
creation maintains only its own self-ignore marker. Init covers
`.state.yaml` before writing it, preserves existing state fields, and
leaves ignore files untouched when state validation fails. Fleet failure
reasons retain `bootstrap_clone`/`uncommitted_changes` while redacting
private paths and capture metadata; the raw L3 payload remains exact.

The root-child classification flags return to the integer-map format
required by the existing acceptance-policy reader. Regression coverage
exercises absent/stale tracked ignore files, real user edits,
zero-valued flags, state-write ordering, and
directory/record/mixed/private-only fleet reasons. The core gateway/stub
and existing snapshot are synchronized; the IO file remains within its
existing line budget.

## Validation

- `pre-commit run --all-files`: passed before each commit.
- Affected task-based checks: 331 passed, 1 skipped.
- Source-inventory guard checks: 39 passed.
- Configured `test_check` (`task test-local-gate`): 51,233 passed, 615
skipped, 29 xfailed on `5854498a6f980782aa0b9f883b1e90a282c7f8f3`.

## Completion requirements for #5258

This PR delivers the code fixes, including the three adjacent defects.
It does not close #5258 by itself.

- [ ] After this integer-map restoration lands in the trusted comparison
base, a maintainer registers `_CLASSIFIED_ELSEWHERE` in
`tests/arch/_acceptance_policy_surfaces.py` as `int_map` with
`default=0`, then runs the configured gate. The authority file is
intentionally untouched here. Adding the row against the current
set-shaped base would fail the policy parser's base comparison.
- [ ] Integrate the completed code and policy registration into #5255's
existing implementation scenario branch and run E-3 twice against the
unchanged sandbox seed. Both envelopes must clear `bootstrap_clone`
without `uncommitted_changes`; record actual run/attempt/head evidence
in #5258's body.
- [ ] Record completed code/policy/replay evidence in #5258 before
closing it. Full #5255 commissioning and its later #5259 allocation
blocker retain their separate acceptance criteria.

Refs #5258. No new tickets or review comments are created by this
change.
Trecek added a commit that referenced this pull request Oct 13, 2026
The runtime-state root classification map was restored in #5263, but it
was still absent from the acceptance-policy registry. Future additions
to its exemptions could therefore escape the policy comparison.

Register `_CLASSIFIED_ELSEWHERE` as an `int_map` with `default=0`. The
registration now compares against the dictionary-shaped base containing
#5263, preserving the existing policy reader and approval mechanism.

Validation: policy comparison against
`dbc689db57d3d847758ec08dd5bad31105f39730` and `pre-commit run
--all-files` passed. The configured MCP `test_check` (`task
test-local-gate`, conservative filtering) passed against that actual PR
base: 11,771 passed, 197 skipped, 12 xfailed on
`3a0a9f8eae7a4d84a7e14d8b0b899ea76b30bd07`.

The first gate compared against stale local `develop` at `0bf84f3`,
whose value still had the old set shape: 51,229 passed and one
policy-comparison test failed. The runner's comparison base was
temporarily pinned to the fetched, dictionary-shaped PR base for the
successful rerun, then restored. Both results are preserved locally.

Refs #5258. After merge, integrate the registered policy and runtime
repairs into the existing #5255 scenario branch, perform both E-3
replays against the unchanged seed, and record the actual
clone-admission evidence in #5258 before closing it. This PR supplies
the remaining policy entry; it does not claim the live replay has
already happened.
Trecek and others added 3 commits October 13, 2026 08:58
…5269)

Real E2E transcripts used the plugin MCP namespace and row-level
structured replies, so the collector missed the claim, published branch,
and launched workers. Teardown could then silently leave a branch behind
when no PR or claim label remained.

This change decodes both canonical namespaces and authoritative
structured receipts, adopts dispatch-valid workers through paired
launch/acknowledgement/recovery receipts, and reports unexplained
sessions. The harness records sandbox head refs before the run and after
cleanup, preserving and failing cleanup for new branches without
ownership proof. Sanitized captured CI evidence exercises ownership,
concurrency, and teardown after a failed run.

Stacked on #5255, targeting `impl-issue-5233-20261008-implementation`,
where the collector was introduced.

Validation:
- All three phases passed `pre-commit run --all-files` and the
configured `task test-local-gate` through the MCP test gate.
- Final gate: 16,021 passed, 207 skipped, 12 xfailed.
- All 15 planned mutation checks passed; temporary test selection was
restored.
- Captured replay adopts both workers and reports peak concurrency of
five. The existing unmatched native stop still fails lifecycle coverage,
as required.

Live GitHub head-ref listing will be confirmed by the first E2E run
after merge.

Fixes #5267

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Trecek and others added 2 commits October 14, 2026 09:27
Unnamed internal-agent `SubagentStop` callbacks made E2E native
lifecycle coverage fail. Install `.+` for agent hooks from the
collector's shared lifecycle declaration, keep session hooks unfiltered,
and reject a main-agent key in user settings before installation.

The collector retains delivered callbacks and adds a
`capture:...:agent_type_missing` violation for unnamed owned agent
records. Regression tests cover settings preservation, main-agent
preflight rejection, resolvable agent names, fleet launch configuration,
named orphan stops, and the historical trace under the declared capture
policy. The README documents the capture precondition and the unchecked
external settings channels, whose named orphan stops still fail
coverage.

Validation: `pre-commit run --all-files` passed. `task test-local-gate`,
invoked through `test_check`, first produced exactly the 10 expected
regression failures before implementation, then passed with 50,926 tests
passing after implementation.

Fixes #5271
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant