Repository navigation
Conversation
Add test/e2e/agent (build tag e2eagent): an mcp-go stdio client replays testdata/replay.jsonl against `nself mcp` started through scripts/e2e/agent/mcp-sandbox.sh, with two loopback simharness hosts in inventory envs qa (tier local-servers) and prod (tier prod). The flow is init, build (plan, confirm), start, the failure-recovery and destructive-gate legs, db migrate up, add ci, async CI (detach, runs wait, evidence), deploy qa, promote dry run, confirmed promote. Every request, result and harness action goes to a JSONL transcript that verify.go checks; bad fixtures prove the verifier rejects non-MCP use, a missing step, a promote apply before its dry run, CI evidence before the wait and a non-v1 envelope. mcp-sandbox.sh runs the server under env -i with a temp HOME holding only the per-run sim key and ssh config, and refuses SSH_AUTH_SOCK, a reachable operator ~/.ssh and any host other than 127.0.0.1. The only key is the one simharness generates at run time; nothing credential-shaped is committed. run.sh modes: --replay, --live (headless agent session with only the MCP server, optional harness-side prod probe), --verify-transcript, --self-test, --tools-list. The Agent E2E workflow runs the replay on ubuntu-latest and uploads agent-e2e-transcript. Vendors the mcp-go client and client/transport packages (go.mod unchanged). The README lists the product gaps the replay hits today. (P7-SHIP-02)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
P7-SHIP-02: the MCP-only agent E2E harness (SHIP Epic D7).
test/e2e/agent/(build tage2eagent): mcp-go stdio client replayingtestdata/replay.jsonlagainst a sandboxednself mcp; two loopback simharness hosts (qatier local-servers,prodtier prod) in the inventory v2 fixture; failure-recovery and destructive-gate legs; transcript writer and verifier with bad fixtures.scripts/e2e/agent/run.sh(--replay,--live,--verify-transcript,--self-test,--tools-list) andmcp-sandbox.sh(env -i, temp HOME, loopback-only allowlist, refuses SSH_AUTH_SOCK and a reachable operator~/.ssh)..github/workflows/agent-e2e.yml(ubuntu-latest, actions pinned by SHA, artifactagent-e2e-transcript).mcp-go/clientandclient/transport; go.mod unchanged.test/e2e/agent/README.md.No committed key or token: the only key is the one simharness generates per run. gitleaks 8.21.2 (gate-pinned) with
.github/gitleaks.toml: clean.Verified locally
go vet -mod=vendor -tags e2eagent ./test/e2e/agent/...run.sh --self-test(unit tests plus four sandbox refusals and a clean accept)run.sh --verify-transcriptexits 1 on all five bad fixtures, 0 on the good onerun.sh --tools-list <bin> --env NSELF_V15=1returns a sorted non-empty arrayNot green yet: product gaps the replay hits
The
Agent E2Ereplay drives the real product and stops at the first failing step. Expected to fail until these close (details in the README):nself mcp(v1.5) will not start outside a project, soinitis unreachable over MCP (harness writes an empty.envmarker as a stopgap).nself_buildon a fresh project fails E451: the gate runs--plan-idon a build that generates secrets.nself_ci*never appear afternself_add ciin the same session; P7-CI-62 (ci machine surface) is not merged.nself_deploy_promoteaccepts one argument only, soqa prodis rejected.nself mcp --timeoutflag.(P7-SHIP-02)