test: paired experimental sealed-catalog binding and refusal checks

[altrudev]

Experimental sealed-catalog consumer acceptance/refusal fixtures

Status: DRAFT / NON-NORMATIVE / MAINTAINER REVIEW REQUIRED. Test-only software evidence associated with agentrust-io/agent-manifest#501 and draft #502; not a production, live-provider, hardware, or conformance claim.

Pinned head: c2fe10d16c692cb01397e897b6f70088c037f1b6. Uses cMCP's own locked Agent Manifest SDK dependency (released 0.15.0 for the independent review), not a Git SHA from the upstream draft.

Evidence includes a signed experimental COSE requirements profile against the runtime ToolCatalog; acceptance/refusal for wrong/missing seals, wrong runtime policy, authenticated subject or server identity changes, expiry, malformed/duplicate/unknown requirements, tampering, and server URL/output schema substitution. The supplemental CLI controls contributed by @dinakarjs (dinakarjs#2) invoke the real Click startup and observe build_server, CMCPProxy construction and mocked upstream forwarding. Valid control constructs/forwards once; refusal controls do neither in the instrumented software harness.

Evidence: Independent Linux/Python 3.12.14 review at this head reports 22 focused tests passed, 2,365 unit/integration tests passed with 6 skips, targeted Ruff passed; process-local bypass mutations caught in reviewer test scope. Latest-head GitHub CI, CodeQL and Confinement reference workflows report success. These are reviewer and hosted-CI results, not a new local run.

EDRG remaining / NOT ESTABLISHED: live provider dispatch observations, complete independent transport-level observations across providers, hardware TPM/TEE, broad verifier bypass mutation matrix, production conformance, normative status, and maintainer acceptance. The contributor check is classified HIGH (Profile HIGH, Credential LOW). Leave security/approval gates unchanged and the PR in draft pending maintainer review.

Requested next step: maintainer review of the experimental evidence and agreement on required additional acceptance controls.

Val Rukhaylo / Altru.dev

[github-actions]

🔴 Contributor Check: HIGH

Check	Result
Profile	HIGH
Credential	LOW
Overall	HIGH

Automated check by AgenTrust Contributor Check.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[altrudev]

Final local EDRG rerun for this draft at head 0581739 (VPS, Python 3.12.15): cMCP 2,353 passed, 6 skipped, 0 failed; new 10-case consumer module passes; Ruff and git diff --check pass. Coverage: conformant signed experimental COSE profile with actual runtime catalog (accepted), missing/mismatched signed seal and wrong runtime policy (refused), real startup accepts conformant COSE and rejects mismatched seal, changed server URL and changed output schema rejected when catalog is reloaded, and a bounded synthetic proxy harness forwards once only after successful authenticated startup; the invalid startup aborts without constructing a proxy or forwarding. Mutations tested separately and reverted: bypassing sealed catalog verifier kills 2 refusal cases; substituting the authenticated subject kills existing startup denial test. The synthetic forwarding is mocked, not a live remote provider call or TPM/TEE attestation. Broader bypass mutation coverage and production dispatch observations remain NOT ESTABLISHED. Upstream CI on final head and independent maintainer review remain pending; keep draft and unmerged.

[dinakarjs]

Independent review at 0581739, Python 3.12.14 with hash-locked dev dependencies and the released Agent Manifest 0.15.0 dependency.

Validation:

• Existing paired module: 10 passed; unit/integration suite: 2,353 passed, 6 skipped.
• A process-local mutation making _require_sealed_catalog_requirement a no-op caused 6 of the 10 paired tests to fail. No mutation was committed.
• Ruff passed. The initial full run encountered inherited SOCKS proxy environment failures; the successful rerun cleared proxy variables without adding an undeclared dependency.

One evidence gap remains in the forwarding test: the refusal branch initializes forwarded = 0 and never constructs a proxy or observes the CLI composition path. It demonstrates startup refusal for the wrong-seal case, but its zero count is not independent instrumentation of construction/dispatch across all negative fixtures.

I prepared a supplemental draft for integration: dinakarjs#2 (6b0db92), based directly on this head. Twelve controls invoke the real Click start command and observe the real build_server call, CMCPProxy construction, and mocked upstream forwarding. The valid control constructs once and forwards once; eleven refusal controls observe no construction and no forwarding (wrong/missing seal, policy, subject, expiry, unknown key, missing/duplicate/unknown requirements, signature tampering, and catalog server substitution).

Supplemental focused: 12 passed; combined unit/integration: 2,365 passed, 6 skipped; Ruff passed. Hosted confinement and contributor checks passed, Vouch skipped; full hosted CI did not trigger on the custom base branch.

These are software tests with mocked transport/AGT, with no live provider or hardware observation. This review does not establish production attestation or conformance, and preserves the draft/maintainer gates.

[altrudev]

Independent reviewer contribution integrated at c2fe10d. Thanks @dinakarjs for the supplemental CLI composition controls (dinakarjs#2, 6b0db92). The twelve tests now reside in this existing draft PR, with source attribution in the commit, rather than a competing upstream submission. Local VPS Python 3.12.15: new CLI cases 12/12 pass; focused combined profile cases 22/22; full cMCP unit/integration suite 2,365 passed / 6 skipped / 0 failed; Ruff and diff checks pass. The tests observe real Click startup, server/proxy construction, and mocked upstream forwarding across one admission and eleven refusals. Transport, hardware and live-provider attestation remain outside this synthetic scope. GitHub CI has restarted on the new head; independent maintainer approval is still needed. Keep draft, no merge.

[dinakarjs]

Verified integration at c2fe10d. The supplemental CLI controls from dinakarjs#2 are present with unchanged acceptance/refusal behavior; formatting adjustments do not alter the observations.

Independent Linux/Python 3.12.14 rerun at this checkout: focused original plus CLI controls 22 passed; full unit/integration 2,365 passed, 6 skipped; targeted Ruff passed. Runtime imports were checked to resolve into this exact checkout; full execution cleared inherited proxy variables, using the existing locked environment and released Agent Manifest dependency.

Process-local sealed-catalog bypass mutation (_require_sealed_catalog_requirement -> no-op) kills three of the twelve CLI controls: wrong_seal, missing_seal, catalog_server; nine pass, including the positive control. No mutation was committed. The earlier instrumentation gap is addressed for these twelve bounded CLI cases: the valid case constructs and forwards once, while eleven invalid cases observe neither construction nor forwarding.

This is reviewer-contributed test code, so I am recording verification rather than providing independent maintainer approval. Mocked transport/AGT, synthetic software only; no hardware/provider or completeness claim. Keep draft pending normal maintainer review.

[carloshvp]

Reviewed exact head c2fe10d16c692cb01397e897b6f70088c037f1b6 in an isolated checkout on macOS, Python 3.12.10, using cMCP's hash-locked dependencies, including released Agent Manifest 0.15.0. The SDK draft branch was not substituted for that dependency.

Requesting changes for the repository's DCO requirement: CONTRIBUTING.md requires every commit to contain Signed-off-by. None of the five commits in this PR has that trailer. Please add the appropriate author's sign-off to every commit; a subsequent signed-off commit does not certify the unsigned ancestors.

Technical validation passed:

• New binding/CLI modules: 22 passed. Full CI test selection (unit, conformance, integration, confinement): 2,461 passed, 29 skipped.
• Repository Ruff, runtime mypy, diff checks, and merge simulation against current main 99b80335c33789bb1d8cd575c2f3681ff82aaa1c passed.
• The CLI controls exercise the real startup/builder path and instrument proxy construction and mocked forwarding. The valid case constructs and forwards once; all eleven refusal cases observe neither.
• Process-local sealed-catalog bypass makes nine of the 22 tests fail, including the wrong/missing-seal and substituted-server CLI controls. Bypassing authenticated-subject matching makes the wrong-subject CLI control fail while the valid control passes. A reject-all startup binding makes the CLI positive control fail. No mutation was committed.

I found no additional blocking defect in this bounded test change. The earlier synthetic forwarding counter by itself is insufficient evidence of non-dispatch; the supplemental CLI module supplies actual instrumentation for its twelve cases. Transport and AGT behavior remain mocked, startup uses development-mode fixtures, and local skips do not establish native Linux/Docker or hardware/provider behavior. This does not promote the experimental profile or establish complete mediation. Draft status, contributor-policy checks, and required approvals remain separate from the DCO correction requested here.

[dinakarjs]

@altrudev I checked Carlos's DCO finding against the current patch series at c2fe10d16c692cb01397e897b6f70088c037f1b6: all 5 commits are authored by Val Rukhaylo and lack Signed-off-by, including the commit integrating my supplemental controls. The source attribution to my contribution should remain intact.

Please make the DCO certification yourself on your fork; I cannot supply an author's certification on your behalf. With a clean checkout of test/edrg-experimental-catalog-binding-501, your Git user.name/user.email set to the identity you intend to certify, and the remote verified to be your fork, the bounded correction is:

git rebase --signoff 99b80335c33789bb1d8cd575c2f3681ff82aaa1c
git log --format='%h %an <%ae>%n%B' 99b80335c33789bb1d8cd575c2f3681ff82aaa1c..HEAD
git diff --exit-code c2fe10d16c692cb01397e897b6f70088c037f1b6 HEAD --
git push --force-with-lease=refs/heads/test/edrg-experimental-catalog-binding-501:c2fe10d16c692cb01397e897b6f70088c037f1b6 origin HEAD:refs/heads/test/edrg-experimental-catalog-binding-501

The log should show a valid author sign-off on every commit; the diff command should confirm no file-content change. If the remote head has moved, inspect that update before retrying. After rewriting, refresh the pinned head in the description and ask Carlos to re-review; rerun normal CI at the new head. Preserve draft status and contributor/maintainer gates. This is a correction route, not a claim that the DCO issue is already fixed.