diff --git a/.codex/skills/verify-decision-contract/SKILL.md b/.codex/skills/verify-decision-contract/SKILL.md index c25b7004..32340262 100644 --- a/.codex/skills/verify-decision-contract/SKILL.md +++ b/.codex/skills/verify-decision-contract/SKILL.md @@ -1,16 +1,16 @@ --- name: verify-decision-contract -description: "Verify the pstack decision contract and canonical routing registry offline against an independently captured cursor/plugins Git tree." +description: "Verify the pstack decision contract, System One wire transport, provider profiles, and canonical routing registry offline against an independently captured cursor/plugins Git tree." disable-model-invocation: true --- # Verify the decision contract -Use this skill to check the Jev contract specifications and `references/decision-routing.json`. The verifier does not call a provider or change routing. +Use this skill to check the Jev contract specifications, provider profiles, class calibration policies, and `references/decision-routing.json`. The verifier does not call external providers during offline verification. ## Launch -Set up the verification environment and verify prerequisite tooling (`python3`, `openspec`, `git`). +Set up the verification environment and verify prerequisite tooling (`python3`, `uv`, `openspec`, `git`). Create an isolated run directory for captured evidence. ```bash @@ -19,26 +19,31 @@ run_dir=$(mktemp -d /tmp/verify-decision-contract.XXXXXX) ## Doctor -Run strict OpenSpec validation checks for the change and specification: +Run OpenSpec validation checks for the specification: ```bash -openspec validate pstack-jev-decision-plane --type change --strict openspec validate pstack-jev-decisions --type spec --strict +uv run pytest tests/test_verify_decision_contract.py tests/test_jev_transport_and_profiles.py tests/test_class_calibration_policy.py ``` ## Drive -Drive the contract verifier against upstream commit and tree captures: +Drive the contract verifier, eval suite, and rollback equivalence against local and upstream captures: ```bash python3 scripts/verify-decision-contract.py --root . --upstream-commit --upstream-tree --evidence-dir "$run_dir" +python3 scripts/eval_decision_routing.py --format json +python3 scripts/eval_blinded_pstack.py --format json +python3 scripts/verify_rollback_equivalence.py --format json ``` ## Proof bar -Assert zero verification errors, 100% route coverage (23/23 playbooks), and strict specification parity between formal spec and change delta. -Enforce three-tier test classification: Tier 1 fast contract unit tests, Tier 2 offline integration fixtures, and Tier 3 host runtime probes. -Reduce scenario verdicts with the strict failure-first lattice: FAIL > BLOCKED > UNTESTED > PASS. +Assert zero verification errors, 100% route coverage across 23 playbooks, and strict specification parity between formal spec and change delta. +Enforce three-tier test classification with Tier 1 fast contract unit tests, Tier 2 offline integration fixtures, and Tier 3 host runtime probes. +Prove lean System One wire protocol compliance without third-party SDK dependencies per `references/jev-provider-options.md`. +Enforce per-class confidence thresholds and >=0.10 probability margins with fail-open baseline fallback. +Reduce scenario verdicts with the strict failure-first lattice where FAIL beats BLOCKED, BLOCKED beats UNTESTED, and UNTESTED beats PASS. ## Evidence diff --git a/.codex/skills/verify-decision-contract/features/decision-adapter.md b/.codex/skills/verify-decision-contract/features/decision-adapter.md index 48d3c222..aeffb7c2 100644 --- a/.codex/skills/verify-decision-contract/features/decision-adapter.md +++ b/.codex/skills/verify-decision-contract/features/decision-adapter.md @@ -3,13 +3,15 @@ ## Sub-features - Single-operation SemanticDecisionProvider contract - Strict egress boundary validation and SafeContext enforcement -- Default-OFF feature gate and fail-open exception handling +- Lean native System One wire protocol over standard library urllib +- Provider profile configuration across opencode-console, opencode-zen, openrouter, and typesafe-direct +- Default-OFF feature gate, paid provider consent gate, and fail-open exception handling ## How to get to it (user POV) -Inspect `scripts/decision_adapter.py` and run `pytest tests/test_decision_adapter.py`. +Inspect `scripts/decision_adapter.py` and `references/jev-provider-options.md`. Run `pytest tests/test_decision_adapter.py` and `pytest tests/test_jev_transport_and_profiles.py`. ## Driving it with Codex -Execute `uv run pytest tests/test_decision_adapter.py tests/test_shadow_routing.py`. +Execute `uv run pytest tests/test_decision_adapter.py tests/test_shadow_routing.py tests/test_jev_transport_and_profiles.py`. ## Gotchas -Provider failures, timeouts, or invalid requests must never raise uncaught exceptions to the caller. +Provider failures, network timeouts, unapproved paid profiles, or invalid responses must never raise uncaught exceptions to the caller. Fail open cleanly to baseline. diff --git a/.codex/skills/verify-decision-contract/features/offline-eval.md b/.codex/skills/verify-decision-contract/features/offline-eval.md index e1a47e0d..a0f16ff7 100644 --- a/.codex/skills/verify-decision-contract/features/offline-eval.md +++ b/.codex/skills/verify-decision-contract/features/offline-eval.md @@ -2,14 +2,16 @@ ## Sub-features - Rerunnable offline routing eval corpus and calibration lever +- Class-specific confidence thresholds and minimum probability margin requirement +- Uncalibrated class fencing and graceful fallback to baseline routing - Revision-bound evidence receipts and mechanical staleness detection - Blinded behavioral evaluation comparing baseline and shadow candidates ## How to get to it (user POV) -Run `python3 scripts/eval_decision_routing.py` and `python3 scripts/eval_blinded_pstack.py`. +Run `python3 scripts/eval_decision_routing.py` and `python3 scripts/eval_blinded_pstack.py`. Inspect `scripts/route_resolver.py` and run `pytest tests/test_class_calibration_policy.py`. ## Driving it with Codex -Execute `python3 scripts/eval_decision_routing.py --format json` and verify verdict is PASS. +Execute `python3 scripts/eval_decision_routing.py --format json` and `uv run pytest tests/test_class_calibration_policy.py`. ## Gotchas -Receipts lacking complete revision bindings or having unknown model identity cannot authorize control. +Receipts lacking complete revision bindings, having uncalibrated classes, or failing margin requirements cannot authorize control. Baseline routing remains active. diff --git a/.codex/skills/verify-decision-contract/scripts/driver.sh b/.codex/skills/verify-decision-contract/scripts/driver.sh index 6f6e18e5..a3705dac 100755 --- a/.codex/skills/verify-decision-contract/scripts/driver.sh +++ b/.codex/skills/verify-decision-contract/scripts/driver.sh @@ -4,23 +4,34 @@ set -euo pipefail TIMEOUT_SECONDS=60 +export PSTACK_TRANSCRIPTS_DIR="${PSTACK_TRANSCRIPTS_DIR:-/tmp/empty-transcripts}" +mkdir -p "${PSTACK_TRANSCRIPTS_DIR}" + +if [ -x ".venv/bin/pytest" ]; then + PYTEST_CMD=".venv/bin/pytest" +else + PYTEST_CMD="python3 -m pytest" +fi run_bounded() { local label="$1" shift echo "==> Running $label (bounded by ${TIMEOUT_SECONDS}s)..." if command -v timeout >/dev/null 2>&1; then - timeout "${TIMEOUT_SECONDS}" "$@" + timeout --kill-after=5s "${TIMEOUT_SECONDS}" "$@" < /dev/null else - "$@" + "$@" < /dev/null fi } cmd="${1:-doctor}" case "$cmd" in doctor) - run_bounded "openspec-validate-change" openspec validate pstack-jev-decision-plane --type change --strict + if [ -d "openspec/changes/pstack-jev-decision-plane" ]; then + run_bounded "openspec-validate-change" openspec validate pstack-jev-decision-plane --type change --strict + fi run_bounded "openspec-validate-spec" openspec validate pstack-jev-decisions --type spec --strict + run_bounded "test-decision-contract" $PYTEST_CMD tests/test_verify_decision_contract.py tests/test_jev_transport_and_profiles.py tests/test_class_calibration_policy.py ;; launch) echo "Launch verified" @@ -29,6 +40,8 @@ case "$cmd" in run_bounded "eval-decision-routing" python3 scripts/eval_decision_routing.py --format json run_bounded "eval-blinded-pstack" python3 scripts/eval_blinded_pstack.py --format json run_bounded "verify-rollback-equivalence" python3 scripts/verify_rollback_equivalence.py --format json + run_bounded "jev-transport-and-profiles" $PYTEST_CMD tests/test_jev_transport_and_profiles.py + run_bounded "class-calibration-policy" $PYTEST_CMD tests/test_class_calibration_policy.py ;; cleanup) echo "Cleanup completed" diff --git a/scripts/verify-decision-contract.py b/scripts/verify-decision-contract.py index 66cf8e3f..115af875 100644 --- a/scripts/verify-decision-contract.py +++ b/scripts/verify-decision-contract.py @@ -257,6 +257,12 @@ def inspect_requirements(text: str, label: str) -> None: def compare_specs(root: Path) -> str: formal_path = root / "openspec" / "specs" / "pstack-jev-decisions" / "spec.md" delta_path = root / "openspec" / "changes" / "pstack-jev-decision-plane" / "specs" / "pstack-jev-decisions" / "spec.md" + if not delta_path.is_file(): + archived_matches = sorted( + root.glob("openspec/changes/archive/*pstack-jev-decision-plane/specs/pstack-jev-decisions/spec.md") + ) + if archived_matches: + delta_path = archived_matches[0] try: formal = formal_path.read_text(encoding="utf-8") delta = delta_path.read_text(encoding="utf-8") diff --git a/tests/test_verify_decision_contract.py b/tests/test_verify_decision_contract.py index bd45194e..54e11b76 100644 --- a/tests/test_verify_decision_contract.py +++ b/tests/test_verify_decision_contract.py @@ -303,3 +303,14 @@ def test_missing_scenario_fails(tmp_path: Path) -> None: formal.write_text(SPEC.split("#### Scenario:", 1)[0], encoding="utf-8") result, output = run_cli(repo, commit_path, tree_path) assert_failure(result, output, "Requirement has no Scenario") + + +def test_archived_spec_fallback_passes(tmp_path: Path) -> None: + repo, commit_path, tree_path = make_repo(tmp_path) + active_delta = repo / "openspec" / "changes" / "pstack-jev-decision-plane" + archive_dir = repo / "openspec" / "changes" / "archive" / "2026-10-13-pstack-jev-decision-plane" + archive_dir.parent.mkdir(parents=True, exist_ok=True) + shutil.move(str(active_delta), str(archive_dir)) + result, output = run_cli(repo, commit_path, tree_path) + assert result.returncode == 0 + assert output["verdict"] == "PASS"