Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 14 additions & 9 deletions .codex/skills/verify-decision-contract/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
---
name: verify-decision-contract
description: "Verify the pstack decision contract and canonical routing registry offline against an independently captured cursor/plugins Git tree."
description: "Verify the pstack decision contract, System One wire transport, provider profiles, and canonical routing registry offline against an independently captured cursor/plugins Git tree."
disable-model-invocation: true
---

# Verify the decision contract

Use this skill to check the Jev contract specifications and `references/decision-routing.json`. The verifier does not call a provider or change routing.
Use this skill to check the Jev contract specifications, provider profiles, class calibration policies, and `references/decision-routing.json`. The verifier does not call external providers during offline verification.

## Launch

Set up the verification environment and verify prerequisite tooling (`python3`, `openspec`, `git`).
Set up the verification environment and verify prerequisite tooling (`python3`, `uv`, `openspec`, `git`).
Create an isolated run directory for captured evidence.

```bash
Expand All @@ -19,26 +19,31 @@ run_dir=$(mktemp -d /tmp/verify-decision-contract.XXXXXX)

## Doctor

Run strict OpenSpec validation checks for the change and specification:
Run OpenSpec validation checks for the specification:

```bash
openspec validate pstack-jev-decision-plane --type change --strict
openspec validate pstack-jev-decisions --type spec --strict
uv run pytest tests/test_verify_decision_contract.py tests/test_jev_transport_and_profiles.py tests/test_class_calibration_policy.py
```

## Drive

Drive the contract verifier against upstream commit and tree captures:
Drive the contract verifier, eval suite, and rollback equivalence against local and upstream captures:

```bash
python3 scripts/verify-decision-contract.py --root . --upstream-commit <commit.json> --upstream-tree <tree.json> --evidence-dir "$run_dir"
python3 scripts/eval_decision_routing.py --format json
python3 scripts/eval_blinded_pstack.py --format json
python3 scripts/verify_rollback_equivalence.py --format json
```

## Proof bar

Assert zero verification errors, 100% route coverage (23/23 playbooks), and strict specification parity between formal spec and change delta.
Enforce three-tier test classification: Tier 1 fast contract unit tests, Tier 2 offline integration fixtures, and Tier 3 host runtime probes.
Reduce scenario verdicts with the strict failure-first lattice: FAIL > BLOCKED > UNTESTED > PASS.
Assert zero verification errors, 100% route coverage across 23 playbooks, and strict specification parity between formal spec and change delta.
Enforce three-tier test classification with Tier 1 fast contract unit tests, Tier 2 offline integration fixtures, and Tier 3 host runtime probes.
Prove lean System One wire protocol compliance without third-party SDK dependencies per `references/jev-provider-options.md`.
Enforce per-class confidence thresholds and >=0.10 probability margins with fail-open baseline fallback.
Reduce scenario verdicts with the strict failure-first lattice where FAIL beats BLOCKED, BLOCKED beats UNTESTED, and UNTESTED beats PASS.

## Evidence

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,15 @@
## Sub-features
- Single-operation SemanticDecisionProvider contract
- Strict egress boundary validation and SafeContext enforcement
- Default-OFF feature gate and fail-open exception handling
- Lean native System One wire protocol over standard library urllib
- Provider profile configuration across opencode-console, opencode-zen, openrouter, and typesafe-direct
- Default-OFF feature gate, paid provider consent gate, and fail-open exception handling

## How to get to it (user POV)
Inspect `scripts/decision_adapter.py` and run `pytest tests/test_decision_adapter.py`.
Inspect `scripts/decision_adapter.py` and `references/jev-provider-options.md`. Run `pytest tests/test_decision_adapter.py` and `pytest tests/test_jev_transport_and_profiles.py`.

## Driving it with Codex
Execute `uv run pytest tests/test_decision_adapter.py tests/test_shadow_routing.py`.
Execute `uv run pytest tests/test_decision_adapter.py tests/test_shadow_routing.py tests/test_jev_transport_and_profiles.py`.

## Gotchas
Provider failures, timeouts, or invalid requests must never raise uncaught exceptions to the caller.
Provider failures, network timeouts, unapproved paid profiles, or invalid responses must never raise uncaught exceptions to the caller. Fail open cleanly to baseline.
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,16 @@

## Sub-features
- Rerunnable offline routing eval corpus and calibration lever
- Class-specific confidence thresholds and minimum probability margin requirement
- Uncalibrated class fencing and graceful fallback to baseline routing
- Revision-bound evidence receipts and mechanical staleness detection
- Blinded behavioral evaluation comparing baseline and shadow candidates

## How to get to it (user POV)
Run `python3 scripts/eval_decision_routing.py` and `python3 scripts/eval_blinded_pstack.py`.
Run `python3 scripts/eval_decision_routing.py` and `python3 scripts/eval_blinded_pstack.py`. Inspect `scripts/route_resolver.py` and run `pytest tests/test_class_calibration_policy.py`.

## Driving it with Codex
Execute `python3 scripts/eval_decision_routing.py --format json` and verify verdict is PASS.
Execute `python3 scripts/eval_decision_routing.py --format json` and `uv run pytest tests/test_class_calibration_policy.py`.

## Gotchas
Receipts lacking complete revision bindings or having unknown model identity cannot authorize control.
Receipts lacking complete revision bindings, having uncalibrated classes, or failing margin requirements cannot authorize control. Baseline routing remains active.
19 changes: 16 additions & 3 deletions .codex/skills/verify-decision-contract/scripts/driver.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,34 @@
set -euo pipefail

TIMEOUT_SECONDS=60
export PSTACK_TRANSCRIPTS_DIR="${PSTACK_TRANSCRIPTS_DIR:-/tmp/empty-transcripts}"
mkdir -p "${PSTACK_TRANSCRIPTS_DIR}"

if [ -x ".venv/bin/pytest" ]; then
PYTEST_CMD=".venv/bin/pytest"
else
PYTEST_CMD="python3 -m pytest"
fi

run_bounded() {
local label="$1"
shift
echo "==> Running $label (bounded by ${TIMEOUT_SECONDS}s)..."
if command -v timeout >/dev/null 2>&1; then
timeout "${TIMEOUT_SECONDS}" "$@"
timeout --kill-after=5s "${TIMEOUT_SECONDS}" "$@" < /dev/null
else
"$@"
"$@" < /dev/null
fi
}

cmd="${1:-doctor}"
case "$cmd" in
doctor)
run_bounded "openspec-validate-change" openspec validate pstack-jev-decision-plane --type change --strict
if [ -d "openspec/changes/pstack-jev-decision-plane" ]; then
run_bounded "openspec-validate-change" openspec validate pstack-jev-decision-plane --type change --strict
fi
run_bounded "openspec-validate-spec" openspec validate pstack-jev-decisions --type spec --strict
run_bounded "test-decision-contract" $PYTEST_CMD tests/test_verify_decision_contract.py tests/test_jev_transport_and_profiles.py tests/test_class_calibration_policy.py
;;
launch)
echo "Launch verified"
Expand All @@ -29,6 +40,8 @@ case "$cmd" in
run_bounded "eval-decision-routing" python3 scripts/eval_decision_routing.py --format json
run_bounded "eval-blinded-pstack" python3 scripts/eval_blinded_pstack.py --format json
run_bounded "verify-rollback-equivalence" python3 scripts/verify_rollback_equivalence.py --format json
run_bounded "jev-transport-and-profiles" $PYTEST_CMD tests/test_jev_transport_and_profiles.py
run_bounded "class-calibration-policy" $PYTEST_CMD tests/test_class_calibration_policy.py
;;
cleanup)
echo "Cleanup completed"
Expand Down
6 changes: 6 additions & 0 deletions scripts/verify-decision-contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -257,6 +257,12 @@ def inspect_requirements(text: str, label: str) -> None:
def compare_specs(root: Path) -> str:
formal_path = root / "openspec" / "specs" / "pstack-jev-decisions" / "spec.md"
delta_path = root / "openspec" / "changes" / "pstack-jev-decision-plane" / "specs" / "pstack-jev-decisions" / "spec.md"
if not delta_path.is_file():
archived_matches = sorted(
root.glob("openspec/changes/archive/*pstack-jev-decision-plane/specs/pstack-jev-decisions/spec.md")
)
if archived_matches:
delta_path = archived_matches[0]
try:
formal = formal_path.read_text(encoding="utf-8")
delta = delta_path.read_text(encoding="utf-8")
Expand Down
11 changes: 11 additions & 0 deletions tests/test_verify_decision_contract.py
Original file line number Diff line number Diff line change
Expand Up @@ -303,3 +303,14 @@ def test_missing_scenario_fails(tmp_path: Path) -> None:
formal.write_text(SPEC.split("#### Scenario:", 1)[0], encoding="utf-8")
result, output = run_cli(repo, commit_path, tree_path)
assert_failure(result, output, "Requirement has no Scenario")


def test_archived_spec_fallback_passes(tmp_path: Path) -> None:
repo, commit_path, tree_path = make_repo(tmp_path)
active_delta = repo / "openspec" / "changes" / "pstack-jev-decision-plane"
archive_dir = repo / "openspec" / "changes" / "archive" / "2026-10-13-pstack-jev-decision-plane"
archive_dir.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(active_delta), str(archive_dir))
result, output = run_cli(repo, commit_path, tree_path)
assert result.returncode == 0
assert output["verdict"] == "PASS"
Loading