Test shared effect semantics without treating agreement as correctness, refusal as useful completion, or a description as permission.
Release series v0.4.0-draft.1. Public development benchmark and evaluation-protocol
proposal. Not a leaderboard, external certification or completed protected study.
Part of Ambit. Action Effect Bench is the conformance and evaluation suite for the effects layer of Ambit, an open protocol for describing, authorizing and proving agent actions. It tests Action Effect Spec implementations and needs no OSuite server or account.
This repository is separate from Action Effect Spec. The runnable part retains the original two offline refund-profile consumers and adds a bounded GitHub-comment pilot: JavaScript production, a separately written Python checker, actual OPA authorization, official MCP transport/Octokit, a local occurrence ledger and read-only provider reconciliation. No OSuite server or account is required. It is not a general-purpose production gateway.
The post-0.4 development work makes the first bounded profile easier to inspect and implement without changing its wire identity. Start with the implementer handoff, the explicit report-contract preflight, and the execution scope. The completed six-task follow-up reports five contract-ready finals, one retained transport failure and increased cost, rather than presenting a perfect or faster result. Separate internal content review finds two reports still need factual-detail additions.
The preflight is an opt-in field-study edition. It retains the six exposed tasks and 6000-byte report limit, counts draft revisions and their cost, and checks the actual final report rather than crediting a different earlier draft. It grants no permission and is not semantic review. Sixteen added compound regressions cover revocation, lost acknowledgements, ambiguous recovery and policy errors in both lanes/strategies, using real local components with a synthetic provider, not additional customer or SaaS experiments.
The 0.4 release, old results and manuscript remain unchanged. External implementation, protected evaluation, real recipient acceptance and stable profile status remain separate gates, not accomplishments implied by this work.
The field-use evidence records real model-led maintenance work through public GitHub REST and the official GitHub MCP server, separate from the older scripted mutation experiment. It retains setup failures, an incorrect report conclusion, a conflicting first-run output contract and post-review repairs. Loop completion is not silently counted as useful output.
- Full technical manuscript and PDF.
- Read-only field-use guide: setup, scope and replay.
- Field-use results and limits.
- Owned native delivery and recovery: two eligible model reports delivered, six separate fault cases, twenty fresh worker processes, five comment calls and eight no-send duplicate probes.
- Live public evidence and versioned no-send replay guide.
- Original 0.3 release: unchanged read-only runs and 39-page manuscript.
- Original 0.2 release: unchanged evidence and short paper.
Offline replay needs neither Azure access nor GitHub credentials. A new model run requires explicitly configured credentials and incurs model costs. These observations are internal self-use, not external certification, arbitrary autonomous-write qualification or evidence that a correct adapter needs this wire format. Four of the six amended model outputs still fail the task budget; successful delivery of two does not reclassify the other four.
- Quickstart: install pinned tools, run locally, and distinguish safe replay from an explicitly authorized live write.
- Runtime guide: two strategies, scope and trust boundaries.
- Results: measured denominators, failures, cost and public provider readback links.
- Working paper: research question, protocol and evidence limits.
- Independent evaluation pathway: custody and separate input/label handling, not a claim that an external study occurred.
The new author-defined corpus has 55 rows, 22 base intents and 108 lane instances. Runtime cases are frozen before candidate execution/results, not before implementation. The structured baseline uses the same Python projection, OPA and dispatch controls without transferring an effect-description envelope. Protocol overhead is reported, not assumed to be a performance win.
Use Node.js 22+ and Python 3.9+ on PATH; supported current releases are
recommended. No dependency installation, OSuite account, credentials or cloud
resource is needed. The qualified CI targets are Linux and macOS, not Windows.
In a new directory:
git clone https://github.com/OndCo/Action-Effect-Bench.git action-effect-bench
git clone https://github.com/OndCo/Action-Effect-Spec.git action-effect-spec
SPEC_COMMIT=$(node -p "require('./action-effect-bench/spec-lock.json').commit")
git -C action-effect-spec checkout --detach "$SPEC_COMMIT"
node action-effect-bench/conformance/case-source.mjs --check
node --test action-effect-spec/implementations/javascript/*.test.mjs action-effect-bench/conformance/runner.test.mjs action-effect-bench/conformance/entrypoints.test.mjs
python3 -B -W error -m unittest discover -s action-effect-spec/implementations/python/tests
node action-effect-bench/conformance/run.mjsThe explicit lowercase sibling directories matter on case-sensitive filesystems. Do not substitute the latest Spec branch for the revision in spec-lock.json. To repeat a historical Bench version, check out that version first and then read its lock. Make changes only in a separate development copy when comparing a new candidate.
The runner exits nonzero on a mismatch, invalid response, timeout, disagreement or source change. It starts real local consumer processes but never executes a refund. New timestamped JSON/Markdown reports preserve expected fields, raw responses, source hashes, runtime versions, failures and separate denominators. See the runner guide and reports.
| Material | Scope |
|---|---|
| 80 frozen public-development cases | Contract-derived expectations authored before candidate inspection, within the same project. |
| 22 post-review regressions | Explicitly added after defects were found; not independent or held out. |
| Two consumers | Separately written JavaScript/Python implementations, not two independent institutions. |
| Unit and harness tests | Parser, binding, projection, framing and scorer checks; not separate customer workloads. |
| 12-case development design index | Proposed broader scenarios, not 12 additional executed cases. |
Every consumer response has authority: not_evaluated. A successful
accepted_partial result is not a safe-action verdict. These original cases do not
establish live authorization, signature verification, real provider outcomes,
arbitrary program analysis or full CORE-001..028 conformance. Per-process timing
is diagnostic, not an end-to-end latency advantage.
Expected-result matching and cross-implementation agreement are scored separately. Two agreeing defects can still exist. A conforming report must state which fields have an independent expected value; the current accepted-description digest is cross-compared, not supplied by an independent full-document canonical oracle.
An implementation tested with this suite can show one of two badges, each linked to a public claim that follows CORE-028 in Action Effect Spec:
- Ambit tested: you ran the Bench yourself and published the claim and raw report.
- Ambit verified: Ond re-ran the Bench version your claim names against the implementation you identify, and published its own report.
The claim names the Bench version and case hash, the Spec revision and profiles, the implementation identity, and passed, failed and unsupported cases. A limited run is labelled as limited. Neither badge is a certification, a security guarantee or conformance with a future stable release, and Ond does not verify its own products. To be listed or to request a re-run, open an issue in this repository. Rules and badge files are at ond.cc/ambit/verified.
- PROTOCOL.md: separate tracks, competent baselines, failure accounting, custody, preregistration and analysis.
- Case provenance: frozen sources and exposure.
- Collaboration contract: proposed roles and limits.
- CONTRIBUTING.md: synthetic counterexamples, new consumers and independently defined future cohorts.
- PUBLICATION.md: selected release scope and rights.
The original code, documentation and synthetic data in this release use the repository's MIT license. No protected research cases, customer traces or unrelated commercial source are included. Confidential reports go through SECURITY.md, not a public issue.