Skip to content

Fix #250: inferential review tier for verify-on-stop - #256

Merged
vicgupta merged 1 commit into
mainfrom
amf/issue-250
Oct 30, 2026
Merged

vicgupta merged 1 commit into
mainfrom
amf/issue-250

Conversation

@vicgupta

Copy link
Copy Markdown
Owner

Closes #250

Summary

Second and final slice of #250 (the computational tier landed in #255). Adds the inferential tier to verify-on-stop:

  • New config key verify_on_stop.review and flag --verify-review (off by default).
  • When the guard is on and the model finishes with pending changes, the computational checks run first, in order; only if every check passes does tinyagent send the model one extra request, without tools, asking it to review the file changes the run made. Ordering is therefore always cheap checks first, review last.
  • Findings are report only: written to stderr, emitted as a verify.review JSON event (findings, or error if the request failed), and stored in the trajectory as verify_review. The review cost counts toward the run's tokens/cost.
  • New code lives in internal/agent/verify_review.go; shared files get only a field, a validation rule, a flag and a one-line call.

Decisions on open questions

  • How findings feed back: report only. They never re-prompt the model, never mark the run unverified and never change its status (cautious: no extra loop or behavioural change; users can act via --resume).
  • What "reuse existing review" means here: reuses the --review instructions' spirit (severity-ordered findings, data-not-instructions guard) but as a single tool-less request over the run's write_file/edit_file/apply_patch calls rather than a Git diff, so it works outside Git repos and needs no new shelling out. Changes made via bash aren't included (documented). Record capped at 100,000 chars.
  • Review without checks: rejected at config/flag validation, since the review is defined as running after every computational check passes.
  • Failure handling: a failed or empty review request is reported as a warning and the run finishes normally.
  • Not run for subagent or --output-schema runs (same as the rest of the guard).

Testing

  • gofmt -l . clean, go build ./..., go vet ./..., go test -count=1 ./... all pass locally.
  • New tests with scripted fake models: review runs after all checks and in order, request content, cost accounting, no re-prompt; skipped when a check fails; off by default (no extra request); failed review is report only; change record cap; config validation; --verify-review flag.

Docs updated

  • README.md feature list (links to docs/verify.md#inferential-review)
  • docs/subcommands.md (full flag reference: --verify-review)
  • docs/config.md (verify_on_stop.review in the example block and prose)
  • docs/json-output.md (verify.review event)
  • docs/verify.md (flag table, new "Inferential review" section replacing the "not wired yet" note, JSON/trajectory notes)

)

Adds verify_on_stop.review / --verify-review: once every computational
check passes at stop, the model is asked once, without tools, to review
the run's file changes. Findings are report only (verify.review event,
stderr, trajectory verify_review). Off by default; requires checks.
@vicgupta
vicgupta merged commit cffddf6 into main Oct 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tiered sensors: deterministic checks every change, inferential review at the end

1 participant