Skip to content

Part of #250: computational checks for verify-on-stop - #255

Merged
vicgupta merged 2 commits into
mainfrom
amf/issue-250
Oct 30, 2026
Merged

vicgupta merged 2 commits into
mainfrom
amf/issue-250

Conversation

@vicgupta

Copy link
Copy Markdown
Owner

Part of #250

Summary

Adds the computational tier of verify-on-stop sensors. verify_on_stop.checks (config) or --verify-check <cmd> (repeatable, replaces the config list) lists fast, deterministic checks such as linters, type checkers, structural checks and tests (pass = exit 0).

When the model gives a final answer while changes are pending, tinyagent runs the checks itself, in the listed order (cheapest first), as bash tool calls so the permission policy, hooks, lockdown and sandbox all apply. It stops at the first failure:

  • All pass: pending changes clear and the run finishes verified, with no re-prompt.
  • One fails: the model is re-prompted with the failing check's position, command and output; this counts against max_reprompts.
  • One is refused: the run finishes unverified straight away.

Each check also counts as a verification command when the model runs it itself. Each check run emits a new verify.check JSON event (index, command, passed). New code is in internal/agent/verify_checks.go; loop.go only passes ctx through to verifyStop.

Not in this slice

The inferential tier (run the existing review once at the end, after all checks pass) is not wired in yet; a comment on #250 lists it as the remaining work.

Testing

gofmt -l clean; go build ./..., go vet ./... and go test -count=1 ./... pass. New tests use the scripted fake model: all checks pass, stop at the first failure with output in the re-prompt and later checks skipped, failure then pass after a model edit, a check run by the model counts as verification, a refused check marks the run unverified, off when the guard is off, config load/validation and flag override.

Docs updated

  • README.md feature list (Verify before stop entry, linking to docs/verify.md#computational-checks)
  • docs/verify.md: flag table plus a new "Computational checks" section
  • docs/subcommands.md: --verify-check in the full flag reference
  • docs/config.md: verify_on_stop.checks in the example config and the Verify before stop section
  • docs/json-output.md: verify.check event

Decisions on open questions

  • Checks run at stop time only (when changes are pending), not automatically after every edit, to keep runs cheap; the model can run them after edits itself, and that counts as verification.
  • Order is the user's listed order (documented as "cheapest first") rather than any automatic cost estimate.
  • Checks go through the normal bash tool path, so they can't bypass permissions, hooks, lockdown or the sandbox; a refused check finishes the run unverified instead of looping.
  • Configuring checks replaces the built-in test/build patterns, same as commands does today.
  • Off by default: with checks empty or verify_on_stop off, behaviour is unchanged. Checks don't run for subagent or --output-schema runs, matching the existing guard.

@vicgupta
vicgupta merged commit b14834d into main Oct 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant