Repository navigation
Part of #250: computational checks for verify-on-stop - #255
Merged
Merged
Conversation
….checks, --verify-check)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #250
Summary
Adds the computational tier of verify-on-stop sensors.
verify_on_stop.checks(config) or--verify-check <cmd>(repeatable, replaces the config list) lists fast, deterministic checks such as linters, type checkers, structural checks and tests (pass = exit 0).When the model gives a final answer while changes are pending, tinyagent runs the checks itself, in the listed order (cheapest first), as
bashtool calls so the permission policy, hooks, lockdown and sandbox all apply. It stops at the first failure:max_reprompts.Each check also counts as a verification command when the model runs it itself. Each check run emits a new
verify.checkJSON event (index,command,passed). New code is ininternal/agent/verify_checks.go;loop.goonly passesctxthrough toverifyStop.Not in this slice
The inferential tier (run the existing review once at the end, after all checks pass) is not wired in yet; a comment on #250 lists it as the remaining work.
Testing
gofmt -lclean;go build ./...,go vet ./...andgo test -count=1 ./...pass. New tests use the scripted fake model: all checks pass, stop at the first failure with output in the re-prompt and later checks skipped, failure then pass after a model edit, a check run by the model counts as verification, a refused check marks the run unverified, off when the guard is off, config load/validation and flag override.Docs updated
--verify-checkin the full flag referenceverify_on_stop.checksin the example config and the Verify before stop sectionverify.checkeventDecisions on open questions
commandsdoes today.checksempty orverify_on_stopoff, behaviour is unchanged. Checks don't run for subagent or--output-schemaruns, matching the existing guard.