Repository navigation
Require optional local verifiers for guided comparisons - #336
Merged
Merged
Conversation
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
AgentLoop replay self-testThis required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance. AgentLoop CI Report
Performance Gates
Trace inputsThis comparison includes synthetic data. It does not establish application performance gains from a pull request.
Metric Impact
Top Findings
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Guided comparisons can check answer shape and values using built-in scorers. Developers also need to apply existing application tests or source checks to the same fixed tasks. A candidate can keep the expected answer while changing a required citation or other behavior.
Changes
improve start --verifier FILEchecks the baseline and freezes the trusted entry script's SHA-256 and timeout.improve finish --verifier FILEchecks the selected source against the session, then runs the verifier for the candidate and its baseline or interleaved control. Both the verifier and built-in scorer must pass.Validation
D:/agentloop/.venv/Scripts/python.exe -m pytest tests/test_guided_verifier.py -q --basetemp D:/agentloop/runs/guided-verifier-focused --tb=short— 5 passed. A faster but incorrectly cited candidate passed the built-in answer check and failed the verifier. Other cases cover passing interleaved results, source mutation, saved-session arguments, a timed-out subprocess, and unavailable output.uv build --out-dir D:/agentloop/runs/guided-verifier-dist --quietand Twine — integrated wheel and sdist passed.D:/agentloop/.venv/Scripts/python.exe D:/agentloop/runs/bugfix_quality_checks.py— all applicable hooks passed after integrating the reviewed repeat-tool advice changes.D:/agentloop/.venv/Scripts/python.exe -m mkdocs build --strict --site-dir D:/agentloop/runs/guided-verifier-site— passed.D:/agentloop/runs/tool-verifier-pilot/venv/Scripts/python.exe -I scripts/smoke_installed_wheel.py --expected-version 0.2.3— full installed CLI, verifier, and evidence smoke passed.venv/Scripts/python.exe -I pilot.pyinD:/agentloop/runs/tool-verifier-pilot— installed wheel selected the reuse finding on the eight baseline attempts. All eight interleaved correct pairs passed, with 24 to 8 tool reads and a 67.8% median paired trace improvement on this fixture. All eight wrong-citation pairs passed the built-in shape check and failed independent verification, despite shorter traced times. Reports preserve the rejection.D:/agentloop/.venv/Scripts/python.exe -m pytest tests/test_guided_verifier.py tests/test_improvement.py tests/test_improvement_init.py tests/test_parallelization_evidence.py -q --basetemp D:/agentloop/runs/guided-verifier-integrated --tb=short— 85 passed on the integrated source.D:/agentloop/.venv/Scripts/python.exe -m pytest tests/test_html_report.py tests/test_guided_validation.py -q --basetemp D:/agentloop/runs/verifier-postmerge-golden --tb=short— 15 passed, 1 skipped on the integrated source.a4eb378779b1061cf25e562700a67156611b2da9, including all four Python suites, native wheel smoke, Docker, CodeQL and docs build.Compatibility and limits
The feature is opt-in. Existing task files, sessions, replay and trace schemas remain valid. A saved session requires an explicit matching CLI selection for verification. Only the entry script and timeout are frozen. The developer supplies a trusted script, versioned test fixtures, and dependencies. The offline exercise uses an owned retrieval workload.
Substantial implementation and validation assistance was provided by Codex.