Skip to content

Require optional local verifiers for guided comparisons - #336

Merged
dipeshbabu merged 2 commits into
mainfrom
codex/guided-local-verifier
Nov 18, 2026
Merged

dipeshbabu merged 2 commits into
mainfrom
codex/guided-local-verifier

Conversation

@dipeshbabu

@dipeshbabu dipeshbabu commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Problem

Guided comparisons can check answer shape and values using built-in scorers. Developers also need to apply existing application tests or source checks to the same fixed tasks. A candidate can keep the expected answer while changing a required citation or other behavior.

Changes

  • improve start --verifier FILE checks the baseline and freezes the trusted entry script's SHA-256 and timeout.
  • improve finish --verifier FILE checks the selected source against the session, then runs the verifier for the candidate and its baseline or interleaved control. Both the verifier and built-in scorer must pass.
  • Run the exact checked Python bytes in a separate process. Give the verifier a bounded task attempt and its recorded output on stdin. Preserve verifications, nonzero exits, timeouts, and unavailable output in the plan and comparison.
  • Extend the existing quality report and replay path with a second quality fixture. Keep verification outside measured runner time.
  • Document usage, including the difference between the entry-script hash and the application's own dependency and fixture versions.

Validation

  • D:/agentloop/.venv/Scripts/python.exe -m pytest tests/test_guided_verifier.py -q --basetemp D:/agentloop/runs/guided-verifier-focused --tb=short — 5 passed. A faster but incorrectly cited candidate passed the built-in answer check and failed the verifier. Other cases cover passing interleaved results, source mutation, saved-session arguments, a timed-out subprocess, and unavailable output.
  • uv build --out-dir D:/agentloop/runs/guided-verifier-dist --quiet and Twine — integrated wheel and sdist passed.
  • D:/agentloop/.venv/Scripts/python.exe D:/agentloop/runs/bugfix_quality_checks.py — all applicable hooks passed after integrating the reviewed repeat-tool advice changes.
  • D:/agentloop/.venv/Scripts/python.exe -m mkdocs build --strict --site-dir D:/agentloop/runs/guided-verifier-site — passed.
  • D:/agentloop/runs/tool-verifier-pilot/venv/Scripts/python.exe -I scripts/smoke_installed_wheel.py --expected-version 0.2.3 — full installed CLI, verifier, and evidence smoke passed.
  • venv/Scripts/python.exe -I pilot.py in D:/agentloop/runs/tool-verifier-pilot — installed wheel selected the reuse finding on the eight baseline attempts. All eight interleaved correct pairs passed, with 24 to 8 tool reads and a 67.8% median paired trace improvement on this fixture. All eight wrong-citation pairs passed the built-in shape check and failed independent verification, despite shorter traced times. Reports preserve the rejection.
  • D:/agentloop/.venv/Scripts/python.exe -m pytest tests/test_guided_verifier.py tests/test_improvement.py tests/test_improvement_init.py tests/test_parallelization_evidence.py -q --basetemp D:/agentloop/runs/guided-verifier-integrated --tb=short — 85 passed on the integrated source.
  • D:/agentloop/.venv/Scripts/python.exe -m pytest tests/test_html_report.py tests/test_guided_validation.py -q --basetemp D:/agentloop/runs/verifier-postmerge-golden --tb=short — 15 passed, 1 skipped on the integrated source.
  • All runnable hosted checks passed on final head a4eb378779b1061cf25e562700a67156611b2da9, including all four Python suites, native wheel smoke, Docker, CodeQL and docs build.
  • A local full run had 2,812 passes, 109 skips, and one snapshot mismatch. I merged the recommendation branch into the worktree while that run was active, so it compared the old renderer already loaded in memory against the new fixture. The integrated HTML run passed. The exact integrated commit then passed the hosted full matrix before squash merge.

Compatibility and limits

The feature is opt-in. Existing task files, sessions, replay and trace schemas remain valid. A saved session requires an explicit matching CLI selection for verification. Only the entry script and timeout are frozen. The developer supplies a trusted script, versioned test fixtures, and dependencies. The offline exercise uses an owned retrieval workload.

Substantial implementation and validation assistance was provided by Codex.

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.68%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.68% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_9e5c976d403641a7 runs/research_agent_baseline.json yes agentloop_demo
candidate run_c31daa79d1994099 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.68%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_dfc1abd1a478: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu marked this pull request as ready for review November 18, 2026 05:32
@dipeshbabu
dipeshbabu merged commit 3eee210 into main Nov 18, 2026
24 checks passed
@dipeshbabu
dipeshbabu deleted the codex/guided-local-verifier branch November 18, 2026 05:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant