Skip to content

Add a runner wall-time gate to guided comparisons - #337

Merged
dipeshbabu merged 1 commit into
mainfrom
codex/guided-runner-regression-gate
Nov 18, 2026
Merged

dipeshbabu merged 1 commit into
mainfrom
codex/guided-runner-regression-gate

Conversation

@dipeshbabu

@dipeshbabu dipeshbabu commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Problem

In a retained offline retrieval comparison, the candidate passed the trace-latency gate even though the median paired runner time rose by 24 ms. Developers need an optional guard for this measurement when total process time matters to their workload.

Changes

  • improve start --max-runner-regression-ms N freezes the allowed wall-time increase for each candidate pair. The comparison uses the interleaved control when supplied.
  • Both decision modes require the configured runner gate on every pair. The trace objective and existing quality checks still apply. Both completed attempts need monotonic runner timings, and legacy timestamps remain visible in the timing report.
  • Record the limit, delta, and rejection reason in CLI and report output. Linked interventions retain failed runner gates.
  • Document the option and exercise it through the installed wheel smoke.

Validation

  • Focused tests cover both decision modes, exact limit boundaries, interleaved-control source selection, legacy timing, and rejection in linked interventions.
  • .venv/Scripts/python.exe -m pytest tests/test_improvement.py tests/test_guided_verifier.py -q --basetemp D:/agentloop/runs/runner-regression-focused --tb=short — 41 passed.
  • .venv/Scripts/python.exe runs/bugfix_quality_checks.py — all applicable hooks passed.
  • uv build --out-dir runs/runner-regression-dist --quiet and Twine checks passed for the wheel and sdist. runs/coding-agent-local-pilot/venv/Scripts/python.exe -I scripts/smoke_installed_wheel.py --expected-version 0.2.3 passed from the locally built wheel.
  • venv/Scripts/python.exe -I pilot.py from D:/agentloop/runs/coding-agent-local-pilot used the locally built wheel and separate standard-library application tests. The equivalent refactor passed all four paired attempts with one scan instead of three per run. Deliberate cleanup delay made the median paired runner time 1.22 seconds slower while the trace was faster, and all four runner gates rejected it. The broken refactor passed the built-in output-shape check and failed all four application verifications. Reports and original checksums are retained locally.
  • All runnable hosted checks passed on d2f92e39bf0bbd039107f6d3da79e0f2a723d33a before squash merge. Both CI and smoke validation use owned offline workloads.
  • .venv/Scripts/python.exe -m pytest -q --basetemp D:/agentloop/runs/runner-regression-full --tb=short — 2,822 passed, 109 skipped.

Compatibility

The gate is opt-in. Existing sessions and reports remain readable, and existing ungated decisions continue to use their frozen checks. For a gated comparison with an older baseline, a fresh interleaved control supplies the required monotonic times.

Substantial implementation and validation assistance was provided by Codex.

@coderabbitai

coderabbitai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.68%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.68% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_fd1fb430deda4bf7 runs/research_agent_baseline.json yes agentloop_demo
candidate run_4d176bf7ecce4721 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.68%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_54db5688581a: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu marked this pull request as ready for review November 18, 2026 18:47
@dipeshbabu
dipeshbabu merged commit 447ea84 into main Nov 18, 2026
24 checks passed
@dipeshbabu
dipeshbabu deleted the codex/guided-runner-regression-gate branch November 18, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant