Skip to content

Add improve compare to run a guided comparison in one command - #339

Merged
dipeshbabu merged 1 commit into
mainfrom
codex/improve-compare
Nov 18, 2026
Merged

dipeshbabu merged 1 commit into
mainfrom
codex/improve-compare

Conversation

@dipeshbabu

Copy link
Copy Markdown
Owner

Why

A fair before/after comparison currently takes four commands (capture, start, capture --control-arg and finish) with matching flags and paths. That is the step most likely to stop a first-time user, and first use by outside developers is the top open gap in docs/ADOPTION_GAPS.md.

What

  • agentloop improve compare --out DIR --change "..." --before-arg ... -- <after runner>:
    1. captures the baseline with the before runner in DIR/baseline;
    2. freezes the session in DIR/session (findings, checks and plan.md);
    3. captures the after runner, alternating with the before runner as an interleaved control, in DIR/paired;
    4. writes the verdict and reports to DIR/results.
  • Input checks first: the change description, decision, objective, minimum improvement, runner-regression limit, verifier and task file are checked before any runner executes, so a typo cannot waste a baseline run. Identical before and after commands are rejected.
  • Early stop: it stops after the baseline when no baseline attempt completed, pointing to the capture and --show-runner-output.
  • Default decision: --decision aggregate, which suits agents whose timing varies. per-pair is still available. The other options match the individual commands.
  • Library API: improvement.compare() exposes the same workflow with a progress callback.
  • Shared printing: finish's result printing moved into _print_report, so both commands print the same verdict and timing table. The output of finish is unchanged.
  • Docs: a new GUIDED_OPTIMIZATION section (with a git worktree example and the trade-off that the plan cannot inform the change), a README pointer and a CHANGELOG entry.

Tests

  • Nine new tests:
    • the full run passes with an interleaved control and the aggregate decision, and every stage's artifacts and progress messages are present;
    • a wrong after runner is rejected;
    • five invalid inputs fail before any directory is created;
    • a broken before runner stops after the baseline;
    • the CLI accepts repeated --before-arg.
  • Improvement, published README, docs site, repetitions and guided validation tests: 77 passed, 1 skipped. Pre-commit is clean.

🤖 Generated with Claude Code

improve compare captures the baseline with the unchanged before runner,
freezes the session, captures the changed runner alternating with the
before runner as an interleaved control, and writes the verdict. It checks
inputs before any runner executes, stops when no baseline attempt
completes, and defaults to the aggregate decision. finish's result printing
moves into a shared helper so both commands print the same verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 18, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.67%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.67% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_ab7942cf13b74ddc runs/research_agent_baseline.json yes agentloop_demo
candidate run_b39e5634b315453e runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.67%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_e7b19c1d355c: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu merged commit 57d3e65 into main Nov 18, 2026
25 checks passed
@dipeshbabu
dipeshbabu deleted the codex/improve-compare branch November 18, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant