Skip to content

Add an aggregate decision for noisy guided comparisons - #329

Merged
dipeshbabu merged 1 commit into
mainfrom
codex/aggregate-decision
Nov 17, 2026
Merged

dipeshbabu merged 1 commit into
mainfrom
codex/aggregate-decision

Conversation

@dipeshbabu

Copy link
Copy Markdown
Owner

Why

improve finish rejects a change if any single pair is slower. Agents that call hosted models vary from run to run. In the first Codex run (docs/CODEX.md), a change that cut tokens in most pairs was rejected because two pairs were slower. Real agent changes will often be rejected by chance under that rule.

What

  • agentloop improve start --decision aggregate (default per-pair). The decision is stored in the hashed session, so it cannot be changed after results are known. Sessions from older versions keep per-pair judgment.
  • Still strict on every pair: completion, quality on both sides, complete analysis, and every replay gate that is not about speed or cost (retry, quality fixtures and so on).
  • Judged across all pairs: the latency and cost gates. The aggregate passes when:
    • the median paired improvement in the objective is positive and at least --minimum-improvement;
    • more pairs improve than regress;
    • the other metric's median regression is within its allowed regression, when every pair can measure it.
  • report.json gains a decision object with counts, the median, reasons and a one-sided exact sign-test p-value, which is reported but does not gate. Markdown, HTML, plan.md and the CLI summarize it. Per-pair failures are still visible in each case's gate table.
  • Per-pair mode is unchanged: same reasons, same order, and decision: {mode: per_pair, passed: true} is added to the report.
  • Docs: a new "Judge noisy agents across all pairs" section in GUIDED_OPTIMIZATION.md, plus a CHANGELOG entry.

Tests

  • Seven new tests:
    • a noisy fixture where per-pair rejects and aggregate accepts (3 improved, 1 regressed, p = 5/16);
    • aggregate still rejects wrong answers;
    • aggregate rejects a median slowdown;
    • the decision cannot be edited after freezing;
    • unknown decisions are rejected;
    • the CLI freezes the decision.
  • test_improvement.py, guided validation, docs site, published README and Codex runner tests: 61 passed, 1 skipped. Pre-commit is clean.

🤖 Generated with Claude Code

improve start --decision aggregate freezes a rule that judges speed or
cost across all pairs: the median paired improvement must meet the
frozen minimum, more pairs must improve than regress, and the other
metric's median regression must stay within its allowance. Completion,
quality and non-performance gates stay strict on every pair. Reports add
the counts and a non-gating one-sided sign-test p-value. The default
per-pair decision and its messages are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 17, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.67%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.67% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_43d8d3dd46bc4635 runs/research_agent_baseline.json yes agentloop_demo
candidate run_3b3666cb84e34882 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.67%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_ad853afff937: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu merged commit f75747c into main Nov 17, 2026
25 checks passed
@dipeshbabu
dipeshbabu deleted the codex/aggregate-decision branch November 17, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant