Skip to content

Warn when a guided comparison plan is too weak to detect a change - #344

Open
dipeshbabu wants to merge 2 commits into
mainfrom
feat/planning-guidance
Open

dipeshbabu wants to merge 2 commits into
mainfrom
feat/planning-guidance

Conversation

@dipeshbabu

Copy link
Copy Markdown
Owner

`improve start` and `improve compare` now warn before the candidate runs when the plan cannot separate a change from noise:

  • fewer than 5 pairs: even if every pair improves, a one-sided sign test cannot fall below p = 0.05;
  • 1 repetition per task: run-to-run variation cannot be measured;
  • noisy baseline latency: repetitions of the same task varied by a median of at least 10%, more than `--minimum-improvement`.

The notes are recorded in the frozen session (`planning_notes`), printed by the CLI and `compare` progress, and listed under Evidence strength in `plan.md`. They do not change decisions.

Tests cover each note, the quiet cases, the session/plan output and compare progress. Docs and changelog updated.

🤖 Generated with Claude Code

improve start and improve compare now note, before the candidate runs,
when fewer than five pairs are planned (a unanimous sign test cannot
reach p < 0.05), when each task runs once (noise is unmeasurable), or
when baseline latency varies between repetitions by at least 10% and
more than the required improvement. Notes are frozen in the session,
printed by the CLI and listed under Evidence strength in plan.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.68%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.68% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_b8331041bd9d4f9a runs/research_agent_baseline.json yes agentloop_demo
candidate run_ecd90c5c79c74fbb runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.68%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_08fba8d7ba01: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant