Skip to content

Fix review findings in repetition detours and the Codex runner - #332

Merged
dipeshbabu merged 1 commit into
mainfrom
codex/review-fixes
Nov 17, 2026
Merged

dipeshbabu merged 1 commit into
mainfrom
codex/review-fixes

Conversation

@dipeshbabu

Copy link
Copy Markdown
Owner

Findings from a high-effort code review of v0.2.2..main.

Fixed

  1. Failed attempts distorted detour comparisons. repetition_detour used every attempt that left a trace, so a failed or interrupted attempt's partial trace could shift the typical repetition, or be flagged itself. start and finish (for controls) now pass only completed attempts.
  2. Fractional titles. Medians of an even number of repetitions can be fractional, which produced titles like "0.5 more model steps". Titles now state the observed and typical counts, for example "took 3 model steps (typically 2) and 2 tool calls (typically 1)".
  3. Stale diagnosis counts. ranking.ready_to_test_count was not recomputed after detour findings were merged. It now is.
  4. Repeated scans. Each trace was rescanned for every other repetition (O(n²)). Each trace is now profiled once.
  5. Codex runner rounds. A tool item reported only on completion, such as a file change, left the model round open, and later extensions measured duration from a moved boundary. The runner now closes the round there and measures each round from its own start.
  6. --decision spelling. An invalid value named per_pair although the CLI documents per-pair. start now accepts both, and the error names per-pair.

Skipped

In aggregate mode, slower pairs keep gates_passed: false in their intervention records. Each record honestly measures its own pair. The overall verdict lives in the report's decision.

Tests

  • New tests:
    • failed attempts are left out of comparisons;
    • fractional typical counts are stated, not subtracted;
    • merged diagnosis counts match their findings;
    • a tool reported only on completion ends the round and durations match the timestamps.
  • test_repetitions.py, test_codex_exec_runner.py and test_improvement.py: 54 passed.
  • Re-checked on the retained Codex traces: the same detours are flagged, with the new titles.

🤖 Generated with Claude Code

- Compare only completed attempts between repetitions, so a failed or
  interrupted attempt's partial trace cannot distort the typical one.
- State observed and typical step counts instead of fractional
  differences, and keep diagnosis ranking counts in step with findings.
- Profile each trace once instead of rescanning every repetition.
- Close a Codex model round when a tool arrives only on completion and
  measure rounds from their own start.
- Accept per-pair in improve start and name that spelling in errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 17, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.67%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.67% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_7d0ac1febe8447b9 runs/research_agent_baseline.json yes agentloop_demo
candidate run_80d42742eb5f44ce runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.67%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_19586fd96753: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu merged commit 944c70b into main Nov 17, 2026
25 checks passed
@dipeshbabu
dipeshbabu deleted the codex/review-fixes branch November 17, 2026 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant