Skip to content

Report detours that only some repetitions take - #330

Merged
dipeshbabu merged 1 commit into
mainfrom
codex/repetition-detour
Nov 17, 2026
Merged

dipeshbabu merged 1 commit into
mainfrom
codex/repetition-detour

Conversation

@dipeshbabu

Copy link
Copy Markdown
Owner

Why

In the first Codex run, the costliest waste was a token spike of about 18–20k in some repetitions. AgentLoop's findings judge one trace at a time, so each spike looked normal on its own and nothing was reported.

Investigating it also corrected the diagnosis. Each spike came from an extra model step: Codex ran one tool, called the model again, then ran another tool, and every model step resends about 19k tokens of context. The RTK.md read previously blamed in docs/CODEX.md cost nothing when it ran in the same step as the requested read.

What

  • New agentloop/repetitions.py: compares the repetitions of each task. An attempt gets a repetition_detour finding when:
    • it took more model steps or more tool calls than the median of its other repetitions; and
    • it used at least 25% more input tokens. Runtime is used only when traces record no tokens, so a slow provider alone is not reported.
  • Evidence: the tool calls that appear in no other repetition (with input excerpts) and the intermediate model steps.
  • Savings estimate: the gap to the typical repetition's tokens and runtime. Assumptions are stated, including that this is an association, not a proven cause.
  • Confidence: medium when two or more attempts in the task set take a detour.
  • Integration:
    • findings merge into each attempt's diagnosis and are re-ranked, so they appear in plan.md and can be selected with --focus repetition_detour;
    • interleaved controls are diagnosed the same way, so intervention records can link the finding.
  • Docs: docs/CODEX.md correction, a GUIDED_OPTIMIZATION section and CHANGELOG entries.

Validation on real traces

Run against the 18 retained Codex traces (baseline, control, candidate):

Set Flagged
Baseline none (no spikes)
Control python-support rep 0 (+20,381 tokens), npm-node-engine rep 1 (+19,363)
Candidate npm-node-engine rep 0 (+17,995)

It caught exactly the three spikes and nothing else. An earlier draft also flagged a slower baseline run that had no extra tokens. That is why tokens decide whenever they are recorded.

Tests

  • tests/test_repetitions.py (7):
    • an outlier with an extra step is reported, with its evidence;
    • a slower run without extra tokens is not reported;
    • runtime decides when no tokens are recorded;
    • single repetitions and identical repetitions are not reported;
    • recurring detours raise confidence;
    • findings merge into ranked diagnoses;
    • end to end through capture, start and plan.md.
  • Improvement, guided validation, findings, ranking, docs and Codex runner tests: 150 passed, 3 skipped. Pre-commit is clean.

🤖 Generated with Claude Code

improve start now compares repetitions of each task. An attempt that took
more model steps or tool calls than its typical repetition and used at
least 25% more input tokens (runtime when tokens are unrecorded) gets a
repetition_detour finding listing the extra calls and intermediate model
steps. Interleaved controls are diagnosed the same way so the finding can
be linked in intervention records.

On the 18 retained Codex traces it flags exactly the three runs that took
an extra model step. docs/CODEX.md now attributes those token spikes to the
extra step rather than to an instruction-file read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 17, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.67%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.67% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_4d744dff1ac4452d runs/research_agent_baseline.json yes agentloop_demo
candidate run_3acba6426a2e4f25 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.67%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_feeef4655410: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu merged commit e042ae5 into main Nov 17, 2026
25 checks passed
@dipeshbabu
dipeshbabu deleted the codex/repetition-detour branch November 17, 2026 06:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant