Skip to content

Add interleaved control capture to the guided workflow - #323

Merged
dipeshbabu merged 1 commit into
mainfrom
codex/interleaved-control-capture
Nov 14, 2026
Merged

dipeshbabu merged 1 commit into
mainfrom
codex/interleaved-control-capture

Conversation

@dipeshbabu

Copy link
Copy Markdown
Owner

Why

A guided comparison runs every baseline attempt before every candidate attempt. Cache warm-up, host load or provider drift can then look like an improvement. docs/ADOPTION_GAPS.md lists this as the causal-performance gap.

What

  • agentloop improve capture --control-arg ARG ... (repeat once per argument) runs an unchanged control runner alongside the candidate. Each planned attempt runs both back to back; the control goes first on even attempt indexes and second on odd ones, and every receipt records its order.
  • The control capture is written to <out>/control/, and its manifest hash is recorded in the candidate capture.json.
  • improve finish detects the control and measures the candidate against it. Findings, focus and gates stay frozen from the original baseline. The control must pass the same quality checks, and intervention records link the control's matching findings, as build_intervention requires.
  • Reports state the comparison basis (comparison_baseline, control_capture_hash, control_runner_hash). Each case records control_ran_first and frozen_baseline_status.
  • Captures without a control behave exactly as before. The capture loop was split into _new_manifest and _attempt helpers so both conditions share it.

Tests

  • New tests check:
    • alternating order and timing;
    • control-based comparison and intervention linkage;
    • a control that fails quality checks;
    • a tampered control manifest;
    • repeated --control-arg on the CLI.
  • tests/test_improvement.py: 24 passed.
  • Related suites (guided validation, init, docs site, published README, interventions, CI command): 57 passed, 1 skipped.
  • ruff check and ruff format --check are clean.

🤖 Generated with Claude Code

improve capture --control-arg runs an unchanged control runner alongside
the candidate, alternating which condition runs first on each attempt.
improve finish then measures the candidate against that control while
keeping findings, focus and gates frozen from the original baseline, so
cache warm-up and host drift affect both conditions instead of favouring
the later candidate run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 14, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.67%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.67% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_f2d0b2f0c1b746d0 runs/research_agent_baseline.json yes agentloop_demo
candidate run_b331602b3c2946d2 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.67%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_829535ba1854: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

@dipeshbabu
dipeshbabu merged commit 6474dca into main Nov 14, 2026
25 checks passed
@dipeshbabu
dipeshbabu deleted the codex/interleaved-control-capture branch November 14, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant