Skip to content

Add a Codex CLI runner for the guided workflow - #324

Merged
dipeshbabu merged 2 commits into
mainfrom
codex/codex-exec-runner
Nov 15, 2026
Merged

dipeshbabu merged 2 commits into
mainfrom
codex/codex-exec-runner

Conversation

@dipeshbabu

@dipeshbabu dipeshbabu commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Why

To let AgentLoop profile and compare OpenAI Codex CLI runs, the first real coding agent tried with the guided workflow outside the Hermes example.

What

  • examples/codex_exec_runner.py: a standard improve capture runner.
    • Runs codex exec --json --ephemeral --sandbox read-only --cd DIR - with the task prompt on stdin.
    • Timestamps each JSONL event on arrival and builds a native trace:
      • command, file-change, MCP and web-search items become tool_call spans; non-zero exit codes are errors;
      • agent_message/reasoning items and silent tool decisions become model_call spans;
      • turn.completed usage goes on the turn's last model span as provider counts (usage_scope: turn_total, cached and reasoning tokens in metadata);
      • turn.failed/error marks the root span as an error.
    • Arguments after a second -- pass to codex exec unchanged, so candidates can vary model, effort or features.
  • examples/codex_repository_tasks.json: three repository metadata lookups with exact expected values.
  • docs/CODEX.md (linked from the nav and INTEGRATIONS.md): the event mapping, usage, and the first observed run.
  • CHANGELOG entry.

First observed run (Codex CLI 0.167.1, Windows)

  • Baseline: 6/6 answers correct, 15.6–29.8 s and about 39k input tokens per one-field lookup. split_large_step was reported on every attempt.
  • Candidate: disabled the apps, browser_use and computer_use features, run against the new interleaved control.
  • Result: all 12 answers correct and 4/6 pairs passed. Input tokens fell about 7% where both conditions took the same path. Both engines.node pairs were slower and failed latency gates, so the change was rejected.
  • Token spikes: the traces show that runs using 16–20k more tokens made an extra shell call to read a user-level instruction file.

Raw traces stay local under runs/ (ignored), not committed.

Tests

  • tests/test_codex_exec_runner.py (8 tests) uses a recorded event shape and needs no Codex or network:
    • span mapping and timing;
    • usage attribution;
    • failed commands and turns;
    • usage without model items;
    • an analyzable round-trip trace;
    • prompt parsing.
  • Docs site tests pass, and ruff is clean on agentloop, examples and tests.

CI fix

The Package artifact and wheel smoke job failed three times in a row with Artifact not found for name: python-package, even though the upload had just succeeded. The download now uses the upload step's artifact-id, and the byte comparison no longer depends on the extracted directory layout. tests/test_release_workflows.py passes.

🤖 Generated with Claude Code

examples/codex_exec_runner.py runs codex exec --json in a read-only
sandbox and converts its event stream into native traces: tool spans
for commands and other tool items, model spans for visible model items
and tool decisions, and provider turn usage on the turn's last model
span. docs/CODEX.md documents the mapping and a first interleaved run,
in which the tool-surface change was rejected on latency.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

AgentLoop replay self-test

This required repository check evaluates generated synthetic demo traces. It tests the replay and CI machinery; it does not show that this PR improves application performance.

AgentLoop CI Report

  • Status: passed
  • Replay: Replay passed: latency improvement 69.68%, cost improvement 78.82%, retry delta -1.
  • Merge recommendation: synthetic self-test only; assess application traces before merge
  • Findings: 1 total, 0 high severity, 1 patchable

Performance Gates

Gate Status Detail
latency_regression pass 0.00% regression <= 0.00% allowed
cost_regression pass 0.00% regression <= 0.00% allowed
latency_improvement pass 69.68% improvement >= 20.00% required
cost_improvement pass 78.82% improvement >= 5.00% required
retry_non_increase pass 0 candidate retries <= 1 baseline retries

Trace inputs

This comparison includes synthetic data. It does not establish application performance gains from a pull request.

Side Run ID Path Synthetic marker Source
baseline run_32a40059972146e9 runs/research_agent_baseline.json yes agentloop_demo
candidate run_d69bcfb6cf604cc8 runs/research_agent_optimized.json yes agentloop_demo

Metric Impact

Metric Impact
Latency improvement 69.68%
Cost improvement 78.82%
Retry delta -1
Candidate schema validity not provided
Candidate quality score not provided

Top Findings

Finding Severity Type Rewrite
al_cache_context_a4e5b713e7a3: Cache repeated prompt/context prefix medium cache_context Move stable instructions into cached prefixes, summaries, or framework-level memory instead of resending full text.

The by-name download immediately after upload kept missing the newly
finalized python-package artifact. Use the upload step's artifact ID and
compare bytes independent of the extracted directory layout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dipeshbabu
dipeshbabu merged commit fa9b854 into main Nov 15, 2026
25 checks passed
@dipeshbabu
dipeshbabu deleted the codex/codex-exec-runner branch November 15, 2026 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant