Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -347,18 +347,21 @@ jobs:
--expected-version "$(uv version --short)"

- name: Upload validated release artifact
id: upload-package
if: ${{ inputs.upload_release_artifact || github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch' }}
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: python-package
path: dist/*
if-no-files-found: error

# Download by the returned ID: a by-name lookup immediately after upload
# can miss the newly finalized artifact.
- name: Download validated release artifact
if: ${{ inputs.upload_release_artifact || github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch' }}
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: python-package
artifact-ids: ${{ steps.upload-package.outputs.artifact-id }}
path: .artifact-download-smoke

- name: Verify downloaded artifact bytes
Expand All @@ -367,7 +370,8 @@ jobs:
run: |
diff -u \
<(cd dist && sha256sum * | sort) \
<(cd .artifact-download-smoke && sha256sum * | sort)
<(find .artifact-download-smoke -type f -exec sha256sum {} + \
| awk '{ n = split($2, part, "/"); print $1 " " part[n] }' | sort)

standalone:
name: Standalone CLI (${{ matrix.target }})
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ trace, study, receipt, API and database schema versions.
`improve finish` then measures the candidate against that control, keeping
findings frozen from the original baseline, so cache warm-up and host drift
affect both conditions.
- `examples/codex_exec_runner.py` runs OpenAI Codex CLI (`codex exec --json`)
on guided tasks and converts its event stream into native traces, with a
repository task set and a documented first run in `docs/CODEX.md`.

## [0.2.1] - 2026-11-12

Expand Down
80 changes: 80 additions & 0 deletions docs/CODEX.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# OpenAI Codex CLI

AgentLoop can profile and compare [Codex CLI](https://github.com/openai/codex)
runs through the guided workflow. The example
[runner](https://github.com/dipeshbabu/agentloop/blob/main/examples/codex_exec_runner.py)
runs `codex exec --json` for each task and converts its event stream into a
native AgentLoop trace. Codex is not modified and no plugin is installed.

Each task attempt is a real Codex run on your ChatGPT plan or API key and counts
against that usage.

## How events become a trace

The runner starts `codex exec --json --ephemeral --skip-git-repo-check
--sandbox read-only --cd DIR -` and sends the task prompt on stdin. Codex events
carry no timestamps, so the runner timestamps each JSONL line as it arrives.

| Codex event | AgentLoop event |
| --- | --- |
| `command_execution`, `file_change`, `mcp_tool_call`, `web_search` items | `tool_call` spanning `item.started` to `item.completed`; non-zero exit codes are errors |
| `agent_message` and `reasoning` items | `model_call` ending when the item arrived |
| Time before a tool call with no visible item | `model_call` named `codex_tool_decision` |
| `turn.completed` usage | Provider token counts on the turn's last model span, with `usage_scope: turn_total`, cached and reasoning tokens in metadata |
| `turn.failed` or `error` | Error on the root `codex_exec` span |

Codex reports usage once per turn, so per-span token counts other than the turn
total are recorded as unavailable instead of being estimated. The final agent
message is the answer scored by the frozen task checks.

## Run the guided workflow

Tasks use a prompt string or an object with `prompt`. The checked-in
[task set](https://github.com/dipeshbabu/agentloop/blob/main/examples/codex_repository_tasks.json)
asks three metadata questions about this repository with exact expected values.

```bash
agentloop improve capture --tasks examples/codex_repository_tasks.json --out runs/codex/baseline --timeout 600 -- python examples/codex_exec_runner.py --cd .
agentloop improve start --tasks examples/codex_repository_tasks.json --baseline runs/codex/baseline/capture.json --out runs/codex/session
```

Arguments after a second `--` go to `codex exec` unchanged. A candidate can
therefore change the model, reasoning effort, profile or features without
editing the runner. Capture it with the unchanged runner as an
[interleaved control](GUIDED_OPTIMIZATION.md#compare-against-an-interleaved-control):

```bash
agentloop improve capture --tasks examples/codex_repository_tasks.json --out runs/codex/paired --timeout 600 --control-arg python --control-arg examples/codex_exec_runner.py --control-arg --cd --control-arg . -- python examples/codex_exec_runner.py --cd . -- -c model_reasoning_effort="low"
agentloop improve finish --session runs/codex/session/session.json --candidate runs/codex/paired/capture.json --change "Lower reasoning effort for lookups" --out runs/codex/results
```

Runner options: `--cd` sets the directory Codex reads, `--instructions FILE`
prepends shared instructions to each prompt, `--model` passes `--model` and
records the label, `--condition` labels the trace, and `--codex` selects the
executable.

## First observed run

On 2026-11-14 the maintainer ran this task set with Codex CLI 0.167.1 on
Windows, using the configured model at its configured reasoning effort. Each of
the three tasks ran twice per condition.

- **Baseline:** all six answers were correct. Attempts took 15.6–29.8 seconds
and used 38,755–39,901 input tokens to read one field. AgentLoop reported
`split_large_step` on every attempt, pointing at the size of the model
context.
- **Candidate:** disabled the unused `apps`, `browser_use` and `computer_use`
features to shrink the tool context. It ran against an interleaved control,
with order alternating between attempts.
- **Outcome:** all twelve answers were correct, and four of six pairs passed.
Input tokens fell by about 2,600 (about 7%) in pairs where both runs took the
same path. Both `engines.node` pairs were slower than their controls, by 12%
and 19%, and failed the latency gates. **The change was rejected.**

In three runs, two controls and one candidate, Codex made an extra shell call to
read a user-level instruction file referenced by the global Codex instructions.
Each of those runs used about 16,000–20,000 more input tokens. The traces make
that call visible as a separate `tool_call` span. This path variation is larger
than the measured context saving. More repetitions would be needed before
attributing a latency change to this configuration. These six pairs describe
one machine, one model configuration and three lookup tasks.
3 changes: 3 additions & 0 deletions docs/INTEGRATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ AgentLoop 0.2.0 also supports 0.22.x and 0.23.x. The
[real runtime pilot](HERMES_RUNTIME_PILOT.md) documents exact host pins and
deterministic-provider validation scope.

For OpenAI Codex CLI, see [the Codex guide](CODEX.md). Its example runner
converts `codex exec --json` events into native traces for the guided workflow.

For framework-neutral pipeline and stage annotations, see
[workflow tracing](WORKFLOW_TRACING.md).

Expand Down
Loading
Loading