From 1ad49521e0f2771ee0eeb75045e64228ae518e68 Mon Sep 17 00:00:00 2001 From: Dipesh Babu Date: Sat, 14 Nov 2026 19:11:14 -0500 Subject: [PATCH 1/2] Add a Codex CLI runner for the guided workflow examples/codex_exec_runner.py runs codex exec --json in a read-only sandbox and converts its event stream into native traces: tool spans for commands and other tool items, model spans for visible model items and tool decisions, and provider turn usage on the turn's last model span. docs/CODEX.md documents the mapping and a first interleaved run, in which the tool-surface change was rejected on latency. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 3 + docs/CODEX.md | 80 ++++++++ docs/INTEGRATIONS.md | 3 + examples/codex_exec_runner.py | 279 +++++++++++++++++++++++++++ examples/codex_repository_tasks.json | 25 +++ mkdocs.yml | 1 + tests/test_codex_exec_runner.py | 163 ++++++++++++++++ 7 files changed, 554 insertions(+) create mode 100644 docs/CODEX.md create mode 100644 examples/codex_exec_runner.py create mode 100644 examples/codex_repository_tasks.json create mode 100644 tests/test_codex_exec_runner.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 5812c66..7ad8511 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,9 @@ trace, study, receipt, API and database schema versions. `improve finish` then measures the candidate against that control, keeping findings frozen from the original baseline, so cache warm-up and host drift affect both conditions. +- `examples/codex_exec_runner.py` runs OpenAI Codex CLI (`codex exec --json`) + on guided tasks and converts its event stream into native traces, with a + repository task set and a documented first run in `docs/CODEX.md`. ## [0.2.1] - 2026-11-12 diff --git a/docs/CODEX.md b/docs/CODEX.md new file mode 100644 index 0000000..d07e18f --- /dev/null +++ b/docs/CODEX.md @@ -0,0 +1,80 @@ +# OpenAI Codex CLI + +AgentLoop can profile and compare [Codex CLI](https://github.com/openai/codex) +runs through the guided workflow. The example +[runner](https://github.com/dipeshbabu/agentloop/blob/main/examples/codex_exec_runner.py) +runs `codex exec --json` for each task and converts its event stream into a +native AgentLoop trace. Codex is not modified and no plugin is installed. + +Each task attempt is a real Codex run on your ChatGPT plan or API key and counts +against that usage. + +## How events become a trace + +The runner starts `codex exec --json --ephemeral --skip-git-repo-check +--sandbox read-only --cd DIR -` and sends the task prompt on stdin. Codex events +carry no timestamps, so the runner timestamps each JSONL line as it arrives. + +| Codex event | AgentLoop event | +| --- | --- | +| `command_execution`, `file_change`, `mcp_tool_call`, `web_search` items | `tool_call` spanning `item.started` to `item.completed`; non-zero exit codes are errors | +| `agent_message` and `reasoning` items | `model_call` ending when the item arrived | +| Time before a tool call with no visible item | `model_call` named `codex_tool_decision` | +| `turn.completed` usage | Provider token counts on the turn's last model span, with `usage_scope: turn_total`, cached and reasoning tokens in metadata | +| `turn.failed` or `error` | Error on the root `codex_exec` span | + +Codex reports usage once per turn, so per-span token counts other than the turn +total are recorded as unavailable instead of being estimated. The final agent +message is the answer scored by the frozen task checks. + +## Run the guided workflow + +Tasks use a prompt string or an object with `prompt`. The checked-in +[task set](https://github.com/dipeshbabu/agentloop/blob/main/examples/codex_repository_tasks.json) +asks three metadata questions about this repository with exact expected values. + +```bash +agentloop improve capture --tasks examples/codex_repository_tasks.json --out runs/codex/baseline --timeout 600 -- python examples/codex_exec_runner.py --cd . +agentloop improve start --tasks examples/codex_repository_tasks.json --baseline runs/codex/baseline/capture.json --out runs/codex/session +``` + +Arguments after a second `--` go to `codex exec` unchanged. A candidate can +therefore change the model, reasoning effort, profile or features without +editing the runner. Capture it with the unchanged runner as an +[interleaved control](GUIDED_OPTIMIZATION.md#compare-against-an-interleaved-control): + +```bash +agentloop improve capture --tasks examples/codex_repository_tasks.json --out runs/codex/paired --timeout 600 --control-arg python --control-arg examples/codex_exec_runner.py --control-arg --cd --control-arg . -- python examples/codex_exec_runner.py --cd . -- -c model_reasoning_effort="low" +agentloop improve finish --session runs/codex/session/session.json --candidate runs/codex/paired/capture.json --change "Lower reasoning effort for lookups" --out runs/codex/results +``` + +Runner options: `--cd` sets the directory Codex reads, `--instructions FILE` +prepends shared instructions to each prompt, `--model` passes `--model` and +records the label, `--condition` labels the trace, and `--codex` selects the +executable. + +## First observed run + +On 2026-11-14 the maintainer ran this task set with Codex CLI 0.167.1 on +Windows, using the configured model at its configured reasoning effort. Each of +the three tasks ran twice per condition. + +- **Baseline:** all six answers were correct. Attempts took 15.6–29.8 seconds + and used 38,755–39,901 input tokens to read one field. AgentLoop reported + `split_large_step` on every attempt, pointing at the size of the model + context. +- **Candidate:** disabled the unused `apps`, `browser_use` and `computer_use` + features to shrink the tool context. It ran against an interleaved control, + with order alternating between attempts. +- **Outcome:** all twelve answers were correct, and four of six pairs passed. + Input tokens fell by about 2,600 (about 7%) in pairs where both runs took the + same path. Both `engines.node` pairs were slower than their controls, by 12% + and 19%, and failed the latency gates. **The change was rejected.** + +In three runs, two controls and one candidate, Codex made an extra shell call to +read a user-level instruction file referenced by the global Codex instructions. +Each of those runs used about 16,000–20,000 more input tokens. The traces make +that call visible as a separate `tool_call` span. This path variation is larger +than the measured context saving. More repetitions would be needed before +attributing a latency change to this configuration. These six pairs describe +one machine, one model configuration and three lookup tasks. diff --git a/docs/INTEGRATIONS.md b/docs/INTEGRATIONS.md index 8db7d11..1025b27 100644 --- a/docs/INTEGRATIONS.md +++ b/docs/INTEGRATIONS.md @@ -7,6 +7,9 @@ AgentLoop 0.2.0 also supports 0.22.x and 0.23.x. The [real runtime pilot](HERMES_RUNTIME_PILOT.md) documents exact host pins and deterministic-provider validation scope. +For OpenAI Codex CLI, see [the Codex guide](CODEX.md). Its example runner +converts `codex exec --json` events into native traces for the guided workflow. + For framework-neutral pipeline and stage annotations, see [workflow tracing](WORKFLOW_TRACING.md). diff --git a/examples/codex_exec_runner.py b/examples/codex_exec_runner.py new file mode 100644 index 0000000..de7397b --- /dev/null +++ b/examples/codex_exec_runner.py @@ -0,0 +1,279 @@ +"""Run OpenAI Codex CLI on guided tasks and record a native AgentLoop trace. + +The runner receives the standard `agentloop improve capture` JSON request on +stdin. It runs `codex exec --json --ephemeral` in a read-only sandbox, timestamps +each JSONL event as it arrives, and converts the stream into AgentLoop events: + +- each shell command, file change, MCP tool call or web search is a tool span; +- each agent message or reasoning item is a model span ending when it arrived; +- Codex reports token usage once per turn, so the turn total is attached to the + last model span of that turn with ``usage_scope = "turn_total"``. + +The final agent message is the task answer that the frozen checks score. +Arguments after ``--`` are passed to ``codex exec`` unchanged, so a candidate can +change the model, profile or configuration without editing this file. + +Example: + + agentloop improve capture --tasks examples/codex_repository_tasks.json \ + --out runs/codex-baseline -- python examples/codex_exec_runner.py --cd . +""" + +from __future__ import annotations + +import argparse +import json +import shutil +import subprocess # nosec B404 - runs the operator's Codex CLI with explicit argv +import sys +import threading +import time +import uuid +from datetime import datetime, timedelta, timezone +from pathlib import Path + +from agentloop.events import AgentEvent +from agentloop.tokens import PROVIDER, UNAVAILABLE +from agentloop.tracer import AgentTrace + +TOOL_ITEMS = {"command_execution", "file_change", "mcp_tool_call", "web_search"} +MODEL_ITEMS = {"agent_message", "reasoning"} +OUTPUT_LIMIT = 4096 + + +def task_prompt(task_input, instructions): + if isinstance(task_input, dict): + task_input = task_input.get("prompt") or task_input.get("question") + if not isinstance(task_input, str) or not task_input.strip(): + raise ValueError("Task input must be a prompt string or an object with 'prompt'") + return f"{instructions.strip()}\n\n{task_input}" if instructions else task_input + + +def clip(value): + if not isinstance(value, str): + return None + return value if len(value) <= OUTPUT_LIMIT else value[:OUTPUT_LIMIT] + "\n[truncated]" + + +def read_stream(process, lines): + """Timestamp each JSONL line on arrival; Codex events carry no timestamps.""" + for raw in process.stdout: + lines.append((time.perf_counter(), raw)) + + +def convert(lines, *, run_id, root_id, origin, clock, model): + def at(moment): + return (origin + timedelta(seconds=moment - clock)).isoformat() + + events, open_tools, output, usage_seen = [], {}, None, False + failure, boundary = None, clock + for moment, raw in lines: + try: + event = json.loads(raw) + except ValueError: + continue + kind, item = event.get("type"), event.get("item") or {} + item_type = item.get("type") + if kind == "turn.started": + boundary = moment + elif kind == "item.started" and item_type in TOOL_ITEMS: + if not open_tools and moment > boundary: + # The model chose this tool call without emitting a visible item. + events.append( + AgentEvent( + event_id="evt_" + uuid.uuid4().hex, + run_id=run_id, + parent_id=root_id, + event_type="model_call", + name="codex_tool_decision", + model=model, + started_at=at(boundary), + ended_at=at(moment), + duration_ms=(moment - boundary) * 1000, + token_provenance=UNAVAILABLE, + metadata={"codex_item_type": "tool_decision"}, + ) + ) + open_tools[item.get("id")] = moment + elif kind == "item.completed" and item_type in TOOL_ITEMS: + started = open_tools.pop(item.get("id"), moment) + exit_code = item.get("exit_code") + failed = item.get("status") == "failed" or exit_code not in (None, 0) + events.append( + AgentEvent( + event_id="evt_" + uuid.uuid4().hex, + run_id=run_id, + parent_id=root_id, + event_type="tool_call", + name="shell" if item_type == "command_execution" else item_type, + started_at=at(started), + ended_at=at(moment), + duration_ms=(moment - started) * 1000, + input_text=clip(item.get("command") or json.dumps(item.get("changes"))), + output_text=clip(item.get("aggregated_output")), + status="error" if failed else "ok", + error=f"exit code {exit_code}" if failed else None, + metadata={"codex_item_type": item_type, "exit_code": exit_code}, + ) + ) + boundary = max(boundary, moment) + elif kind == "item.completed" and item_type in MODEL_ITEMS: + # The model was working from the previous visible boundary until this + # item arrived, unless a tool was still running in that interval. + started = max([boundary, *open_tools.values()]) + events.append( + AgentEvent( + event_id="evt_" + uuid.uuid4().hex, + run_id=run_id, + parent_id=root_id, + event_type="model_call", + name=f"codex_{item_type}", + model=model, + started_at=at(started), + ended_at=at(moment), + duration_ms=(moment - started) * 1000, + token_provenance=UNAVAILABLE, + output_text=clip(item.get("text")), + metadata={"codex_item_type": item_type}, + ) + ) + if item_type == "agent_message": + output = item.get("text") + boundary = moment + elif kind == "turn.completed": + usage = event.get("usage") or {} + models = [item for item in events if item.event_type == "model_call"] + if not models: + models.append( + AgentEvent( + event_id="evt_" + uuid.uuid4().hex, + run_id=run_id, + parent_id=root_id, + event_type="model_call", + name="codex_turn", + model=model, + started_at=at(boundary), + ended_at=at(moment), + duration_ms=(moment - boundary) * 1000, + ) + ) + events.append(models[0]) + last = models[-1] + last.input_tokens = int(usage.get("input_tokens") or 0) + last.output_tokens = int(usage.get("output_tokens") or 0) + last.token_provenance = PROVIDER + last.metadata.update( + usage_scope="turn_total", + cached_input_tokens=int(usage.get("cached_input_tokens") or 0), + reasoning_output_tokens=int(usage.get("reasoning_output_tokens") or 0), + ) + usage_seen = True + elif kind in {"turn.failed", "error"}: + detail = event.get("error") or event.get("message") or kind + failure = detail.get("message", str(detail)) if isinstance(detail, dict) else detail + return events, output, usage_seen, failure + + +def run(args, codex_args, request): + codex = shutil.which(args.codex) + if codex is None: + raise SystemExit(f"Cannot find the Codex CLI executable {args.codex!r}") + instructions = args.instructions.read_text(encoding="utf-8") if args.instructions else "" + prompt = task_prompt(request["input"], instructions) + command = [ + codex, + "exec", + "--json", + "--ephemeral", + "--skip-git-repo-check", + "--sandbox", + "read-only", + "--cd", + str(args.cd.resolve()), + *codex_args, + "-", + ] + trace = AgentTrace( + f"codex exec: {request['task_id']}", + metadata={ + "integration": "codex_exec", + "condition": args.condition, + "task_id": request["task_id"], + "repetition": request["repetition"], + "codex_args": codex_args, + }, + ) + root_id = "evt_" + uuid.uuid4().hex + origin, clock = datetime.now(timezone.utc), time.perf_counter() + process = subprocess.Popen( # nosec B603 - explicit Codex argv, no shell + command, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + encoding="utf-8", + errors="replace", + ) + lines = [] + reader = threading.Thread(target=read_stream, args=(process, lines), daemon=True) + reader.start() + stderr = threading.Thread(target=lambda: process.stderr.read(), daemon=True) + stderr.start() + process.stdin.write(prompt) + process.stdin.close() + returncode = process.wait() + reader.join() + ended = time.perf_counter() + events, output, usage_seen, failure = convert( + lines, run_id=trace.run_id, root_id=root_id, origin=origin, clock=clock, model=args.model + ) + if returncode != 0 and failure is None: + failure = f"codex exec exited with code {returncode}" + trace.add_event( + AgentEvent( + event_id=root_id, + run_id=trace.run_id, + event_type="agent_step", + name="codex_exec", + started_at=origin.isoformat(), + ended_at=(origin + timedelta(seconds=ended - clock)).isoformat(), + duration_ms=(ended - clock) * 1000, + input_text=prompt, + output_text=clip(output), + status="error" if failure else "ok", + error=failure, + metadata={"operation_kind": "agent", "usage_reported": usage_seen}, + ) + ) + for event in events: + trace.add_event(event) + trace.metadata.update( + output=(output or "").strip() if output is not None else None, + success=failure is None and output is not None, + ) + trace.finish() + trace.export_json(request["trace_path"]) + return 1 if failure else 0 + + +def main(): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--cd", type=Path, default=Path("."), help="Directory Codex reads.") + parser.add_argument("--codex", default="codex", help="Codex CLI executable.") + parser.add_argument("--model", default=None, help="Model label recorded in the trace.") + parser.add_argument("--condition", default="unlabelled", help="Label recorded in the trace.") + parser.add_argument( + "--instructions", type=Path, default=None, help="Text file prepended to each task prompt." + ) + argv = sys.argv[1:] + split = argv.index("--") if "--" in argv else len(argv) + args = parser.parse_args(argv[:split]) + codex_args = argv[split + 1 :] + if args.model: + codex_args = ["--model", args.model, *codex_args] + request = json.loads(sys.stdin.readline()) + raise SystemExit(run(args, codex_args, request)) + + +if __name__ == "__main__": + main() diff --git a/examples/codex_repository_tasks.json b/examples/codex_repository_tasks.json new file mode 100644 index 0000000..49f7af4 --- /dev/null +++ b/examples/codex_repository_tasks.json @@ -0,0 +1,25 @@ +{ + "schema_version": "1.0", + "name": "Codex repository metadata questions", + "repetitions": 2, + "tasks": [ + { + "id": "python-support", + "input": {"prompt": "Read pyproject.toml in the current directory. What is the exact value of project.requires-python, including comparison operators? Reply with only that value."}, + "expected": ">=3.11", + "scorer": {"type": "contains"} + }, + { + "id": "npm-package-name", + "input": {"prompt": "Read sdk/typescript/package.json in the current directory. What is the exact value of the name field? Reply with only that value."}, + "expected": "agentloop-profiler", + "scorer": {"type": "contains"} + }, + { + "id": "npm-node-engine", + "input": {"prompt": "Read sdk/typescript/package.json in the current directory. What is the exact value of engines.node? Reply with only that value."}, + "expected": ">=22", + "scorer": {"type": "contains"} + } + ] +} diff --git a/mkdocs.yml b/mkdocs.yml index 40e631f..68155a4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -105,6 +105,7 @@ nav: - Integrations: - Framework overview: INTEGRATIONS.md - Hermes: HERMES.md + - Codex CLI: CODEX.md - LangGraph: LANGGRAPH_HARNESS.md - Harbor: - Completed trials: HARBOR_TRIALS.md diff --git a/tests/test_codex_exec_runner.py b/tests/test_codex_exec_runner.py new file mode 100644 index 0000000..bc662f6 --- /dev/null +++ b/tests/test_codex_exec_runner.py @@ -0,0 +1,163 @@ +from __future__ import annotations + +import json +from datetime import datetime, timezone + +import pytest + +from agentloop.events import AgentEvent +from agentloop.findings import build_diagnosis +from agentloop.tracer import AgentTrace +from examples.codex_exec_runner import convert, task_prompt + +# Shape recorded from `codex exec --json` (Codex CLI 0.167.1), shortened. +STREAM = [ + (0.0, {"type": "thread.started", "thread_id": "thread"}), + (0.5, {"type": "turn.started"}), + ( + 2.0, + { + "type": "item.started", + "item": {"id": "item_1", "type": "command_execution", "command": "cat pyproject.toml"}, + }, + ), + ( + 2.4, + { + "type": "item.completed", + "item": { + "id": "item_1", + "type": "command_execution", + "command": "cat pyproject.toml", + "aggregated_output": 'requires-python = ">=3.11"', + "exit_code": 0, + "status": "completed", + }, + }, + ), + ( + 3.0, + { + "type": "item.completed", + "item": {"id": "item_2", "type": "agent_message", "text": ">=3.11"}, + }, + ), + ( + 3.1, + { + "type": "turn.completed", + "usage": { + "input_tokens": 40000, + "cached_input_tokens": 32000, + "output_tokens": 60, + "reasoning_output_tokens": 20, + }, + }, + ), +] + + +def lines(stream): + return [(moment, json.dumps(event) + "\n") for moment, event in stream] + + +def run_convert(stream): + return convert( + lines(stream), + run_id="run_test", + root_id="evt_root", + origin=datetime(2026, 11, 14, tzinfo=timezone.utc), + clock=0.0, + model="test-model", + ) + + +def test_stream_becomes_tool_and_model_spans_with_turn_usage(): + events, output, usage_seen, failure = run_convert(STREAM) + assert output == ">=3.11" + assert usage_seen is True + assert failure is None + assert [(event.event_type, event.name) for event in events] == [ + ("model_call", "codex_tool_decision"), + ("tool_call", "shell"), + ("model_call", "codex_agent_message"), + ] + decision, tool, answer = events + assert decision.duration_ms == pytest.approx(1500) + assert tool.duration_ms == pytest.approx(400) + assert tool.input_text == "cat pyproject.toml" + assert answer.duration_ms == pytest.approx(600) + assert decision.input_tokens == 0 and decision.token_provenance == "unavailable" + assert (answer.input_tokens, answer.output_tokens, answer.token_provenance) == ( + 40000, + 60, + "provider", + ) + assert answer.metadata["usage_scope"] == "turn_total" + assert answer.metadata["cached_input_tokens"] == 32000 + + +def test_failed_commands_and_turns_are_retained_as_errors(): + stream = [ + *STREAM[:3], + ( + 2.4, + { + "type": "item.completed", + "item": {"id": "item_1", "type": "command_execution", "exit_code": 2}, + }, + ), + (3.0, {"type": "turn.failed", "error": {"message": "usage limit reached"}}), + ] + events, output, usage_seen, failure = run_convert(stream) + tool = next(event for event in events if event.event_type == "tool_call") + assert tool.status == "error" and tool.error == "exit code 2" + assert output is None + assert usage_seen is False + assert failure == "usage limit reached" + + +def test_usage_without_model_items_gets_its_own_model_span(): + events, _, usage_seen, _ = run_convert([STREAM[1], STREAM[-1]]) + assert usage_seen is True + assert [(event.name, event.input_tokens) for event in events] == [("codex_turn", 40000)] + + +def test_converted_events_form_an_analyzable_trace(): + events, output, _, _ = run_convert(STREAM) + trace = AgentTrace("codex test", metadata={"output": output}) + trace.add_event( + AgentEvent( + event_id="evt_root", + run_id=trace.run_id, + event_type="agent_step", + name="codex_exec", + started_at=events[0].started_at, + ended_at=events[-1].ended_at, + duration_ms=3100, + metadata={"operation_kind": "agent"}, + ) + ) + for event in events: + event.run_id = trace.run_id + trace.add_event(event) + trace.finish() + restored = AgentTrace.from_dict(json.loads(json.dumps(trace.to_dict()))) + assert build_diagnosis(restored)["run_id"] == trace.run_id + + +@pytest.mark.parametrize( + ("value", "instructions", "expected"), + [ + ("plain", "", "plain"), + ({"prompt": "object"}, "", "object"), + ({"question": "legacy"}, "Be brief.", "Be brief.\n\nlegacy"), + ], +) +def test_task_prompt_accepts_strings_and_prompt_objects(value, instructions, expected): + assert task_prompt(value, instructions) == expected + + +def test_task_prompt_rejects_missing_prompts(): + with pytest.raises(ValueError, match="prompt"): + task_prompt({"other": 1}, "") From 7ee547c2b58212fa8e485fd35fcb99a5eb966b1d Mon Sep 17 00:00:00 2001 From: Dipesh Babu Date: Sat, 14 Nov 2026 19:18:54 -0500 Subject: [PATCH 2/2] Download the smoke artifact by its upload ID The by-name download immediately after upload kept missing the newly finalized python-package artifact. Use the upload step's artifact ID and compare bytes independent of the extracted directory layout. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6e5ad39..94a8bf7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -347,6 +347,7 @@ jobs: --expected-version "$(uv version --short)" - name: Upload validated release artifact + id: upload-package if: ${{ inputs.upload_release_artifact || github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch' }} uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 with: @@ -354,11 +355,13 @@ jobs: path: dist/* if-no-files-found: error + # Download by the returned ID: a by-name lookup immediately after upload + # can miss the newly finalized artifact. - name: Download validated release artifact if: ${{ inputs.upload_release_artifact || github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch' }} uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 with: - name: python-package + artifact-ids: ${{ steps.upload-package.outputs.artifact-id }} path: .artifact-download-smoke - name: Verify downloaded artifact bytes @@ -367,7 +370,8 @@ jobs: run: | diff -u \ <(cd dist && sha256sum * | sort) \ - <(cd .artifact-download-smoke && sha256sum * | sort) + <(find .artifact-download-smoke -type f -exec sha256sum {} + \ + | awk '{ n = split($2, part, "/"); print $1 " " part[n] }' | sort) standalone: name: Standalone CLI (${{ matrix.target }})