[fergusfinn]

Adds batchbench-compare-trace-replay, which scores a replay of Dynamo request traces against production's own record of the same requests.

It joins three inputs on each request's source row:

• the manifest's metadata.requests (feat(export): name each trace request's source row in trajectory metadata #42), which names each request's source row;
• the agent's --request-events-jsonl (feat(agent): record per-request timings with --request-events-jsonl #41), which holds the replay's client-side timings;
• the trace rows themselves.

Over a scoring window of source time, it reports for both sides:

• requests and failures;
• first-token, time-per-output-token and end-to-end percentiles;
• the share that succeeded within a first-token bound and a time-per-token bound.

A failed replayed request counts against that share. --replay-gpus adds the replay's throughput per GPU. An optional production context file maps decode worker ids to worker sets and gives each set's GPUs. With it, production's figures are split by set, and with --model each set's throughput per GPU over the window is reported beside the replay's.

Only requests whose source arrived inside the window count. That lets an export that starts earlier warm the replay's caches before scoring begins.

Stacked on #42.

Checked end to end on a real manifest: a 1/16 session sample of GLM-5.3-Flash, 12 Nov 17:30–19:00, scored over 18:00–19:00.

• With an events file that replays production exactly, all 8,403 scored requests matched.
• Every replay figure equals production's, overall and per set.
• The per-set throughput agrees with an independent count: Isambard 0.1127 req/s/GPU, 110 output tok/s/GPU, 65.2% within 15 s / 33 ms.

Tests: 44 pass, 2 skipped (they need the compiled agent).

🤖 Generated with Claude Code

https://claude.ai/code/session_01VDPqeFzGgF3vkuiyimwMxX

[cubic-dev-ai]

3 issues found across 5 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/batchbench/trace_replay.py">

<violation number="1" location="src/batchbench/trace_replay.py:108">
P2: `within_slo` counts every successful request with missing TPOT as within SLO, even when it has two or more output tokens. Restrict the missing-TPOT exception to requests with fewer than two output tokens.</violation>

<violation number="2" location="src/batchbench/trace_replay.py:123">
P2: A malformed or truncated trace line containing `"request_end"` crashes the comparison before it can report valid rows from the file. Parse the row defensively and skip malformed or non-mapping lines, matching `load_trace_records` behavior.</violation>

<violation number="3" location="src/batchbench/trace_replay.py:225">
P1: The documented replay-events input cannot be produced by `batchbench-agent`: it exposes only `--agent-events-jsonl`, which writes agent lifecycle records without `invocation` or request timings. Add the per-request event output (or consume the existing format) and update the command/docs consistently.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: The documented replay-events input cannot be produced by batchbench-agent: it exposes only --agent-events-jsonl, which writes agent lifecycle records without invocation or request timings. Add the per-request event output (or consume the existing format) and update the command/docs consistently.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/batchbench/trace_replay.py, line 225:

<comment>The documented replay-events input cannot be produced by `batchbench-agent`: it exposes only `--agent-events-jsonl`, which writes agent lifecycle records without `invocation` or request timings. Add the per-request event output (or consume the existing format) and update the command/docs consistently.</comment>

<file context>
@@ -0,0 +1,272 @@
+        description="Compare a replay of Dynamo request traces with production's record of the same requests.",
+    )
+    parser.add_argument("--manifest", required=True, help="the replayed manifest (batchbench-export-plans --dynamo-traces)")
+    parser.add_argument("--request-events", required=True, help="the replay's batchbench-agent --request-events-jsonl")
+    parser.add_argument("--dynamo-traces", action="append", required=True, metavar="PATH",
+                        help="production trace files: a directory, file or glob; repeatable")
</file context>

[cubic-dev-ai]

P2: A malformed or truncated trace line containing "request_end" crashes the comparison before it can report valid rows from the file. Parse the row defensively and skip malformed or non-mapping lines, matching load_trace_records behavior.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/batchbench/trace_replay.py, line 123:

<comment>A malformed or truncated trace line containing `"request_end"` crashes the comparison before it can report valid rows from the file. Parse the row defensively and skip malformed or non-mapping lines, matching `load_trace_records` behavior.</comment>

<file context>
@@ -0,0 +1,272 @@
+        for line in _trace_lines(path, stats):
+            if f'"{TRACE_EVENT}"' not in line:
+                continue
+            request = json.loads(line).get("request") or {}
+            request_id = request.get("request_id")
+            in_window = model is not None and request.get("model") == model and lo_ms <= (request.get("request_received_ms") or 0) < hi_ms
</file context>

Suggested change

	request = json.loads(line).get("request") or {}
	try:
	row = json.loads(line)
	except ValueError:
	continue
	if not isinstance(row, Mapping):
	continue
	request = row.get("request") or {}
	if not isinstance(request, Mapping):
	continue

[cubic-dev-ai]

P2: within_slo counts every successful request with missing TPOT as within SLO, even when it has two or more output tokens. Restrict the missing-TPOT exception to requests with fewer than two output tokens.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At src/batchbench/trace_replay.py, line 108:

<comment>`within_slo` counts every successful request with missing TPOT as within SLO, even when it has two or more output tokens. Restrict the missing-TPOT exception to requests with fewer than two output tokens.</comment>

<file context>
@@ -0,0 +1,272 @@
+    """Succeeded, first token within the bound, and time per output token within its bound (or under two tokens)."""
+
+    return (timing.ok and timing.ttft_s is not None and timing.ttft_s <= slo_ttft_s
+            and (timing.tpot_s is None or timing.tpot_s <= slo_tpot_s))
+
+
</file context>

[fergusfinn]

Virtual Fergus: reviewed, looks right.

• Sources come from metadata.requests with a 1-based invocation, which lines up with feat(agent): record per-request timings with --request-events-jsonl #41's events. Null request ids (feat(export): name each trace request's source row in trajectory metadata #42's fix) are skipped. Only rows whose source arrived in [score-start, score-end) count.
• A failed replay counts against the within-bound share. Production's "ok" is "has a first token", and its time per token is avg_itl_ms.
• Per-set throughput is requests / window seconds / set_gpus, so it is right only if set_gpus is the GPUs averaged over the whole window. internal#6192's script currently averages only the minutes a set was present (comment there), which overstates a set's GPUs when it was absent for part of the window.
• python3 -m unittest discover -s tests at 8d43f70: 44 ran, OK, 2 skipped.

It lands after #40 and #42, which it is stacked on.

[cubic-dev-ai]

0 issues found across 3 files (changes from recent commits).

<sub>Requires human review: Auto-approval blocked by 3 unresolved issues from previous reviews.

View guided diff | Turn on auto-fix | Re-trigger cubic</sub>

[fergusfinn]

Virtual Fergus: re-reviewed at 59ab413 (--stream). The change is right: it is opt-in, it runs after sampling, it sets stream: true on every request so the agent measures first-token time and time per output token, and the test covers both with and without the flag. python3 -m unittest discover -s tests: 45 ran, OK, 2 skipped.

One wording fix. The --stream help says "the records don't say whether the original was streamed". That holds for Dynamo traces (docs/export-plans.md:228), but prompt-chain rows carry stream (docs/export-plans.md:32, copied at export_plans.py:441). For those rows the flag overrides a recorded stream: false. Something like "mark every request stream: true, overriding any recorded value" would be accurate for both sources.