Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,28 @@ perfect prefix caching and the end-to-end session latency. `--results-csv` persi
the same figures; `--agent-events-jsonl` records per-trajectory admission and
completion times.

`--request-events-jsonl` writes one line per agent request, success or failure, when
the run ends. Times are milliseconds measured by the client:

| Field | Meaning |
|---|---|
| `trajectory_id`, `agent_id` | The trajectory the request belongs to. |
| `invocation` | The request's 1-based position in its trajectory. |
| `sent_at_ms` | Start of the first attempt, from the start of the run (the clock `--agent-events-jsonl` uses). |
| `ttft_ms` | From `sent_at_ms` to the first streamed chunk whose delta carries `content`, `reasoning_content` or `tool_calls`. Null for non-streamed requests and when no such chunk arrived. |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The timing definition should say that content and reasoning_content must be non-empty and that tool_calls must contain a delta; otherwise readers can count the initial empty stream chunk as the first token. Match the documentation to delta_carries_output.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At README.md, line 139:

<comment>The timing definition should say that `content` and `reasoning_content` must be non-empty and that `tool_calls` must contain a delta; otherwise readers can count the initial empty stream chunk as the first token. Match the documentation to `delta_carries_output`.</comment>

<file context>
@@ -128,6 +128,28 @@ perfect prefix caching and the end-to-end session latency. `--results-csv` persi
+| `trajectory_id`, `agent_id` | The trajectory the request belongs to. |
+| `invocation` | The request's 1-based position in its trajectory. |
+| `sent_at_ms` | Start of the first attempt, from the start of the run (the clock `--agent-events-jsonl` uses). |
+| `ttft_ms` | From `sent_at_ms` to the first streamed chunk whose delta carries `content`, `reasoning_content` or `tool_calls`. Null for non-streamed requests and when no such chunk arrived. |
+| `last_token_ms` | From `sent_at_ms` to the last such chunk. |
+| `e2e_ms` | From `sent_at_ms` until the request succeeded or its last attempt failed. |
</file context>
Suggested change
| `ttft_ms` | From `sent_at_ms` to the first streamed chunk whose delta carries `content`, `reasoning_content` or `tool_calls`. Null for non-streamed requests and when no such chunk arrived. |
| `ttft_ms` | From `sent_at_ms` to the first streamed chunk whose delta has non-empty `content` or `reasoning_content`, or a non-empty `tool_calls` array. Null for non-streamed requests and when no such chunk arrived. |

| `last_token_ms` | From `sent_at_ms` to the last such chunk. |
| `e2e_ms` | From `sent_at_ms` until the request succeeded or its last attempt failed. |
| `tpot_ms` | `(last_token_ms - ttft_ms) / (completion_tokens - 1)`; null below two completion tokens. |
| `prompt_tokens`, `completion_tokens` | From the response's `usage`; null on failure. |
| `cached_tokens` | `usage.prompt_tokens_details.cached_tokens`, or null when the backend omits it. |
| `attempts` | Attempts made, retries included. |
| `status`, `error` | `"ok"` with a null error, or `"error"` with the last attempt's error. |
| `http_status` | Status of the last attempt's response; null when none arrived. |

For a retried request, `ttft_ms` and `last_token_ms` come from the attempt that
succeeded and are measured from the first attempt's start. A trajectory stops at its
first failed request, so its failure is its last line.

`batchbench-harness` drives runs on remote GPU hosts; see [HARNESS.md](HARNESS.md).

## Building from source
Expand Down
Loading
Loading