BatchBench

Load generation for OpenAI-compatible chat endpoints. A Rust core with a thin Python wrapper, published as prebuilt wheels:

uv pip install batchbench

Three entrypoints:

Command	What it does
batchbench	Independent requests at a fixed concurrency with controlled token shapes
batchbench-agent	Stateful multi-turn sessions, sampled or replayed from a manifest
batchbench-export-plans	Builds a replay manifest from prompt-chain records in ClickHouse

Independent requests

batchbench \
  --model gpt-4o-mini \
  --users 8 \
  --requests-per-user 2 \
  --input-tokens 256 \
  --output-tokens 64 \
  --output-vary 0

Output length is enforced with min_tokens/max_tokens; pass --sglang for backends that use min_new_tokens/max_new_tokens. Ctrl+C cancels active requests and prints a partial summary. The same run is available from Python:

import batchbench

report = batchbench.run_benchmark({
    "endpoint": "https://example.com/v1/chat/completions",
    "user_count": 1,
    "mode": batchbench.finite_mode(requests_per_user=1),
    "request_body": batchbench.request_entry(
        {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "ping"}]},
        line_idx=0, input_tokens=1),
    "requests": [batchbench.request_entry(
        {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "ping"}], "max_tokens": 4},
        line_idx=0, input_tokens=1)],
    "dry_run": True,
})

Multi-turn sessions

batchbench-agent runs concurrent agent loops. Within a loop, each request appends the model's reply and a synthetic tool result to the conversation and sends it again, so every turn reuses the previous prompt as a prefix. Token shapes are fixed or log-normal:

batchbench-agent \
  --model Qwen/Qwen3-8B \
  --host http://localhost:8000 \
  --agents 16 \
  --input-tokens 256 \
  --output-tokens 64 \
  --environment-tokens 128 \
  --tool-invocations 8 \
  --tool-call-latency-ms 250

Each --*-tokens flag has a --*-lognorm-median / --*-lognorm-sigma / --*-lognorm-max family; --seed makes sampling reproducible. --tokenizer-model names the tokenizer when the endpoint's model name is not a Hugging Face identifier (a model id, a tokenizer.json, or a directory containing one). Every request carries a per-agent user field and routing header so a gateway can keep a session on one worker; --user-prefix, --disable-user-tagging, and --dp-rank-perfect-routing control that.

Replaying a manifest

A manifest is a JSONL file with one trajectory per line: its requests, their prompt and output token counts, the gaps between them, and optionally the prompt's content blocks. The agent binary replays it instead of sampling:

batchbench-agent \
  --model Qwen/Qwen3-8B \
  --host http://localhost:8000 \
  --agent-plans-jsonl plans.jsonl \
  --admission open-loop \
  --time-scale 2 \
  --agent-events-jsonl events.jsonl \
  --results-csv results.csv

--admission open-loop starts each trajectory at its recorded offset; the default closed-loop admission fills --max-active-agents slots in file order. --time-scale N replays a recorded hour in 1/N hours. Block text is generated from each block's seed at exactly the requested token count, so blocks with equal seeds send identical bytes and shared prefixes exercise prefix caching. --dry-run checks a manifest without sending requests. The format is specified in docs/manifest.md; examples/trajectory-replay/ has one file per schema version.

Building a manifest from records

batchbench-export-plans reads a window of prompt-chain records from ClickHouse, rebuilds sessions from the chains, and writes a schema version 2 manifest:

uv pip install "batchbench[export]"

CLICKHOUSE_URL=https://warehouse.example.com:8443 CLICKHOUSE_USER=... CLICKHOUSE_PASSWORD=... \
batchbench-export-plans \
  --chains-table chains_current --analytics-table requests \
  --start 2026-09-01T09:00:00Z --end 2026-09-01T10:00:00Z \
  --model vendor/model-a \
  --sample 0.1 --seed 7 --stratify-by-session-length \
  --output plans.jsonl

The input tables it expects, and how sessions are reconstructed, are described in docs/export-plans.md.

Report

Each run prints total prompt and completion tokens from the responses' usage and request latency p50/p90/p99. Agent runs add the estimated cached prompt tokens under perfect prefix caching and the end-to-end session latency. --results-csv persists the same figures; --agent-events-jsonl records per-trajectory admission and completion times.

--request-events-jsonl writes one line per agent request, success or failure, when the run ends. Times are milliseconds measured by the client:

Field	Meaning
trajectory_id, agent_id	The trajectory the request belongs to.
invocation	The request's 1-based position in its trajectory.
sent_at_ms	Start of the first attempt, from the start of the run (the clock --agent-events-jsonl uses).
ttft_ms	From sent_at_ms to the first streamed chunk whose delta carries content, reasoning_content or tool_calls. Null for non-streamed requests and when no such chunk arrived.
last_token_ms	From sent_at_ms to the last such chunk.
e2e_ms	From sent_at_ms until the request succeeded or its last attempt failed.
tpot_ms	(last_token_ms - ttft_ms) / (completion_tokens - 1); null below two completion tokens.
prompt_tokens, completion_tokens	From the response's usage; null on failure.
cached_tokens	usage.prompt_tokens_details.cached_tokens, or null when the backend omits it.
attempts	Attempts made, retries included.
status, error	"ok" with a null error, or "error" with the last attempt's error.
http_status	Status of the last attempt's response; null when none arrived.

For a retried request, ttft_ms and last_token_ms come from the attempt that succeeded and are measured from the first attempt's start. A trajectory stops at its first failed request, so its failure is its last line.

batchbench-harness drives runs on remote GPU hosts; see HARNESS.md.

Building from source

cargo build --release --manifest-path rust/Cargo.toml --bin batchbench
cargo build --release --manifest-path rust/Cargo.toml --bin batchbench-agent

Releases are cut by Release Please; a v* tag builds platform wheels and publishes them to PyPI.