From 9fe9cdd65e29576ec58fd972299d61069c2bdfe2 Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 16:49:04 +0000 Subject: [PATCH 1/9] feat(export): build manifests from Dynamo request traces Add --dynamo-traces, which reads Dynamo frontend request_end rows (dynamo.request.trace.v1, *.jsonl.gz) from local files and writes the same schema v2 manifest as the prompt-chain source. - Rows are windowed by request_received_ms, not file name, and attempts sharing an x_request_id collapse to the last one that finished. - Sessions link by the strict-prefix rule on the block hashes, within one model, block size, principal and agent session. - Prompts are cut where chains diverge or end; each segment is a user block sized trace_block_size x hashes, the partial tail is a final block, and an extension carries the previous tail, a live reply, then new content. - An optional join on x_request_id (--analytics-table or --analytics-rows-jsonl) fills principal and alias. - Docs list the fidelity lost against prompt chains. --- README.md | 6 +- docs/export-plans.md | 122 +++++- src/batchbench/dynamo_traces.py | 643 ++++++++++++++++++++++++++++++++ src/batchbench/export_plans.py | 175 ++++++--- tests/test_dynamo_traces.py | 385 +++++++++++++++++++ 5 files changed, 1280 insertions(+), 51 deletions(-) create mode 100644 src/batchbench/dynamo_traces.py create mode 100644 tests/test_dynamo_traces.py diff --git a/README.md b/README.md index 9a79954..825a09e 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Three entrypoints: |---|---| | `batchbench` | Independent requests at a fixed concurrency with controlled token shapes | | `batchbench-agent` | Stateful multi-turn sessions, sampled or replayed from a manifest | -| `batchbench-export-plans` | Builds a replay manifest from prompt-chain records in ClickHouse | +| `batchbench-export-plans` | Builds a replay manifest from prompt-chain records in ClickHouse or from Dynamo request traces | ## Independent requests @@ -117,7 +117,9 @@ batchbench-export-plans \ --output plans.jsonl ``` -The input tables it expects, and how sessions are reconstructed, are described in +It can also build one from a directory of Dynamo frontend request traces +(`--dynamo-traces`), with no ClickHouse needed. The input tables and trace rows it +expects, and how sessions are reconstructed, are described in [docs/export-plans.md](docs/export-plans.md). ## Report diff --git a/docs/export-plans.md b/docs/export-plans.md index 7949444..4d68f3d 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -1,7 +1,8 @@ # Exporting a manifest from prompt-chain records -`batchbench-export-plans` reads prompt-chain records from ClickHouse and writes a -[schema version 2 manifest](manifest.md). It needs the `export` extra: +`batchbench-export-plans` reads prompt-chain records from ClickHouse, or Dynamo +request traces from local files ([below](#dynamo-request-traces)), and writes a +[schema version 2 manifest](manifest.md). Reading ClickHouse needs the `export` extra: ```bash uv pip install "batchbench[export]" @@ -79,3 +80,120 @@ Replay with: ```bash batchbench-agent --agent-plans-jsonl plans.jsonl --admission open-loop --model --host ``` + +## Dynamo request traces + +`--dynamo-traces` replaces the prompt-chain source with the `request_end` rows a Dynamo +frontend writes when request tracing is enabled (schema `dynamo.request.trace.v1`, +gzipped or plain JSONL). It takes a directory (read recursively for `*.jsonl.gz` and +`*.jsonl`), a file, or a glob, and may be repeated. No ClickHouse is needed: + +```bash +batchbench-export-plans \ + --dynamo-traces ./traces \ + --model example-org/model-a \ + --start 2026-09-01T09:00:00Z --end 2026-09-01T10:00:00Z \ + --output plans.jsonl +``` + +Fields read from each row: + +| Field | Use | +|---|---| +| `request.model` | Dynamo model name; `--model` matches it exactly | +| `request.request_received_ms` | Request time, used for the window, ordering and offsets | +| `request.total_time_ms` | Duration, for `delay_after_ms` | +| `request.input_tokens` | `prompt_tokens` (`replay.input_length` when absent or short of the hashed blocks) | +| `request.output_tokens` | `output_tokens`; rows without a positive count are dropped | +| `request.replay.trace_block_size` | Tokens per hashed block; varies by deployment | +| `request.replay.input_sequence_hashes` | Cumulative hash of each full block of the prompt | +| `request.x_request_id` | The caller's `x-request-id` header, when it sent one | +| `agent_context.session_id` | Agent session, when present | +| `request.finish_reason_metadata.finish_reason` | Picks among retried attempts | + +Trace files are named by upload time, and a file can hold rows received long before it +was written, so every file is read and rows are windowed by `request_received_ms`: +rows in `[--start - --link-window-hours, --end)` are loaded, and those before `--start` +only serve as link targets. Rows with the same `x_request_id` are attempts of one +request (a gateway retry or failover reuses the header); the last attempt that finished +is kept and the others are counted as `duplicate_attempts`. + +### Sessions and blocks + +Requests link with the same strict-prefix rule as prompt chains, applied to the block +hashes, and only within the same model, block size, principal and agent session id. +Rows with different block sizes are never linked or merged, even within one model. + +Each prompt is cut after every hash where some chain of the same model and block size +ends or where two such chains continue differently. Each piece becomes one `user` block +whose seed is the piece's last hash and whose size is the block size times its hash +count, so a prefix shared by many requests (a system prompt, a common document) replays +with identical bytes everywhere. The tokens past the last full block +(`input_tokens - block_size * hashes`) become a final `user` block with a seed of its +own, so `sum(blocks.tokens) = prompt_tokens` and `overhead_tokens` is 0. A final block +rather than overhead keeps the replayed prompt the recorded length, which matters for +block sizes in the thousands. + +A request that extends the previous request of its trajectory repeats that request's +blocks, including its tail block, and the tokens that follow start with a `live` +assistant block of the previous request's `output_tokens`, then the rest as `user` +blocks. When the extension holds fewer tokens than the previous output, the live block +is clamped to fit and counted as `live_clamped`. Each trajectory carries +`metadata` with the model name, block size and, when joined, the alias and principal. + +### Joining analytics on `x_request_id` + +With `--analytics-table` (and `--clickhouse-url`) the rows that carry an `x_request_id` +are joined to a gateway analytics table on `--analytics-id-column` (default +`x_request_id`). The query reads that table's id, `instance_id`, `correlation_id`, +`model`, `user_id` and `api_key_id`, bounded by `--analytics-ts-column` (default +`timestamp`) to the time span of the rows with an id, widened by 15 minutes on each +side. The id is compared as a string, case-insensitively. A matched row's `user_id` +(else its `api_key_id`) becomes the principal, which links sessions and is what +`--principal-id` filters on, and its `model` becomes the trajectory's `alias`. +Unmatched rows keep no principal and link by agent session id alone, if they have one. +`--dump-rows-jsonl` saves the analytics rows and `--analytics-rows-jsonl` joins against +such a file instead of ClickHouse: + +```bash +CLICKHOUSE_URL=https://warehouse.example.com:8443 CLICKHOUSE_USER=... CLICKHOUSE_PASSWORD=... \ +batchbench-export-plans \ + --dynamo-traces ./traces --model example-org/model-a \ + --analytics-table requests \ + --start 2026-09-01T09:00:00Z --end 2026-09-01T10:00:00Z \ + --output plans.jsonl +``` + +Besides the usual counts, the summary line reports, for traces: `files`, +`unreadable_files` (a truncated file keeps the rows before the damage), `lines`, +`bad_lines`, `not_request_end`, `no_replay`, `other_model`, `outside_window`, +`malformed_hashes`, `loaded`, `with_x_request_id`, `duplicate_attempts`, +`analytics_rows`, `analytics_matched`, `with_session_id`, `live_blocks`, +`live_clamped`, and `block_sizes` (rows loaded per block size). + +### Fidelity lost against prompt chains + +- **No roles.** Every block is `user` except the live reply: system prompts, tool + results and tool calls are not told apart. +- **No tools array.** Tool definitions are part of the hashed prompt, so they replay as + user text and the replayed request sends no `tools`. +- **Template overhead is merged and double counted.** Trace token counts include the + chat template. The replay sends every block as its own message and the backend adds + its template on top, so the replayed prompt exceeds `prompt_tokens` by the replay's + per-message template cost, and the recorded per-message overhead cannot be separated + from content. +- **Granularity is one trace block.** Cuts fall only on block boundaries, so a block + straddles message boundaries, and the live reply starts where the previous prompt's + tokens end rather than at a message boundary. +- **Large blocks hide divergence.** Two prompts that differ only inside their last + partial block have the same hashes. A turn that adds less than one block keeps its + chain unchanged, so it is not a strict extension and starts a new trajectory; with + block sizes in the thousands, many sessions fall apart into single requests. Without + a principal or session id, a short request whose distinctive content sits in its tail + can be linked to an unrelated longer request that shares its full blocks. +- **Reply length is approximate.** The live block is the previous `output_tokens`; + when a backend drops reasoning from the history, the reply in the next prompt is + shorter than that, and the new user content is undercounted by the difference. +- **No `stream` or `max_tokens`.** Traces do not record them, so they are omitted. +- **Timing starts at Dynamo.** `request_received_ms` is when the frontend received the + request, after any gateway queueing. diff --git a/src/batchbench/dynamo_traces.py b/src/batchbench/dynamo_traces.py new file mode 100644 index 0000000..745ab71 --- /dev/null +++ b/src/batchbench/dynamo_traces.py @@ -0,0 +1,643 @@ +"""Build schema version 2 manifests from Dynamo frontend request traces. + +A Dynamo frontend with request tracing enabled writes one ``request_end`` row +(``dynamo.request.trace.v1``) per request: timings, token counts, and the KV router's +cumulative block hashes of the prompt (``replay.input_sequence_hashes``, one per full block of +``replay.trace_block_size`` tokens). The hashes play the role of a prompt chain, at block rather +than message granularity, so sessions are rebuilt with the same strict-prefix rule as +:mod:`batchbench.export_plans`: + +* Requests link only within the same model, block size, principal, and agent session. +* Every prompt is cut wherever its chain diverges from another chain of the same model and block + size, or where another chain ends. Each segment becomes one ``user`` block whose seed is the + segment's last hash, so prompts that share a prefix replay with identical bytes. +* The tokens past the last full block become a final ``user`` block of their own. +* In a request that extends the previous one, the first ``output_tokens`` tokens after the + previous prompt (and its tail) become the ``live`` assistant block. + +An optional analytics join on ``x_request_id`` fills the principal and the model alias. The +expected input is documented in ``docs/export-plans.md``. +""" + +from __future__ import annotations + +import glob +import gzip +import json +import os +import uuid +import zlib +from array import array +from dataclasses import dataclass, field +from datetime import datetime, timedelta, timezone +from typing import Any, Iterable, Iterator, Mapping, Optional, Sequence + +from .export_plans import ( + DEFAULT_LINK_WINDOW, + SCHEMA_VERSION, + ChainRecord, + ExportError, + ExportStats, + Trajectory, + build_trajectories, + extends, + link_parents, + scale_ms, + usable_records, +) + +TRACE_SCHEMA = "dynamo.request.trace.v1" +TRACE_EVENT = "request_end" +TRACE_SUFFIXES = (".jsonl.gz", ".jsonl") +DEFAULT_ANALYTICS_ID_COLUMN = "x_request_id" +# The analytics timestamp is when the gateway received the request, the trace's when Dynamo did; +# retries and failover land later still. +ANALYTICS_PAD = timedelta(minutes=15) +ANALYTICS_COLUMNS = ("x_request_id", "instance_id", "correlation_id", "model", "user_id", "api_key_id") + + +# --------------------------------------------------------------------------- +# Records + + +@dataclass +class TraceRecord(ChainRecord): + """One ``request_end`` row. ``chain`` holds the block hashes as unsigned 64-bit integers.""" + + block_size: int = 0 + tail_tokens: int = 0 + request_id: str = "" + x_request_id: Optional[str] = None + session_id: Optional[str] = None + alias: Optional[str] = None + + @property + def link_key(self) -> Any: + return (self.model, self.block_size, self.principal_id, self.session_id or "") + + @property + def end_ms(self) -> float: + return self.ts.timestamp() * 1000.0 + (self.duration_ms or 0.0) + + +@dataclass +class TraceStats: + files: int = 0 + unreadable_files: int = 0 + lines: int = 0 + bad_lines: int = 0 + not_request_end: int = 0 + no_replay: int = 0 + other_model: int = 0 + outside_window: int = 0 + malformed_hashes: int = 0 + loaded: int = 0 + with_x_request_id: int = 0 + duplicate_attempts: int = 0 + analytics_rows: int = 0 + analytics_matched: int = 0 + with_session_id: int = 0 + live_blocks: int = 0 + live_clamped: int = 0 + block_sizes: dict[int, int] = field(default_factory=dict) + + def as_dict(self) -> dict[str, Any]: + values: dict[str, Any] = {key: value for key, value in self.__dict__.items() if key != "block_sizes"} + values["block_sizes"] = ",".join(f"{size}:{count}" for size, count in sorted(self.block_sizes.items())) or "none" + return values + + +def normalize_request_id(value: Any) -> Optional[str]: + """Canonical lower-case form of an ``x-request-id``; UUIDs in their hyphenated form.""" + + if value is None: + return None + text = str(value).strip().lower() + if not text: + return None + try: + return str(uuid.UUID(text)) + except ValueError: + return text + + +def _is_count(value: Any) -> bool: + return isinstance(value, int) and not isinstance(value, bool) and value >= 0 + + +def record_from_trace( + row: Mapping[str, Any], + stats: TraceStats, + model: Optional[str] = None, + window_ms: Optional[tuple[float, float]] = None, +) -> Optional[TraceRecord]: + """Build a :class:`TraceRecord` from one parsed trace line, or count why it was skipped.""" + + if row.get("schema") != TRACE_SCHEMA or row.get("event_type") != TRACE_EVENT: + stats.not_request_end += 1 + return None + request = row.get("request") + replay = request.get("replay") if isinstance(request, Mapping) else None + received_ms = request.get("request_received_ms") if isinstance(request, Mapping) else None + if not isinstance(replay, Mapping) or not _is_count(received_ms): + stats.no_replay += 1 + return None + row_model = str(request.get("model") or "") + if model is not None and row_model != model: + stats.other_model += 1 + return None + if window_ms is not None and not (window_ms[0] <= received_ms < window_ms[1]): + stats.outside_window += 1 + return None + + block_size = replay.get("trace_block_size") + hashes = replay.get("input_sequence_hashes") or [] + input_length = replay.get("input_length") + if not _is_count(block_size) or block_size == 0 or not isinstance(hashes, list): + stats.malformed_hashes += 1 + return None + try: + chain = array("Q", hashes) + except (TypeError, OverflowError): + stats.malformed_hashes += 1 + return None + full_tokens = block_size * len(chain) + prompt_tokens = request.get("input_tokens") + if not _is_count(prompt_tokens) or prompt_tokens < full_tokens: + prompt_tokens = input_length + if not _is_count(prompt_tokens) or prompt_tokens < full_tokens: + stats.malformed_hashes += 1 + return None + + agent = row.get("agent_context") if isinstance(row.get("agent_context"), Mapping) else {} + session_id = agent.get("session_id") + finish = request.get("finish_reason_metadata") + finish_reason = finish.get("finish_reason") if isinstance(finish, Mapping) else None + total_time_ms = request.get("total_time_ms") + output_tokens = request.get("output_tokens") + record = TraceRecord( + ts=datetime.fromtimestamp(received_ms / 1000.0, tz=timezone.utc), + instance_id="", + correlation_id=0, + principal_id="", + model=row_model, + chain=chain, # type: ignore[arg-type] + block_roles=[], + block_tokens=[], + prompt_tokens=int(prompt_tokens), + completion_tokens=int(output_tokens) if _is_count(output_tokens) else None, + duration_ms=float(total_time_ms) if isinstance(total_time_ms, (int, float)) and not isinstance(total_time_ms, bool) else None, + stream=None, + max_tokens=None, + finish_reason=None if finish_reason is None else str(finish_reason), + block_size=int(block_size), + tail_tokens=int(prompt_tokens) - full_tokens, + request_id=str(request.get("request_id") or ""), + x_request_id=normalize_request_id(request.get("x_request_id")), + session_id=None if session_id in (None, "") else str(session_id), + ) + stats.loaded += 1 + stats.block_sizes[record.block_size] = stats.block_sizes.get(record.block_size, 0) + 1 + if record.x_request_id: + stats.with_x_request_id += 1 + if record.session_id: + stats.with_session_id += 1 + return record + + +def trace_files(paths: Sequence[str]) -> list[str]: + """Expand directories (recursively) and glob patterns into trace files, sorted.""" + + found: set[str] = set() + for path in paths: + if os.path.isdir(path): + for directory, _, names in os.walk(path): + found.update(os.path.join(directory, name) for name in names if name.endswith(TRACE_SUFFIXES)) + elif os.path.isfile(path): + found.add(path) + else: + matches = [match for match in glob.glob(path, recursive=True) if os.path.isfile(match)] + if not matches: + raise ExportError(f"no trace files match {path!r}") + found.update(matches) + if not found: + raise ExportError("no trace files found (expected *.jsonl.gz or *.jsonl)") + return sorted(found) + + +def _trace_lines(path: str, stats: TraceStats) -> Iterator[str]: + stats.files += 1 + try: + if path.endswith(".gz"): + handle = gzip.open(path, "rt", encoding="utf-8") + else: + handle = open(path, "r", encoding="utf-8") + with handle: + for line in handle: + yield line + except (OSError, EOFError, zlib.error, UnicodeDecodeError): + # A truncated upload keeps the rows read before the damage. + stats.unreadable_files += 1 + + +def load_trace_records( + paths: Sequence[str], + window_start: datetime, + window_end: datetime, + link_window: timedelta = DEFAULT_LINK_WINDOW, + model: Optional[str] = None, + stats: Optional[TraceStats] = None, +) -> list[TraceRecord]: + """Read ``request_end`` rows received in ``[window_start - link_window, window_end)``. + + Files are named by upload time and can hold rows received long before it, so every file is + read and rows are windowed by ``request_received_ms``. + """ + + stats = stats if stats is not None else TraceStats() + window_ms = ( + (window_start - link_window).timestamp() * 1000.0, + window_end.timestamp() * 1000.0, + ) + model_needle = None if model is None else json.dumps(model, ensure_ascii=False) + records: list[TraceRecord] = [] + for path in trace_files(paths): + for line in _trace_lines(path, stats): + if not line.strip(): + continue + stats.lines += 1 + if f'"{TRACE_EVENT}"' not in line: + stats.not_request_end += 1 + continue + if model_needle is not None and model_needle not in line: + stats.other_model += 1 + continue + try: + row = json.loads(line) + except ValueError: + stats.bad_lines += 1 + continue + if not isinstance(row, Mapping): + stats.bad_lines += 1 + continue + record = record_from_trace(row, stats, model, window_ms) + if record is not None: + records.append(record) + return records + + +def _attempt_rank(record: TraceRecord) -> tuple: + return ( + bool(record.completion_tokens), + record.finish_reason is not None, + record.end_ms, + record.request_id, + ) + + +def drop_duplicate_attempts(records: Iterable[TraceRecord], stats: TraceStats) -> list[TraceRecord]: + """Keep one row per ``x_request_id``: the last attempt that finished. + + A gateway retry or failover inside one client request reuses its ``x-request-id``, so the + failed attempts and the final one all have rows. Rows are then ordered and re-indexed. + """ + + kept: list[TraceRecord] = [] + best: dict[str, TraceRecord] = {} + with_id = 0 + for record in records: + if not record.x_request_id: + kept.append(record) + continue + with_id += 1 + current = best.get(record.x_request_id) + if current is None or _attempt_rank(record) > _attempt_rank(current): + best[record.x_request_id] = record + stats.duplicate_attempts += with_id - len(best) + kept.extend(best.values()) + kept.sort(key=lambda record: (record.ts, record.request_id, record.x_request_id or "")) + for index, record in enumerate(kept): + record.index = index + return kept + + +# --------------------------------------------------------------------------- +# Analytics join + + +def build_analytics_query(table: str, id_column: str = DEFAULT_ANALYTICS_ID_COLUMN, ts_column: str = "timestamp") -> str: + """Analytics rows with an ``x-request-id`` in ``[{start}, {end})``; columns as ``ANALYTICS_COLUMNS``.""" + + return f""" +SELECT + toString({id_column}) AS x_request_id, + toString(instance_id) AS instance_id, + correlation_id, + model, + toString(user_id) AS user_id, + toString(api_key_id) AS api_key_id +FROM {table} +WHERE {ts_column} >= {{start:DateTime64(3)}} + AND {ts_column} < {{end:DateTime64(3)}} + AND {id_column} IS NOT NULL +""".strip() + + +def analytics_window(records: Iterable[TraceRecord]) -> Optional[tuple[datetime, datetime]]: + """The analytics time range that covers every record with an ``x_request_id``.""" + + starts = [] + ends = [] + for record in records: + if record.x_request_id: + starts.append(record.ts) + ends.append(record.end_ts) + if not starts: + return None + return min(starts) - ANALYTICS_PAD, max(ends) + ANALYTICS_PAD + + +def fetch_analytics_rows( + url: str, + user: Optional[str], + password: Optional[str], + sql: str, + parameters: Mapping[str, Any], + wanted: set[str], +) -> list[dict[str, Any]]: + """Stream the analytics query and keep the rows whose id is in ``wanted``.""" + + try: + import clickhouse_connect # type: ignore[import-not-found] + except ImportError as error: # pragma: no cover - exercised only without the extra + raise ExportError("clickhouse-connect is required to read ClickHouse; install batchbench[export]") from error + + client_args: dict[str, Any] = {"dsn": url} + if user: + client_args["username"] = user + if password: + client_args["password"] = password + client = clickhouse_connect.get_client(**client_args) + rows = [] + try: + with client.query_rows_stream(sql, parameters=dict(parameters)) as stream: + for values in stream: + row = dict(zip(ANALYTICS_COLUMNS, values)) + if normalize_request_id(row["x_request_id"]) in wanted: + rows.append(row) + finally: + client.close() + return rows + + +def apply_analytics(records: Iterable[TraceRecord], rows: Iterable[Mapping[str, Any]], stats: TraceStats) -> None: + """Fill principal, alias and gateway ids from analytics rows matched on ``x_request_id``. + + The principal is the analytics ``user_id``, else its ``api_key_id``. + """ + + by_id: dict[str, Mapping[str, Any]] = {} + for row in rows: + request_id = normalize_request_id(row.get("x_request_id")) + if request_id: + by_id[request_id] = row + stats.analytics_rows += len(by_id) + for record in records: + row = by_id.get(record.x_request_id) if record.x_request_id else None + if row is None: + continue + stats.analytics_matched += 1 + user_id = row.get("user_id") + api_key_id = row.get("api_key_id") + record.principal_id = str(user_id or api_key_id or "") + record.user_id = None if user_id is None else str(user_id) + record.api_key_id = None if api_key_id is None else str(api_key_id) + record.alias = None if row.get("model") is None else str(row["model"]) + if row.get("instance_id") is not None: + record.instance_id = str(row["instance_id"]) + if row.get("correlation_id") is not None: + record.correlation_id = int(row["correlation_id"]) + + +# --------------------------------------------------------------------------- +# Blocks + + +def _common_prefix_length(left: array, right: array) -> int: + low, high = 0, min(len(left), len(right)) + while low < high: + middle = (low + high + 1) // 2 + if left[:middle] == right[:middle]: + low = middle + else: + high = middle - 1 + return low + + +def segment_cuts(records: Iterable[TraceRecord]) -> dict[tuple[str, int], set[int]]: + """Per (model, block size), the hashes after which a prompt is cut into a new segment. + + A prompt is cut after a hash where some chain ends or where two chains continue differently. + In lexicographic order, chains through one hash are contiguous, so both cases show up as the + common prefix of two neighbouring chains. + """ + + groups: dict[tuple[str, int], list[array]] = {} + for record in records: + if len(record.chain): + groups.setdefault((record.model, record.block_size), []).append(record.chain) # type: ignore[arg-type] + cuts: dict[tuple[str, int], set[int]] = {} + for key, chains in groups.items(): + chains.sort() + group_cuts = {chain[-1] for chain in chains} + for left, right in zip(chains, chains[1:]): + shared = _common_prefix_length(left, right) + if shared: + group_cuts.add(left[shared - 1]) + cuts[key] = group_cuts + return cuts + + +def hash_seed(value: int) -> str: + return f"{value:016x}" + + +def record_segments(record: TraceRecord, cuts: set[int]) -> list[tuple[int, str, int]]: + """``(end position, seed, tokens)`` for each segment of the record's full blocks.""" + + chain = record.chain + positions = sorted(chain.index(hash_) for hash_ in cuts.intersection(chain)) # type: ignore[attr-defined] + segments = [] + start = 0 + for position in positions: + end = position + 1 + segments.append((end, hash_seed(chain[position]), record.block_size * (end - start))) # type: ignore[arg-type] + start = end + if start < len(chain): # unreachable when cuts include every leaf; keep the blocks anyway + segments.append((len(chain), hash_seed(chain[-1]), record.block_size * (len(chain) - start))) # type: ignore[arg-type] + return segments + + +def _request_anchor(record: TraceRecord) -> str: + leaf = hash_seed(record.chain[-1]) if len(record.chain) else "none" # type: ignore[arg-type] + return f"{leaf}:{record.request_id or record.index}" + + +def tail_seed(record: TraceRecord) -> str: + """The tokens past the last full block are this request's own; the seed says so.""" + + return f"{_request_anchor(record)}:tail{record.tail_tokens}" + + +def _carve(pieces: list[tuple[str, int, int]], tokens: int) -> list[tuple[str, int, int]]: + """Remove ``tokens`` from the front of ``(seed, full tokens, tokens left)`` pieces.""" + + remaining = [] + for seed, full, left in pieces: + if tokens >= left: + tokens -= left + continue + remaining.append((seed, full, left - tokens)) + tokens = 0 + return remaining + + +def _piece_block(seed: str, full: int, left: int) -> dict[str, Any]: + # A piece cut short is a different block from the whole piece, so it gets its own seed. + return _user(seed if left == full else f"{seed}:rest{left}", left) + + +def _user(seed: str, tokens: int) -> dict[str, Any]: + return {"seed": seed, "tokens": int(tokens), "role": "user"} + + +def trace_request_blocks( + record: TraceRecord, + previous: Optional[TraceRecord], + cuts: set[int], + stats: TraceStats, +) -> list[dict[str, Any]]: + """Blocks for one request; ``previous`` is the request it extends, if any. + + In an extension the tokens after the previous prompt's full blocks are, in order, the + previous prompt's tail (same seed as there), the previous reply (``live``, as many tokens as + the previous output, clamped to what the extension holds), and new user content. + """ + + segments = record_segments(record, cuts) + tail = [(tail_seed(record), record.tail_tokens)] if record.tail_tokens > 0 else [] + if previous is None: + return [_user(seed, tokens) for _, seed, tokens in segments] + [_user(seed, tokens) for seed, tokens in tail] + + shared = len(previous.chain) + blocks = [_user(seed, tokens) for end, seed, tokens in segments if end <= shared] + extension = [(seed, tokens, tokens) for end, seed, tokens in segments if end > shared] + extension += [(seed, tokens, tokens) for seed, tokens in tail] + if previous.tail_tokens > 0: + blocks.append(_user(tail_seed(previous), previous.tail_tokens)) + extension = _carve(extension, previous.tail_tokens) + reply = int(previous.completion_tokens or 0) + live = min(reply, sum(left for _, _, left in extension)) + if live < reply: + stats.live_clamped += 1 + if live > 0: + blocks.append({"seed": f"{_request_anchor(previous)}:reply{live}", "tokens": live, "role": "assistant", "live": True}) + extension = _carve(extension, live) + stats.live_blocks += 1 + blocks.extend(_piece_block(*piece) for piece in extension) + return blocks + + +# --------------------------------------------------------------------------- +# Plans + + +def _trajectory_stem(record: TraceRecord) -> str: + if len(record.chain): + return hash_seed(record.chain[-1]) # type: ignore[arg-type] + return f"req-{record.request_id or record.index}" + + +def trace_trajectory_plan( + trajectory: Trajectory, + trajectory_id: str, + window_start: datetime, + time_scale: float, + cuts: Mapping[tuple[str, int], set[int]], + stats: TraceStats, +) -> dict[str, Any]: + requests: list[TraceRecord] = trajectory.requests # type: ignore[assignment] + root: TraceRecord = trajectory.root # type: ignore[assignment] + group_cuts = cuts.get((root.model, root.block_size), set()) + first_offset_ms = (requests[0].ts - window_start).total_seconds() * 1000.0 + plan_requests = [] + previous: Optional[TraceRecord] = None + for position, record in enumerate(requests): + if position == 0: + reset_before = trajectory.began_before_window + else: + reset_before = not extends(previous.chain, record.chain) # type: ignore[union-attr] + blocks = trace_request_blocks(record, None if reset_before else previous, group_cuts, stats) + prompt_tokens = int(record.prompt_tokens or 0) + request: dict[str, Any] = { + "prompt_tokens": prompt_tokens, + "output_tokens": int(record.completion_tokens or 0), + } + if position + 1 < len(requests): + gap_ms = (requests[position + 1].ts - record.end_ts).total_seconds() * 1000.0 + request["delay_after_ms"] = scale_ms(gap_ms, time_scale) + if reset_before: + request["reset_before"] = True + request["overhead_tokens"] = max(prompt_tokens - sum(block["tokens"] for block in blocks), 0) + request["blocks"] = blocks + plan_requests.append(request) + previous = record + + metadata: dict[str, Any] = {"source": "dynamo-request-trace", "model": root.model, "block_size": root.block_size} + if root.alias: + metadata["alias"] = root.alias + if root.principal_id: + metadata["principal"] = root.principal_id + return { + "schema_version": SCHEMA_VERSION, + "trajectory_id": trajectory_id, + "start_after_ms": scale_ms(first_offset_ms, time_scale), + "requests": plan_requests, + "metadata": metadata, + } + + +def build_trace_plans( + records: Sequence[TraceRecord], + window_start: datetime, + window_end: datetime, + time_scale: float = 1.0, + link_window: timedelta = DEFAULT_LINK_WINDOW, + stats: Optional[ExportStats] = None, + trace_stats: Optional[TraceStats] = None, +) -> list[dict[str, Any]]: + """Reconstruct sessions from trace records and convert them to manifest lines.""" + + stats = stats if stats is not None else ExportStats() + trace_stats = trace_stats if trace_stats is not None else TraceStats() + if time_scale <= 0 or time_scale != time_scale or time_scale == float("inf"): + raise ExportError("time scale must be a positive finite number") + usable = usable_records(records, stats) + cuts = segment_cuts(usable) # type: ignore[arg-type] + parents = link_parents(usable, link_window) + trajectories = build_trajectories(usable, parents, window_start, window_end) + stats.in_window = sum(len(trajectory.requests) for trajectory in trajectories) + stats.trajectories = len(trajectories) + stats.prewindow_sessions = sum(1 for trajectory in trajectories if trajectory.began_before_window) + + counters: dict[str, int] = {} + plans = [] + for trajectory in trajectories: + stem = _trajectory_stem(trajectory.root) # type: ignore[arg-type] + counter = counters.get(stem, 0) + counters[stem] = counter + 1 + plans.append( + trace_trajectory_plan(trajectory, f"{stem}-{counter}", window_start, time_scale, cuts, trace_stats) + ) + plans.sort(key=lambda plan: (plan["start_after_ms"], plan["trajectory_id"])) + return plans diff --git a/src/batchbench/export_plans.py b/src/batchbench/export_plans.py index 87c3732..77aef1a 100644 --- a/src/batchbench/export_plans.py +++ b/src/batchbench/export_plans.py @@ -17,7 +17,8 @@ * ``start_after_ms`` and ``delay_after_ms`` reproduce the recorded arrival pattern, divided by ``--time-scale``. -The expected input schema is documented in ``docs/export-plans.md``. The ClickHouse client is +:mod:`batchbench.dynamo_traces` builds the same manifests from Dynamo request traces. The expected +input schemas are documented in ``docs/export-plans.md``. The ClickHouse client is confined to :func:`fetch_rows`; everything else works on plain row dictionaries so it can be tested with fixtures. """ @@ -90,6 +91,12 @@ class ChainRecord: def leaf(self) -> Optional[str]: return self.chain[-1] if self.chain else None + @property + def link_key(self) -> Any: + """Requests link only to earlier requests with the same key.""" + + return self.principal_id + @property def end_ts(self) -> datetime: return self.ts + timedelta(milliseconds=self.duration_ms or 0.0) @@ -247,19 +254,19 @@ def link_parents( """ parents: dict[int, Optional[int]] = {} - by_principal: dict[str, list[ChainRecord]] = {} + by_principal: dict[Any, list[ChainRecord]] = {} for record in records: - by_principal.setdefault(record.principal_id, []).append(record) + by_principal.setdefault(record.link_key, []).append(record) for principal_records in by_principal.values(): principal_records.sort(key=_record_order) - latest_by_leaf: dict[str, ChainRecord] = {} + latest_by_leaf: dict[Any, ChainRecord] = {} + leaves: set[Any] = set() for record in principal_records: parent: Optional[ChainRecord] = None - for hash_ in record.chain[:-1]: - candidate = latest_by_leaf.get(hash_) - if candidate is None: - continue + # A set intersection keeps this linear in C for long block-level chains. + for hash_ in leaves.intersection(record.chain[:-1]): + candidate = latest_by_leaf[hash_] if record.ts - candidate.ts > link_window: continue if parent is None or _record_order(candidate) > _record_order(parent): @@ -267,6 +274,7 @@ def link_parents( parents[record.index] = None if parent is None else parent.index if record.leaf is not None: latest_by_leaf[record.leaf] = record + leaves.add(record.leaf) return parents @@ -763,7 +771,7 @@ def fetch_rows( def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( prog="batchbench export-plans", - description="Export a batchbench schema v2 trajectory manifest from ClickHouse prompt-chain records.", + description="Export a batchbench schema v2 trajectory manifest from ClickHouse prompt-chain records or Dynamo request traces.", ) source = parser.add_argument_group("source") source.add_argument("--clickhouse-url", default=os.environ.get("CLICKHOUSE_URL"), help="ClickHouse HTTP(S) URL (env CLICKHOUSE_URL)") @@ -774,13 +782,18 @@ def build_parser() -> argparse.ArgumentParser: source.add_argument("--analytics-table", help="per-request usage table joined on (instance_id, correlation_id)") source.add_argument("--analytics-ts-column", default=DEFAULT_ANALYTICS_TS_COLUMN, help=f"timestamp column used to bound the analytics side of the join (default {DEFAULT_ANALYTICS_TS_COLUMN})") source.add_argument("--rows-jsonl", help="read previously fetched rows from this JSONL file instead of ClickHouse") - source.add_argument("--dump-rows-jsonl", help="also write the fetched rows to this JSONL file") + source.add_argument("--dump-rows-jsonl", help="also write the fetched rows (analytics rows with --dynamo-traces) to this JSONL file") + + traces = parser.add_argument_group("dynamo request traces") + traces.add_argument("--dynamo-traces", action="append", metavar="PATH", help="read Dynamo request_end traces (*.jsonl.gz or *.jsonl) from this directory, file, or glob instead of prompt chains; repeatable") + traces.add_argument("--analytics-id-column", default="x_request_id", help="analytics column holding the gateway's x-request-id (default x_request_id)") + traces.add_argument("--analytics-rows-jsonl", help="join traces against analytics rows from this JSONL file instead of ClickHouse") window = parser.add_argument_group("window and filters") window.add_argument("--start", required=True, help="window start (RFC 3339)") window.add_argument("--end", required=True, help="window end (RFC 3339, exclusive)") window.add_argument("--principal-id", help="restrict to one principal") - window.add_argument("--model", help="restrict to one model alias") + window.add_argument("--model", help="restrict to one model alias (with --dynamo-traces: the Dynamo model name)") window.add_argument("--served-by", help="restrict to requests served by this upstream") window.add_argument("--link-window-hours", type=float, default=DEFAULT_LINK_WINDOW.total_seconds() / 3600.0, help="how far back a continuation may look for its parent (default 24)") @@ -813,45 +826,16 @@ def main(argv: Optional[Sequence[str]] = None) -> int: raise ExportError("--link-window-hours must be non-negative") link_window = timedelta(hours=args.link_window_hours) - if args.rows_jsonl: - rows = read_rows_jsonl(args.rows_jsonl) - else: - if not args.clickhouse_url: - raise ExportError("--clickhouse-url (or CLICKHOUSE_URL) is required without --rows-jsonl") - if not args.chains_table or not args.analytics_table: - raise ExportError("--chains-table and --analytics-table are required without --rows-jsonl") - sql, parameters = build_query( - args.chains_table, - args.analytics_table, - args.principal_id, - args.model, - args.served_by, - chains_final=args.chains_final, - analytics_ts_column=args.analytics_ts_column, - ) - parameters["lookback_start"] = (window_start - link_window).replace(tzinfo=None) - parameters["window_end"] = window_end.replace(tzinfo=None) - rows = fetch_rows(args.clickhouse_url, args.clickhouse_user, args.clickhouse_password, sql, parameters) - if args.dump_rows_jsonl: - with open(args.dump_rows_jsonl, "w", encoding="utf-8") as handle: - for row in rows: - handle.write(json.dumps(row, default=_json_default) + "\n") - stats = ExportStats() - records = records_from_rows(rows) - if args.rows_jsonl: - records = [ - record - for record in records - if (not args.principal_id or record.principal_id == args.principal_id) - and (not args.model or record.model == args.model) - and (not args.served_by or record.served_by == args.served_by) - ] - plans = build_plans(records, window_start, window_end, args.time_scale, link_window, stats) + extra_stats: dict[str, Any] = {} + if args.dynamo_traces: + plans, extra_stats = _export_dynamo_traces(args, window_start, window_end, link_window, stats) + else: + plans = _export_chains(args, window_start, window_end, link_window, stats) plans = sample_plans(plans, args.sample, args.seed, args.stratify_by_session_length) stats.sampled = len(plans) + summary = ", ".join(f"{key}={value}" for key, value in {**stats.as_dict(), **extra_stats}.items()) if not plans: - summary = ", ".join(f"{key}={value}" for key, value in stats.as_dict().items()) raise ExportError(f"no trajectories to export ({summary})") problems = validate_plans(plans) if problems: @@ -862,10 +846,107 @@ def main(argv: Optional[Sequence[str]] = None) -> int: print(f"error: {error}", file=sys.stderr) return 1 - summary = ", ".join(f"{key}={value}" for key, value in stats.as_dict().items()) print(f"wrote {written} trajectories to {args.output} ({summary})", file=sys.stderr) return 0 +def _dump_rows(rows: Iterable[Mapping[str, Any]], path: str) -> None: + with open(path, "w", encoding="utf-8") as handle: + for row in rows: + handle.write(json.dumps(row, default=_json_default) + "\n") + + +def _export_dynamo_traces( + args: argparse.Namespace, + window_start: datetime, + window_end: datetime, + link_window: timedelta, + stats: ExportStats, +) -> tuple[list[dict[str, Any]], dict[str, Any]]: + from . import dynamo_traces + + if args.rows_jsonl or args.chains_table: + raise ExportError("--dynamo-traces replaces --rows-jsonl and --chains-table") + if args.served_by: + raise ExportError("--served-by is not available with --dynamo-traces") + if args.analytics_rows_jsonl and args.analytics_table: + raise ExportError("use --analytics-rows-jsonl or --analytics-table, not both") + trace_stats = dynamo_traces.TraceStats() + records = dynamo_traces.load_trace_records( + args.dynamo_traces, window_start, window_end, link_window, args.model, trace_stats + ) + records = dynamo_traces.drop_duplicate_attempts(records, trace_stats) + analytics_rows: Optional[list[dict[str, Any]]] = None + if args.analytics_rows_jsonl: + analytics_rows = read_rows_jsonl(args.analytics_rows_jsonl) + elif args.analytics_table: + if not args.clickhouse_url: + raise ExportError("--analytics-table needs --clickhouse-url (or CLICKHOUSE_URL)") + bounds = dynamo_traces.analytics_window(records) + analytics_rows = [] + if bounds is not None: + sql = dynamo_traces.build_analytics_query( + args.analytics_table, args.analytics_id_column, args.analytics_ts_column + ) + parameters = {"start": bounds[0].replace(tzinfo=None), "end": bounds[1].replace(tzinfo=None)} + wanted = {record.x_request_id for record in records if record.x_request_id} + analytics_rows = dynamo_traces.fetch_analytics_rows( + args.clickhouse_url, args.clickhouse_user, args.clickhouse_password, sql, parameters, wanted + ) + if args.dump_rows_jsonl: + _dump_rows(analytics_rows, args.dump_rows_jsonl) + if analytics_rows is not None: + dynamo_traces.apply_analytics(records, analytics_rows, trace_stats) + if args.principal_id: + records = [record for record in records if record.principal_id == args.principal_id] + plans = dynamo_traces.build_trace_plans( + records, window_start, window_end, args.time_scale, link_window, stats, trace_stats + ) + return plans, trace_stats.as_dict() + + +def _export_chains( + args: argparse.Namespace, + window_start: datetime, + window_end: datetime, + link_window: timedelta, + stats: ExportStats, +) -> list[dict[str, Any]]: + if args.analytics_rows_jsonl: + raise ExportError("--analytics-rows-jsonl needs --dynamo-traces") + if args.rows_jsonl: + rows = read_rows_jsonl(args.rows_jsonl) + else: + if not args.clickhouse_url: + raise ExportError("--clickhouse-url (or CLICKHOUSE_URL) is required without --rows-jsonl") + if not args.chains_table or not args.analytics_table: + raise ExportError("--chains-table and --analytics-table are required without --rows-jsonl") + sql, parameters = build_query( + args.chains_table, + args.analytics_table, + args.principal_id, + args.model, + args.served_by, + chains_final=args.chains_final, + analytics_ts_column=args.analytics_ts_column, + ) + parameters["lookback_start"] = (window_start - link_window).replace(tzinfo=None) + parameters["window_end"] = window_end.replace(tzinfo=None) + rows = fetch_rows(args.clickhouse_url, args.clickhouse_user, args.clickhouse_password, sql, parameters) + if args.dump_rows_jsonl: + _dump_rows(rows, args.dump_rows_jsonl) + + records = records_from_rows(rows) + if args.rows_jsonl: + records = [ + record + for record in records + if (not args.principal_id or record.principal_id == args.principal_id) + and (not args.model or record.model == args.model) + and (not args.served_by or record.served_by == args.served_by) + ] + return build_plans(records, window_start, window_end, args.time_scale, link_window, stats) + + if __name__ == "__main__": raise SystemExit(main()) diff --git a/tests/test_dynamo_traces.py b/tests/test_dynamo_traces.py new file mode 100644 index 0000000..bc05470 --- /dev/null +++ b/tests/test_dynamo_traces.py @@ -0,0 +1,385 @@ +"""Tests for the Dynamo request-trace source. All traces are synthetic: hashes are made here.""" + +from __future__ import annotations + +import gzip +import hashlib +import json +import shutil +import subprocess +import tempfile +import unittest +from datetime import datetime, timedelta, timezone +from pathlib import Path + +from batchbench import export_plans +from batchbench.dynamo_traces import ( + TraceStats, + analytics_window, + apply_analytics, + build_analytics_query, + build_trace_plans, + drop_duplicate_attempts, + hash_seed, + load_trace_records, + normalize_request_id, + record_from_trace, + segment_cuts, +) +from batchbench.export_plans import ExportStats, validate_plans + +from test_export_plans import find_agent_binary, word_level_tokenizer_json + +T0 = datetime(2026, 9, 1, 12, 0, 0, tzinfo=timezone.utc) +WINDOW_START = T0 +WINDOW_END = T0 + timedelta(hours=1) +MODEL = "example-org/model-a" +OTHER_MODEL = "example-org/model-b" +USER_A = "aaaaaaaa-0000-4000-8000-000000000001" +USER_B = "bbbbbbbb-0000-4000-8000-000000000002" + + +def chain_of(labels, block_size=64): + """Cumulative 64-bit hashes over block labels, like Dynamo's sequence-aware hashes.""" + + hashes = [] + previous = f"bs{block_size}".encode() + for label in labels: + digest = hashlib.blake2b(previous + b"|" + label.encode(), digest_size=8).digest() + hashes.append(int.from_bytes(digest, "big")) + previous = digest + return hashes + + +_request_counter = [0] + + +def trace( + received, + labels, + block_size=64, + tail=10, + output_tokens=20, + total_time_ms=1000.0, + model=MODEL, + x_request_id=None, + session_id=None, + hashes=None, + finish_reason="stop", +): + _request_counter[0] += 1 + hashes = chain_of(labels, block_size) if hashes is None else hashes + request = { + "request_id": f"req-{_request_counter[0]:04d}", + "model": model, + "request_received_ms": int(received.timestamp() * 1000), + "total_time_ms": total_time_ms, + "ttft_ms": 50.0, + "input_tokens": block_size * len(hashes) + tail, + "output_tokens": output_tokens, + "cached_tokens": 0, + "worker": {"decode_worker_id": 1}, + "replay": { + "trace_block_size": block_size, + "input_length": block_size * len(hashes) + tail, + "input_sequence_hashes": hashes, + }, + } + if finish_reason: + request["finish_reason_metadata"] = {"finish_reason": finish_reason} + if x_request_id is not None: + request["x_request_id"] = x_request_id + row = { + "schema": "dynamo.request.trace.v1", + "event_type": "request_end", + "event_time_unix_ms": request["request_received_ms"] + int(total_time_ms), + "event_source": "dynamo", + "request": request, + } + if session_id is not None: + row["agent_context"] = {"session_id": session_id, "input_trigger": "user"} + return row + + +def records_for(rows, stats=None): + stats = stats if stats is not None else TraceStats() + records = [record_from_trace(row, stats) for row in rows] + return drop_duplicate_attempts([record for record in records if record is not None], stats) + + +def plans_for(rows, analytics=None, stats=None, trace_stats=None): + trace_stats = trace_stats if trace_stats is not None else TraceStats() + records = records_for(rows, trace_stats) + if analytics is not None: + apply_analytics(records, analytics, trace_stats) + return build_trace_plans(records, WINDOW_START, WINDOW_END, stats=stats, trace_stats=trace_stats) + + +def seeds(labels, block_size=64): + return [hash_seed(value) for value in chain_of(labels, block_size)] + + +class BlockTest(unittest.TestCase): + def test_prompts_are_cut_where_chains_diverge_or_end(self): + first = trace(T0, ["s1", "s2", "u1"], session_id="sess-1") + second = trace(T0 + timedelta(seconds=5), ["s1", "s2", "u1", "x1", "x2"], tail=5, session_id="sess-1") + branch = trace(T0 + timedelta(seconds=9), ["s1", "s2", "v1"], tail=0) + trace_stats = TraceStats() + plans = plans_for([first, second, branch], trace_stats=trace_stats) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(len(plans), 2) + session, other = plans + s2, u1 = seeds(["s1", "s2", "u1"])[1:] + x2 = seeds(["s1", "s2", "u1", "x1", "x2"])[-1] + v1 = seeds(["s1", "s2", "v1"])[-1] + + turn1, turn2 = session["requests"] + # The shared system prefix is one segment ending where the branch diverges. + self.assertEqual(turn1["blocks"][:2], [ + {"seed": s2, "tokens": 128, "role": "user"}, + {"seed": u1, "tokens": 64, "role": "user"}, + ]) + tail1 = turn1["blocks"][2] + self.assertEqual((tail1["tokens"], tail1["role"]), (10, "user")) + self.assertEqual(turn1["prompt_tokens"], 3 * 64 + 10) + self.assertEqual(turn1["overhead_tokens"], 0) + + # The extension replays the previous tail, then the live reply, then the new content. + self.assertNotIn("reset_before", turn2) + self.assertEqual(turn2["blocks"][:3], turn1["blocks"]) + live = turn2["blocks"][3] + self.assertEqual((live["tokens"], live["role"], live["live"]), (20, "assistant", True)) + rest = turn2["blocks"][4] + self.assertEqual(rest, {"seed": f"{x2}:rest{128 - 10 - 20}", "tokens": 98, "role": "user"}) + self.assertEqual(turn2["blocks"][5]["tokens"], 5) + self.assertEqual(sum(block["tokens"] for block in turn2["blocks"]), turn2["prompt_tokens"]) + self.assertEqual(turn2["overhead_tokens"], 0) + self.assertEqual(trace_stats.live_blocks, 1) + + # The other branch shares the system segment's seed, so its bytes replay identically. + self.assertEqual([block["seed"] for block in other["requests"][0]["blocks"]], [s2, v1]) + self.assertEqual(session["metadata"], {"source": "dynamo-request-trace", "model": MODEL, "block_size": 64}) + + def test_cuts_are_per_model_and_block_size(self): + records = records_for([ + trace(T0, ["a", "b"]), + trace(T0, ["a", "c"]), + trace(T0, ["a", "b"], model=OTHER_MODEL), + ]) + cuts = segment_cuts(records) + a, b = chain_of(["a", "b"]) + c = chain_of(["a", "c"])[1] + self.assertEqual(cuts[(MODEL, 64)], {a, b, c}) + self.assertEqual(cuts[(OTHER_MODEL, 64)], {b}) + + def test_a_reply_longer_than_the_extension_is_clamped(self): + first = trace(T0, ["s1"], tail=30, output_tokens=500, session_id="sess-1") + second = trace(T0 + timedelta(seconds=5), ["s1", "u2"], tail=7, session_id="sess-1") + trace_stats = TraceStats() + plans = plans_for([first, second], trace_stats=trace_stats) + blocks = plans[0]["requests"][1]["blocks"] + # 64 + 7 extension tokens: 30 are the previous tail, the remaining 41 the clamped reply. + self.assertEqual([(block["tokens"], block["role"], block.get("live", False)) for block in blocks], [ + (64, "user", False), (30, "user", False), (41, "assistant", True), + ]) + self.assertEqual(trace_stats.live_clamped, 1) + self.assertEqual(validate_plans(plans), []) + + +class LinkingTest(unittest.TestCase): + def test_non_default_block_size_and_mixed_block_sizes_never_link(self): + short = trace(T0, ["p1", "p2"], block_size=1056, tail=100) + hashes = chain_of(["p1", "p2", "p3"], 1056) + longer = trace(T0 + timedelta(seconds=3), [], block_size=1056, hashes=hashes, tail=0) + # Same hash values under another block size: a different pool, never the same session. + foreign = trace(T0 + timedelta(seconds=6), [], block_size=128, hashes=hashes + [7], tail=0) + plans = plans_for([short, longer, foreign]) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(sorted(len(plan["requests"]) for plan in plans), [1, 2]) + linked = next(plan for plan in plans if len(plan["requests"]) == 2) + self.assertEqual(linked["metadata"]["block_size"], 1056) + self.assertEqual(linked["requests"][0]["blocks"][0]["tokens"], 2 * 1056) + alone = next(plan for plan in plans if len(plan["requests"]) == 1) + self.assertEqual(alone["metadata"]["block_size"], 128) + self.assertEqual(sum(block["tokens"] for block in alone["requests"][0]["blocks"]), 4 * 128) + self.assertFalse(any(block.get("live") for block in alone["requests"][0]["blocks"])) + + def test_sessions_and_principals_separate_otherwise_matching_chains(self): + first = trace(T0, ["s1", "u1"], x_request_id="00000000-0000-7000-8000-00000000000A") + second = trace(T0 + timedelta(seconds=5), ["s1", "u1", "x1"], x_request_id="00000000-0000-7000-8000-00000000000b") + # Without the join both rows have no principal and link on their chains alone. + self.assertEqual([len(plan["requests"]) for plan in plans_for([first, second])], [2]) + + analytics = [ + {"x_request_id": "00000000-0000-7000-8000-00000000000a", "instance_id": "i-1", "correlation_id": 1, "model": "alias-a", "user_id": USER_A, "api_key_id": None}, + {"x_request_id": "00000000-0000-7000-8000-00000000000b", "instance_id": "i-1", "correlation_id": 2, "model": "alias-a", "user_id": USER_B, "api_key_id": None}, + ] + trace_stats = TraceStats() + plans = plans_for([first, second], analytics=analytics, trace_stats=trace_stats) + self.assertEqual([len(plan["requests"]) for plan in plans], [1, 1]) + self.assertEqual(trace_stats.analytics_matched, 2) + self.assertEqual(plans[0]["metadata"]["alias"], "alias-a") + self.assertEqual(plans[0]["metadata"]["principal"], USER_A) + + # Agent session ids keep two sessions apart even without a principal. + one = trace(T0, ["s1", "u1"], session_id="sess-1") + two = trace(T0 + timedelta(seconds=5), ["s1", "u1", "x1"], session_id="sess-2") + self.assertEqual([len(plan["requests"]) for plan in plans_for([one, two])], [1, 1]) + + def test_retried_attempts_keep_the_last_one_that_finished(self): + request_id = "00000000-0000-7000-8000-0000000000c1" + failed = trace(T0, ["s1", "u1"], output_tokens=0, finish_reason=None, x_request_id=request_id) + final = trace(T0 + timedelta(seconds=2), ["s1", "u1"], output_tokens=33, x_request_id=request_id.upper()) + unrelated = trace(T0 + timedelta(seconds=3), ["s1", "w1"]) + trace_stats = TraceStats() + records = records_for([failed, final, unrelated], trace_stats) + self.assertEqual(trace_stats.duplicate_attempts, 1) + self.assertEqual(trace_stats.with_x_request_id, 2) + self.assertEqual(len(records), 2) + kept = next(record for record in records if record.x_request_id) + self.assertEqual((kept.completion_tokens, kept.x_request_id), (33, request_id)) + self.assertEqual([record.index for record in records], [0, 1]) + + +class SourceTest(unittest.TestCase): + def setUp(self): + self.directory = Path(tempfile.mkdtemp(prefix="batchbench-traces-")) + + def tearDown(self): + shutil.rmtree(self.directory, ignore_errors=True) + + def write(self, name, rows, gz=True): + path = self.directory / name + path.parent.mkdir(parents=True, exist_ok=True) + data = "".join(json.dumps(row) + "\n" for row in rows).encode() + path.write_bytes(gzip.compress(data) if gz else data) + return path + + def test_rows_are_windowed_by_receipt_not_file_name(self): + # Files are named by upload time; the 13:10 upload holds a row received at 11:55. + self.write("2026/09/01/frontend-a-121000-run1-0.jsonl.gz", [trace(T0 + timedelta(minutes=5), ["s1", "u1", "x1"], session_id="sess-1")]) + self.write("2026/09/01/frontend-a-131000-run1-1.jsonl.gz", [ + trace(T0 - timedelta(minutes=5), ["s1", "u1"], session_id="sess-1"), + trace(T0 + timedelta(hours=1, minutes=1), ["s1", "late"]), + trace(T0 - timedelta(hours=30), ["s1", "stale"]), + trace(T0 + timedelta(minutes=7), ["s1", "u1"], model=OTHER_MODEL), + {"schema": "dynamo.request.trace.v1", "event_type": "tool_start", "event_time_unix_ms": 0}, + ]) + (self.directory / "2026/09/01/broken.jsonl.gz").write_bytes(gzip.compress(b"{}\n")[:-12]) + trace_stats = TraceStats() + records = load_trace_records([str(self.directory)], WINDOW_START, WINDOW_END, model=MODEL, stats=trace_stats) + self.assertEqual(trace_stats.files, 3) + self.assertEqual(trace_stats.unreadable_files, 1) + self.assertEqual(trace_stats.loaded, 2) + self.assertEqual(trace_stats.outside_window, 2) + self.assertEqual(trace_stats.other_model, 1) + self.assertEqual(trace_stats.not_request_end, 1) + self.assertEqual(trace_stats.block_sizes, {64: 2}) + + stats = ExportStats() + plans = build_trace_plans(drop_duplicate_attempts(records, trace_stats), WINDOW_START, WINDOW_END, stats=stats, trace_stats=trace_stats) + self.assertEqual(len(plans), 1) + self.assertEqual(stats.prewindow_sessions, 1) + first = plans[0]["requests"][0] + self.assertTrue(first["reset_before"]) + self.assertFalse(any(block.get("live") for block in first["blocks"])) + self.assertEqual(plans[0]["start_after_ms"], 5 * 60_000) + + def test_glob_and_missing_paths(self): + self.write("a.jsonl", [trace(T0, ["s1"])], gz=False) + records = load_trace_records([str(self.directory / "*.jsonl")], WINDOW_START, WINDOW_END) + self.assertEqual(len(records), 1) + with self.assertRaises(export_plans.ExportError): + load_trace_records([str(self.directory / "nothing-*.gz")], WINDOW_START, WINDOW_END) + + def run_cli(self, *extra): + output = self.directory / "plans.jsonl" + code = export_plans.main([ + "--dynamo-traces", str(self.directory / "traces"), + "--start", WINDOW_START.isoformat(), + "--end", WINDOW_END.isoformat(), + "--model", MODEL, + "--output", str(output), + *extra, + ]) + lines = [json.loads(line) for line in output.read_text().splitlines()] if output.exists() else [] + return code, lines + + def session_rows(self): + return [ + trace(T0, ["s1", "u1"], x_request_id="00000000-0000-7000-8000-000000000001", session_id="sess-1"), + trace(T0 + timedelta(seconds=4), ["s1", "u1", "x1"], x_request_id="00000000-0000-7000-8000-000000000002", session_id="sess-1"), + trace(T0 + timedelta(seconds=2), ["s1", "q1"], block_size=256, tail=200), + trace(T0 + timedelta(seconds=3), ["s1", "q1", "q2"], block_size=256, tail=3), + ] + + def test_cli_exports_from_traces_alone_and_with_analytics_rows(self): + self.write("traces/frontend-a-120100-run1-0.jsonl.gz", self.session_rows()) + code, plans = self.run_cli() + self.assertEqual(code, 0) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(sorted(len(plan["requests"]) for plan in plans), [2, 2]) + self.assertTrue(all("principal" not in plan["metadata"] for plan in plans)) + + analytics = self.directory / "analytics.jsonl" + analytics.write_text(json.dumps({"x_request_id": "00000000-0000-7000-8000-000000000001", "instance_id": "i-1", "correlation_id": 9, "model": "alias-a", "user_id": USER_A, "api_key_id": None}) + "\n") + code, plans = self.run_cli("--analytics-rows-jsonl", str(analytics)) + self.assertEqual(code, 0) + # One turn matched a principal, the other did not, so the session splits. + self.assertEqual(sorted(len(plan["requests"]) for plan in plans), [1, 1, 2]) + self.assertEqual(validate_plans(plans), []) + code, plans = self.run_cli("--analytics-rows-jsonl", str(analytics), "--principal-id", USER_A) + self.assertEqual((code, len(plans)), (0, 1)) + self.assertEqual(plans[0]["metadata"]["alias"], "alias-a") + + def test_cli_rejects_mixed_sources(self): + self.write("traces/a.jsonl.gz", self.session_rows()) + code, _ = self.run_cli("--served-by", "upstream-a") + self.assertEqual(code, 1) + code, _ = self.run_cli("--analytics-table", "requests") + self.assertEqual(code, 1) # no ClickHouse URL + + def test_exported_manifest_dry_runs_in_batchbench_agent(self): + binary = find_agent_binary() + if binary is None: + self.skipTest("batchbench-agent binary not built") + self.write("traces/a.jsonl.gz", self.session_rows()) + code, _ = self.run_cli() + self.assertEqual(code, 0) + tokenizer = self.directory / "tokenizer.json" + tokenizer.write_text(json.dumps(word_level_tokenizer_json()), encoding="utf-8") + result = subprocess.run( + [binary, "--model", "test-model", "--tokenizer-model", str(tokenizer), "--host", "http://127.0.0.1:9", + "--agent-plans-jsonl", str(self.directory / "plans.jsonl"), "--admission", "open-loop", + "--time-scale", "1000", "--dry-run"], + capture_output=True, text=True, timeout=120, + ) + self.assertEqual(result.returncode, 0, result.stderr + result.stdout) + self.assertIn("Agents: 2 completed / 2 total", result.stdout) + # Each session's second turn replays the first turn's real reply. + self.assertEqual(result.stdout.count("live_blocks=1"), 2) + + +class AnalyticsQueryTest(unittest.TestCase): + def test_query_and_window(self): + sql = build_analytics_query("requests", "x_request_id", "timestamp") + self.assertIn("toString(x_request_id) AS x_request_id", sql) + self.assertIn("FROM requests", sql) + self.assertIn("timestamp >= {start:DateTime64(3)}", sql) + self.assertIn("x_request_id IS NOT NULL", sql) + records = records_for([ + trace(T0, ["a"], x_request_id="00000000-0000-7000-8000-000000000001"), + trace(T0 + timedelta(minutes=30), ["b"], total_time_ms=60_000.0, x_request_id="00000000-0000-7000-8000-000000000002"), + trace(T0 + timedelta(hours=5), ["c"]), + ]) + start, end = analytics_window(records) + self.assertEqual(start, T0 - timedelta(minutes=15)) + self.assertEqual(end, T0 + timedelta(minutes=46)) + self.assertIsNone(analytics_window(records_for([trace(T0, ["d"])]))) + self.assertEqual(normalize_request_id(" 0000000A-0000-7000-8000-000000000001 "), "0000000a-0000-7000-8000-000000000001") + self.assertEqual(normalize_request_id("not-a-uuid"), "not-a-uuid") + self.assertIsNone(normalize_request_id("")) + + +if __name__ == "__main__": + unittest.main() From c43fa98de0cbedaac7f35b060f649a7900c8f076 Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 16:51:54 +0000 Subject: [PATCH 2/9] fix(export): keep earlier turns' blocks and do not replay a clamped reply live A continuation now repeats the blocks emitted for the previous request, so the third and later turns keep earlier replies and tails instead of re-cutting the shared prefix into plain user segments. A previous reply that does not fit in the extension is generated at the size that fits, since a live block always replays the whole real reply. --- docs/export-plans.md | 15 +++++++------ src/batchbench/dynamo_traces.py | 38 ++++++++++++++++++++------------- tests/test_dynamo_traces.py | 28 ++++++++++++++++++++---- 3 files changed, 56 insertions(+), 25 deletions(-) diff --git a/docs/export-plans.md b/docs/export-plans.md index 4d68f3d..88ef474 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -134,11 +134,13 @@ own, so `sum(blocks.tokens) = prompt_tokens` and `overhead_tokens` is 0. A final rather than overhead keeps the replayed prompt the recorded length, which matters for block sizes in the thousands. -A request that extends the previous request of its trajectory repeats that request's -blocks, including its tail block, and the tokens that follow start with a `live` -assistant block of the previous request's `output_tokens`, then the rest as `user` -blocks. When the extension holds fewer tokens than the previous output, the live block -is clamped to fit and counted as `live_clamped`. Each trajectory carries +A request that extends the previous request of its trajectory repeats the blocks +emitted for that request (its replies and tail keep their seeds and roles), and the +tokens that follow start with a `live` assistant block of the previous request's +`output_tokens`, then the rest as `user` blocks. A live block replays the whole real +reply, so when the extension holds fewer tokens than the previous output, the reply is +generated at the size that fits instead, with no live block, and counted as +`live_clamped`. Each trajectory carries `metadata` with the model name, block size and, when joined, the alias and principal. ### Joining analytics on `x_request_id` @@ -193,7 +195,8 @@ Besides the usual counts, the summary line reports, for traces: `files`, can be linked to an unrelated longer request that shares its full blocks. - **Reply length is approximate.** The live block is the previous `output_tokens`; when a backend drops reasoning from the history, the reply in the next prompt is - shorter than that, and the new user content is undercounted by the difference. + shorter than that, and the new user content is undercounted by the difference. When + the reply does not fit at all, it is generated rather than replayed live. - **No `stream` or `max_tokens`.** Traces do not record them, so they are omitted. - **Timing starts at Dynamo.** `request_received_ms` is when the frontend received the request, after any gateway queueing. diff --git a/src/batchbench/dynamo_traces.py b/src/batchbench/dynamo_traces.py index 745ab71..7165bf8 100644 --- a/src/batchbench/dynamo_traces.py +++ b/src/batchbench/dynamo_traces.py @@ -12,8 +12,8 @@ size, or where another chain ends. Each segment becomes one ``user`` block whose seed is the segment's last hash, so prompts that share a prefix replay with identical bytes. * The tokens past the last full block become a final ``user`` block of their own. -* In a request that extends the previous one, the first ``output_tokens`` tokens after the - previous prompt (and its tail) become the ``live`` assistant block. +* A request that extends the previous one repeats that request's blocks, and the first + ``output_tokens`` tokens after them become the ``live`` assistant block. An optional analytics join on ``x_request_id`` fills the principal and the model alias. The expected input is documented in ``docs/export-plans.md``. @@ -514,14 +514,16 @@ def _user(seed: str, tokens: int) -> dict[str, Any]: def trace_request_blocks( record: TraceRecord, previous: Optional[TraceRecord], + previous_blocks: Sequence[Mapping[str, Any]], cuts: set[int], stats: TraceStats, ) -> list[dict[str, Any]]: """Blocks for one request; ``previous`` is the request it extends, if any. - In an extension the tokens after the previous prompt's full blocks are, in order, the - previous prompt's tail (same seed as there), the previous reply (``live``, as many tokens as - the previous output, clamped to what the extension holds), and new user content. + An extension repeats ``previous_blocks``, the blocks emitted for the previous request (so + earlier replies and tails keep their seeds and roles). The tokens after them are the previous + reply, as a ``live`` block of the previous output, and then new user content. The previous + tail is already in ``previous_blocks``, so it is carved off the extension first. """ segments = record_segments(record, cuts) @@ -530,19 +532,21 @@ def trace_request_blocks( return [_user(seed, tokens) for _, seed, tokens in segments] + [_user(seed, tokens) for seed, tokens in tail] shared = len(previous.chain) - blocks = [_user(seed, tokens) for end, seed, tokens in segments if end <= shared] + blocks = [dict(block) for block in previous_blocks] extension = [(seed, tokens, tokens) for end, seed, tokens in segments if end > shared] extension += [(seed, tokens, tokens) for seed, tokens in tail] - if previous.tail_tokens > 0: - blocks.append(_user(tail_seed(previous), previous.tail_tokens)) - extension = _carve(extension, previous.tail_tokens) + extension = _carve(extension, previous.tail_tokens) reply = int(previous.completion_tokens or 0) - live = min(reply, sum(left for _, _, left in extension)) - if live < reply: + available = sum(left for _, _, left in extension) + if reply > available: + # The live block replays the whole real reply, which would not fit: generate it instead. stats.live_clamped += 1 - if live > 0: - blocks.append({"seed": f"{_request_anchor(previous)}:reply{live}", "tokens": live, "role": "assistant", "live": True}) - extension = _carve(extension, live) + if available > 0: + blocks.append({"seed": f"{_request_anchor(previous)}:reply{available}", "tokens": available, "role": "assistant"}) + extension = [] + elif reply > 0: + blocks.append({"seed": f"{_request_anchor(previous)}:reply{reply}", "tokens": reply, "role": "assistant", "live": True}) + extension = _carve(extension, reply) stats.live_blocks += 1 blocks.extend(_piece_block(*piece) for piece in extension) return blocks @@ -572,12 +576,15 @@ def trace_trajectory_plan( first_offset_ms = (requests[0].ts - window_start).total_seconds() * 1000.0 plan_requests = [] previous: Optional[TraceRecord] = None + previous_blocks: list[dict[str, Any]] = [] for position, record in enumerate(requests): if position == 0: reset_before = trajectory.began_before_window else: reset_before = not extends(previous.chain, record.chain) # type: ignore[union-attr] - blocks = trace_request_blocks(record, None if reset_before else previous, group_cuts, stats) + blocks = trace_request_blocks( + record, None if reset_before else previous, previous_blocks, group_cuts, stats + ) prompt_tokens = int(record.prompt_tokens or 0) request: dict[str, Any] = { "prompt_tokens": prompt_tokens, @@ -592,6 +599,7 @@ def trace_trajectory_plan( request["blocks"] = blocks plan_requests.append(request) previous = record + previous_blocks = blocks metadata: dict[str, Any] = {"source": "dynamo-request-trace", "model": root.model, "block_size": root.block_size} if root.alias: diff --git a/tests/test_dynamo_traces.py b/tests/test_dynamo_traces.py index bc05470..61e142c 100644 --- a/tests/test_dynamo_traces.py +++ b/tests/test_dynamo_traces.py @@ -172,17 +172,37 @@ def test_cuts_are_per_model_and_block_size(self): self.assertEqual(cuts[(MODEL, 64)], {a, b, c}) self.assertEqual(cuts[(OTHER_MODEL, 64)], {b}) - def test_a_reply_longer_than_the_extension_is_clamped(self): + def test_later_turns_repeat_the_blocks_sent_before(self): + rows = [ + trace(T0, ["s1", "u1"], tail=10, session_id="sess-1"), + trace(T0 + timedelta(seconds=5), ["s1", "u1", "x1", "x2"], tail=20, session_id="sess-1"), + trace(T0 + timedelta(seconds=9), ["s1", "u1", "x1", "x2", "y1", "y2"], tail=3, session_id="sess-1"), + ] + trace_stats = TraceStats() + plans = plans_for(rows, trace_stats=trace_stats) + self.assertEqual(validate_plans(plans), []) + turn1, turn2, turn3 = plans[0]["requests"] + # Turn three keeps turn two's reply and tails instead of re-cutting its prefix. + self.assertEqual(turn3["blocks"][: len(turn2["blocks"])], turn2["blocks"]) + self.assertEqual(turn2["blocks"][: len(turn1["blocks"])], turn1["blocks"]) + self.assertEqual([block.get("live", False) for block in turn3["blocks"]].count(True), 2) + self.assertTrue(turn3["blocks"][len(turn2["blocks"])]["live"]) + for request in (turn1, turn2, turn3): + self.assertEqual(sum(block["tokens"] for block in request["blocks"]), request["prompt_tokens"]) + self.assertEqual(trace_stats.live_blocks, 2) + + def test_a_reply_longer_than_the_extension_is_generated_not_live(self): first = trace(T0, ["s1"], tail=30, output_tokens=500, session_id="sess-1") second = trace(T0 + timedelta(seconds=5), ["s1", "u2"], tail=7, session_id="sess-1") trace_stats = TraceStats() plans = plans_for([first, second], trace_stats=trace_stats) blocks = plans[0]["requests"][1]["blocks"] - # 64 + 7 extension tokens: 30 are the previous tail, the remaining 41 the clamped reply. + # 64 + 7 extension tokens: 30 are the previous tail, and the 500-token reply would not fit + # in the other 41, so they are generated rather than replaying the real reply. self.assertEqual([(block["tokens"], block["role"], block.get("live", False)) for block in blocks], [ - (64, "user", False), (30, "user", False), (41, "assistant", True), + (64, "user", False), (30, "user", False), (41, "assistant", False), ]) - self.assertEqual(trace_stats.live_clamped, 1) + self.assertEqual((trace_stats.live_clamped, trace_stats.live_blocks), (1, 0)) self.assertEqual(validate_plans(plans), []) From d72fc48cd7e508db5f5e6cfecba8c28ce1e94adb Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 16:53:59 +0000 Subject: [PATCH 3/9] fix(export): split off Dynamo's partial last-block hash Dynamo appends a hash for the partial last block when the prompt length is not a multiple of the block size, so a row has ceil(length / block size) hashes. Such rows were dropped as malformed. The partial hash is now split off: linking and segment cuts use full blocks only, and the partial hash seeds the tail block. --- docs/export-plans.md | 16 ++++++-- src/batchbench/dynamo_traces.py | 13 +++++- tests/test_dynamo_traces.py | 70 +++++++++++++++++++++++++++++++-- 3 files changed, 91 insertions(+), 8 deletions(-) diff --git a/docs/export-plans.md b/docs/export-plans.md index 88ef474..43c07b4 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -106,11 +106,19 @@ Fields read from each row: | `request.input_tokens` | `prompt_tokens` (`replay.input_length` when absent or short of the hashed blocks) | | `request.output_tokens` | `output_tokens`; rows without a positive count are dropped | | `request.replay.trace_block_size` | Tokens per hashed block; varies by deployment | -| `request.replay.input_sequence_hashes` | Cumulative hash of each full block of the prompt | +| `request.replay.input_length` | Hashed prompt length | +| `request.replay.input_sequence_hashes` | Cumulative hash of each full block of the prompt, then of the partial last block when one remains | | `request.x_request_id` | The caller's `x-request-id` header, when it sent one | | `agent_context.session_id` | Agent session, when present | | `request.finish_reason_metadata.finish_reason` | Picks among retried attempts | +There are `ceil(input_length / trace_block_size)` hashes. When the length is not a +multiple of the block size, the last hash covers the partial last block; it changes as +soon as a later turn fills that block, so it is split off and never used for linking or +cutting, only to seed the tail block. Rows that hash only the full blocks are read the +same way. A row with more hashes than that, or whose full blocks hold more tokens than +the prompt, is counted as `malformed_hashes`. + Trace files are named by upload time, and a file can hold rows received long before it was written, so every file is read and rows are windowed by `request_received_ms`: rows in `[--start - --link-window-hours, --end)` are loaded, and those before `--start` @@ -129,8 +137,8 @@ ends or where two such chains continue differently. Each piece becomes one `user whose seed is the piece's last hash and whose size is the block size times its hash count, so a prefix shared by many requests (a system prompt, a common document) replays with identical bytes everywhere. The tokens past the last full block -(`input_tokens - block_size * hashes`) become a final `user` block with a seed of its -own, so `sum(blocks.tokens) = prompt_tokens` and `overhead_tokens` is 0. A final block +(`input_tokens - block_size * full blocks`) become a final `user` block, seeded by the +partial-block hash (so identical prompts share it) or, without one, by the request, so `sum(blocks.tokens) = prompt_tokens` and `overhead_tokens` is 0. A final block rather than overhead keeps the replayed prompt the recorded length, which matters for block sizes in the thousands. @@ -188,7 +196,7 @@ Besides the usual counts, the summary line reports, for traces: `files`, straddles message boundaries, and the live reply starts where the previous prompt's tokens end rather than at a message boundary. - **Large blocks hide divergence.** Two prompts that differ only inside their last - partial block have the same hashes. A turn that adds less than one block keeps its + partial block have the same full-block hashes. A turn that adds less than one block keeps its chain unchanged, so it is not a strict extension and starts a new trajectory; with block sizes in the thousands, many sessions fall apart into single requests. Without a principal or session id, a short request whose distinctive content sits in its tail diff --git a/src/batchbench/dynamo_traces.py b/src/batchbench/dynamo_traces.py index 7165bf8..4184d11 100644 --- a/src/batchbench/dynamo_traces.py +++ b/src/batchbench/dynamo_traces.py @@ -70,6 +70,7 @@ class TraceRecord(ChainRecord): x_request_id: Optional[str] = None session_id: Optional[str] = None alias: Optional[str] = None + partial_hash: Optional[int] = None @property def link_key(self) -> Any: @@ -161,6 +162,12 @@ def record_from_trace( except (TypeError, OverflowError): stats.malformed_hashes += 1 return None + # Dynamo hashes every full block and then, when tokens remain, the partial last block. The + # partial hash changes as soon as a later turn fills that block, so it never links or cuts. + hashed_tokens = input_length if _is_count(input_length) else request.get("input_tokens") + partial_hash = None + if _is_count(hashed_tokens) and hashed_tokens % block_size and len(chain) == hashed_tokens // block_size + 1: + partial_hash = chain.pop() full_tokens = block_size * len(chain) prompt_tokens = request.get("input_tokens") if not _is_count(prompt_tokens) or prompt_tokens < full_tokens: @@ -192,6 +199,7 @@ def record_from_trace( finish_reason=None if finish_reason is None else str(finish_reason), block_size=int(block_size), tail_tokens=int(prompt_tokens) - full_tokens, + partial_hash=partial_hash, request_id=str(request.get("request_id") or ""), x_request_id=normalize_request_id(request.get("x_request_id")), session_id=None if session_id in (None, "") else str(session_id), @@ -484,8 +492,11 @@ def _request_anchor(record: TraceRecord) -> str: def tail_seed(record: TraceRecord) -> str: - """The tokens past the last full block are this request's own; the seed says so.""" + """The tokens past the last full block: seeded by Dynamo's partial-block hash when the row + has one (identical prompts share it), otherwise by the request itself.""" + if record.partial_hash is not None: + return f"{hash_seed(record.partial_hash)}:tail{record.tail_tokens}" return f"{_request_anchor(record)}:tail{record.tail_tokens}" diff --git a/tests/test_dynamo_traces.py b/tests/test_dynamo_traces.py index 61e142c..1983259 100644 --- a/tests/test_dynamo_traces.py +++ b/tests/test_dynamo_traces.py @@ -51,6 +51,12 @@ def chain_of(labels, block_size=64): return hashes +def partial_of(labels, tail_label, block_size=64): + """The hash Dynamo appends for a partial last block: the full chain plus the tail's tokens.""" + + return chain_of(list(labels) + [f"partial:{tail_label}"], block_size)[-1] + + _request_counter = [0] @@ -66,22 +72,28 @@ def trace( session_id=None, hashes=None, finish_reason="stop", + partial=True, ): + """A request_end row shaped like production: with ``partial`` (the default) and a non-empty + tail, the hashes end with one for the partial last block, so there are ceil(length / block + size) of them.""" + _request_counter[0] += 1 - hashes = chain_of(labels, block_size) if hashes is None else hashes + full = chain_of(labels, block_size) if hashes is None else list(hashes) + hashes = full + ([partial_of(labels, tail, block_size) if labels else 1] if partial and tail else []) request = { "request_id": f"req-{_request_counter[0]:04d}", "model": model, "request_received_ms": int(received.timestamp() * 1000), "total_time_ms": total_time_ms, "ttft_ms": 50.0, - "input_tokens": block_size * len(hashes) + tail, + "input_tokens": block_size * len(full) + tail, "output_tokens": output_tokens, "cached_tokens": 0, "worker": {"decode_worker_id": 1}, "replay": { "trace_block_size": block_size, - "input_length": block_size * len(hashes) + tail, + "input_length": block_size * len(full) + tail, "input_sequence_hashes": hashes, }, } @@ -206,6 +218,58 @@ def test_a_reply_longer_than_the_extension_is_generated_not_live(self): self.assertEqual(validate_plans(plans), []) +class HashShapeTest(unittest.TestCase): + def test_partial_last_block_hash_is_split_off(self): + stats = TraceStats() + ceil_row = trace(T0, ["a", "b"], tail=10) + self.assertEqual(len(ceil_row["request"]["replay"]["input_sequence_hashes"]), 3) + record = record_from_trace(ceil_row, stats) + self.assertEqual(list(record.chain), chain_of(["a", "b"])) + self.assertEqual((record.tail_tokens, record.partial_hash), (10, partial_of(["a", "b"], 10))) + + exact = trace(T0, ["a", "b"], tail=0) + self.assertEqual(len(exact["request"]["replay"]["input_sequence_hashes"]), 2) + record = record_from_trace(exact, stats) + self.assertEqual((len(record.chain), record.tail_tokens, record.partial_hash), (2, 0, None)) + + # Rows without the partial hash (floor) still load, with the tail unhashed. + floor_row = trace(T0, ["a", "b"], tail=10, partial=False) + record = record_from_trace(floor_row, stats) + self.assertEqual((len(record.chain), record.tail_tokens, record.partial_hash), (2, 10, None)) + self.assertEqual(stats.malformed_hashes, 0) + + # More hashes than ceil(length / block size), or a partial hash on an exact multiple. + too_many = trace(T0, ["a", "b"], tail=10) + too_many["request"]["replay"]["input_sequence_hashes"].append(5) + exact_with_extra = trace(T0, ["a", "b"], tail=0) + exact_with_extra["request"]["replay"]["input_sequence_hashes"].append(5) + self.assertIsNone(record_from_trace(too_many, stats)) + self.assertIsNone(record_from_trace(exact_with_extra, stats)) + self.assertEqual(stats.malformed_hashes, 2) + + def test_a_turn_extending_a_partial_last_block_links_and_seeds_its_tail(self): + first = trace(T0, ["s1", "u1"], tail=40) + # Turn two fills turn one's partial block, so its third hash differs from turn one's + # partial hash: only full blocks may be required to match. + second = trace(T0 + timedelta(seconds=5), ["s1", "u1", "u1-filled", "x1"], tail=12) + self.assertNotIn( + first["request"]["replay"]["input_sequence_hashes"][-1], + second["request"]["replay"]["input_sequence_hashes"], + ) + trace_stats = TraceStats() + plans = plans_for([first, second], trace_stats=trace_stats) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(len(plans), 1) + turn1, turn2 = plans[0]["requests"] + tail_block = turn1["blocks"][-1] + self.assertEqual(tail_block, {"seed": f"{hash_seed(partial_of(['s1', 'u1'], 40))}:tail40", "tokens": 40, "role": "user"}) + self.assertNotIn("reset_before", turn2) + self.assertEqual(turn2["blocks"][: len(turn1["blocks"])], turn1["blocks"]) + self.assertTrue(turn2["blocks"][len(turn1["blocks"])]["live"]) + self.assertEqual(sum(block["tokens"] for block in turn2["blocks"]), turn2["prompt_tokens"]) + self.assertEqual(trace_stats.live_blocks, 1) + + class LinkingTest(unittest.TestCase): def test_non_default_block_size_and_mixed_block_sizes_never_link(self): short = trace(T0, ["p1", "p2"], block_size=1056, tail=100) From ca98538a362a9025f982b4aff9d296ec31b2a8f9 Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 16:59:08 +0000 Subject: [PATCH 4/9] fix(export): reuse sent blocks on branches; tighten trace parsing - A branch in a trajectory starts with the blocks already sent for the longest prefix it shares with an earlier request, so it replays the same bytes, live replies included. - Segments are found in one pass over the chain instead of one index search per cut. - Rows with more hashes than ceil(input_length / block size) are malformed, whatever input_tokens says. - The model prefilter is used only for ASCII names, which no writer escapes. - --principal-id with --dynamo-traces needs an analytics join. - Docstrings name link keys and both ClickHouse readers. --- docs/export-plans.md | 12 ++++--- src/batchbench/dynamo_traces.py | 58 +++++++++++++++++++++++++++++---- src/batchbench/export_plans.py | 17 ++++++---- tests/test_dynamo_traces.py | 43 +++++++++++++++++++++++- 4 files changed, 112 insertions(+), 18 deletions(-) diff --git a/docs/export-plans.md b/docs/export-plans.md index 43c07b4..936b643 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -116,8 +116,8 @@ There are `ceil(input_length / trace_block_size)` hashes. When the length is not multiple of the block size, the last hash covers the partial last block; it changes as soon as a later turn fills that block, so it is split off and never used for linking or cutting, only to seed the tail block. Rows that hash only the full blocks are read the -same way. A row with more hashes than that, or whose full blocks hold more tokens than -the prompt, is counted as `malformed_hashes`. +same way. A row with more hashes than `ceil(input_length / trace_block_size)`, or whose full +blocks hold more tokens than the prompt, is counted as `malformed_hashes`. Trace files are named by upload time, and a file can hold rows received long before it was written, so every file is read and rows are windowed by `request_received_ms`: @@ -148,7 +148,10 @@ tokens that follow start with a `live` assistant block of the previous request's `output_tokens`, then the rest as `user` blocks. A live block replays the whole real reply, so when the extension holds fewer tokens than the previous output, the reply is generated at the size that fits instead, with no live block, and counted as -`live_clamped`. Each trajectory carries +`live_clamped`. Any other request in a trajectory (a branch, which carries +`reset_before`) starts with the blocks already sent for the longest prefix it shares +with an earlier request of the trajectory, so a branch replays the same bytes as the +turn it branches from, earlier live replies included. Each trajectory carries `metadata` with the model name, block size and, when joined, the alias and principal. ### Joining analytics on `x_request_id` @@ -160,7 +163,8 @@ are joined to a gateway analytics table on `--analytics-id-column` (default `timestamp`) to the time span of the rows with an id, widened by 15 minutes on each side. The id is compared as a string, case-insensitively. A matched row's `user_id` (else its `api_key_id`) becomes the principal, which links sessions and is what -`--principal-id` filters on, and its `model` becomes the trajectory's `alias`. +`--principal-id` filters on, and its `model` becomes the trajectory's `alias`. Principals come only from the join, so +`--principal-id` needs one. Unmatched rows keep no principal and link by agent session id alone, if they have one. `--dump-rows-jsonl` saves the analytics rows and `--analytics-rows-jsonl` joins against such a file instead of ClickHouse: diff --git a/src/batchbench/dynamo_traces.py b/src/batchbench/dynamo_traces.py index 4184d11..1216aaa 100644 --- a/src/batchbench/dynamo_traces.py +++ b/src/batchbench/dynamo_traces.py @@ -166,8 +166,12 @@ def record_from_trace( # partial hash changes as soon as a later turn fills that block, so it never links or cuts. hashed_tokens = input_length if _is_count(input_length) else request.get("input_tokens") partial_hash = None - if _is_count(hashed_tokens) and hashed_tokens % block_size and len(chain) == hashed_tokens // block_size + 1: - partial_hash = chain.pop() + if _is_count(hashed_tokens): + if len(chain) > -(-hashed_tokens // block_size): + stats.malformed_hashes += 1 + return None + if hashed_tokens % block_size and len(chain) == hashed_tokens // block_size + 1: + partial_hash = chain.pop() full_tokens = block_size * len(chain) prompt_tokens = request.get("input_tokens") if not _is_count(prompt_tokens) or prompt_tokens < full_tokens: @@ -267,7 +271,8 @@ def load_trace_records( (window_start - link_window).timestamp() * 1000.0, window_end.timestamp() * 1000.0, ) - model_needle = None if model is None else json.dumps(model, ensure_ascii=False) + # A cheap substring test before parsing. Only for ASCII names: a writer may escape others. + model_needle = json.dumps(model) if model is not None and model.isascii() else None records: list[TraceRecord] = [] for path in trace_files(paths): for line in _trace_lines(path, stats): @@ -474,7 +479,7 @@ def record_segments(record: TraceRecord, cuts: set[int]) -> list[tuple[int, str, """``(end position, seed, tokens)`` for each segment of the record's full blocks.""" chain = record.chain - positions = sorted(chain.index(hash_) for hash_ in cuts.intersection(chain)) # type: ignore[attr-defined] + positions = [position for position, hash_ in enumerate(chain) if hash_ in cuts] segments = [] start = 0 for position in positions: @@ -522,12 +527,18 @@ def _user(seed: str, tokens: int) -> dict[str, Any]: return {"seed": seed, "tokens": int(tokens), "role": "user"} +# Per trajectory: for a segment seed, the blocks already sent that render the prompt exactly up +# to the end of that segment, as (blocks, count). +Renderings = dict[str, tuple[list[dict[str, Any]], int]] + + def trace_request_blocks( record: TraceRecord, previous: Optional[TraceRecord], previous_blocks: Sequence[Mapping[str, Any]], cuts: set[int], stats: TraceStats, + renderings: Optional[Renderings] = None, ) -> list[dict[str, Any]]: """Blocks for one request; ``previous`` is the request it extends, if any. @@ -535,12 +546,28 @@ def trace_request_blocks( earlier replies and tails keep their seeds and roles). The tokens after them are the previous reply, as a ``live`` block of the previous output, and then new user content. The previous tail is already in ``previous_blocks``, so it is carved off the extension first. + + Any other request starts from the longest prefix it shares with a request already sent in the + trajectory, rendered with the blocks sent then (``renderings``), so a branch replays the same + bytes as the turn it branches from. ``renderings`` is updated with this request's blocks. """ segments = record_segments(record, cuts) tail = [(tail_seed(record), record.tail_tokens)] if record.tail_tokens > 0 else [] + renderings = renderings if renderings is not None else {} if previous is None: - return [_user(seed, tokens) for _, seed, tokens in segments] + [_user(seed, tokens) for seed, tokens in tail] + blocks: list[dict[str, Any]] = [] + shared = 0 + for end, seed, _ in reversed(segments): + rendering = renderings.get(seed) + if rendering is not None: + blocks = [dict(block) for block in rendering[0][: rendering[1]]] + shared = end + break + blocks += [_user(seed, tokens) for end, seed, tokens in segments if end > shared] + blocks += [_user(seed, tokens) for seed, tokens in tail] + _record_renderings(record, segments, blocks, renderings) + return blocks shared = len(previous.chain) blocks = [dict(block) for block in previous_blocks] @@ -560,9 +587,27 @@ def trace_request_blocks( extension = _carve(extension, reply) stats.live_blocks += 1 blocks.extend(_piece_block(*piece) for piece in extension) + _record_renderings(record, segments, blocks, renderings) return blocks +def _record_renderings( + record: TraceRecord, + segments: Sequence[tuple[int, str, int]], + blocks: list[dict[str, Any]], + renderings: Renderings, +) -> None: + """Remember, for each segment end that falls on a block boundary, the blocks up to it.""" + + ends = {record.block_size * end: seed for end, seed, _ in segments} + total = 0 + for count, block in enumerate(blocks, start=1): + total += block["tokens"] + seed = ends.get(total) + if seed is not None: + renderings.setdefault(seed, (blocks, count)) + + # --------------------------------------------------------------------------- # Plans @@ -588,13 +633,14 @@ def trace_trajectory_plan( plan_requests = [] previous: Optional[TraceRecord] = None previous_blocks: list[dict[str, Any]] = [] + renderings: Renderings = {} for position, record in enumerate(requests): if position == 0: reset_before = trajectory.began_before_window else: reset_before = not extends(previous.chain, record.chain) # type: ignore[union-attr] blocks = trace_request_blocks( - record, None if reset_before else previous, previous_blocks, group_cuts, stats + record, None if reset_before else previous, previous_blocks, group_cuts, stats, renderings ) prompt_tokens = int(record.prompt_tokens or 0) request: dict[str, Any] = { diff --git a/src/batchbench/export_plans.py b/src/batchbench/export_plans.py index 77aef1a..371cf3d 100644 --- a/src/batchbench/export_plans.py +++ b/src/batchbench/export_plans.py @@ -18,9 +18,9 @@ ``--time-scale``. :mod:`batchbench.dynamo_traces` builds the same manifests from Dynamo request traces. The expected -input schemas are documented in ``docs/export-plans.md``. The ClickHouse client is -confined to :func:`fetch_rows`; everything else works on plain row dictionaries so it can be -tested with fixtures. +input schemas are documented in ``docs/export-plans.md``. ClickHouse is read only by +:func:`fetch_rows` here and ``fetch_analytics_rows`` there; everything else works on plain row +dictionaries so it can be tested with fixtures. """ from __future__ import annotations @@ -248,17 +248,18 @@ def link_parents( ) -> dict[int, Optional[int]]: """Map each record index to the index of the request it continues, or ``None`` for a root. - Within a principal, the parent of B is the latest earlier request (within ``link_window``) + Within a link key (the principal, for chain records), the parent of B is the latest earlier + request (within ``link_window``) whose full chain is a strict prefix of B's chain. Because chain hashes are cumulative, A's chain is a prefix of B's exactly when A's leaf hash appears in B's chain before B's leaf. """ parents: dict[int, Optional[int]] = {} - by_principal: dict[Any, list[ChainRecord]] = {} + by_link_key: dict[Any, list[ChainRecord]] = {} for record in records: - by_principal.setdefault(record.link_key, []).append(record) + by_link_key.setdefault(record.link_key, []).append(record) - for principal_records in by_principal.values(): + for principal_records in by_link_key.values(): principal_records.sort(key=_record_order) latest_by_leaf: dict[Any, ChainRecord] = {} leaves: set[Any] = set() @@ -869,6 +870,8 @@ def _export_dynamo_traces( raise ExportError("--dynamo-traces replaces --rows-jsonl and --chains-table") if args.served_by: raise ExportError("--served-by is not available with --dynamo-traces") + if args.principal_id and not (args.analytics_table or args.analytics_rows_jsonl): + raise ExportError("--principal-id with --dynamo-traces needs an analytics join, the only source of principals") if args.analytics_rows_jsonl and args.analytics_table: raise ExportError("use --analytics-rows-jsonl or --analytics-table, not both") trace_stats = dynamo_traces.TraceStats() diff --git a/tests/test_dynamo_traces.py b/tests/test_dynamo_traces.py index 1983259..b385fa7 100644 --- a/tests/test_dynamo_traces.py +++ b/tests/test_dynamo_traces.py @@ -203,6 +203,31 @@ def test_later_turns_repeat_the_blocks_sent_before(self): self.assertEqual(sum(block["tokens"] for block in request["blocks"]), request["prompt_tokens"]) self.assertEqual(trace_stats.live_blocks, 2) + def test_a_branch_replays_the_blocks_already_sent_for_its_shared_prefix(self): + first = trace(T0, ["a"], tail=10, session_id="sess-1") + extension = trace(T0 + timedelta(seconds=5), ["a", "b", "c"], tail=4, session_id="sess-1") + branch = trace(T0 + timedelta(seconds=9), ["a", "b", "d"], tail=6, session_id="sess-1") + plans = plans_for([first, extension, branch]) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(len(plans), 1) + _, turn2, turn3 = plans[0]["requests"] + self.assertTrue(turn3["reset_before"]) + # Turn two renders blocks a and b as a, the first tail, the live reply and the rest of b. + b_end = 2 * 64 + prefix, total = [], 0 + for block in turn2["blocks"]: + if total == b_end: + break + prefix.append(block) + total += block["tokens"] + self.assertEqual(total, b_end) + self.assertTrue(any(block.get("live") for block in prefix)) + # The branch shares those 128 hashed tokens, so it reuses the same seeds (the agent + # replays a seed it has sent, live reply included, with the same bytes). + self.assertEqual(turn3["blocks"][: len(prefix)], prefix) + self.assertEqual(turn3["blocks"][len(prefix)]["seed"], seeds(["a", "b", "d"])[-1]) + self.assertEqual(sum(block["tokens"] for block in turn3["blocks"]), turn3["prompt_tokens"]) + def test_a_reply_longer_than_the_extension_is_generated_not_live(self): first = trace(T0, ["s1"], tail=30, output_tokens=500, session_id="sess-1") second = trace(T0 + timedelta(seconds=5), ["s1", "u2"], tail=7, session_id="sess-1") @@ -243,9 +268,14 @@ def test_partial_last_block_hash_is_split_off(self): too_many["request"]["replay"]["input_sequence_hashes"].append(5) exact_with_extra = trace(T0, ["a", "b"], tail=0) exact_with_extra["request"]["replay"]["input_sequence_hashes"].append(5) + # A large input_tokens does not excuse more hashes than input_length has blocks. + inflated = trace(T0, ["a", "b"], tail=10) + inflated["request"]["replay"]["input_sequence_hashes"].append(5) + inflated["request"]["input_tokens"] = 10_000 self.assertIsNone(record_from_trace(too_many, stats)) self.assertIsNone(record_from_trace(exact_with_extra, stats)) - self.assertEqual(stats.malformed_hashes, 2) + self.assertIsNone(record_from_trace(inflated, stats)) + self.assertEqual(stats.malformed_hashes, 3) def test_a_turn_extending_a_partial_last_block_links_and_seeds_its_tail(self): first = trace(T0, ["s1", "u1"], tail=40) @@ -369,6 +399,15 @@ def test_rows_are_windowed_by_receipt_not_file_name(self): self.assertFalse(any(block.get("live") for block in first["blocks"])) self.assertEqual(plans[0]["start_after_ms"], 5 * 60_000) + def test_non_ascii_model_names_match_escaped_or_not(self): + name = "example-org/modèle-a" + escaped = json.dumps(trace(T0, ["s1"], model=name)) # ensure_ascii escapes the name + plain = json.dumps(trace(T0 + timedelta(seconds=1), ["s2"], model=name), ensure_ascii=False) + path = self.directory / "mixed.jsonl" + path.write_text(escaped + "\n" + plain + "\n", encoding="utf-8") + records = load_trace_records([str(path)], WINDOW_START, WINDOW_END, model=name) + self.assertEqual(len(records), 2) + def test_glob_and_missing_paths(self): self.write("a.jsonl", [trace(T0, ["s1"])], gz=False) records = load_trace_records([str(self.directory / "*.jsonl")], WINDOW_START, WINDOW_END) @@ -422,6 +461,8 @@ def test_cli_rejects_mixed_sources(self): self.assertEqual(code, 1) code, _ = self.run_cli("--analytics-table", "requests") self.assertEqual(code, 1) # no ClickHouse URL + code, _ = self.run_cli("--principal-id", USER_A) + self.assertEqual(code, 1) # principals come only from the join def test_exported_manifest_dry_runs_in_batchbench_agent(self): binary = find_agent_binary() From 8fe2d05948f01c8aafb9cb715b0a4d001ff12ad7 Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 17:01:43 +0000 Subject: [PATCH 5/9] docs(export): explain reply sizing for reasoning models --- docs/export-plans.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/export-plans.md b/docs/export-plans.md index 936b643..ec52ae8 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -205,10 +205,16 @@ Besides the usual counts, the summary line reports, for traces: `files`, block sizes in the thousands, many sessions fall apart into single requests. Without a principal or session id, a short request whose distinctive content sits in its tail can be linked to an unrelated longer request that shares its full blocks. -- **Reply length is approximate.** The live block is the previous `output_tokens`; - when a backend drops reasoning from the history, the reply in the next prompt is - shorter than that, and the new user content is undercounted by the difference. When - the reply does not fit at all, it is generated rather than replayed live. +- **Reply length is overstated for reasoning models.** The reply is sized from the + previous `output_tokens`, which for a reasoning model include the reasoning tokens. + Clients usually do not send reasoning back, so the copy of the reply in the next + prompt is shorter than that. The trace has no reasoning token count to subtract, so + the reply block is oversized and eats into the new user or tool content that follows. + When the reply does not fit in the extension at all, it is generated at the size that + fits rather than replayed live. `live_clamped` in the summary counts those cases: for + a reasoning model expect most extensions to be clamped (for example, most of the + extensions of a hypothetical `example-org/reasoner-a`), which means the live replies + are not replayed and each turn's new content is undercounted. - **No `stream` or `max_tokens`.** Traces do not record them, so they are omitted. - **Timing starts at Dynamo.** `request_received_ms` is when the frontend received the request, after any gateway queueing. From b5290359ce4b1c39746b324fae16537fc9f04b66 Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 17:01:52 +0000 Subject: [PATCH 6/9] docs(export): state the live_clamped signal plainly --- docs/export-plans.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/export-plans.md b/docs/export-plans.md index ec52ae8..887ab1a 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -211,10 +211,9 @@ Besides the usual counts, the summary line reports, for traces: `files`, prompt is shorter than that. The trace has no reasoning token count to subtract, so the reply block is oversized and eats into the new user or tool content that follows. When the reply does not fit in the extension at all, it is generated at the size that - fits rather than replayed live. `live_clamped` in the summary counts those cases: for - a reasoning model expect most extensions to be clamped (for example, most of the - extensions of a hypothetical `example-org/reasoner-a`), which means the live replies - are not replayed and each turn's new content is undercounted. + fits rather than replayed live. `live_clamped` in the summary counts those cases. When it is + a large share of `live_clamped + live_blocks`, as is typical for a reasoning model, + most replies are not replayed live and each turn's new content is undercounted. - **No `stream` or `max_tokens`.** Traces do not record them, so they are omitted. - **Timing starts at Dynamo.** `request_received_ms` is when the frontend received the request, after any gateway queueing. From 73edfb3b6b036b1056ec13a4121ba3a74a22718a Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 17:04:18 +0000 Subject: [PATCH 7/9] fix(export): mark only the newest reply live Repeated earlier replies were kept live, so one request could carry dozens of live blocks. They now repeat as plain assistant blocks under the seed they were sent with. The agent replays an already-sent seed with the same bytes, live reply included, so the replay is unchanged within a trajectory and each request has at most one live block. --- docs/export-plans.md | 12 ++++++---- src/batchbench/dynamo_traces.py | 18 ++++++++++----- tests/test_dynamo_traces.py | 39 ++++++++++++++++++++++++--------- 3 files changed, 50 insertions(+), 19 deletions(-) diff --git a/docs/export-plans.md b/docs/export-plans.md index 887ab1a..405b75d 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -148,10 +148,14 @@ tokens that follow start with a `live` assistant block of the previous request's `output_tokens`, then the rest as `user` blocks. A live block replays the whole real reply, so when the extension holds fewer tokens than the previous output, the reply is generated at the size that fits instead, with no live block, and counted as -`live_clamped`. Any other request in a trajectory (a branch, which carries -`reset_before`) starts with the blocks already sent for the longest prefix it shares -with an earlier request of the trajectory, so a branch replays the same bytes as the -turn it branches from, earlier live replies included. Each trajectory carries +`live_clamped`. Only the newest reply in a request is `live`: earlier replies +repeat as plain assistant blocks under the seeds they were sent with, and the agent +replays a seed it has already sent in the trajectory with the same bytes, a real reply +included. Any other request in a trajectory (a branch, which carries `reset_before`) +starts with the blocks already sent for the longest prefix it shares with an earlier +request of the trajectory. A reset clears the agent's cache, so the branch's generated +blocks come out the same from their seeds, but its copies of earlier replies are +generated rather than the model's real text. Each trajectory carries `metadata` with the model name, block size and, when joined, the alias and principal. ### Joining analytics on `x_request_id` diff --git a/src/batchbench/dynamo_traces.py b/src/batchbench/dynamo_traces.py index 1216aaa..bc36dc1 100644 --- a/src/batchbench/dynamo_traces.py +++ b/src/batchbench/dynamo_traces.py @@ -523,6 +523,13 @@ def _piece_block(seed: str, full: int, left: int) -> dict[str, Any]: return _user(seed if left == full else f"{seed}:rest{left}", left) +def _repeated(block: Mapping[str, Any]) -> dict[str, Any]: + """A block sent before, repeated: only the newest reply in a request is ``live``. Within a + trajectory the agent replays a seed it has sent with the same bytes, a live reply included.""" + + return {key: value for key, value in block.items() if key != "live"} + + def _user(seed: str, tokens: int) -> dict[str, Any]: return {"seed": seed, "tokens": int(tokens), "role": "user"} @@ -543,9 +550,10 @@ def trace_request_blocks( """Blocks for one request; ``previous`` is the request it extends, if any. An extension repeats ``previous_blocks``, the blocks emitted for the previous request (so - earlier replies and tails keep their seeds and roles). The tokens after them are the previous - reply, as a ``live`` block of the previous output, and then new user content. The previous - tail is already in ``previous_blocks``, so it is carved off the extension first. + earlier replies and tails keep their seeds and roles, but are no longer ``live``). The tokens + after them are the previous reply, as a ``live`` block of the previous output, and then new + user content. The previous tail is already in ``previous_blocks``, so it is carved off the + extension first. Any other request starts from the longest prefix it shares with a request already sent in the trajectory, rendered with the blocks sent then (``renderings``), so a branch replays the same @@ -561,7 +569,7 @@ def trace_request_blocks( for end, seed, _ in reversed(segments): rendering = renderings.get(seed) if rendering is not None: - blocks = [dict(block) for block in rendering[0][: rendering[1]]] + blocks = [_repeated(block) for block in rendering[0][: rendering[1]]] shared = end break blocks += [_user(seed, tokens) for end, seed, tokens in segments if end > shared] @@ -570,7 +578,7 @@ def trace_request_blocks( return blocks shared = len(previous.chain) - blocks = [dict(block) for block in previous_blocks] + blocks = [_repeated(block) for block in previous_blocks] extension = [(seed, tokens, tokens) for end, seed, tokens in segments if end > shared] extension += [(seed, tokens, tokens) for seed, tokens in tail] extension = _carve(extension, previous.tail_tokens) diff --git a/tests/test_dynamo_traces.py b/tests/test_dynamo_traces.py index b385fa7..b47809d 100644 --- a/tests/test_dynamo_traces.py +++ b/tests/test_dynamo_traces.py @@ -194,10 +194,20 @@ def test_later_turns_repeat_the_blocks_sent_before(self): plans = plans_for(rows, trace_stats=trace_stats) self.assertEqual(validate_plans(plans), []) turn1, turn2, turn3 = plans[0]["requests"] + + def sent(blocks): + return [{key: value for key, value in block.items() if key != "live"} for block in blocks] + # Turn three keeps turn two's reply and tails instead of re-cutting its prefix. - self.assertEqual(turn3["blocks"][: len(turn2["blocks"])], turn2["blocks"]) - self.assertEqual(turn2["blocks"][: len(turn1["blocks"])], turn1["blocks"]) - self.assertEqual([block.get("live", False) for block in turn3["blocks"]].count(True), 2) + self.assertEqual(turn3["blocks"][: len(turn2["blocks"])], sent(turn2["blocks"])) + self.assertEqual(turn2["blocks"][: len(turn1["blocks"])], sent(turn1["blocks"])) + # Only the newest reply is live; turn one's reply repeats under the seed turn two sent + # it with, which the agent replays with the same bytes. + for request in (turn2, turn3): + self.assertEqual([block.get("live", False) for block in request["blocks"]].count(True), 1) + first_reply = next(block for block in turn2["blocks"] if block.get("live")) + repeated = turn3["blocks"][turn2["blocks"].index(first_reply)] + self.assertEqual(repeated, {key: value for key, value in first_reply.items() if key != "live"}) self.assertTrue(turn3["blocks"][len(turn2["blocks"])]["live"]) for request in (turn1, turn2, turn3): self.assertEqual(sum(block["tokens"] for block in request["blocks"]), request["prompt_tokens"]) @@ -222,9 +232,10 @@ def test_a_branch_replays_the_blocks_already_sent_for_its_shared_prefix(self): total += block["tokens"] self.assertEqual(total, b_end) self.assertTrue(any(block.get("live") for block in prefix)) - # The branch shares those 128 hashed tokens, so it reuses the same seeds (the agent - # replays a seed it has sent, live reply included, with the same bytes). - self.assertEqual(turn3["blocks"][: len(prefix)], prefix) + # The branch shares those 128 hashed tokens, so it reuses the same seeds; the repeated + # reply is no longer live. + self.assertEqual(turn3["blocks"][: len(prefix)], [{k: v for k, v in block.items() if k != "live"} for block in prefix]) + self.assertFalse(any(block.get("live") for block in turn3["blocks"])) self.assertEqual(turn3["blocks"][len(prefix)]["seed"], seeds(["a", "b", "d"])[-1]) self.assertEqual(sum(block["tokens"] for block in turn3["blocks"]), turn3["prompt_tokens"]) @@ -468,7 +479,12 @@ def test_exported_manifest_dry_runs_in_batchbench_agent(self): binary = find_agent_binary() if binary is None: self.skipTest("batchbench-agent binary not built") - self.write("traces/a.jsonl.gz", self.session_rows()) + three_turns = [ + trace(T0 + timedelta(seconds=1), ["t1", "t2"], session_id="sess-3"), + trace(T0 + timedelta(seconds=6), ["t1", "t2", "t3"], session_id="sess-3"), + trace(T0 + timedelta(seconds=11), ["t1", "t2", "t3", "t4", "t5"], session_id="sess-3"), + ] + self.write("traces/a.jsonl.gz", self.session_rows() + three_turns) code, _ = self.run_cli() self.assertEqual(code, 0) tokenizer = self.directory / "tokenizer.json" @@ -480,9 +496,12 @@ def test_exported_manifest_dry_runs_in_batchbench_agent(self): capture_output=True, text=True, timeout=120, ) self.assertEqual(result.returncode, 0, result.stderr + result.stdout) - self.assertIn("Agents: 2 completed / 2 total", result.stdout) - # Each session's second turn replays the first turn's real reply. - self.assertEqual(result.stdout.count("live_blocks=1"), 2) + self.assertIn("Agents: 3 completed / 3 total", result.stdout) + # Every later turn has exactly one live block, and none falls back to generated text: + # the three-turn session's third turn repeats the first reply as an already-sent seed. + self.assertEqual(result.stdout.count("live_blocks=1"), 4) + self.assertNotIn("live_blocks=2", result.stdout) + self.assertNotIn("Live block fallbacks", result.stdout) class AnalyticsQueryTest(unittest.TestCase): From 8594ccec0c53f58aeb590e8618ca9468a5c361bc Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 17:07:17 +0000 Subject: [PATCH 8/9] fix(export): split a generated reply where segments end A clamped reply was one generated block, so when it spanned a segment cut no rendering was recorded there and a later branch lost the shared prefix. It is now one generated block per remaining piece. --- docs/export-plans.md | 7 ++++--- src/batchbench/dynamo_traces.py | 8 +++++--- tests/test_dynamo_traces.py | 19 +++++++++++++++++-- 3 files changed, 26 insertions(+), 8 deletions(-) diff --git a/docs/export-plans.md b/docs/export-plans.md index 405b75d..8f54854 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -147,15 +147,16 @@ emitted for that request (its replies and tail keep their seeds and roles), and tokens that follow start with a `live` assistant block of the previous request's `output_tokens`, then the rest as `user` blocks. A live block replays the whole real reply, so when the extension holds fewer tokens than the previous output, the reply is -generated at the size that fits instead, with no live block, and counted as -`live_clamped`. Only the newest reply in a request is `live`: earlier replies +generated at the size that fits instead (split where segments end), with no live +block, and counted as `live_clamped`. Only the newest reply in a request is `live`: earlier replies repeat as plain assistant blocks under the seeds they were sent with, and the agent replays a seed it has already sent in the trajectory with the same bytes, a real reply included. Any other request in a trajectory (a branch, which carries `reset_before`) starts with the blocks already sent for the longest prefix it shares with an earlier request of the trajectory. A reset clears the agent's cache, so the branch's generated blocks come out the same from their seeds, but its copies of earlier replies are -generated rather than the model's real text. Each trajectory carries +generated rather than the model's real text. A live reply cannot be split, so a +branch that diverges inside one reuses the blocks only up to the reply's start. Each trajectory carries `metadata` with the model name, block size and, when joined, the alias and principal. ### Joining analytics on `x_request_id` diff --git a/src/batchbench/dynamo_traces.py b/src/batchbench/dynamo_traces.py index bc36dc1..6ade22a 100644 --- a/src/batchbench/dynamo_traces.py +++ b/src/batchbench/dynamo_traces.py @@ -585,10 +585,12 @@ def trace_request_blocks( reply = int(previous.completion_tokens or 0) available = sum(left for _, _, left in extension) if reply > available: - # The live block replays the whole real reply, which would not fit: generate it instead. + # The live block replays the whole real reply, which would not fit: generate it instead, + # one block per remaining piece so segment ends stay block boundaries for later branches. stats.live_clamped += 1 - if available > 0: - blocks.append({"seed": f"{_request_anchor(previous)}:reply{available}", "tokens": available, "role": "assistant"}) + for seed, full, left in extension: + piece = seed if left == full else f"{seed}:rest{left}" + blocks.append({"seed": f"{piece}:reply", "tokens": left, "role": "assistant"}) extension = [] elif reply > 0: blocks.append({"seed": f"{_request_anchor(previous)}:reply{reply}", "tokens": reply, "role": "assistant", "live": True}) diff --git a/tests/test_dynamo_traces.py b/tests/test_dynamo_traces.py index b47809d..6423ce8 100644 --- a/tests/test_dynamo_traces.py +++ b/tests/test_dynamo_traces.py @@ -239,6 +239,21 @@ def test_a_branch_replays_the_blocks_already_sent_for_its_shared_prefix(self): self.assertEqual(turn3["blocks"][len(prefix)]["seed"], seeds(["a", "b", "d"])[-1]) self.assertEqual(sum(block["tokens"] for block in turn3["blocks"]), turn3["prompt_tokens"]) + def test_a_branch_keeps_a_generated_reply_that_spans_its_cut(self): + first = trace(T0, ["a"], tail=10, output_tokens=500, session_id="sess-1") + extension = trace(T0 + timedelta(seconds=5), ["a", "b", "c"], tail=4, session_id="sess-1") + branch = trace(T0 + timedelta(seconds=9), ["a", "b", "d"], tail=6, session_id="sess-1") + plans = plans_for([first, extension, branch]) + self.assertEqual(validate_plans(plans), []) + _, turn2, turn3 = plans[0]["requests"] + # The 500-token reply does not fit in the 54 + 64 + 4 tokens after the first tail, so it + # is generated, split where block b ends; the branch repeats it up to that cut. + self.assertEqual([(block["tokens"], block["role"]) for block in turn2["blocks"]], [ + (64, "user"), (10, "user"), (54, "assistant"), (64, "assistant"), (4, "assistant"), + ]) + self.assertEqual(turn3["blocks"][:3], turn2["blocks"][:3]) + self.assertEqual(turn3["blocks"][3]["seed"], seeds(["a", "b", "d"])[-1]) + def test_a_reply_longer_than_the_extension_is_generated_not_live(self): first = trace(T0, ["s1"], tail=30, output_tokens=500, session_id="sess-1") second = trace(T0 + timedelta(seconds=5), ["s1", "u2"], tail=7, session_id="sess-1") @@ -246,9 +261,9 @@ def test_a_reply_longer_than_the_extension_is_generated_not_live(self): plans = plans_for([first, second], trace_stats=trace_stats) blocks = plans[0]["requests"][1]["blocks"] # 64 + 7 extension tokens: 30 are the previous tail, and the 500-token reply would not fit - # in the other 41, so they are generated rather than replaying the real reply. + # in the other 41, so they are generated (split where block u2 ends) rather than live. self.assertEqual([(block["tokens"], block["role"], block.get("live", False)) for block in blocks], [ - (64, "user", False), (30, "user", False), (41, "assistant", False), + (64, "user", False), (30, "user", False), (34, "assistant", False), (7, "assistant", False), ]) self.assertEqual((trace_stats.live_clamped, trace_stats.live_blocks), (1, 0)) self.assertEqual(validate_plans(plans), []) From 87b2e78b42e974c1c0553f8061888a3f7d348c10 Mon Sep 17 00:00:00 2001 From: Steffen Ridderbusch Date: Fri, 6 Nov 2026 17:13:41 +0000 Subject: [PATCH 9/9] docs(export): replies are assistant blocks whether live or not --- docs/export-plans.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/export-plans.md b/docs/export-plans.md index 8f54854..b5bba1e 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -192,8 +192,9 @@ Besides the usual counts, the summary line reports, for traces: `files`, ### Fidelity lost against prompt chains -- **No roles.** Every block is `user` except the live reply: system prompts, tool - results and tool calls are not told apart. +- **No roles.** Every block is `user` except the previous requests' replies, which are + `assistant` (live, repeated, or generated): system prompts, tool results and tool + calls are not told apart. - **No tools array.** Tool definitions are part of the hashed prompt, so they replay as user text and the replayed request sends no `tools`. - **Template overhead is merged and double counted.** Trace token counts include the