diff --git a/README.md b/README.md index 9a79954..825a09e 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ Three entrypoints: |---|---| | `batchbench` | Independent requests at a fixed concurrency with controlled token shapes | | `batchbench-agent` | Stateful multi-turn sessions, sampled or replayed from a manifest | -| `batchbench-export-plans` | Builds a replay manifest from prompt-chain records in ClickHouse | +| `batchbench-export-plans` | Builds a replay manifest from prompt-chain records in ClickHouse or from Dynamo request traces | ## Independent requests @@ -117,7 +117,9 @@ batchbench-export-plans \ --output plans.jsonl ``` -The input tables it expects, and how sessions are reconstructed, are described in +It can also build one from a directory of Dynamo frontend request traces +(`--dynamo-traces`), with no ClickHouse needed. The input tables and trace rows it +expects, and how sessions are reconstructed, are described in [docs/export-plans.md](docs/export-plans.md). ## Report diff --git a/docs/export-plans.md b/docs/export-plans.md index 7949444..b5bba1e 100644 --- a/docs/export-plans.md +++ b/docs/export-plans.md @@ -1,7 +1,8 @@ # Exporting a manifest from prompt-chain records -`batchbench-export-plans` reads prompt-chain records from ClickHouse and writes a -[schema version 2 manifest](manifest.md). It needs the `export` extra: +`batchbench-export-plans` reads prompt-chain records from ClickHouse, or Dynamo +request traces from local files ([below](#dynamo-request-traces)), and writes a +[schema version 2 manifest](manifest.md). Reading ClickHouse needs the `export` extra: ```bash uv pip install "batchbench[export]" @@ -79,3 +80,146 @@ Replay with: ```bash batchbench-agent --agent-plans-jsonl plans.jsonl --admission open-loop --model --host ``` + +## Dynamo request traces + +`--dynamo-traces` replaces the prompt-chain source with the `request_end` rows a Dynamo +frontend writes when request tracing is enabled (schema `dynamo.request.trace.v1`, +gzipped or plain JSONL). It takes a directory (read recursively for `*.jsonl.gz` and +`*.jsonl`), a file, or a glob, and may be repeated. No ClickHouse is needed: + +```bash +batchbench-export-plans \ + --dynamo-traces ./traces \ + --model example-org/model-a \ + --start 2026-09-01T09:00:00Z --end 2026-09-01T10:00:00Z \ + --output plans.jsonl +``` + +Fields read from each row: + +| Field | Use | +|---|---| +| `request.model` | Dynamo model name; `--model` matches it exactly | +| `request.request_received_ms` | Request time, used for the window, ordering and offsets | +| `request.total_time_ms` | Duration, for `delay_after_ms` | +| `request.input_tokens` | `prompt_tokens` (`replay.input_length` when absent or short of the hashed blocks) | +| `request.output_tokens` | `output_tokens`; rows without a positive count are dropped | +| `request.replay.trace_block_size` | Tokens per hashed block; varies by deployment | +| `request.replay.input_length` | Hashed prompt length | +| `request.replay.input_sequence_hashes` | Cumulative hash of each full block of the prompt, then of the partial last block when one remains | +| `request.x_request_id` | The caller's `x-request-id` header, when it sent one | +| `agent_context.session_id` | Agent session, when present | +| `request.finish_reason_metadata.finish_reason` | Picks among retried attempts | + +There are `ceil(input_length / trace_block_size)` hashes. When the length is not a +multiple of the block size, the last hash covers the partial last block; it changes as +soon as a later turn fills that block, so it is split off and never used for linking or +cutting, only to seed the tail block. Rows that hash only the full blocks are read the +same way. A row with more hashes than `ceil(input_length / trace_block_size)`, or whose full +blocks hold more tokens than the prompt, is counted as `malformed_hashes`. + +Trace files are named by upload time, and a file can hold rows received long before it +was written, so every file is read and rows are windowed by `request_received_ms`: +rows in `[--start - --link-window-hours, --end)` are loaded, and those before `--start` +only serve as link targets. Rows with the same `x_request_id` are attempts of one +request (a gateway retry or failover reuses the header); the last attempt that finished +is kept and the others are counted as `duplicate_attempts`. + +### Sessions and blocks + +Requests link with the same strict-prefix rule as prompt chains, applied to the block +hashes, and only within the same model, block size, principal and agent session id. +Rows with different block sizes are never linked or merged, even within one model. + +Each prompt is cut after every hash where some chain of the same model and block size +ends or where two such chains continue differently. Each piece becomes one `user` block +whose seed is the piece's last hash and whose size is the block size times its hash +count, so a prefix shared by many requests (a system prompt, a common document) replays +with identical bytes everywhere. The tokens past the last full block +(`input_tokens - block_size * full blocks`) become a final `user` block, seeded by the +partial-block hash (so identical prompts share it) or, without one, by the request, so `sum(blocks.tokens) = prompt_tokens` and `overhead_tokens` is 0. A final block +rather than overhead keeps the replayed prompt the recorded length, which matters for +block sizes in the thousands. + +A request that extends the previous request of its trajectory repeats the blocks +emitted for that request (its replies and tail keep their seeds and roles), and the +tokens that follow start with a `live` assistant block of the previous request's +`output_tokens`, then the rest as `user` blocks. A live block replays the whole real +reply, so when the extension holds fewer tokens than the previous output, the reply is +generated at the size that fits instead (split where segments end), with no live +block, and counted as `live_clamped`. Only the newest reply in a request is `live`: earlier replies +repeat as plain assistant blocks under the seeds they were sent with, and the agent +replays a seed it has already sent in the trajectory with the same bytes, a real reply +included. Any other request in a trajectory (a branch, which carries `reset_before`) +starts with the blocks already sent for the longest prefix it shares with an earlier +request of the trajectory. A reset clears the agent's cache, so the branch's generated +blocks come out the same from their seeds, but its copies of earlier replies are +generated rather than the model's real text. A live reply cannot be split, so a +branch that diverges inside one reuses the blocks only up to the reply's start. Each trajectory carries +`metadata` with the model name, block size and, when joined, the alias and principal. + +### Joining analytics on `x_request_id` + +With `--analytics-table` (and `--clickhouse-url`) the rows that carry an `x_request_id` +are joined to a gateway analytics table on `--analytics-id-column` (default +`x_request_id`). The query reads that table's id, `instance_id`, `correlation_id`, +`model`, `user_id` and `api_key_id`, bounded by `--analytics-ts-column` (default +`timestamp`) to the time span of the rows with an id, widened by 15 minutes on each +side. The id is compared as a string, case-insensitively. A matched row's `user_id` +(else its `api_key_id`) becomes the principal, which links sessions and is what +`--principal-id` filters on, and its `model` becomes the trajectory's `alias`. Principals come only from the join, so +`--principal-id` needs one. +Unmatched rows keep no principal and link by agent session id alone, if they have one. +`--dump-rows-jsonl` saves the analytics rows and `--analytics-rows-jsonl` joins against +such a file instead of ClickHouse: + +```bash +CLICKHOUSE_URL=https://warehouse.example.com:8443 CLICKHOUSE_USER=... CLICKHOUSE_PASSWORD=... \ +batchbench-export-plans \ + --dynamo-traces ./traces --model example-org/model-a \ + --analytics-table requests \ + --start 2026-09-01T09:00:00Z --end 2026-09-01T10:00:00Z \ + --output plans.jsonl +``` + +Besides the usual counts, the summary line reports, for traces: `files`, +`unreadable_files` (a truncated file keeps the rows before the damage), `lines`, +`bad_lines`, `not_request_end`, `no_replay`, `other_model`, `outside_window`, +`malformed_hashes`, `loaded`, `with_x_request_id`, `duplicate_attempts`, +`analytics_rows`, `analytics_matched`, `with_session_id`, `live_blocks`, +`live_clamped`, and `block_sizes` (rows loaded per block size). + +### Fidelity lost against prompt chains + +- **No roles.** Every block is `user` except the previous requests' replies, which are + `assistant` (live, repeated, or generated): system prompts, tool results and tool + calls are not told apart. +- **No tools array.** Tool definitions are part of the hashed prompt, so they replay as + user text and the replayed request sends no `tools`. +- **Template overhead is merged and double counted.** Trace token counts include the + chat template. The replay sends every block as its own message and the backend adds + its template on top, so the replayed prompt exceeds `prompt_tokens` by the replay's + per-message template cost, and the recorded per-message overhead cannot be separated + from content. +- **Granularity is one trace block.** Cuts fall only on block boundaries, so a block + straddles message boundaries, and the live reply starts where the previous prompt's + tokens end rather than at a message boundary. +- **Large blocks hide divergence.** Two prompts that differ only inside their last + partial block have the same full-block hashes. A turn that adds less than one block keeps its + chain unchanged, so it is not a strict extension and starts a new trajectory; with + block sizes in the thousands, many sessions fall apart into single requests. Without + a principal or session id, a short request whose distinctive content sits in its tail + can be linked to an unrelated longer request that shares its full blocks. +- **Reply length is overstated for reasoning models.** The reply is sized from the + previous `output_tokens`, which for a reasoning model include the reasoning tokens. + Clients usually do not send reasoning back, so the copy of the reply in the next + prompt is shorter than that. The trace has no reasoning token count to subtract, so + the reply block is oversized and eats into the new user or tool content that follows. + When the reply does not fit in the extension at all, it is generated at the size that + fits rather than replayed live. `live_clamped` in the summary counts those cases. When it is + a large share of `live_clamped + live_blocks`, as is typical for a reasoning model, + most replies are not replayed live and each turn's new content is undercounted. +- **No `stream` or `max_tokens`.** Traces do not record them, so they are omitted. +- **Timing starts at Dynamo.** `request_received_ms` is when the frontend received the + request, after any gateway queueing. diff --git a/src/batchbench/dynamo_traces.py b/src/batchbench/dynamo_traces.py new file mode 100644 index 0000000..6ade22a --- /dev/null +++ b/src/batchbench/dynamo_traces.py @@ -0,0 +1,718 @@ +"""Build schema version 2 manifests from Dynamo frontend request traces. + +A Dynamo frontend with request tracing enabled writes one ``request_end`` row +(``dynamo.request.trace.v1``) per request: timings, token counts, and the KV router's +cumulative block hashes of the prompt (``replay.input_sequence_hashes``, one per full block of +``replay.trace_block_size`` tokens). The hashes play the role of a prompt chain, at block rather +than message granularity, so sessions are rebuilt with the same strict-prefix rule as +:mod:`batchbench.export_plans`: + +* Requests link only within the same model, block size, principal, and agent session. +* Every prompt is cut wherever its chain diverges from another chain of the same model and block + size, or where another chain ends. Each segment becomes one ``user`` block whose seed is the + segment's last hash, so prompts that share a prefix replay with identical bytes. +* The tokens past the last full block become a final ``user`` block of their own. +* A request that extends the previous one repeats that request's blocks, and the first + ``output_tokens`` tokens after them become the ``live`` assistant block. + +An optional analytics join on ``x_request_id`` fills the principal and the model alias. The +expected input is documented in ``docs/export-plans.md``. +""" + +from __future__ import annotations + +import glob +import gzip +import json +import os +import uuid +import zlib +from array import array +from dataclasses import dataclass, field +from datetime import datetime, timedelta, timezone +from typing import Any, Iterable, Iterator, Mapping, Optional, Sequence + +from .export_plans import ( + DEFAULT_LINK_WINDOW, + SCHEMA_VERSION, + ChainRecord, + ExportError, + ExportStats, + Trajectory, + build_trajectories, + extends, + link_parents, + scale_ms, + usable_records, +) + +TRACE_SCHEMA = "dynamo.request.trace.v1" +TRACE_EVENT = "request_end" +TRACE_SUFFIXES = (".jsonl.gz", ".jsonl") +DEFAULT_ANALYTICS_ID_COLUMN = "x_request_id" +# The analytics timestamp is when the gateway received the request, the trace's when Dynamo did; +# retries and failover land later still. +ANALYTICS_PAD = timedelta(minutes=15) +ANALYTICS_COLUMNS = ("x_request_id", "instance_id", "correlation_id", "model", "user_id", "api_key_id") + + +# --------------------------------------------------------------------------- +# Records + + +@dataclass +class TraceRecord(ChainRecord): + """One ``request_end`` row. ``chain`` holds the block hashes as unsigned 64-bit integers.""" + + block_size: int = 0 + tail_tokens: int = 0 + request_id: str = "" + x_request_id: Optional[str] = None + session_id: Optional[str] = None + alias: Optional[str] = None + partial_hash: Optional[int] = None + + @property + def link_key(self) -> Any: + return (self.model, self.block_size, self.principal_id, self.session_id or "") + + @property + def end_ms(self) -> float: + return self.ts.timestamp() * 1000.0 + (self.duration_ms or 0.0) + + +@dataclass +class TraceStats: + files: int = 0 + unreadable_files: int = 0 + lines: int = 0 + bad_lines: int = 0 + not_request_end: int = 0 + no_replay: int = 0 + other_model: int = 0 + outside_window: int = 0 + malformed_hashes: int = 0 + loaded: int = 0 + with_x_request_id: int = 0 + duplicate_attempts: int = 0 + analytics_rows: int = 0 + analytics_matched: int = 0 + with_session_id: int = 0 + live_blocks: int = 0 + live_clamped: int = 0 + block_sizes: dict[int, int] = field(default_factory=dict) + + def as_dict(self) -> dict[str, Any]: + values: dict[str, Any] = {key: value for key, value in self.__dict__.items() if key != "block_sizes"} + values["block_sizes"] = ",".join(f"{size}:{count}" for size, count in sorted(self.block_sizes.items())) or "none" + return values + + +def normalize_request_id(value: Any) -> Optional[str]: + """Canonical lower-case form of an ``x-request-id``; UUIDs in their hyphenated form.""" + + if value is None: + return None + text = str(value).strip().lower() + if not text: + return None + try: + return str(uuid.UUID(text)) + except ValueError: + return text + + +def _is_count(value: Any) -> bool: + return isinstance(value, int) and not isinstance(value, bool) and value >= 0 + + +def record_from_trace( + row: Mapping[str, Any], + stats: TraceStats, + model: Optional[str] = None, + window_ms: Optional[tuple[float, float]] = None, +) -> Optional[TraceRecord]: + """Build a :class:`TraceRecord` from one parsed trace line, or count why it was skipped.""" + + if row.get("schema") != TRACE_SCHEMA or row.get("event_type") != TRACE_EVENT: + stats.not_request_end += 1 + return None + request = row.get("request") + replay = request.get("replay") if isinstance(request, Mapping) else None + received_ms = request.get("request_received_ms") if isinstance(request, Mapping) else None + if not isinstance(replay, Mapping) or not _is_count(received_ms): + stats.no_replay += 1 + return None + row_model = str(request.get("model") or "") + if model is not None and row_model != model: + stats.other_model += 1 + return None + if window_ms is not None and not (window_ms[0] <= received_ms < window_ms[1]): + stats.outside_window += 1 + return None + + block_size = replay.get("trace_block_size") + hashes = replay.get("input_sequence_hashes") or [] + input_length = replay.get("input_length") + if not _is_count(block_size) or block_size == 0 or not isinstance(hashes, list): + stats.malformed_hashes += 1 + return None + try: + chain = array("Q", hashes) + except (TypeError, OverflowError): + stats.malformed_hashes += 1 + return None + # Dynamo hashes every full block and then, when tokens remain, the partial last block. The + # partial hash changes as soon as a later turn fills that block, so it never links or cuts. + hashed_tokens = input_length if _is_count(input_length) else request.get("input_tokens") + partial_hash = None + if _is_count(hashed_tokens): + if len(chain) > -(-hashed_tokens // block_size): + stats.malformed_hashes += 1 + return None + if hashed_tokens % block_size and len(chain) == hashed_tokens // block_size + 1: + partial_hash = chain.pop() + full_tokens = block_size * len(chain) + prompt_tokens = request.get("input_tokens") + if not _is_count(prompt_tokens) or prompt_tokens < full_tokens: + prompt_tokens = input_length + if not _is_count(prompt_tokens) or prompt_tokens < full_tokens: + stats.malformed_hashes += 1 + return None + + agent = row.get("agent_context") if isinstance(row.get("agent_context"), Mapping) else {} + session_id = agent.get("session_id") + finish = request.get("finish_reason_metadata") + finish_reason = finish.get("finish_reason") if isinstance(finish, Mapping) else None + total_time_ms = request.get("total_time_ms") + output_tokens = request.get("output_tokens") + record = TraceRecord( + ts=datetime.fromtimestamp(received_ms / 1000.0, tz=timezone.utc), + instance_id="", + correlation_id=0, + principal_id="", + model=row_model, + chain=chain, # type: ignore[arg-type] + block_roles=[], + block_tokens=[], + prompt_tokens=int(prompt_tokens), + completion_tokens=int(output_tokens) if _is_count(output_tokens) else None, + duration_ms=float(total_time_ms) if isinstance(total_time_ms, (int, float)) and not isinstance(total_time_ms, bool) else None, + stream=None, + max_tokens=None, + finish_reason=None if finish_reason is None else str(finish_reason), + block_size=int(block_size), + tail_tokens=int(prompt_tokens) - full_tokens, + partial_hash=partial_hash, + request_id=str(request.get("request_id") or ""), + x_request_id=normalize_request_id(request.get("x_request_id")), + session_id=None if session_id in (None, "") else str(session_id), + ) + stats.loaded += 1 + stats.block_sizes[record.block_size] = stats.block_sizes.get(record.block_size, 0) + 1 + if record.x_request_id: + stats.with_x_request_id += 1 + if record.session_id: + stats.with_session_id += 1 + return record + + +def trace_files(paths: Sequence[str]) -> list[str]: + """Expand directories (recursively) and glob patterns into trace files, sorted.""" + + found: set[str] = set() + for path in paths: + if os.path.isdir(path): + for directory, _, names in os.walk(path): + found.update(os.path.join(directory, name) for name in names if name.endswith(TRACE_SUFFIXES)) + elif os.path.isfile(path): + found.add(path) + else: + matches = [match for match in glob.glob(path, recursive=True) if os.path.isfile(match)] + if not matches: + raise ExportError(f"no trace files match {path!r}") + found.update(matches) + if not found: + raise ExportError("no trace files found (expected *.jsonl.gz or *.jsonl)") + return sorted(found) + + +def _trace_lines(path: str, stats: TraceStats) -> Iterator[str]: + stats.files += 1 + try: + if path.endswith(".gz"): + handle = gzip.open(path, "rt", encoding="utf-8") + else: + handle = open(path, "r", encoding="utf-8") + with handle: + for line in handle: + yield line + except (OSError, EOFError, zlib.error, UnicodeDecodeError): + # A truncated upload keeps the rows read before the damage. + stats.unreadable_files += 1 + + +def load_trace_records( + paths: Sequence[str], + window_start: datetime, + window_end: datetime, + link_window: timedelta = DEFAULT_LINK_WINDOW, + model: Optional[str] = None, + stats: Optional[TraceStats] = None, +) -> list[TraceRecord]: + """Read ``request_end`` rows received in ``[window_start - link_window, window_end)``. + + Files are named by upload time and can hold rows received long before it, so every file is + read and rows are windowed by ``request_received_ms``. + """ + + stats = stats if stats is not None else TraceStats() + window_ms = ( + (window_start - link_window).timestamp() * 1000.0, + window_end.timestamp() * 1000.0, + ) + # A cheap substring test before parsing. Only for ASCII names: a writer may escape others. + model_needle = json.dumps(model) if model is not None and model.isascii() else None + records: list[TraceRecord] = [] + for path in trace_files(paths): + for line in _trace_lines(path, stats): + if not line.strip(): + continue + stats.lines += 1 + if f'"{TRACE_EVENT}"' not in line: + stats.not_request_end += 1 + continue + if model_needle is not None and model_needle not in line: + stats.other_model += 1 + continue + try: + row = json.loads(line) + except ValueError: + stats.bad_lines += 1 + continue + if not isinstance(row, Mapping): + stats.bad_lines += 1 + continue + record = record_from_trace(row, stats, model, window_ms) + if record is not None: + records.append(record) + return records + + +def _attempt_rank(record: TraceRecord) -> tuple: + return ( + bool(record.completion_tokens), + record.finish_reason is not None, + record.end_ms, + record.request_id, + ) + + +def drop_duplicate_attempts(records: Iterable[TraceRecord], stats: TraceStats) -> list[TraceRecord]: + """Keep one row per ``x_request_id``: the last attempt that finished. + + A gateway retry or failover inside one client request reuses its ``x-request-id``, so the + failed attempts and the final one all have rows. Rows are then ordered and re-indexed. + """ + + kept: list[TraceRecord] = [] + best: dict[str, TraceRecord] = {} + with_id = 0 + for record in records: + if not record.x_request_id: + kept.append(record) + continue + with_id += 1 + current = best.get(record.x_request_id) + if current is None or _attempt_rank(record) > _attempt_rank(current): + best[record.x_request_id] = record + stats.duplicate_attempts += with_id - len(best) + kept.extend(best.values()) + kept.sort(key=lambda record: (record.ts, record.request_id, record.x_request_id or "")) + for index, record in enumerate(kept): + record.index = index + return kept + + +# --------------------------------------------------------------------------- +# Analytics join + + +def build_analytics_query(table: str, id_column: str = DEFAULT_ANALYTICS_ID_COLUMN, ts_column: str = "timestamp") -> str: + """Analytics rows with an ``x-request-id`` in ``[{start}, {end})``; columns as ``ANALYTICS_COLUMNS``.""" + + return f""" +SELECT + toString({id_column}) AS x_request_id, + toString(instance_id) AS instance_id, + correlation_id, + model, + toString(user_id) AS user_id, + toString(api_key_id) AS api_key_id +FROM {table} +WHERE {ts_column} >= {{start:DateTime64(3)}} + AND {ts_column} < {{end:DateTime64(3)}} + AND {id_column} IS NOT NULL +""".strip() + + +def analytics_window(records: Iterable[TraceRecord]) -> Optional[tuple[datetime, datetime]]: + """The analytics time range that covers every record with an ``x_request_id``.""" + + starts = [] + ends = [] + for record in records: + if record.x_request_id: + starts.append(record.ts) + ends.append(record.end_ts) + if not starts: + return None + return min(starts) - ANALYTICS_PAD, max(ends) + ANALYTICS_PAD + + +def fetch_analytics_rows( + url: str, + user: Optional[str], + password: Optional[str], + sql: str, + parameters: Mapping[str, Any], + wanted: set[str], +) -> list[dict[str, Any]]: + """Stream the analytics query and keep the rows whose id is in ``wanted``.""" + + try: + import clickhouse_connect # type: ignore[import-not-found] + except ImportError as error: # pragma: no cover - exercised only without the extra + raise ExportError("clickhouse-connect is required to read ClickHouse; install batchbench[export]") from error + + client_args: dict[str, Any] = {"dsn": url} + if user: + client_args["username"] = user + if password: + client_args["password"] = password + client = clickhouse_connect.get_client(**client_args) + rows = [] + try: + with client.query_rows_stream(sql, parameters=dict(parameters)) as stream: + for values in stream: + row = dict(zip(ANALYTICS_COLUMNS, values)) + if normalize_request_id(row["x_request_id"]) in wanted: + rows.append(row) + finally: + client.close() + return rows + + +def apply_analytics(records: Iterable[TraceRecord], rows: Iterable[Mapping[str, Any]], stats: TraceStats) -> None: + """Fill principal, alias and gateway ids from analytics rows matched on ``x_request_id``. + + The principal is the analytics ``user_id``, else its ``api_key_id``. + """ + + by_id: dict[str, Mapping[str, Any]] = {} + for row in rows: + request_id = normalize_request_id(row.get("x_request_id")) + if request_id: + by_id[request_id] = row + stats.analytics_rows += len(by_id) + for record in records: + row = by_id.get(record.x_request_id) if record.x_request_id else None + if row is None: + continue + stats.analytics_matched += 1 + user_id = row.get("user_id") + api_key_id = row.get("api_key_id") + record.principal_id = str(user_id or api_key_id or "") + record.user_id = None if user_id is None else str(user_id) + record.api_key_id = None if api_key_id is None else str(api_key_id) + record.alias = None if row.get("model") is None else str(row["model"]) + if row.get("instance_id") is not None: + record.instance_id = str(row["instance_id"]) + if row.get("correlation_id") is not None: + record.correlation_id = int(row["correlation_id"]) + + +# --------------------------------------------------------------------------- +# Blocks + + +def _common_prefix_length(left: array, right: array) -> int: + low, high = 0, min(len(left), len(right)) + while low < high: + middle = (low + high + 1) // 2 + if left[:middle] == right[:middle]: + low = middle + else: + high = middle - 1 + return low + + +def segment_cuts(records: Iterable[TraceRecord]) -> dict[tuple[str, int], set[int]]: + """Per (model, block size), the hashes after which a prompt is cut into a new segment. + + A prompt is cut after a hash where some chain ends or where two chains continue differently. + In lexicographic order, chains through one hash are contiguous, so both cases show up as the + common prefix of two neighbouring chains. + """ + + groups: dict[tuple[str, int], list[array]] = {} + for record in records: + if len(record.chain): + groups.setdefault((record.model, record.block_size), []).append(record.chain) # type: ignore[arg-type] + cuts: dict[tuple[str, int], set[int]] = {} + for key, chains in groups.items(): + chains.sort() + group_cuts = {chain[-1] for chain in chains} + for left, right in zip(chains, chains[1:]): + shared = _common_prefix_length(left, right) + if shared: + group_cuts.add(left[shared - 1]) + cuts[key] = group_cuts + return cuts + + +def hash_seed(value: int) -> str: + return f"{value:016x}" + + +def record_segments(record: TraceRecord, cuts: set[int]) -> list[tuple[int, str, int]]: + """``(end position, seed, tokens)`` for each segment of the record's full blocks.""" + + chain = record.chain + positions = [position for position, hash_ in enumerate(chain) if hash_ in cuts] + segments = [] + start = 0 + for position in positions: + end = position + 1 + segments.append((end, hash_seed(chain[position]), record.block_size * (end - start))) # type: ignore[arg-type] + start = end + if start < len(chain): # unreachable when cuts include every leaf; keep the blocks anyway + segments.append((len(chain), hash_seed(chain[-1]), record.block_size * (len(chain) - start))) # type: ignore[arg-type] + return segments + + +def _request_anchor(record: TraceRecord) -> str: + leaf = hash_seed(record.chain[-1]) if len(record.chain) else "none" # type: ignore[arg-type] + return f"{leaf}:{record.request_id or record.index}" + + +def tail_seed(record: TraceRecord) -> str: + """The tokens past the last full block: seeded by Dynamo's partial-block hash when the row + has one (identical prompts share it), otherwise by the request itself.""" + + if record.partial_hash is not None: + return f"{hash_seed(record.partial_hash)}:tail{record.tail_tokens}" + return f"{_request_anchor(record)}:tail{record.tail_tokens}" + + +def _carve(pieces: list[tuple[str, int, int]], tokens: int) -> list[tuple[str, int, int]]: + """Remove ``tokens`` from the front of ``(seed, full tokens, tokens left)`` pieces.""" + + remaining = [] + for seed, full, left in pieces: + if tokens >= left: + tokens -= left + continue + remaining.append((seed, full, left - tokens)) + tokens = 0 + return remaining + + +def _piece_block(seed: str, full: int, left: int) -> dict[str, Any]: + # A piece cut short is a different block from the whole piece, so it gets its own seed. + return _user(seed if left == full else f"{seed}:rest{left}", left) + + +def _repeated(block: Mapping[str, Any]) -> dict[str, Any]: + """A block sent before, repeated: only the newest reply in a request is ``live``. Within a + trajectory the agent replays a seed it has sent with the same bytes, a live reply included.""" + + return {key: value for key, value in block.items() if key != "live"} + + +def _user(seed: str, tokens: int) -> dict[str, Any]: + return {"seed": seed, "tokens": int(tokens), "role": "user"} + + +# Per trajectory: for a segment seed, the blocks already sent that render the prompt exactly up +# to the end of that segment, as (blocks, count). +Renderings = dict[str, tuple[list[dict[str, Any]], int]] + + +def trace_request_blocks( + record: TraceRecord, + previous: Optional[TraceRecord], + previous_blocks: Sequence[Mapping[str, Any]], + cuts: set[int], + stats: TraceStats, + renderings: Optional[Renderings] = None, +) -> list[dict[str, Any]]: + """Blocks for one request; ``previous`` is the request it extends, if any. + + An extension repeats ``previous_blocks``, the blocks emitted for the previous request (so + earlier replies and tails keep their seeds and roles, but are no longer ``live``). The tokens + after them are the previous reply, as a ``live`` block of the previous output, and then new + user content. The previous tail is already in ``previous_blocks``, so it is carved off the + extension first. + + Any other request starts from the longest prefix it shares with a request already sent in the + trajectory, rendered with the blocks sent then (``renderings``), so a branch replays the same + bytes as the turn it branches from. ``renderings`` is updated with this request's blocks. + """ + + segments = record_segments(record, cuts) + tail = [(tail_seed(record), record.tail_tokens)] if record.tail_tokens > 0 else [] + renderings = renderings if renderings is not None else {} + if previous is None: + blocks: list[dict[str, Any]] = [] + shared = 0 + for end, seed, _ in reversed(segments): + rendering = renderings.get(seed) + if rendering is not None: + blocks = [_repeated(block) for block in rendering[0][: rendering[1]]] + shared = end + break + blocks += [_user(seed, tokens) for end, seed, tokens in segments if end > shared] + blocks += [_user(seed, tokens) for seed, tokens in tail] + _record_renderings(record, segments, blocks, renderings) + return blocks + + shared = len(previous.chain) + blocks = [_repeated(block) for block in previous_blocks] + extension = [(seed, tokens, tokens) for end, seed, tokens in segments if end > shared] + extension += [(seed, tokens, tokens) for seed, tokens in tail] + extension = _carve(extension, previous.tail_tokens) + reply = int(previous.completion_tokens or 0) + available = sum(left for _, _, left in extension) + if reply > available: + # The live block replays the whole real reply, which would not fit: generate it instead, + # one block per remaining piece so segment ends stay block boundaries for later branches. + stats.live_clamped += 1 + for seed, full, left in extension: + piece = seed if left == full else f"{seed}:rest{left}" + blocks.append({"seed": f"{piece}:reply", "tokens": left, "role": "assistant"}) + extension = [] + elif reply > 0: + blocks.append({"seed": f"{_request_anchor(previous)}:reply{reply}", "tokens": reply, "role": "assistant", "live": True}) + extension = _carve(extension, reply) + stats.live_blocks += 1 + blocks.extend(_piece_block(*piece) for piece in extension) + _record_renderings(record, segments, blocks, renderings) + return blocks + + +def _record_renderings( + record: TraceRecord, + segments: Sequence[tuple[int, str, int]], + blocks: list[dict[str, Any]], + renderings: Renderings, +) -> None: + """Remember, for each segment end that falls on a block boundary, the blocks up to it.""" + + ends = {record.block_size * end: seed for end, seed, _ in segments} + total = 0 + for count, block in enumerate(blocks, start=1): + total += block["tokens"] + seed = ends.get(total) + if seed is not None: + renderings.setdefault(seed, (blocks, count)) + + +# --------------------------------------------------------------------------- +# Plans + + +def _trajectory_stem(record: TraceRecord) -> str: + if len(record.chain): + return hash_seed(record.chain[-1]) # type: ignore[arg-type] + return f"req-{record.request_id or record.index}" + + +def trace_trajectory_plan( + trajectory: Trajectory, + trajectory_id: str, + window_start: datetime, + time_scale: float, + cuts: Mapping[tuple[str, int], set[int]], + stats: TraceStats, +) -> dict[str, Any]: + requests: list[TraceRecord] = trajectory.requests # type: ignore[assignment] + root: TraceRecord = trajectory.root # type: ignore[assignment] + group_cuts = cuts.get((root.model, root.block_size), set()) + first_offset_ms = (requests[0].ts - window_start).total_seconds() * 1000.0 + plan_requests = [] + previous: Optional[TraceRecord] = None + previous_blocks: list[dict[str, Any]] = [] + renderings: Renderings = {} + for position, record in enumerate(requests): + if position == 0: + reset_before = trajectory.began_before_window + else: + reset_before = not extends(previous.chain, record.chain) # type: ignore[union-attr] + blocks = trace_request_blocks( + record, None if reset_before else previous, previous_blocks, group_cuts, stats, renderings + ) + prompt_tokens = int(record.prompt_tokens or 0) + request: dict[str, Any] = { + "prompt_tokens": prompt_tokens, + "output_tokens": int(record.completion_tokens or 0), + } + if position + 1 < len(requests): + gap_ms = (requests[position + 1].ts - record.end_ts).total_seconds() * 1000.0 + request["delay_after_ms"] = scale_ms(gap_ms, time_scale) + if reset_before: + request["reset_before"] = True + request["overhead_tokens"] = max(prompt_tokens - sum(block["tokens"] for block in blocks), 0) + request["blocks"] = blocks + plan_requests.append(request) + previous = record + previous_blocks = blocks + + metadata: dict[str, Any] = {"source": "dynamo-request-trace", "model": root.model, "block_size": root.block_size} + if root.alias: + metadata["alias"] = root.alias + if root.principal_id: + metadata["principal"] = root.principal_id + return { + "schema_version": SCHEMA_VERSION, + "trajectory_id": trajectory_id, + "start_after_ms": scale_ms(first_offset_ms, time_scale), + "requests": plan_requests, + "metadata": metadata, + } + + +def build_trace_plans( + records: Sequence[TraceRecord], + window_start: datetime, + window_end: datetime, + time_scale: float = 1.0, + link_window: timedelta = DEFAULT_LINK_WINDOW, + stats: Optional[ExportStats] = None, + trace_stats: Optional[TraceStats] = None, +) -> list[dict[str, Any]]: + """Reconstruct sessions from trace records and convert them to manifest lines.""" + + stats = stats if stats is not None else ExportStats() + trace_stats = trace_stats if trace_stats is not None else TraceStats() + if time_scale <= 0 or time_scale != time_scale or time_scale == float("inf"): + raise ExportError("time scale must be a positive finite number") + usable = usable_records(records, stats) + cuts = segment_cuts(usable) # type: ignore[arg-type] + parents = link_parents(usable, link_window) + trajectories = build_trajectories(usable, parents, window_start, window_end) + stats.in_window = sum(len(trajectory.requests) for trajectory in trajectories) + stats.trajectories = len(trajectories) + stats.prewindow_sessions = sum(1 for trajectory in trajectories if trajectory.began_before_window) + + counters: dict[str, int] = {} + plans = [] + for trajectory in trajectories: + stem = _trajectory_stem(trajectory.root) # type: ignore[arg-type] + counter = counters.get(stem, 0) + counters[stem] = counter + 1 + plans.append( + trace_trajectory_plan(trajectory, f"{stem}-{counter}", window_start, time_scale, cuts, trace_stats) + ) + plans.sort(key=lambda plan: (plan["start_after_ms"], plan["trajectory_id"])) + return plans diff --git a/src/batchbench/export_plans.py b/src/batchbench/export_plans.py index 87c3732..371cf3d 100644 --- a/src/batchbench/export_plans.py +++ b/src/batchbench/export_plans.py @@ -17,9 +17,10 @@ * ``start_after_ms`` and ``delay_after_ms`` reproduce the recorded arrival pattern, divided by ``--time-scale``. -The expected input schema is documented in ``docs/export-plans.md``. The ClickHouse client is -confined to :func:`fetch_rows`; everything else works on plain row dictionaries so it can be -tested with fixtures. +:mod:`batchbench.dynamo_traces` builds the same manifests from Dynamo request traces. The expected +input schemas are documented in ``docs/export-plans.md``. ClickHouse is read only by +:func:`fetch_rows` here and ``fetch_analytics_rows`` there; everything else works on plain row +dictionaries so it can be tested with fixtures. """ from __future__ import annotations @@ -90,6 +91,12 @@ class ChainRecord: def leaf(self) -> Optional[str]: return self.chain[-1] if self.chain else None + @property + def link_key(self) -> Any: + """Requests link only to earlier requests with the same key.""" + + return self.principal_id + @property def end_ts(self) -> datetime: return self.ts + timedelta(milliseconds=self.duration_ms or 0.0) @@ -241,25 +248,26 @@ def link_parents( ) -> dict[int, Optional[int]]: """Map each record index to the index of the request it continues, or ``None`` for a root. - Within a principal, the parent of B is the latest earlier request (within ``link_window``) + Within a link key (the principal, for chain records), the parent of B is the latest earlier + request (within ``link_window``) whose full chain is a strict prefix of B's chain. Because chain hashes are cumulative, A's chain is a prefix of B's exactly when A's leaf hash appears in B's chain before B's leaf. """ parents: dict[int, Optional[int]] = {} - by_principal: dict[str, list[ChainRecord]] = {} + by_link_key: dict[Any, list[ChainRecord]] = {} for record in records: - by_principal.setdefault(record.principal_id, []).append(record) + by_link_key.setdefault(record.link_key, []).append(record) - for principal_records in by_principal.values(): + for principal_records in by_link_key.values(): principal_records.sort(key=_record_order) - latest_by_leaf: dict[str, ChainRecord] = {} + latest_by_leaf: dict[Any, ChainRecord] = {} + leaves: set[Any] = set() for record in principal_records: parent: Optional[ChainRecord] = None - for hash_ in record.chain[:-1]: - candidate = latest_by_leaf.get(hash_) - if candidate is None: - continue + # A set intersection keeps this linear in C for long block-level chains. + for hash_ in leaves.intersection(record.chain[:-1]): + candidate = latest_by_leaf[hash_] if record.ts - candidate.ts > link_window: continue if parent is None or _record_order(candidate) > _record_order(parent): @@ -267,6 +275,7 @@ def link_parents( parents[record.index] = None if parent is None else parent.index if record.leaf is not None: latest_by_leaf[record.leaf] = record + leaves.add(record.leaf) return parents @@ -763,7 +772,7 @@ def fetch_rows( def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( prog="batchbench export-plans", - description="Export a batchbench schema v2 trajectory manifest from ClickHouse prompt-chain records.", + description="Export a batchbench schema v2 trajectory manifest from ClickHouse prompt-chain records or Dynamo request traces.", ) source = parser.add_argument_group("source") source.add_argument("--clickhouse-url", default=os.environ.get("CLICKHOUSE_URL"), help="ClickHouse HTTP(S) URL (env CLICKHOUSE_URL)") @@ -774,13 +783,18 @@ def build_parser() -> argparse.ArgumentParser: source.add_argument("--analytics-table", help="per-request usage table joined on (instance_id, correlation_id)") source.add_argument("--analytics-ts-column", default=DEFAULT_ANALYTICS_TS_COLUMN, help=f"timestamp column used to bound the analytics side of the join (default {DEFAULT_ANALYTICS_TS_COLUMN})") source.add_argument("--rows-jsonl", help="read previously fetched rows from this JSONL file instead of ClickHouse") - source.add_argument("--dump-rows-jsonl", help="also write the fetched rows to this JSONL file") + source.add_argument("--dump-rows-jsonl", help="also write the fetched rows (analytics rows with --dynamo-traces) to this JSONL file") + + traces = parser.add_argument_group("dynamo request traces") + traces.add_argument("--dynamo-traces", action="append", metavar="PATH", help="read Dynamo request_end traces (*.jsonl.gz or *.jsonl) from this directory, file, or glob instead of prompt chains; repeatable") + traces.add_argument("--analytics-id-column", default="x_request_id", help="analytics column holding the gateway's x-request-id (default x_request_id)") + traces.add_argument("--analytics-rows-jsonl", help="join traces against analytics rows from this JSONL file instead of ClickHouse") window = parser.add_argument_group("window and filters") window.add_argument("--start", required=True, help="window start (RFC 3339)") window.add_argument("--end", required=True, help="window end (RFC 3339, exclusive)") window.add_argument("--principal-id", help="restrict to one principal") - window.add_argument("--model", help="restrict to one model alias") + window.add_argument("--model", help="restrict to one model alias (with --dynamo-traces: the Dynamo model name)") window.add_argument("--served-by", help="restrict to requests served by this upstream") window.add_argument("--link-window-hours", type=float, default=DEFAULT_LINK_WINDOW.total_seconds() / 3600.0, help="how far back a continuation may look for its parent (default 24)") @@ -813,45 +827,16 @@ def main(argv: Optional[Sequence[str]] = None) -> int: raise ExportError("--link-window-hours must be non-negative") link_window = timedelta(hours=args.link_window_hours) - if args.rows_jsonl: - rows = read_rows_jsonl(args.rows_jsonl) - else: - if not args.clickhouse_url: - raise ExportError("--clickhouse-url (or CLICKHOUSE_URL) is required without --rows-jsonl") - if not args.chains_table or not args.analytics_table: - raise ExportError("--chains-table and --analytics-table are required without --rows-jsonl") - sql, parameters = build_query( - args.chains_table, - args.analytics_table, - args.principal_id, - args.model, - args.served_by, - chains_final=args.chains_final, - analytics_ts_column=args.analytics_ts_column, - ) - parameters["lookback_start"] = (window_start - link_window).replace(tzinfo=None) - parameters["window_end"] = window_end.replace(tzinfo=None) - rows = fetch_rows(args.clickhouse_url, args.clickhouse_user, args.clickhouse_password, sql, parameters) - if args.dump_rows_jsonl: - with open(args.dump_rows_jsonl, "w", encoding="utf-8") as handle: - for row in rows: - handle.write(json.dumps(row, default=_json_default) + "\n") - stats = ExportStats() - records = records_from_rows(rows) - if args.rows_jsonl: - records = [ - record - for record in records - if (not args.principal_id or record.principal_id == args.principal_id) - and (not args.model or record.model == args.model) - and (not args.served_by or record.served_by == args.served_by) - ] - plans = build_plans(records, window_start, window_end, args.time_scale, link_window, stats) + extra_stats: dict[str, Any] = {} + if args.dynamo_traces: + plans, extra_stats = _export_dynamo_traces(args, window_start, window_end, link_window, stats) + else: + plans = _export_chains(args, window_start, window_end, link_window, stats) plans = sample_plans(plans, args.sample, args.seed, args.stratify_by_session_length) stats.sampled = len(plans) + summary = ", ".join(f"{key}={value}" for key, value in {**stats.as_dict(), **extra_stats}.items()) if not plans: - summary = ", ".join(f"{key}={value}" for key, value in stats.as_dict().items()) raise ExportError(f"no trajectories to export ({summary})") problems = validate_plans(plans) if problems: @@ -862,10 +847,109 @@ def main(argv: Optional[Sequence[str]] = None) -> int: print(f"error: {error}", file=sys.stderr) return 1 - summary = ", ".join(f"{key}={value}" for key, value in stats.as_dict().items()) print(f"wrote {written} trajectories to {args.output} ({summary})", file=sys.stderr) return 0 +def _dump_rows(rows: Iterable[Mapping[str, Any]], path: str) -> None: + with open(path, "w", encoding="utf-8") as handle: + for row in rows: + handle.write(json.dumps(row, default=_json_default) + "\n") + + +def _export_dynamo_traces( + args: argparse.Namespace, + window_start: datetime, + window_end: datetime, + link_window: timedelta, + stats: ExportStats, +) -> tuple[list[dict[str, Any]], dict[str, Any]]: + from . import dynamo_traces + + if args.rows_jsonl or args.chains_table: + raise ExportError("--dynamo-traces replaces --rows-jsonl and --chains-table") + if args.served_by: + raise ExportError("--served-by is not available with --dynamo-traces") + if args.principal_id and not (args.analytics_table or args.analytics_rows_jsonl): + raise ExportError("--principal-id with --dynamo-traces needs an analytics join, the only source of principals") + if args.analytics_rows_jsonl and args.analytics_table: + raise ExportError("use --analytics-rows-jsonl or --analytics-table, not both") + trace_stats = dynamo_traces.TraceStats() + records = dynamo_traces.load_trace_records( + args.dynamo_traces, window_start, window_end, link_window, args.model, trace_stats + ) + records = dynamo_traces.drop_duplicate_attempts(records, trace_stats) + analytics_rows: Optional[list[dict[str, Any]]] = None + if args.analytics_rows_jsonl: + analytics_rows = read_rows_jsonl(args.analytics_rows_jsonl) + elif args.analytics_table: + if not args.clickhouse_url: + raise ExportError("--analytics-table needs --clickhouse-url (or CLICKHOUSE_URL)") + bounds = dynamo_traces.analytics_window(records) + analytics_rows = [] + if bounds is not None: + sql = dynamo_traces.build_analytics_query( + args.analytics_table, args.analytics_id_column, args.analytics_ts_column + ) + parameters = {"start": bounds[0].replace(tzinfo=None), "end": bounds[1].replace(tzinfo=None)} + wanted = {record.x_request_id for record in records if record.x_request_id} + analytics_rows = dynamo_traces.fetch_analytics_rows( + args.clickhouse_url, args.clickhouse_user, args.clickhouse_password, sql, parameters, wanted + ) + if args.dump_rows_jsonl: + _dump_rows(analytics_rows, args.dump_rows_jsonl) + if analytics_rows is not None: + dynamo_traces.apply_analytics(records, analytics_rows, trace_stats) + if args.principal_id: + records = [record for record in records if record.principal_id == args.principal_id] + plans = dynamo_traces.build_trace_plans( + records, window_start, window_end, args.time_scale, link_window, stats, trace_stats + ) + return plans, trace_stats.as_dict() + + +def _export_chains( + args: argparse.Namespace, + window_start: datetime, + window_end: datetime, + link_window: timedelta, + stats: ExportStats, +) -> list[dict[str, Any]]: + if args.analytics_rows_jsonl: + raise ExportError("--analytics-rows-jsonl needs --dynamo-traces") + if args.rows_jsonl: + rows = read_rows_jsonl(args.rows_jsonl) + else: + if not args.clickhouse_url: + raise ExportError("--clickhouse-url (or CLICKHOUSE_URL) is required without --rows-jsonl") + if not args.chains_table or not args.analytics_table: + raise ExportError("--chains-table and --analytics-table are required without --rows-jsonl") + sql, parameters = build_query( + args.chains_table, + args.analytics_table, + args.principal_id, + args.model, + args.served_by, + chains_final=args.chains_final, + analytics_ts_column=args.analytics_ts_column, + ) + parameters["lookback_start"] = (window_start - link_window).replace(tzinfo=None) + parameters["window_end"] = window_end.replace(tzinfo=None) + rows = fetch_rows(args.clickhouse_url, args.clickhouse_user, args.clickhouse_password, sql, parameters) + if args.dump_rows_jsonl: + _dump_rows(rows, args.dump_rows_jsonl) + + records = records_from_rows(rows) + if args.rows_jsonl: + records = [ + record + for record in records + if (not args.principal_id or record.principal_id == args.principal_id) + and (not args.model or record.model == args.model) + and (not args.served_by or record.served_by == args.served_by) + ] + return build_plans(records, window_start, window_end, args.time_scale, link_window, stats) + + if __name__ == "__main__": raise SystemExit(main()) diff --git a/tests/test_dynamo_traces.py b/tests/test_dynamo_traces.py new file mode 100644 index 0000000..6423ce8 --- /dev/null +++ b/tests/test_dynamo_traces.py @@ -0,0 +1,544 @@ +"""Tests for the Dynamo request-trace source. All traces are synthetic: hashes are made here.""" + +from __future__ import annotations + +import gzip +import hashlib +import json +import shutil +import subprocess +import tempfile +import unittest +from datetime import datetime, timedelta, timezone +from pathlib import Path + +from batchbench import export_plans +from batchbench.dynamo_traces import ( + TraceStats, + analytics_window, + apply_analytics, + build_analytics_query, + build_trace_plans, + drop_duplicate_attempts, + hash_seed, + load_trace_records, + normalize_request_id, + record_from_trace, + segment_cuts, +) +from batchbench.export_plans import ExportStats, validate_plans + +from test_export_plans import find_agent_binary, word_level_tokenizer_json + +T0 = datetime(2026, 9, 1, 12, 0, 0, tzinfo=timezone.utc) +WINDOW_START = T0 +WINDOW_END = T0 + timedelta(hours=1) +MODEL = "example-org/model-a" +OTHER_MODEL = "example-org/model-b" +USER_A = "aaaaaaaa-0000-4000-8000-000000000001" +USER_B = "bbbbbbbb-0000-4000-8000-000000000002" + + +def chain_of(labels, block_size=64): + """Cumulative 64-bit hashes over block labels, like Dynamo's sequence-aware hashes.""" + + hashes = [] + previous = f"bs{block_size}".encode() + for label in labels: + digest = hashlib.blake2b(previous + b"|" + label.encode(), digest_size=8).digest() + hashes.append(int.from_bytes(digest, "big")) + previous = digest + return hashes + + +def partial_of(labels, tail_label, block_size=64): + """The hash Dynamo appends for a partial last block: the full chain plus the tail's tokens.""" + + return chain_of(list(labels) + [f"partial:{tail_label}"], block_size)[-1] + + +_request_counter = [0] + + +def trace( + received, + labels, + block_size=64, + tail=10, + output_tokens=20, + total_time_ms=1000.0, + model=MODEL, + x_request_id=None, + session_id=None, + hashes=None, + finish_reason="stop", + partial=True, +): + """A request_end row shaped like production: with ``partial`` (the default) and a non-empty + tail, the hashes end with one for the partial last block, so there are ceil(length / block + size) of them.""" + + _request_counter[0] += 1 + full = chain_of(labels, block_size) if hashes is None else list(hashes) + hashes = full + ([partial_of(labels, tail, block_size) if labels else 1] if partial and tail else []) + request = { + "request_id": f"req-{_request_counter[0]:04d}", + "model": model, + "request_received_ms": int(received.timestamp() * 1000), + "total_time_ms": total_time_ms, + "ttft_ms": 50.0, + "input_tokens": block_size * len(full) + tail, + "output_tokens": output_tokens, + "cached_tokens": 0, + "worker": {"decode_worker_id": 1}, + "replay": { + "trace_block_size": block_size, + "input_length": block_size * len(full) + tail, + "input_sequence_hashes": hashes, + }, + } + if finish_reason: + request["finish_reason_metadata"] = {"finish_reason": finish_reason} + if x_request_id is not None: + request["x_request_id"] = x_request_id + row = { + "schema": "dynamo.request.trace.v1", + "event_type": "request_end", + "event_time_unix_ms": request["request_received_ms"] + int(total_time_ms), + "event_source": "dynamo", + "request": request, + } + if session_id is not None: + row["agent_context"] = {"session_id": session_id, "input_trigger": "user"} + return row + + +def records_for(rows, stats=None): + stats = stats if stats is not None else TraceStats() + records = [record_from_trace(row, stats) for row in rows] + return drop_duplicate_attempts([record for record in records if record is not None], stats) + + +def plans_for(rows, analytics=None, stats=None, trace_stats=None): + trace_stats = trace_stats if trace_stats is not None else TraceStats() + records = records_for(rows, trace_stats) + if analytics is not None: + apply_analytics(records, analytics, trace_stats) + return build_trace_plans(records, WINDOW_START, WINDOW_END, stats=stats, trace_stats=trace_stats) + + +def seeds(labels, block_size=64): + return [hash_seed(value) for value in chain_of(labels, block_size)] + + +class BlockTest(unittest.TestCase): + def test_prompts_are_cut_where_chains_diverge_or_end(self): + first = trace(T0, ["s1", "s2", "u1"], session_id="sess-1") + second = trace(T0 + timedelta(seconds=5), ["s1", "s2", "u1", "x1", "x2"], tail=5, session_id="sess-1") + branch = trace(T0 + timedelta(seconds=9), ["s1", "s2", "v1"], tail=0) + trace_stats = TraceStats() + plans = plans_for([first, second, branch], trace_stats=trace_stats) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(len(plans), 2) + session, other = plans + s2, u1 = seeds(["s1", "s2", "u1"])[1:] + x2 = seeds(["s1", "s2", "u1", "x1", "x2"])[-1] + v1 = seeds(["s1", "s2", "v1"])[-1] + + turn1, turn2 = session["requests"] + # The shared system prefix is one segment ending where the branch diverges. + self.assertEqual(turn1["blocks"][:2], [ + {"seed": s2, "tokens": 128, "role": "user"}, + {"seed": u1, "tokens": 64, "role": "user"}, + ]) + tail1 = turn1["blocks"][2] + self.assertEqual((tail1["tokens"], tail1["role"]), (10, "user")) + self.assertEqual(turn1["prompt_tokens"], 3 * 64 + 10) + self.assertEqual(turn1["overhead_tokens"], 0) + + # The extension replays the previous tail, then the live reply, then the new content. + self.assertNotIn("reset_before", turn2) + self.assertEqual(turn2["blocks"][:3], turn1["blocks"]) + live = turn2["blocks"][3] + self.assertEqual((live["tokens"], live["role"], live["live"]), (20, "assistant", True)) + rest = turn2["blocks"][4] + self.assertEqual(rest, {"seed": f"{x2}:rest{128 - 10 - 20}", "tokens": 98, "role": "user"}) + self.assertEqual(turn2["blocks"][5]["tokens"], 5) + self.assertEqual(sum(block["tokens"] for block in turn2["blocks"]), turn2["prompt_tokens"]) + self.assertEqual(turn2["overhead_tokens"], 0) + self.assertEqual(trace_stats.live_blocks, 1) + + # The other branch shares the system segment's seed, so its bytes replay identically. + self.assertEqual([block["seed"] for block in other["requests"][0]["blocks"]], [s2, v1]) + self.assertEqual(session["metadata"], {"source": "dynamo-request-trace", "model": MODEL, "block_size": 64}) + + def test_cuts_are_per_model_and_block_size(self): + records = records_for([ + trace(T0, ["a", "b"]), + trace(T0, ["a", "c"]), + trace(T0, ["a", "b"], model=OTHER_MODEL), + ]) + cuts = segment_cuts(records) + a, b = chain_of(["a", "b"]) + c = chain_of(["a", "c"])[1] + self.assertEqual(cuts[(MODEL, 64)], {a, b, c}) + self.assertEqual(cuts[(OTHER_MODEL, 64)], {b}) + + def test_later_turns_repeat_the_blocks_sent_before(self): + rows = [ + trace(T0, ["s1", "u1"], tail=10, session_id="sess-1"), + trace(T0 + timedelta(seconds=5), ["s1", "u1", "x1", "x2"], tail=20, session_id="sess-1"), + trace(T0 + timedelta(seconds=9), ["s1", "u1", "x1", "x2", "y1", "y2"], tail=3, session_id="sess-1"), + ] + trace_stats = TraceStats() + plans = plans_for(rows, trace_stats=trace_stats) + self.assertEqual(validate_plans(plans), []) + turn1, turn2, turn3 = plans[0]["requests"] + + def sent(blocks): + return [{key: value for key, value in block.items() if key != "live"} for block in blocks] + + # Turn three keeps turn two's reply and tails instead of re-cutting its prefix. + self.assertEqual(turn3["blocks"][: len(turn2["blocks"])], sent(turn2["blocks"])) + self.assertEqual(turn2["blocks"][: len(turn1["blocks"])], sent(turn1["blocks"])) + # Only the newest reply is live; turn one's reply repeats under the seed turn two sent + # it with, which the agent replays with the same bytes. + for request in (turn2, turn3): + self.assertEqual([block.get("live", False) for block in request["blocks"]].count(True), 1) + first_reply = next(block for block in turn2["blocks"] if block.get("live")) + repeated = turn3["blocks"][turn2["blocks"].index(first_reply)] + self.assertEqual(repeated, {key: value for key, value in first_reply.items() if key != "live"}) + self.assertTrue(turn3["blocks"][len(turn2["blocks"])]["live"]) + for request in (turn1, turn2, turn3): + self.assertEqual(sum(block["tokens"] for block in request["blocks"]), request["prompt_tokens"]) + self.assertEqual(trace_stats.live_blocks, 2) + + def test_a_branch_replays_the_blocks_already_sent_for_its_shared_prefix(self): + first = trace(T0, ["a"], tail=10, session_id="sess-1") + extension = trace(T0 + timedelta(seconds=5), ["a", "b", "c"], tail=4, session_id="sess-1") + branch = trace(T0 + timedelta(seconds=9), ["a", "b", "d"], tail=6, session_id="sess-1") + plans = plans_for([first, extension, branch]) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(len(plans), 1) + _, turn2, turn3 = plans[0]["requests"] + self.assertTrue(turn3["reset_before"]) + # Turn two renders blocks a and b as a, the first tail, the live reply and the rest of b. + b_end = 2 * 64 + prefix, total = [], 0 + for block in turn2["blocks"]: + if total == b_end: + break + prefix.append(block) + total += block["tokens"] + self.assertEqual(total, b_end) + self.assertTrue(any(block.get("live") for block in prefix)) + # The branch shares those 128 hashed tokens, so it reuses the same seeds; the repeated + # reply is no longer live. + self.assertEqual(turn3["blocks"][: len(prefix)], [{k: v for k, v in block.items() if k != "live"} for block in prefix]) + self.assertFalse(any(block.get("live") for block in turn3["blocks"])) + self.assertEqual(turn3["blocks"][len(prefix)]["seed"], seeds(["a", "b", "d"])[-1]) + self.assertEqual(sum(block["tokens"] for block in turn3["blocks"]), turn3["prompt_tokens"]) + + def test_a_branch_keeps_a_generated_reply_that_spans_its_cut(self): + first = trace(T0, ["a"], tail=10, output_tokens=500, session_id="sess-1") + extension = trace(T0 + timedelta(seconds=5), ["a", "b", "c"], tail=4, session_id="sess-1") + branch = trace(T0 + timedelta(seconds=9), ["a", "b", "d"], tail=6, session_id="sess-1") + plans = plans_for([first, extension, branch]) + self.assertEqual(validate_plans(plans), []) + _, turn2, turn3 = plans[0]["requests"] + # The 500-token reply does not fit in the 54 + 64 + 4 tokens after the first tail, so it + # is generated, split where block b ends; the branch repeats it up to that cut. + self.assertEqual([(block["tokens"], block["role"]) for block in turn2["blocks"]], [ + (64, "user"), (10, "user"), (54, "assistant"), (64, "assistant"), (4, "assistant"), + ]) + self.assertEqual(turn3["blocks"][:3], turn2["blocks"][:3]) + self.assertEqual(turn3["blocks"][3]["seed"], seeds(["a", "b", "d"])[-1]) + + def test_a_reply_longer_than_the_extension_is_generated_not_live(self): + first = trace(T0, ["s1"], tail=30, output_tokens=500, session_id="sess-1") + second = trace(T0 + timedelta(seconds=5), ["s1", "u2"], tail=7, session_id="sess-1") + trace_stats = TraceStats() + plans = plans_for([first, second], trace_stats=trace_stats) + blocks = plans[0]["requests"][1]["blocks"] + # 64 + 7 extension tokens: 30 are the previous tail, and the 500-token reply would not fit + # in the other 41, so they are generated (split where block u2 ends) rather than live. + self.assertEqual([(block["tokens"], block["role"], block.get("live", False)) for block in blocks], [ + (64, "user", False), (30, "user", False), (34, "assistant", False), (7, "assistant", False), + ]) + self.assertEqual((trace_stats.live_clamped, trace_stats.live_blocks), (1, 0)) + self.assertEqual(validate_plans(plans), []) + + +class HashShapeTest(unittest.TestCase): + def test_partial_last_block_hash_is_split_off(self): + stats = TraceStats() + ceil_row = trace(T0, ["a", "b"], tail=10) + self.assertEqual(len(ceil_row["request"]["replay"]["input_sequence_hashes"]), 3) + record = record_from_trace(ceil_row, stats) + self.assertEqual(list(record.chain), chain_of(["a", "b"])) + self.assertEqual((record.tail_tokens, record.partial_hash), (10, partial_of(["a", "b"], 10))) + + exact = trace(T0, ["a", "b"], tail=0) + self.assertEqual(len(exact["request"]["replay"]["input_sequence_hashes"]), 2) + record = record_from_trace(exact, stats) + self.assertEqual((len(record.chain), record.tail_tokens, record.partial_hash), (2, 0, None)) + + # Rows without the partial hash (floor) still load, with the tail unhashed. + floor_row = trace(T0, ["a", "b"], tail=10, partial=False) + record = record_from_trace(floor_row, stats) + self.assertEqual((len(record.chain), record.tail_tokens, record.partial_hash), (2, 10, None)) + self.assertEqual(stats.malformed_hashes, 0) + + # More hashes than ceil(length / block size), or a partial hash on an exact multiple. + too_many = trace(T0, ["a", "b"], tail=10) + too_many["request"]["replay"]["input_sequence_hashes"].append(5) + exact_with_extra = trace(T0, ["a", "b"], tail=0) + exact_with_extra["request"]["replay"]["input_sequence_hashes"].append(5) + # A large input_tokens does not excuse more hashes than input_length has blocks. + inflated = trace(T0, ["a", "b"], tail=10) + inflated["request"]["replay"]["input_sequence_hashes"].append(5) + inflated["request"]["input_tokens"] = 10_000 + self.assertIsNone(record_from_trace(too_many, stats)) + self.assertIsNone(record_from_trace(exact_with_extra, stats)) + self.assertIsNone(record_from_trace(inflated, stats)) + self.assertEqual(stats.malformed_hashes, 3) + + def test_a_turn_extending_a_partial_last_block_links_and_seeds_its_tail(self): + first = trace(T0, ["s1", "u1"], tail=40) + # Turn two fills turn one's partial block, so its third hash differs from turn one's + # partial hash: only full blocks may be required to match. + second = trace(T0 + timedelta(seconds=5), ["s1", "u1", "u1-filled", "x1"], tail=12) + self.assertNotIn( + first["request"]["replay"]["input_sequence_hashes"][-1], + second["request"]["replay"]["input_sequence_hashes"], + ) + trace_stats = TraceStats() + plans = plans_for([first, second], trace_stats=trace_stats) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(len(plans), 1) + turn1, turn2 = plans[0]["requests"] + tail_block = turn1["blocks"][-1] + self.assertEqual(tail_block, {"seed": f"{hash_seed(partial_of(['s1', 'u1'], 40))}:tail40", "tokens": 40, "role": "user"}) + self.assertNotIn("reset_before", turn2) + self.assertEqual(turn2["blocks"][: len(turn1["blocks"])], turn1["blocks"]) + self.assertTrue(turn2["blocks"][len(turn1["blocks"])]["live"]) + self.assertEqual(sum(block["tokens"] for block in turn2["blocks"]), turn2["prompt_tokens"]) + self.assertEqual(trace_stats.live_blocks, 1) + + +class LinkingTest(unittest.TestCase): + def test_non_default_block_size_and_mixed_block_sizes_never_link(self): + short = trace(T0, ["p1", "p2"], block_size=1056, tail=100) + hashes = chain_of(["p1", "p2", "p3"], 1056) + longer = trace(T0 + timedelta(seconds=3), [], block_size=1056, hashes=hashes, tail=0) + # Same hash values under another block size: a different pool, never the same session. + foreign = trace(T0 + timedelta(seconds=6), [], block_size=128, hashes=hashes + [7], tail=0) + plans = plans_for([short, longer, foreign]) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(sorted(len(plan["requests"]) for plan in plans), [1, 2]) + linked = next(plan for plan in plans if len(plan["requests"]) == 2) + self.assertEqual(linked["metadata"]["block_size"], 1056) + self.assertEqual(linked["requests"][0]["blocks"][0]["tokens"], 2 * 1056) + alone = next(plan for plan in plans if len(plan["requests"]) == 1) + self.assertEqual(alone["metadata"]["block_size"], 128) + self.assertEqual(sum(block["tokens"] for block in alone["requests"][0]["blocks"]), 4 * 128) + self.assertFalse(any(block.get("live") for block in alone["requests"][0]["blocks"])) + + def test_sessions_and_principals_separate_otherwise_matching_chains(self): + first = trace(T0, ["s1", "u1"], x_request_id="00000000-0000-7000-8000-00000000000A") + second = trace(T0 + timedelta(seconds=5), ["s1", "u1", "x1"], x_request_id="00000000-0000-7000-8000-00000000000b") + # Without the join both rows have no principal and link on their chains alone. + self.assertEqual([len(plan["requests"]) for plan in plans_for([first, second])], [2]) + + analytics = [ + {"x_request_id": "00000000-0000-7000-8000-00000000000a", "instance_id": "i-1", "correlation_id": 1, "model": "alias-a", "user_id": USER_A, "api_key_id": None}, + {"x_request_id": "00000000-0000-7000-8000-00000000000b", "instance_id": "i-1", "correlation_id": 2, "model": "alias-a", "user_id": USER_B, "api_key_id": None}, + ] + trace_stats = TraceStats() + plans = plans_for([first, second], analytics=analytics, trace_stats=trace_stats) + self.assertEqual([len(plan["requests"]) for plan in plans], [1, 1]) + self.assertEqual(trace_stats.analytics_matched, 2) + self.assertEqual(plans[0]["metadata"]["alias"], "alias-a") + self.assertEqual(plans[0]["metadata"]["principal"], USER_A) + + # Agent session ids keep two sessions apart even without a principal. + one = trace(T0, ["s1", "u1"], session_id="sess-1") + two = trace(T0 + timedelta(seconds=5), ["s1", "u1", "x1"], session_id="sess-2") + self.assertEqual([len(plan["requests"]) for plan in plans_for([one, two])], [1, 1]) + + def test_retried_attempts_keep_the_last_one_that_finished(self): + request_id = "00000000-0000-7000-8000-0000000000c1" + failed = trace(T0, ["s1", "u1"], output_tokens=0, finish_reason=None, x_request_id=request_id) + final = trace(T0 + timedelta(seconds=2), ["s1", "u1"], output_tokens=33, x_request_id=request_id.upper()) + unrelated = trace(T0 + timedelta(seconds=3), ["s1", "w1"]) + trace_stats = TraceStats() + records = records_for([failed, final, unrelated], trace_stats) + self.assertEqual(trace_stats.duplicate_attempts, 1) + self.assertEqual(trace_stats.with_x_request_id, 2) + self.assertEqual(len(records), 2) + kept = next(record for record in records if record.x_request_id) + self.assertEqual((kept.completion_tokens, kept.x_request_id), (33, request_id)) + self.assertEqual([record.index for record in records], [0, 1]) + + +class SourceTest(unittest.TestCase): + def setUp(self): + self.directory = Path(tempfile.mkdtemp(prefix="batchbench-traces-")) + + def tearDown(self): + shutil.rmtree(self.directory, ignore_errors=True) + + def write(self, name, rows, gz=True): + path = self.directory / name + path.parent.mkdir(parents=True, exist_ok=True) + data = "".join(json.dumps(row) + "\n" for row in rows).encode() + path.write_bytes(gzip.compress(data) if gz else data) + return path + + def test_rows_are_windowed_by_receipt_not_file_name(self): + # Files are named by upload time; the 13:10 upload holds a row received at 11:55. + self.write("2026/09/01/frontend-a-121000-run1-0.jsonl.gz", [trace(T0 + timedelta(minutes=5), ["s1", "u1", "x1"], session_id="sess-1")]) + self.write("2026/09/01/frontend-a-131000-run1-1.jsonl.gz", [ + trace(T0 - timedelta(minutes=5), ["s1", "u1"], session_id="sess-1"), + trace(T0 + timedelta(hours=1, minutes=1), ["s1", "late"]), + trace(T0 - timedelta(hours=30), ["s1", "stale"]), + trace(T0 + timedelta(minutes=7), ["s1", "u1"], model=OTHER_MODEL), + {"schema": "dynamo.request.trace.v1", "event_type": "tool_start", "event_time_unix_ms": 0}, + ]) + (self.directory / "2026/09/01/broken.jsonl.gz").write_bytes(gzip.compress(b"{}\n")[:-12]) + trace_stats = TraceStats() + records = load_trace_records([str(self.directory)], WINDOW_START, WINDOW_END, model=MODEL, stats=trace_stats) + self.assertEqual(trace_stats.files, 3) + self.assertEqual(trace_stats.unreadable_files, 1) + self.assertEqual(trace_stats.loaded, 2) + self.assertEqual(trace_stats.outside_window, 2) + self.assertEqual(trace_stats.other_model, 1) + self.assertEqual(trace_stats.not_request_end, 1) + self.assertEqual(trace_stats.block_sizes, {64: 2}) + + stats = ExportStats() + plans = build_trace_plans(drop_duplicate_attempts(records, trace_stats), WINDOW_START, WINDOW_END, stats=stats, trace_stats=trace_stats) + self.assertEqual(len(plans), 1) + self.assertEqual(stats.prewindow_sessions, 1) + first = plans[0]["requests"][0] + self.assertTrue(first["reset_before"]) + self.assertFalse(any(block.get("live") for block in first["blocks"])) + self.assertEqual(plans[0]["start_after_ms"], 5 * 60_000) + + def test_non_ascii_model_names_match_escaped_or_not(self): + name = "example-org/modèle-a" + escaped = json.dumps(trace(T0, ["s1"], model=name)) # ensure_ascii escapes the name + plain = json.dumps(trace(T0 + timedelta(seconds=1), ["s2"], model=name), ensure_ascii=False) + path = self.directory / "mixed.jsonl" + path.write_text(escaped + "\n" + plain + "\n", encoding="utf-8") + records = load_trace_records([str(path)], WINDOW_START, WINDOW_END, model=name) + self.assertEqual(len(records), 2) + + def test_glob_and_missing_paths(self): + self.write("a.jsonl", [trace(T0, ["s1"])], gz=False) + records = load_trace_records([str(self.directory / "*.jsonl")], WINDOW_START, WINDOW_END) + self.assertEqual(len(records), 1) + with self.assertRaises(export_plans.ExportError): + load_trace_records([str(self.directory / "nothing-*.gz")], WINDOW_START, WINDOW_END) + + def run_cli(self, *extra): + output = self.directory / "plans.jsonl" + code = export_plans.main([ + "--dynamo-traces", str(self.directory / "traces"), + "--start", WINDOW_START.isoformat(), + "--end", WINDOW_END.isoformat(), + "--model", MODEL, + "--output", str(output), + *extra, + ]) + lines = [json.loads(line) for line in output.read_text().splitlines()] if output.exists() else [] + return code, lines + + def session_rows(self): + return [ + trace(T0, ["s1", "u1"], x_request_id="00000000-0000-7000-8000-000000000001", session_id="sess-1"), + trace(T0 + timedelta(seconds=4), ["s1", "u1", "x1"], x_request_id="00000000-0000-7000-8000-000000000002", session_id="sess-1"), + trace(T0 + timedelta(seconds=2), ["s1", "q1"], block_size=256, tail=200), + trace(T0 + timedelta(seconds=3), ["s1", "q1", "q2"], block_size=256, tail=3), + ] + + def test_cli_exports_from_traces_alone_and_with_analytics_rows(self): + self.write("traces/frontend-a-120100-run1-0.jsonl.gz", self.session_rows()) + code, plans = self.run_cli() + self.assertEqual(code, 0) + self.assertEqual(validate_plans(plans), []) + self.assertEqual(sorted(len(plan["requests"]) for plan in plans), [2, 2]) + self.assertTrue(all("principal" not in plan["metadata"] for plan in plans)) + + analytics = self.directory / "analytics.jsonl" + analytics.write_text(json.dumps({"x_request_id": "00000000-0000-7000-8000-000000000001", "instance_id": "i-1", "correlation_id": 9, "model": "alias-a", "user_id": USER_A, "api_key_id": None}) + "\n") + code, plans = self.run_cli("--analytics-rows-jsonl", str(analytics)) + self.assertEqual(code, 0) + # One turn matched a principal, the other did not, so the session splits. + self.assertEqual(sorted(len(plan["requests"]) for plan in plans), [1, 1, 2]) + self.assertEqual(validate_plans(plans), []) + code, plans = self.run_cli("--analytics-rows-jsonl", str(analytics), "--principal-id", USER_A) + self.assertEqual((code, len(plans)), (0, 1)) + self.assertEqual(plans[0]["metadata"]["alias"], "alias-a") + + def test_cli_rejects_mixed_sources(self): + self.write("traces/a.jsonl.gz", self.session_rows()) + code, _ = self.run_cli("--served-by", "upstream-a") + self.assertEqual(code, 1) + code, _ = self.run_cli("--analytics-table", "requests") + self.assertEqual(code, 1) # no ClickHouse URL + code, _ = self.run_cli("--principal-id", USER_A) + self.assertEqual(code, 1) # principals come only from the join + + def test_exported_manifest_dry_runs_in_batchbench_agent(self): + binary = find_agent_binary() + if binary is None: + self.skipTest("batchbench-agent binary not built") + three_turns = [ + trace(T0 + timedelta(seconds=1), ["t1", "t2"], session_id="sess-3"), + trace(T0 + timedelta(seconds=6), ["t1", "t2", "t3"], session_id="sess-3"), + trace(T0 + timedelta(seconds=11), ["t1", "t2", "t3", "t4", "t5"], session_id="sess-3"), + ] + self.write("traces/a.jsonl.gz", self.session_rows() + three_turns) + code, _ = self.run_cli() + self.assertEqual(code, 0) + tokenizer = self.directory / "tokenizer.json" + tokenizer.write_text(json.dumps(word_level_tokenizer_json()), encoding="utf-8") + result = subprocess.run( + [binary, "--model", "test-model", "--tokenizer-model", str(tokenizer), "--host", "http://127.0.0.1:9", + "--agent-plans-jsonl", str(self.directory / "plans.jsonl"), "--admission", "open-loop", + "--time-scale", "1000", "--dry-run"], + capture_output=True, text=True, timeout=120, + ) + self.assertEqual(result.returncode, 0, result.stderr + result.stdout) + self.assertIn("Agents: 3 completed / 3 total", result.stdout) + # Every later turn has exactly one live block, and none falls back to generated text: + # the three-turn session's third turn repeats the first reply as an already-sent seed. + self.assertEqual(result.stdout.count("live_blocks=1"), 4) + self.assertNotIn("live_blocks=2", result.stdout) + self.assertNotIn("Live block fallbacks", result.stdout) + + +class AnalyticsQueryTest(unittest.TestCase): + def test_query_and_window(self): + sql = build_analytics_query("requests", "x_request_id", "timestamp") + self.assertIn("toString(x_request_id) AS x_request_id", sql) + self.assertIn("FROM requests", sql) + self.assertIn("timestamp >= {start:DateTime64(3)}", sql) + self.assertIn("x_request_id IS NOT NULL", sql) + records = records_for([ + trace(T0, ["a"], x_request_id="00000000-0000-7000-8000-000000000001"), + trace(T0 + timedelta(minutes=30), ["b"], total_time_ms=60_000.0, x_request_id="00000000-0000-7000-8000-000000000002"), + trace(T0 + timedelta(hours=5), ["c"]), + ]) + start, end = analytics_window(records) + self.assertEqual(start, T0 - timedelta(minutes=15)) + self.assertEqual(end, T0 + timedelta(minutes=46)) + self.assertIsNone(analytics_window(records_for([trace(T0, ["d"])]))) + self.assertEqual(normalize_request_id(" 0000000A-0000-7000-8000-000000000001 "), "0000000a-0000-7000-8000-000000000001") + self.assertEqual(normalize_request_id("not-a-uuid"), "not-a-uuid") + self.assertIsNone(normalize_request_id("")) + + +if __name__ == "__main__": + unittest.main()