fix: prevent OOM when processing large Codex rollouts

[zaiddreakh]

Summary

• stream rollout JSONL using a fixed file-size snapshot instead of loading the entire transcript
• skip oversized records and bound retained turn/context/routing data, explicitly labelling incomplete traces instead of blocking the session
• keep system/current-user messages and a bounded recent-history window in generation inputs, evicting whole turns/steps to preserve tool-call relationships
• track sampled-span delivery per flush window; preprocessing drops leave only affected turns retryable without poisoning later turns
• batch across turns at 32 ended spans or approximately 4 MiB of span attributes, checkpointing only complete, confirmed turns
• preserve confirmed checkpoints through transport rejection or process interruption, with stable retry identities

Validation

• pnpm test with OTEL_TRACES_SAMPLER unset — 145 tests passed
• pnpm run lint — formatting and tsc --noEmit passed
• regression coverage: oversized records/turns/context, bounded routing, preprocessing drops, intentional non-sampling, rejected batches, interruption/retry, current prompts after history eviction, and UTF-8/escaped input byte limits

Stress and latency benchmark

Reproduce:

node scripts/benchmark-rollout.mjs plugins/tracing/dist/index.mjs 200 200000 200

~40.2 MB / 200-turn rollout under --max-old-space-size=128, with 200 ms added to every HTTP request:

• exit 0, 800 spans, 200 checkpointed turns
• 29 HTTP requests, down from 200 in the original draft
• 7.4 seconds, within the hook's 30-second timeout
• approximately 114 MiB peak RSS (the heap limit and RSS are distinct)

Truncation is explicit via codex.rollout.truncated, codex.system_prompt.truncated, codex.routing.truncated, and codex.generation_input.truncated. Individual oversized history groups or tool schemas may be omitted; no bounded representation can preserve arbitrarily large inputs verbatim.

Addresses langfuse/langfuse#18053.

[CLAassistant]

All committers have signed the CLA.