[hooks-logging] Compact events.jsonl on session end, keeping only resume-relevant accounting lines

[danshapiro]

Summary

Related: #332 (default raw: true injection) and #333 (no retention/rotation/size-cap) — this one is a distinct, complementary knob: compact an ended session's events.jsonl down to its resume-relevant accounting lines instead of asking users to choose between full payloads forever and whole-session deletion.

Two facts about the read side of events.jsonl that make a "small retained history" well-defined:

1. The only component that reads the file at runtime is cost_history.py in amplifier-app-cli, which sums data.usage.cost_usd across llm:response events to re-seed cumulative session cost on resume. It tolerates a file containing only those lines (substring pre-filter; extra lines skipped; missing file → graceful skip).
2. Every other reference (session repair / truncate utilities, fork/export) only truncates or copies the file proportionally — line content beyond llm:response is never interpreted. In-session consumers (web UI streaming, etc.) read from memory, not the file.

So for a finished session, ~99% of the file is write-only data: tool:pre/post, content_block:*, provider:*, llm:request, context:compaction, artifact:*. Already-filtered/raw: false settings help the inflow rate (#332) but don't reclaim the structural bulk.

Measured

Compacting 12,791 end-of-life session logs (>1 week untouched) — keep each "event": "llm:response" line trimmed to {ts, event, duration_ms, status, data.usage, data.model, data.provider}, drop all other lines:

• 44.21 GB → 0.06 GB (≈750× smaller)
• Cost accounting identical: summed cost_usd from a compacted file matches the untrimmed original exactly, so cumulative-cost restore on resume (sale.order: Mostrar Comisión, Costo Envío e Ingreso Neto en tab Financiero Canal #284) is preserved byte-for-byte.

Proposal

Module hooks-logging, opt-in config, consistent with the existing exclude_events (#8) style:

hooks:
  - module: hooks-logging
    config:
      compact_on_session_end: true
      compact_keep_events: ["llm:response"]                       # default
      compact_keep_data_fields: ["usage", "model", "provider"]    # default

Behavior: on session:end, after the final write, the hook rewrites its own session log atomically (temp file + rename, preserve mtime) dropping lines not matching compact_keep_events (fnmatch) and pruning kept lines to the whitelisted fields. Idempotent; a second compact is a no-op.

Also worth considering in the same pass: session_log_enabled: false as a documented master switch (strictly cleaner than exclude_events: ["*"], which works today but reads as an accident).

Why not the existing knobs

• exclude_events (refactor: move subagent logs from .data/ to logs/ directory #8) and strip_raw act at write time: the immediately-debuggable recent tail is lost, which is the part you usually want. Compaction keeps full detail for the live session and shrinks it only once it's over.
• Truncation-based mechanisms ([hooks-logging] No retention/rotation/size-cap option for events.jsonl — append-only files reach 1.1 GB per session #333) drop the llm:response tail along with everything else, so cost resume loses data; compaction keeps it.

Interaction notes

• amplifier-bundle-team-tracking's session sync can upload events.jsonl (include_events: true); docs should note compaction reduces what gets synced, so default stays off (opt-in).
• Fork/session-copy utilities that duplicate events.jsonl keep working on compacted files (they only copy/truncate).

Happy to send a PR against microsoft/amplifier-module-hooks-logging if the shape looks right.