Record time as well as money for every non-free run (#143)

.gitignore

@@ -82,6 +82,12 @@ analysis_out/op/*
 # becomes unrecoverable rather than merely inconvenient. Append-only; never rewrite
 # a past line, since each one is a measurement.
 !analysis_out/usage_log.jsonl
+# The compute half of the same record (scripts/analysis/slurm_usage.py, #143): one
+# row per Slurm job allocation -- elapsed, GPUs, GPU-hours, and dollars on the
+# clusters that bill. Unlike the token ledger this one is partly back-fillable from
+# `sacct`, but only while the cluster still retains the job records. Same rules:
+# append-only, one row per (cluster, job id, start), last row per key wins.
+!analysis_out/compute_log.jsonl
 # Crop-window sizing eval summaries (#114): per-rule containment/context stats plus
 # the sha256 of the per-box CSV each summarizes (the CSVs themselves regenerate).
 # The glob also covers per-bundle runs against box_gallery extent gold (#116),

CLAUDE.md

@@ -53,6 +53,50 @@ When a task involves human raters, assume from the start that **a second rater w
 the two will be compared** — per-rater files, a stable item list, and an agreement script, not a
 single blob that has to be reverse-engineered later.
 
+## Record what every experiment cost — time *and* money — as it runs
+
+**Any experiment on a non-free model or non-free compute records both its wall-clock time and
+its spend, at the time it runs.** A paper reports cost alongside accuracy, and neither number
+survives being left for later: a re-run reads the detection cache, makes no calls, and returns
+in seconds, so it reproduces neither the token counts nor the runtime.
+
+- **Paid APIs.** `compare.py` appends one row per leg to `analysis_out/usage_log.jsonl`
+  (committed): tokens, dollars, wall-clock, seconds-per-panorama, the rig, and the build that
+  actually served it. `--usage-log none` requires `--allow-unrecorded-spend`. Prices live in
+  `scripts/model_comparison/pricing.py` and are **verified-only** — each rate carries the date
+  it was checked and, for compute, the page it came from, never memory.
+- **Free models still cost time.** OWLv2, Grounding DINO, Qwen, Molmo and the YOLO arms bill no
+  tokens and burn real GPU-hours. They write the same row with `paid: false`, because "what did
+  this cost" has two units and only one of them is dollars.
+- **Cluster jobs.** `scripts/analysis/slurm_usage.py` scrapes `sacct` into
+  `analysis_out/compute_log.jsonl` — elapsed, GPUs, GPU-hours, and dollars on the clusters that
+  bill. Run it after a cluster run, not at the end of the project.
+- **A ledger is not a commit, and it has to outlive the run.** Both default into the **main
+  checkout**, never a scratch worktree: a worktree is deleted when its session ends, and that is
+  exactly how one leg spent $70.41 and left no row (#139, #143). Committing the ledger is part
+  of finishing the experiment.
+- **A missing record is an emergency with a deadline.** Vertex telemetry retains ~6 weeks
+  (`scripts/analysis/vertex_usage.py`) and `sacct` retention is finite too, so recover a missing
+  number the day you notice it. Recovery is per-model per-day, so **per-split attribution is
+  permanently gone** even after a successful pull — layer 3 rescues the total, never the
+  breakdown.
+- **A recovered number belongs in the ledger, and belongs marked.** Write it as a row with
+  `kind: "recovered"` (`rampnet.ledger.RECOVERED`) carrying the billed total *minus what the
+  surviving rows already account for*, so the ledger sums to the bill rather than double
+  counting. Two rules follow and they pull opposite ways: **cost totals include it** — omitting
+  #139's $70.41 under-reported the benchmark's Claude spend by ~200x — but **reconciliation must
+  never count it as logged**, because a recovered row was read off that same bill, so counting it
+  that way compares the bill against itself and reports `ok` for the exact gap the check exists to
+  find. `--reconcile` totals it in a separate column and subtracts it before the verdict, so a gap
+  already found and written down reads as `ok (1 recovered)` and only the unexplained remainder is
+  called out — a check that reports the same handled gap as an emergency every run is a check
+  people stop reading. A recovered row carries no `bundle` and no `elapsed_s`; those were never in
+  the telemetry.
+- **Token counts and GPU-hours are the durable facts; dollars are estimates** and the billing
+  console is authoritative. Estimate input tokens from geometry when you must (deterministic —
+  it came within 0.02% for a 984-panorama leg); **never estimate output**, which is thinking
+  plus box count and ran 62% high when tried.
+
 ## Environment & commands
 
 - Conda env (Linux + CUDA 11.8; env file pins linux-64 packages — this does not run natively on Windows):