The paired MDE on manual_gold is 0.0063, measured -- and Run A's curve declines after epoch 6 (#135) - #138
The paired MDE on manual_gold is 0.0063, measured -- and Run A's curve declines after epoch 6 (#135)#138jonfroehlich wants to merge 11 commits into
Conversation
…does not help (#135) Answers the prerequisite #135 put before the 1,675-3,350 GPU-hour spend, from committed data alone: no cluster time, no .model_cache, no GPU, no network. Two results decide the question. Pairing is worth 2-3x and it is free. Both checkpoints are scored on the same 1,000 panoramas against the same ground truth, so the pano-to-pano difficulty that dominates the unpaired noise cancels. Measured on manual_gold, the s.e. of a difference drops 1.8-3.6x, and the MDE on max-F1 goes from 0.0117 unpaired to a bracket of 0.0043-0.0135 paired. The quantity that governs it is the discordance between the two detectors, not n*p*(1-p), which is why the 0.01 tie bar -- an unpaired bar -- is the wrong instrument for an epoch-vs-epoch comparison. Pooling is worth 7%. Ten splits hold 6,560 instances to manual_gold's 3,919 and the MDE moves 0.0117 -> 0.0109; the nine city splits alone are worse than manual_gold alone (0.0219). Not a lever, and it would mix an independently-labeled GT with nine RampNet-anchored ones. Two things fall out along the way. Run A's plateau, re-read paired, is real on capability and not on calibration. Epochs 2 vs 6 are identical on max-F1 to four decimals and 3 vs 6 is unreadable even at the favourable end of the bracket -- so the #84 finding survives a sharper instrument. But the recall gaps at the fixed 0.30 point need 20-34% discordance to be unreadable against 2.1-9.1% observed, so the operating-point drift #84 described is measured, not hedged: the epoch-3 preference under the recall-first stance is a result rather than a judgment call. The 0.008 the tie bar was derived from is sqrt(0.25/3919), the binomial worst case at p=0.5. At RampNet's actual recall it is 0.0048, and the measured clustered value is 0.0055 -- so the bar was conservative by ~45%, and lands near the right unpaired answer for the wrong reason. Panorama clustering is real: design effect 1.23-1.88, which is why the bootstrap resamples panos and not instances. Instrument check, committed rather than noted: the RampNet-vs-RampNet pair straddles records.jsonl and op_cache, and #132's exclude_border artifact puts 30% of its disagreements at the seam against a 2.1% baseline. The two pairs drawn from one source read the baseline exactly, which is the control saying the diagnostic does not fire spuriously. The limit that actually binds is not the benchmark: this is panorama sampling variance only, and Run B is n=1. Nothing here or anywhere in the repo measures training-seed variance, so a gain below ~0.01 max-F1 would be measurable but not attributable to the anneal. Recorded next to the number, with the recommendation. Verified: metrics() under all-ones weights reproduces detection_eval.aggregate to 1e-12 on four (split, model) combinations spanning both GT schemes and both detection sources. Suite 1234 passed, 1 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#135) The #135 write-up bracketed the paired MDE at [0.0043, 0.0135] because the epoch-to-epoch discordance of two RampNet checkpoints has never been measured -- only bounded -- and said closing it would cost about a GPU-hour. It costs none. The 2026-08-17 scoring cache survived on makelab2 intact: 13 GB at run_a_84/evaluate_cache/heatmaps/, all EIGHT epochs single-pass, 1,000 panos each. Peak extraction from a cached heatmap is CPU-only numpy, so the measurement needs no model, no panorama images (they are not on that host anyway), no GPU and no network -- and it covers the whole curve rather than the two checkpoints the doc proposed. A separate script rather than a flag on stage_two/evaluate.py, deliberately: that evaluator produced every committed Stage 2 number and its heatmap cache key is <fingerprint>_<dataset>_<tta> and nothing else, so the cheapest way to guarantee this analysis cannot perturb either is to not touch it. What it does share is the part that must not diverge -- extract_peaks_from_heatmap, PEAK_MIN_DISTANCE and MODEL_HEATMAP_SIZE are imported from it, not copied. --threshold truncates the tail on purpose. Run A scored at --threshold 0.0, which keeps every local maximum: ~511,000 predictions over 1,000 panos, nearly all noise floor. The 0.05 default matches op_cache's floor, keeps files ~200 KB instead of ~40 MB, and sits far below everything this is used for (protocol point 0.30; Run A max-F1 lands at 0.268-0.582). The cost is that AP is not recoverable below it, which is stated in the file and beside the output. --verify re-scores each dump against manual_labels and checks it reproduces docs/data/run_a_84_manual_gold/summary.csv. That is the load-bearing part: it is what proves the dump is the same instrument that produced the committed curve rather than a second, similar one. Output lands in the published-detections shape, so benchmark_power_135.py and every other reader of benchmark/model_detections/ takes it with no special case. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…135) The write-up could only BRACKET the paired standard error for two Stage 2 checkpoints at [0.0043, 0.0135], because Run A's committed artifacts are aggregate PR curves and carry nothing per-panorama. That bracket is now a measurement, and it cost no GPU: the 2026-08-17 scoring cache survived on makelab2 with all eight epochs at 1,000 panos each, so dump_peaks_from_cache.py recovered the detections in 4 minutes with no model, no images and no network. MEASURED across all 28 pairs of Run A checkpoints on manual_gold: se(d max-F1) 0.0016-0.0029, median 0.0021; discordance 2.5-6.6%. For pairs >= 3 epochs apart -- the closest analogue to Run B vs Run A -- median se 0.0022, i.e. MDE 0.0063 at 80% power against the unpaired bar's 0.0117. The old bracket contained it and the working assumption (4-6% discordance => 0.006-0.009) was nearly exact. s.e. grows with epoch separation, 0.0016 at gap 1 to 0.0029 at gap 7, so a Run B checkpoint -- further from any Run A checkpoint than any pair here -- should be planned at se ~0.003, MDE ~0.008. That is an extrapolation of a measured trend and is flagged as one. AND IT CORRECTS #84's CURVE SHAPE. Each pair now has its own measured s.e. instead of one global bracket, and epochs 7 and 8 are measurably BELOW the plateau: 3 vs 7 at z=3.9, 3 vs 8 at 3.0, 6 vs 7 at 2.8. The unpaired 0.01 bar could not see it. So the curve is not "steps up once from 1 to 2 then flat" -- it is epoch 1 low, epochs 2-6 a genuine plateau (2 vs 6 identical to 4 dp, 3 vs 6 at z=1.4), then a measurable decline. A shallow inverted U. The headline is unchanged -- still no resolvable peak, 2-6 remain mutually indistinguishable -- but at constant LR the model does begin to LOSE capability after ~epoch 6. That sharpens what Run B is testing: not "does a longer budget add to a flat curve" but "does cosine decay arrest a decline that is now measured". It cuts both ways -- that decline is the classic signature of an LR left too high, which is what an anneal fixes; but a 30-epoch run whose decay does not bite could land BELOW epoch 3. It also makes the #84 amendment's 8-epoch cosine rung (~450 GPU-h, budget-matched) more attractive than it was, because B-A does not attribute on its own. Verification is the load-bearing part: re-scoring each dump against manual_labels reproduces the committed summary.csv max-F1 to 5e-9..4e-7 across all eight epochs. max-F1 is a property of the whole PR curve, so agreement at 1e-7 says the peak extraction is the same operation, not a similar one. The uniform 1.2-1.4e-4 offset on F1@0.30 is the committed table's own 0.005-grid downsampling, which its provenance note already documents. The dumps land in docs/data/run_a_84_detections/, NOT benchmark/model_detections/: roster.py asserts every file there belongs to a registered challenger leg (#122), and Run A's epochs are internal checkpoints, not entries in the RampNet-vs-VLM comparison. The suite caught the first attempt to put them there -- the registry working as intended. They are committed because the 13 GB cache they came from cannot be, which is stated beside them as the one input a clean clone cannot get. Suite 1236 passed, 1 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jonfroehlich
changed the title
Deep reviewRe-ran everything from the branch head (
So everything below is a labeling/interpretation problem sitting on top of an artifact that is exactly what the script produces. One of them makes a headline number wrong by ~2×. 1. High — the nine city bundles are hard-truncated at conf 0.55, so "the #54 operating point of 0.30" is not what was measured on 9 of 10 splits
Four consequences:
The repo already documents the truncation at Measured impact, re-running with
"Pooling is worth 7%" is understated by ~2× — it is ~14%. The conclusion holds (14% is still not a lever; POOLED cities alone stays far worse than manual_gold alone), but the number in the summary table and the PR body is wrong. Caveat on that check: The headline paired numbers are all 2. Medium — the "LOWER BOUND" section cannot bound what it claims
That matters because the module docstring assigns this section exactly that job ("…so they give the lower bound"). For max-F1 — the metric the headline bracket and the whole recommendation are stated in — it cannot. No published number is wrong (the real 0.0015 comes from 3. Medium — the doc's own reproducibility claim fails on its headline numberLine 281: "Every derived number in this document is in It errs safe (interpolating the three committed points to 4–6% gives about 0.0056–0.0068), but per the repo's replication rule it should be computed by the script, and that sentence should not stand as written. Related: the stated mechanism "the standard error tracks the discordance" does not hold on its own three points — 8.1% gives 0.0029 but 9.1% gives 0.0048, a 12% rise in discordance against a 66% rise in s.e. For max-F1 there is no McNemar form, as the doc itself says. 4. Medium-low — the summary table mixes F1 and max-F1 standard errors under a max-F1 headerThe headline table (and the PR body verbatim) is headed "s.e. on Numerically immaterial, but it is the one table people will quote, and "0.01 is very nearly the correct unpaired MDE of 0.0117" is stated about the wrong column. (The separate "1.8×–3.6× gain" claim is like-for-like — both sides F1.) 5. Medium-low — the #84 correction is not discoverable from #84's own docThe PR's best secondary result is that the epoch-3 preference is now a measured result rather than a judgment call. 6. One statistical cautionThe load-bearing assumption is hedged in one place and dropped in another. The max-F1 bracket is carefully "bracketed rather than point-estimated." But the recall table prints a flat "resolvable" for gaps needing 20–34% discordance, justified by "real pairs on this benchmark run 2.1–9.1%" — where those pairs are two cross-architecture YOLO comparisons and one seam-contaminated self-pair, none of them epoch-vs-epoch. Run A's own data suggests epoch-to-epoch calibration is unstable ( 7. Low
What holds upStating this because a defect list on its own misrepresents the PR:
The irony worth naming: finding 1 is exactly what "verify the instrument before judging" exists to catch, and the evidence was sitting in this PR's own committed JSON as 18 rows of Fixes for 1–7 to follow on this branch. 🤖 Generated with Claude Code (claude-opus-5[1m]) |
…safe schedule (#135) Run A measured that constant LR does not merely flatten -- epochs 7-8 are measurably BELOW the plateau (3v7 z=3.9, 3v8 z=3.0, 6v7 z=2.8). Drifting back up after settling is the signature of a learning rate left too high to settle with, which is what a decay fixes. So there is now something for an 8-epoch anneal to act on; before that measurement the curve looked flat and this rung would have had nothing to demonstrate against. It is also the only comparison that ATTRIBUTES. Run B is 30-60 epochs AND cosine, so B - A confounds length with schedule -- the #84 amendment says so explicitly. This rung holds the budget at Run A's 8 epochs and changes only the schedule, at ~480 GPU-h and ~34 h calendar against Run B's ~1,800 and ~5-7 days. It is not a substitute: an 8-epoch cosine decays fast and cannot show a benefit that needs length. It answers the narrower question for a quarter of the cost. THE SCHEDULE IS STATELESS AND THAT IS THE DESIGN. train.py's new lr_at_step computes the rate from the absolute step index, not from a scheduler object. Stage 2 runs on preemptible ckpt-all and resumes from latest_checkpoint.pth (Run A was requeued twice); a stateful scheduler restarts its decay from the peak on every requeue unless its state is ALSO serialized, turning a cosine into a sawtooth. That failure is silent -- the job completes and the loss curve looks plausible. Reading global_step, which is already checkpointed and already broadcast to every rank, makes it impossible by construction instead of by remembering one more field. tests/test_train_lr_schedule.py pins it by simulating a run chopped into segments by requeues and asserting the LR sequence matches an uninterrupted one. train.py now logs per-step LR to tensorboard, because a sawtooth is invisible in the loss curve and obvious there. Read that scalar before trusting the run. Default stays --lr-schedule constant, so a re-run of Run A or the paper recipe is unaffected; a test asserts that default rather than trusting it. No warmup: it would be a second change, and Run A was stable from step 0. The decision rule is pre-registered BEFORE launch, in docs/stage2_cosine_rung_135.md -- primary test (cosine ep8 vs Run A ep8, paired, max-F1, each pair against its own bootstrap s.e.), secondary mechanism test (is Run A's own ep3->ep8 decline arrested?), and the gate on whether Run B follows. A tie on both is explicitly NOT an automatic cancellation; it is a judgment call, recorded as one rather than dressed as a rule. What it cannot settle is stated in the same file: seed variance is unmeasured and this is n=1, so a difference below ~0.01 max-F1 is measured but not attributable. Suite 1243 passed, 1 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…not assumed (#135) klone's train.py was byte-identical to main before the upload (ccce228b), so what runs is main plus the scheduler and nothing else -- its post-upload hash 051e3256 matches this branch exactly. The launcher landed as ASCII, not CRLF, which would have died with a bad-interpreter error. Dataset counted rather than stat'ed (300,126 entries under train/ = 150,063 pairs = Run A's 9,378 steps/epoch at world size 16), since the directory predates scrubbed's purge window. 270 GB free on /gscratch/makelab for a ~10 GB run. Env reused from Run A's durable copy, so the environment is identical to the arm being compared against. Fresh start confirmed at submit: no latest_checkpoint.pth in the new run dir, which is deliberately NOT Run A's -- pointing it there would have resumed Run A's weights and overwritten its artefacts. Recorded the first thing to check once it starts: the per-step LR scalar must fall smoothly from 1e-5. A sawtooth means a requeue reset the schedule, and that is invisible in the loss curve. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cosine rung's whole premise is that `lr_at_step` is stateless, so a preemption cannot restart the decay from the peak. That was argued in a docstring and covered by a unit test of the *formula*; nothing checked what the running job actually applied after a resume. The failure mode is silent -- a sawtooth job completes and its loss curve looks fine -- so "it should be correct by construction" is exactly the claim that needs an instrument pointed at it. `scripts/analysis/check_lr_schedule_135.py` reads the per-step LR scalar train.py logs and asks four things a mis-resume would fail: is the merged series monotone, do overlapping incarnations agree on the same step index, what is the rate at each resume as a ratio of the peak (1.000000 is the sawtooth signature), and does every value match `lr_at_step` at that step. It reuses `stage2_train_cost.read_scalars` via `stage2_epoch_curve.read_scalars_by_file` -- stdlib only, no tensorboard install, and the per-file split is what makes incarnations visible at all. Two things worth naming: - **The off-by-one is real.** train.py computes `lr_at_step(n)` then logs it under step `n + 1`. Reading it without that shift makes a correct run look wrong by one step's worth of decay, so the script names a whole-series shift as a convention change rather than reporting it as a broken schedule. - **The lift out of train.py now has one home.** `tests/test_train_lr_schedule.py` had its own AST copy; it imports the script's `load_schedule` instead. That test covers the formula, this script covers where the formula was applied. The tests are mostly negative, on purpose: a checker that passed vacuously would be worse than none, because a PASS would be read as evidence. `test_a_sawtooth_resume_ fails` builds the exact defect and asserts it is caught -- note the teeth are only 0.7% deep 4,000 steps into a 75,024-step cosine, which is why this is a ratio and a monotonicity test rather than an eyeball. Validated against job 38640313's first incarnation (4,003 steps): 0 non-decreasing violations, max deviation 4.547e-13, reproducing the ad-hoc 49-step check that preceded it at three orders of magnitude more data. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tions (#135) `sacct` collapses a requeued job to its LAST incarnation, so the count in these three docstrings was read off a view that cannot show the others. `sacct -D` on Run A's two job ids returns six incarnations, i.e. five requeues: 38541865 1s ran 8:12:19 REQUEUED 38541865 2m04s ran 0:59:35 PREEMPTED 38541865 2m26s ran 8:09:12 REQUEUED 38541865 2m01s ran 2:47:04 FAILED (its own error, not the scheduler) 38566413 16m02s ran 8:06:45 REQUEUED 38566413 2m03s ran 4:47:18 COMPLETED This strengthens rather than weakens the argument the docstrings are making: the stateless schedule had to survive five resume boundaries on the run that motivated it, not two. Comment-only. The copy of train.py under test on klone is untouched and still hashes to c359a45, matching the PROVENANCE.txt pin for job 38640313, so the run in flight is unaffected and its provenance still resolves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ng everything (#135) The cosine rung spent 8h54m on 2026-08-19/20 advancing its durable checkpoint by zero steps, across nine allocations, four of which never logged a single training step. Two independent causes, both fixed here. **1. Resuming decoded every batch it meant to skip.** The fast-forward was for i, (images, target_heatmaps) in enumerate(train_loader): if epoch == start_epoch and i < batch_idx_in_epoch: continue which pulls each skipped batch all the way through the DataLoader -- reading and decoding a 2048x4096 panorama per skipped step -- and throws it away. Measured from event wall-times against sacct -D start times: fresh start (step 0) 4.6 min to first logged step resume from step 4,000 23.2, 23.4 min resume from step 9,000 29.6 min `ResumeSkipSampler` drops the *indices* instead, so the workers never fetch them. The batches that remain and their order are identical -- DistributedSampler's permutation is a pure function of (seed, epoch), so an islice off the front leaves the rest untouched. This is a speed fix, not a change to what the model sees, and `test_skipping_matches_the_old_discard_loop` is what holds that claim up: if it ever breaks, a resumed run trains on a different sample of the epoch than an uninterrupted one and nothing downstream would reveal it. **2. Checkpoints every 1,000 steps are 22 min apart.** Now `--checkpoint-interval-steps`, still defaulting to 1000 so every committed Stage 2 number is untouched; the rung's launcher passes 200 (~4.5 min). A test pins the default and asserts the launcher opts down explicitly rather than inheriting it. Together: a slice had to last ~45-52 min to bank anything, against observed slices of 36 s to 6 min. That is a livelock, not slow progress. Two traps worth naming, both silent if got wrong: - **The LR horizon must not move on resume.** `total_train_steps` now reads `train_sampler.epoch_length` (full, unskipped) rather than `len(train_loader)`, which shrinks on the epoch a resume lands in. Had it moved, a resumed run would decay over a shorter horizon than the one it is continuing and the schedule under test would quietly not be the pre-registered one. - **`batch_idx_in_epoch` is now `resume_offset + i + 1`.** With the skip moved into the sampler, `i` restarts at 0, so checkpointing bare `i + 1` would rewind further on every successive preemption. This is a latent defect in every preemptible Stage 2 run, not just this rung -- Run B would have hit it harder, being longer. It does not invalidate anything already measured: the LR verification across four resume boundaries stands on its own. Suite 1361 passed, 1 skipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The red CI was a real matcher divergence, and the seam is worth 0.000264 to this analysisProvenance note first: this work landed inside commit fa02b37, whose message describes What was failingFailing since 2026-08-18, three runs, both Python versions. Not a flake. This branch predates
What it moved: one numberRe-scoring all eight committed epoch dumps with the wrapping matcher:
One prediction on one panorama now claims a ground-truth ramp across the seam instead of The conclusions hold, and sharpen slightly0.000264 is 4% of the 0.0063 paired MDE the recommendation rests on, and it moves epoch 7
So "epoch 1 low → 2–6 a genuine plateau → 7–8 measurably declining" stands, with 3 vs 7 now One thing this exposed that was not the seamThe max-F1 table was mixing matchers: point estimates read from Both matchers are now pinned. The historical curve is checked against the historical matcher at Spun outRe-scoring the whole roster to check the blast radius turned up three committed YOLO 🤖 Generated with Claude Code (claude-opus-5[1m]) |
…it (#135) Job 38640313 finished 2026-08-21T19:07:16 after 21 restarts. It was never cancelled and never resubmitted -- the requeue livelock diagnosed on 08-20 broke on its own when ckpt-all capacity freed up and incarnations went from minutes to eight-hour slices. Recorded as a caveat rather than a success: fa02b37 was NEVER applied to this job. The train.py that ran carries no ResumeSkipSampler and still has checkpoint_interval_steps hardcoded to 1000. The rung finished on capacity weather, which says the livelock is a property of the partition regime -- it will recur -- and that a run whose completion depends on luck is not one to schedule deliberately. **The check the pre-registration named as load-bearing now covers the whole run.** Only five resume boundaries had ever been checked against 21 restarts. Re-run against the complete event set, with the AS-RUN train.py as the lift source rather than this branch's copy: 75,024 of 75,024 unique steps (100.00%), zero non-decreasing violations, max |logged - lr_at_step(step-1)| = 4.547e-13 against a 1e-11 tolerance. Every boundary lands at the rate its step index predicts and never at 1.000000, which is the sawtooth signature. The schedule under test is the schedule that was applied. The auto-label half of the curve, budget-matched against Run A (one difference: the schedule decays): - **Annealing does not move the optimum.** Both arms bottom at epoch 5. - **Annealing damps the post-minimum decline.** Run A rises +4.6% from its own minimum to epoch 8; the rung rises +1.4%. About 70% of the late-epoch degradation is schedule, not overfitting. The reason to believe the second is the shape, not the size: the arms are indistinguishable through epochs 1-3, where cosine is still between 1.00x and 0.69x of peak, and separate monotonically from epoch 4 as the anneal bites. With n=1 per arm, seed variance remains the unmeasured term #138 identified as binding. manual_gold is NOT scored yet and this settles nothing about #135's actual question. That sweep is running on makelab2 with the protocol copied verbatim from run_a_84/run_evals.sh and the eval host's repo pinned at dc7450e -- the commit Run A was scored under -- so the comparison isolates the schedule rather than confounding it with the #140 matcher change (#148). A prediction is recorded in the doc before the numbers arrive, so it can be wrong. Committed so this re-derives from a clean clone with no cluster access, per Run A's precedent: - stage_two/cosine_rung_135_events/ -- all 18 event files, 7.4 MB, with SHA256SUMS. Verified byte-identical to the klone originals after transfer, and the LR check reproduces the same PASS from these committed copies on a different OS. - docs/data/stage2_cosine_rung_135.csv -- the curve, script-derived. stage2_epoch_curve.py grows --curve-label / --reference-label. Its CSV column names were hardcoded to run_a_val_loss / paper_val_loss, so writing a second run through it would have produced an artifact headed with the name of a run it does not contain -- worse than no artifact, because nothing downstream could tell. Defaults are unchanged and Run A's committed CSV regenerates byte-identically; the full suite passes (1361 passed, 1 skipped). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pre-registered question is answered, and the prediction recorded in this file before the numbers existed was right: annealing changes nothing measurable on the benchmark. Largest |delta| on either pre-registered metric is 0.0042, against #138's measured paired MDE of 0.0063. No epoch separates the arms. The F1@0.30 peaks are 0.9161 (Run A, epoch 6) and 0.9163 (cosine, epoch 4) -- the same number at a different epoch -- and both arms replicate #84's finding of no resolvable human peak. **The finding is a negative one about the selection signal, not about the schedule.** Annealing measurably improves auto-label validation loss: up to 3.980% at epoch 8, monotone in the anneal, and mechanistically consistent, with the arms indistinguishable while the cosine is still near peak and separating as it bites. That is not noise. None of it reaches human-labelled F1. The ~1% auto-val gain at the shared optimum buys nothing on 3,919 human-placed instances, consistent with #84's exchange rate (13.5% auto-val bought ~0.009 F1), which is what the prediction was scaled from. Auto-label val loss is a real optimisation signal only loosely coupled to the thing we care about. **AP disagrees with F1, and that is recorded rather than quietly dropped.** At epoch 7 the AP gap is +0.0092 while max-F1 is +0.0042 -- more than double. AP integrates the whole PR curve, F1 is read at a point, so the natural reading is that annealing improves the low-confidence tail more than the operating region. Quoting the AP delta as the headline would overstate the result 2x. Nothing above rests on AP; it is not a pre-registered metric here. One directional hint, flagged as a hint: from each arm's own F1@0.30 peak to epoch 8, Run A declines -0.0073 and the cosine arm -0.0039, so the late-epoch damping visible in auto-val survives into F1 at about half size. Both straddle the tie bar, so this is a thing to test at length, not to claim at n=1. For the Run B gate: this settles that at 8 epochs, budget- and seed-matched, the schedule does not change manual_gold. The mechanism-based argument for the annealed arm now has no benchmark support at this length. It does NOT settle whether a 30-epoch annealed run helps -- that changes length and schedule together, the confound the #84 amendment flagged, and this rung was built not to answer it. Provenance: scored on makelab2 with the repo pinned at dc7450e, the commit Run A was scored under, so the comparison isolates the schedule rather than confounding it with the #140 matcher change (#148). The summary script is byte-identical (c42c23dc10f3) to the one that produced Run A's committed table. All 8 checkpoints were sha256-verified after transfer and evaluate.py stamps each fingerprint into its metrics file; the eight fingerprints in summary.csv match the eight source hashes, so every row ties to specific weights rather than a directory name. Downsampled PR-vs-confidence curves committed at docs/data/cosine_rung_135_manual_gold/ (a few KB each; full curves are ~4 MB x 8 and the checkpoints 8.6 GB, neither committable). They re-derive F1 to three decimals, well inside the tie bar, so the table is checkable without cluster access. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 participant
Answers the prerequisite #135 set before the 1,675–3,350 GPU-hour spend: can
manual_goldresolve the effect Run B might produce, and does pooling the ten splits raise it?No GPU time was spent. Run A was not re-run.
The answer
manual_goldmax-F1Pairing is worth ~2× and it is free. Both checkpoints are scored on the same 1,000 panoramas against the same GT, so pano difficulty cancels. What governs a paired comparison is the discordance between the two detectors — measured at 2.5–6.6% across Run A's epochs — not
n·p·(1−p).Pooling is worth 7% (0.0117 → 0.0109). The nine city splits alone are worse than
manual_goldalone (0.0219) despite more panoramas. #135's per-split instruction is right.The benchmark is not a reason to cancel Run B. At MDE 0.0063 measured — ~0.008 extrapolated to Run B's larger epoch separation — it resolves an effect the size of #51's annealed tail (+0.024 here, 8σ) with room to spare.
The bracket was closed for no GPU at all
An earlier revision could only bracket the paired s.e. at [0.0043, 0.0135] using stand-in pairs, because Run A's committed artifacts are aggregate PR curves. Then the 2026-08-17 scoring cache turned out to have survived on makelab2 — 13 GB, all eight epochs, 1,000 panos each — and peak extraction from a cached heatmap is CPU-only numpy. So the measurement needed no model, no panorama images (they are not on that host), no GPU and no network, took 4 minutes, and covered all 28 epoch pairs instead of the two originally proposed.
The old bracket contained the answer, and the working assumption ("4–6% discordance ⇒ 0.006–0.009") was nearly exact.
Verification is the load-bearing part. Re-scoring each dump against
manual_labels/reproduces the committedsummary.csv:max-F1 is a property of the whole PR curve, so agreement at 10⁻⁷ says the peak extraction is the same operation, not a similar one. The uniform 10⁻⁴ offset on F1@0.30 is the committed table's own 0.005-grid downsampling, which its provenance note already documents.
It corrects #84's curve shape
With each pair read against its own measured s.e. rather than a global bracket, epochs 7 and 8 are measurably below the plateau — 3 vs 7 at z = 3.9, 3 vs 8 at 3.0, 6 vs 7 at 2.8. The unpaired 0.01 bar could not see it.
So the curve is not "steps up once from epoch 1 to 2 and is then flat":
The headline is unchanged — still no resolvable peak; 2, 3, 4, 5, 6 remain mutually indistinguishable (2 vs 6 identical to 4dp, 3 vs 6 at z = 1.4). What changes is that at constant LR the model begins to lose capability after ~epoch 6.
That sharpens what Run B tests: not "does a longer budget add to a flat curve" but "does cosine decay arrest a decline that is now measured." It cuts both ways — that decline is the classic signature of an LR left too high, which is what an anneal fixes; but a 30-epoch run whose decay does not bite could land below epoch 3.
Recommendation
Run the 30-epoch arm; not 60 on spec. The plateau ends at 6, so the extra 30 epochs buy schedule shape rather than useful steps, and a 60-epoch cosine spends proportionally longer at the high LRs the epoch 7–8 decline is evidence against.
Pre-register three free amendments: read Run B against Run A paired on max-F1 with the tie bar replaced by the measured MDE (0.008 at Run B's separation); per-split, not pooled; attribute cautiously below ~0.01.
Also flagged, not relitigated: the #84 amendment's 8-epoch cosine rung (~450 GPU-h, budget-matched to Run A) is more attractive than it was, because constant-LR is now known to peak at ~3 and decline by 7, so that rung isolates schedule from length — the confound the amendment already noted B − A carries.
The limit that binds is not the benchmark
Panorama sampling variance only, and Run B is n = 1. Nothing measures training-seed variance, so a gain below ~0.01 max-F1 is measurable but not attributable to the anneal without a seed control that doubles the bill.
Notes
docs/data/run_a_84_detections/, notbenchmark/model_detections/—rampnet/roster.pyasserts every file there belongs to a registered challenger leg (#122), and these are internal checkpoints, not comparison entries. The suite caught the first attempt to put them there.docs/data/run_a_84_manual_gold/.exclude_borderartifact puts 30% of the RampNet-vs-RampNet pair's disagreements at the seam against a 2.1% baseline; the pairs drawn from one source read the baseline exactly.metrics()under all-ones weights reproducesrampnet.detection_eval.aggregateto 1e-12 on four (split, model) combinations. Suite 1236 passed, 1 skipped.Files
docs/stage2_run_b_power_135.md— the write-up, caveats beside the numbersscripts/analysis/benchmark_power_135.py— the analysis, seeded and deterministicscripts/analysis/dump_peaks_from_cache.py— cache → per-pano detections,--verifydocs/data/benchmark_power_135.json,docs/data/run_a_84_detections/— LF-pinnedtests/test_benchmark_power_135.py— 10 testsInput to the #135 decision; closes nothing on its own.
🤖 Generated with Claude Code (claude-opus-5[1m])