Fetch walk rows a run at a time, not a segment at a time

[turner]

Increment E. Closes #75. Unblocked by #74, whose write-up is #80.

_write_walks_mc called fetch(".", id-1, id) once per S line. It now reads the GFA through in one pass, remembering the segment ids, and fetches each run of consecutive ids in a second pass. Across the five committed fixtures that is 1,226 lookups down to 11.

fixture	segments	fetches
chr8:78,771,162	9	9 → 1
chr1:25,331,046	75	75 → 2
chr8:10,079,054	92	92 → 1
chr1:25,301,271	770	770 → 2
chr1:25,331,646	280	280 → 5

#74 measured what a lookup costs on the 8.0 kb region: 770 point fetches take 90.0 s where two range fetches take 0.31 s, and the parse inside them is 0.12 s of that.

The parse body is unchanged character for character, and so is the W-line emission phase. contiguous_runs is not new — the v1 path thirty lines above has grouped ids into runs since before this was written, and the v2 path never picked it up.

One deliberate departure from the design doc

docs/increment-e-batched-walk-reads §3 proposed grouping sorted(segment_ids), and §4(b) noted the risk that comes with it — parse order changes from S-line order to ascending id, coord_table is built by appending, and the emission phase's sort is stable, so parse order breaks ties. Its mitigation was an ascending-ids assertion with a per-segment fallback.

This groups in S-line order instead. A run then forms only where the file already ascends by one, so rows always arrive in the order the per-segment loop saw them — unconditionally. Difference (b) stops existing rather than being guarded, and there is no second code path to keep in agreement. On the five fixtures the two produce identical runs; the difference is that this one cannot reorder a W line on a fixture nobody has captured yet.

The oracle

Byte-identical W lines across all five committed fixtures, as #75 asked — and running in CI, which needed solving, because the real hprc-v2.0-mc-grch38-v2.2.walk.gz is team-generated, multi-gigabyte, and on no machine that runs these tests. That is the same reason #74 had to run on the server.

_write_walks_mc's output determines its input closely enough to invert: a W line names an assembly, a contig, a start and an ordered walk, the S lines give every segment's length, and a walk is contiguous by construction, so each segment's coordinate accumulates. Rebuild a derivative from a fixture's own W lines, feed it back in, and the same W lines must come out. All five do.

Its one gap is stated in the test module: column 5 (repeat placements) is rebuilt as ., because a W line cannot say which coordinates arrived that way. Harmless for the round trip — a dup that survived into the real output was reinserted into the regular lists and comes back as a regular placement — but it means the dedupe at main.py:373 runs against an empty table there. test_the_duplicate_quirks_are_preserved covers it head-on instead, and pins some genuinely strange behaviour: a surviving dup gets reinserted next to the identical regular placement, and the gap check then splits one unbroken path into two W lines. That is #76's, not this ticket's, and the byte-identical oracle depends on it staying exactly that wrong.

Verified by mutation

mutation	result
off-by-one on the run start (drops each run's first segment)	7 failed
revert to one fetch per segment	5 failed
group sorted(segment_ids) (the doc's version)	18 passed

The third passing is the honest limit and is stated as such: all five fixtures ascend, so sorting is a no-op on them. test_runs_break_on_a_gap_and_follow_file_order pins the ordering directly instead.

Also

• test_an_interrupted_extraction_leaves_no_subgraph now uses ids 1,2,5. Its stand-in fails on the second fetch, and 1,2,3 is now a single fetch, so there was no longer a midway for it to fail at. The gap restores the test's meaning, with a comment saying why.
• §8's fetch-count logging is in: one log.info per request with the segment count and the run count, so the win is legible in production rather than only in a benchmark.
• Full suite: 55 passed, 5 skipped (the skips are vg, which isn't on this machine).

Not in this PR

• The cold-region speed number. Split GenerateWalksMC's per-segment cost between the lookup and the parse #74's 293.8x is a warm-cache ratio and is not published as a speedup. GenerateWalksMC's stage time on a cold ~10 kb region against the published 30.1 s of a 38.9 s request needs the server, like Split GenerateWalksMC's per-segment cost between the lookup and the parse #74 did.
• The sparse-region question. chr1:25,331,646 is 280 segments across a span of 5,569 in 5 runs, where a range fetch decompresses far more rows than it uses. Correctness there is covered above; whether it is a speed win is the roadmap's open bullet, and it is the measurement that would set the gap-coalescing parameter §6 ships at zero. Best taken against this implementation.

🤖 Generated with Claude Code

https://claude.ai/code/session_013Q66fG2JF4diSxfYMEsGCW