Repository navigation
#74 reports: the cost is the seek, not the parse - #80
Merged
Merged
Conversation
Records the measurement on the server and resolves everything that was written as waiting on it. 770 segments fetched one at a time take 90.0 s; the same rows through two range fetches, parsed identically, take 0.31 s. So increment E is the range fetch already described, and it needs no ADR. The script's own printed `parse` line says 16.8 s / 19%, and that figure is an artifact of its subtraction: the identical parse measured across the two range arms is 0.121 s. The two point arms differ from the two range arms in exactly one respect, so the 16.8 s is more lookup cost, not parse work batching would leave behind. The new perf record says so rather than reprinting the number unqualified, and the real split is ~99.9% lookup. Two things the run did not deliver, both noted and neither blocking #75: the sparse fixture (chr1:25,331,646, 280 segments across a span of 5,569) was not measured, and #76's dup count did not come back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013Q66fG2JF4diSxfYMEsGCW
This was referenced Sep 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #74.
The measurement ran on the server on 2026-09-06 against the real walk derivative. This branch records it and resolves the four places that were written as waiting on it.
The result
Over
chr1:25,301,271— 770 segments, ids79335081–857, 2 contiguous runs:point+parse(today)pointrange+parse(the proposal)rangeThe lookup dominates. So #75 keeps the shape it was written in — a range fetch behind a byte-identical oracle — and increment E needs no ADR. The parse-dominated branch, which would have meant
numpy/pandas/a C extension in a project that has none, is closed.One correction to the script's own output
walk-lookup-split.pyderives the parse cost aspoint+parse − point= 16.811 s, and prints it as 19% of today. That is an artifact of the subtraction. The identical parse, over the same 770 rows into the same tables, measured asrange+parse − rangeis 0.121 s — 0.157 ms/segment against 21.8, a factor of 139 for identical work. Both are medians of three passes.The two arms differ in exactly one respect, so the 16.8 s is more lookup cost, not parse work that batching leaves behind. The real split is ~99.9% lookup. The perf record states this rather than reprinting 19% unqualified; the verdict is unchanged, only stronger.
What is deliberately not claimed
293.8xis a warm-cache ratio on one fixture and is not published as a speedup. Increment E's number isGenerateWalksMC's stage time on a cold ~10 kb region against the published 30.1 s, measured when #75 ships.Two gaps, neither blocking #75
chr1:25,331,646is 280 segments across a span of 5,569 in 5 runs, where a range fetch decompresses ~5,569 rows to use 280. Correctness there is covered by Stop reading the walk table one segment at a time #75's byte-identical oracle across all five fixtures; speed is not. It is the same measurement that would set the gap-coalescing parameter increment E ships at zero, so it is better taken against Stop reading the walk table one segment at a time #75's implementation than against the harness. Recorded as a new "Still open" bullet.needs-info. Per its own "Note for Stop reading the walk table one segment at a time #75", it blocks nothing here.Files
docs/perf/walk-lookup-split-2026-09-06.md— new; raw output verbatim plus how to read itdocs/tube-map-pipeline.html— "what is still unmeasured" → the answerdocs/increment-e-batched-walk-reads.{html,md}— §7 collapses from a fork to a result; dateline updated; orphaned.forkCSS droppeddocs/seqtubemap-roadmap.md— status paragraph, frontier table, dependency graph, Split GenerateWalksMC's per-segment cost between the lookup and the parse #74 ✅, Stop reading the walk table one segment at a time #75 unblocked, "Still open" bullet struck and replaced with the sparse-region question🤖 Generated with Claude Code
https://claude.ai/code/session_013Q66fG2JF4diSxfYMEsGCW