You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
streaming: fold the record-backend mixed track window decode into the producer #400
The #375 Track B work added RecordStreamEngine.window_realign_inputs, which decodes a window
synchronously in the calling thread so Python can size the deletion-extended track query
(t_ends_ext) before pulling that window's first batch.
A mixed VCF/PGEN stream therefore decodes each window twice: once here, once in the producer
thread that fills the ping-pong slots.
How it is made safe today._mixed_engine() (python/genvarloader/_dataset/_streaming.py)
gives the mixed path its ownRecordStreamEngine, built plan-less via build_engine([], 1, 1).
That engine has its own filler, its own reader and its own lock, so the synchronous decode and the
drive's producer never actually contend — they share no mutable state. PgenWindowFiller's reader_lock remains in place regardless, because it is what would make a shared-filler
arrangement safe: apply_sample_subset mutates a single pgenlib reader and the GIL is released
between that mutation and the read, so two concurrent fills on one filler would silently swap
sample columns. VCF is exempt either way — VcfWindowFiller::fill opens a fresh VcfRecordSource per call.
The cost. Roughly 2x decode on the mixed path, plus a second engine object per backend.
Folding the variant table + CSR into what the producer already hands the consumer would remove the
second decode and collapse the two engines into one. That consolidation is precisely the case reader_lock exists to protect, so it should be done with the lock, not by removing it.
Not required for v1 parity, which is why it was deferred.
Implemented by #420, merged into streaming as 7a768f8. Gate: mixed record window decode now 2 transpose word-reads (was 4) and 5 PGEN variant decodes (was 10) per window, with realign on or off.
Activity
docs(streaming): link the #400 PR
docs(streaming): make the retired oracle's docs self-consistent (#400)
test(streaming): tighten the #400 gate and guard tests (#400)
Merge pull request #420 from mcvickerlab/feat/400-mixed-record-decode…
d-laub commentedon Sep 21, 2026
Implemented by #420, merged into
streamingas 7a768f8. Gate: mixed record window decode now 2 transpose word-reads (was 4) and 5 PGEN variant decodes (was 10) per window, with realign on or off.