Repository navigation
Add light microscopy and light-sheet formats behind a seeking reader (0.6.0) - #4
Merged
Merged
Conversation
…(0.6.0) Pointer-based formats could not be decoded from a fixed leading or trailing read, so decoders may now register Read(seek=budget) and receive a bounded Source (read_at / read_tail, byte budget, read cap, HTTP Range request cap, 64 KiB cached chunks). decode_bytes still serves them from memory, and sparse recordings of real files are the golden fixtures. New decoders: TIFF / BigTIFF / OME-TIFF / ImageJ / Micro-Manager, Zeiss CZI, HDF5 without libhdf5 (superblock v0-v3, object headers v1/v2, symbol tables, dense links and attributes via fractal heaps and v2 B-trees) with Imaris, BigDataViewer, Luxendo and generic flavors, Leica LIF, Nikon ND2 (Ver2 XML and Ver3 LiteVariant), Imspector OBF/.msr, KLB, ICS, OME-Zarr / Zarr v2+v3 / N5 sidecars (a store directory decodes through them), and an .xml dispatcher for BigDataViewer SpimData, OME-XML and EPU. Verified on real sample files from the OME image repository, Zenodo, IDR and OpenOrganelle, cross-checked against tifffile, czifile, h5py, readlif, nd2, obffile and zarr. Two Range requests describe a 51 GB BigDataViewer file.
…sed, HDF5 cycle guard A sweep of 390 files on the OME sample server by URL found one real failure: Ver3 ND2 writers pad the chunk-map chunk's name field to 4128 bytes and the decoder rejected names over 4096, so the whole file read as None. Fixed, with a sparse fixture of that file cross-checked against the nd2 package. The same sweep showed OME-TIFFs with trailing directories spending one Range request per plane (48, the cap, on a 105-plane file). The XML already states the plane count, so the chain is no longer chased for OME-TIFF; pageCount stops at 1 and is marked truncated, planeCount comes from the OME block. Hostile files (self-referencing IFDs, HDF5 continuation and B-tree cycles, CZI and ND2 offsets past EOF, billion-laughs XML) all terminated within the read cap; the HDF5 v1 B-tree walk now stops a cycle at once instead of spending the cap on it. Those cases are kept as tests/test_adversarial.py. Also run: 200k random and magic-seeded buffers, 14k mutations of every fixture, a 900-file parallel batch identical to serial.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The library decoded one of the formats a light-sheet rig produces (MRC). Its README listed TIFF and HDF5 as out of scope because their metadata sits behind internal pointers. This PR adds a bounded seeking reader and the decoders that need it.
Reader model
Read(seek=budget)registers a decoder that takes aSource(read_at,read_tail,size,key) instead of bytes.FileSource,UrlSource(HTTP Range, 64 KiB cached chunks,URL_MAX_REQUESTScap) andBytesSource(bytes, or sparse ranges for fixtures) implement it.decode_source(key, src)serves every format from a Source.tests/fixtures/*.sparse.json).Formats
Spec-only (no public sample): Micro-Manager-native TIFF, ImSpector TIFF, Luxendo
.lux.h5(shape only). Not decoded: OIB/OIR, DCIMG, raw.bin/.dat.Measured remote cost
Behavior change
.xmlis now claimed by a root-element dispatcher; unknown XML decodes to None, andhas_decoder_for("x.xml")is True. Illumina file-name dispatch still wins. One assertion intests/test_illumina.pythat.xmlwas unclaimed is removed.Tests
pytest -q: 425 passed, 2 skipped (fuzz covers every registered extension and name; magic-seeded fuzz added for the TIFF, HDF5 and CZI pointer walks).