Repository navigation
Fix SZ and SZ3 from_filter_options for chunks with 3 or more dimensions - #412
Open
aaronkanzer wants to merge 1 commit into
Open
aaronkanzer wants to merge 1 commit into
aaronkanzer wants to merge 1 commit into
Conversation
The SZ filters prepend (ndim, dtype, dims...) to the cd_values: 4 values for 1D and 2D chunks, ndim + 2 values otherwise. _from_filter_options always read the error bound mode at index 4, so it raised "Unsupported sz_mode" or returned the wrong config for 3D+ chunks.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SZ and SZ3 filters put the chunk shape at the start of the stored cd_values:
(ndim, dtype, dims...), which is 4 values for 1D and 2D chunks andndim + 2values otherwise (seeSZ_cdArrayToMetaDataErrinH5Z_SZ.candH5Z_SZ3.cpp).SZ._from_filter_optionsandSZ3._from_filter_optionsalways read the error bound mode at index 4, so for 3D or larger chunks they read a chunk dimension instead. Depending on the shape this raises or quietly returns the wrong filter.Observed on 7.1.0 and main:
ValueError: Unsupported sz_mode: 4. Chunks of(1, 2, 16, 16)giveSZ3(norm2=0.0)instead ofSZ3(absolute=0.001). Expected:SZ3(absolute=0.001)in both cases. 1D and 2D chunks work today, which is why the existing tests pass.The fix drops the extra shape values before parsing, so the existing index layout still applies. New test cases cover 2D, 3D and 4D chunks for both filters.