Skip to content

Support negative slice steps on fields stored in HDF5 - #277

Open
aaronkanzer wants to merge 1 commit into
nexpy:mainfrom
aaronkanzer:fix/negative-step-file-slices
Open

aaronkanzer wants to merge 1 commit into
nexpy:mainfrom
aaronkanzer:fix/negative-step-file-slices

Conversation

@aaronkanzer

Copy link
Copy Markdown

Slicing an NXfield with a negative step works when the data are in memory. It fails once the data are in a file, or in the HDF5 core memory file used for large fields. So whether field[::-1] works depends on the size of the array.

import numpy as np
from nexusformat.nexus import NXroot, NXentry, NXfield, nxload

a = np.arange(2000 * 1000.).reshape(2000, 1000)
NXroot(NXentry(NXfield(a, name="x"))).save("big.nxs", "w")
x = nxload("big.nxs")["entry/x"]
x[::-1]

Observed: ValueError: Step must be >= 1 (got -1). Writes such as x[::-2] = ... on a file opened with "rw" fail the same way. So does NXfield(shape=(200, 100), dtype=float)[::-1], because fields larger than maxsize are kept in an HDF5 core file.

Expected: the same result as a[::-1], which is what a small array returns.

Cause: NXFile.readvalue, NXFile.writevalue, _get_memdata and _put_memdata pass the index straight to h5py. HDF5 hyperslabs only accept positive steps.

Fix: a small helper turns each slice with a negative step into the matching slice with a positive step. It reads or writes that slice, then reverses the affected axes with NumPy. Indices without a negative step, and fancy indices, still go to h5py unchanged. Tests cover reads and writes on file-backed and memory-file fields. The helpers were also checked against NumPy on 5000 random int, slice and Ellipsis indices, with no mismatches.

HDF5 hyperslabs require positive steps, so field[::-1] raised
'Step must be >= 1' whenever the field data lived in a file or in the
core memory file used for large fields, although the same index works
on small fields held in memory. Read and write the equivalent slice
with a positive step and reverse the affected axes in NumPy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant