Add trpx filter

CHANGELOG.rst

@@ -1,3 +1,14 @@
+Unreleased
+----------
+
+- Compression filters and libraries:
+
+  * Added **Terse** filter: `TERSE/PROLIX (TRPX) v2.0 <https://github.com/senikm/trpx2.0>`_ (commit 93cf8e4)
+
+- Build:
+
+  * Added support of C++20 compile flag handling for filter extensions (previously only for static libraries)
+
 7.1.0: 01/09/02026
 ------------------
 

README.rst

@@ -4,7 +4,7 @@ hdf5plugin
 .. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.7257761.svg
    :target: https://doi.org/10.5281/zenodo.7257761
 
-*hdf5plugin* provides `HDF5 compression filters <https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md#list-of-filters-registered-with-the-hdf-group>`_ (namely: Blosc, Blosc2, BitShuffle, BZip2, Htj2k, FciDecomp, LZ4, Sperr, SZ, SZ3, Zfp, ZStd) and makes them usable from `h5py <https://www.h5py.org>`_.
+*hdf5plugin* provides `HDF5 compression filters <https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md#list-of-filters-registered-with-the-hdf-group>`_ (namely: Blosc, Blosc2, BitShuffle, BZip2, Htj2k, FciDecomp, LZ4, Sperr, SZ, SZ3, TERSE/PROLIX, Zfp, ZStd) and makes them usable from `h5py <https://www.h5py.org>`_.
 
 See `documentation <http://www.silx.org/doc/hdf5plugin/latest/>`_.
 

doc/contribute.rst

@@ -282,6 +282,20 @@ compression_opts:
 - **psnr high** (big endian float64)
 - **psnr low**
 
+terse
+.....
+
+No options can be provided by the user, but the filter stores the following ``cd_values``:
+
+- **type_code**: Code of the dataset's integer data type:
+
+  * 0: int16
+  * 1: uint16
+  * 2: int32
+  * 3: uint32
+  * 4: int8
+  * 5: uint8
+
 zfp
 ...
 

doc/index.rst

@@ -1,7 +1,7 @@
 hdf5plugin
 ==========
 
-*hdf5plugin* provides `HDF5 compression filters <https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md#list-of-filters-registered-with-the-hdf-group>`_ (namely: Blosc, Blosc2, BitShuffle, BZip2, FciDecomp, Htj2k, LZ4, Sperr, SZ, SZ3, Zfp, ZStd) and makes them usable from `h5py <https://www.h5py.org>`_.
+*hdf5plugin* provides `HDF5 compression filters <https://github.com/HDFGroup/hdf5_plugins/blob/master/docs/RegisteredFilterPlugins.md#list-of-filters-registered-with-the-hdf-group>`_ (namely: Blosc, Blosc2, BitShuffle, BZip2, FciDecomp, Htj2k, LZ4, Sperr, SZ, SZ3, TERSE/PROLIX, Zfp, ZStd) and makes them usable from `h5py <https://www.h5py.org>`_.
 
 
 * Supported operating systems: Linux, Windows, macOS.

doc/information.rst

@@ -83,6 +83,8 @@ HDF5 compression filters and compression libraries sources were obtained from:
 * `SZ3 plugin <https://github.com/szcompressor/SZ3>`_
   (commit `4bbe9df7e4bcb <https://github.com/szcompressor/SZ3/commit/4bbe9df7e4bcb6ae6339fcb3033100da07fe7434>`_)
   using `SZ3 <https://github.com/szcompressor/SZ3>`_ and ZStd.
+* `TERSE/PROLIX (TRPX v2.0) plugin <https://github.com/senikm/trpx2.0>`_
+  (commit `93cf8e4 <https://github.com/senikm/trpx2.0/tree/93cf8e4b7e832da3e318e9d37d73f70e3ce08eeb>`_).
 * `HDF5-ZFP plugin <https://github.com/LLNL/H5Z-ZFP>`_ (v1.1.1) using ZFP.
 * `ZStd plugin <https://github.com/HDFGroup/hdf5_plugins/tree/master/ZSTD>`_
   (commit `5573db8 <https://github.com/HDFGroup/hdf5_plugins/tree/5573db8f9706e1df262752a9247e90a3d4bd57bd/ZSTD>`_) using ZStd.
@@ -120,6 +122,7 @@ Please read the different licenses:
   * SPERR: `lib/H5Z-SPERR/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/lib/H5Z-SPERR/LICENSE>`_ and `lib/SPERR/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/lib/SPERR/LICENSE>`_
   * SZ: `lib/SZ/copyright-and-BSD-license.txt <https://github.com/silx-kit/hdf5plugin/blob/main/lib/SZ/copyright-and-BSD-license.txt>`_
   * SZ3: `lib/SZ3/copyright-and-BSD-license.txt <https://github.com/silx-kit/hdf5plugin/blob/main/lib/SZ3/copyright-and-BSD-license.txt>`_
+  * TERSE/PROLIX: `lib/trpx2.0/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/lib/trpx2.0/LICENSE>`_
   * zfp: `lib/H5Z-ZFP/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/lib/H5Z-ZFP/LICENSE>`_
   * zstd: `lib/HDF5Plugin-Zstandard/LICENSE <https://github.com/silx-kit/hdf5plugin/blob/main/lib/HDF5Plugin-Zstandard/LICENSE>`_
 

doc/usage.rst

@@ -30,7 +30,7 @@ As for reading compressed datasets, ``import hdf5plugin`` is required to enable
 
 To create a compressed dataset use `h5py.Group.create_dataset`_ and set the ``compression`` and ``compression_opts`` arguments.
 
-``hdf5plugin`` provides helpers to prepare those compression options: `Bitshuffle`_, `Blosc`_, `Blosc2`_, `BZip2`_, `FciDecomp`_, `Htj2k`_, `LZ4`_, `Sperr`_, `SZ`_, `SZ3`_, `Zfp`_, `Zstd`_.
+``hdf5plugin`` provides helpers to prepare those compression options: `Bitshuffle`_, `Blosc`_, `Blosc2`_, `BZip2`_, `FciDecomp`_, `Htj2k`_, `LZ4`_, `Sperr`_, `SZ`_, `SZ3`_, `Terse`_, `Zfp`_, `Zstd`_.
 
 Sample code:
 
@@ -133,6 +133,14 @@ SZ3
    :inherited-members: FilterRefBase, Mapping
    :undoc-members:
 
+Terse
+=====
+
+.. autoclass:: Terse
+   :members:
+   :inherited-members: FilterRefBase, Mapping
+   :undoc-members:
+
 Zfp
 ===
 

lib/trpx2.0/.gitignore

@@ -0,0 +1,5 @@
+.DS_Store
+.vscode/
+build/
+
+

lib/trpx2.0/CMakeLists.txt

@@ -0,0 +1,22 @@
+cmake_minimum_required(VERSION 3.15)
+project(pyterse_and_tersecodec LANGUAGES CXX)
+
+
+option(BUILD_PYTERSE "Build the pyterse Python extension" ON)
+option(BUILD_TERSECODEC "Build the HDF5 filter library (tersecodec) plugin" ON)
+
+
+set(CMAKE_CXX_STANDARD 20)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+if(NOT CMAKE_BUILD_TYPE)
+  set(CMAKE_BUILD_TYPE Release)
+endif()
+
+
+if(BUILD_PYTERSE)
+  add_subdirectory(pyterse)
+endif()
+
+if(BUILD_TERSECODEC)
+  add_subdirectory(tersecodec)
+endif()

lib/trpx2.0/DIALS_integration/FormatTRPX.py

@@ -0,0 +1,171 @@
+from __future__ import annotations
+
+import os
+import re
+import xml.etree.ElementTree as ET
+import numpy as np
+import sys
+
+
+sys.path.append(os.path.join(os.getcwd(), 'build'))
+
+from pyterse import Terse  
+
+from scitbx.array_family import flex
+from dxtbx.format.Format import Format
+from dxtbx.model import ScanFactory
+from dxtbx.model.beam import Probe
+
+
+class FormatTRPX(Format):
+    @staticmethod
+    def understand(image_file):
+        try:
+            with FormatTRPX.open_file(image_file, "rb") as fh:
+                header = fh.read(24)
+                print(f"Header: {header}")  
+
+        except OSError:
+            return False
+
+        # Check if header starts with '<Terse'
+        is_trpx = header.startswith(b'<Terse')
+
+        return is_trpx
+
+    @staticmethod
+    def _read_metadata(image_file):
+        hd = {}
+        with FormatTRPX.open_file(image_file, "rb") as f:
+            header_data = b''
+            while True:
+                chunk = f.read(1024)
+                if not chunk:
+                    # End of file reached without finding '/>'
+                    raise RuntimeError("Failed to find the end of the header.")
+                header_data += chunk
+                if b'/>' in header_data:
+                    end_index = header_data.find(b'/>') + 2
+                    header_data = header_data[:end_index]
+                    break
+
+            try:
+                header_str = header_data.decode('utf-8')
+            except UnicodeDecodeError as e:
+                raise RuntimeError(f"Failed to decode header data as UTF-8: {e}")
+
+            try:
+                root = ET.fromstring(header_str)
+            except ET.ParseError as e:
+                raise RuntimeError(f"Failed to parse XML header: {e}")
+
+            hd['prolix_bits'] = int(root.attrib.get('prolix_bits', '12'))
+            hd['signed'] = int(root.attrib.get('signed', '0'))
+            hd['block'] = int(root.attrib.get('block', '12'))
+            hd['memory_size'] = int(root.attrib.get('memory_size', '0'))
+            hd['number_of_values'] = int(root.attrib.get('number_of_values', '0'))
+            hd['number_of_frames'] = int(root.attrib.get('number_of_frames', '1'))
+
+            hd['pixel_size'] = (0.055, 0.055)
+            hd['trusted_range'] = (0, 65535)
+            hd['distance'] = 478.0
+            hd['start_tilt'] = 0.0
+            hd['delta_tilt'] = 0.0
+            hd['exposure'] = 1.0
+
+            if 'dimensions' in root.attrib:
+                dimensions_str = root.attrib['dimensions']
+                dimensions = list(map(int, dimensions_str.strip().split()))
+                hd['image_size'] = dimensions
+            else:
+                num_values = hd['number_of_values']
+                image_dimension = int(np.sqrt(num_values))
+                hd['image_size'] = (image_dimension, image_dimension)
+
+            hd['distance'] = float(root.attrib.get('distance', '478.0'))
+            pixel_size_str = root.attrib.get('pixel_size', '0.055 0.055')
+            hd['pixel_size'] = tuple(map(float, pixel_size_str.strip().split()))
+            trusted_range_str = root.attrib.get('trusted_range', '0 65535')
+            hd['trusted_range'] = tuple(map(float, trusted_range_str.strip().split()))
+
+        return hd
+
+
+    def _start(self):
+        """Open the image file and read useful metadata into an internal dictionary"""
+        self._header_dictionary = self._read_metadata(self._image_file)
+
+    def _goniometer(self):
+        """Dummy goniometer, 'vertical' as the images are viewed."""
+        goniometer = self._goniometer_factory.known_axis((0, -1, 0))
+        return goniometer
+
+    def _detector(self):
+        beam_centre = [
+            (p * i) / 2
+            for p, i in zip(
+                self._header_dictionary["pixel_size"],
+                self._header_dictionary["image_size"],
+            )
+        ]
+        d = self._detector_factory.simple(
+            "PAD",
+            self._header_dictionary["distance"],
+            beam_centre,
+            "+x",
+            "-y",
+            self._header_dictionary["pixel_size"],
+            self._header_dictionary["image_size"],
+            self._header_dictionary["trusted_range"],
+        )
+        return d
+
+
+    def _beam(self):
+        """Unpolarized beam, default energy 200 keV"""
+        beam = self._beam_factory.make_polarized_beam(
+            sample_to_source=(0.0, 0.0, 1.0),
+            wavelength=0.02508,
+            polarization=(0, 1, 0),
+            polarization_fraction=0.5,
+            probe=Probe.electron,
+        ) 
+        return beam
+
+    def _scan(self):
+        """Scan model for this image, filling out any unavailable items with dummy values"""
+        alpha = self._header_dictionary.get("start_tilt", 0.0)
+        dalpha = self._header_dictionary.get("delta_tilt", 1.0)
+        exposure = self._header_dictionary.get("exposure", 0.0)
+        oscillation = (alpha, dalpha)
+        fname = os.path.split(self._image_file)[-1]
+        s = fname.split("_")[-1].split(".")[0]
+        try:
+            index = int(re.match(".*?([0-9]+)$", s).group(1))
+        except AttributeError:
+            index = 1
+        scan = ScanFactory.make_scan((index, index), exposure, oscillation, {index: 0}) 
+        return scan
+
+    def get_raw_data(self):
+        terse = Terse.load(self._image_file)
+        decompressed_data = terse.prolix()
+        raw_data_flex = flex.double(decompressed_data)
+        return raw_data_flex
+
+if __name__ == "__main__":
+    import sys
+    if len(sys.argv) < 2:
+        print("Usage: python script.py image_file.trpx")
+        sys.exit(1)
+    image_file = sys.argv[1]
+    try:
+        if FormatTRPX.understand(image_file):
+            format_instance = FormatTRPX(image_file)
+            format_instance._start()
+            raw_data = format_instance.get_raw_data()
+            print("Raw data:", raw_data)
+        else:
+            print("File format not understood.")
+    except Exception as e:
+        print(f"An error occurred: {e}")