Repository navigation
Release 0.14.0 - #325
Merged
Merged
Release 0.14.0#325
Conversation
On baseline x86-64 builds, scalar f32 floor, round, round_ties_even and mul_add compile to calls into libm (floorf, roundf, rintf, fmaf) for every sample; aarch64 has each as an instruction. A cross-compiled audit found 71 such call sites in decode functions outside the SIMD dispatch, and a -C target-cpu=native build of j2k decoded 16-bit HTJ2K 1.31-1.53x faster on a Ryzen 7 5800X3D. - The scalar reversible 5-3 lifting (horizontal rows, the banded and vertical scalar forms) uses integer arithmetic: exact without floor or fused multiply-add instructions. - The horizontal row filter, the native 8/16-bit sample output loops (whole planes, interleaved and streamed rows), the 8-bit packers, the direct-route color and store loops and the deferred-ICT remainder run inlined at the runtime-dispatched SIMD level, where these are instructions. Decode functions that never dispatch now make no libm calls (palette lookup aside); the remaining calls sit in the SSE2/SSE4.2 levels that only pre-AVX2 CPUs select. Output is unchanged: all j2k-native and j2k tests pass, including new checks of the f32 5-3 filters against the exact integer reference, and T.803 CPU Part 1 (90/90) and Part 15 (70/70) pass. On an M4 Mac mini the decode times are unchanged (0.99-1.01x).
Cleanups from a review of 035d304; behavior and output are unchanged. - Explain the `_at`-inside-`dispatch!` convention once in `math` instead of in every helper's docs and lint reasons. - Drop the banded `lift_53_low`/`lift_53_high` wrappers and pass the integer 5-3 helpers directly. - Fold `write_rgb8_rows_at` back into `write_rgb8_region` as a dispatch block, removing an eight-argument helper. - Dispatch once per streamed row write instead of once per row. - Pack RGB through `pack_rgb8` (which dispatches itself) before the dispatched one-, two- and four-component paths, not inside them. - Build the IDWT tests' `pseudo_random_coefficients` on `pseudo_random_integers` (identical values).
- Scalar floor, round and round-ties-even compute the exact result in plain f32 arithmetic where the target has no instruction for them (baseline x86-64, no-std); aarch64 and SSE4.1 builds keep the std methods. Checked bit for bit against std over all 2^32 inputs. - Replace the x86-64-only output kernels with the loops every target uses, dispatched once: the RGB8 pack, the fused 9-7 colour pack and the centred rounding. The AVX2 builds of those loops vectorize. - Move `SampleShift` to `mct` and route the direct decode's store and colour transform through it and `DeferredMct`, instead of copies of the same arithmetic; add `From<J2kWaveletTransform>`. - Pick the native sample converter in one `with_native_converter!` match instead of three copies. - Pass the banded 5-3 vertical lifting steps as #[inline(always)] closures. As function items their Fn shims ran out of line from the AVX2 dispatch, calling out for every vector operation. The encoder's lifting and colour-transform helpers had the same problem and now inline or dispatch; nested helpers replace closures that were outlined. Output is unchanged: 1640 j2k-native and j2k tests pass, and T.803 CPU Part 1 (90/90) and Part 15 (70/70) pass. Not yet timed.
`cargo xtask x86-codegen-audit` builds j2k-native for x86_64-unknown-linux-gnu without LTO, disassembles it and fails on a scalar floor/round/fma library call outside a SIMD dispatch, or on an AVX2/AVX-512 vector operation compiled out of line. The PR checks' Rust quality job runs it.
…ounding e1465d2 replaced x86-64's hand-written RGB8, fused 9-7 colour pack and centred-rounding kernels with the loops other targets use, and rounded in plain f32 arithmetic on baseline x86-64. Measured on a Ryzen 7 5800X3D (AVX2), RGB8 tile decodes got up to 1.45x slower: about 5% from the arithmetic rounding and the rest from the loops, which LLVM vectorizes less well than the kernels. - Hosted builds round with the std methods again, which are instructions inside the SIMD dispatch; only no-std builds use the exact arithmetic rounding, in place of its old approximations. - The three loops run x86-64's eight-pixel blocks first, then the shared loop takes the remainder; other targets run only the shared loop, which measured unchanged on an M4. With this change x86-64 decode times match 035d304's (0.97-1.01x) on every A/B case.
With hosted builds rounding through the std methods again (a163211), the quantizer's per-coefficient floor and round were C library calls on baseline x86-64, as `cargo xtask x86-codegen-audit` reports. The loops now run at the dispatched SIMD level, in parallel chunks when the subband is large, and the step encoding, once per subband, uses the `libm` crate's exact floorf and roundf. quantize_subband reuses Quantizer instead of a copy of its arithmetic. Also formats a163211's mct.rs change.
The scalar row transform descales p0 directly, (p0 + 16) >> 5, when a first-pass row has no AC inputs. The x86 32-bit kernel always ran the full transform, whose p0 << 13 wraps once p0 reaches 2^18, so crafted blocks with one nonzero column (dequantized values clamp to any i16) decoded to different bytes on AVX2 hosts than on the scalar path. Select the scalar's DC value per lane in pass 2, as the NEON kernel already does. Parity proptests now cover single-column blocks and full-range right-half, quadrant and 2x2 shapes on x86; the first two failed before this change.
The x86 IDCT was a 128-bit, 32-bit-lane port of the NEON kernel. The new kernel keeps samples in 16-bit lanes of ymm registers and multiplies interleaved (p[k], p[k+4]) pairs with vpmaddwd, so every product and sum is exact in 32 bits; it never adds two inputs in 16 bits. Pass 1 computes two output rows per register with lane-specific gains, narrows to i16 with column m paired with column m+4, and an in-lane 32-bit transpose feeds pass 2 the same pairing. The +128 level shift is folded into the pass-2 rounding and the output is clamped by saturating packs. The only narrowing is the 16-bit hand-off between passes. A per-block check accepts a block only when every column's sum of |coefficient| is at most 5,905: with the largest per-input gain 11,363, every first-pass output then lies in [-32,764, 32,763]; the bound is tight (x[1][c] = 5,906 gives 32,769). Other blocks run the 32-bit kernel, which matches the scalar path for every input. The derivation sits beside the check. The per-input gains of the ISLOW transform are defined once, beside the scalar reference and pinned to its column pass by a test; the AVX2 kernel and NEON's 2x2 kernel take their constants from it. On cuda (Ryzen 7 5800X3D): 33.8 -> 17.0 ns per block in the micro bench. No block of the A/B corpus (4.0M) exceeds the bound (largest column sum 2,173), and decoded bytes are identical to the previous build.
Blocks the entropy decoder classifies as bottom-half-zero, right-half-zero, top-left quadrant or top-left 2x2 ran the general kernel on x86, as NEON already specializes them. The 16-bit kernel is now generic over the rows and columns that may hold coefficients: zero rows are not loaded, zero columns 4..7 skip half of pass 1, and zero first-pass columns 2..7 drop their pass-2 products. The range check and the 32-bit fallback are shared, so every entry point stays exact for any block of its shape. BenchAvx2Idct gains the four entry points for the micro bench and the parity proptests; the structured unit tests run every entry point whose shape admits the block.
x86 colour conversion ran eight pixels at a time with 32-bit vpmulld and a scalar RGB store, and 4:2:0 fancy upsampling was scalar. Both now use 16-bit lanes and stay bit-exact with the scalar formulas. Colour: 32 pixels per step. Each fixed-point gain splits into a whole multiple of 2^16, added exactly, and a 16-bit fraction: R = Y + Cr + ((26345 Cr + 2^15) >> 16) and B = Y + 2 Cb + ((-14942 Cb + 2^15) >> 16) via vpmulhw on the doubled input, plus one, halved; G = Y - Cr - ((22554 Cb - 18734 Cr + 2^15) >> 16) via vpmaddwd. Unsigned saturating packs are the clamp, and three byte-shuffle passes interleave 96 bytes of RGB. A partial last chunk runs through padded copies. The triple cursor now yields 32-byte chunks; the 8-pixel 32-bit path is gone. Upsampling: the interior pairs of each row run sixteen chroma samples per step through the shared emit_h2v2_row_with driver, mirroring the NEON hook, for the row-pair path and Backend::upsample_h2v2_fancy_row. The vpmaddwd gain packing is shared with the IDCT as simd::x86::madd_pair. The every-chroma-pair test now covers all 2^24 (Y, Cb, Cr) triples, and a new test covers every ordered pair of adjacent column sums and every (curr, near) sample pair. cuda (Ryzen 7 5800X3D), kodak24 q90 matched batch vs the previous commit, median of 3 interleaved runs: 4:2:0 36.22 -> 32.86 ms on one worker and 5.50 -> 5.22 ms on 16; 4:4:4 40.57 -> 38.58 ms on one worker, 5.97 -> 6.02 ms (+0.8%) on 16. Decoded bytes are identical over the A/B corpus.
The x86 YCbCr->RGB kernel converts 32 pixels per step and a shorter tail through zero-padded copies. Check rows of 31, 32, 33, 63, 64, 65 and 97 pixels against the scalar converter, on every backend, alongside the existing short and 255/256-pixel widths.
…ffolding Remove the duplication the last four commits added or kept, with no change to decoded bytes. IDCT constants. src/idct/mod.rs defines the 8-bit shifts and roundings (CONST_BITS, PASS1_BITS as i32, PASS1_SHIFT, PASS2_SHIFT, their roundings, and DC_SHIFT/DC_ROUNDING of the row DC shortcut) and the 12-bit pass shifts once. The scalar, reduced-size, NEON and AVX2 transforms use them; the per-kernel shift re-declarations, the twelve FIX aliases in avx2.rs and neon.rs, and downscale.rs's hard-coded 13, 2 and four codec-math FIX values are gone. downscale.rs uses scalar's descale_i64 instead of a copy, and the AVX2 32-bit kernel uses the shared pass roundings. Gain bounds. scalar::islow_gain_bounds() computes the largest gain and largest L1 norm of ISLOW_GAINS once; AVX2's MAX_GAIN/MAX_GAIN_L1 come from it, and NEON const-asserts its hand-written "largest L1 gain is 61,214" pass-1 overflow claim against it. x86 masks. x86_memory::broadcast_u8x16 loads [u8; 16] pshufb masks into both lanes; the RGB interleave masks and the IDCT's 4x4 byte transpose are [u8; 16] consts and lane_mask is gone. Upsampling. color::upsample::h2v2_fancy_pair_blocks is the one 16-pair loop (windows, bounds, chunking); the AVX2 and NEON kernels supply only the arithmetic. Backend::upsample_h2v2_fancy_row picks `near` once and passes the backend's pair kernel to emit_h2v2_row_with, the same shape for every backend. Butterfly. The AVX2 16-bit kernel's two passes share one butterfly::<INPUTS, LOW, HIGH>, generic over how many inputs may be nonzero (the sparse ROWS/COLS skips) and which output each 128-bit lane computes: pass 1 runs it per four columns with lanes (0, 1), pass 2 twice with (0, 0) and (1, 1). Kept after an A/B against the same refactor without it: equal or faster in every case (4:2:0 one worker 31.42 vs 31.70 ms, the rest within 0.5%). Tests and benches, every test kept. idct::avx2::SHAPES is the one shape-to-kernel list: the unit tests use it directly, and BenchAvx2Idct::SHAPES/run_shape expose it to the parity proptests and the micro bench (this also removes the type_complexity error the micro bench's own list raised under the non-lib clippy gate). The parity generators all go through shaped_blocks, and their in-bound ranges derive from BenchAvx2Idct::MAX_COLUMN_ABS_SUM instead of literal 5,905/738/1,476/2,952. Backend tests share available_backends() and assert_h2v2_row_matches_reference(). Verified on cuda against dae7106: tests 994 default / 965 scalar-only, strict clippy findings unchanged, decoded bytes identical over the A/B corpus (1850/1850). On studio (NEON): tests 972 default / 972 scalar-only / 986 bench-internals, lib and non-lib clippy clean. cuda (Ryzen 7 5800X3D), kodak24 q90 matched batch, median of 5 interleaved rounds, dae7106 -> this commit: 4:2:0 32.40 -> 31.42 ms on one worker and 5.17 -> 5.00 ms on 16; 4:4:4 38.62 -> 37.56 and 6.00 -> 5.84 ms. The 2-3% gain is outside the run-to-run spread and also present without the butterfly change; its source was not isolated. studio (NEON), 4:2:0 on one worker: 16.61 -> 16.60 ms.
Each MCU row is decoded into a single stripe that keeps the two rows the fancy upsampler needs from the row above, and its output rows are written immediately through the existing Backend row-pair kernels. The three rotating stripe buffers and the second upsample/colour pass are gone. Restart intervals and region requests use the same path; all seeks go through restart::scan_position_before_mcu and skip_to_mcu_with. Removes emit/region420.rs, Fast420RegionStripe, the old full and region 4:2:0 drivers, and Backend::prefers_cropped_420_region (the streaming emitter keeps its crop condition), with the two tests of that removed policy. Verified on Apple M4/M5: byte-identical on 540 files x 23 decode modes, j2k-jpeg tests default and scalar-only, x86-64 clippy type-check.
…ions 4:4:4 YCbCr and identity-RGB scans (R,G,B component IDs or Adobe APP14 transform 0) share one column-window row decoder with the 4:2:0 streaming path and write each MCU's pixels straight to the output. Region requests use it too. decode_scan_fast_rgb_444 and emit_stripe_rgb_444 are generalised in place into the _v1 functions; matches_fast_444_device_shape and matches_fast_rgb_v1_shape mirror the existing 4:2:0 CPU/device split. Verified on Apple M4/M5: byte-identical on 540 files x 23 decode modes, j2k-jpeg tests default and scalar-only, x86-64 clippy type-check.
4:2:2 scans use the shared column-window row decoder from the 4:4:4 streaming path, with horizontal-only fancy upsampling per row, for full images and region requests. Verified on Apple M4/M5: byte-identical on 540 files x 23 decode modes, j2k-jpeg tests default and scalar-only, x86-64 clippy type-check.
next_entropy_marker replaces the two byte-at-a-time restart loops and four other marker scanners. For restart-coded scans, Decoder::checkpoint_for_mcu returns the restart-interval start from a RestartIndex built on the second seek and kept on the decoder; malformed streams are not retried. scan_position_before_mcu uses that checkpoint only when it is exactly the right interval start and otherwise scans for the marker as before, and in both cases reads the bytes after the marker, so errors and offsets are unchanged. bit_reader.rs gains data_position/from_data_position. Verified on studio: 444 files (corpus, large RST variants, 60 corrupt variants) x full decode, restart index, regions cold/warm and DCT windows: 22,644 result lines identical to base; tests default and scalar-only, fmt, clippy, x86-64 clippy, Metal tests, CUDA check.
…s the skip CpuCheckpointCache stores checkpoints compactly on a grid of MCU-row starts plus spans within each row, with constant-time lookup, and publishes finished segments instead of holding its mutex across the skip pass. DeviceCheckpoint's layout (the GPU packet ABI) is unchanged and is still the seek descriptor. Checkpoint storage accounting reuses the existing reserve/reconcile helpers, made generic; checkpoint_allocation_bytes is removed. McuSkipper is passed to skip_to_mcu_with. Verified on studio as for the restart index: 22,644 result lines identical to base; tests, fmt, clippy, x86-64 clippy, Metal tests, CUDA check. The serial grid build is 0-4% slower on a cold first touch of plain scans.
The device DCT window path asked scan_position_before_mcu for no checkpoint and so skipped from the scan start; it now passes the decoder's checkpoint. The consumer contract for checkpoints is documented on ScanPosition. Verified on studio as for the restart index: result lines identical to base; tests, fmt, clippy, x86-64 clippy, Metal tests, CUDA check.
8208c32 staged the whole CHANGELOG.md and so committed six Unreleased entries (decode_srgb8, bounded row warnings, lenient batch preparation, Python device batch preparation, reduced-resolution origins, the README Grok note) that describe uncommitted changes in other crates. This removes them from the committed file only; the working tree keeps them as uncommitted edits for the change that owns them.
Without the cuda-runtime feature the hand-written Default only zeroed the submission counter, which clippy::derivable_impls rejects. Derive Default in that configuration and keep the hand-written impl, which builds the owned packet cache, only with cuda-runtime. Checked on cuda: clippy -p j2k-jpeg-cuda --lib -D warnings and the non-lib targets (disallowed_* allowed, as xtask does), with and without cuda-runtime; cargo fmt --check.
…and 8-bit output On x86-64 the j2k facade converted streamed Gray16 rows (and its 8-bit and owned-batch row conversions) in code built for baseline x86-64, outside any runtime SIMD dispatch, so each sample called the C library's roundf, and the saturating float-to-integer cast ran one lane at a time even under AVX2. Profiles on a Ryzen 7 5800X3D put 40-55% of 12-bit CT, radiograph and mammography decode time in that conversion. j2k_native::at_simd_level (doc-hidden) runs a closure at the dispatched SIMD level. round_clamped_native moves from image/native.rs into math.rs (pub, doc-hidden) and on x86-64 rounds by adding 1.5 x 2^23 and taking the low bits, which vectorizes; elsewhere it keeps the saturating cast (one fcvtzs on aarch64). The native packers and the facade's U16Conversion both round through it, removing the facade's duplicate round().clamp(). The x86 codegen audit now also disassembles j2k, and target_bytes_for_bpp uses an exact libm-free ceil so the audit stays clean. cuda (j2k ms, 1 thread): ct58 104.5 -> 69.3, dx2 122.3 -> 69.2, mg4 137.6 -> 68.6, mr64 21.8 -> 17.9; 12-bit sets now beat or tie OpenJPH, Grok and OpenHTJ2K. 43,616 outputs byte-identical on x86-64 and aarch64; fmt, three clippy levels, 1640 j2k-native/j2k tests, x86-codegen-audit, no-std and aarch64 checks pass; aarch64 timings unchanged.
… state for streaming decode Fixes the duplication found in the streaming commits (9ad65f1..c4bbc39): - one shape rule (fast_packet_family_for_shape) behind matches_fast_{420, 422,444}_shape and fast_rgb_route; the matches_fast_tile_shape alias and the separate device/CPU predicates are gone. Identity-RGB 4:4:4 now takes the Metal 4:4:4 fast path too (CUDA's route still requires YCbCr); - enter_scan_at_mcu{,_with} replace four copies of the seek-and-skip prologue, on top of scan_position_before_mcu, still the only seek path; - McuColumns, CropWindow, RestartState, FastRgbRoute/FastRgbRequest, FastTile420Components::{resolve,skip_mcu}, uses_fused_422_row and rgb420_chroma_rows replace repeated window, crop, restart, route, component and chroma-row code; sequential_mcu_grid and checked_add_allocation_bytes are reused; skip_one_mcu_with_entries is removed; *_rgb_v1 names become *_fast_rgb_rows. Two in-kernel MCU skips and the 4:4:4 kernel's IDCT dispatch stay inline: sharing them made the kernels spill (+1.6-4.5% on Apple M4). Byte-identical (540 files x 23 modes) on studio and cuda; region and DCT window probes identical; tests pass on studio, mini and cuda; strict clippy unchanged. Neutral on cuda and M5 Max; about 1% slower on M4.
The resident batch decoder is the only caller of PreparedPlanCacheKey::prepared_operation and the only producer of the PreparedOperationGray and PreparedOperationColor kinds, and it compiles only on macOS. On Linux both were dead code, so clippy-strict and cargo xtask ci failed there at 43b742b. Gate them on the same target as their caller.
The 9/7 precision pins carried a second fixture and pin set for six synthetic RGB JPEGs, because the baseline encoder's floating-point tie fallback took its cosines from the platform libm and encoded them differently on x86_64. The integer forward DCT removed that split, and it also re-encoded seven of the nine synthetic fixtures on every architecture, so neither old set matched. Drop the x86_64 tables and pin the new fixture bytes (the FNV-1a 64 values the JPEG determinism series reports for them) with one set of pins. The six conformance fixtures and the two grayscale synthetic fixtures are unchanged, and so are their 32 pins. The other 28 come from the same binary64 (F64) and canonical binary32 (F32) paths on the new bytes. On these inputs two two-layer outputs differ in length between F64 and F32 (420_512 and 420_1100x1030, by 2 and 4 bytes), so a pin now records the length and digest per precision. 19 of the 60 outputs differ between F64 and F32. Generated on aarch64 (M4); x86_64 checks the same values in ci and release-cpu.
…y is gone The constant-ownership policy allowlisted 39 known copies of canonical codec literals: the CUDA and Metal JPEG colour conversion and the CUDA J2K encode forward ICT. The JPEG determinism and CUDA encode series replaced all of them with imports from j2k-codec-math or generated Metal fragments, so every entry now counts zero. Remove the allowlist, its ceiling and its checks: any copy outside the owner now fails repo-lint.
…encode work Add the Unreleased entries for the JPEG encode determinism, GPU JPEG 2000 parity, canonical 9/7 transcode, CUDA HTJ2K encode, CPU decode, CPU encode, signed 16-bit output, reserved Adobe transform and IIIF series, including the two output changes and the CPython 3.11 minimum.
Regenerate the stable API snapshots and the reviewed public API diff for the integrated GPU parity, canonical 9/7 transcode, JPEG encode determinism and IIIF series, and merge their review entries. - j2k-codec-math adds 85 items: 43 JPEG ISLOW helpers (already reviewed), 4 GPU-parity owners (htj2k::MEL_EXPONENTS and three generated Metal fragments), 16 canonical binary32 9/7 transcode items (transcode97) and 22 JPEG determinism items (jpeg::fdct, jpeg::ycbcr forward weights, jpeg::upsample). Additions only. - j2k-transcode adds TranscodePrecision, JpegToHtj2kOptions::precision and DctToWaveletStageAccelerator::check_dwt97_precision (a provided method); struct-literal builders of JpegToHtj2kOptions must add the field. - j2k-native and j2k add the coding-style inspection that IIIF needs. stable-api and semver regenerated on aarch64-apple-darwin with the pinned tools; every other package's fingerprints are unchanged from the per-series reviews.
The JPEG determinism series moved the decode (91881, 22554, 46802, 116130) and encode (19595, 7471, 11059, 5329) YCbCr fixed-point constants into j2k_codec_math::jpeg::ycbcr, which the CPU, Metal and CUDA codecs now import. The ownership policy still named codec-math's jpeg.rs and j2k-jpeg's sample_planes.rs, so repo-lint reported the new owner as a copy. Name jpeg/ycbcr.rs as the single owner of all eight.
clippy 1.99 on Linux rejects chunks_exact with a constant size (chunks_exact_to_as_chunks) in the canonical 9/7 reference added by the CUDA transcode series, which failed clippy-strict there. Take the blocks as [f32; 64] arrays directly, which also drops the slice-to-array conversion.
After the integer forward DCT, the only truncating cast left in every_quantizing_path_matches_the_canonical_transform_including_ties sits in its aarch64 NEON block, so on x86_64 its cast_possible_truncation expectation went unfulfilled and clippy-strict failed. Convert the sum with i16::try_from (64 eight-bit samples fit) and drop the expectation, so the test lints the same on every architecture.
Each classic Tier-1 wave was one Rayon parallel iterator over its blocks, and every block allocated its column samples and flags. Equal splits of a wave finish at different times on cores of different speeds, and the allocations repeat for every block of an image. As the HT path already does, one workspace per worker now lives for a wave, and workers pull batches of up to eight blocks off one queue; batches shrink to one block as the queue empties, so the last blocks spread over every worker. Waves with fewer than two blocks per worker keep one Rayon task per block with fresh scratch, as before: at about one block per worker the queue made some small encodes slower. Results still land in their ordered slots, so output is unchanged. The workspaces start empty, grow to exactly the largest block a worker codes and are released at the end of each wave, so the wave frontier (every block's output bound plus one scratch per worker) still covers them; their owner vector is charged like the HT one. A test checks blocks of mixed sizes and sub-bands against fresh encodes in a small wave and in a queued one, and the column-word differential test reuses one workspace across all of its cases. Ported from the archived 06bbdf218 onto 0.14.0. Interleaved A/B against 43b742b, 4096x2048 tissue mosaic (mirror-extended or cropped), medians, A/A = base against itself. Large images, 5 rounds x 7 encodes, measured before the small-wave fallback (their waves stay on the queue): M5 Max (studio), 18 threads: classic lossless RGB8 4096^2 132.2 -> 129.2 ms (-2.3%, A/A -0.3%) classic lossless RGB8 2048^2 32.1 -> 31.5 ms (-2.0%, A/A -0.3%) classic lossless Gray8 2048^2 16.8 -> 16.6 ms (-1.2%, A/A +0.3%) classic 9/7 RGB8 4096^2 123.9 -> 121.5 ms (-1.9%, A/A +0.2%) classic 9/7 RGB8 4096^2 1024 tiles 2 bpp 254.7 -> 229.5 ms (-9.9%, A/A -0.8%) other classic and HT cases within the A/A range; one thread unchanged (all within +/-0.6%). Ryzen 7 5800X3D (x86-64), 16 threads: classic 9/7 RGB8 4096^2 415.6 -> 394.9 ms (-5.0%, A/A -1.5%) classic 9/7 RGB8 4096^2 2 bpp 558.0 -> 530.6 ms (-4.9%, A/A +1.0%) classic 9/7 RGB8 4096^2 1024 tiles 2 bpp 1042.1 -> 1021.1 ms (-2.0%, A/A +0.2%) other cases within the A/A range. 3 and 8 concurrent callers: within the A/A range on both hosts. Rechecked with the fallback: 9/7 RGB8 4096^2 1024 tiles 2 bpp -8.8% on the M5 Max (A/A +0.1%, 5 rounds); on the Ryzen (9 rounds) 9/7 RGB8 4096^2 412.8 -> 402.6 ms (-2.5%, A/A -0.4%, 8 of 9 rounds faster) and 2 bpp 575.5 -> 530.8 ms (-7.8%, A/A -2.3%). Small images (9 rounds x 301-401 encodes; their classic waves hold 4 to 192 blocks), with the fallback below two blocks per worker: M5 Max: lossless 256x256 RGB8 (48 blocks) -8.9%, 512x512 Gray8 (64) -6.6%, 512x512 RGB8 (192) -6.5%; the others within the A/A range. Ryzen: 256x256 RGB8 -5.2%, 512x512 Gray8 -5.0%, 512x512 RGB8 -4.3%; the others within the A/A range, 128x128 RGB8 0.99 -> 0.98 ms (-0.5%, A/A -0.2%). Without the fallback, lossless 181x181 Gray8 (16 blocks on 16 workers) was 3.2% (M5 Max) and 4.8% (Ryzen) slower and 128x128 RGB8 (12 on 12) up to 3.8% slower on the Ryzen, while 9/7 256x256 Gray8 at 1 bpp (16 on 16) was 4.4% and 8.8% faster; the fallback gives that gain up. A fallback below 4 or 8 blocks per worker also gave up the 256x256 RGB8 gain on both hosts. One thread on the Ryzen: classic lossless RGB8 4096^2 +1.0% for this commit alone (7 rounds), +0.2% for the whole series (5 rounds, within the A/A range). Codestreams are byte-identical (57 cases, 1 thread and all threads, both hosts).
Every input other than 8-bit RGB was converted to f32 planes by one thread
in a loop that read each sample through a byte loop of runtime length. For
a 2048x2048 Gray16 HTJ2K lossless encode that loop took 55% of the encode
on the Ryzen (14.4 of 26.3 ms at 16 threads). Inlined into its callers, its
speed also moved between builds that did not change it: by 9% on the Ryzen
(Gray16 2048^2, one thread) and by 53% on the M5 Max (Gray8 128^2, 17 vs
26 us), so unrelated commits measured as regressions on Gray images.
The serial fill is now one out-of-line function whose loops are specialized
to samples of one to four bytes, filling a plane at a time; wider samples
keep a runtime-width loop. Images of at least 3 x 65536 samples, the floor
of the 8-bit RGB path counted in samples, fill each plane on the Rayon
pool instead, as 8-bit RGB already does; with a floor of 65536 samples a
256x256 Gray8 HTJ2K encode took 4.7% longer on the Ryzen. The per-sample
conversion is unchanged. Planes are reserved and filled instead of zeroed
and overwritten, and a short pixel buffer still leaves the rest of each
plane zero. A test checks one to four components at 8 to 38 bits, signed
and unsigned, on and off the pool and with a short buffer, against the
reference deinterleave.
New in this port, not from the archive: found while checking the next
commit, whose build made the old loop slower.
Interleaved A/B against the previous commit, 4096x2048 tissue mosaic
(cropped), 7 rounds x 9 encodes, medians, A/A = the previous commit
against itself (within +/-0.9% on the M5 Max, +/-4.2% on the Ryzen):
M5 Max, 18 threads Ryzen 7 5800X3D, 16
HT lossless Gray8 2048^2 6.85 -> 2.69 ms -61% 24.57 -> 13.31 -46%
HT lossless Gray16 2048^2 7.96 -> 3.24 ms -59% 26.04 -> 13.77 -47%
HT 9/7 Gray8 2048^2 7.02 -> 2.88 ms -59% 24.59 -> 13.91 -43%
classic lossless Gray8 4096^2 64.67 -> 47.66 ms -26% 205.67 ->156.46 -24%
classic lossless Gray8 2048^2 16.41 -> 12.26 ms -25% 47.15 -> 36.53 -23%
classic lossless Gray16 2048^2 30.84 -> 26.24 ms -15% 86.48 -> 78.47 -9%
classic 9/7 Gray8 2048^2 1 bpp 25.35 -> 21.00 ms -17% 76.48 -> 65.63 -14%
classic 9/7 Gray16 2048^2 32.23 -> 27.46 ms -15% 93.34 -> 82.14 -12%
Small Gray images, 7 rounds x 301 (M5 Max) or 201 (Ryzen) encodes,
128x128 to 512x512: every case faster on the M5 Max (-5.7% to -21.0%);
on the Ryzen every case faster (-5.3% to -20.0%) except 512x512 classic
lossless, unchanged (-0.5%). One thread, the whole series against
43b742b (the other two commits are within +/-0.6% there): Gray16 HT
2048^2 -23.5%, Gray8 classic 2048^2 -2.1% on the M5 Max; -22.3% and
-4.2% on the Ryzen.
8-bit RGB inputs take the unchanged RGB path. Codestreams are
byte-identical.
An encode called from outside the pool ran each of its many short parallel stages, one per sub-band and DWT level, as an injected job: the calling thread slept on a latch and the workers were woken for every stage. The CPU entry points (encode, encode_htj2k, encode_htj2k_with_qfactor, encode_with_roi_regions and the component-plane encoders) and each rate-search candidate (IrreversibleScaleEncoder::encode_at_scale) now run through one helper, profile::on_pool_worker. It moves the encode to a pool worker when the pool has more than one thread and no other call is in flight, so consecutive stages find the workers awake. Beside other callers, a moved encode's waiting worker would take their jobs and hold the moved encode until they ended, so concurrent callers keep their own threads as before; calls from a pool thread, or on a one-thread pool, run in place. Encodes of less than 64 KiB of input samples also run in place: moved, they lose more to the hop than their few stages gain. Without that floor, on the M5 Max, 128x128 RGB8 classic (48 KiB) took 5.1% longer, 181x181 Gray8 classic (32 KiB) 2.6% and 128x128 Gray8 HT (16 KiB) 20.6%, while 256x256 Gray8 HT (64 KiB) took 14.2% less and 256x256 classic was unchanged. The calling thread's stage summary moves to the worker for the duration and back, so J2K_PROFILE_STAGES=summary rows follow the same path. A panic in the moved work reaches the caller and releases the in-flight count. Tests cover the summary hand-off, a panicking call, and identical bytes from the test thread, a pool thread and four concurrent callers, below and above the 64 KiB floor. Ported from the archived da4ad967b, the most complete of its three versions: 09bb2ff1a depends on an unlanded decoder helper (29b7f1ab) that moves one call even beside others, which the archive measured 4% slower for 3 or 8 concurrent encodes; 5534ba9d1 is the same patch as da4ad967b. This depends on the classic worker-queue commit two before it. On 43b742b alone, it made classic rate-target encodes 35-40% slower on the M5 Max (7-11% on x86-64): a parallel iterator whose root runs on a worker splits more coarsely than one injected from outside, and the classic Tier-1 waves were one such iterator. With that commit's worker queue, the loss is gone. Interleaved A/B, 4096x2048 tissue mosaic (mirror-extended or cropped), medians, A/A = the reference against itself. 8-bit RGB, against the classic worker-queue commit (the previous commit does not touch 8-bit RGB inputs), 5 rounds x 7 encodes; these sizes are far above the 64 KiB floor, which was added after this run and does not change them: M5 Max (studio), 18 threads: HT lossless RGB8 4096^2, 1024 tiles 52.6 -> 40.6 ms (-22.9%, A/A +1.2%) HT lossless RGB8 2048^2 6.18 -> 5.72 ms (-7.4%, A/A +0.3%) HT 9/7 RGB8 4096^2 2 bpp 36.5 -> 33.8 ms (-7.4%, A/A -0.2%) HT 9/7 RGB8 4096^2 Qfactor 90 14.1 -> 13.2 ms (-6.6%, A/A +0.8%) classic 9/7 RGB8 4096^2 1024 tiles 2 bpp 231.4 -> 207.1 ms (-10.5%, A/A +0.3%) classic lossless RGB8 4096^2 1024 tiles 153.8 -> 145.9 ms (-5.1%, A/A -0.1%) other RGB8 cases within the A/A range. Ryzen 7 5800X3D (x86-64), 16 threads: HT lossless RGB8 2048^2 36.0 -> 31.7 ms (-12.0%, A/A +4.5%) HT lossless RGB8 4096^2, 1024 tiles 224.6 -> 203.1 ms (-9.6%, A/A -0.8%) HT 9/7 RGB8 4096^2 2 bpp 206.0 -> 195.9 ms (-4.9%, A/A +0.8%) classic 9/7 RGB8 4096^2 1024 tiles 2 bpp 1017.6 -> 968.3 ms (-4.9%, A/A +0.2%) classic 9/7 RGB8 4096^2 2 bpp 539.1 -> 523.8 ms (-2.8%, A/A +0.0%) other RGB8 cases within the A/A range. Gray8 and Gray16 2048^2/4096^2 against the previous commit: on the M5 Max (7 rounds x 9 encodes) HT 5-8% faster, classic 0.5-2.6% faster or within the A/A range; on the Ryzen (9 rounds x 15 encodes) within the A/A range, HT 3-6% faster. Small images, 7 rounds x 301 (M5 Max) or 201 (Ryzen) encodes, against the reference that lacks only this commit: below 64 KiB (128x128 Gray8/RGB8 classic and Gray8 HT, 181x181 Gray8) and 256x256 classic within the A/A range on both hosts; M5 Max Ryzen HT lossless Gray8 256^2 0.41 -> 0.35 (-15.1%) 1.41 -> 1.29 (-8.3%) classic lossless Gray8 512^2 1.33 -> 1.16 (-12.6%) 4.73 -> 4.45 (-5.9%) classic lossless RGB8 512^2 3.22 -> 2.86 (-11.2%) 9.13 -> 8.48 (-7.1%) HT lossless RGB8 512^2 1.52 -> 1.11 (-27.3%) 7.91 -> 6.78 (-14.3%) (ms; A/A on these rows within -0.8%..+0.5% on both hosts) Criterion (crates/j2k/benches/public_api) on the M5 Max, 5 rounds, the whole series against 43b742b before the first commit's small-wave fallback: cpu_encode_matrix rgb8_512 classic external 3.93 -> 3.40 ms (-13.4%), htj2k external 1.50 -> 1.05 ms (-30.2%), classic roundtrip 8.89 -> 8.59 ms (-3.4%) (A/A within -2.1%..+1.7%). With the fallback, the A/B harness puts the whole series at 128x128 RGB8 classic lossless within the A/A range on both hosts (+0.5% M5 Max, -0.1% Ryzen) and 128x128 Gray8 at -6.4% and -6.2%. One thread, and 3 or 8 concurrent callers (3x12 and 8x6 encodes, 7 rounds): within the A/A range. Codestreams are byte-identical (57 cases, 1 thread and all threads, both hosts).
Encodes through `encode_with_accelerator` stayed on the caller's thread even when the accelerator declined every remaining stage, as Metal's Auto route does for lossless input and outside its measured sizes, so they missed the pool-worker gain of the CPU-only entry points. Once the whole-tile hooks have declined and `declines_every_stage` holds, a single-tile encode of at least 64 KiB now runs its staged transform, Tier-1 and Tier-2 work through `on_pool_worker_for`. A stand-in keeps the caller's two CPU policies and declines every hook, so the caller's accelerator need not be `Send`. Smaller encodes keep calling the caller's hooks. M5 Max, 10 interleaved rounds against a1cb1536 (A/A 98.9-101.3%): 4096x2048 RGB8 HTJ2K lossless through Metal Auto 15.06 -> 13.47 ms (89.4%, every round 88.4-90.1%). The CPU encodes and Metal Auto Qfactor 90 are within noise. Output is unchanged.
Replace the Unreleased heading and staged-version line with the dated 0.14.0 heading, so the release integrity check passes in publish mode. The release comparisons table drops its "unreleased" marker, and README and SECURITY name 0.14.0 as the current release candidate, as the 0.13.1 release did. 0.13.1 stays the latest published release until the tag.
The 0.14.0 entry was one list of 126 changes in merge order. Add an "Upgrading from 0.13" section that collects what can need an edit or change output (the three API changes, the Python 3.11 minimum, the security fix, the output changes, stricter validation, files that now decode and the reserved Adobe APP14 handling), and a "Highlights" section for the new features and the release's measured speed-ups, with the two measured slowdowns. The end-to-end figures come from the release-vs-main benchmark on an M5 Max, an M4 and a Ryzen 7 5800X3D with an RTX 4070 SUPER. The existing entries follow under "All changes", unchanged.
The PR diff check rejects the blank line each file gained at its end in the comparator row-packing change.
The pclr parser accepted any 7-bit depth descriptor, up to 128 bits, while ISO/IEC 15444-1 I.5.3.4 stops at 38 and ihdr and bpcc already reject more. The depth reaches the 8-bit packer's `1 << depth`: from 64 bits a build with overflow checks panicked and a release build returned every pixel as 255. Reject a column above 38 bits as an invalid box. Lenient parsing skips the palette and records the recovery, as it does for other malformed palettes.
`Rect::is_within` accepts a rectangle with zero width or height, so an empty region passed the six region bounds checks. The fast 4:2:0 route then computed `y + h - 1`, which underflows: a panic with overflow checks and a wrapped row index without them. Component rows and tile planning returned `Ok` for the same request. Check the region once in `ensure_decodable_rect` and return `RectOutOfBounds` for an empty one, as `lossless_crop` already does. The six call sites share it instead of repeating the bounds check.
…ot complete Resident lossless encode commits separate prepare and Tier-1 command buffers and waited only for the final one. The earlier buffers were kept for GPU timestamps and their status was never read. They write coefficients into pooled buffers, so if one failed the final command buffer still assembled a well-formed codestream from the pool's stale contents and reported an OK status. After the final command buffer completes, require every retained command buffer to have completed too. The split-command decode used for stage profiling waited only for its last stage; it now checks the four earlier ones the same way.
`--threshold-percent NaN` and `inf` parse as `f64` and passed the non-negative check. No regression compares greater than either, so the guard could not fail.
`cargo xtask clippy` fails on `doc_markdown` here wherever libjpeg-turbo 3 is installed, because the test only compiles with it.
The crate documentation still linked dicom-toolkit's PNG example, which uses that crate's API, and repeated its testing and performance statements. Point to the support matrix, the T.803 results and the benchmark evidence instead, and say which default features bring in dependencies that use unsafe code. The provenance paragraphs are unchanged. A forward 5/3 helper was documented as integer arithmetic; it computes on `f32`.
No workspace member references it. The lockfile is unchanged.
… a known issue Add the three fixes to the 0.14.0 entry and its upgrade notes. Note a known issue in the changelog and the support matrix: reversible coding of 23- and 24-bit samples runs in `f32` and is not exact for every input. Lossless encodes checked against OpenJPEG 2.5.4 differed from the source at 23 and 24 bits and matched at 22 bits and below, and a 24-bit lossless codestream written by OpenJPEG decoded with samples off by one.
Reversible coding of 21- to 24-bit samples ran the colour transform and the 5/3 wavelet in `f32`. A lifting step sums two neighbours, and through any number of decomposition levels that sum reaches almost 16 times the largest sample magnitude, so past 20 bits it no longer fits the 24 bits an `f32` holds exactly. Lossless encodes of such images could fail their round-trip check and decodes could return samples off by one, on ordinary content as well as on worst-case input. One rule, `reversible_coding_needs_integers`, now sends reversible samples deeper than 20 bits down the integer path that 25- to 38-bit samples already took, on encode and on every decode route. Decoded planes of at most 24 bits are handed back as `f32` planes, which hold the finished samples exactly, so component-plane, region and bounded-row decodes still serve these images. Samples of 20 bits and fewer take the same routes as before. The integer path is slower and needs more memory. On a 2048x2048 24-bit gray image on an M4 Pro, classic decode takes 318 ms where the inexact path took 47 ms, and a full decode peaks at about 34 bytes per sample. Requests with no integer path are refused instead of coded in `f32`: resident, precomputed and device reversible encodes deeper than 20 bits (`RequireDevice` included), raw-pixel reversible encodes deeper than 20 bits with subsampled or per-component metadata, and decodes of such images whose tiles mix wavelets or colour transforms. Rules that several backends kept their own copies of now have one source, and four of those copies had drifted: - The unsigned level shift of 32- to 38-bit samples overflowed a 32-bit shift: a debug panic, and wrong samples in release. `ComponentInfo` computes it for the full decode, the direct plan and direct-CPU colour. - Irreversible ROI decoding classified background coefficients below the shift as region. `roi_maxshift_is_region` in j2k-codec-math decides it for the native reversible and irreversible paths and the CUDA kernels, and the Metal shader mirrors it. - The CUDA transcode IDCT skipped the CPU's DC shortcuts, and the test reference for IDCT-then-5/3 called the code under test. Both follow the CPU oracle and T.800 Annex F. - Direct-CPU colour saturated samples deeper than 16 bits when writing 8-bit output. One `sample_as_u8_native` conversion serves it and the facade. The facade takes its sample limits from j2k-types: 24 bits for `f32` planes, 20 for reversible coding in `f32`, and 38 for Part 1. Component index widths, the CUDA classic kernel name and the Metal encode failure details are each named once, with tests that hold host and shader copies equal. Tests hold the `f32` path equal to the integer path on worst-case input at 20 bits and show it differ at 21, round-trip worst-case samples at every precision from 8 to 24 bits, compare 17- to 24-bit lossless output with OpenJPEG in both directions, and fail on the old level-shift, ROI and 8-bit conversion code. The support matrix names them, the changelog records the fixes, and the stable API snapshots gain the new items. The tile-batch region-scaled benchmark encodes its fixtures with two decomposition levels, as the other quarter-scale benchmarks do. With the default single level every quarter-scale request was refused and the host performance lane stopped there.
The release is tagged on 2026-11-12, not 2026-11-10.
The 0.14.0 review records the roi module and roi_maxshift_is_region in j2k-codec-math, seven hidden sample and coding limits in j2k-types, the hidden CUDA classic kernel name in j2k-cuda-j2k-engine and the hidden sample_as_u8_native in j2k-native, with their regenerated fingerprints. The reviewed diff report is regenerated. Nothing is removed.
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release candidate for j2k 0.14.0: 183 commits on
main(f1a5441), each scoped to one change.The 0.14.0 entry in
CHANGELOG.mdstarts with Upgrading from 0.13 (API edits, the Python 3.11 minimum, the ICC security fix, output changes and stricter validation) and Highlights (new features and measured speed-ups). The full per-change list follows under All changes.Closes #300, closes #304, closes #308.
Merging
Please don't squash or create a merge commit.
mainis fast-forwarded to exactly this head (61740d5), so the release tag,full-validation.ymlandgpu-validation.ymlall refer to the commit that was validated.Validation already run on this head
Every gate below ran on 61740d5 itself.
public-support --final;cargo xtask ci: 249 suites, 6108 passed, 0 failed;release-cpu: 97 suites, 3030 passed, 0 failed;release-metal --mode full: 82 suites, 1840 passed, 0 failed.cargo xtask ci: 249 suites, 5219 passed, 0 failed;release-cpu: 97 suites, 3022 passed, 0 failed;x86-codegen-audit;release-cuda --mode quick: 37 suites, 808 passed, 0 failed;release-cuda --mode full: 54 suites, 1089 passed, 0 failed, 0 skipped;bench-build --lane cuda.release-integrity, in normal and--publishmode;cargo xtask package;gpu-validation.yml, target all, mode quick: run 43228112975, CUDA and Metal both passed.Semver ran in full only in CI. Locally the API review passed and 15 of 16 packages reported no semver update required before the crates.io index timed out.
j2k-perf-guardhost and Metal lanes against 7219321 found no regressions.Still to run
gpu-validation.yml, target all, mode quick, on this head. The PR's GPU evidence check needs it.mainis fast-forwarded:full-validation.ymlon push, thengpu-validation.ymlwith target all and mode full.cargo xtask release-status --sha 61740d590c44283fb1c05ae7d4ca44d820897790 --scope all, then the annotatedv0.14.0tag.