diff --git a/CHANGELOG.md b/CHANGELOG.md index c2749a9ed..3d429fd80 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,118 @@ Staged workspace version: `0.13.2`. status, detail and batch source index. CUDA keeps its typed source and job attribution. Unsupported statuses, raw runtime failures and combined cleanup or release failures keep their existing classes. +- Row-streamed decodes on serial bands finish directly from the inverse + transform's own rows. The streamed inverse transform lifts every level + with the banded transform's kernels, one arrival at a time on each level's + ring of rows; on serial bands, a row is finished directly from the ring + (inverse colour transform, level shift, rounding and packing) once no + lifting step reads it, in place of a copy into scratch rows, and the 8-bit + RGB finish is shared with the component planes' fused last level. A tile + on the worker pool with one reader finishes its rows the same way instead + of copying them into batches of up to 32 rows, and an HTJ2K tile whose + components reconstruct on readers of their own keeps each of its two + pipelined row batches to at most 256 rows. On an 18-core ARM64 machine a + 5428 x 4460 RGB image at 1.5 bits per pixel streams in 0.92 of its + previous time on 6 threads, a 4096 x 2048 HTJ2K image in 0.90 to 0.95 on 1 + to 18 threads and with 48 MiB in place of 72 MiB on a pool, an 18000 x + 12000 HTJ2K image in 0.88 on 18 threads and 1024 x 1024 HTJ2K regions in + 0.90 to 0.94. Output is unchanged; a decode that fails partway can leave a + different number of rows written, each of them a row the successful decode + writes. +- Streamed decodes keep to the memory they charge and hand each row over + once. A tile whose components reconstruct on readers of their own sized + its batch of rows by the region's width while the decode charged the + tile's width, so a tile just narrower than its region could hold twice its + charge (an 8,192-wide tile of an 8,200-wide image: 6.4 M against 3.4 M + samples); its batch now keeps to the charge, and such an image decodes in + 0.98 of the time on 12 and 18 threads, with 60.7 MiB in place of 72.6 MiB + on 12. A serial HTJ2K region decode allocated the scratch for code-blocks + its region clips when it met the first one, so a decode short of budget + could fail after its first rows reached the sink, where the retries + without retained scratch or without the packet directory no longer run; + the scratch is now sized with the rest of the Tier-1 workspace before the + first row. A tile packing two batches of rows into an RGB8 destination + counted a packed batch only once the next batch had reconstructed, so a + scheduler fault there could decode the tile again and write those rows + twice; the packed rows now count at once. Output of decodes that succeed + is unchanged. +- Images that a batch with an explicit worker count decodes together on + its stream pool, when it plans as many images at once as the pool has + threads and the pool has fewer than 12, hold two bands in every ring of + the pool's bands in place of three at the finest level, and keep their + bands whole: a pool thread with no band of one image to decode takes + another image's bands. Eight 2048 x 2048 RGB images peak at 116.9, 138.3 + and 182.6 MiB in place of 121.4, 147.3 and 200.6 MiB on 2, 4 and 8 + workers, and ten on 10 workers at 229.5 in place of 252.0 MiB, in the + same batch time. From 12 threads, where a lone tile holds deeper rings, + the images keep the usual rings, as do an image encoded in more than one + and a half times its group's mean bytes, the likeliest to decode last + with the pool to itself, groups planned with fewer images at once than + pool threads, and single-image decodes. Output is unchanged. +- Classic tiles of several components that stream their rows from the + worker pool's bands with a reader per component and full-width reads, + but without the deeper band rings of 12 threads and more, as on 10 and + 11 threads or at a reduced resolution, hold two bands in each + finest-level ring in place of three when the pool has more threads than + three per component. The bands ahead of the readers are then fewer than + the pool's threads, so they split into parts that every thread takes. A + 5661 x 8287 RGB image on 10 threads peaks at 49.2 MiB in place of 55.3 + MiB and an 18000 x 12000 one at 148 MiB in place of 168 MiB, and the + latter at half resolution on 12 threads at 79 MiB in place of 89 MiB, + with the same or slightly shorter decode times. Output is unchanged. +- `decode_rows_u8_bounded` and `decode_rows_u16_bounded` decode an image + whose tiles span its width, and whose headers let its region decode stream + its rows, once, as a region decode of the whole image that streams its + rows into a buffer of at most the requested stripe rows and hands them + over a stripe at a time. They decoded every stripe as a region decode of + its own, which decoded each code-block again for every stripe its wavelet + support reaches: at the default 64-row stripes 6.0 to 7.4 times the + code-block samples of one full decode on 2048- to 8287-row images. On an + 18-core ARM64 machine the default stripes of a 5661 x 8287 RGB image take + 1.34 s instead of 12.9 s on one thread and 0.11 s instead of 1.52 s on 18. + The decode's peak heap is then that of the streamed full-image decode + (22.5 MiB instead of 10.3 on one thread, 69 instead of 12.8 on 18); the + process's peak resident memory beside its input and output falls on such + classic images (25.6 MiB instead of 31.8, 72.8 instead of 77.4) and rises + on HTJ2K images on a pool (a 4096 x 2048 image on 18 threads: 76.9 MiB + instead of 42.6). Rows, bytes, the stripe bound and errors are unchanged: + a decode that fails before its first row reaches the sink decodes a stripe + at a time instead, and a sink that refuses a row still stops the rows + there, though a one-pass decode now goes on until it completes or fails. + Whether an image decodes once is planned from its main and tile headers: + images whose tiles sit side by side, whose headers keep region decodes on + component planes, or whose main header codes no decomposition level, and + 16-bit rows of images deeper than 24 bits, still decode a stripe at a + time, and an image whose tiles turn out not to stream (a tile-level coding + style without decomposition levels, or HTJ2K code-blocks mixed with + classic ones) decodes into component planes of the whole image and hands + their rows over a stripe at a time. The stripe bound covers the packed + output stripe, not the decoder's working memory. +- Classic code-blocks of several codeword segments, with selective bypass + or per-pass termination (as archival images and multi-layer encodes of + this crate often have), decode on the worker pool's tile route and pool + bands like single-segment ones: each worker collects a block's segment + descriptors into its own scratch, reserved before it starts, from the + segment records it already reads, with no per-block allocation or plan + growth. Region decodes and full-image decodes that stream their rows now + stream such images on the pool without component planes, and full-image + decodes into planes decode their blocks in place instead of on the + per-sub-band path. Output is unchanged. +- A region decode on the worker pool that streams its rows to a sink, such + as an 8-bit full-image decode whose component planes would pass the + decode budget, now streams them from serial bands when the pool's bands + cannot decode a tile but serial bands can, and the region's planes would + pass the budget. Such decodes of classic images with BYPASS or per-pass + termination failed with a resource limit on pools of two or more threads, + and decode as on one thread now. Decodes whose planes fit keep their + route. +- Parallel band decoding charges each component's plan and the band + executor's records to one decode budget, across all components when a + tile's rows stream together, and releases the plans and the records before + whole-window error recovery or the serial fallback. Band bookkeeping no + longer rescans completed bands, and components whose bands average too + little work for the pool decode over whole windows instead, or on serial + bands when their rows stream. - Classic code-blocks with BYPASS, per-pass termination, context resets or segmentation symbols use the packed-column passes, including raw segments. Vertically causal blocks retain the per-coefficient passes. Clean and corrupt @@ -34,6 +146,32 @@ Staged workspace version: `0.13.2`. term the charge grows to match; elsewhere it is unchanged and the IDWT holds less within it (a 1024 x 768 gray image on 4 threads: 3.9 MB in place of 4.5 MB, against a 4.1 MB charge). Output is unchanged. +- On a pool of up to 10 threads, decodes take fresh component planes, + coefficient storage and IDWT buffers zeroed from the allocator instead of + zeroing them on the calling thread, so the page faults of a process's first + decode are spread over the threads that fill those buffers instead of + running one after another while the pool waits. In a fresh process, + 4096-wide RGB images that decode to planes, HTJ2K and classic, and a + 1/4-scale HTJ2K decode take 5 to 24% less time at 4 to 10 threads. Warm + decodes, output and peak heap are unchanged, and so are decodes on larger + pools, where the faults would contend in the kernel. New dependency: + `bytemuck`. +- `CpuBatchDecoder` streams a Gray8/RGB8 image's rows only when its region + decode can stream them. An image with a channel definition box, or with + image or tile offsets, keeps its offset plan, and is rejected up front when + that plan exceeds the per-decode bound, as other outputs are; before, it was + admitted as streamed but decoded whole component planes. A group whose + images decode with inner parallelism no longer flattens their code-block + payloads first. +- `CpuBatchDecoder` with an explicit worker count of two or more decodes a + group's streamed Gray8/RGB8 images on a pool of exactly that many threads, + so threads that finish their images help the images still running instead + of idling: eight 2048 x 2048 RGB images on six workers kept four threads + busy before. Every image whose rows stream does so at every worker count, + repeated or not. A group of plans that do not stream still decodes on the + staged route at one worker, but one image at a time when its flattened + code-block payloads would exceed the staging caps, instead of failing the + batch. Output is unchanged. - Decodes that would exceed a resource limit now fail with `J2kError::ResourceLimit { requested, cap, source }` instead of the generic `J2kError::NativeDecode`, on every decode route: `J2kDecoder`, the tile @@ -54,12 +192,150 @@ Staged workspace version: `0.13.2`. and exclude those items from the prepared groups. Backend-neutral `prepare_batch` and `prepare_batch_from_images` keep their existing behavior: decoding them on the CPU can still fail the whole call when one - image's output exceeds the allocation cap. On either preparation path, a - group whose combined output exceeds that cap still fails the whole decode - call. + image's output exceeds the allocation cap. On either preparation path, + a group whose combined output exceeds that cap is decoded as several + result groups, each with an output buffer within the cap. Staged windows are admitted by planned execution memory, and an admission claim that cannot fit beside the shared plan fails instead of waiting indefinitely. Successful output is unchanged. +- 16-bit full-image decodes (`J2kDecoder::decode_into`, `decode_scaled_into` + and the decoders behind `decode_tiles_into` at `Rgb16`, `Rgba16` and + `Gray16`) and `J2kDecoder::decode_native_components` whose component + planes, or the planes together with their packed native samples, would + pass the native decode budget now run as a region decode of the whole + image that streams its rows, as 8-bit decodes do, instead of failing with + a resource-limit error. `decode_native_components` packs the rows into its + native planes as they are finished. This covers images without palette, + channel definitions, sYCC or CIELab whose unsubsampled components share one + precision of up to 24 bits, signed or unsigned; other images keep the + planes route and its error. Output is unchanged apart from the fix to + images deeper than 16 bits below. A 5661 x 8287 RGB image decodes to `Rgb16` in about 130 ms with + 103 MiB peak heap on an 18-thread pool (1.39 s and 26 MiB on one thread), + where the planes route with its budget raised to fit takes about 196 ms + and 984 MiB (1.49 s and 952 MiB); its native components take about 122 ms + and 96 MiB besides the 141 MB of planes returned. +- 16-bit region decodes (`decode_region_into`, `decode_region_scaled_into` + and their power-of-two forms) and `decode_native_region_components` of RGB + images without palette, channel definitions, sYCC or CIELab stream their + rows instead of holding the region's `f32` planes and native samples, and + so do those of other such images whose planes would pass the native decode + budget. `decode_native_region_components` takes this route only for images + of one to four components without image or tile offsets, whose region + decodes stream; others keep the planes route, which meets the decode budget + before it allocates the returned planes. Output is unchanged. +- Native component planes (`decode_native_components`, + `decode_native_region_components` and the native decoder's component-plane + decodes) are the decode's output on both routes: the decode budget holds + the `f32` planes they are packed from, and the returned planes have an + allowance of their own, each held to the per-allocation cap, as a + caller's buffer is. Before, the planes route charged them beside the + `f32` planes, so an image whose `f32` planes fit the budget but not + together with their output (one that cannot stream its rows, such as one + of five components or with image offsets) failed with a resource-limit + error; it now decodes. The streamed rows route allocates its planes when + the first rows arrive, once per decode, so a region decode that falls back + to component planes (a component without decomposition levels, or classic + code-blocks of several segments on the pool) meets the decode budget before + any output exists. Output is unchanged. +- 16-bit outputs of images deeper than 16 bits hold each sample's value + clamped to the 16-bit range (two's complement when signed), as scaled + decodes already wrote them. Full-image and region decodes (and row decodes + of images deeper than 24 bits) wrote each byte of the wider native samples + as a sample of its own. 8-bit outputs of such images on the streamed + route, in scaled decodes and in `decode_rows_u8_bounded` quantize each + sample over its component's range, as full-image decodes on the planes + route do, instead of over 16 bits, which saturated most samples. Images + of at most 16 bits are unaffected. +- 16-bit output and `decode_native` on the planes route are faster: two to + four components pack into native samples a block of pixels at a time on + the thread pool instead of one sample at a time, and 8-bit native samples + widen to 16 bits from a table instead of a division per sample. `Rgb16` + `decode_into` of a 4096 x 4096 RGB image takes about 40% of the time it + did on an 18-thread pool. Output is unchanged. +- Region decodes of a tile whose coarse resolutions are empty while finer + ones are not (a one-sample tile row or column at an odd position with two + or more levels) reconstruct it from the first level that has samples, as + full-image decodes do. Before, 8-bit region decodes of such tiles failed + with `IncompletePlaneStore` or `CodeBlockDecodeFailure`, and 16-bit and + component-plane region decodes returned mid-gray samples for them. +- An 8-bit full-image decode (`J2kDecoder::decode_into`, `decode_scaled_into` + and the decoders behind `decode_tiles_into`) whose component planes and + largest tile's workspace would pass the native decode budget now runs as a + region decode of the whole image, which writes rows as the streamed IDWT + finishes them, instead of failing with a resource-limit error. The native + decoder plans the planes route's charge from the codestream's tiles before + decoding (no planes or coefficients are allocated for it), so an image the + planes route decodes keeps that route, its output and its speed. A + 5661 x 8287 RGB image now decodes at full resolution with about 100 MiB + peak heap on the default thread pool (26 MiB on one thread), where the + planes route with its budget raised to fit needs about 980 MiB. Untiled RGB + images decode within 8% of that route's time, 1024 x 1024-tiled RGB takes + up to 45% longer and grayscale up to 2.8 times as long. Images whose tiles + have empty coarse resolutions (a one-sample tile row or column at an odd + position) stream too, at 8 and 16 bits and into native component planes, + since region decodes reconstruct such tiles: a 20000 x 7000 gray image of + 2857 x 2333 tiles no longer fails at the decode budget. +- Full-image 8-bit decodes also stream when streaming was measured to be the + faster route for the image, chosen before decoding from the codestream's + header and tiles: reduced-resolution decodes (`decode_scaled_into`) of + untiled three-component images with an output of 1024 x 512 or more, classic + at any thread count and HTJ2K on two threads or more (a 5661 x 8287 RGB + image at 1/2 to 1/8 takes 5-47% less time on 6 threads or more, the same on + one, in 4 to 13 times less memory; HTJ2K images at 1/2 and 1/4 9-53% less), + smaller classic outputs from 256 x 256 on 2 to 11 pool threads; + full-resolution classic images coded at 4 bits per pixel or more from + 512 x 512 on 2 to 11 pool threads (5-17% faster on 6 threads) and, below + 2^19 samples, on larger pools too; full-resolution classic images coded at + fewer bits per pixel from 4096 x 2048 on 2 to 6 pool threads and from 12 + (a 4096 x 2048 RGB image at 2 bpp 1% faster on 6 threads and 4-5% on + 18, in about half the memory); and full-resolution images with a tile + for every thread. Those now decode their tiles at once on the calling thread + and scoped worker threads, each tile on one thread, instead of one tile after + another: a 4096 x 8192 RGB image of 1024 x 1024 tiles decodes 8% faster than + on the planes with 6 threads and 23% faster with 18, in less than half the + memory. Tiled classic images with fewer tiles stream them one after another + on 2 to 11 pool threads when each tile is a megapixel or more. Tiled images + past the decode budget decode their tiles at once too (an 8192 x 8192 RGB + image of 1024 x 1024 tiles 22% faster with 18 threads, a 16384 x 16384 gray + one 2.9 times as fast). Untiled gray classic images stream at full + resolution from 2896 x 2896 below 12 pool threads and from 5792 x 5792 on + larger pools (12-15% faster on 6 threads, as fast elsewhere, in a fifth to a + seventh of the memory). On one thread (`CpuDecodeParallelism::Serial` or a + one-thread pool) decodes stream wherever streaming was measured as fast or + nearly so in much less memory: classic RGB images of 512 x 512 or more at + 4 bits per pixel or more and of 4096 x 2048 or more at fewer (a + 5428 x 4460 RGB image at 1.5 bits per pixel 2% slower, in 16 instead of + 309 MiB); classic images in tiles of 256 x 256 or more, one tile after + another; reduced classic outputs of 512 x 256 or more, tiled or not; + reduced HTJ2K outputs of 1024 x 512 or more at 1/4 and below; gray classic + images from 1024 x 1024 or in tiles of 256 x 256 or more, and reduced gray + outputs from 512 x 256; and 16-bit outputs of 512 x 512 samples or more of + any coding, untiled or in tiles of 256 x 256 or more, which the planes + route packs from a second set of planes (HTJ2K images 21-25% faster + streamed). On one thread 8-bit decodes of untiled HTJ2K images at full + resolution, of HTJ2K images at 1/2 and of HTJ2K gray images keep the + planes route, as do small images; on two threads or more reduced and + tiled gray images, untiled full-resolution HTJ2K images, reduced tiled + images, small images and images with few tiles on a large pool do. A + stream chosen this way whose own buffers pass the decode budget, as a very + wide and shallow image's line buffers can while its planes fit, now + decodes on the planes route instead of failing, on any thread count. + Output is unchanged. +- Row-streamed decodes of three 8-bit components into RGB8 output write the + reconstructed rows straight into the output: 9/7 images with the + multiple-component transform pack each row through the planes route's + fused inverse-ICT pack instead of copying it out of the inverse wavelet + transform, transforming it in place and packing it through the row sink, + and other images pack into the output after the in-place transform. + Parallel row-streamed decodes keep a classic component's bands on the + worker pool when the component has fewer rows of code-blocks than the pool + has threads, where they decoded on one thread (a whole-image region of a + 512 x 512 RGB image at 2 bpp decodes 4.9 times as fast on 18 threads), + and serial ones look each sub-band up directly instead of searching the + tile's sub-bands several times a row (whole-image regions of 256 x 256 + and 512 x 512 tiles 4-9% faster on one thread). An 18000 x 12000 RGB + image decodes 1-2% faster on one thread and 3-5% faster on 18. Output is + unchanged. - x86-64 RGB8 decodes of 9/7 images round and pack the inverse ICT output with the runtime SIMD level (AVX2 where present) instead of a scalar library call per sample: the centered rounding of the inverse MCT runs eight @@ -94,6 +370,78 @@ Staged workspace version: `0.13.2`. faster lossless at 1 and 6 threads (5% and 13% at 18), and 1024 x 1024 regions and reduced-resolution decodes of an 18000 x 12000 image 2 to 10% faster. Output and peak heap are unchanged. +- Parallel CPU region decodes of classic tiles whose rows stream to a row + sink run every component's bands on the pool at once, and the IDWT + advances the components together and hands the rows to the sink, instead + of decoding one component's bands, then its planes, at a time. At reduced + resolutions, where each sub-band holds few bands, one component's bands + left most of the pool idle. On an 18000 x 12000 RGB image, overviews at + 1/16 and 1/8 scale decode 53% and 47% faster at 18 threads and 1024 x 1024 + regions up to 5% faster, with 29 to 59% less peak heap. HTJ2K tiles are + unchanged. +- The row-streamed region IDWT synthesizes rows with the banded IDWT's SIMD + horizontal kernel, reading the stored band rows in place, instead of + interleaving each row and lifting it with the scalar filter. Output is + bit-identical. On an 18000 x 12000 RGB image, HTJ2K regions decode 27 to + 52% faster and classic 1024 x 1024 regions 16 to 36% faster at 6 and 18 + threads, and 3 to 35% faster at one thread. +- Parallel CPU region decodes streaming rows from pool bands reconstruct + their components in batches of rows, each component on its own pool task + for HTJ2K tiles and for classic tiles with 12 threads or more, and pack + three 8-bit components straight into interleaved RGB8 output. HTJ2K tiles + now stream rows from pool bands too. `ComponentRowSink` gains a + rustdoc-hidden `rgb8_region` method, `None` by default, through which a + sink offers such a destination. On an 18000 x 12000 RGB image the HTJ2K + overview at 1/4 scale decodes 32 to 35% faster, HTJ2K 1024 x 1024 regions + 16 to 20%, and classic 1024 x 1024 regions at 18 threads 9 to 18%. +- Parallel CPU region decodes with a reader per component transform and pack + each batch of rows on every reader's thread instead of the calling thread + alone. The HTJ2K overview at 1/4 scale of an 18000 x 12000 RGB image + decodes 10 to 15% faster. +- Parallel CPU region decodes with a reader per component transform and pack + one batch of rows on the pool while the next reconstructs, in batches of + up to 2 Mi samples per component and a quarter of the tile's rows; a + single reader takes a row of each component at a time. On an 18000 x 12000 + RGB image the HTJ2K overviews at 1/4 and 1/8 scale decode 6 to 29% faster, + and the classic overview at 1/16 scale 12% faster at 6 threads. +- Parallel CPU region decodes of classic tiles that reconstruct each + component on its own reader hold one band more in every ring and one + batch of rows instead of two. On an 18000 x 12000 RGB image, 1024 x 1024 + regions and overviews at 1/8 and 1/16 scale decode 4 to 6% faster at 18 + threads, with less peak heap. +- Those tiles reconstruct half as many rows at a time, while that leaves + at least 128 rows, since they hold only one batch. A full-image region + decode of a 5661 x 8287 RGB image to RGB8 peaks 12 MiB (12%) lower at 18 + threads, at the same speed. +- The row-streamed IDWT reads band rows of up to 16384 samples that hold no + code-block as zeros through its SIMD kernel; such rows wider than 4096 + samples took the scalar path. An 18000 x 12000 RGB image, whose chroma + sub-bands have many such rows, decodes 11 to 12% faster at 18 threads + (441 to 391 ms) and 5% faster at 10. Output is bit-identical. +- Parallel CPU region decodes of classic tiles of several components and + at least 4096 x 4096 samples per component reconstruct each component on + its own pool reader from 10 pool threads, instead of 12, holding one + batch of rows as before. With one reader that reconstructs every + component and packs every row, such tiles were bound by that reader on + 10 and 11 threads: on a 10-thread pool an 18000 x 12000 RGB image now + decodes 11% faster (980 to 868 ms), 5661 x 8287 RGB 8%, 16384 x 16384 + RGB 7% and 8192 x 8192 RGB 5%, at the same peak heap. Smaller tiles, which + spend more of their time in Tier-1, keep one reader. Output is + bit-identical. +- Parallel CPU region decodes that reconstruct every component through one + reader (gray images, and classic tiles on fewer than 12 pool threads that + keep one reader) hold at most 32 rows of each component at a time instead + of up to 512, since that reader transforms and hands over each batch + before it reconstructs the next. A full-image decode of a 5661 x 8287 RGB + image to RGB8 on 6 threads peaks 22 MiB (33%) lower, at the same speed, + and 8 to 16 MiB lower at 1/4 and 1/2 scale; 4096 x 2048, 16384 x 16384 and + 18000 x 12000 RGB images drop 17 to 23 MiB, gray images 7 MiB at any pool + size. +- Classic tiles with a reader per component hold the extra band in every + ring only when they read whole rows at full resolution; overviews and + narrower regions keep one batch of rows without it. A 5661 x 8287 RGB + image's overviews at 1/2, 1/4 and 1/8 scale peak about 5.5, 2.5 and 1 MiB + lower at 12 threads or more, and 1024 x 1024 regions about 2.3 MiB lower. ## [0.13.1] - 2026-10-29 diff --git a/Cargo.lock b/Cargo.lock index f03e74a51..a2b893b38 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2772,8 +2772,10 @@ dependencies = [ name = "j2k-native" version = "0.13.2" dependencies = [ + "bytemuck", "criterion", "fearless_simd", + "j2k-alloc-probe", "j2k-codec-math", "j2k-profile", "j2k-types", diff --git a/Cargo.toml b/Cargo.toml index 61574394d..aaae971c0 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -103,6 +103,7 @@ objc2-metal-performance-shaders-graph = { version = "=0.3.2", default-features = ] } block2 = { version = "=0.6.2", default-features = false, features = ["std"] } rayon = "1" +bytemuck = { version = "1.25", default-features = false } cc = "1.2" libc = "0.2" windows-sys = "0.61" diff --git a/crates/j2k-alloc-probe/src/lib.rs b/crates/j2k-alloc-probe/src/lib.rs index a71c17de0..451e065e0 100644 --- a/crates/j2k-alloc-probe/src/lib.rs +++ b/crates/j2k-alloc-probe/src/lib.rs @@ -8,9 +8,13 @@ //! delayed allocator call from reporting into a later measurement, and each //! measurement waits for already-admitted reporters before taking its //! snapshot. +//! +//! The allocator also keeps the bytes live at any moment and their high +//! water, always on and independent of measurements, for probes that bound +//! working memory rather than allocation calls (see [`reset_peak`]). use std::alloc::{GlobalAlloc, Layout, System}; -use std::sync::atomic::{AtomicU64, AtomicU8, Ordering}; +use std::sync::atomic::{AtomicU64, AtomicU8, AtomicUsize, Ordering}; #[cfg(test)] use std::sync::atomic::AtomicBool; @@ -33,6 +37,8 @@ static ALLOCATIONS: AtomicU64 = AtomicU64::new(0); static REALLOCATIONS: AtomicU64 = AtomicU64::new(0); static DEALLOCATIONS: AtomicU64 = AtomicU64::new(0); static REQUESTED_BYTES: AtomicU64 = AtomicU64::new(0); +static LIVE_BYTES: AtomicUsize = AtomicUsize::new(0); +static PEAK_BYTES: AtomicUsize = AtomicUsize::new(0); #[cfg(test)] static TEST_HOOK: AtomicU8 = AtomicU8::new(TEST_HOOK_IDLE); @@ -60,6 +66,7 @@ unsafe impl GlobalAlloc for CountingAllocator { let pointer = unsafe { System.alloc(layout) }; if !pointer.is_null() { reporter.record_allocation(layout.size()); + grow_live(layout.size()); } pointer } @@ -70,6 +77,7 @@ unsafe impl GlobalAlloc for CountingAllocator { let pointer = unsafe { System.alloc_zeroed(layout) }; if !pointer.is_null() { reporter.record_allocation(layout.size()); + grow_live(layout.size()); } pointer } @@ -77,6 +85,7 @@ unsafe impl GlobalAlloc for CountingAllocator { unsafe fn dealloc(&self, pointer: *mut u8, layout: Layout) { let reporter = Reporter::begin(); reporter.record_deallocation(); + shrink_live(layout.size()); // SAFETY: forwards the pointer and its original layout unchanged. unsafe { System.dealloc(pointer, layout) }; } @@ -88,6 +97,11 @@ unsafe impl GlobalAlloc for CountingAllocator { let replacement = unsafe { System.realloc(pointer, layout, new_size) }; if !replacement.is_null() { reporter.record_reallocation(new_size); + if new_size >= layout.size() { + grow_live(new_size - layout.size()); + } else { + shrink_live(layout.size() - new_size); + } } replacement } @@ -191,6 +205,44 @@ pub fn assert_allocations(label: &str, budget: Budget, operation: impl FnOnce result } +/// Bytes held through the global allocator right now, by every thread. +#[must_use] +pub fn live_bytes() -> usize { + LIVE_BYTES.load(Ordering::Relaxed) +} + +/// Start a new high-water window at the bytes live now and return them. +/// +/// The window is process-wide like a measurement: allocations on every +/// thread raise [`peak_bytes`], so unrelated threads must stay idle while it +/// is open. Windows need no guard and never block the allocator. +pub fn reset_peak() -> usize { + let live = LIVE_BYTES.load(Ordering::Relaxed); + PEAK_BYTES.store(live, Ordering::Relaxed); + // An allocation between the load and the store raised the live bytes + // past `live`; keep it in the new window. + PEAK_BYTES.fetch_max(LIVE_BYTES.load(Ordering::Relaxed), Ordering::Relaxed); + live +} + +/// The most bytes live at once since the last [`reset_peak`] (or since the +/// process started). +#[must_use] +pub fn peak_bytes() -> usize { + PEAK_BYTES.load(Ordering::Relaxed) +} + +fn grow_live(bytes: usize) { + let live = LIVE_BYTES + .fetch_add(bytes, Ordering::Relaxed) + .wrapping_add(bytes); + PEAK_BYTES.fetch_max(live, Ordering::Relaxed); +} + +fn shrink_live(bytes: usize) { + LIVE_BYTES.fetch_sub(bytes, Ordering::Relaxed); +} + struct MeasurementGuard { finished: bool, } diff --git a/crates/j2k-alloc-probe/tests/main.rs b/crates/j2k-alloc-probe/tests/main.rs index 2886631e3..6e2ffeac4 100644 --- a/crates/j2k-alloc-probe/tests/main.rs +++ b/crates/j2k-alloc-probe/tests/main.rs @@ -4,7 +4,7 @@ use std::hint::black_box; use std::panic::{catch_unwind, AssertUnwindSafe}; use std::sync::{mpsc, OnceLock}; -use j2k_alloc_probe::{assert_allocations, measure, Budget}; +use j2k_alloc_probe::{assert_allocations, live_bytes, measure, peak_bytes, reset_peak, Budget}; use j2k_native::{ decode_ht_code_block_scalar_with_workspace, encode, encode_ht_code_block_scalar_with_passes_and_workspace, @@ -32,6 +32,10 @@ fn main() { harness_counts_allocation_and_deallocation, ), ("harness_counts_reallocation", harness_counts_reallocation), + ( + "harness_tracks_live_bytes_and_their_peak", + harness_tracks_live_bytes_and_their_peak, + ), ( "preexisting_deallocation_cannot_credit_budget", preexisting_deallocation_cannot_credit_budget, @@ -131,6 +135,44 @@ fn harness_counts_reallocation() { assert!(stats.requested_bytes >= 4096); } +fn harness_tracks_live_bytes_and_their_peak() { + const BLOCK: usize = 64 * 1024; + let base = reset_peak(); + assert_eq!(peak_bytes(), base, "a new window starts at the live bytes"); + + let first = black_box(Box::<[u8]>::new_uninit_slice(BLOCK)); + let second = black_box(Box::<[u8]>::new_uninit_slice(BLOCK / 2)); + assert_eq!(live_bytes(), base + BLOCK + BLOCK / 2); + drop(first); + assert_eq!( + live_bytes(), + base + BLOCK / 2, + "a free lowers the live bytes" + ); + assert_eq!( + peak_bytes(), + base + BLOCK + BLOCK / 2, + "a free never lowers the peak" + ); + + let mut grown: Vec = Vec::new(); + grown + .try_reserve_exact(BLOCK / 4) + .expect("small allocation must succeed"); + grown + .try_reserve_exact(2 * BLOCK) + .expect("growth allocation must succeed"); + assert_eq!(live_bytes(), base + BLOCK / 2 + grown.capacity()); + assert_eq!(peak_bytes(), base + BLOCK / 2 + grown.capacity()); + // Nothing was pushed: shrinking the empty vector frees its buffer. + grown.shrink_to_fit(); + drop(second); + assert_eq!(live_bytes(), base, "every byte allocated here was freed"); + + assert_eq!(reset_peak(), base); + assert_eq!(peak_bytes(), base, "a reset forgets the earlier peak"); +} + fn preexisting_deallocation_cannot_credit_budget() { let preexisting = black_box(Box::new([0_u8; 8192])); let violation = without_panic_output(|| { diff --git a/crates/j2k-native/Cargo.toml b/crates/j2k-native/Cargo.toml index 89bc3796c..941beb81e 100644 --- a/crates/j2k-native/Cargo.toml +++ b/crates/j2k-native/Cargo.toml @@ -26,6 +26,7 @@ name = "j2k_native" path = "src/lib.rs" [dependencies] +bytemuck = { workspace = true, features = ["extern_crate_alloc"] } fearless_simd = { workspace = true, optional = true } libm = { workspace = true } log = { workspace = true, optional = true } @@ -36,6 +37,8 @@ j2k-types = { path = "../j2k-types", version = "=0.13.2" } [dev-dependencies] criterion = { workspace = true } +# Counts the bytes the band executor allocates, against what it charges. +j2k-alloc-probe = { path = "../j2k-alloc-probe" } # Lets SIMD kernel tests also run on the portable fallback level. fearless_simd = { workspace = true, features = ["force_support_fallback"] } diff --git a/crates/j2k-native/fixtures/multi_segment/all_modes_gray_97.j2k b/crates/j2k-native/fixtures/multi_segment/all_modes_gray_97.j2k new file mode 100644 index 000000000..6e25168ef Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/all_modes_gray_97.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/bypass_gray_53.j2k b/crates/j2k-native/fixtures/multi_segment/bypass_gray_53.j2k new file mode 100644 index 000000000..be895bb94 Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/bypass_gray_53.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/bypass_rgb_53_blocks_per_component.j2k b/crates/j2k-native/fixtures/multi_segment/bypass_rgb_53_blocks_per_component.j2k new file mode 100644 index 000000000..0775b67a4 Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/bypass_rgb_53_blocks_per_component.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/bypass_rgb_97.j2k b/crates/j2k-native/fixtures/multi_segment/bypass_rgb_97.j2k new file mode 100644 index 000000000..9b74d4b17 Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/bypass_rgb_97.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/bypass_rgb_97_64x64.j2k b/crates/j2k-native/fixtures/multi_segment/bypass_rgb_97_64x64.j2k new file mode 100644 index 000000000..1c656db30 Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/bypass_rgb_97_64x64.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/bypass_termall_gray_97.j2k b/crates/j2k-native/fixtures/multi_segment/bypass_termall_gray_97.j2k new file mode 100644 index 000000000..492d1848b Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/bypass_termall_gray_97.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/bypass_termall_rgb_53.j2k b/crates/j2k-native/fixtures/multi_segment/bypass_termall_rgb_53.j2k new file mode 100644 index 000000000..48c359121 Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/bypass_termall_rgb_53.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/reset_rgb_97.j2k b/crates/j2k-native/fixtures/multi_segment/reset_rgb_97.j2k new file mode 100644 index 000000000..4ae893afd Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/reset_rgb_97.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/segmark_gray_53.j2k b/crates/j2k-native/fixtures/multi_segment/segmark_gray_53.j2k new file mode 100644 index 000000000..2f53bc632 Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/segmark_gray_53.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/termall_gray_97.j2k b/crates/j2k-native/fixtures/multi_segment/termall_gray_97.j2k new file mode 100644 index 000000000..3e497f23d Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/termall_gray_97.j2k differ diff --git a/crates/j2k-native/fixtures/multi_segment/termall_rgb_53.j2k b/crates/j2k-native/fixtures/multi_segment/termall_rgb_53.j2k new file mode 100644 index 000000000..ff43b510e Binary files /dev/null and b/crates/j2k-native/fixtures/multi_segment/termall_rgb_53.j2k differ diff --git a/crates/j2k-native/src/color/mod.rs b/crates/j2k-native/src/color/mod.rs index 7e494a759..058257a11 100644 --- a/crates/j2k-native/src/color/mod.rs +++ b/crates/j2k-native/src/color/mod.rs @@ -23,6 +23,7 @@ pub use output_planes::{ DecodedComponentsParts, DecodedNativeComponents, DecodedNativeComponentsParts, NativeComponentPlane, NativeComponentPlaneParts, }; +pub use packing::pack_rgb8; #[cfg(all(feature = "simd", target_arch = "x86_64"))] pub(crate) use packing::store_rgb8_x8; pub(crate) use packing::{ diff --git a/crates/j2k-native/src/color/output_planes.rs b/crates/j2k-native/src/color/output_planes.rs index 78c79bd91..dde051e32 100644 --- a/crates/j2k-native/src/color/output_planes.rs +++ b/crates/j2k-native/src/color/output_planes.rs @@ -208,6 +208,17 @@ impl<'a> ComponentPlane<'a> { pub trait ComponentRowSink { /// Receive the samples of one rectangle of the region. fn write_rows(&mut self, x: u32, y: u32, planes: &[ComponentPlane<'_>]); + + /// The sink's destination for a `width` x `height` region of three 8-bit + /// components, when it is interleaved RGB8 rows of `3 * width` bytes back + /// to back that the sink would fill from such planes by rounding and + /// saturating each sample. A decode may then pack its rows straight into + /// this buffer instead of calling [`Self::write_rows`]. The default, + /// `None`, always uses `write_rows`. + fn rgb8_region(&mut self, width: u32, height: u32) -> Option<&mut [u8]> { + let _ = (width, height); + None + } } /// Borrowed decoded component planes for an image. diff --git a/crates/j2k-native/src/color/packing.rs b/crates/j2k-native/src/color/packing.rs index f30414294..1797a2924 100644 --- a/crates/j2k-native/src/color/packing.rs +++ b/crates/j2k-native/src/color/packing.rs @@ -301,7 +301,7 @@ pub(crate) fn interleave_and_convert( } /// Pack 8-bit RGB planes as interleaved bytes, rounded and saturated. -fn pack_rgb8(planes: [&[f32]; 3], output: &mut [u8]) { +pub fn pack_rgb8(planes: [&[f32]; 3], output: &mut [u8]) { #[cfg(all(feature = "simd", target_arch = "x86_64"))] { use fearless_simd::{dispatch, Level}; diff --git a/crates/j2k-native/src/direct_cpu.rs b/crates/j2k-native/src/direct_cpu.rs index f3b8fd142..83edd3e1d 100644 --- a/crates/j2k-native/src/direct_cpu.rs +++ b/crates/j2k-native/src/direct_cpu.rs @@ -95,12 +95,21 @@ pub struct J2kDirectCpuScratch { ht_task_workspaces: Vec, /// Extra classic workspaces for parallel referenced classic execution, /// one per additional Rayon worker. - classic_task_workspaces: Vec, + classic_task_workspaces: Vec, /// Whether referenced execution may spread work over the Rayon pool. parallel: bool, staged_state: Option, } +/// One extra Rayon worker's state for parallel referenced classic execution: +/// its code-block workspace and the buffer it joins a code block's payload +/// fragments into when a multi-layer codestream splits them. +#[derive(Debug, Default)] +struct ClassicTaskWorkspace { + decode: J2kCodeBlockDecodeWorkspace, + payload: Vec, +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum StagedDirectRoute { Classic, @@ -170,12 +179,21 @@ impl J2kDirectCpuScratch { .iter() .try_fold( self.classic_workspace.allocated_bytes().ok(), - |bytes, workspace| Some(bytes?.checked_add(workspace.allocated_bytes().ok()?)), + |bytes, task| Some(bytes?.checked_add(task.decode.allocated_bytes().ok()?)), ) .flatten() .unwrap_or(usize::MAX) } + /// Length and capacity of each extra classic task's payload buffer. + #[cfg(test)] + fn classic_task_payloads(&self) -> Vec<(usize, usize)> { + self.classic_task_workspaces + .iter() + .map(|task| (task.payload.len(), task.payload.capacity())) + .collect() + } + /// Number of coefficient/IDWT buffer owners retained by this scratch. /// /// Referenced multi-tile execution retains the maximum live tile shape, diff --git a/crates/j2k-native/src/direct_cpu/allocation.rs b/crates/j2k-native/src/direct_cpu/allocation.rs index 6f7c22fa8..656942eb6 100644 --- a/crates/j2k-native/src/direct_cpu/allocation.rs +++ b/crates/j2k-native/src/direct_cpu/allocation.rs @@ -452,8 +452,9 @@ fn parallel_ht_task_workspace_count( } /// Largest code block and most code-block rows of any sub-band a parallel -/// referenced classic execution decodes. One dispatch decodes at most one -/// row per Rayon worker, so it never needs more workspaces than rows. +/// referenced classic execution decodes, and its largest payload split over +/// several input fragments. One dispatch decodes at most one row per Rayon +/// worker, so it never needs more workspaces than rows. #[derive(Debug, Clone, Copy)] struct ClassicTaskShape { dimensions: (u32, u32), @@ -462,6 +463,9 @@ struct ClassicTaskShape { expect(dead_code, reason = "rows bound parallel dispatch only") )] rows: usize, + /// Bytes each task joins one fragmented payload into; zero when every + /// payload is one input slice. + fragmented_payload_bytes: usize, } /// Extra classic workspaces beside the main one: one per additional Rayon @@ -478,36 +482,44 @@ fn classic_task_workspace_count(parallel: bool, classic_tasks: Option, scratch: &J2kDirectCpuScratch, ) -> Result<()> { let target_count = classic_task_workspace_count(scratch.parallel, classic_tasks); - budget.include_capacity::( + budget.include_capacity::( scratch.classic_task_workspaces.capacity().max(target_count), )?; - let target_bytes = classic_tasks.map_or(Ok(0), |shape| { + let target_workspace_bytes = classic_tasks.map_or(Ok(0), |shape| { bitplane::classic_decode_workspace_bytes(shape.dimensions.0, shape.dimensions.1) })?; + let target_payload_bytes = classic_tasks.map_or(0, |shape| shape.fragmented_payload_bytes); for index in 0..scratch.classic_task_workspaces.len().max(target_count) { - let retained = scratch + let (retained_workspace, retained_payload) = scratch .classic_task_workspaces .get(index) - .map_or(Ok(0), crate::J2kCodeBlockDecodeWorkspace::allocated_bytes)?; - let target = if index < target_count { - target_bytes + .map_or(Ok((0, 0)), |task| { + task.decode + .allocated_bytes() + .map(|bytes| (bytes, task.payload.capacity())) + })?; + let (target_workspace, target_payload) = if index < target_count { + (target_workspace_bytes, target_payload_bytes) } else { - 0 + (0, 0) }; - budget.include_bytes(retained.max(target))?; + budget.include_bytes(retained_workspace.max(target_workspace))?; + budget.include_bytes(retained_payload.max(target_payload))?; } Ok(()) } /// Keep exactly the extra classic workspaces parallel referenced classic -/// execution uses, each prepared for the plan's largest code block. +/// execution uses, each prepared for the plan's largest code block and +/// holding a buffer for its largest fragmented payload. fn reserve_classic_task_workspaces( classic_tasks: Option, scratch: &mut J2kDirectCpuScratch, @@ -518,15 +530,23 @@ fn reserve_classic_task_workspaces( while scratch.classic_task_workspaces.len() < count { scratch .classic_task_workspaces - .push(crate::J2kCodeBlockDecodeWorkspace::default()); + .push(super::ClassicTaskWorkspace::default()); } if let Some(ClassicTaskShape { dimensions: (width, height), + fragmented_payload_bytes, .. }) = classic_tasks { - for workspace in &mut scratch.classic_task_workspaces { - workspace.prepare(width, height)?; + for task in &mut scratch.classic_task_workspaces { + task.decode.prepare(width, height)?; + task.payload.clear(); + if task.payload.capacity() < fragmented_payload_bytes { + // Release before growing so the old and new buffers never + // coexist beyond the charged maximum. + task.payload = Vec::new(); + } + try_reserve_decode_elements(&mut task.payload, fragmented_payload_bytes)?; } } Ok(()) diff --git a/crates/j2k-native/src/direct_cpu/allocation/referenced.rs b/crates/j2k-native/src/direct_cpu/allocation/referenced.rs index 491a767ca..a7603445e 100644 --- a/crates/j2k-native/src/direct_cpu/allocation/referenced.rs +++ b/crates/j2k-native/src/direct_cpu/allocation/referenced.rs @@ -15,25 +15,94 @@ use super::{ use budget::validate_referenced_aggregate_plan; use storage::{normalize_referenced_scratch, reserve_referenced_scratch}; use view::{ - classic_plan_has_fragmented_payloads, max_referenced_classic_payload_bytes, + max_referenced_classic_fragmented_payload_bytes, max_referenced_classic_payload_bytes, max_referenced_classic_rows, max_referenced_dimensions, max_referenced_payload_bytes, ReferencedPlanView, }; +/// Accounting inputs shared by an estimator and the matching reservation. +#[derive(Clone, Copy)] +struct ReferencedRequirements<'a> { + plan: ReferencedPlanView<'a>, + parallel: bool, + retained_plan_bytes: usize, + compressed_payload_bytes: usize, + retained_classic_workspace_dimensions: Option<(u32, u32)>, + retained_ht_workspace_dimensions: Option<(u32, u32)>, + classic_tasks: Option, +} + +impl<'a> ReferencedRequirements<'a> { + fn htj2k(plan: &'a J2kReferencedHtj2kPlan, parallel: bool) -> Result { + let view = ReferencedPlanView::Htj2k(plan); + Ok(Self { + plan: view, + parallel, + retained_plan_bytes: plan.retained_allocation_bytes()?, + compressed_payload_bytes: max_referenced_payload_bytes(plan)?, + retained_classic_workspace_dimensions: max_referenced_dimensions(view, false), + retained_ht_workspace_dimensions: max_referenced_dimensions(view, true), + classic_tasks: None, + }) + } + + fn classic(plan: &'a J2kReferencedClassicPlan, parallel: bool) -> Result { + let dimensions = max_referenced_classic_dimensions(plan); + Ok(Self { + plan: ReferencedPlanView::Classic(plan), + parallel, + retained_plan_bytes: plan.retained_allocation_bytes()?, + compressed_payload_bytes: max_referenced_classic_payload_bytes(plan), + retained_classic_workspace_dimensions: dimensions, + retained_ht_workspace_dimensions: None, + classic_tasks: classic_task_shape(plan), + }) + } + + fn staged_htj2k(plan: &'a J2kReferencedHtj2kPlan) -> Result { + Ok(Self { + plan: ReferencedPlanView::Htj2k(plan), + parallel: false, + retained_plan_bytes: plan.retained_allocation_bytes()?, + compressed_payload_bytes: 0, + retained_classic_workspace_dimensions: None, + retained_ht_workspace_dimensions: None, + classic_tasks: None, + }) + } + + fn staged_classic(plan: &'a J2kReferencedClassicPlan) -> Result { + Ok(Self { + plan: ReferencedPlanView::Classic(plan), + parallel: false, + retained_plan_bytes: plan.retained_allocation_bytes()?, + compressed_payload_bytes: 0, + retained_classic_workspace_dimensions: None, + retained_ht_workspace_dimensions: None, + classic_tasks: None, + }) + } + + fn validate(self, scratch: &J2kDirectCpuScratch) -> Result { + validate_referenced_aggregate_plan( + self.plan, + self.parallel, + self.retained_plan_bytes, + self.compressed_payload_bytes, + self.retained_classic_workspace_dimensions, + self.retained_ht_workspace_dimensions, + self.classic_tasks, + scratch, + ) + } +} + pub(in super::super) fn prepare_referenced_direct_scratch( plan: &J2kReferencedHtj2kPlan, scratch: &mut J2kDirectCpuScratch, ) -> Result { - let view = ReferencedPlanView::Htj2k(plan); - let parallel = scratch.parallel; prepare_referenced_component_scratch( - view, - parallel, - plan.retained_allocation_bytes()?, - max_referenced_payload_bytes(plan)?, - max_referenced_dimensions(view, false), - max_referenced_dimensions(view, true), - None, + ReferencedRequirements::htj2k(plan, scratch.parallel)?, scratch, ) } @@ -44,20 +113,11 @@ pub(in super::super) fn planned_referenced_htj2k_execution_bytes( plan: &J2kReferencedHtj2kPlan, parallel: bool, ) -> Result { - let view = ReferencedPlanView::Htj2k(plan); let mut scratch = J2kDirectCpuScratch::new(); scratch.parallel = parallel; - validate_referenced_aggregate_plan( - view, - parallel, - plan.retained_allocation_bytes()?, - max_referenced_payload_bytes(plan)?, - max_referenced_dimensions(view, false), - max_referenced_dimensions(view, true), - None, - &scratch, - ) - .map(DirectWorkspaceBudget::peak_bytes) + ReferencedRequirements::htj2k(plan, parallel)? + .validate(&scratch) + .map(DirectWorkspaceBudget::peak_bytes) } /// Planned peak bytes of one referenced classic execution into empty @@ -68,30 +128,17 @@ pub(in super::super) fn planned_referenced_classic_execution_bytes( ) -> Result { let mut scratch = J2kDirectCpuScratch::new(); scratch.parallel = parallel; - let classic_dimensions = max_referenced_classic_dimensions(plan); - validate_referenced_aggregate_plan( - ReferencedPlanView::Classic(plan), - parallel, - plan.retained_allocation_bytes()?, - max_referenced_classic_payload_bytes(plan), - classic_dimensions, - None, - classic_task_shape(plan), - &scratch, - ) - .map(DirectWorkspaceBudget::peak_bytes) + ReferencedRequirements::classic(plan, parallel)? + .validate(&scratch) + .map(DirectWorkspaceBudget::peak_bytes) } -/// Shape of the extra classic workspaces parallel execution of `plan` uses: -/// none for plans whose payloads are split over fragments, which parallel -/// dispatch cannot borrow as one slice. +/// Shape of the extra classic workspaces parallel execution of `plan` uses. fn classic_task_shape(plan: &J2kReferencedClassicPlan) -> Option { - if classic_plan_has_fragmented_payloads(plan) { - return None; - } Some(ClassicTaskShape { dimensions: max_referenced_classic_dimensions(plan)?, rows: max_referenced_classic_rows(plan), + fragmented_payload_bytes: max_referenced_classic_fragmented_payload_bytes(plan), }) } @@ -99,18 +146,8 @@ pub(in super::super) fn prepare_referenced_classic_scratch( plan: &J2kReferencedClassicPlan, scratch: &mut J2kDirectCpuScratch, ) -> Result { - let view = ReferencedPlanView::Classic(plan); - let parallel = scratch.parallel; - let classic_dimensions = max_referenced_classic_dimensions(plan); - let classic_tasks = classic_task_shape(plan); prepare_referenced_component_scratch( - view, - parallel, - plan.retained_allocation_bytes()?, - max_referenced_classic_payload_bytes(plan), - classic_dimensions, - None, - classic_tasks, + ReferencedRequirements::classic(plan, scratch.parallel)?, scratch, ) } @@ -119,66 +156,29 @@ pub(in super::super) fn prepare_referenced_htj2k_staged_scratch( plan: &J2kReferencedHtj2kPlan, scratch: &mut J2kDirectCpuScratch, ) -> Result { - prepare_referenced_component_scratch( - ReferencedPlanView::Htj2k(plan), - false, - plan.retained_allocation_bytes()?, - 0, - None, - None, - None, - scratch, - ) + prepare_referenced_component_scratch(ReferencedRequirements::staged_htj2k(plan)?, scratch) } pub(in super::super) fn prepare_referenced_classic_staged_scratch( plan: &J2kReferencedClassicPlan, scratch: &mut J2kDirectCpuScratch, ) -> Result { - prepare_referenced_component_scratch( - ReferencedPlanView::Classic(plan), - false, - plan.retained_allocation_bytes()?, - 0, - None, - None, - None, - scratch, - ) + prepare_referenced_component_scratch(ReferencedRequirements::staged_classic(plan)?, scratch) } -#[expect( - clippy::too_many_arguments, - reason = "normalization, accounting, and reservation share every retained owner shape explicitly" -)] fn prepare_referenced_component_scratch( - plan: ReferencedPlanView<'_>, - parallel: bool, - retained_plan_bytes: usize, - compressed_payload_bytes: usize, - retained_classic_workspace_dimensions: Option<(u32, u32)>, - retained_ht_workspace_dimensions: Option<(u32, u32)>, - classic_tasks: Option, + requirements: ReferencedRequirements<'_>, scratch: &mut J2kDirectCpuScratch, ) -> Result { normalize_referenced_scratch( - plan, - compressed_payload_bytes, - retained_classic_workspace_dimensions.is_some(), - retained_ht_workspace_dimensions.is_some(), - classic_tasks.is_some(), + requirements.plan, + requirements.compressed_payload_bytes, + requirements.retained_classic_workspace_dimensions.is_some(), + requirements.retained_ht_workspace_dimensions.is_some(), + requirements.classic_tasks.is_some(), scratch, )?; - if let Err(error) = validate_referenced_aggregate_plan( - plan, - parallel, - retained_plan_bytes, - compressed_payload_bytes, - retained_classic_workspace_dimensions, - retained_ht_workspace_dimensions, - classic_tasks, - scratch, - ) { + if let Err(error) = requirements.validate(scratch) { if !matches!( error, DecodeError::Validation(ValidationError::ImageTooLarge) @@ -186,39 +186,21 @@ fn prepare_referenced_component_scratch( return Err(error); } scratch.clear(); - validate_referenced_aggregate_plan( - plan, - parallel, - retained_plan_bytes, - compressed_payload_bytes, - retained_classic_workspace_dimensions, - retained_ht_workspace_dimensions, - classic_tasks, - scratch, - )?; + requirements.validate(scratch)?; } if let Err(error) = reserve_referenced_scratch( - plan, - compressed_payload_bytes, - retained_classic_workspace_dimensions, - retained_ht_workspace_dimensions, - classic_tasks, + requirements.plan, + requirements.compressed_payload_bytes, + requirements.retained_classic_workspace_dimensions, + requirements.retained_ht_workspace_dimensions, + requirements.classic_tasks, scratch, ) { scratch.clear(); return Err(error); } - match validate_referenced_aggregate_plan( - plan, - parallel, - retained_plan_bytes, - compressed_payload_bytes, - retained_classic_workspace_dimensions, - retained_ht_workspace_dimensions, - classic_tasks, - scratch, - ) { + match requirements.validate(scratch) { Ok(workspace_budget) => Ok(workspace_budget), Err(error) => { scratch.clear(); diff --git a/crates/j2k-native/src/direct_cpu/allocation/referenced/view.rs b/crates/j2k-native/src/direct_cpu/allocation/referenced/view.rs index 73e1425a2..c37a08ca9 100644 --- a/crates/j2k-native/src/direct_cpu/allocation/referenced/view.rs +++ b/crates/j2k-native/src/direct_cpu/allocation/referenced/view.rs @@ -221,10 +221,7 @@ fn referenced_parallel_classic_dispatch_bytes( parallel: bool, ) -> Result { let worker_count = rayon::current_num_threads(); - let ReferencedPlanView::Classic(classic) = plan else { - return Ok(0); - }; - if !parallel || worker_count <= 1 || classic_plan_has_fragmented_payloads(classic) { + if !matches!(plan, ReferencedPlanView::Classic(_)) || !parallel || worker_count <= 1 { return Ok(0); } let mut maximum_jobs = 0usize; @@ -240,10 +237,11 @@ fn referenced_parallel_classic_dispatch_bytes( if maximum_jobs == 0 { return Ok(0); } - super::super::super::referenced::rows::row_dispatch_transient_bytes::< - crate::J2kOwnedCodeBlockBatchJob, - crate::J2kCodeBlockDecodeWorkspace, - >(maximum_jobs, maximum_jobs, worker_count) + super::super::super::referenced_classic::parallel_classic_decode_transient_bytes( + maximum_jobs, + maximum_jobs, + worker_count, + ) } #[cfg(not(feature = "parallel"))] @@ -333,13 +331,18 @@ pub(in super::super::super) fn max_referenced_classic_dimensions( max_referenced_dimensions(ReferencedPlanView::Classic(plan), false) } -/// Whether some classic payload is split over several input fragments (a -/// multi-layer codestream). Parallel classic dispatch borrows each payload -/// as one input slice, so such plans decode their blocks serially. -pub(super) fn classic_plan_has_fragmented_payloads(plan: &J2kReferencedClassicPlan) -> bool { +/// Largest classic payload split over several input fragments (a +/// multi-layer codestream), which a parallel task joins into its own buffer; +/// zero when every payload is one input slice. +pub(super) fn max_referenced_classic_fragmented_payload_bytes( + plan: &J2kReferencedClassicPlan, +) -> usize { plan.payloads() .iter() - .any(|payload| payload.range_count > 1) + .filter(|payload| payload.range_count > 1) + .map(|payload| payload.combined_length) + .max() + .unwrap_or(0) } /// Most runs of code-block rows (consecutive jobs sharing `output_y`) in any diff --git a/crates/j2k-native/src/direct_cpu/referenced/component.rs b/crates/j2k-native/src/direct_cpu/referenced/component.rs index e33293a51..0e3ba2953 100644 --- a/crates/j2k-native/src/direct_cpu/referenced/component.rs +++ b/crates/j2k-native/src/direct_cpu/referenced/component.rs @@ -206,7 +206,7 @@ fn execute_ht_sub_band_referenced( } #[cfg(feature = "parallel")] -type RowJob<'p, 'i> = super::rows::RowJob<'p, 'i, crate::HtOwnedCodeBlockBatchJob>; +type RowJob<'p, 'i> = super::rows::RowJob<'p, crate::HtOwnedCodeBlockBatchJob, &'i [u8]>; /// Bytes owned together while a referenced HT sub-band is dispatched in /// parallel. Planning and execution share this helper so the private dispatch @@ -219,7 +219,8 @@ pub(in crate::direct_cpu) fn parallel_ht_decode_transient_bytes( ) -> Result { super::rows::row_dispatch_transient_bytes::< crate::HtOwnedCodeBlockBatchJob, - HtCodeBlockDecodeWorkspace, + &[u8], + &mut HtCodeBlockDecodeWorkspace, >(entry_capacity, row_capacity, workspace_capacity) } @@ -244,7 +245,7 @@ fn decode_ht_sub_band_rows_parallel( }; entries.sort_unstable_by_key(|entry| (entry.job.output_y, entry.job.output_x)); let entry_capacity = entries.capacity(); - let Some(rows) = super::rows::carve_rows::<_, HtCodeBlockDecodeWorkspace>( + let Some(rows) = super::rows::carve_rows::<_, _, &mut HtCodeBlockDecodeWorkspace>( &entries, output, sub_band_width, @@ -260,7 +261,7 @@ fn decode_ht_sub_band_rows_parallel( rows, entry_capacity, &mut *ht.main, - ht.tasks, + ht.tasks.iter_mut(), ht.budget, |entry, output, workspace| { let job = entry.job; diff --git a/crates/j2k-native/src/direct_cpu/referenced/rows.rs b/crates/j2k-native/src/direct_cpu/referenced/rows.rs index 7910af285..7c0087302 100644 --- a/crates/j2k-native/src/direct_cpu/referenced/rows.rs +++ b/crates/j2k-native/src/direct_cpu/referenced/rows.rs @@ -2,7 +2,7 @@ //! Row-parallel code-block dispatch shared by referenced HT and classic //! execution: borrowed jobs are grouped into disjoint rows of one sub-band -//! and decoded on the Rayon pool, each row with a pooled workspace. +//! and decoded on the Rayon pool, each row with a pooled task workspace. use crate::error::{DecodingError, Result, ValidationError}; use crate::{HtOwnedCodeBlockBatchJob, J2kOwnedCodeBlockBatchJob}; @@ -47,37 +47,39 @@ impl CodeBlockRowGeometry for J2kOwnedCodeBlockBatchJob { } } -/// One code block of a parallel sub-band decode with its borrowed payload and -/// output range in the sub-band. -pub(in crate::direct_cpu) struct RowJob<'p, 'i, J> { +/// One code block of a parallel sub-band decode with its payload `D` and +/// output range in the sub-band. HT borrows each payload as one input slice; +/// classic carries the payload's input fragments for its task to join. +pub(in crate::direct_cpu) struct RowJob<'p, J, D> { pub(in crate::direct_cpu) job: &'p J, - pub(in crate::direct_cpu) data: &'i [u8], + pub(in crate::direct_cpu) data: D, pub(in crate::direct_cpu) start: usize, pub(in crate::direct_cpu) end: usize, } /// A row of code blocks: its output rows, its jobs, and its first sample. -pub(in crate::direct_cpu) type CodeBlockRow<'o, 'j, 'p, 'i, J> = - (&'o mut [f32], &'j [RowJob<'p, 'i, J>], usize); +pub(in crate::direct_cpu) type CodeBlockRow<'o, 'j, 'p, J, D> = + (&'o mut [f32], &'j [RowJob<'p, J, D>], usize); -/// Bytes owned together while one sub-band is dispatched in parallel. -/// Planning and execution share this helper so the private dispatch types -/// cannot drift from their accounting. -pub(in crate::direct_cpu) fn row_dispatch_transient_bytes( +/// Bytes owned together while one sub-band is dispatched in parallel, with +/// `W` the pooled per-task workspace handle. Planning and execution share +/// this helper so the private dispatch types cannot drift from their +/// accounting. +pub(in crate::direct_cpu) fn row_dispatch_transient_bytes( entry_capacity: usize, row_capacity: usize, workspace_capacity: usize, ) -> Result { entry_capacity - .checked_mul(core::mem::size_of::>()) + .checked_mul(core::mem::size_of::>()) .and_then(|bytes| { row_capacity - .checked_mul(core::mem::size_of::>()) + .checked_mul(core::mem::size_of::>()) .and_then(|rows| bytes.checked_add(rows)) }) .and_then(|bytes| { workspace_capacity - .checked_mul(core::mem::size_of::<&mut W>()) + .checked_mul(core::mem::size_of::()) .and_then(|workspaces| bytes.checked_add(workspaces)) }) .ok_or_else(|| ValidationError::ImageTooLarge.into()) @@ -85,16 +87,16 @@ pub(in crate::direct_cpu) fn row_dispatch_transient_bytes( /// Split `output` into one slice per row of code blocks, or `None` when the /// jobs, sorted by position, do not form disjoint rows of equal-height blocks. -pub(in crate::direct_cpu) fn carve_rows<'o, 'j, 'p, 'i, J: CodeBlockRowGeometry, W>( - entries: &'j [RowJob<'p, 'i, J>], +pub(in crate::direct_cpu) fn carve_rows<'o, 'j, 'p, J: CodeBlockRowGeometry, D, W>( + entries: &'j [RowJob<'p, J, D>], output: &'o mut [f32], sub_band_width: usize, entry_capacity: usize, budget: DirectWorkspaceBudget, -) -> Result>>> { +) -> Result>>> { let mut rows = alloc::vec::Vec::new(); crate::try_reserve_decode_elements(&mut rows, entries.len())?; - budget.validate_workspace(row_dispatch_transient_bytes::( + budget.validate_workspace(row_dispatch_transient_bytes::( entry_capacity, rows.capacity(), 0, @@ -131,20 +133,21 @@ pub(in crate::direct_cpu) fn carve_rows<'o, 'j, 'p, 'i, J: CodeBlockRowGeometry, Ok(Some(rows)) } -/// Decode rows of code blocks on the Rayon pool, each with a pooled +/// Decode rows of code blocks on the Rayon pool, each with a pooled task /// workspace taken from `main` and `tasks`. -pub(in crate::direct_cpu) fn decode_rows( - rows: alloc::vec::Vec>, +pub(in crate::direct_cpu) fn decode_rows( + rows: alloc::vec::Vec>, entry_capacity: usize, - main: &mut W, - tasks: &mut [W], + main: W, + tasks: impl ExactSizeIterator, budget: DirectWorkspaceBudget, decode: F, ) -> Result<()> where J: Sync, + D: Sync, W: Send, - F: Fn(&RowJob<'_, '_, J>, &mut [f32], &mut W) -> Result<()> + Sync, + F: Fn(&RowJob<'_, J, D>, &mut [f32], &mut W) -> Result<()> + Sync, { use rayon::prelude::*; use std::sync::Mutex; @@ -157,8 +160,8 @@ where let mut workspaces = alloc::vec::Vec::new(); crate::try_reserve_decode_elements(&mut workspaces, workspace_count)?; workspaces.push(main); - workspaces.extend(tasks.iter_mut()); - budget.validate_workspace(row_dispatch_transient_bytes::( + workspaces.extend(tasks); + budget.validate_workspace(row_dispatch_transient_bytes::( entry_capacity, row_capacity, workspaces.capacity(), @@ -173,10 +176,10 @@ where .lock() .map_err(|_| DecodingError::CodeBlockDecodeFailure)? .pop(); - let workspace = pooled.ok_or(DecodingError::CodeBlockDecodeFailure)?; + let mut workspace = pooled.ok_or(DecodingError::CodeBlockDecodeFailure)?; let result = jobs.iter().try_for_each(|entry| { let output = &mut row[entry.start - row_start..entry.end - row_start]; - decode(entry, output, workspace) + decode(entry, output, &mut workspace) }); pool.lock() .map_err(|_| DecodingError::CodeBlockDecodeFailure)? diff --git a/crates/j2k-native/src/direct_cpu/referenced_classic.rs b/crates/j2k-native/src/direct_cpu/referenced_classic.rs index d0d2928da..8ad63306e 100644 --- a/crates/j2k-native/src/direct_cpu/referenced_classic.rs +++ b/crates/j2k-native/src/direct_cpu/referenced_classic.rs @@ -14,9 +14,9 @@ use super::allocation::prepare_referenced_classic_scratch; use super::referenced::{decoded_color_components, decoded_plane, payload_slice}; use super::{ apply_inverse_mct_region, checked_sub_band_job_output_range, execute_idwt_step, - prepare_sub_band_output, rounds_at_store, store_component, DirectComponentBandScratch, - DirectComponentPlane, DirectWorkspaceBudget, J2kDirectCpuScratch, J2kDirectDecodedComponents, - SubBandJobOutputRange, + prepare_sub_band_output, rounds_at_store, store_component, ClassicTaskWorkspace, + DirectComponentBandScratch, DirectComponentPlane, DirectWorkspaceBudget, J2kDirectCpuScratch, + J2kDirectDecodedComponents, SubBandJobOutputRange, }; /// Execute retained per-tile Gray/RGB/RGBA classic JPEG 2000 geometry @@ -281,7 +281,7 @@ struct ClassicWorkspaces<'a> { not(feature = "parallel"), expect(dead_code, reason = "extra workspaces serve parallel execution only") )] - tasks: &'a mut [J2kCodeBlockDecodeWorkspace], + tasks: &'a mut [ClassicTaskWorkspace], parallel: bool, #[cfg_attr( not(feature = "parallel"), @@ -310,6 +310,7 @@ fn execute_classic_sub_band_with_workspaces( output, sub_band_width, payloads, + compressed_payload, workspaces, )? { @@ -395,23 +396,51 @@ fn decode_classic_sub_band_jobs( #[cfg(feature = "parallel")] type ClassicRowJob<'p, 'i> = - super::referenced::rows::RowJob<'p, 'i, crate::J2kOwnedCodeBlockBatchJob>; + super::referenced::rows::RowJob<'p, crate::J2kOwnedCodeBlockBatchJob, ClassicFragments<'i>>; + +/// One row task's classic state: its code-block workspace and the buffer it +/// joins fragmented payloads into. +#[cfg(feature = "parallel")] +struct ClassicRowTask<'a> { + workspace: &'a mut J2kCodeBlockDecodeWorkspace, + payload: &'a mut alloc::vec::Vec, +} + +/// Bytes owned together while a referenced classic sub-band is dispatched in +/// parallel. Planning and execution share this helper so the private +/// dispatch types cannot drift from their accounting. +#[cfg(feature = "parallel")] +pub(in crate::direct_cpu) fn parallel_classic_decode_transient_bytes( + entry_capacity: usize, + row_capacity: usize, + workspace_capacity: usize, +) -> Result { + super::referenced::rows::row_dispatch_transient_bytes::< + crate::J2kOwnedCodeBlockBatchJob, + ClassicFragments<'_>, + ClassicRowTask<'_>, + >(entry_capacity, row_capacity, workspace_capacity) +} /// Decode a classic sub-band's rows of code blocks in parallel, each writing -/// its own rows of `output`. Returns `false` without consuming payloads when a -/// block's bytes are not one contiguous input slice or the blocks do not tile -/// the sub-band in rows; the serial path then decodes the sub-band. Every -/// block decodes exactly as on the serial path, so the samples are identical. +/// its own rows of `output`. Each row task joins fragmented (multi-layer) +/// payloads in its own buffer: the main task in `compressed_payload`, every +/// extra one in its task workspace. Returns `false` without consuming +/// payloads when the blocks do not tile the sub-band in rows; the serial path +/// then decodes the sub-band. Every block decodes exactly as on the serial +/// path, so the samples are identical. #[cfg(feature = "parallel")] fn decode_classic_sub_band_rows_parallel( plan: &J2kOwnedSubBandPlan, output: &mut [f32], sub_band_width: usize, payloads: &mut ClassicPayloadCursor<'_>, + compressed_payload: &mut alloc::vec::Vec, workspaces: &mut ClassicWorkspaces<'_>, ) -> Result { use super::referenced::rows; + let encoded_input = payloads.encoded_input; let checkpoint = payloads.position(); let Some(mut entries) = collect_classic_row_jobs( plan, @@ -426,7 +455,7 @@ fn decode_classic_sub_band_rows_parallel( }; entries.sort_unstable_by_key(|entry| (entry.job.output_y, entry.job.output_x)); let entry_capacity = entries.capacity(); - let Some(rows) = rows::carve_rows::<_, J2kCodeBlockDecodeWorkspace>( + let Some(rows) = rows::carve_rows::<_, _, ClassicRowTask<'_>>( &entries, output, sub_band_width, @@ -438,16 +467,25 @@ fn decode_classic_sub_band_rows_parallel( return Ok(false); }; let irreversible_midpoint = plan.irreversible_midpoint; + let main = ClassicRowTask { + workspace: &mut *workspaces.main, + payload: compressed_payload, + }; + let tasks = workspaces.tasks.iter_mut().map(|task| ClassicRowTask { + workspace: &mut task.decode, + payload: &mut task.payload, + }); rows::decode_rows( rows, entry_capacity, - &mut *workspaces.main, - workspaces.tasks, + main, + tasks, workspaces.budget, - |entry, output, workspace| { + |entry, output, task| { + let ClassicRowTask { workspace, payload } = task; let job = entry.job; let code_block = J2kCodeBlockDecodeJob { - data: entry.data, + data: entry.data.join(encoded_input, payload)?, segments: &job.segments, width: job.width, height: job.height, @@ -471,8 +509,8 @@ fn decode_classic_sub_band_rows_parallel( Ok(true) } -/// Validate every job's output range and borrow its payload, or `None` when -/// a payload is split over several input fragments. +/// Validate every job's output range and borrow its payload fragments, or +/// `None` when a job's output stride is not the sub-band width. #[cfg(feature = "parallel")] fn collect_classic_row_jobs<'p, 'i>( plan: &'p J2kOwnedSubBandPlan, @@ -483,10 +521,11 @@ fn collect_classic_row_jobs<'p, 'i>( ) -> Result>>> { let mut entries = alloc::vec::Vec::new(); try_reserve_decode_elements(&mut entries, plan.jobs.len())?; - budget.validate_workspace(super::referenced::rows::row_dispatch_transient_bytes::< - crate::J2kOwnedCodeBlockBatchJob, - J2kCodeBlockDecodeWorkspace, - >(entries.capacity(), 0, 0)?)?; + budget.validate_workspace(parallel_classic_decode_transient_bytes( + entries.capacity(), + 0, + 0, + )?)?; for job in &plan.jobs { if !job.data.is_empty() { bail!(DecodingError::CodeBlockDecodeFailure); @@ -505,12 +544,9 @@ fn collect_classic_row_jobs<'p, 'i>( if job.output_stride != sub_band_width { return Ok(None); } - let Some(data) = payloads.next_contiguous()? else { - return Ok(None); - }; entries.push(ClassicRowJob { job, - data, + data: payloads.next_fragments()?, start: range.start, end: range.end, }); @@ -552,6 +588,45 @@ pub(super) fn validate_payload_ranges( Ok(()) } +/// One code block's compressed payload as ordered input fragments; a +/// multi-layer codestream splits it over one fragment per contributing +/// layer. +#[derive(Clone, Copy)] +struct ClassicFragments<'plan> { + ranges: &'plan [crate::J2kCodestreamRange], + combined_length: usize, +} + +impl<'plan> ClassicFragments<'plan> { + /// The payload bytes: the input slice itself for one fragment, else the + /// fragments joined in `combined`. + fn join<'scratch>( + self, + encoded_input: &'plan [u8], + combined: &'scratch mut alloc::vec::Vec, + ) -> Result<&'scratch [u8]> + where + 'plan: 'scratch, + { + if let [range] = self.ranges { + if range.length != self.combined_length { + bail!(DecodingError::CodeBlockDecodeFailure); + } + return payload_slice(encoded_input, *range); + } + + combined.clear(); + try_reserve_decode_elements(combined, self.combined_length)?; + for range in self.ranges { + combined.extend_from_slice(payload_slice(encoded_input, *range)?); + } + if combined.len() != self.combined_length { + bail!(DecodingError::CodeBlockDecodeFailure); + } + Ok(combined) + } +} + pub(super) struct ClassicPayloadCursor<'plan> { encoded_input: &'plan [u8], payloads: &'plan [crate::J2kClassicCodeBlockPayload], @@ -574,9 +649,17 @@ impl<'plan> ClassicPayloadCursor<'plan> { } fn next_data<'scratch>( - &'scratch mut self, + &mut self, combined: &'scratch mut alloc::vec::Vec, - ) -> Result<&'scratch [u8]> { + ) -> Result<&'scratch [u8]> + where + 'plan: 'scratch, + { + self.next_fragments()?.join(self.encoded_input, combined) + } + + /// The next payload's input fragments. + fn next_fragments(&mut self) -> Result> { let payload = self .payloads .get(self.next) @@ -588,27 +671,14 @@ impl<'plan> ClassicPayloadCursor<'plan> { let end_range = payload .end_range() .ok_or(DecodingError::CodeBlockDecodeFailure)?; - let fragments = self + let ranges = self .ranges .get(payload.first_range..end_range) .ok_or(DecodingError::CodeBlockDecodeFailure)?; - - if let [range] = fragments { - if range.length != payload.combined_length { - bail!(DecodingError::CodeBlockDecodeFailure); - } - return payload_slice(self.encoded_input, *range); - } - - combined.clear(); - try_reserve_decode_elements(combined, payload.combined_length)?; - for range in fragments { - combined.extend_from_slice(payload_slice(self.encoded_input, *range)?); - } - if combined.len() != payload.combined_length { - bail!(DecodingError::CodeBlockDecodeFailure); - } - Ok(combined) + Ok(ClassicFragments { + ranges, + combined_length: payload.combined_length, + }) } /// Position for [`Self::rewind`] when a parallel attempt falls back. @@ -622,36 +692,6 @@ impl<'plan> ClassicPayloadCursor<'plan> { self.next = position; } - /// The next payload borrowed from the input when it is one contiguous - /// fragment (or empty), else `None` without consuming it. - #[cfg(feature = "parallel")] - fn next_contiguous(&mut self) -> Result> { - let payload = self - .payloads - .get(self.next) - .ok_or(DecodingError::CodeBlockDecodeFailure)?; - let end_range = payload - .end_range() - .ok_or(DecodingError::CodeBlockDecodeFailure)?; - let fragments = self - .ranges - .get(payload.first_range..end_range) - .ok_or(DecodingError::CodeBlockDecodeFailure)?; - let data = match fragments { - [] if payload.combined_length == 0 => &[][..], - [range] if range.length == payload.combined_length => { - payload_slice(self.encoded_input, *range)? - } - [_] => bail!(DecodingError::CodeBlockDecodeFailure), - _ => return Ok(None), - }; - self.next = self - .next - .checked_add(1) - .ok_or(DecodingError::CodeBlockDecodeFailure)?; - Ok(Some(data)) - } - pub(super) fn ensure_exhausted(&self) -> Result<()> { if self.next == self.payloads.len() { Ok(()) diff --git a/crates/j2k-native/src/direct_cpu/referenced_classic/tests.rs b/crates/j2k-native/src/direct_cpu/referenced_classic/tests.rs index 823825f70..e015c0cde 100644 --- a/crates/j2k-native/src/direct_cpu/referenced_classic/tests.rs +++ b/crates/j2k-native/src/direct_cpu/referenced_classic/tests.rs @@ -7,6 +7,7 @@ use crate::{ J2kDirectCpuScratch, }; +#[cfg(feature = "parallel")] fn rgb_fixture(width: u32, height: u32, reversible: bool) -> Vec { rgb_fixture_with_layers(width, height, reversible, 1) } @@ -29,14 +30,41 @@ fn rgb_fixture_with_layers(width: u32, height: u32, reversible: bool, num_layers encode(&samples, width, height, 3, 8, false, &options).expect("encode classic RGB fixture") } -/// Every reconstructed plane as raw bits, plus the retained classic -/// workspace bytes after the execution. -fn execute_whole_plan(bytes: &[u8], parallel: bool) -> (Vec>, usize) { +/// Reconstructed planes as raw bits, the retained classic workspace bytes, +/// and each extra task's payload buffer (length, capacity) after one +/// execution. +struct Execution { + planes: Vec>, + workspace_bytes: usize, + task_payloads: Vec<(usize, usize)>, +} + +type Region = (u32, u32, u32, u32); + +fn whole_image(bytes: &[u8]) -> Region { + let image = Image::new(bytes, &DecodeSettings::default()).expect("classic image"); + (0, 0, image.width(), image.height()) +} + +/// The largest payload split over several input fragments, if any. +#[cfg(feature = "parallel")] +fn largest_fragmented_payload(bytes: &[u8], region: Region) -> Option { + let image = Image::new(bytes, &DecodeSettings::default()).expect("classic image"); + let plan = image + .build_referenced_classic_plan_region_with_context(&mut DecoderContext::default(), region) + .expect("classic region plan"); + plan.payloads() + .iter() + .filter(|payload| payload.range_count > 1) + .map(|payload| payload.combined_length) + .max() +} + +fn execute_region(bytes: &[u8], region: Region, parallel: bool) -> Execution { let image = Image::new(bytes, &DecodeSettings::default()).expect("classic image"); - let region = (0, 0, image.width(), image.height()); let plan = image .build_referenced_classic_plan_region_with_context(&mut DecoderContext::default(), region) - .expect("classic whole-image plan"); + .expect("classic region plan"); let mut scratch = J2kDirectCpuScratch::new(); scratch.set_parallel_decode(parallel); let decoded = execute_referenced_classic_plan(&plan, bytes, false, &mut scratch) @@ -52,7 +80,11 @@ fn execute_whole_plan(bytes: &[u8], parallel: bool) -> (Vec>, usize) { .collect() }) .collect(); - (planes, scratch.retained_classic_workspace_bytes()) + Execution { + planes, + workspace_bytes: scratch.retained_classic_workspace_bytes(), + task_payloads: scratch.classic_task_payloads(), + } } #[cfg(feature = "parallel")] @@ -65,16 +97,24 @@ fn parallel_referenced_classic_execution_matches_serial_bits() { for reversible in [true, false] { // Several rows of 64 x 64 code blocks per high-resolution sub-band. let bytes = rgb_fixture(300, 260, reversible); - let (serial, serial_workspace_bytes) = execute_whole_plan(&bytes, false); - let (parallel, parallel_workspace_bytes) = - pool.install(|| execute_whole_plan(&bytes, true)); - assert_eq!(serial, parallel, "reversible={reversible}"); + let region = whole_image(&bytes); + assert_eq!(largest_fragmented_payload(&bytes, region), None); + let serial = execute_region(&bytes, region, false); + let parallel = pool.install(|| execute_region(&bytes, region, true)); + assert_eq!(serial.planes, parallel.planes, "reversible={reversible}"); // The tallest sub-band (130 rows) holds three rows of 64-high // blocks, so of the four workers only two extra ones can take a row // and keep their own prepared workspace. assert_eq!( - parallel_workspace_bytes, - serial_workspace_bytes * 3, + parallel.workspace_bytes, + serial.workspace_bytes * 3, + "reversible={reversible}" + ); + // Single-fragment payloads are borrowed from the input, so the extra + // tasks keep no payload buffer. + assert_eq!( + parallel.task_payloads, + [(0, 0); 2], "reversible={reversible}" ); } @@ -82,35 +122,64 @@ fn parallel_referenced_classic_execution_matches_serial_bits() { #[cfg(feature = "parallel")] #[test] -fn fragmented_payloads_decode_serially_without_extra_workspaces() { - let bytes = rgb_fixture_with_layers(300, 260, false, 4); - let image = Image::new(&bytes, &DecodeSettings::default()).expect("classic image"); - let plan = image - .build_referenced_classic_plan_region_with_context( - &mut DecoderContext::default(), - (0, 0, image.width(), image.height()), - ) - .expect("classic whole-image plan"); - assert!( - plan.payloads() - .iter() - .any(|payload| payload.range_count > 1), - "four quality layers split code-block payloads over fragments" - ); +fn fragmented_payloads_decode_in_parallel_with_per_task_buffers() { let pool = rayon::ThreadPoolBuilder::new() .num_threads(4) .build() .expect("four-thread pool"); - let (serial, serial_workspace_bytes) = execute_whole_plan(&bytes, false); - let (parallel, parallel_workspace_bytes) = pool.install(|| execute_whole_plan(&bytes, true)); - assert_eq!(serial, parallel); - assert_eq!(parallel_workspace_bytes, serial_workspace_bytes); + for (reversible, layers) in [(false, 4), (true, 8), (false, 8)] { + let bytes = rgb_fixture_with_layers(300, 260, reversible, layers); + let crop = (37, 21, 230, 211); + for region in [whole_image(&bytes), crop] { + let case = format!("reversible={reversible} layers={layers} region={region:?}"); + let largest = largest_fragmented_payload(&bytes, region) + .unwrap_or_else(|| panic!("{case}: layers split payloads over fragments")); + let serial = execute_region(&bytes, region, false); + let parallel = pool.install(|| execute_region(&bytes, region, true)); + assert_eq!(serial.planes, parallel.planes, "{case}"); + // One prepared workspace per extra task, each beside its buffer. + let tasks = parallel.task_payloads.len(); + assert!(tasks >= 1, "{case}: no extra task"); + assert_eq!( + parallel.workspace_bytes, + serial.workspace_bytes * (tasks + 1), + "{case}" + ); + for &(_, capacity) in ¶llel.task_payloads { + assert!( + capacity >= largest, + "{case}: task buffer {capacity} < {largest}" + ); + } + // The pool hands out the extra tasks first, so the parallel rows + // joined fragments in their own buffers. + assert!( + parallel.task_payloads.iter().any(|&(len, _)| len > 0), + "{case}: no extra task joined a fragmented payload" + ); + } + } } #[test] fn parallel_request_on_the_global_pool_matches_serial_bits() { - let bytes = rgb_fixture(200, 136, false); - let (serial, _) = execute_whole_plan(&bytes, false); - let (parallel, _) = execute_whole_plan(&bytes, true); - assert_eq!(serial, parallel); + for layers in [1, 6] { + let bytes = rgb_fixture_with_layers(200, 136, false, layers); + let region = whole_image(&bytes); + let serial = execute_region(&bytes, region, false); + let parallel = execute_region(&bytes, region, true); + assert_eq!(serial.planes, parallel.planes, "layers={layers}"); + assert_eq!( + serial.task_payloads, + Vec::<(usize, usize)>::new(), + "layers={layers}" + ); + // However many workers the global pool has, each extra task keeps + // one prepared workspace. + assert_eq!( + parallel.workspace_bytes, + serial.workspace_bytes * (parallel.task_payloads.len() + 1), + "layers={layers}" + ); + } } diff --git a/crates/j2k-native/src/image/native.rs b/crates/j2k-native/src/image/native.rs index 84f48964f..e3ea7b995 100644 --- a/crates/j2k-native/src/image/native.rs +++ b/crates/j2k-native/src/image/native.rs @@ -15,6 +15,8 @@ use crate::{ use super::Image; mod allocation; +mod interleave; +mod rows; pub(super) use allocation::{try_clone_color_space, NativeOutputBudget}; impl<'a> Image<'a> { @@ -26,11 +28,14 @@ impl<'a> Image<'a> { retained_baseline_bytes: usize, parallel: bool, ) -> Result { - let mut budget = NativeOutputBudget::for_decoded_channels( + // The decode budget holds the `f32` planes the pack reads; the packed + // planes are the output and have their own allowance. + NativeOutputBudget::for_decoded_channels( retained_baseline_bytes, components, component_owner_capacity, )?; + let mut budget = NativeOutputBudget::for_output(); budget.include_elements::(components.len())?; for (component_idx, component) in components.iter().enumerate() { let sampling = self.component_plane_sampling_at(component_idx); @@ -93,12 +98,7 @@ impl<'a> Image<'a> { has_alpha: self.has_alpha, planes, }; - NativeOutputBudget::validate_component_pack( - retained_baseline_bytes, - components, - component_owner_capacity, - &packed, - )?; + NativeOutputBudget::validate_component_pack(&packed)?; Ok(packed) } diff --git a/crates/j2k-native/src/image/native/allocation.rs b/crates/j2k-native/src/image/native/allocation.rs index 089bafe1c..3df7ebdf0 100644 --- a/crates/j2k-native/src/image/native/allocation.rs +++ b/crates/j2k-native/src/image/native/allocation.rs @@ -64,14 +64,18 @@ impl NativeOutputBudget { Ok(budget) } - pub(super) fn validate_component_pack( - retained_image_bytes: usize, - components: &[ComponentData], - component_owner_capacity: usize, - packed: &DecodedNativeComponents, - ) -> Result<()> { - let mut budget = - Self::for_decoded_channels(retained_image_bytes, components, component_owner_capacity)?; + /// The allowance of a decode's returned native component planes. They + /// are its output, like a caller's buffer, so the decode budget, which + /// holds the working memory they are packed from, does not charge them; + /// they are held to this allowance of their own before they are + /// allocated, and each allocation to the per-allocation cap. The planes + /// route and the rows route share it. + pub(super) fn for_output() -> Self { + Self::default() + } + + pub(super) fn validate_component_pack(packed: &DecodedNativeComponents) -> Result<()> { + let mut budget = Self::for_output(); budget.include_elements::(packed.planes.capacity())?; for plane in &packed.planes { budget.include_elements::(plane.data.capacity())?; diff --git a/crates/j2k-native/src/image/native/allocation/tests.rs b/crates/j2k-native/src/image/native/allocation/tests.rs index 41d552fe3..9f99a9af2 100644 --- a/crates/j2k-native/src/image/native/allocation/tests.rs +++ b/crates/j2k-native/src/image/native/allocation/tests.rs @@ -2,10 +2,13 @@ use super::{bit_capacity_bytes, NativeOutputBudget}; use crate::error::DecodeError; -use crate::j2c::ComponentData; +use crate::j2c::{ComponentData, DecoderContext}; use crate::math::{SimdBuffer, SIMD_WIDTH}; use crate::DEFAULT_MAX_DECODE_BYTES; -use crate::{ColorSpace, ComponentPlane, DecodedComponents, RawBitmap}; +use crate::{ + encode, ColorSpace, ComponentPlane, CpuDecodeParallelism, DecodeErrorClass, DecodeSettings, + DecodedComponents, EncodeOptions, Image, RawBitmap, +}; use alloc::{vec, vec::Vec}; use core::mem::size_of; @@ -169,3 +172,56 @@ fn bit_vector_capacity_is_counted_in_storage_bytes() { if requested > cap && cap == DEFAULT_MAX_DECODE_BYTES )); } + +/// The packed planes of a native component decode are its output, like the +/// rows route's planes: the decode budget holds the `f32` planes they are +/// packed from, not the output beside them. An image that does not stream +/// its rows (five components), given a budget with room for its working +/// memory but not for its `f32` planes and output together, decodes to the +/// bytes a decode with the whole budget gives; one with no room for the +/// `f32` planes is still refused. +#[test] +fn native_component_planes_are_output_beside_the_decode_budget() { + let (width, height, components) = (256_u32, 256_u32, 5_u16); + let samples: Vec = (0..width * height * u32::from(components) * 2) + .map(|index| (index.wrapping_mul(2_654_435_761) >> 24) as u8) + .collect(); + let options = EncodeOptions { + num_decomposition_levels: 2, + tile_size: Some((64, 64)), + ..EncodeOptions::default() + }; + let bytes = encode(&samples, width, height, components, 12, false, &options).expect("encode"); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("image"); + assert!(!image.region_decode_streams_rows()); + let planes = |retained_baseline_bytes| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); + image + .decode_native_components_with_context_and_retained_baseline( + &mut context, + retained_baseline_bytes, + ) + .map(|decoded| { + decoded + .planes + .into_iter() + .map(|plane| (plane.data, plane.bit_depth, plane.bytes_per_sample)) + .collect::>() + }) + }; + let expected = planes(0).expect("a decode with the whole budget"); + let output: usize = expected.iter().map(|(data, ..)| data.len()).sum(); + assert_eq!(output, 256 * 256 * 5 * 2); + // The `f32` planes without padding or owners: less than the decode holds. + let f32_planes = 256 * 256 * 5 * size_of::(); + let decoded = planes(DEFAULT_MAX_DECODE_BYTES - (f32_planes + output - 1)) + .expect("the output is not charged to the decode budget"); + assert!(decoded == expected, "same planes"); + let error = planes(DEFAULT_MAX_DECODE_BYTES - f32_planes / 2) + .expect_err("the f32 planes pass the budget"); + assert!( + matches!(error.classify(), DecodeErrorClass::ResourceLimit { .. }), + "{error:?}" + ); +} diff --git a/crates/j2k-native/src/image/native/interleave.rs b/crates/j2k-native/src/image/native/interleave.rs new file mode 100644 index 000000000..90b97daac --- /dev/null +++ b/crates/j2k-native/src/image/native/interleave.rs @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! Interleaved native samples of several components, packed a block of +//! pixels at a time. + +use alloc::vec::Vec; + +use super::{round_clamped_native, signed_native_bounds, unsigned_native_max}; +use crate::j2c::ComponentData; + +use super::Image; + +/// Pixels packed per block, and so per task when the pack is split across +/// threads. +const BLOCK_PIXELS: usize = 16 * 1024; + +impl Image<'_> { + /// Append samples `0..pixel_count` of every component, pixel by pixel, + /// as packed little-endian native samples of `bit_depth`. Produces the + /// same bytes as [`Self::push_component_native_sample_bytes`] for every + /// index and component, but two to four `f32` components of one + /// signedness and at most 16 bits are converted as + /// [`Self::extend_component_native_sample_bytes`] converts one, a block + /// of pixels at a time, the blocks split across threads when `parallel`. + #[expect( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + reason = "samples are rounded and clamped to the component's 8- or 16-bit range before packing" + )] + pub(crate) fn extend_interleaved_native_sample_bytes( + out: &mut Vec, + components: &[ComponentData], + pixel_count: usize, + bit_depth: u8, + parallel: bool, + ) { + let Some(planes) = float_planes(components, pixel_count, bit_depth) else { + for index in 0..pixel_count { + for component in components { + Self::push_component_native_sample_bytes(out, component, index, bit_depth); + } + } + return; + }; + let planes = &planes[..components.len()]; + let start = out.len(); + let bytes = if bit_depth <= 8 { 1 } else { 2 }; + // Callers reserve the full output first, so this does not reallocate. + out.resize(start + pixel_count * components.len() * bytes, 0); + let destination = &mut out[start..]; + match (bit_depth, components[0].signed) { + (1..=8, true) => { + let (min, max) = signed_native_bounds(bit_depth); + interleave_converted(destination, planes, parallel, move |sample| { + [(round_clamped_native(sample, min, max) as i8) as u8] + }); + } + (1..=8, false) => { + let max = unsigned_native_max(bit_depth); + interleave_converted(destination, planes, parallel, move |sample| { + [round_clamped_native(sample, 0.0, max) as u8] + }); + } + (_, true) => { + let (min, max) = signed_native_bounds(bit_depth); + interleave_converted(destination, planes, parallel, move |sample| { + ((round_clamped_native(sample, min, max) as i16) as u16).to_le_bytes() + }); + } + (_, false) => { + let max = unsigned_native_max(bit_depth); + interleave_converted(destination, planes, parallel, move |sample| { + (round_clamped_native(sample, 0.0, max) as u16).to_le_bytes() + }); + } + } + } +} + +/// The first `pixel_count` samples of each component, when there are two +/// to four `f32` components of one signedness and `bit_depth` is 1 to 16. +fn float_planes( + components: &[ComponentData], + pixel_count: usize, + bit_depth: u8, +) -> Option<[&[f32]; 4]> { + let first = components.first()?; + if !(2..=4).contains(&components.len()) || !(1..=16).contains(&bit_depth) { + return None; + } + let mut planes: [&[f32]; 4] = [&[]; 4]; + for (plane, component) in planes.iter_mut().zip(components) { + if component.integer_container.is_some() || component.signed != first.signed { + return None; + } + *plane = component.container.truncated().get(..pixel_count)?; + } + Some(planes) +} + +/// Write `convert(sample)` of every plane's sample, pixel by pixel, to +/// `destination`, a block of pixels at a time, the blocks split across +/// threads when `parallel` and the planes are large. +fn interleave_converted( + destination: &mut [u8], + planes: &[&[f32]], + parallel: bool, + convert: impl Fn(f32) -> [u8; N] + Send + Sync, +) { + let pixel_bytes = planes.len() * N; + let block = |(index, destination): (usize, &mut [u8])| { + let first = index * BLOCK_PIXELS; + for (offset, pixel) in destination.chunks_exact_mut(pixel_bytes).enumerate() { + for (samples, bytes) in planes.iter().zip(pixel.as_chunks_mut::().0) { + *bytes = convert(samples[first + offset]); + } + } + }; + #[cfg(feature = "parallel")] + if parallel && planes[0].len() >= super::PARALLEL_PACK_SAMPLES { + use rayon::prelude::*; + destination + .par_chunks_mut(pixel_bytes * BLOCK_PIXELS) + .enumerate() + .for_each(block); + return; + } + #[cfg(not(feature = "parallel"))] + let _ = parallel; + destination + .chunks_mut(pixel_bytes * BLOCK_PIXELS) + .enumerate() + .for_each(block); +} + +#[cfg(test)] +mod tests { + use alloc::vec::Vec; + + use crate::image::Image; + use crate::j2c::ComponentData; + use crate::math::{SimdBuffer, SIMD_WIDTH}; + + /// Samples around the bounds and rounding ties of 1 to 16 bits. + #[expect(clippy::cast_precision_loss, reason = "test values fit f32 exactly")] + fn samples(count: usize) -> Vec { + let mut samples = Vec::from([f32::NAN, f32::INFINITY, f32::NEG_INFINITY, -0.0]); + let mut value = -70_000_i32; + while samples.len() < count { + samples.extend([value as f32, value as f32 + 0.5, value as f32 - 0.5]); + value += 97; + } + samples.truncate(count); + samples + } + + #[test] + fn interleaved_pack_matches_the_per_sample_pack() { + let count = 5000; + let base = samples(count); + for components in 2..=5 { + for bit_depth in [1, 4, 8, 9, 12, 16, 17] { + for signed in [false, true] { + for parallel in [false, true] { + let planes: Vec = (0..components) + .map(|component| { + let mut samples = base.clone(); + samples.rotate_left(component * 7); + ComponentData { + container: SimdBuffer::::new(samples), + integer_container: None, + bit_depth, + signed, + } + }) + .collect(); + let mut expected = Vec::new(); + for index in 0..count { + for component in &planes { + Image::push_component_native_sample_bytes( + &mut expected, + component, + index, + bit_depth, + ); + } + } + let mut packed = Vec::from([7_u8]); + packed.reserve(expected.len()); + Image::extend_interleaved_native_sample_bytes( + &mut packed, + &planes, + count, + bit_depth, + parallel, + ); + assert_eq!( + packed[1..], + expected[..], + "{components} x {bit_depth} bits, signed {signed}, parallel {parallel}" + ); + assert_eq!(packed[0], 7); + } + } + } + } + } +} diff --git a/crates/j2k-native/src/image/native/rows.rs b/crates/j2k-native/src/image/native/rows.rs new file mode 100644 index 000000000..711cb8be9 --- /dev/null +++ b/crates/j2k-native/src/image/native/rows.rs @@ -0,0 +1,460 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! Native-bit-depth component planes packed from a region decode's +//! streamed rows instead of from region-sized `f32` planes. + +use alloc::vec::Vec; + +use super::{ + round_clamped_native, signed_native_bounds, try_clone_color_space, unsigned_native_max, + NativeOutputBudget, +}; +use crate::color::{ + ColorSpace, ComponentPlane, ComponentRowSink, DecodedNativeComponents, NativeComponentPlane, +}; +use crate::error::{bail, DecodeError, DecodingError, Result, ValidationError}; +use crate::j2c::codestream::ComponentInfo; +use crate::j2c::DecoderContext; +use crate::roi::validate_roi; +use crate::{ + checked_decode_byte_len2, checked_decode_sample_count, native_bytes_per_sample, + try_reserve_decode_elements, try_resize_decode_elements, +}; + +use super::Image; + +impl<'a> Image<'a> { + /// The precision of every component when a region decode's rows (see + /// [`Self::decode_region_rows_for_integer_output_with_context`]) carry + /// the samples the native-bit-depth decodes pack: the image has no + /// palette, channel definition, sYCC or `CIELab`, which only those + /// decodes' planes resolve or convert, and its components are at full + /// sampling with one precision of at most 24 bits. `None` otherwise. + #[doc(hidden)] + #[must_use] + pub fn streamed_rows_native_precision(&self) -> Option { + if !self.permits_line_region_idwt() { + return None; + } + let precision = self.header.component_infos.first()?.size_info.precision; + let uniform = self.header.component_infos.iter().all(|component| { + component.size_info.precision == precision + && component.size_info.horizontal_resolution == 1 + && component.size_info.vertical_resolution == 1 + }); + (uniform && (1..=24).contains(&precision)).then_some(precision) + } + + /// [`Self::decode_native_region_components_with_context`] on the + /// region decode that hands its rows to a sink (see + /// [`Self::decode_region_rows_for_integer_output_with_context`]), which + /// packs them into the returned planes as they are finished: the decode + /// holds no region-sized `f32` planes. The planes hold the same bytes. + /// Only for an image with a [`Self::streamed_rows_native_precision`] + /// whose region decodes stream their rows (see + /// [`Self::region_decode_streams_rows`]). + /// The region of the whole image gives + /// [`Self::decode_native_components_with_context`]'s planes. + /// + /// The returned planes are the decode's output: like a caller's buffer, + /// and like the planes the planes route packs, they are not charged to + /// the decode budget but to an allowance of their own (see + /// `NativeOutputBudget::for_output`), checked before the decode, and + /// each is held to the per-allocation cap. They are allocated when the + /// first rows arrive, so a region decode that falls back to component + /// planes meets the decode budget before they exist. + /// + /// # Errors + /// + /// Returns an error when the region is invalid, the image has no + /// streamed rows precision or does not stream its rows, a plane cannot + /// be allocated, or decoding fails. + #[doc(hidden)] + pub fn decode_native_region_components_rows_with_context( + &self, + roi: (u32, u32, u32, u32), + decoder_context: &mut DecoderContext<'a>, + ) -> Result { + let dimensions = (roi.2, roi.3); + let mut sink = NativePlanesSink::new(self, dimensions); + self.decode_native_rows(roi, decoder_context, &mut sink)?; + let planes = sink.finish()?; + let color_space = try_clone_color_space(&self.color_space)?; + Ok(DecodedNativeComponents { + dimensions, + color_space, + has_alpha: self.has_alpha, + planes, + }) + } + + /// The region decode of [`Self::decode_native_region_components_rows_with_context`] + /// into `sink`. + fn decode_native_rows( + &self, + roi: (u32, u32, u32, u32), + decoder_context: &mut DecoderContext<'a>, + sink: &mut NativePlanesSink<'_>, + ) -> Result<()> { + validate_roi((self.width(), self.height()), roi)?; + if self.streamed_rows_native_precision().is_none() || !self.region_decode_streams_rows() { + bail!(DecodingError::UnsupportedFeature( + "streamed native component planes need one to four unsubsampled components of one precision up to 24 bits, no image or tile offsets, and no palette, channel definition, sYCC or CIELab" + )); + } + // The planes are allocated when the first rows arrive, but their + // allowance is known now: an output that cannot exist fails before + // the decode. + sink.output_allowance()?; + self.decode_region_rows_for_integer_output_with_context(roi, decoder_context, sink) + } +} + +/// Packs each rectangle of a region decode into the native planes it +/// covers. The planes are allocated when the first rectangle arrives, once +/// per decode. The first error, an allocation or a rectangle that does not +/// fit the planes, stops every later write and fails the decode. +struct NativePlanesSink<'i> { + components: &'i [ComponentInfo], + color_space: &'i ColorSpace, + dimensions: (u32, u32), + planes: Vec, + /// One row of samples wider than 16 bits, packed before it is copied. + scratch: Vec, + error: Option, + /// Times the planes were allocated. + #[cfg(test)] + allocations: usize, + /// Rectangles handed to the sink. + #[cfg(test)] + writes: usize, +} + +impl ComponentRowSink for NativePlanesSink<'_> { + fn write_rows(&mut self, x: u32, y: u32, planes: &[ComponentPlane<'_>]) { + #[cfg(test)] + { + self.writes += 1; + } + if self.error.is_some() { + return; + } + if self.planes.is_empty() { + if let Err(error) = self.allocate() { + self.error = Some(error); + return; + } + } + if self.try_write_rows(x, y, planes).is_none() { + self.error = Some(DecodingError::CodeBlockDecodeFailure.into()); + } + } +} + +impl<'i> NativePlanesSink<'i> { + /// A sink for the `dimensions` of a region of `image`, with no planes + /// yet. + fn new(image: &'i Image<'_>, dimensions: (u32, u32)) -> Self { + Self { + components: &image.header.component_infos, + color_space: &image.color_space, + dimensions, + planes: Vec::new(), + scratch: Vec::new(), + error: None, + #[cfg(test)] + allocations: 0, + #[cfg(test)] + writes: 0, + } + } + + /// The planes held, as the planes route holds its packed planes, to the + /// output allowance, and each to the per-allocation cap. + fn output_allowance(&self) -> Result { + let samples = checked_decode_sample_count(self.dimensions.0, self.dimensions.1)?; + let mut allowance = NativeOutputBudget::for_output(); + allowance.include_elements::(self.components.len())?; + for component in self.components { + let bytes_per_sample = native_bytes_per_sample(component.size_info.precision)?; + let bytes = checked_decode_byte_len2(samples, bytes_per_sample)?; + allowance.include_elements::(bytes)?; + } + allowance.include_color_space_clone(self.color_space)?; + Ok(allowance) + } + + /// Allocate every component's plane within the output allowance. + fn allocate(&mut self) -> Result<()> { + let samples = checked_decode_sample_count(self.dimensions.0, self.dimensions.1)?; + let mut allowance = self.output_allowance()?; + let mut planes = Vec::new(); + try_reserve_decode_elements(&mut planes, self.components.len())?; + allowance.include_capacity_overage::( + self.components.len(), + planes.capacity(), + )?; + for component in self.components { + let bit_depth = component.size_info.precision; + let bytes_per_sample = native_bytes_per_sample(bit_depth)?; + let bytes = checked_decode_byte_len2(samples, bytes_per_sample)?; + let mut data = Vec::new(); + try_resize_decode_elements(&mut data, bytes, 0)?; + allowance.include_capacity_overage::(bytes, data.capacity())?; + planes.push(NativeComponentPlane { + data, + dimensions: self.dimensions, + bit_depth, + signed: component.size_info.signed, + sampling: (1, 1), + bytes_per_sample: u8::try_from(bytes_per_sample) + .map_err(|_| ValidationError::ImageTooLarge)?, + }); + } + self.planes = planes; + #[cfg(test)] + { + self.allocations += 1; + } + Ok(()) + } + + /// The packed planes, or the first error a rectangle met. A decode that + /// handed over no rows has no planes. + fn finish(self) -> Result> { + if let Some(error) = self.error { + return Err(error); + } + if self.planes.is_empty() { + bail!(DecodingError::CodeBlockDecodeFailure); + } + Ok(self.planes) + } + + fn try_write_rows(&mut self, x: u32, y: u32, sources: &[ComponentPlane<'_>]) -> Option<()> { + if sources.len() != self.planes.len() { + return None; + } + for (source, target) in sources.iter().zip(self.planes.iter_mut()) { + if source.bit_depth != target.bit_depth || source.signed != target.signed { + return None; + } + let width = source.dimensions.0 as usize; + let height = source.dimensions.1 as usize; + if width == 0 || height == 0 { + continue; + } + let bytes_per_sample = usize::from(target.bytes_per_sample); + let row_bytes = width.checked_mul(bytes_per_sample)?; + let stride = (target.dimensions.0 as usize).checked_mul(bytes_per_sample)?; + let mut start = (y as usize) + .checked_mul(stride)? + .checked_add((x as usize).checked_mul(bytes_per_sample)?)?; + let rows = source.samples.get(..width.checked_mul(height)?)?; + for samples in rows.chunks_exact(width) { + let end = start.checked_add(row_bytes)?; + let destination = target.data.get_mut(start..end)?; + pack_native_samples( + destination, + samples, + target.bit_depth, + target.signed, + &mut self.scratch, + ) + .ok()?; + start = start.checked_add(stride)?; + } + } + Some(()) + } +} + +/// Write `samples` to `destination` as the packed little-endian native +/// samples `Image::extend_component_native_sample_bytes` appends for an +/// `f32` component of `bit_depth`. Samples wider than 16 bits are packed +/// in `scratch` first. +#[expect( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + reason = "samples are rounded and clamped to the component's 8- or 16-bit range before packing" +)] +pub(super) fn pack_native_samples( + destination: &mut [u8], + samples: &[f32], + bit_depth: u8, + signed: bool, + scratch: &mut Vec, +) -> Result<()> { + match (bit_depth, signed) { + (1..=8, true) => { + let (min, max) = signed_native_bounds(bit_depth); + for (destination, &sample) in destination.iter_mut().zip(samples) { + *destination = (round_clamped_native(sample, min, max) as i8) as u8; + } + } + (1..=8, false) => { + let max = unsigned_native_max(bit_depth); + for (destination, &sample) in destination.iter_mut().zip(samples) { + *destination = round_clamped_native(sample, 0.0, max) as u8; + } + } + (9..=16, true) => { + let (min, max) = signed_native_bounds(bit_depth); + for (destination, &sample) in destination.as_chunks_mut::<2>().0.iter_mut().zip(samples) + { + *destination = + ((round_clamped_native(sample, min, max) as i16) as u16).to_le_bytes(); + } + } + (9..=16, false) => { + let max = unsigned_native_max(bit_depth); + for (destination, &sample) in destination.as_chunks_mut::<2>().0.iter_mut().zip(samples) + { + *destination = (round_clamped_native(sample, 0.0, max) as u16).to_le_bytes(); + } + } + _ => { + scratch.clear(); + try_reserve_decode_elements(scratch, destination.len())?; + for &sample in samples { + Image::push_native_sample_bytes(scratch, sample, bit_depth, signed); + } + let packed = scratch + .get(..destination.len()) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + destination.copy_from_slice(packed); + } + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use alloc::vec::Vec; + + use super::{pack_native_samples, NativePlanesSink}; + use crate::image::Image; + use crate::j2c::DecoderContext; + use crate::{ + encode, native_bytes_per_sample, CpuDecodeParallelism, DecodeErrorClass, DecodeSettings, + EncodeOptions, + }; + + /// Samples around every bound and rounding tie of each precision, plus + /// values far outside it and NaN. + #[expect(clippy::cast_precision_loss, reason = "bounds of at most 24 bits")] + fn samples(bit_depth: u8) -> Vec { + let mut samples = Vec::from([f32::NAN, f32::INFINITY, f32::NEG_INFINITY, 0.0, -0.0]); + for bound in [ + -(1_i64 << (bit_depth - 1)), + (1_i64 << (bit_depth - 1)) - 1, + (1_i64 << bit_depth) - 1, + 0, + 1, + ] { + for offset in [-1.5, -1.0, -0.5, -0.49, 0.0, 0.49, 0.5, 1.0, 1.5] { + samples.push(bound as f32 + offset); + } + } + samples.extend([1.0e9, -1.0e9, 0.499_999_97, 2.5, -2.5, 3.5]); + samples + } + + #[test] + fn packed_rows_match_the_planes_pack_at_every_precision() { + let mut scratch = Vec::new(); + for bit_depth in 1..=24 { + for signed in [false, true] { + let samples = samples(bit_depth); + let mut expected = Vec::new(); + for &sample in &samples { + Image::push_native_sample_bytes(&mut expected, sample, bit_depth, signed); + } + let bytes = native_bytes_per_sample(bit_depth).expect("bytes per sample"); + let mut packed = alloc::vec![0xa5; samples.len() * bytes]; + pack_native_samples(&mut packed, &samples, bit_depth, signed, &mut scratch) + .expect("pack"); + assert_eq!(packed, expected, "{bit_depth} bits, signed {signed}"); + } + } + } + + fn context<'a>(parallelism: CpuDecodeParallelism) -> DecoderContext<'a> { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(parallelism); + context + } + + /// The rows route allocates its planes when the first rows arrive, once + /// per decode. A region decode that falls back to component planes (here + /// an RGB image without decomposition levels, which has no Tier-1 bands + /// to stream) meets the decode budget before any plane exists: past it, + /// the decode fails with a resource limit and the sink allocated + /// nothing. Headers only: its SIZ claims 12000 x 12000 samples, whose + /// `f32` planes pass the budget before any packet is read. + #[test] + fn native_rows_allocate_the_planes_when_the_first_rows_arrive() { + let options = EncodeOptions { + num_decomposition_levels: 0, + ..EncodeOptions::default() + }; + let mut bytes = encode(&[0; 16 * 16 * 3], 16, 16, 3, 8, false, &options).expect("encode"); + let siz = bytes + .windows(2) + .position(|marker| marker == [0xff, 0x51]) + .expect("SIZ marker"); + for field in [6, 10, 22, 26] { + bytes[siz + field..siz + field + 4].copy_from_slice(&12_000_u32.to_be_bytes()); + } + let image = Image::new(&bytes, &DecodeSettings::default()).expect("headers parse"); + assert!(image.region_decode_streams_rows()); + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + let mut sink = NativePlanesSink::new(&image, (12_000, 12_000)); + let error = image + .decode_native_rows((0, 0, 12_000, 12_000), &mut context(parallelism), &mut sink) + .expect_err("the planes pass the budget"); + assert!( + matches!(error.classify(), DecodeErrorClass::ResourceLimit { .. }), + "{parallelism:?}: {error:?}" + ); + assert_eq!((sink.allocations, sink.writes), (0, 0), "{parallelism:?}"); + assert!(sink.planes.is_empty(), "{parallelism:?}"); + } + } + + /// A streamed region decode hands its rows over in many rectangles, one + /// or more per tile, and the sink allocates its planes for the first of + /// them only. The planes hold the planes route's bytes. + #[test] + fn streamed_native_rows_allocate_the_planes_once() { + let (width, height) = (96_u32, 80_u32); + let samples: Vec = (0..width * height * 3) + .map(|index| (index.wrapping_mul(2_654_435_761) >> 24) as u8) + .collect(); + let options = EncodeOptions { + num_decomposition_levels: 2, + tile_size: Some((32, 32)), + ..EncodeOptions::default() + }; + let bytes = encode(&samples, width, height, 3, 8, false, &options).expect("encode"); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("image"); + let roi = (0, 0, width, height); + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + let mut sink = NativePlanesSink::new(&image, (width, height)); + image + .decode_native_rows(roi, &mut context(parallelism), &mut sink) + .expect("streamed decode"); + assert_eq!(sink.allocations, 1, "{parallelism:?}"); + let writes = sink.writes; + assert!(writes >= 9, "{parallelism:?}: {writes} rectangles"); + let streamed = sink.finish().expect("planes"); + let expected = image + .decode_native_region_components_with_context(roi, &mut context(parallelism)) + .expect("planes route"); + assert_eq!(streamed.len(), expected.planes.len(), "{parallelism:?}"); + for (streamed, expected) in streamed.iter().zip(&expected.planes) { + assert!(streamed.data == expected.data, "{parallelism:?}"); + } + } + } +} diff --git a/crates/j2k-native/src/image/output_api.rs b/crates/j2k-native/src/image/output_api.rs index ec455c834..22ad2aa0d 100644 --- a/crates/j2k-native/src/image/output_api.rs +++ b/crates/j2k-native/src/image/output_api.rs @@ -198,13 +198,13 @@ impl<'a> Image<'a> { parallel, ); } else { - for index in 0..pixel_count { - for component in components { - Self::push_component_native_sample_bytes( - &mut data, component, index, bit_depth, - ); - } - } + Self::extend_interleaved_native_sample_bytes( + &mut data, + components, + pixel_count, + bit_depth, + parallel, + ); } if data.len() != capacity { bail!(DecodingError::CodeBlockDecodeFailure); @@ -288,13 +288,13 @@ impl<'a> Image<'a> { parallel, ); } else { - for index in 0..pixel_count { - for component in components { - Self::push_component_native_sample_bytes( - &mut data, component, index, bit_depth, - ); - } - } + Self::extend_interleaved_native_sample_bytes( + &mut data, + components, + pixel_count, + bit_depth, + parallel, + ); } if data.len() != capacity { bail!(DecodingError::CodeBlockDecodeFailure); @@ -476,9 +476,10 @@ impl<'a> Image<'a> { /// [`Self::decode_region_components_for_integer_output_with_context`] /// that hands the region's samples to `sink` instead of returning - /// component planes. A plain region decode on one thread streams them as - /// they are finished and never holds the whole planes; any other decode - /// hands over its planes as one rectangle at the end. + /// component planes. A plain region decode streams them as they are + /// finished and never holds the whole planes, its Tier-1 bands decoded + /// serially or on the worker pool; any other decode hands over its planes + /// as one rectangle at the end. /// /// # Errors /// @@ -522,6 +523,95 @@ impl<'a> Image<'a> { Ok(()) } + /// Whether [`Self::decode_region_rows_for_integer_output_with_context`] + /// can stream this image's rows, as far as its main header and boxes + /// say: the image has no palette, channel definition, sYCC or `CIELab` + /// (see `permits_line_region_idwt`), its components are at full sampling + /// with at most 24 bits, and the image and its tiles have no offsets and + /// one to four components (see `j2c::image_streams_rows`, which the + /// decode checks before it streams). Any other region decode falls back + /// to component planes of the region and hands them over at the end. + /// A tile's own coding can still keep it there (see + /// [`Self::full_decode_planes_floor`], which plans the full image only + /// when this holds). + #[doc(hidden)] + #[must_use] + pub fn region_decode_streams_rows(&self) -> bool { + self.permits_line_region_idwt() + && self.header.component_infos.iter().all(|component| { + component.size_info.precision <= 24 + && component.size_info.horizontal_resolution == 1 + && component.size_info.vertical_resolution == 1 + }) + && j2c::image_streams_rows(&self.header) + } + + /// The bytes a full-image decode into component planes charges the + /// decode budget at least, planned from the codestream's tiles without + /// decoding, when [`Self::decode_region_rows_for_integer_output_with_context`] + /// over the whole image could stream its rows instead and so need no + /// planes. A caller holding the planes to `budget` bytes runs that + /// region decode when the floor passes it: the planes decode would be + /// refused, and an image it can finish never has a floor past the + /// budget. + /// + /// `None` when the region decode could not stream (see + /// [`Self::region_decode_streams_rows`], or a tile with an inverse MCT + /// over fewer than three components or over wavelets that differ), or + /// when the header alone puts the planes and one tile's coefficients and + /// IDWT buffers within half of `budget`. + /// + /// # Errors + /// + /// Returns the error parsing the tiles meets, which the planes decode + /// meets as well. + #[doc(hidden)] + pub fn full_decode_planes_floor(&self, budget: usize) -> Result> { + if !self.region_decode_streams_rows() { + return Ok(None); + } + j2c::full_decode_planes_floor( + self.codestream, + &self.header, + self.retained_header_metadata_bytes()?, + budget, + ) + } + + /// The output rectangles `(x, y, width, height)` of the image's tiles at + /// this image's resolution, in raster order, leaving out tiles with no + /// output samples. A region decode of each rectangle decodes that tile + /// alone, and together they cover the image once. + /// + /// # Errors + /// + /// Returns an error when the header's sampling factors are invalid. + #[doc(hidden)] + pub fn output_tile_rects(&self) -> Result> { + let size = &self.header.size_data; + let (x_shrink, y_shrink) = ( + size.checked_x_shrink_factor()?, + size.checked_y_shrink_factor()?, + ); + let (x_origin, y_origin) = j2c::output_grid_offset(size); + let mut rects = Vec::new(); + for ty in 0..size.num_y_tiles() { + for tx in 0..size.num_x_tiles() { + let rect = j2c::tile_rect(size, tx, ty); + let x = rect.x0.div_ceil(x_shrink).saturating_sub(x_origin); + let y = rect.y0.div_ceil(y_shrink).saturating_sub(y_origin); + let x1 = rect.x1.div_ceil(x_shrink).saturating_sub(x_origin); + let y1 = rect.y1.div_ceil(y_shrink).saturating_sub(y_origin); + let width = x1.saturating_sub(x); + let height = y1.saturating_sub(y); + if width > 0 && height > 0 { + rects.push((x, y, width, height)); + } + } + } + Ok(rects) + } + /// Rounded-for-integer-output counterpart of /// [`Self::decode_components_with_ht_decoder`]. /// diff --git a/crates/j2k-native/src/image/packet_directory.rs b/crates/j2k-native/src/image/packet_directory.rs index 81736acae..66b1fa1f3 100644 --- a/crates/j2k-native/src/image/packet_directory.rs +++ b/crates/j2k-native/src/image/packet_directory.rs @@ -118,21 +118,46 @@ impl PacketDirectoryCache { #[cfg(test)] mod tests { use crate::{ - ColorSpace, CpuDecodeParallelism, DecodeSettings, DecoderContext, Image, - DEFAULT_MAX_DECODE_BYTES, + CpuDecodeParallelism, DecodeSettings, DecoderContext, Image, DEFAULT_MAX_DECODE_BYTES, }; + /// An owned native component decode beside the packet cache: one with + /// room for its working memory and the cache keeps the cache, and one + /// whose working memory fits only without it evicts the cache and + /// retries. (Its planes are output, which the decode budget does not + /// charge, so the cache competes with the working memory only.) #[test] fn packet_directory_output_allocation_retries_successfully() { let bytes = include_bytes!("../../fixtures/packet_directory/openjpeg_rgb_origin5x3_rpcl.j2k"); - let mut image = Image::new(bytes, &DecodeSettings::default()).unwrap(); - // A retained profile is cloned only during output packing. Its size - // makes that phase, rather than coefficient decoding, the peak owner. - image.color_space = ColorSpace::Icc { - profile: vec![0; 1 << 20], - num_channels: 3, + let decode = |image: &Image<'_>, baseline| { + let mut ctx = DecoderContext::default(); + ctx.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); + image + .decode_native_components_with_context_and_retained_baseline(&mut ctx, baseline) + .map(|decoded| { + decoded + .planes() + .iter() + .map(|plane| plane.data().to_vec()) + .collect::>() + }) }; + let mut control = Image::new(bytes, &DecodeSettings::default()).unwrap(); + control.packet_cache = super::PacketDirectoryCache::disabled(); + let (mut fits, mut fails) = (0, DEFAULT_MAX_DECODE_BYTES); + while fails - fits > 1 { + let candidate = fits + (fails - fits) / 2; + match decode(&control, candidate) { + Ok(_) => fits = candidate, + Err(error) => { + assert!(crate::j2c::is_capacity_error(&error), "{error:?}"); + fails = candidate; + } + } + } + let expected = decode(&control, fits).unwrap(); + let image = Image::new(bytes, &DecodeSettings::default()).unwrap(); let metadata = image.retained_allocation_bytes().unwrap(); let mut ctx = DecoderContext::default(); ctx.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); @@ -141,31 +166,19 @@ mod tests { .decode_region_components_with_context((60, 40, 16, 16), &mut ctx) .unwrap(); } - let expected = image - .decode_native_components_with_context(&mut ctx) - .unwrap(); - let channels = &ctx.tile_decode_context.channel_data; - let channel_bytes = crate::image::allocation::DecodeOwnerBudget::for_components( - 0, - channels, - channels.capacity(), - ) - .unwrap() - .bytes(); - let baseline = - DEFAULT_MAX_DECODE_BYTES - channel_bytes - expected.allocated_bytes().unwrap(); - // Isolate the late failure: the core already fits with the cache live. - image - .run_decode(&mut ctx, None, None, true, baseline) - .expect("core fits with cache"); - assert!(image.retained_allocation_bytes().unwrap() > metadata); - let actual = image - .decode_native_components_with_context_and_retained_baseline(&mut ctx, baseline) - .expect("owned output must evict and retry"); + let cache = image.retained_allocation_bytes().unwrap() - metadata; + assert!(cache > 0); + // Room for the working memory beside the cache. + assert_eq!( + decode(&image, fits - cache).expect("the cache can stay"), + expected + ); + assert_eq!(image.retained_allocation_bytes().unwrap(), metadata + cache); + assert_eq!( + decode(&image, fits).expect("eviction must restore a fitting request"), + expected + ); assert_eq!(image.retained_allocation_bytes().unwrap(), metadata); - for (actual, expected) in actual.planes().iter().zip(expected.planes()) { - assert_eq!(actual.data(), expected.data()); - } } #[test] diff --git a/crates/j2k-native/src/j2c/bitplane.rs b/crates/j2k-native/src/j2c/bitplane.rs index 6bdd2d7c1..d46a85e49 100644 --- a/crates/j2k-native/src/j2c/bitplane.rs +++ b/crates/j2k-native/src/j2c/bitplane.rs @@ -16,7 +16,6 @@ pub(crate) use facade::{ decode, decode_code_block_segments_validated, decode_code_block_segments_validated_profiled, }; pub(crate) use observer::J2kBlockDecodeStats; -pub(crate) use reconstruction::MidpointReconstructor; #[cfg(test)] pub(crate) use state::FORCE_PER_COEFFICIENT; pub(crate) use state::{ diff --git a/crates/j2k-native/src/j2c/bitplane/reconstruction.rs b/crates/j2k-native/src/j2c/bitplane/reconstruction.rs index fd2898e1d..813f72f94 100644 --- a/crates/j2k-native/src/j2c/bitplane/reconstruction.rs +++ b/crates/j2k-native/src/j2c/bitplane/reconstruction.rs @@ -84,6 +84,34 @@ impl MidpointReconstructor { } } + /// Convert a decoded row with the block's midpoint, ROI and scaling policy. + #[inline] + #[expect( + clippy::cast_precision_loss, + reason = "reconstructed integer coefficients enter the codec f32 domain" + )] + pub(crate) fn reconstruct_row( + self, + coefficients: &[Coefficient], + output: &mut [f32], + step: f32, + irreversible_midpoint: bool, + ) { + if irreversible_midpoint { + self.dequantize_row(coefficients, output, step); + } else if self.roi_shift == 0 && self.reversible_is_exact() { + Self::scale_exact_reversible_row(coefficients, output, step); + } else { + for (sample, &coefficient) in output.iter_mut().zip(coefficients) { + *sample = crate::apply_roi_maxshift_inverse_i64( + self.reconstruct_reversible(coefficient), + self.roi_shift, + ) as f32 + * step; + } + } + } + /// Reconstruct and dequantize a row: `output[i] = reconstruct(c[i]) * step` /// for as many elements as both slices hold. /// diff --git a/crates/j2k-native/src/j2c/bitplane/schedule.rs b/crates/j2k-native/src/j2c/bitplane/schedule.rs index c9799b8ff..fdcd184ba 100644 --- a/crates/j2k-native/src/j2c/bitplane/schedule.rs +++ b/crates/j2k-native/src/j2c/bitplane/schedule.rs @@ -46,7 +46,7 @@ fn decode_inner_with_observer( // Build a list so that we can associate coding passes with their segments // and data more easily. for layer in storage.layers.get(code_block.layers.clone())? { - if let Some(range) = layer.segments.clone() { + if let Some(range) = layer.segments() { let layer_segments = storage.segments.get(range)?; for segment in layer_segments { if segment.idx != last_segment_idx { diff --git a/crates/j2k-native/src/j2c/build.rs b/crates/j2k-native/src/j2c/build.rs index 700ad4d2d..8a2e010d9 100644 --- a/crates/j2k-native/src/j2c/build.rs +++ b/crates/j2k-native/src/j2c/build.rs @@ -6,7 +6,7 @@ use super::roi::RoiPlan; use super::tag_tree::{TagNode, TagTree}; use super::tile::{ResolutionTile, Tile}; use crate::error::{DecodingError, Result, ValidationError}; -use crate::try_resize_decode_elements; +use crate::{try_resize_decode_elements, try_zero_fill_decode_elements}; use core::{iter, ops::Range}; mod allocation; @@ -19,6 +19,7 @@ use allocation::{ code_block_grid, prepare_decomposition_storage, push_preallocated, tag_tree_node_count, DecompositionAllocationPlan, }; +pub(in crate::j2c) use allocation::{global_code_block_count, IdwtWorkspaceTracker, RecordCounts}; pub(crate) use selection::{select_region_precincts, PrecinctSelection, PrecinctUse}; /// Build and allocate all necessary structures to process the code-blocks @@ -222,11 +223,7 @@ fn resize_planned_coefficients( storage: &mut DecompositionStorage<'_>, plan: &DecompositionAllocationPlan, ) -> Result<()> { - try_resize_decode_elements( - &mut storage.coefficients, - plan.coefficient_storage_len(), - 0.0, - )?; + try_zero_fill_decode_elements(&mut storage.coefficients, plan.coefficient_storage_len())?; if storage.exact_integer_decode { try_resize_decode_elements( &mut storage.coefficients_i64, @@ -444,14 +441,10 @@ fn build_code_blocks( if end > storage.layers.capacity() { return Err(DecodingError::HostAllocationFailed.into()); } - storage.layers.extend(iter::repeat_n( - Layer { - // This will be updated once we actually read the - // layer segments. - segments: None, - }, - layer_count, - )); + // Each layer is updated once its packet's segments are read. + storage + .layers + .extend(iter::repeat_n(Layer::UNREAD, layer_count)); if storage.layers.len() != end { return Err(DecodingError::InvalidPrecinct.into()); } @@ -744,7 +737,42 @@ pub(crate) struct Segment<'a> { pub(crate) data: &'a [u8], } -#[derive(Clone)] +/// The segments one layer of a code-block contributed: a range of the +/// tile's segment records, once the layer's packet has been read. A tile +/// holds one per layer of every code-block, so it is kept to 8 bytes. +#[derive(Clone, Copy)] pub(crate) struct Layer { - pub(crate) segments: Option>, + start: u32, + /// The range's length, or [`Layer::UNREAD_LEN`] before the packet is + /// read. + len: u32, +} + +impl Layer { + const UNREAD_LEN: u32 = u32::MAX; + + /// A layer whose packet has not been read. + pub(crate) const UNREAD: Self = Self { + start: 0, + len: Self::UNREAD_LEN, + }; + + /// A layer that contributed the segments `range`, or `None` when the + /// range does not fit the record. + pub(crate) fn with_segments(range: Range) -> Option { + let start = u32::try_from(range.start).ok()?; + let len = u32::try_from(range.end.checked_sub(range.start)?) + .ok() + .filter(|len| *len != Self::UNREAD_LEN)?; + start.checked_add(len)?; + Some(Self { start, len }) + } + + /// The segments the layer contributed, once its packet has been read. + pub(crate) fn segments(self) -> Option> { + (self.len != Self::UNREAD_LEN).then(|| { + let start = self.start as usize; + start..start + self.len as usize + }) + } } diff --git a/crates/j2k-native/src/j2c/build/allocation.rs b/crates/j2k-native/src/j2c/build/allocation.rs index 804e7e94d..e78ad2690 100644 --- a/crates/j2k-native/src/j2c/build/allocation.rs +++ b/crates/j2k-native/src/j2c/build/allocation.rs @@ -15,6 +15,7 @@ mod plan; mod reuse; pub(super) use plan::{code_block_grid, tag_tree_node_count}; +pub(in crate::j2c) use plan::{global_code_block_count, IdwtWorkspaceTracker}; pub(super) use reuse::push_preallocated; /// How sub-band coefficient ranges share [`DecompositionStorage::coefficients`]. @@ -62,6 +63,40 @@ impl BuildWorkspace { } } +/// Logical counts of the records every built tile owns. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(in crate::j2c) struct RecordCounts { + pub(in crate::j2c) tile_decompositions: usize, + pub(in crate::j2c) decompositions: usize, + pub(in crate::j2c) sub_bands: usize, + pub(in crate::j2c) precincts: usize, + pub(in crate::j2c) code_blocks: usize, + pub(in crate::j2c) layers: usize, + pub(in crate::j2c) tag_tree_nodes: usize, +} + +impl RecordCounts { + /// These records' exact bytes, independent of the decode allocation cap. + pub(in crate::j2c) fn checked_bytes(self) -> Result { + let mut total = 0usize; + let mut include = |size: usize, count: usize| -> Result<()> { + total = size + .checked_mul(count) + .and_then(|bytes| total.checked_add(bytes)) + .ok_or(ValidationError::ImageTooLarge)?; + Ok(()) + }; + include(size_of::(), self.tile_decompositions)?; + include(size_of::(), self.decompositions)?; + include(size_of::(), self.sub_bands)?; + include(size_of::(), self.precincts)?; + include(size_of::(), self.code_blocks)?; + include(size_of::(), self.layers)?; + include(size_of::(), self.tag_tree_nodes)?; + Ok(total) + } +} + pub(super) struct DecompositionAllocationPlan { retained_baseline_bytes: usize, pub(super) total_bytes: usize, @@ -166,6 +201,18 @@ impl DecompositionAllocationPlan { Ok(()) } + fn record_counts(&self) -> RecordCounts { + RecordCounts { + tile_decompositions: self.tile_decompositions, + decompositions: self.decompositions, + sub_bands: self.sub_bands, + precincts: self.precincts, + code_blocks: self.code_blocks, + layers: self.layers, + tag_tree_nodes: self.tag_tree_nodes, + } + } + /// Charge the unbuilt precincts' records and tag-tree roots. fn include_unbuilt_precincts(&self, total_bytes: &mut usize) -> Result<()> { checked_include_elements::(total_bytes, self.unbuilt_precincts)?; @@ -179,14 +226,8 @@ impl DecompositionAllocationPlan { /// Reject a logically impossible request before walking every precinct. fn validate_minimum_live_workspace(&self, idwt_workspace_elements: usize) -> Result<()> { let mut total_bytes = self.retained_baseline_bytes; - checked_include_elements::(&mut total_bytes, self.tile_decompositions)?; - checked_include_elements::(&mut total_bytes, self.decompositions)?; - checked_include_elements::(&mut total_bytes, self.sub_bands)?; - checked_include_elements::(&mut total_bytes, self.precincts)?; - checked_include_elements::(&mut total_bytes, self.code_blocks)?; - checked_include_elements::(&mut total_bytes, self.layers)?; + checked_include_elements::(&mut total_bytes, self.record_counts().checked_bytes()?)?; checked_include_elements::(&mut total_bytes, self.parse_only_layers)?; - checked_include_elements::(&mut total_bytes, self.tag_tree_nodes)?; checked_include_elements::(&mut total_bytes, self.parse_only_code_blocks)?; checked_include_elements::(&mut total_bytes, self.parse_only_tag_tree_nodes)?; self.include_unbuilt_precincts(&mut total_bytes)?; @@ -380,8 +421,8 @@ pub(crate) fn allocate_deferred_coefficients( .unwrap_or(0) }; let mut budget = reuse::ReallocationBudget::from_live_bytes(live_bytes)?; - budget.reserve(&mut storage.coefficients, coefficient_count)?; - crate::try_resize_decode_elements(&mut storage.coefficients, coefficient_count, 0.0)?; + budget.reserve_zeroed(&mut storage.coefficients, coefficient_count)?; + crate::try_zero_fill_decode_elements(&mut storage.coefficients, coefficient_count)?; storage.structural_workspace_bytes = budget.live_bytes(); Ok(()) } @@ -393,6 +434,35 @@ pub(crate) fn allocate_deferred_coefficients( pub(crate) const FINEST_LEVEL_BAND_SLOTS: u32 = 3; pub(crate) const OTHER_LEVEL_BAND_SLOTS: u32 = 2; +/// Band slots of each level with `deep_band_rings`: one more band of +/// lookahead for the pool's workers, held in place of a second batch of +/// reconstructed rows. +const DEEP_RING_EXTRA_SLOTS: u32 = 1; + +/// Band slots each sub-band of the finest active level holds with +/// `shallow_band_rings`: one fewer, as the pool's threads split their bands +/// into parts instead of waiting for whole bands further ahead. +const SHALLOW_FINEST_LEVEL_BAND_SLOTS: u32 = 2; + +/// The band slots of each sub-band's ring with parallel bands, at the +/// finest active level and at the others. With `shared_pool_band_rings` the +/// finest level holds the shallow rings (and no ring the deep rings' extra +/// band, which needs a larger pool): pool threads that find no band of the +/// image to start take another image's bands instead of bands further ahead. +fn pool_band_slots(storage: &DecompositionStorage<'_>) -> (u32, u32) { + let finest = if storage.shallow_band_rings || storage.shared_pool_band_rings { + SHALLOW_FINEST_LEVEL_BAND_SLOTS + } else { + FINEST_LEVEL_BAND_SLOTS + }; + let extra = if storage.deep_band_rings { + DEEP_RING_EXTRA_SLOTS + } else { + 0 + }; + (finest + extra, OTHER_LEVEL_BAND_SLOTS + extra) +} + /// Give each sub-band of a region decode the coefficient range for its /// stored rectangle: the required code-blocks, clipped to the IDWT windows /// unless `clip_to_idwt_windows` is false. With banded Tier-1, a range holds @@ -407,6 +477,7 @@ fn lay_out_region_coefficients( return Err(DecodingError::CodeBlockDecodeFailure.into()); }; let mut coefficient_count = 0usize; + let (finest_slots, other_slots) = pool_band_slots(storage); // Build stores each component's sub-bands together, starting with its // LL band, so a component starts where its LL band does. let per_component = storage.coefficient_layout == CoefficientLayout::Component; @@ -455,9 +526,9 @@ fn lay_out_region_coefficients( .contains(&sub_band_idx) }); if finest { - FINEST_LEVEL_BAND_SLOTS + finest_slots } else { - OTHER_LEVEL_BAND_SLOTS + other_slots } } else { 1 @@ -505,11 +576,14 @@ pub(crate) fn lay_out_whole_region_coefficients( ) -> Result<()> { storage.banded_tier1 = false; storage.parallel_bands = false; + storage.deep_band_rings = false; + storage.shallow_band_rings = false; + storage.shared_pool_band_rings = false; let coefficient_count = lay_out_region_coefficients(storage, true, 0)?; let mut budget = reuse::ReallocationBudget::from_live_bytes(storage.structural_workspace_bytes)?; - budget.reserve(&mut storage.coefficients, coefficient_count)?; - crate::try_resize_decode_elements(&mut storage.coefficients, coefficient_count, 0.0)?; + budget.reserve_zeroed(&mut storage.coefficients, coefficient_count)?; + crate::try_zero_fill_decode_elements(&mut storage.coefficients, coefficient_count)?; storage.structural_workspace_bytes = budget.live_bytes(); Ok(()) } diff --git a/crates/j2k-native/src/j2c/build/allocation/plan.rs b/crates/j2k-native/src/j2c/build/allocation/plan.rs index 663394d94..f1a0a4e4f 100644 --- a/crates/j2k-native/src/j2c/build/allocation/plan.rs +++ b/crates/j2k-native/src/j2c/build/allocation/plan.rs @@ -318,7 +318,7 @@ fn include_code_blocks( DecompositionAllocationPlan::checked_add_count(&mut plan.layers, layer_count) } -fn global_code_block_count( +pub(in crate::j2c) fn global_code_block_count( resolution_tile: &ResolutionTile<'_>, sub_band_rect: IntRect, ) -> Result { @@ -453,7 +453,7 @@ fn checked_rect_elements(rect: IntRect) -> Result { } #[derive(Clone, Copy)] -pub(super) struct IdwtWorkspaceTracker { +pub(in crate::j2c) struct IdwtWorkspaceTracker { active_decomposition_count: Option, transform: WaveletTransform, observed_count: usize, @@ -463,7 +463,7 @@ pub(super) struct IdwtWorkspaceTracker { } impl IdwtWorkspaceTracker { - pub(super) fn new( + pub(in crate::j2c) fn new( active_decomposition_count: Option, transform: WaveletTransform, ) -> Self { @@ -477,7 +477,7 @@ impl IdwtWorkspaceTracker { } } - pub(super) fn observe(&mut self, rect: IntRect) { + pub(in crate::j2c) fn observe(&mut self, rect: IntRect) { if self .active_decomposition_count .is_some_and(|count| self.observed_count < count) @@ -489,7 +489,7 @@ impl IdwtWorkspaceTracker { self.observed_count = self.observed_count.saturating_add(1); } - pub(super) fn finish(self, ll_rect: IntRect) -> Result> { + pub(in crate::j2c) fn finish(self, ll_rect: IntRect) -> Result> { if self.active_decomposition_count.is_none() { return Ok(None); } diff --git a/crates/j2k-native/src/j2c/build/allocation/reuse.rs b/crates/j2k-native/src/j2c/build/allocation/reuse.rs index 0577021fe..d094157fe 100644 --- a/crates/j2k-native/src/j2c/build/allocation/reuse.rs +++ b/crates/j2k-native/src/j2c/build/allocation/reuse.rs @@ -65,6 +65,54 @@ impl ReallocationBudget { self.live_bytes } + /// [`Self::reserve`] for storage that starts as zeros. A buffer that + /// needs a new allocation is replaced by a zeroed one of `target_len` + /// elements (see [`crate::try_zero_fill_decode_elements`]); its old + /// contents are dead, so it is released first and never held beside it. + pub(super) fn reserve_zeroed( + &mut self, + values: &mut Vec, + target_len: usize, + ) -> Result<()> { + values.clear(); + if target_len <= values.capacity() { + return Ok(()); + } + let old_bytes = values + .capacity() + .checked_mul(size_of::()) + .ok_or(ValidationError::ImageTooLarge)?; + *values = Vec::new(); + self.live_bytes = self + .live_bytes + .checked_sub(old_bytes) + .ok_or(ValidationError::ImageTooLarge)?; + let requested_bytes = target_len + .checked_mul(size_of::()) + .ok_or(ValidationError::ImageTooLarge)?; + if self + .live_bytes + .checked_add(requested_bytes) + .is_none_or(|bytes| bytes > DEFAULT_MAX_DECODE_BYTES) + { + return Err(ValidationError::ImageTooLarge.into()); + } + crate::try_zero_fill_decode_elements(values, target_len)?; + let new_live_bytes = values + .capacity() + .checked_mul(size_of::()) + .and_then(|bytes| self.live_bytes.checked_add(bytes)) + .ok_or(ValidationError::ImageTooLarge)?; + if new_live_bytes > DEFAULT_MAX_DECODE_BYTES { + // As in `reserve`: never keep a larger allocation than requested + // after reporting the cap breach. + *values = Vec::new(); + return Err(ValidationError::ImageTooLarge.into()); + } + self.live_bytes = new_live_bytes; + Ok(()) + } + pub(super) fn reserve(&mut self, values: &mut Vec, target_len: usize) -> Result<()> { values.clear(); if target_len <= values.capacity() { @@ -134,7 +182,7 @@ pub(super) fn reserve_decomposition_storage( budget.reserve(&mut storage.code_blocks, plan.code_blocks)?; budget.reserve(&mut storage.layers, plan.layers)?; budget.reserve(&mut storage.tag_tree_nodes, plan.tag_tree_nodes)?; - budget.reserve(&mut storage.coefficients, plan.coefficient_storage_len())?; + budget.reserve_zeroed(&mut storage.coefficients, plan.coefficient_storage_len())?; if storage.exact_integer_decode { budget.reserve( &mut storage.coefficients_i64, diff --git a/crates/j2k-native/src/j2c/build/allocation/tests.rs b/crates/j2k-native/src/j2c/build/allocation/tests.rs index b5285a0a2..2d0005cad 100644 --- a/crates/j2k-native/src/j2c/build/allocation/tests.rs +++ b/crates/j2k-native/src/j2c/build/allocation/tests.rs @@ -4,7 +4,7 @@ use super::plan::{tag_tree_node_count, IdwtWorkspaceTracker}; use super::reuse::{discard_stale_capacity, ReallocationBudget}; use super::{ release_unused_roi_workspace, roi_workspace_bytes, BuildWorkspace, DecompositionAllocationPlan, - Segment, DEFAULT_MAX_DECODE_BYTES, + RecordCounts, Segment, DEFAULT_MAX_DECODE_BYTES, }; use crate::error::{DecodeError, ValidationError}; use crate::j2c::build::{Decomposition, SubBand, SubBandType}; @@ -24,6 +24,19 @@ fn coefficient_plan(baseline: usize, coefficients: usize) -> DecompositionAlloca plan } +#[test] +fn logical_record_bytes_are_independent_of_the_decode_cap() { + let count = DEFAULT_MAX_DECODE_BYTES / size_of::() + 1; + let bytes = RecordCounts { + sub_bands: count, + ..RecordCounts::default() + } + .checked_bytes() + .expect("record byte count fits usize"); + assert_eq!(bytes, count * size_of::()); + assert!(bytes > DEFAULT_MAX_DECODE_BYTES); +} + #[test] fn metadata_plan_retains_logical_samples_without_allocating_pixel_storage() { let mut storage = DecompositionStorage { @@ -441,6 +454,66 @@ fn reusable_capacity_growth_rejects_a_logical_target_over_the_aggregate_cap() { ); } +#[test] +fn zeroed_growth_releases_the_old_buffer_before_taking_the_new_one() { + let mut storage = DecompositionStorage::default(); + storage.coefficients.push(9.0); + let old_bytes = storage.coefficients.capacity() * size_of::(); + let target = storage.coefficients.capacity() + 1; + // Room for the new buffer only once the old one is gone. + let baseline = DEFAULT_MAX_DECODE_BYTES - target * size_of::(); + let mut reallocation = + ReallocationBudget::for_storage(&storage, baseline).expect("live baseline"); + assert_eq!(reallocation.live_bytes(), baseline + old_bytes); + + reallocation + .reserve_zeroed(&mut storage.coefficients, target) + .expect("the old buffer is released first"); + assert_eq!(storage.coefficients, vec![0.0; target]); + assert_eq!( + reallocation.live_bytes(), + baseline + storage.coefficients.capacity() * size_of::() + ); +} + +#[test] +fn zeroed_reuse_keeps_the_allocation_and_none_of_its_samples() { + let mut storage = DecompositionStorage::default(); + storage.coefficients.extend([5.0_f32; 8]); + let capacity = storage.coefficients.capacity(); + let baseline = DEFAULT_MAX_DECODE_BYTES - capacity * size_of::(); + let mut reallocation = + ReallocationBudget::for_storage(&storage, baseline).expect("live baseline"); + + reallocation + .reserve_zeroed(&mut storage.coefficients, 6) + .expect("the retained allocation has room"); + crate::try_zero_fill_decode_elements(&mut storage.coefficients, 6).expect("zero fill"); + assert_eq!(storage.coefficients.capacity(), capacity); + assert_eq!(storage.coefficients, vec![0.0; 6]); + assert_eq!(reallocation.live_bytes(), DEFAULT_MAX_DECODE_BYTES); +} + +#[test] +fn zeroed_growth_rejects_a_target_over_the_aggregate_cap() { + let mut storage = DecompositionStorage::default(); + storage + .coefficients + .try_reserve_exact(1) + .expect("small test capacity"); + let coefficient_bytes = storage.coefficients.capacity() * size_of::(); + let baseline = DEFAULT_MAX_DECODE_BYTES - coefficient_bytes; + let mut reallocation = + ReallocationBudget::for_storage(&storage, baseline).expect("live baseline"); + let target = storage.coefficients.capacity() + 1; + + assert_eq!( + reallocation.reserve_zeroed(&mut storage.coefficients, target), + Err(DecodeError::Validation(ValidationError::ImageTooLarge)) + ); + assert_eq!(storage.coefficients.capacity(), 0); +} + #[test] fn region_decode_scratch_preserves_the_allocation_boundary() { use crate::{ @@ -520,3 +593,27 @@ fn region_decode_scratch_preserves_the_allocation_boundary() { } } } + +/// A layer record keeps its segment range in 8 bytes, tells a layer whose +/// packet is unread from one that contributed no segments, and refuses a +/// range it can't hold rather than truncating it. +#[test] +fn layer_records_keep_their_segment_ranges() { + use crate::j2c::build::Layer; + + assert_eq!(size_of::(), 8); + assert_eq!(Layer::UNREAD.segments(), None); + for range in [0..0, 0..3, 7..7, 41..48] { + let layer = Layer::with_segments(range.clone()).expect("range fits"); + assert_eq!(layer.segments(), Some(range)); + } + let last = u32::MAX as usize; + assert_eq!( + Layer::with_segments(last - 2..last - 1).and_then(Layer::segments), + Some(last - 2..last - 1) + ); + assert!(Layer::with_segments(last..last + 1).is_none()); + assert!(Layer::with_segments(0..last).is_none()); + let start = core::hint::black_box(5); + assert!(Layer::with_segments(start..start - 2).is_none()); +} diff --git a/crates/j2k-native/src/j2c/decode.rs b/crates/j2k-native/src/j2c/decode.rs index 7c2087236..d4c87655a 100644 --- a/crates/j2k-native/src/j2c/decode.rs +++ b/crates/j2k-native/src/j2c/decode.rs @@ -39,6 +39,7 @@ mod direct_plan; #[cfg(feature = "parallel")] mod parallel_bands; mod peak; +mod planes_floor; mod reuse; mod state; mod store; @@ -59,6 +60,7 @@ pub(crate) use self::direct_plan::{ build_referenced_htj2k_grayscale_plan, build_referenced_htj2k_rgba_plan, }; pub(crate) use self::peak::{record_workspace, WorkspaceBytes, WorkspacePeak}; +pub(crate) use self::planes_floor::full_decode_planes_floor; #[cfg(feature = "std")] pub(crate) use self::reuse::is_capacity_error; pub(crate) use self::state::{DecompositionStorage, TileDecodeContext, TileDecompositions}; @@ -314,7 +316,7 @@ fn decode_parsed_tiles<'a>( bail!(TileError::Invalid); } ensure_tiles_share_sample_type(tiles)?; - let (structural_workspace_bytes, reused_baseline) = choose_region_idwt( + let (structural_workspace_bytes, reused_baseline, cpu_decode_parallelism) = choose_region_idwt( header, tiles, ctx, @@ -346,7 +348,6 @@ fn decode_parsed_tiles<'a>( let retained_decode_base_without_scratch = retained_decode_baseline .checked_sub(reused_baseline.scratch_capacity) .ok_or(ValidationError::ImageTooLarge)?; - let cpu_decode_parallelism = ctx.cpu_decode_parallelism; let (tile_ctx, storage) = (&mut ctx.tile_decode_context, &mut ctx.storage); // Under Auto, the tiles may decode at once on worker contexts. On any @@ -569,7 +570,12 @@ fn region_decode_without_hook_or_special_samples( /// if it `wants_rows`, whether it streams them (see [`streams_rows`]). A /// decode that streams gives up the context's retained capacity first (see /// `release_retained_before_streaming`), and the decode goes on with the -/// returned structural bytes and baseline. +/// returned structural bytes, baseline and parallelism. +/// +/// Rows that stream only from serial bands, under `Auto` because the pool's +/// bands can't take a tile, stream serially when the region's planes would +/// pass the decode budget even in a fresh context: the planes decode would +/// be refused, so this costs no decode that succeeds today its parallelism. fn choose_region_idwt( header: &Header<'_>, tiles: &ParsedTiles<'_>, @@ -577,7 +583,7 @@ fn choose_region_idwt( (hooked, wants_rows): (bool, bool), structural_workspace_bytes: usize, reused_baseline: reuse::ReusedDecodeBaseline, -) -> Result<(usize, reuse::ReusedDecodeBaseline)> { +) -> Result<(usize, reuse::ReusedDecodeBaseline, CpuDecodeParallelism)> { let line_region_idwt = ctx.line_region_idwt && ctx.tile_decode_context.line_region_idwt_permitted && region_decode_without_hook_or_special_samples( @@ -586,29 +592,67 @@ fn choose_region_idwt( hooked, ); ctx.tile_decode_context.line_region_idwt = line_region_idwt; - let stream_rows = wants_rows - && line_region_idwt - && streams_rows( + let streams = |tile_ctx: &TileDecodeContext, parallelism| { + wants_rows && line_region_idwt && streams_rows(header, tiles, tile_ctx, parallelism) + }; + let mut parallelism = ctx.cpu_decode_parallelism; + let mut stream_rows = streams(&ctx.tile_decode_context, parallelism); + if !stream_rows + && parallelism == CpuDecodeParallelism::Auto + && streams(&ctx.tile_decode_context, CpuDecodeParallelism::Serial) + && planes_pass_budget( header, - tiles, &ctx.tile_decode_context, - ctx.cpu_decode_parallelism, - ); + structural_workspace_bytes, + reused_baseline, + ) + { + parallelism = CpuDecodeParallelism::Serial; + stream_rows = true; + } ctx.tile_decode_context.stream_rows = stream_rows; ctx.tile_decode_context.streamed_samples = 0; - if stream_rows { - ctx.release_retained_before_streaming(structural_workspace_bytes, reused_baseline) + let (structural_workspace_bytes, reused_baseline) = if stream_rows { + ctx.release_retained_before_streaming(structural_workspace_bytes, reused_baseline)? } else { - Ok((structural_workspace_bytes, reused_baseline)) - } + (structural_workspace_bytes, reused_baseline) + }; + Ok((structural_workspace_bytes, reused_baseline, parallelism)) +} + +/// Whether the output region's component planes would pass the decode +/// budget beside `structural_workspace_bytes` without the context's +/// retained capacity, as `reset` charges them; planes too large to count +/// pass it too. +fn planes_pass_budget( + header: &Header<'_>, + tile_ctx: &TileDecodeContext, + structural_workspace_bytes: usize, + reused_baseline: reuse::ReusedDecodeBaseline, +) -> bool { + let (width, height) = tile_ctx.output_region.map_or( + ( + header.size_data.image_width(), + header.size_data.image_height(), + ), + OutputRegion::dimensions, + ); + let fresh = structural_workspace_bytes + .saturating_sub(reused_baseline.channel_capacity) + .saturating_sub(reused_baseline.scratch_capacity); + planes_floor::region_planes_bytes(header, width, height).map_or(true, |planes| { + planes.saturating_add(fresh) > tile_ctx.planes_budget() + }) } /// Whether a region decode on the row-streamed IDWT can hand its samples to /// a row sink instead of component planes: every tile decodes its Tier-1 in -/// bands (see [`tile_bands`]), so all components can advance together; the -/// image and tiles have no offsets; there are one to four components; and -/// each tile with an inverse MCT has three components or more, the first -/// three with one wavelet, as `apply_inverse_mct_per_tile` requires. +/// bands, serially (see [`tile_bands`]) or on the pool (see [`pool_bands`]), +/// so all components can advance together; the image and tiles have no +/// offsets and there are one to four components (see +/// [`image_streams_rows`]); and each tile with an inverse MCT has three +/// components or more, the first three with one wavelet, as +/// `apply_inverse_mct_per_tile` requires. /// `ensure_every_sample_streamed` checks afterwards that the rows covered /// the region. fn streams_rows( @@ -617,18 +661,14 @@ fn streams_rows( tile_ctx: &TileDecodeContext, parallelism: CpuDecodeParallelism, ) -> bool { - let size = &header.size_data; let components = header.component_infos.len(); - size.image_area_x_offset == 0 - && size.image_area_y_offset == 0 - && size.tile_x_offset == 0 - && size.tile_y_offset == 0 - && (1..=4).contains(&components) + image_streams_rows(header) && tiles.iter().all(|tile| { let infos = &tile.component_infos; infos.len() == components // The row-streamed IDWT already excludes hooks. - && tile_bands(header, tile, tile_ctx, false, parallelism) + && (tile_bands(header, tile, tile_ctx, false, parallelism) + || pool_bands(header, tile, tile_ctx, false, parallelism)) && infos .iter() .all(|component| !component.requires_exact_integer_decode()) @@ -643,6 +683,19 @@ fn streams_rows( }) } +/// The conditions of [`streams_rows`] that the main header settles: the +/// image and its tiles have no offsets, and there are one to four +/// components. The plans that choose a streamed route before decoding +/// check them here too (see `Image::region_decode_streams_rows`). +pub(crate) fn image_streams_rows(header: &Header<'_>) -> bool { + let size = &header.size_data; + size.image_area_x_offset == 0 + && size.image_area_y_offset == 0 + && size.tile_x_offset == 0 + && size.tile_y_offset == 0 + && (1..=4).contains(&header.component_infos.len()) +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub(crate) struct OutputRegion { pub(crate) x: u32, @@ -696,6 +749,8 @@ pub(crate) struct DecodeDebugCounters { /// Components that gave up the pool for whole windows because a band /// was left running with no worker to finish it. pub(crate) pool_band_fallbacks: usize, + /// Pool bands planned in parts, summed over decoded tiles. + pub(crate) pool_band_parts: usize, /// The bytes the decode's budget starts from after `reset`, retained /// capacity included. pub(crate) decode_baseline_bytes: usize, @@ -708,6 +763,43 @@ pub(crate) struct DecodeDebugCounters { pub(crate) ht_task_workspace_growths: usize, #[cfg(all(test, feature = "parallel"))] pub(crate) classic_task_workspace_growths: usize, + /// The most readers a tile streaming its rows from pool bands + /// reconstructed them on. + #[cfg(all(test, feature = "parallel"))] + pub(crate) pool_band_readers: usize, + /// The most band slots a ring of pool bands held that did not hold its + /// whole window, at the finest active level and at the others. + #[cfg(all(test, feature = "parallel"))] + pub(crate) pool_band_finest_slots: usize, + #[cfg(all(test, feature = "parallel"))] + pub(crate) pool_band_other_slots: usize, + /// The most line-buffer samples a tile streaming its rows from pool + /// bands held beyond the IDWT samples its ROI plan charged. + #[cfg(all(test, feature = "parallel"))] + pub(crate) uncharged_line_samples: usize, + /// The most samples of row batches a tile streaming from pool bands held + /// beside its IDWT rings. + #[cfg(all(test, feature = "parallel"))] + pub(crate) line_batch_samples: usize, + /// Bytes of the pool band plans recorded in the workspace peak, summed + /// over components and tiles. + #[cfg(all(test, feature = "parallel"))] + pub(crate) pool_band_plan_bytes: usize, + /// Bytes that budget charged for its threads' scratch, and the bytes + /// that scratch held once the bands decoded, summed over tiles. + #[cfg(all(test, feature = "parallel"))] + pub(crate) thread_scratch_charged: usize, + #[cfg(all(test, feature = "parallel"))] + pub(crate) thread_scratch_held: usize, + /// The most block-scratch samples a thread of such a tile held. + #[cfg(all(test, feature = "parallel"))] + pub(crate) thread_block_samples: usize, + /// Tiles whose failed bands decoded again over whole windows, and the + /// pool threads' scratches still held when they did, summed. + #[cfg(all(test, feature = "parallel"))] + pub(crate) whole_window_reruns: usize, + #[cfg(all(test, feature = "parallel"))] + pub(crate) scratches_live_in_reruns: usize, #[cfg(all(test, feature = "parallel"))] pub(crate) parallel_coefficients: ParallelCoefficientStats, } @@ -891,13 +983,26 @@ fn prepare_region_storage( let clip_to_idwt_windows = clips_to_idwt_windows(tile_ctx, hooked); storage.banded_tier1 = storage.line_region_idwt && tile_bands(header, tile, tile_ctx, hooked, cpu_decode_parallelism); - storage.lockstep_lines = storage.banded_tier1 && tile_ctx.stream_rows; #[cfg(feature = "parallel")] { storage.parallel_bands = storage.line_region_idwt && !storage.banded_tier1 && parallel_bands::applies(header, tile, tile_ctx, hooked, cpu_decode_parallelism); } + storage.lockstep_lines = + (storage.banded_tier1 || storage.parallel_bands) && tile_ctx.stream_rows; + #[cfg(feature = "parallel")] + { + storage.deep_band_rings = storage.parallel_bands + && storage.lockstep_lines + && parallel_bands::deep_rings(tile) + && parallel_bands::reads_whole_rows(header, storage); + storage.one_synthesis_reader = storage.parallel_bands + && storage.lockstep_lines + && parallel_bands::one_reader(tile, storage); + storage.shallow_band_rings = parallel_bands::shallow_rings(tile, storage); + storage.shared_pool_band_rings = parallel_bands::shared_rings(tile_ctx, storage); + } build::allocate_deferred_coefficients( storage, header.skipped_resolution_levels, @@ -940,6 +1045,24 @@ fn tile_bands( bands } +/// Whether `tile` decodes its bands on the worker pool ahead of this +/// decode's row-streamed IDWT (see [`parallel_bands::applies`]). +fn pool_bands( + header: &Header<'_>, + tile: &Tile<'_>, + tile_ctx: &TileDecodeContext, + hooked: bool, + parallelism: CpuDecodeParallelism, +) -> bool { + #[cfg(feature = "parallel")] + return parallel_bands::applies(header, tile, tile_ctx, hooked, parallelism); + #[cfg(not(feature = "parallel"))] + { + let _ = (header, tile, tile_ctx, hooked, parallelism); + false + } +} + /// Where a decoded tile's samples go. enum TileOutput<'o, 'p> { /// The decode's component planes. @@ -960,7 +1083,8 @@ enum TileOutput<'o, 'p> { /// planes, and with [`TileOutput::Sink`], the tile's rows go to the sink. #[expect( clippy::too_many_arguments, - reason = "the per-component stages share decode_tile's state, adapter, and profiling explicitly" + clippy::too_many_lines, + reason = "the per-component stages share decode_tile's state, adapter, and profiling explicitly, and the sink routes and the Tier-1, IDWT and store order stay in one place" )] fn decode_components<'a>( tile: &Tile<'a>, @@ -974,6 +1098,24 @@ fn decode_components<'a>( mut output: TileOutput<'_, '_>, ) -> Result<()> { if let TileOutput::Sink(sink) = output { + #[cfg(feature = "parallel")] + if storage.parallel_bands { + if parallel_bands::decode_tile_rows( + tile, + header, + tile_ctx, + storage, + sink, + profile_enabled, + profile_timings, + )? { + return Ok(()); + } + // The route declined the bands: decode them serially, which the + // pool's band slots also hold. + storage.parallel_bands = false; + storage.banded_tier1 = true; + } return banded_tier1::decode_tile_rows( tile, header, @@ -1444,9 +1586,9 @@ mod tests { #[test] fn collect_classic_code_block_data_preserves_zero_length_segments() { let mut storage = DecompositionStorage::default(); - storage.layers.push(Layer { - segments: Some(0..3), - }); + storage + .layers + .push(Layer::with_segments(0..3).expect("test segment range")); storage.segments.push(Segment { idx: 0, coding_pases: 1, @@ -1494,9 +1636,9 @@ mod tests { #[test] fn collect_classic_code_block_data_rejects_non_contiguous_segment_indices() { let mut storage = DecompositionStorage::default(); - storage.layers.push(Layer { - segments: Some(0..2), - }); + storage + .layers + .push(Layer::with_segments(0..2).expect("test segment range")); storage.segments.push(Segment { idx: 0, coding_pases: 1, diff --git a/crates/j2k-native/src/j2c/decode/banded_tier1.rs b/crates/j2k-native/src/j2c/decode/banded_tier1.rs index d64209f4b..320e3467f 100644 --- a/crates/j2k-native/src/j2c/decode/banded_tier1.rs +++ b/crates/j2k-native/src/j2c/decode/banded_tier1.rs @@ -27,9 +27,10 @@ use crate::error::{DecodeError, DecodingError, ValidationError}; use crate::j2c::build; use crate::j2c::codestream::WaveletTransform; use crate::j2c::idwt::{ComponentLines, LinePlan, LineReconstruction, SubBandRows}; +use crate::j2c::mct::DeferredMct; use crate::j2c::rect::IntRect; use crate::profile; -use crate::{try_resize_decode_elements, ComponentPlane, ComponentRowSink}; +use crate::{try_zero_fill_decode_elements, ComponentPlane, ComponentRowSink}; /// Whether a tile's Tier-1 decodes in bands, given that it feeds the /// streamed IDWT: in stored windows clipped to it, on the calling thread @@ -159,68 +160,42 @@ fn stream_rows<'a>( profile_enabled: bool, tier1_us: &mut u128, ) -> Result<()> { - let components = tile.component_infos.len(); let mut bands = Vec::new(); - let line_plans = plan_tile_lines(tile, header, tile_ctx, storage, &mut bands)?; + let active_levels = (0..tile.component_infos.len()) + .map(|component| push_component_bands(storage, header, component, &mut bands)); + let line_plans = plan_tile_lines(tile, header, tile_ctx, storage, active_levels)?; + let band_of = band_lookup(&bands)?; let empty = IntRect::from_xywh(0, 0, 0, 0); let (copy, region) = line_plans .first() .map_or((empty, empty), |plan| (plan.copy, plan.region)); - let scratch_len = (copy.width() as usize) - .checked_mul(components) - .ok_or(ValidationError::ImageTooLarge)?; // The index first: once the ring buffer is out of `tile_ctx`, every // error path below must put it back. let (precinct_rows, index_bytes) = index_bands(storage, tile_ctx, &bands)?; - let (mut buffer, ring_lens) = - match take_line_buffer(tile_ctx, storage, &line_plans, scratch_len) { - Ok(taken) => taken, - Err(error) => { - storage.structural_workspace_bytes -= index_bytes; - return Err(error); - } - }; - let mut output = RowOutput { - tile, - sink, - round_irreversible_output: tile_ctx.round_irreversible_output, - depths: core::array::from_fn(|index| { - tile_ctx - .channel_data - .get(index) - .map_or((0, false), |component| { - (component.bit_depth, component.signed) - }) - }), - copy, - region, - streamed: 0, + let (mut buffer, ring_lens) = match take_line_buffer(tile_ctx, storage, &line_plans, 0) { + Ok(taken) => taken, + Err(error) => { + storage.structural_workspace_bytes -= index_bytes; + return Err(error); + } }; + let mut output = RowOutput::new(tile, sink, tile_ctx, copy, region); let mut output_samples = 0usize; let result = (|| { - let rings_len = buffer.len() - scratch_len; - let (mut rings, scratch) = buffer.split_at_mut(rings_len); - let mut lines = Vec::new(); - lines - .try_reserve_exact(line_plans.len()) - .map_err(|_| DecodingError::HostAllocationFailed)?; - for (plan, len) in line_plans.iter().zip(&ring_lens) { - let (ring, tail) = core::mem::take(&mut rings).split_at_mut(*len); - rings = tail; - lines.push(plan.reconstruction(ring)?); - } + let mut lines = line_reconstructions(&line_plans, &ring_lens, &mut buffer)?; let mut rows = BandedRows { header, component_infos: &tile.component_infos, tile_ctx: &mut *tile_ctx, storage: &mut *storage, bands, + band_of, precinct_rows, profile_enabled, tier1_us: 0, }; let result = output - .write(&mut lines, &mut rows, scratch) + .write(&mut lines, &mut rows) .and_then(|()| rows.decode_remaining_bands()); *tier1_us = rows.tier1_us; output_samples = lines.iter().map(LineReconstruction::output_samples).sum(); @@ -236,40 +211,40 @@ fn stream_rows<'a>( result } -/// Add every component's bands to `bands` and plan its reconstruction. -/// Returns no plans when no component has rows to reconstruct. -fn plan_tile_lines( +/// Plan every component's streamed reconstruction of its active levels. +/// No plans when no component has rows to reconstruct; every component of +/// a 1x1-sampled tile reconstructs the same rows. +/// Pull each component's active levels before constructing its line plan, +/// preserving the serial caller's band-push and error order. +pub(super) fn plan_tile_lines( tile: &Tile<'_>, header: &Header<'_>, tile_ctx: &TileDecodeContext, storage: &DecompositionStorage<'_>, - bands: &mut Vec, + active_levels: impl ExactSizeIterator>, ) -> Result> { let mut line_plans = Vec::new(); line_plans - .try_reserve_exact(tile.component_infos.len()) + .try_reserve_exact(active_levels.len()) .map_err(|_| DecodingError::HostAllocationFailed)?; - let mut missing = 0; - for (component_idx, component_info) in tile.component_infos.iter().enumerate() { - let active_levels = push_component_bands(storage, header, component_idx, bands)?; - match LinePlan::new( + for (component_idx, (component_info, levels)) in + tile.component_infos.iter().zip(active_levels).enumerate() + { + if let Some(plan) = LinePlan::new( storage, tile_ctx, component_idx, - active_levels, + levels?, header, component_info.wavelet_transform(), )? { - Some(plan) => line_plans.push(plan), - None => missing += 1, + line_plans.push(plan); } } - if missing == tile.component_infos.len() { - line_plans.clear(); + if line_plans.is_empty() { return Ok(line_plans); } - // Every component of a 1x1-sampled tile reconstructs the same rows. - if missing > 0 + if line_plans.len() != tile.component_infos.len() || line_plans .iter() .any(|plan| plan.copy != line_plans[0].copy || plan.region != line_plans[0].region) @@ -279,9 +254,27 @@ fn plan_tile_lines( Ok(line_plans) } +/// Build every component's reconstruction in its own slice of the ring buffer. +pub(super) fn line_reconstructions<'a>( + plans: &[LinePlan], + ring_lens: &[usize], + mut rings: &'a mut [f32], +) -> Result>> { + let mut lines = Vec::new(); + lines + .try_reserve_exact(plans.len()) + .map_err(|_| DecodingError::HostAllocationFailed)?; + for (plan, len) in plans.iter().zip(ring_lens) { + let (ring, tail) = core::mem::take(&mut rings).split_at_mut(*len); + rings = tail; + lines.push(plan.reconstruction(ring)?); + } + Ok(lines) +} + /// Take the context's ring buffer, grown to hold every plan's rings and /// `scratch_len` more samples, and return it with each plan's ring length. -fn take_line_buffer( +pub(super) fn take_line_buffer( tile_ctx: &mut TileDecodeContext, storage: &DecompositionStorage<'_>, line_plans: &[LinePlan], @@ -305,7 +298,7 @@ fn take_line_buffer( ); let mut buffer = core::mem::take(&mut tile_ctx.idwt_line_rings); let resized = if buffer.len() < total { - try_resize_decode_elements(&mut buffer, total, 0.0) + try_zero_fill_decode_elements(&mut buffer, total) } else { Ok(()) }; @@ -318,7 +311,7 @@ fn take_line_buffer( } /// Where a tile's reconstructed rows go: transformed, then to the sink. -struct RowOutput<'o, 't> { +pub(super) struct RowOutput<'o, 't> { tile: &'o Tile<'t>, sink: &'o mut dyn ComponentRowSink, round_irreversible_output: bool, @@ -327,42 +320,129 @@ struct RowOutput<'o, 't> { /// The rows, in the coordinates of `region`, that the tile reconstructs. copy: IntRect, region: IntRect, + /// The inverse MCT of three 8-bit components, which finishes their rows + /// for an RGB8 destination (see [`DeferredMct::pack_rgb8_row`]). + mct: Option, /// Samples handed to the sink. - streamed: usize, + pub(super) streamed: usize, } -impl RowOutput<'_, '_> { - /// Advance every component's reconstruction a row at a time, copying - /// each row into `scratch`, one row per component, and hand them over. - fn write( +impl<'o, 't> RowOutput<'o, 't> { + /// The output of `tile`'s rows `copy`, in the coordinates of `region`, + /// to `sink`. + pub(super) fn new( + tile: &'o Tile<'t>, + sink: &'o mut dyn ComponentRowSink, + tile_ctx: &TileDecodeContext, + copy: IntRect, + region: IntRect, + ) -> Self { + let depths: [(u8, bool); 4] = core::array::from_fn(|index| { + tile_ctx + .channel_data + .get(index) + .map_or((0, false), |component| { + (component.bit_depth, component.signed) + }) + }); + let rgb8 = + tile.component_infos.len() == 3 && depths[..3].iter().all(|(depth, _)| *depth == 8); + Self { + tile, + sink, + round_irreversible_output: tile_ctx.round_irreversible_output, + depths, + copy, + region, + mct: (rgb8 && tile.mct).then(|| { + DeferredMct::for_rows(&tile.component_infos, tile_ctx.round_irreversible_output) + }), + streamed: 0, + } + } + + /// The sink's RGB8 rows for `rows` rows of the tile from row `y`, when + /// it has a destination for three 8-bit components. + fn rgb8_rows(&mut self, y: u32, rows: usize) -> Result>> { + let (copy, region) = (self.copy, self.region); + let rgb8 = self.depths.len() >= 3 + && self.tile.component_infos.len() == 3 + && self.depths[..3].iter().all(|(depth, _)| *depth == 8); + if !rgb8 { + return Ok(None); + } + let Some(out) = self.sink.rgb8_region(region.width(), region.height()) else { + return Ok(None); + }; + let stride = region.width() as usize * 3; + let start = (y - region.y0) as usize * stride; + let out = out + .get_mut(start..start + rows * stride) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + Ok(Some(Rgb8Rows { + out, + #[cfg(feature = "parallel")] + stride, + x: (copy.x0 - region.x0) as usize * 3, + })) + } + + /// Advance every component's reconstruction a row at a time and finish + /// each row in the IDWT's ring once no lifting step reads it: transformed + /// in place and packed into the sink's RGB8 rows when it has them, the + /// inverse MCT fused into the pack where [`DeferredMct::pack_rgb8_row`] + /// fuses it, else handed to `write_rows`. + pub(super) fn write( &mut self, lines: &mut [LineReconstruction<'_>], - rows: &mut BandedRows<'_, '_>, - scratch: &mut [f32], + rows: &mut dyn SubBandRows, ) -> Result<()> { let (copy, components) = (self.copy, lines.len()); let width = copy.width() as usize; if lines.is_empty() || width == 0 { return Ok(()); } + let (tile, round_irreversible_output, mct) = + (self.tile, self.round_irreversible_output, self.mct); for y in copy.y0..copy.y1 { + // Every component's row final before any takes the arrival that + // frees it, so the bands decode in the order rows handed over + // final read them, and a failing tile meets the same band first. + for line in lines.iter_mut() { + line.finish_next_row(rows)?; + } let mut outputs: [&mut [f32]; 4] = Default::default(); - for ((line, output), chunk) in lines - .iter_mut() - .zip(outputs.iter_mut()) - .zip(scratch.chunks_exact_mut(width)) - { + for (line, output) in lines.iter_mut().zip(outputs.iter_mut()) { let (row_y, row) = line - .next_row(rows)? + .next_row_mut(rows)? .ok_or(DecodingError::CodeBlockDecodeFailure)?; if row_y != y || row.len() != width { return Err(DecodingError::CodeBlockDecodeFailure.into()); } - chunk.copy_from_slice(row); - *output = chunk; + *output = row; } let outputs = &mut outputs[..components]; - store::transform_streamed_rows(self.tile, self.round_irreversible_output, outputs)?; + if let Some(rgb8) = self.rgb8_rows(y, 1)? { + let pixels = rgb8 + .out + .get_mut(rgb8.x..rgb8.x + width * 3) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + match (mct, outputs) { + (Some(mct), [red, green, blue]) => { + mct.pack_rgb8_row([red, green, blue], pixels, crate::color::pack_rgb8); + } + (_, outputs) => { + store::transform_streamed_rows(tile, round_irreversible_output, outputs)?; + let [red, green, blue, ..] = outputs else { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + }; + crate::color::pack_rgb8([red, green, blue], pixels); + } + } + self.streamed = self.streamed.saturating_add(width * components); + continue; + } + store::transform_streamed_rows(tile, round_irreversible_output, outputs)?; let component_rows: [ComponentPlane<'_>; 4] = core::array::from_fn(|index| ComponentPlane { samples: outputs.get(index).map_or(&[][..], |row| &**row), @@ -382,17 +462,440 @@ impl RowOutput<'_, '_> { } } +#[cfg(feature = "parallel")] +impl RowOutput<'_, '_> { + /// [`Self::write`] through one reader per component, each on its own + /// pool task, `batch` rows at a time. Each component's reconstruction + /// advances `batch` rows into its own rows of `scratch`, then the rows + /// are transformed and handed over, packed straight into the sink's RGB8 + /// destination when it has one for three 8-bit components. With room for + /// two batches in `scratch`, a batch is transformed and packed on the + /// pool while the next one reconstructs. + pub(super) fn write_batches( + &mut self, + lines: &mut [LineReconstruction<'_>], + readers: &mut [&mut (dyn SubBandRows + Send)], + scratch: &mut [f32], + batch: usize, + ) -> Result<()> { + let (copy, components) = (self.copy, lines.len()); + let width = copy.width() as usize; + if lines.is_empty() || width == 0 || batch == 0 { + return Ok(()); + } + let batch_len = width + .checked_mul(batch) + .ok_or(ValidationError::ImageTooLarge)?; + let half = batch_len + .checked_mul(components) + .ok_or(ValidationError::ImageTooLarge)?; + let transform = (self.tile, self.round_irreversible_output); + let tasks = readers.len(); + let (now, rest) = scratch.split_at_mut(half.min(scratch.len())); + // With room for two batches, a batch is transformed and handed over + // on the pool while the next one reconstructs in the other half of + // `scratch`. + let pipelined = rest.len() >= half; + let mut halves = [now, rest]; + // The rows, from a row, of the batch in `halves[1]` left to finish. + let mut pending = None; + let mut y = copy.y0; + loop { + let rows = (copy.y1.saturating_sub(y) as usize).min(batch); + let [now, before] = &mut halves; + let done = pending.take(); + if rows == 0 { + if let Some(done) = done { + self.finish_batch( + transform, + before, + done, + (batch_len, components, tasks), + None, + )?; + } + return Ok(()); + } + let mut parts = batch_parts(now, batch_len, rows * width, components)?; + let mut reconstruct = + || reconstruct_batch(lines, readers, &mut parts[..components], y, width); + match done { + Some(done) => self.finish_batch( + transform, + before, + done, + (batch_len, components, tasks), + Some(&mut reconstruct), + )?, + None => reconstruct()?, + } + if pipelined { + pending = Some((y, rows)); + halves.swap(0, 1); + } else { + let [now, _] = &mut halves; + self.finish_batch( + transform, + now, + (y, rows), + (batch_len, components, tasks), + None, + )?; + } + #[expect( + clippy::cast_possible_truncation, + reason = "a batch holds at most the `copy.y1 - y` rows left, a u32" + )] + let rows = rows as u32; + y += rows; + } + } + + /// Transform the `rows` rows of each component in `buffer`, `batch_len` + /// samples apart, from row `y`, in up to `tasks` ranges, and hand them + /// over: packed into the sink's RGB8 rows when it has them, else through + /// `write_rows`. With `alongside`, the rows finish on the pool while + /// `alongside` runs here. + fn finish_batch( + &mut self, + transform: (&Tile<'_>, bool), + buffer: &mut [f32], + (y, rows): (u32, usize), + (batch_len, components, tasks): (usize, usize, usize), + alongside: Option<&mut dyn FnMut() -> Result<()>>, + ) -> Result<()> { + let width = self.copy.width() as usize; + let mut parts = batch_parts(buffer, batch_len, rows * width, components)?; + let mct = self.mct; + let destination = self.rgb8_rows(y, rows)?; + let packed = destination.is_some(); + let mct = mct.filter(|_| packed); + let finished = match alongside { + Some(alongside) => { + let mut finished = Ok(()); + let mut alongside_result = Ok(()); + overlap( + || { + finished = finish_rows( + (transform, mct), + &mut parts[..components], + width, + destination, + tasks, + ); + }, + || alongside_result = alongside(), + ); + alongside_result.and(finished) + } + None => finish_rows( + (transform, mct), + &mut parts[..components], + width, + destination, + tasks, + ), + }; + let samples = rows * width * components; + if packed { + // The rows are in the destination once they are packed, whatever + // failed alongside, so they count as handed over: a tile that + // then fails must not decode them again. + self.streamed = self.streamed.saturating_add(samples); + return finished; + } + finished?; + self.write_component_rows(&parts[..components], y, rows); + self.streamed = self.streamed.saturating_add(samples); + Ok(()) + } + + /// Hand `rows` rows of each component in `parts`, from row `y`, to the + /// sink. + fn write_component_rows(&mut self, parts: &[&mut [f32]], y: u32, rows: usize) { + let (copy, region) = (self.copy, self.region); + #[expect( + clippy::cast_possible_truncation, + reason = "`rows` is one batch of the copy rectangle's rows, at most its u32 height" + )] + let component_rows: [ComponentPlane<'_>; 4] = + core::array::from_fn(|index| ComponentPlane { + samples: parts.get(index).map_or(&[][..], |part| &**part), + dimensions: (copy.width(), rows as u32), + bit_depth: self.depths[index].0, + signed: self.depths[index].1, + sampling: (1, 1), + }); + self.sink.write_rows( + copy.x0 - region.x0, + y - region.y0, + &component_rows[..parts.len()], + ); + } +} + +/// Each component's first `len` samples of its `batch_len` in `buffer`. +#[cfg(feature = "parallel")] +fn batch_parts( + buffer: &mut [f32], + batch_len: usize, + len: usize, + components: usize, +) -> Result<[&mut [f32]; 4]> { + let mut parts: [&mut [f32]; 4] = Default::default(); + for (part, chunk) in parts + .iter_mut() + .zip(buffer.chunks_exact_mut(batch_len)) + .take(components) + { + *part = chunk + .get_mut(..len) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + } + if parts[..components.min(4)] + .iter() + .any(|part| part.len() != len) + { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + Ok(parts) +} + +/// Run `spawned` on the pool while `inline` runs here, and return once both +/// have. +#[cfg(feature = "parallel")] +fn overlap(spawned: impl FnOnce() + Send, inline: impl FnOnce()) { + rayon::in_place_scope(|scope| { + scope.spawn(move |_| spawned()); + inline(); + }); +} + +/// Reconstruct the next `parts[c].len() / width` rows of each component `c`, +/// from row `first_y`, into `parts[c]`, through its own reader: on the pool, +/// one component per task. +#[cfg(feature = "parallel")] +fn reconstruct_batch( + lines: &mut [LineReconstruction<'_>], + readers: &mut [&mut (dyn SubBandRows + Send)], + parts: &mut [&mut [f32]], + first_y: u32, + width: usize, +) -> Result<()> { + if readers.len() < lines.len() { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + let mut results: [Result<()>; 4] = core::array::from_fn(|_| Ok(())); + rayon::in_place_scope(|scope| { + let mut tasks = lines + .iter_mut() + .zip(readers.iter_mut()) + .zip(parts.iter_mut()) + .zip(results.iter_mut()); + let first = tasks.next(); + for (((line, reader), part), result) in tasks { + scope.spawn(move |_| { + *result = reconstruct_rows_into(line, &mut **reader, part, first_y, width); + }); + } + if let Some((((line, reader), part), result)) = first { + *result = reconstruct_rows_into(line, &mut **reader, part, first_y, width); + } + }); + results.into_iter().collect() +} + +/// The next rows of `line`, from row `first_y`, through `rows`, into `out`, +/// `width` samples each. +#[cfg(feature = "parallel")] +fn reconstruct_rows_into( + line: &mut LineReconstruction<'_>, + rows: &mut dyn SubBandRows, + out: &mut [f32], + first_y: u32, + width: usize, +) -> Result<()> { + for (index, chunk) in out.chunks_exact_mut(width).enumerate() { + let (row_y, row) = line + .next_row(rows)? + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let expected = u32::try_from(index) + .ok() + .and_then(|index| first_y.checked_add(index)); + if Some(row_y) != expected || row.len() != width { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + chunk.copy_from_slice(row); + } + Ok(()) +} + +/// Output rows that take three 8-bit components packed as RGB8. +struct Rgb8Rows<'o> { + /// Whole output rows, `stride` bytes each, one per reconstructed row. + out: &'o mut [u8], + #[cfg(feature = "parallel")] + stride: usize, + /// Where the reconstructed samples start in each output row, in bytes. + x: usize, +} + +/// One range of a batch's rows: each component's rows and their output. +#[cfg(feature = "parallel")] +struct BatchRows<'p, 'o> { + parts: [&'p mut [f32]; 4], + rgb8: Option>, +} + +#[cfg(feature = "parallel")] +impl BatchRows<'_, '_> { + /// Transform the rows one at a time, as [`RowOutput::write`] does, then + /// pack them into the RGB8 rows if there are any; with the `mct` of + /// three components and RGB8 rows, each row finishes through + /// [`DeferredMct::pack_rgb8_row`] instead. + fn finish( + mut self, + ((tile, round_irreversible_output), mct): ((&Tile<'_>, bool), Option), + components: usize, + width: usize, + ) -> Result<()> { + let rows = self.parts[0].len().checked_div(width).unwrap_or(0); + if let (Some(mct), Some(rgb8)) = (mct, self.rgb8.as_mut()) { + let [red, green, blue, _] = &mut self.parts; + let samples = red + .chunks_exact_mut(width) + .zip(green.chunks_exact_mut(width)) + .zip(blue.chunks_exact_mut(width)); + for (line, ((red, green), blue)) in rgb8.out.chunks_exact_mut(rgb8.stride).zip(samples) + { + let pixels = line + .get_mut(rgb8.x..rgb8.x + width * 3) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + mct.pack_rgb8_row([red, green, blue], pixels, crate::color::pack_rgb8); + } + return Ok(()); + } + for row in 0..rows { + let mut outputs: [&mut [f32]; 4] = Default::default(); + for (output, part) in outputs.iter_mut().zip(self.parts.iter_mut()) { + *output = part + .get_mut(row * width..(row + 1) * width) + .unwrap_or_default(); + } + store::transform_streamed_rows( + tile, + round_irreversible_output, + &mut outputs[..components], + )?; + } + let Some(rgb8) = self.rgb8 else { + return Ok(()); + }; + let [red, green, blue, _] = &self.parts; + let samples = red + .chunks_exact(width) + .zip(green.chunks_exact(width)) + .zip(blue.chunks_exact(width)); + for (line, ((red, green), blue)) in rgb8.out.chunks_exact_mut(rgb8.stride).zip(samples) { + let pixels = line + .get_mut(rgb8.x..rgb8.x + width * 3) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + crate::color::pack_rgb8([red, green, blue], pixels); + } + Ok(()) + } +} + +/// Transform each component's rows in `parts`, `width` samples each, and +/// pack them into `rgb8` when given, in up to `tasks` ranges of rows, all but +/// the first spawned on the pool. +#[cfg(feature = "parallel")] +fn finish_rows( + transform: ((&Tile<'_>, bool), Option), + parts: &mut [&mut [f32]], + width: usize, + mut rgb8: Option>, + tasks: usize, +) -> Result<()> { + let components = parts.len(); + let rows = parts + .first() + .map_or(0, |part| part.len().checked_div(width).unwrap_or(0)); + if components == 0 || components > 4 || rows == 0 { + return Ok(()); + } + let range_rows = rows.div_ceil(tasks.clamp(1, 4)); + let mut rest: [&mut [f32]; 4] = Default::default(); + for (rest, part) in rest.iter_mut().zip(parts.iter_mut()) { + *rest = &mut **part; + } + let mut ranges: [Option>; 4] = Default::default(); + for range in &mut ranges { + let take = range_rows.min(rest[0].len() / width); + if take == 0 { + break; + } + let mut range_parts: [&mut [f32]; 4] = Default::default(); + for (part, rest) in range_parts.iter_mut().zip(rest.iter_mut()) { + let samples = core::mem::take(rest); + let (head, tail) = samples.split_at_mut((take * width).min(samples.len())); + *part = head; + *rest = tail; + } + let range_rgb8 = match rgb8.take() { + Some(Rgb8Rows { out, stride, x }) => { + let (head, tail) = out.split_at_mut((take * stride).min(out.len())); + rgb8 = Some(Rgb8Rows { + out: tail, + stride, + x, + }); + Some(Rgb8Rows { + out: head, + stride, + x, + }) + } + None => None, + }; + *range = Some(BatchRows { + parts: range_parts, + rgb8: range_rgb8, + }); + } + let mut results: [Result<()>; 4] = core::array::from_fn(|_| Ok(())); + rayon::in_place_scope(|scope| { + let mut jobs = ranges.into_iter().zip(results.iter_mut()); + let first = jobs.next(); + for (range, result) in jobs { + if let Some(range) = range { + scope.spawn(move |_| *result = range.finish(transform, components, width)); + } + } + if let Some((Some(range), result)) = first { + *result = range.finish(transform, components, width); + } + }); + results.into_iter().collect() +} + /// When a lockstep decode fails, rerun Tier-1 over whole windows one /// component at a time, in the order the planes path decodes them, and /// return the first error. If the whole windows don't fit the cap, or the /// rerun succeeds, keep the banded error. -fn whole_window_tile_error<'a>( +pub(super) fn whole_window_tile_error<'a>( tile: &Tile<'a>, header: &Header<'_>, tile_ctx: &mut TileDecodeContext, storage: &mut DecompositionStorage<'a>, banded_error: DecodeError, ) -> DecodeError { + #[cfg(all(test, feature = "parallel"))] + { + let counters = &mut tile_ctx.debug_counters; + counters.whole_window_reruns += 1; + counters.scratches_live_in_reruns += super::subband::BlockScratch::live(); + } storage.coefficient_layout = build::CoefficientLayout::Component; if build::lay_out_whole_region_coefficients(storage).is_err() { return banded_error; @@ -431,6 +934,7 @@ fn decode_bands<'a>( ) -> Result<()> { let mut bands = Vec::new(); let active_levels = push_component_bands(storage, header, component_idx, &mut bands)?; + let band_of = band_lookup(&bands)?; let (precinct_rows, index_bytes) = index_bands(storage, tile_ctx, &bands)?; tile_ctx.idwt_output.placement = crate::j2c::idwt::IdwtPlacement::Coefficients; @@ -457,6 +961,7 @@ fn decode_bands<'a>( tile_ctx: &mut *tile_ctx, storage: &mut *storage, bands, + band_of, precinct_rows, profile_enabled, tier1_us: 0, @@ -586,6 +1091,30 @@ struct SubBandBands { last_band: Option, } +/// The [`band_lookup`] entry of a sub-band id without bands. +const NO_BAND: usize = usize::MAX; + +/// For every sub-band id up to the largest in `bands`, the index of its +/// first entry there, or [`NO_BAND`]: the IDWT looks its sub-bands up +/// several times a row. +fn band_lookup(bands: &[SubBandBands]) -> Result> { + let len = bands + .iter() + .map(|bands| bands.sub_band + 1) + .max() + .unwrap_or(0); + let mut band_of = Vec::new(); + band_of + .try_reserve_exact(len) + .map_err(|_| DecodingError::HostAllocationFailed)?; + band_of.resize(len, NO_BAND); + // From the last, so that the first entry of a sub-band wins. + for (index, bands) in bands.iter().enumerate().rev() { + band_of[bands.sub_band] = index; + } + Ok(band_of) +} + struct PrecinctRow { first_y: u32, last_y: u32, @@ -640,6 +1169,8 @@ struct BandedRows<'r, 'a> { tile_ctx: &'r mut TileDecodeContext, storage: &'r mut DecompositionStorage<'a>, bands: Vec, + /// [`band_lookup`] of `bands`. + band_of: Vec, precinct_rows: Option>>, profile_enabled: bool, tier1_us: u128, @@ -647,9 +1178,10 @@ struct BandedRows<'r, 'a> { impl BandedRows<'_, '_> { fn position(&self, sub_band: usize) -> Result { - self.bands - .iter() - .position(|bands| bands.sub_band == sub_band) + self.band_of + .get(sub_band) + .copied() + .filter(|&index| index != NO_BAND) .ok_or(DecodingError::CodeBlockDecodeFailure.into()) } @@ -711,6 +1243,13 @@ impl BandedRows<'_, '_> { /// Decode the code-blocks that start at `rows.start` into the /// sub-band's coefficients, which then hold the band's stored rows. fn decode_band(&mut self, index: usize, rows: Range) -> Result<()> { + #[cfg(test)] + if let Some(left) = self.tile_ctx.fail_band_after { + self.tile_ctx.fail_band_after = left.checked_sub(1); + if left == 0 { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + } let start = profile::profile_now(self.profile_enabled); let bands = &self.bands[index]; let (sub_band, resolution, stored) = (bands.sub_band, bands.resolution, bands.stored); diff --git a/crates/j2k-native/src/j2c/decode/direct_plan.rs b/crates/j2k-native/src/j2c/decode/direct_plan.rs index 99f9d47a4..64367a503 100644 --- a/crates/j2k-native/src/j2c/decode/direct_plan.rs +++ b/crates/j2k-native/src/j2c/decode/direct_plan.rs @@ -22,7 +22,9 @@ use crate::{ mod classic; #[cfg(feature = "parallel")] -pub(super) use self::classic::classic_allocation_counts; +pub(super) use self::classic::{ + classic_allocation_counts, extend_reserved, push_reserved, walk_classic_segments, +}; pub(super) use self::classic::{ collect_classic_code_block_data, collect_referenced_classic_code_block_data, }; diff --git a/crates/j2k-native/src/j2c/decode/direct_plan/classic.rs b/crates/j2k-native/src/j2c/decode/direct_plan/classic.rs index e555c4388..74179f9ad 100644 --- a/crates/j2k-native/src/j2c/decode/direct_plan/classic.rs +++ b/crates/j2k-native/src/j2c/decode/direct_plan/classic.rs @@ -4,7 +4,7 @@ use super::{bail, DecodeAllocationBudget, DecodingError, DecompositionStorage, Result}; use crate::error::ValidationError; -use crate::j2c::build::CodeBlock; +use crate::j2c::build::{CodeBlock, Layer, Segment}; use crate::j2c::codestream::CodeBlockStyle; use crate::{try_reserve_decode_elements, J2kCodeBlockSegment}; use alloc::vec::Vec; @@ -12,7 +12,8 @@ use alloc::vec::Vec; #[derive(Clone, Copy)] pub(crate) struct ClassicAllocationCounts { pub(crate) data_bytes: usize, - segments: usize, + /// Codeword segments. + pub(crate) segments: usize, /// Segments that hold data. pub(crate) fragments: usize, /// Coding passes over all segments. @@ -47,67 +48,17 @@ pub(crate) fn collect_classic_code_block_data( collected_segments.capacity(), )?; - let mut last_segment_idx = 0u8; - let mut segment_start_offset = 0usize; - let mut segment_start_coding_pass = 0u8; - let mut coding_passes = 0u8; - let is_normal_mode = style.single_codeword_segment(); - - for layer in &storage.layers[code_block.layers.start..code_block.layers.end] { - let Some(range) = layer.segments.clone() else { - continue; - }; - - for segment in &storage.segments[range] { - if segment.idx != last_segment_idx { - validate_next_segment_index(last_segment_idx, segment.idx)?; - if coding_passes > segment_start_coding_pass - || combined_data.len() > segment_start_offset - { - collected_segments.push(classic_segment( - *style, - segment_start_offset, - combined_data.len(), - segment_start_coding_pass, - coding_passes, - )?); - } - segment_start_offset = combined_data.len(); - segment_start_coding_pass = coding_passes; - last_segment_idx = segment.idx; - } - - combined_data.extend_from_slice(segment.data); - coding_passes = coding_passes.saturating_add(segment.coding_pases); - } - } - - if coding_passes > segment_start_coding_pass || combined_data.len() > segment_start_offset { - collected_segments.push(classic_segment( - *style, - segment_start_offset, - combined_data.len(), - segment_start_coding_pass, - coding_passes, - )?); - } - - if is_normal_mode { - collected_segments.clear(); - collected_segments.push(J2kCodeBlockSegment { - data_offset: 0, - data_length: u32::try_from(combined_data.len()) - .map_err(|_| DecodingError::CodeBlockDecodeFailure)?, - start_coding_pass: 0, - end_coding_pass: coding_passes, - use_arithmetic: true, - }); - } - - if coding_passes != code_block.number_of_coding_passes { - bail!(DecodingError::CodeBlockDecodeFailure); - } - if combined_data.len() != counts.data_bytes || collected_segments.len() > counts.segments { + let (data_bytes, _, coding_passes) = walk_classic_segments( + code_block, + *style, + (&storage.layers, &storage.segments), + |fragment| extend_reserved(&mut combined_data, fragment), + |segment| push_reserved(&mut collected_segments, segment), + )?; + if coding_passes != code_block.number_of_coding_passes + || data_bytes != counts.data_bytes + || collected_segments.len() > counts.segments + { bail!(DecodingError::CodeBlockDecodeFailure); } @@ -123,7 +74,7 @@ pub(crate) fn collect_referenced_classic_code_block_data( style: &CodeBlockStyle, storage: &DecompositionStorage<'_>, budget: &mut DecodeAllocationBudget, - mut append_fragment: impl FnMut(&[u8]) -> Result<()>, + append_fragment: impl FnMut(&[u8]) -> Result<()>, ) -> Result<(usize, usize, Vec)> { let counts = classic_allocation_counts(code_block, *style, storage)?; budget.include_elements::(counts.segments)?; @@ -135,140 +86,153 @@ pub(crate) fn collect_referenced_classic_code_block_data( collected_segments.capacity(), )?; + let (data_bytes, fragment_count, coding_passes) = walk_classic_segments( + code_block, + *style, + (&storage.layers, &storage.segments), + append_fragment, + |segment| push_reserved(&mut collected_segments, segment), + )?; + if coding_passes != code_block.number_of_coding_passes + || data_bytes != counts.data_bytes + || fragment_count != counts.fragments + || collected_segments.len() > counts.segments + { + bail!(DecodingError::CodeBlockDecodeFailure); + } + + Ok((fragment_count, data_bytes, collected_segments)) +} + +/// Payload bytes, segments, data fragments and coding passes of a classic +/// block, rejecting segment indices that skip one, without allocating. +pub(crate) fn classic_allocation_counts( + code_block: &CodeBlock, + style: CodeBlockStyle, + storage: &DecompositionStorage<'_>, +) -> Result { + let mut segments = 0_usize; + let (data_bytes, fragments, coding_passes) = walk_classic_segments( + code_block, + style, + (&storage.layers, &storage.segments), + |_| Ok(()), + |_| { + segments = segments + .checked_add(1) + .ok_or(ValidationError::ImageTooLarge)?; + Ok(()) + }, + )?; + Ok(ClassicAllocationCounts { + data_bytes, + segments, + fragments, + coding_passes, + }) +} + +/// Walk the codeword segments of a classic block without allocating, as +/// [`collect_classic_code_block_data`] collects them: pass each nonempty +/// data fragment, layer by layer, to `fragment`, and each codeword segment, +/// its contributions across layers joined, to `segment`, at its offset into +/// the fragments joined. A segment that holds coding passes and no bytes is +/// kept; one that holds neither is not. A block of one codeword segment is +/// one arithmetic segment over all its passes. Returns the payload bytes and +/// data fragments, and the coding passes, rejecting segment indices that +/// skip one. +pub(crate) fn walk_classic_segments( + code_block: &CodeBlock, + style: CodeBlockStyle, + (layers, segments): (&[Layer], &[Segment<'_>]), + mut fragment: impl FnMut(&[u8]) -> Result<()>, + mut segment: impl FnMut(J2kCodeBlockSegment) -> Result<()>, +) -> Result<(usize, usize, u8)> { + let single = style.single_codeword_segment(); let mut data_bytes = 0usize; - let mut fragment_count = 0usize; + let mut fragments = 0usize; let mut last_segment_idx = 0u8; let mut segment_start_offset = 0usize; let mut segment_start_coding_pass = 0u8; let mut coding_passes = 0u8; - let is_normal_mode = style.single_codeword_segment(); - - for layer in &storage.layers[code_block.layers.start..code_block.layers.end] { - let Some(range) = layer.segments.clone() else { + let layers = layers + .get(code_block.layers.clone()) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + for layer in layers { + let Some(range) = layer.segments() else { continue; }; - - for segment in &storage.segments[range] { - if segment.idx != last_segment_idx { - validate_next_segment_index(last_segment_idx, segment.idx)?; - if coding_passes > segment_start_coding_pass || data_bytes > segment_start_offset { - collected_segments.push(classic_segment( - *style, + let records = segments + .get(range) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + for record in records { + if record.idx != last_segment_idx { + validate_next_segment_index(last_segment_idx, record.idx)?; + if !single + && (coding_passes > segment_start_coding_pass + || data_bytes > segment_start_offset) + { + segment(classic_segment( + style, segment_start_offset, data_bytes, segment_start_coding_pass, coding_passes, - )?); + )?)?; } segment_start_offset = data_bytes; segment_start_coding_pass = coding_passes; - last_segment_idx = segment.idx; + last_segment_idx = record.idx; } - - if !segment.data.is_empty() { - append_fragment(segment.data)?; - fragment_count = fragment_count + if !record.data.is_empty() { + fragment(record.data)?; + fragments = fragments .checked_add(1) .ok_or(ValidationError::ImageTooLarge)?; } data_bytes = data_bytes - .checked_add(segment.data.len()) + .checked_add(record.data.len()) .ok_or(ValidationError::ImageTooLarge)?; - coding_passes = coding_passes.saturating_add(segment.coding_pases); + coding_passes = coding_passes.saturating_add(record.coding_pases); } } - - if coding_passes > segment_start_coding_pass || data_bytes > segment_start_offset { - collected_segments.push(classic_segment( - *style, - segment_start_offset, - data_bytes, - segment_start_coding_pass, - coding_passes, - )?); - } - - if is_normal_mode { - collected_segments.clear(); - collected_segments.push(J2kCodeBlockSegment { + if single { + segment(J2kCodeBlockSegment { data_offset: 0, data_length: u32::try_from(data_bytes) .map_err(|_| DecodingError::CodeBlockDecodeFailure)?, start_coding_pass: 0, end_coding_pass: coding_passes, use_arithmetic: true, - }); + })?; + } else if coding_passes > segment_start_coding_pass || data_bytes > segment_start_offset { + segment(classic_segment( + style, + segment_start_offset, + data_bytes, + segment_start_coding_pass, + coding_passes, + )?)?; } + Ok((data_bytes, fragments, coding_passes)) +} - if coding_passes != code_block.number_of_coding_passes - || data_bytes != counts.data_bytes - || fragment_count != counts.fragments - || collected_segments.len() > counts.segments - { +/// Append `data` to `values` within the capacity reserved for it. +pub(crate) fn extend_reserved(values: &mut Vec, data: &[u8]) -> Result<()> { + if values.capacity() - values.len() < data.len() { bail!(DecodingError::CodeBlockDecodeFailure); } - - Ok((fragment_count, data_bytes, collected_segments)) + values.extend_from_slice(data); + Ok(()) } -/// Payload bytes, segments, data fragments and coding passes of a classic -/// block, rejecting segment indices that skip one, without allocating. -pub(crate) fn classic_allocation_counts( - code_block: &CodeBlock, - style: CodeBlockStyle, - storage: &DecompositionStorage<'_>, -) -> Result { - let mut data_bytes = 0_usize; - let mut segment_count = 0_usize; - let mut last_segment_idx = 0_u8; - let mut segment_start_offset = 0_usize; - let mut segment_start_coding_pass = 0_u8; - let mut coding_passes = 0_u8; - let mut fragment_count = 0_usize; - - for layer in &storage.layers[code_block.layers.start..code_block.layers.end] { - let Some(range) = layer.segments.clone() else { - continue; - }; - for segment in &storage.segments[range] { - if segment.idx != last_segment_idx { - validate_next_segment_index(last_segment_idx, segment.idx)?; - if coding_passes > segment_start_coding_pass || data_bytes > segment_start_offset { - segment_count = segment_count - .checked_add(1) - .ok_or(ValidationError::ImageTooLarge)?; - } - segment_start_offset = data_bytes; - segment_start_coding_pass = coding_passes; - last_segment_idx = segment.idx; - } - data_bytes = data_bytes - .checked_add(segment.data.len()) - .ok_or(ValidationError::ImageTooLarge)?; - if !segment.data.is_empty() { - fragment_count = fragment_count - .checked_add(1) - .ok_or(ValidationError::ImageTooLarge)?; - } - coding_passes = coding_passes.saturating_add(segment.coding_pases); - } - } - - if coding_passes > segment_start_coding_pass || data_bytes > segment_start_offset { - segment_count = segment_count - .checked_add(1) - .ok_or(ValidationError::ImageTooLarge)?; - } - if style.single_codeword_segment() { - segment_count = 1; +/// Push `value` onto `values` within the capacity reserved for it. +pub(crate) fn push_reserved(values: &mut Vec, value: T) -> Result<()> { + if values.len() == values.capacity() { + bail!(DecodingError::CodeBlockDecodeFailure); } - - Ok(ClassicAllocationCounts { - data_bytes, - segments: segment_count, - fragments: fragment_count, - coding_passes, - }) + values.push(value); + Ok(()) } fn validate_next_segment_index(previous: u8, current: u8) -> Result<()> { diff --git a/crates/j2k-native/src/j2c/decode/parallel_bands.rs b/crates/j2k-native/src/j2c/decode/parallel_bands.rs index 0be196c26..d300c8d70 100644 --- a/crates/j2k-native/src/j2c/decode/parallel_bands.rs +++ b/crates/j2k-native/src/j2c/decode/parallel_bands.rs @@ -17,12 +17,22 @@ //! - The IDWT waits only for a band a worker is decoding. A band nobody has //! started it decodes itself, so it never depends on a pool thread being //! free, even when the pool is busy with other work. +//! - When the finest level's slots ahead of the IDWT hold fewer bands than +//! the pool has threads, as with one component, a classic band splits into +//! parts, runs of its blocks that threads decode at once into the band's +//! slot, so every thread has work without holding more bands. //! //! Every band is decoded once, including bands the IDWT never reads, so a //! decode visits, decodes and rejects the code-blocks the whole-window route //! does. Workers fail in any order, so a failure reruns whole-window Tier-1 //! serially for the error the whole-window path reports. //! +//! A region decode that streams its rows to a sink runs every component's +//! bands on one executor instead, while the IDWT advances the components +//! together, a row of each at a time, as the serial bands of +//! `banded_tier1::decode_tile_rows` do. The pool then holds every +//! component's next bands at once, and the region needs no planes. +//! //! A running band always belongs to a worker that is still active. A wait //! that finds a running band with no worker active would otherwise last //! forever, so it gives up the pool instead, and the component decodes over @@ -35,19 +45,25 @@ use core::mem::size_of; use core::ops::Range; use std::sync::{Condvar, Mutex, MutexGuard}; -use super::banded_tier1; -use super::subband::{plan_band_route, routes_tile, BandRoute, BandRouting}; +use super::banded_tier1::{self, RowOutput}; +use super::subband::{ + plan_band_route, plan_band_route_beside, pool_entry_bytes, routes_tile, BandOutput, BandRoute, + BandRouting, BlockScratch, PayloadSource, LOCK_BYTES, +}; use super::workspace::{TaskScratch, TaskWorkspace}; use super::{ - clips_to_idwt_windows, CpuDecodeParallelism, DecodeProfileTimings, DecompositionStorage, - Header, Result, Tile, TileDecodeContext, + clips_to_idwt_windows, record_workspace, CpuDecodeParallelism, DecodeAllocationBudget, + DecodeProfileTimings, DecompositionStorage, Header, Result, Tile, TileDecodeContext, }; use crate::error::{DecodeError, DecodingError, ValidationError}; use crate::j2c::build::{self, CodeBlockCoding}; use crate::j2c::codestream::WaveletTransform; -use crate::j2c::idwt::{ComponentLines, SubBandRows}; +use crate::j2c::idwt::{ + streamed_batch_rows, ComponentLines, LinePlan, LineReconstruction, SubBandRows, +}; use crate::j2c::rect::IntRect; use crate::profile; +use crate::ComponentRowSink; /// Sub-band rows of support on each side of the rows a level's lifting /// reads, in rows of that level, for ordering bands: a 9/7 lift reads two @@ -102,7 +118,7 @@ pub(super) fn decode_component<'a>( profile_timings: &mut DecodeProfileTimings, ) -> Result { let stage_start = profile::profile_now(profile_enabled); - let routing = plan_band_route(tile, component_idx, tile_ctx, storage, header); + let routing = plan_band_route(tile, component_idx, tile_ctx, storage, header, false); let route = match routing { // Planning can fail in a later band before earlier blocks decode. Err(error) => { @@ -148,7 +164,7 @@ pub(super) fn decode_component<'a>( &mut tier1_us, ) .and_then(|completed| { - if completed { + if completed == Some(true) { route.record(tile_ctx, storage)?; } Ok(completed) @@ -159,11 +175,26 @@ pub(super) fn decode_component<'a>( profile_timings.codeblock_us += tier1_us; profile_timings.idwt_us += elapsed.saturating_sub(tier1_us); match result { - Ok(true) => { + Ok(Some(true)) => { tile_ctx.debug_counters.pool_band_components += 1; Ok(true) } - Ok(false) => { + Ok(None) => { + // The executor did not fit beside the plan, before anything + // decoded; the serial bands need neither. + banded_tier1::decode_component( + tile, + component_idx, + header, + tile_ctx, + storage, + transform, + profile_enabled, + profile_timings, + )?; + Ok(true) + } + Ok(Some(false)) => { // A band was left running with no worker to finish it. #[cfg(test)] let simulated = tile_ctx.strand_pool_band; @@ -186,6 +217,11 @@ pub(super) fn decode_component<'a>( } } +/// Decode the bands of `route`'s component on the pool while the streamed +/// IDWT reads them. Returns `None`, before anything decodes, when the +/// rings and bands, the executor's records or its threads' scratch do not +/// fit beside the route's plan, and `Some(false)` when a band was left +/// running with no worker to finish it. #[expect( clippy::too_many_arguments, reason = "the banded stage shares decode_tile's state and profiling explicitly" @@ -199,13 +235,40 @@ fn decode_route( transform: WaveletTransform, profile_enabled: bool, tier1_us: &mut u128, -) -> Result { - let (rings, bands, active_levels) = plan_rings(route, storage, header, component_idx)?; - // The executor's records live beside the route's plan and scratch while - // the bands decode. - route - .budget() - .include_bytes(executor_bytes(&rings, &bands)?)?; +) -> Result> { + // The rings and bands live beside the route's plan and scratch while + // the bands decode; planning them holds their row scales beside them. + let mut budget = route.budget(); + let (planned, scales) = ring_plan_bytes(route, storage, header, component_idx)?; + let mut planning = budget; + if declines(planning.include_bytes(planned + scales))? + || declines(budget.include_bytes(planned))? + { + return Ok(None); + } + let (rings, mut bands, active_levels) = plan_rings(route, storage, header, component_idx)?; + #[cfg(test)] + count_ring_slots(tile_ctx, &rings); + if route.coding() == CodeBlockCoding::Classic { + split_bands( + tile_ctx, + storage.shared_pool_band_rings, + &rings, + &mut bands, + route.workers() + 1, + |band, parts| route.strip_parts(band.strip, parts), + )?; + } + // So do the executor's records and each thread's scratch, the route's + // workers' and this thread's. + if declines(budget.include_bytes(executor_bytes(&rings, &bands, (route.workers(), 1))?))? { + return Ok(None); + } + let mut scratches = + match BlockScratch::for_threads(route.workers() + 1, route.scratch_sizes()?, &mut budget) { + Err(error) if super::reuse::is_capacity_error(&error) => return Ok(None), + scratches => scratches?, + }; tile_ctx.idwt_output.placement = crate::j2c::idwt::IdwtPlacement::Coefficients; tile_ctx.idwt_output.coefficients = Vec::new(); tile_ctx.idwt_output.rect = IntRect::from_xywh(0, 0, 0, 0); @@ -217,44 +280,45 @@ fn decode_route( header, transform, )?; - #[cfg(test)] - let strand_once = tile_ctx.strand_pool_band; - #[cfg(not(test))] - let strand_once = false; + let strand = test_strand(tile_ctx); let mut coefficients = core::mem::take(&mut storage.coefficients); let slots = split_slots(&rings, &bands, storage, &mut coefficients); + let source = PayloadSource::new(storage); let result = slots.and_then(|slots| { - let run = |rows: &mut dyn SubBandRows| -> Result<()> { - lines.as_mut().map_or(Ok(()), |lines| lines.run(rows)) + let run = |rows: &mut [&mut (dyn SubBandRows + Send)]| -> Result<()> { + match (lines.as_mut(), rows.first_mut()) { + (Some(lines), Some(rows)) => lines.run(&mut **rows), + _ => Ok(()), + } }; match route.coding() { CodeBlockCoding::Classic => run_bands( - route, + (route.workers(), 1, false), &rings, &bands, slots, - &mut tile_ctx.classic_task_workspaces, + (&mut tile_ctx.classic_task_workspaces, &mut scratches), |index, output, workspace, scratch| { - route.decode_classic(index, output, workspace, scratch) + route.decode_classic(index, output, workspace, (scratch, source)) }, run, profile_enabled, tier1_us, - strand_once, + strand, ), CodeBlockCoding::HighThroughput => run_bands( - route, + (route.workers(), 1, false), &rings, &bands, slots, - &mut tile_ctx.ht_task_workspaces, + (&mut tile_ctx.ht_task_workspaces, &mut scratches), |index, output, workspace, scratch| { - route.decode_ht(index, output, workspace, scratch) + route.decode_ht(index, output, workspace, (scratch, source)) }, run, profile_enabled, tier1_us, - strand_once, + strand, ), } }); @@ -263,41 +327,760 @@ fn decode_route( // theirs written. storage.stale_coefficients = true; let Some(lines) = lines else { - return result; + return result.map(Some); }; match result { - Ok(true) => lines.finish(tile_ctx, Ok(())).map(|()| true), + Ok(true) => lines.finish(tile_ctx, Ok(())).map(|()| Some(true)), Ok(false) => { // Return the rings and plane; the whole-window path refills them. match lines.finish(tile_ctx, Err(DecodingError::CodeBlockDecodeFailure.into())) { - Ok(()) | Err(_) => Ok(false), + Ok(()) | Err(_) => Ok(Some(false)), } } - Err(error) => lines.finish(tile_ctx, Err(error)).map(|()| true), + Err(error) => lines.finish(tile_ctx, Err(error)).map(|()| Some(true)), } } -/// The bytes the executor holds for `rings` and `bands`, besides the slots: -/// the records, each band's status, slot, queue entry and decoded-list entry, -/// and each ring's cursors and slot list. -fn executor_bytes(rings: &[Ring], bands: &[Band]) -> Result { - let per_band = size_of::() - + size_of::() - + size_of::() - + 2 * size_of::>() - + size_of::, usize)>>(); - let per_ring = size_of::() +/// Whether `result` failed for want of budget, which declines the pool's +/// bands; other errors pass on. +fn declines(result: Result<()>) -> Result { + match result { + Err(error) if super::reuse::is_capacity_error(&error) => Ok(true), + result => result.map(|()| false), + } +} + +/// Decode every component of a tile whose rows stream to a sink: the bands +/// of all its components on the pool at once, while the streamed IDWT +/// advances the components together on this thread and hands their rows +/// over, as `banded_tier1::decode_tile_rows` does with serial bands. +/// Returns `false`, before anything is decoded or streamed, when the route +/// declines a component, or when a band was stranded before any row was +/// streamed; the tile's bands then decode serially. +#[expect( + clippy::too_many_lines, + reason = "the streamed tile keeps its plans, executor, rings, and failure handling together" +)] +pub(super) fn decode_tile_rows<'a>( + tile: &Tile<'a>, + header: &Header<'_>, + tile_ctx: &mut TileDecodeContext, + storage: &mut DecompositionStorage<'a>, + sink: &mut dyn ComponentRowSink, + profile_enabled: bool, + profile_timings: &mut DecodeProfileTimings, +) -> Result { + let stage_start = profile::profile_now(profile_enabled); + let components = tile.component_infos.len(); + // Every component's route stays live beside the others' while the + // executor runs them all, so each is planned in the budget the last one + // left, as one budget, which starts with the list of routes. + // A budget too small for the pool leaves the tile to serial bands. + let mut live = DecodeAllocationBudget::for_storage(storage)?; + let mut routes = Vec::new(); + if declines(live.reserve_new(&mut routes, components))? { + return Ok(false); + } + for component_idx in 0..components { + match plan_band_route_beside( + tile, + component_idx, + tile_ctx, + storage, + header, + Some(live), + true, + ) { + Ok(BandRouting::Route(route)) => { + live = route.budget(); + routes.push(route); + } + Ok(BandRouting::Serial | BandRouting::WholeWindows) => return Ok(false), + // Planning can fail in a later band before earlier blocks decode. + Err(error) => { + // The whole-window rerun's budget leaves the plans out. + drop(routes); + return Err(banded_tier1::whole_window_tile_error( + tile, header, tile_ctx, storage, error, + )); + } + } + } + // Every component's rings and bands live beside the plans while the + // bands decode; planning each holds its own rings, bands and row scales + // beside them. + let (planned, planning) = tile_ring_plan_bytes(&routes, storage, header)?; + let mut peak = live; + if declines(peak.include_bytes(planned + planning))? || declines(live.include_bytes(planned))? { + return Ok(false); + } + let (rings, mut bands, strip_starts, active_levels) = + plan_tile_rings(&routes, storage, header)?; + // So do the streamed IDWT's plans, its rings' lengths and its + // reconstructions, one of each per component. + let line_lists = components + .checked_mul( + size_of::() + size_of::() + size_of::>(), + ) + .ok_or(ValidationError::ImageTooLarge)?; + if declines(live.include_bytes(line_lists))? { + return Ok(false); + } + let line_plans = banded_tier1::plan_tile_lines( + tile, + header, + tile_ctx, + storage, + active_levels.iter().copied().map(Ok), + )?; + let empty = IntRect::from_xywh(0, 0, 0, 0); + let (copy, region) = line_plans + .first() + .map_or((empty, empty), |plan| (plan.copy, plan.region)); + // Every route plans the tile's one block coding (see `routes_tile`). + let coding = routes + .first() + .map_or(CodeBlockCoding::Classic, |route| route.coding()); + // Each component's reconstruction reads through its own view of the + // bands, with a workspace of its own from the bank the routes prepared + // for their workers and the calling thread. + let bank = match coding { + CodeBlockCoding::Classic => tile_ctx.classic_task_workspaces.len(), + CodeBlockCoding::HighThroughput => tile_ctx.ht_task_workspaces.len(), + }; + let route_workers = routes + .iter() + .map(|route| route.workers()) + .max() + .unwrap_or(0); + let samples = (copy.width() as usize).saturating_mul(copy.height() as usize); + let readers = if parallel_synthesis(components, coding, bank, route_workers, samples) { + components + } else { + 1 + }; + let workers = route_workers.min(bank.saturating_sub(readers)); + #[cfg(test)] + count_ring_slots(tile_ctx, &rings); + if coding == CodeBlockCoding::Classic { + split_bands( + tile_ctx, + storage.shared_pool_band_rings, + &rings, + &mut bands, + workers + readers, + |band, parts| { + let (route, strip) = route_strip(&routes, &strip_starts, band.strip)?; + route.strip_parts(strip, parts) + }, + )?; + } + // The executor's records live beside every route's plan and scratch + // while the bands decode. + if declines(live.include_bytes(executor_bytes(&rings, &bands, (workers, readers))?))? { + return Ok(false); + } + // Each thread's scratch serves every component's bands, so it holds, + // and is charged once for, the largest of the routes'; the serial bands + // need none. + let threads = workers + readers; + #[cfg(test)] + let unscratched = live.live_bytes(); + let Some(mut scratches) = BlockScratch::charged_for_threads( + threads, + BandRoute::shared_scratch(routes.iter().map(|route| &**route))?, + &mut live, + )? + else { + return Ok(false); + }; + #[cfg(test)] + { + tile_ctx.debug_counters.thread_scratch_charged += live.live_bytes() - unscratched; + } + // A reader per component reconstructs its rows in batches, as many as + // fit rows of the region's width but never more than the tile's width is + // charged for (see `roi_idwt_peak_elements`): two of them for an HTJ2K + // tile, so one finishes while the next reconstructs; a classic + // tile's readers wait on Tier-1 instead, and hold one batch of half the + // rows, with deep band rings when they read whole rows. One reader + // finishes each row in the IDWT's ring, as serial bands do. + let pipelined = readers > 1 && coding == CodeBlockCoding::HighThroughput; + let batch_rows = + |width: u32| streamed_batch_rows(width as usize, copy.height() as usize, !pipelined); + let batch = if readers > 1 { + batch_rows(region.width()).min(batch_rows(copy.width())) + } else { + 0 + }; + let scratch_len = (copy.width() as usize) + .checked_mul(components) + .and_then(|samples| samples.checked_mul(batch)) + .and_then(|samples| samples.checked_mul(if pipelined { 2 } else { 1 })) + .ok_or(ValidationError::ImageTooLarge)?; + let (mut buffer, ring_lens) = + banded_tier1::take_line_buffer(tile_ctx, storage, &line_plans, scratch_len)?; + #[cfg(test)] + { + let held = ring_lens + .iter() + .fold(scratch_len, |held, len| held.saturating_add(*len)); + let counters = &mut tile_ctx.debug_counters; + counters.pool_band_readers = counters.pool_band_readers.max(readers); + counters.line_batch_samples = counters.line_batch_samples.max(scratch_len); + counters.uncharged_line_samples = counters + .uncharged_line_samples + .max(held.saturating_sub(storage.roi_idwt_charge_elements)); + } + let mut output = RowOutput::new(tile, sink, tile_ctx, copy, region); + // One reader for every component of a classic tile leaves the pool's + // threads to Tier-1, and decodes bands itself while it waits. + let helps = readers == 1 && coding == CodeBlockCoding::Classic; + let strand = test_strand(tile_ctx); + #[cfg(test)] + let fail_after = test_countdown(tile_ctx.fail_band_after); + #[cfg(test)] + let failed_band = || test_countdown_ends(&fail_after); + #[cfg(not(test))] + let failed_band = || false; + let mut tier1_us = 0; + let mut output_samples = 0usize; + let mut coefficients = core::mem::take(&mut storage.coefficients); + let slots = split_slots(&rings, &bands, storage, &mut coefficients); + let source = PayloadSource::new(storage); + let result = slots.and_then(|slots| { + let rings_len = buffer.len() - scratch_len; + let (line_rings, scratch) = buffer.split_at_mut(rings_len); + let mut lines = banded_tier1::line_reconstructions(&line_plans, &ring_lens, line_rings)?; + let run = |readers: &mut [&mut (dyn SubBandRows + Send)]| -> Result<()> { + let written = match readers { + [reader] => output.write(&mut lines, &mut **reader), + readers => output.write_batches(&mut lines, readers, scratch, batch), + }; + output_samples = lines.iter().map(LineReconstruction::output_samples).sum(); + written + }; + match coding { + CodeBlockCoding::Classic => run_bands( + (workers, readers, helps), + &rings, + &bands, + slots, + (&mut tile_ctx.classic_task_workspaces, &mut scratches), + |strip, output, workspace, scratch| { + if failed_band() { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + let (route, strip) = route_strip(&routes, &strip_starts, strip)?; + route.decode_classic(strip, output, workspace, (scratch, source)) + }, + run, + profile_enabled, + &mut tier1_us, + strand, + ), + CodeBlockCoding::HighThroughput => run_bands( + (workers, readers, helps), + &rings, + &bands, + slots, + (&mut tile_ctx.ht_task_workspaces, &mut scratches), + |strip, output, workspace, scratch| { + if failed_band() { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + let (route, strip) = route_strip(&routes, &strip_starts, strip)?; + route.decode_ht(strip, output, workspace, (scratch, source)) + }, + run, + profile_enabled, + &mut tier1_us, + strand, + ), + } + }); + storage.coefficients = coefficients; + // Each band zeroes its slot before it decodes, and the last bands leave + // theirs written. + storage.stale_coefficients = true; + tile_ctx.idwt_line_rings = buffer; + #[cfg(test)] + { + tile_ctx.fail_band_after = usize::try_from(fail_after.into_inner()).ok(); + let counters = &mut tile_ctx.debug_counters; + counters.thread_scratch_held += BlockScratch::held_bytes(&scratches); + counters.thread_block_samples = counters + .thread_block_samples + .max(BlockScratch::block_samples(&scratches)); + } + let streamed = output.streamed; + tile_ctx.streamed_samples = tile_ctx.streamed_samples.saturating_add(streamed); + tile_ctx.debug_counters.idwt_output_samples = tile_ctx + .debug_counters + .idwt_output_samples + .saturating_add(output_samples); + let elapsed = profile::elapsed_us(stage_start); + profile_timings.codeblock_us += tier1_us; + profile_timings.idwt_us += elapsed.saturating_sub(tier1_us); + let result = result.and_then(|completed| { + if completed { + BandRoute::record_live( + routes.iter().map(|route| &**route), + threads, + tile_ctx, + storage, + )?; + } + Ok(completed) + }); + // Neither the serial bands' nor the whole-window rerun's budget includes + // the plans, the executor's records or the threads' scratch, so they go + // first. + drop(( + routes, + rings, + bands, + strip_starts, + active_levels, + line_plans, + ring_lens, + scratches, + )); + match result { + Ok(true) => { + tile_ctx.debug_counters.pool_band_components += components; + record_workspace(tile_ctx, storage, 0)?; + Ok(true) + } + Ok(false) => { + // A band was left running with no worker to finish it. + #[cfg(test)] + let simulated = tile_ctx.strand_pool_band; + #[cfg(not(test))] + let simulated = false; + debug_assert!(simulated, "a running band had no active worker"); + tile_ctx.debug_counters.pool_band_fallbacks += 1; + // Rows the sink holds can't be taken back. + if streamed > 0 { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + Ok(false) + } + Err(error) => Err(banded_tier1::whole_window_tile_error( + tile, header, tile_ctx, storage, error, + )), + } +} + +/// Whether a tile streaming its rows from pool bands may hold deeper band +/// rings (see `DecompositionStorage::deep_band_rings`): a classic tile of +/// several components on a pool of [`MIN_CLASSIC_SYNTHESIS_THREADS`] threads +/// or more, which reconstructs each on its own reader (see +/// [`parallel_synthesis`]). Its readers wait on classic Tier-1, which more +/// bands in flight keep busier when it reads whole rows (see +/// [`reads_whole_rows`]); it holds one batch of rows instead of two. HTJ2K +/// bands keep up with the shallower rings, and its readers gain more from +/// the second batch. +pub(super) fn deep_rings(tile: &Tile<'_>) -> bool { + tile.component_infos.len() > 1 + && rayon::current_num_threads() >= MIN_CLASSIC_SYNTHESIS_THREADS + && tile.component_infos.iter().all(|component| { + !component + .coding_style + .parameters + .code_block_style + .uses_high_throughput_block_coding() + }) +} + +/// Whether a tile reads whole rows at full resolution: no resolution level +/// skipped, and full-width windows (see [`reads_full_width`]). Only such +/// tiles hold deep band rings: a reduced or narrower read decodes as fast +/// without the extra band in every ring. +pub(super) fn reads_whole_rows(header: &Header<'_>, storage: &DecompositionStorage<'_>) -> bool { + header.skipped_resolution_levels == 0 && reads_full_width(storage) +} + +/// Whether every sub-band window a tile's IDWT reads spans its sub-band. +fn reads_full_width(storage: &DecompositionStorage<'_>) -> bool { + storage.roi_plan.as_ref().is_some_and(|plan| { + storage + .sub_bands + .iter() + .enumerate() + .all(|(index, sub_band)| { + plan.sub_band_window(index) + .is_none_or(|window| window.width() == sub_band.rect.width()) + }) + }) +} + +/// Whether a tile streaming its rows from pool bands holds one band fewer in +/// each ring of its finest level (see +/// `DecompositionStorage::shallow_band_rings`): a classic tile of several +/// components that reconstructs each on its own reader without deep band +/// rings, as on 10 and 11 threads or at a reduced resolution, reading +/// full-width windows, on a pool with more threads than the tile has +/// finest-level sub-bands. Its readers wait on classic Tier-1. With two slots +/// the bands ahead of them are then fewer than the pool's threads, so the +/// bands split into parts (see [`plan_parts`]) that every thread takes, which +/// keeps the pool as busy as a band more in every ring did. Narrower regions +/// keep their rings. +pub(super) fn shallow_rings(tile: &Tile<'_>, storage: &DecompositionStorage<'_>) -> bool { + let components = tile.component_infos.len(); + storage.parallel_bands + && storage.lockstep_lines + && !storage.deep_band_rings + && !storage.one_synthesis_reader + && components > 1 + && 3 * components < rayon::current_num_threads() + && tile.component_infos.iter().all(|component| { + !component + .coding_style + .parameters + .code_block_style + .uses_high_throughput_block_coding() + }) + && reads_full_width(storage) +} + +/// Whether a tile streaming its rows from pool bands holds the shared rings +/// (see `DecompositionStorage::shared_pool_band_rings`): its decode shares +/// the pool with other images (see `TileDecodeContext::shared_pool`), on +/// fewer than [`MIN_CLASSIC_SYNTHESIS_THREADS`] threads. From there a lone +/// classic tile holds deep rings, and two bands a ring left a batch filling +/// an 18-thread pool 1-2% slower. +pub(super) fn shared_rings( + tile_ctx: &TileDecodeContext, + storage: &DecompositionStorage<'_>, +) -> bool { + storage.parallel_bands + && tile_ctx.shared_pool + && rayon::current_num_threads() < MIN_CLASSIC_SYNTHESIS_THREADS +} + +/// Whether a tile streaming its rows from pool bands reconstructs them +/// through one reader whatever workers its routes plan (see +/// [`parallel_synthesis`]): it has one component, or classic code-blocks on +/// too few pool threads for its size to reconstruct each component on its +/// own (see [`reader_per_component`]). The size is the largest final window +/// of a component, which holds the rows [`parallel_synthesis`] counts. +pub(super) fn one_reader(tile: &Tile<'_>, storage: &DecompositionStorage<'_>) -> bool { + let components = tile.component_infos.len(); + let classic = tile.component_infos.iter().all(|component| { + !component + .coding_style + .parameters + .code_block_style + .uses_high_throughput_block_coding() + }); + let coding = if classic { + CodeBlockCoding::Classic + } else { + CodeBlockCoding::HighThroughput + }; + let samples = storage.roi_plan.as_ref().map_or(usize::MAX, |plan| { + (0..components) + .filter_map(|component_idx| plan.final_window(component_idx)) + .map(|window| (window.width() as usize).saturating_mul(window.height() as usize)) + .max() + .unwrap_or(0) + }); + !reader_per_component(components, coding, rayon::current_num_threads(), samples) +} + +/// Pool threads from which a classic tile reconstructs each component on its +/// own reader whatever its size, and its band rings are deep. +const MIN_CLASSIC_SYNTHESIS_THREADS: usize = 12; + +/// Pool threads from which a classic tile of [`MIN_CLASSIC_READER_SAMPLES`] +/// or more reconstructs each component on its own reader below +/// [`MIN_CLASSIC_SYNTHESIS_THREADS`]. A reader waiting for a band holds its +/// thread, and classic Tier-1 keeps readers waiting, so a smaller pool does +/// better with one reader that decodes bands while it waits (4-7% on 8 +/// threads). From 10 threads that one reader, which reconstructs every +/// component and packs every row alone, bounds a large tile: a reader per +/// component decodes 18000 x 12000 RGB 11-19% and 5661 x 8287 RGB 8-13% +/// faster on 10 and 11 threads. +const MIN_CLASSIC_READER_THREADS: usize = 10; + +/// Samples of each component from which a classic tile takes a reader per +/// component on [`MIN_CLASSIC_READER_THREADS`] threads. Smaller tiles spend +/// more of their time in Tier-1, from which the readers take two pool +/// threads: on 10 threads a 4096 x 2048 lossless RGB image and the 2830 x +/// 4143 to 708 x 1036 reduced decodes of a 5661 x 8287 one were 1-11% slower +/// with them. +#[cfg(not(test))] +const MIN_CLASSIC_READER_SAMPLES: usize = 1 << 24; +/// The unit tests' fixtures are far below the size, so their gate lies +/// between a 203 x 157 component and its 102 x 79 half. +#[cfg(test)] +const MIN_CLASSIC_READER_SAMPLES: usize = 1 << 14; + +/// Whether a streamed tile's `components` reconstruct on the pool, one per +/// task, given its block coding, a workspace bank of `bank`, routes planned +/// for `route_workers` pool threads and `samples` reconstructed per component. +fn parallel_synthesis( + components: usize, + coding: CodeBlockCoding, + bank: usize, + route_workers: usize, + samples: usize, +) -> bool { + route_workers >= 2 + && bank > components + && reader_per_component(components, coding, rayon::current_num_threads(), samples) +} + +/// Whether a streamed tile of `components` of `coding` blocks may reconstruct +/// each component on its own reader on a pool of `threads`, with `samples` +/// reconstructed per component: HTJ2K tiles on any pool, classic ones from +/// [`MIN_CLASSIC_SYNTHESIS_THREADS`], or from [`MIN_CLASSIC_READER_THREADS`] +/// at [`MIN_CLASSIC_READER_SAMPLES`] or more. [`parallel_synthesis`] decides +/// by it and [`one_reader`] charges by it, so a tile charged for one reader +/// never runs several. +fn reader_per_component( + components: usize, + coding: CodeBlockCoding, + threads: usize, + samples: usize, +) -> bool { + components > 1 + && (coding == CodeBlockCoding::HighThroughput + || threads >= MIN_CLASSIC_SYNTHESIS_THREADS + || (threads >= MIN_CLASSIC_READER_THREADS && samples >= MIN_CLASSIC_READER_SAMPLES)) +} + +/// The route of a band's `strip`, counted over every component's route from +/// `strip_starts`, and the strip in that route. +fn route_strip<'r, 'a>( + routes: &'r [Box>], + strip_starts: &[usize], + strip: usize, +) -> Result<(&'r BandRoute<'a>, usize)> { + let component = strip_starts + .partition_point(|start| *start <= strip) + .checked_sub(1) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let route = routes + .get(component) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + Ok((route, strip - strip_starts[component])) +} + +/// Every component's rings and bands, in component order, as one executor +/// takes them, with the first strip of each component's route among the +/// strips of all routes, and the levels each component reconstructs. +type TileRings = (Vec, Vec, Vec, Vec); + +fn plan_tile_rings( + routes: &[Box>], + storage: &DecompositionStorage<'_>, + header: &Header<'_>, +) -> Result { + let (mut rings, mut bands) = (Vec::new(), Vec::new()); + let (mut ring_count, mut band_count) = (0usize, 0usize); + for (component_idx, route) in routes.iter().enumerate() { + let (component_rings, component_bands) = + ring_counts(route, storage, header, component_idx)?; + ring_count = ring_count.saturating_add(component_rings); + band_count = band_count.saturating_add(component_bands); + } + rings + .try_reserve_exact(ring_count) + .map_err(|_| DecodingError::HostAllocationFailed)?; + bands + .try_reserve_exact(band_count) + .map_err(|_| DecodingError::HostAllocationFailed)?; + let mut strip_starts = Vec::new(); + strip_starts + .try_reserve_exact(routes.len()) + .map_err(|_| DecodingError::HostAllocationFailed)?; + let mut active_levels = Vec::new(); + active_levels + .try_reserve_exact(routes.len()) + .map_err(|_| DecodingError::HostAllocationFailed)?; + let mut strips = 0usize; + for (component_idx, route) in routes.iter().enumerate() { + let (component_rings, component_bands, levels) = + plan_rings(route, storage, header, component_idx)?; + let (ring_offset, band_offset) = (rings.len(), bands.len()); + if component_rings.len() > rings.capacity() - ring_offset + || component_bands.len() > bands.capacity() - band_offset + { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + rings.extend(component_rings.into_iter().map(|ring| Ring { + bands: ring.bands.start + band_offset..ring.bands.end + band_offset, + ..ring + })); + bands.extend(component_bands.into_iter().map(|band| Band { + strip: band.strip + strips, + ring: band.ring + ring_offset, + ..band + })); + strip_starts.push(strips); + strips += route.strips().count(); + active_levels.push(levels); + } + Ok((rings, bands, strip_starts, active_levels)) +} + +/// The bytes the executor holds for `rings` and `bands` while `workers` +/// pool threads and `readers` readers decode them, besides the rings and +/// bands themselves (see [`ring_plan_bytes`]) and the slots' samples: each +/// band's status, held and owned slot, part counters, decoded-list entry +/// and shared output; a queue entry for each part of each band, as many as +/// its parts put in it; each ring's cursors, slot list and slots; each +/// reader's view, with its band being read and next band in every ring, and +/// its entry in the readers' list; each thread's workspace and scratch +/// pair; the pool's job for each worker running or queued, and for each +/// pool thread whose worker finished before the pool freed its job; what +/// the shared output of each band decoding in parts, the shared state's +/// lock and its condition variable allocate when first used; and what the +/// pool takes when entered from outside it. +fn executor_bytes( + rings: &[Ring], + bands: &[Band], + (workers, readers): (usize, usize), +) -> Result { + let per_band = size_of::() + 3 * size_of::() - + size_of::>() - + size_of::>(); + + 2 * size_of::>() + + size_of::>(); + let per_ring = 2 * size_of::() + size_of::>(); + // A view holds references to its workspace, scratch and decode, + // whatever their types. + let per_reader = rings + .len() + .checked_mul(size_of::() + size_of::>()) + .ok_or(ValidationError::ImageTooLarge)? + + size_of::>() + + size_of::<&mut (dyn SubBandRows + Send)>(); + let per_thread = size_of::<(&mut (), &mut BlockScratch)>(); let slots: usize = rings.iter().map(|ring| ring.slots).sum(); + [ + bands.len().checked_mul(per_band), + queue_entries(bands).checked_mul(size_of::()), + rings.len().checked_mul(per_ring), + slots.checked_mul(size_of::<&mut [f32]>()), + readers.checked_mul(per_reader), + workers + .checked_add(readers) + .and_then(|threads| threads.checked_mul(per_thread)), + workers + .checked_add(rayon::current_num_threads()) + .and_then(|jobs| jobs.checked_mul(SENT_WORKER_BYTES)), + // Called from outside the pool, its workers queue on the pool's + // shared queue, and this thread waits for them to finish. + Some(pool_entry_bytes()), + bands + .iter() + .filter(|band| band.parts > 1) + .count() + .checked_add(2) + .and_then(|locks| locks.checked_mul(LOCK_BYTES)), + ] + .into_iter() + .try_fold(0usize, |bytes, part| bytes.checked_add(part?)) + .ok_or(ValidationError::ImageTooLarge.into()) +} + +/// An entry of the executor's queue of bands that can start or hand out a +/// part: soonest read first, then largest first, and the band. +type QueueEntry = Reverse<(i64, Reverse, usize)>; + +/// The most entries the executor's queue holds for `bands`: a band enters +/// it once when its ring has a slot for it and again for each part after +/// one it hands out, and a reader may start a band without taking its +/// entry, so a band puts as many entries in it as it has parts. +fn queue_entries(bands: &[Band]) -> usize { bands + .iter() + .fold(0usize, |entries, band| entries.saturating_add(band.parts)) +} + +/// Bytes of the job the pool boxes for each worker the executor sends: the +/// worker's closure, its two references, with the scope it runs in and the +/// job's context. +const SENT_WORKER_BYTES: usize = 4 * size_of::(); + +/// The bytes the rings and bands of the component of `route` hold once +/// planned, and the rows' scales planning them holds beside them. +fn ring_plan_bytes( + route: &BandRoute<'_>, + storage: &DecompositionStorage<'_>, + header: &Header<'_>, + component_idx: usize, +) -> Result<(usize, usize)> { + let (rings, bands) = ring_counts(route, storage, header, component_idx)?; + let planned = rings + .checked_mul(size_of::()) + .zip(bands.checked_mul(size_of::())) + .and_then(|(rings, bands)| rings.checked_add(bands)) + .ok_or(ValidationError::ImageTooLarge)?; + let scales = rings + .checked_mul(size_of::()) + .ok_or(ValidationError::ImageTooLarge)?; + Ok((planned, scales)) +} + +/// The bytes every route's rings and bands hold once [`plan_tile_rings`] +/// planned them, with each component's first strip and levels, and the +/// most it holds beside them while it plans one component: that one's own +/// rings, bands and row scales. +fn tile_ring_plan_bytes( + routes: &[Box>], + storage: &DecompositionStorage<'_>, + header: &Header<'_>, +) -> Result<(usize, usize)> { + let mut planned = routes + .len() + .checked_mul(2 * size_of::()) + .ok_or(ValidationError::ImageTooLarge)?; + let mut planning = 0usize; + for (component_idx, route) in routes.iter().enumerate() { + let (own, scales) = ring_plan_bytes(route, storage, header, component_idx)?; + planned = planned + .checked_add(own) + .ok_or(ValidationError::ImageTooLarge)?; + planning = planning.max(own.saturating_add(scales)); + } + Ok((planned, planning)) +} + +/// The rings and bands [`plan_rings`] plans for the component of `route`: +/// one ring for each sub-band it reconstructs, one band for each strip. +fn ring_counts( + route: &BandRoute<'_>, + storage: &DecompositionStorage<'_>, + header: &Header<'_>, + component_idx: usize, +) -> Result<(usize, usize)> { + let levels = active_levels(storage, header, component_idx)?; + let rings = levels + .checked_mul(3) + .and_then(|rings| rings.checked_add(1)) + .ok_or(ValidationError::ImageTooLarge)?; + Ok((rings, route.strips().count())) +} + +/// The levels the component reconstructs: its decompositions, but the +/// resolution levels the decode skips. +fn active_levels( + storage: &DecompositionStorage<'_>, + header: &Header<'_>, + component_idx: usize, +) -> Result { + storage + .tile_decompositions + .get(component_idx) + .ok_or(DecodingError::CodeBlockDecodeFailure)? + .decompositions .len() - .checked_mul(per_band) - .zip(rings.len().checked_mul(per_ring)) - .and_then(|(bands, rings)| bands.checked_add(rings)) - .and_then(|bytes| bytes.checked_add(slots.checked_mul(size_of::<&mut [f32]>())?)) - .ok_or(ValidationError::ImageTooLarge.into()) + .checked_sub(usize::from(header.skipped_resolution_levels)) + .filter(|levels| *levels > 0) + .ok_or_else(|| DecodingError::CodeBlockDecodeFailure.into()) } /// A sub-band's ring of band slots and its bands, in row order. @@ -305,6 +1088,8 @@ struct Ring { sub_band: usize, stored: IntRect, slots: usize, + /// Whether the sub-band belongs to the finest level the IDWT reads. + finest: bool, /// Whether the ring holds the sub-band's whole window, because its bands /// fit in fewer rows than its slots would: each band then decodes into /// its own rows of the window, and all of them can be in flight at once. @@ -322,6 +1107,10 @@ struct Band { /// component, approximately: bands it reads sooner start sooner. order: i64, work: usize, + /// Runs of its blocks the band decodes in, on as many threads at once. + parts: usize, + /// The count its blocks were split for, which gives those `parts` runs. + split: usize, } /// The component's rings, its bands grouped by ring in row order, and the @@ -333,12 +1122,7 @@ fn plan_rings( component_idx: usize, ) -> Result<(Vec, Vec, usize)> { let tile_decompositions = &storage.tile_decompositions[component_idx]; - let active_levels = tile_decompositions - .decompositions - .len() - .checked_sub(usize::from(header.skipped_resolution_levels)) - .filter(|levels| *levels > 0) - .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let active_levels = active_levels(storage, header, component_idx)?; let mut rings = Vec::new(); rings .try_reserve_exact(1 + 3 * active_levels) @@ -372,6 +1156,7 @@ fn plan_rings( sub_band, stored, slots, + finest: resolution == active_levels, whole_window, bands: 0..0, }); @@ -403,6 +1188,8 @@ fn plan_rings( rows: first..last, order: (i64::from(first) - SUPPORT_ROWS) * scales[index], work: strip.work, + parts: 1, + split: 1, }); } bands[start..].sort_unstable_by_key(|band| band.rows.start); @@ -421,13 +1208,113 @@ fn plan_rings( Ok((rings, bands, active_levels)) } -/// Each ring's free slots and the slot each band of a whole-window ring -/// owns. -type Slots<'c> = (Vec>, Vec>); - -/// Split each ring's coefficients into its slots: equal slots as long as its -/// longest band, or in a whole-window ring, each band's own rows. -fn split_slots<'c>( +/// Count the deepest of `rings` that hold no whole window, at the finest +/// level and at the others, in the decode's debug counters. +#[cfg(test)] +fn count_ring_slots(tile_ctx: &mut TileDecodeContext, rings: &[Ring]) { + let counters = &mut tile_ctx.debug_counters; + for ring in rings.iter().filter(|ring| !ring.whole_window) { + let deepest = if ring.finest { + &mut counters.pool_band_finest_slots + } else { + &mut counters.pool_band_other_slots + }; + *deepest = (*deepest).max(ring.slots); + } +} + +/// Most parts a band splits into. +const MAX_BAND_PARTS: usize = 8; + +/// Parts of finest-level bands that a split aims at per thread: with two, a +/// thread that finishes a part finds another while the band's last parts +/// finish. One part per thread left 5792 x 5792 gray 14% slower (80 against +/// 70 ms on 18 threads). +const BAND_PARTS_PER_THREAD: usize = 2; + +/// Split a classic tile's bands into parts for `threads` threads (see +/// [`plan_parts`]), and count the bands that split. A tile on the shared +/// rings (see `DecompositionStorage::shared_pool_band_rings`) keeps its bands +/// whole: the other images' bands keep the pool's threads busy, and the parts +/// left a batch filling a 10-thread pool 1% slower. +fn split_bands( + tile_ctx: &mut TileDecodeContext, + shared: bool, + rings: &[Ring], + bands: &mut [Band], + threads: usize, + split: impl Fn(&Band, usize) -> Result, +) -> Result<()> { + #[cfg(test)] + let forced = tile_ctx.band_parts; + #[cfg(not(test))] + let forced = None; + if shared && forced.is_none() { + return Ok(()); + } + plan_parts(rings, bands, threads, forced, split)?; + tile_ctx.debug_counters.pool_band_parts += bands.iter().filter(|band| band.parts > 1).count(); + Ok(()) +} + +/// Split `bands` into parts, through `split` (which returns the runs a band +/// splits into for a wanted count), when the finest level's slots ahead of +/// the bands being read hold fewer bands than `threads`: each finest band +/// into as many parts as give [`BAND_PARTS_PER_THREAD`] parts per thread, and +/// every other band into parts of about the same work. A tile whose rings +/// hold bands enough keeps them whole. `forced` splits every band into that +/// many parts instead. +fn plan_parts( + rings: &[Ring], + bands: &mut [Band], + threads: usize, + forced: Option, + split: impl Fn(&Band, usize) -> Result, +) -> Result<()> { + if let Some(parts) = forced { + for band in bands { + band.split = parts.max(1); + band.parts = split(band, band.split)?; + } + return Ok(()); + } + let ahead: usize = rings + .iter() + .filter(|ring| ring.finest && !ring.whole_window) + .map(|ring| ring.slots.saturating_sub(1)) + .sum(); + if ahead == 0 || ahead >= threads { + return Ok(()); + } + let parts = threads + .saturating_mul(BAND_PARTS_PER_THREAD) + .div_ceil(ahead) + .min(MAX_BAND_PARTS); + let (finest_work, finest_bands) = rings + .iter() + .filter(|ring| ring.finest) + .flat_map(|ring| &bands[ring.bands.clone()]) + .fold((0usize, 0usize), |(work, count), band| { + (work.saturating_add(band.work), count + 1) + }); + let part_work = (finest_work / finest_bands.max(1) / parts).max(1); + for band in bands { + let wanted = band.work.div_ceil(part_work).clamp(1, parts); + if wanted > 1 { + band.split = wanted; + band.parts = split(band, wanted)?; + } + } + Ok(()) +} + +/// Each ring's free slots and the slot each band of a whole-window ring +/// owns. +type Slots<'c> = (Vec>, Vec>); + +/// Split each ring's coefficients into its slots: equal slots as long as its +/// longest band, or in a whole-window ring, each band's own rows. +fn split_slots<'c>( rings: &[Ring], bands: &[Band], storage: &DecompositionStorage<'_>, @@ -483,9 +1370,15 @@ fn split_slots<'c>( offset = start + len; } } else { + // Each slot holds whole rows of the sub-band, as many as the + // ring's share of them: at least its longest band's, which + // can be fewer than the rows the ring was laid out for. + let width = ring.stored.width() as usize; let slot_len = range .len() .checked_div(ring.slots) + .and_then(|len| len.checked_div(width)) + .and_then(|rows| rows.checked_mul(width)) .ok_or(DecodingError::CodeBlockDecodeFailure)?; for _ in 0..ring.slots { let (slot, tail) = core::mem::take(&mut ring_samples).split_at_mut(slot_len); @@ -520,11 +1413,16 @@ struct State<'c, 'w, W> { /// The rows of the window each band of a whole-window ring decodes into, /// until it starts. owned: Vec>, - /// Bands that can start, soonest read first, then largest first. - /// Entries for bands since started are skipped. - ready: BinaryHeap, usize)>>, + /// Bands that can start, or hand out another part, soonest read first, + /// then largest first. Entries for bands with nothing left to hand out + /// are skipped. + ready: BinaryHeap, /// Each ring's next band to make ready. next_ready: Vec, + /// Parts of each band handed out, and parts still decoding or to hand + /// out. + next_part: Vec, + parts_left: Vec, /// How many of each ring's bands, from its first, have been freed. freed_prefix: Vec, /// Bands that finished decoding, for the drain to free; bands the IDWT @@ -532,7 +1430,7 @@ struct State<'c, 'w, W> { decoded: Vec, running: usize, freed: usize, - workspaces: Vec<&'w mut W>, + workspaces: Vec<(&'w mut W, &'w mut BlockScratch)>, /// Workers on the pool. active: usize, error: Option, @@ -543,10 +1441,13 @@ struct State<'c, 'w, W> { struct Shared<'p, 'c, 'w, W> { rings: &'p [Ring], bands: &'p [Band], + /// The slot of each band decoding in parts, which its parts share. + outputs: Vec>, max_workers: usize, - /// Test-only: the first worker to start a band abandons it. + /// Test-only: the worker that starts the band task after this many + /// abandons it (a negative count: none does). #[cfg(test)] - strand_once: core::sync::atomic::AtomicBool, + strand_after: core::sync::atomic::AtomicIsize, state: Mutex>, /// Signalled when a band is decoded or a decode fails. decoded: Condvar, @@ -575,17 +1476,22 @@ impl<'c, 'w, W> Shared<'_, 'c, 'w, W> { } } - /// Take the next ready band, marked running, with a slot to decode into. - fn start_next(&self, state: &mut State<'c, 'w, W>) -> Result> { + /// Take the next ready band, marked running, with a slot to decode + /// into, or the next part of a running band. + fn start_next(&self, state: &mut State<'c, 'w, W>) -> Result>> { while let Some(Reverse((_, _, index))) = state.ready.pop() { - if state.status[index] == Status::Waiting { - return self.start(state, index).map(Some); + match state.status[index] { + Status::Waiting => return self.start(state, index).map(Some), + Status::Running if self.has_part(state, index) => { + return Ok(Some(self.next_part(state, index))); + } + _ => {} } } Ok(None) } - fn start(&self, state: &mut State<'c, 'w, W>, index: usize) -> Result<(usize, &'c mut [f32])> { + fn start(&self, state: &mut State<'c, 'w, W>, index: usize) -> Result> { let slot = match state.owned[index].take() { Some(slot) => slot, None => state.free_slots[self.bands[index].ring] @@ -594,7 +1500,62 @@ impl<'c, 'w, W> Shared<'_, 'c, 'w, W> { }; state.status[index] = Status::Running; state.running += 1; - Ok((index, slot)) + let parts = self.bands[index].parts; + if parts <= 1 { + return Ok(Task::Whole(index, slot)); + } + *self.output(index)? = slot; + state.parts_left[index] = parts; + Ok(self.next_part(state, index)) + } + + /// Whether running band `index` decodes in parts and has parts left to + /// hand out. + fn has_part(&self, state: &State<'c, 'w, W>, index: usize) -> bool { + let parts = self.bands[index].parts; + parts > 1 && state.next_part[index] < parts + } + + /// Hand out the next part of running band `index`, leaving the band + /// ready while it has more. + fn next_part(&self, state: &mut State<'c, 'w, W>, index: usize) -> Task<'c> { + let part = state.next_part[index]; + state.next_part[index] += 1; + if self.has_part(state, index) { + let band = &self.bands[index]; + state + .ready + .push(Reverse((band.order, Reverse(band.work), index))); + } + Task::Part(index, part) + } + + fn output(&self, index: usize) -> Result> { + self.outputs[index] + .lock() + .map_err(|_| DecodingError::CodeBlockDecodeFailure.into()) + } + + /// Record a part of band `index`, decoded outside the lock; the last + /// part to finish leaves the band decoded. + fn finish_part(&self, index: usize, result: Result<()>) -> Result<()> { + let mut state = self.lock()?; + match result { + Ok(()) => { + state.parts_left[index] -= 1; + if state.parts_left[index] == 0 { + let slot = core::mem::take(&mut *self.output(index)?); + state.held[index] = Some(slot); + Self::decoded(&mut state, index); + } + } + Err(error) => { + state.error.get_or_insert(error); + } + } + drop(state); + self.decoded.notify_all(); + Ok(()) } /// Band `index`, running, has decoded. @@ -641,43 +1602,53 @@ impl<'c, 'w, W> Shared<'_, 'c, 'w, W> { } } -/// Decode `bands` into `slots` on up to `route.workers()` pool threads -/// while `run` drives the IDWT through them on this thread, then decode the -/// bands it didn't read. Returns `false` when a band was stranded. +/// A thread's next decode: a band of one part, straight into its slot, or +/// part `.1` of band `.0`, into the slot its parts share. +enum Task<'c> { + Whole(usize, &'c mut [f32]), + Part(usize, usize), +} + +/// Decode `bands` into `slots` on up to `workers` pool threads while `run` +/// drives the IDWT through `readers` views of them, then decode the bands it +/// didn't read. Each view reads its own sub-bands and decodes a band nobody +/// has started with a workspace and scratch of its own, so views may run on +/// different threads. Views that `help` decode other bands while they wait. +/// Returns `false` when a band was stranded. #[expect( clippy::too_many_arguments, - reason = "the executor takes the plan, its storage and its two drivers explicitly" + clippy::too_many_lines, + reason = "the executor takes the plan, its storage and its two drivers explicitly, and sets up its shared state and runs its views in one scope" )] fn run_bands( - route: &BandRoute<'_>, + (workers, readers, helps): (usize, usize, bool), rings: &[Ring], bands: &[Band], slots: Slots<'_>, - bank: &mut [TaskWorkspace], - decode: impl Fn(usize, &mut [f32], &mut W, &mut Vec) -> Result<()> + Sync, - run: impl FnOnce(&mut dyn SubBandRows) -> Result<()>, + (bank, scratches): (&mut [TaskWorkspace], &mut [BlockScratch]), + decode: impl Fn(usize, BandOutput<'_, '_>, &mut W, &mut BlockScratch) -> Result<()> + Sync, + run: impl FnOnce(&mut [&mut (dyn SubBandRows + Send)]) -> Result<()>, profile_enabled: bool, tier1_us: &mut u128, - strand_once: bool, + strand: Option, ) -> Result { #[cfg(not(test))] - let _ = strand_once; - let workspaces_needed = route.workers() + 1; - let mut workspaces: Vec<&mut W> = Vec::new(); + let _ = strand; + let workspaces_needed = workers + readers.max(1); + let mut workspaces: Vec<(&mut W, &mut BlockScratch)> = Vec::new(); workspaces .try_reserve_exact(workspaces_needed) .map_err(|_| DecodingError::HostAllocationFailed)?; workspaces.extend( bank.iter_mut() - .take(workspaces_needed) - .map(|slot| &mut slot.workspace), + .map(|slot| &mut slot.workspace) + .zip(scratches.iter_mut()) + .take(workspaces_needed), ); if workspaces.len() != workspaces_needed { return Err(DecodingError::CodeBlockDecodeFailure.into()); } - let own_workspace = workspaces - .pop() - .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let reader_workspaces = workspaces.split_off(workers); let mut state = State { status: try_filled(bands.len(), Status::Waiting)?, held: try_filled_with(bands.len(), || None)?, @@ -685,6 +1656,8 @@ fn run_bands( owned: slots.1, ready: BinaryHeap::new(), next_ready: try_collect(rings.iter().map(|ring| ring.bands.start), rings.len())?, + next_part: try_filled(bands.len(), 0)?, + parts_left: try_filled(bands.len(), 0)?, freed_prefix: try_filled(rings.len(), 0)?, decoded: Vec::new(), running: 0, @@ -696,7 +1669,7 @@ fn run_bands( }; state .ready - .try_reserve(bands.len()) + .try_reserve_exact(queue_entries(bands)) .map_err(|_| DecodingError::HostAllocationFailed)?; state .decoded @@ -705,12 +1678,17 @@ fn run_bands( let shared = Shared { rings, bands, - max_workers: route.workers(), + outputs: try_filled_with(bands.len(), || Mutex::new(<&mut [f32]>::default()))?, + max_workers: workers, #[cfg(test)] - strand_once: core::sync::atomic::AtomicBool::new(strand_once), + strand_after: test_countdown(strand), state: Mutex::new(state), decoded: Condvar::new(), }; + // Where locks allocate when first used, the state's is first locked + // below, and the condition variable is first used here, before any + // worker races to allocate it. + shared.decoded.notify_all(); { let mut state = shared.lock()?; for ring in 0..rings.len() { @@ -720,20 +1698,45 @@ fn run_bands( let decode = &decode; let shared = &shared; let scoped = rayon::in_place_scope(|scope| -> Result<()> { - let mut rows = PoolRows { - shared, - scope, - decode, - workspace: own_workspace, - clipped_scratch: Vec::new(), - current: try_filled_with(rings.len(), || None)?, - next: try_collect(rings.iter().map(|ring| ring.bands.start), rings.len())?, - profile_enabled, - tier1_us: 0, - }; - rows.send_workers()?; - let result = run(&mut rows).and_then(|()| rows.drain()); - *tier1_us = rows.tier1_us; + let mut views = Vec::new(); + views + .try_reserve_exact(reader_workspaces.len()) + .map_err(|_| DecodingError::HostAllocationFailed)?; + for (workspace, scratch) in reader_workspaces { + views.push(PoolRows { + shared, + scope, + decode, + workspace, + scratch, + current: try_filled_with(rings.len(), || None)?, + next: try_collect(rings.iter().map(|ring| ring.bands.start), rings.len())?, + helps, + profile_enabled, + tier1_us: 0, + }); + } + views + .first() + .ok_or(DecodingError::CodeBlockDecodeFailure)? + .send_workers()?; + let mut readers: Vec<&mut (dyn SubBandRows + Send)> = Vec::new(); + readers + .try_reserve_exact(views.len()) + .map_err(|_| DecodingError::HostAllocationFailed)?; + readers.extend( + views + .iter_mut() + .map(|view| view as &mut (dyn SubBandRows + Send)), + ); + let result = run(&mut readers).and_then(|()| { + // Every view hands its bands back before the first drains. + for view in &mut views[1..] { + view.release_all()?; + } + views[0].drain() + }); + *tier1_us = views.iter().map(|view| view.tier1_us).sum(); if result.is_err() { // Let the workers stop at their next band. if let Ok(mut state) = shared.lock() { @@ -752,6 +1755,35 @@ fn run_bands( error.map_or(Ok(true), Err) } +/// The band task a test strands (see `TileDecodeContext::strand_pool_band`). +#[cfg(test)] +fn test_strand(tile_ctx: &TileDecodeContext) -> Option { + tile_ctx + .strand_pool_band + .then_some(tile_ctx.strand_pool_band_after) +} + +/// No band task is stranded outside tests. +#[cfg(not(test))] +const fn test_strand(_: &TileDecodeContext) -> Option { + None +} + +/// Test-only: count one more event down on `left`, and whether it was the +/// one the count waited for (a negative count waits for none). +#[cfg(test)] +fn test_countdown_ends(left: &core::sync::atomic::AtomicIsize) -> bool { + left.fetch_sub(1, core::sync::atomic::Ordering::Relaxed) == 0 +} + +/// Test-only: a countdown of `events` for [`test_countdown_ends`]. +#[cfg(test)] +fn test_countdown(events: Option) -> core::sync::atomic::AtomicIsize { + core::sync::atomic::AtomicIsize::new( + events.map_or(-1, |events| isize::try_from(events).unwrap_or(isize::MAX)), + ) +} + fn try_filled(len: usize, value: T) -> Result> { let mut values = Vec::new(); values @@ -779,27 +1811,54 @@ fn try_filled_with(len: usize, value: impl FnMut() -> T) -> Result> { Ok(values) } +/// Run `task` with `workspace` and `scratch`, and record it. +fn run_task<'c, W>( + shared: &Shared<'_, 'c, '_, W>, + decode: &(impl Fn(usize, BandOutput<'_, '_>, &mut W, &mut BlockScratch) -> Result<()> + Sync), + task: Task<'c>, + workspace: &mut W, + scratch: &mut BlockScratch, +) -> Result<()> { + match task { + Task::Whole(index, slot) => { + slot.fill(0.0); + let strip = shared.bands[index].strip; + let result = decode(strip, BandOutput::Whole(&mut *slot), workspace, scratch); + shared.finish(index, slot, result) + } + Task::Part(index, part) => { + let band = &shared.bands[index]; + let output = BandOutput::Part { + part, + parts: band.split, + slot: &shared.outputs[index], + }; + let result = decode(band.strip, output, workspace, scratch); + shared.finish_part(index, result) + } + } +} + /// A pool worker: decode ready bands until none is left to start. fn work( shared: &Shared<'_, '_, '_, W>, - decode: &(impl Fn(usize, &mut [f32], &mut W, &mut Vec) -> Result<()> + Sync), + decode: &(impl Fn(usize, BandOutput<'_, '_>, &mut W, &mut BlockScratch) -> Result<()> + Sync), ) { let Ok(mut state) = shared.lock() else { return; }; - let Some(workspace) = state.workspaces.pop() else { + let Some((workspace, scratch)) = state.workspaces.pop() else { state.active -= 1; return; }; - let mut clipped_scratch = Vec::new(); loop { let next = if state.error.is_some() { Ok(None) } else { shared.start_next(&mut state) }; - let (index, slot) = match next { - Ok(Some(started)) => started, + let task = match next { + Ok(Some(task)) => task, Ok(None) => break, Err(error) => { state.error.get_or_insert(error); @@ -808,24 +1867,14 @@ fn work( }; drop(state); #[cfg(test)] - if shared - .strand_once - .swap(false, core::sync::atomic::Ordering::Relaxed) - { + if test_countdown_ends(&shared.strand_after) { // Leave the band running, as the stranding bug would. if let Ok(mut state) = shared.lock() { state.active -= 1; } return; } - slot.fill(0.0); - let result = decode( - shared.bands[index].strip, - slot, - workspace, - &mut clipped_scratch, - ); - if shared.finish(index, slot, result).is_err() { + if run_task(shared, decode, task, workspace, scratch).is_err() { return; } state = match shared.lock() { @@ -833,7 +1882,7 @@ fn work( Err(_) => return, }; } - state.workspaces.push(workspace); + state.workspaces.push((workspace, scratch)); state.active -= 1; drop(state); shared.decoded.notify_all(); @@ -846,11 +1895,14 @@ struct PoolRows<'s, 'p, 'c, 'w, W, D> { scope: &'s rayon::Scope<'p>, decode: &'p D, workspace: &'w mut W, - clipped_scratch: Vec, + scratch: &'w mut BlockScratch, /// Each ring's band being read, with its slot. current: Vec>, /// Each ring's next band to read. next: Vec, + /// Whether a wait for a running band first decodes the next band + /// nobody has started. + helps: bool, profile_enabled: bool, tier1_us: u128, } @@ -858,7 +1910,7 @@ struct PoolRows<'s, 'p, 'c, 'w, W, D> { impl<'p, 'c, W, D> PoolRows<'_, 'p, 'c, '_, W, D> where W: TaskScratch + 'p, - D: Fn(usize, &mut [f32], &mut W, &mut Vec) -> Result<()> + Sync, + D: Fn(usize, BandOutput<'_, '_>, &mut W, &mut BlockScratch) -> Result<()> + Sync, 'c: 'p, { /// Send pool workers, up to the plan's count, while bands are ready. @@ -901,50 +1953,75 @@ where fn acquire(&mut self, index: usize) -> Result<&'c mut [f32]> { let mut state = self.shared.lock()?; loop { - if let Some(error) = state.error.take() { + // The error stays for the other readers, whose bands it may + // have left running. + if let Some(error) = state.error { return Err(error); } - match state.status[index] { + let task = match state.status[index] { Status::Done => { return state.held[index] .take() .ok_or(DecodingError::CodeBlockDecodeFailure.into()); } - Status::Running => { - state = self - .shared - .decoded - .wait_timeout(state, STRANDED_CHECK) - .map_err(|_| DecodingError::CodeBlockDecodeFailure)? - .0; - // A failed band stays running; its error ends the wait. - if state.error.is_none() - && state.status[index] == Status::Running - && state.active == 0 - { - state.stranded = true; - return Err(DecodingError::CodeBlockDecodeFailure.into()); - } - } - Status::Waiting => { - let (index, slot) = self.shared.start(&mut state, index)?; - drop(state); - self.decode_here(index, slot)?; - return Ok(slot); - } Status::Freed => return Err(DecodingError::CodeBlockDecodeFailure.into()), + // A band nobody has started, or its first part, decodes + // here; its other parts go to the pool. + Status::Waiting => Some(self.shared.start(&mut state, index)?), + // A reader decodes a part of this band nobody has started, + // and one that helps the next band or part nobody has + // started, while this one runs. + Status::Running if self.shared.has_part(&state, index) => { + Some(self.shared.next_part(&mut state, index)) + } + Status::Running if self.helps => self.shared.start_next(&mut state)?, + Status::Running => None, + }; + if let Some(task) = task { + // A reader decoding a band counts as an active worker for + // anyone waiting on it. + state.active += 1; + drop(state); + self.send_workers()?; + let decoded = self.decode_for_pool(task); + state = self.shared.lock()?; + state.active -= 1; + decoded?; + continue; + } + state = self + .shared + .decoded + .wait_timeout(state, STRANDED_CHECK) + .map_err(|_| DecodingError::CodeBlockDecodeFailure)? + .0; + // A failed band stays running; its error ends the wait. + if state.error.is_none() && state.status[index] == Status::Running && state.active == 0 + { + state.stranded = true; + drop(state); + return Err(DecodingError::CodeBlockDecodeFailure.into()); } } } + /// Decode `task`, a band or part this reader, another one or the drain + /// will take, and hand it over as a worker does. + fn decode_for_pool(&mut self, task: Task<'c>) -> Result<()> { + let start = profile::profile_now(self.profile_enabled); + let result = run_task(self.shared, self.decode, task, self.workspace, self.scratch); + self.tier1_us += profile::elapsed_us(start); + result + } + fn decode_here(&mut self, index: usize, slot: &mut [f32]) -> Result<()> { let start = profile::profile_now(self.profile_enabled); slot.fill(0.0); let result = (self.decode)( self.shared.bands[index].strip, - slot, + BandOutput::Whole(slot), self.workspace, - &mut self.clipped_scratch, + self.scratch, ); self.tier1_us += profile::elapsed_us(start); result?; @@ -954,15 +2031,23 @@ where Ok(()) } - /// After the IDWT, free its bands and decode every band it didn't read, - /// here and on the pool. - fn drain(&mut self) -> Result<()> { + /// Hand back every band this view is reading. + fn release_all(&mut self) -> Result<()> { for ring in 0..self.current.len() { self.release(ring)?; } + Ok(()) + } + + /// After the IDWT, free its bands and decode every band it didn't read, + /// here and on the pool. + fn drain(&mut self) -> Result<()> { + self.release_all()?; loop { let mut state = self.shared.lock()?; - if let Some(error) = state.error.take() { + // The error stays for the other readers, whose bands it may + // have left running. + if let Some(error) = state.error { return Err(error); } // Free every decoded band, which lets its ring's next band start. @@ -980,12 +2065,17 @@ where let next = self.shared.start_next(&mut state)?; let running = state.running > 0; match next { - Some((index, slot)) => { + Some(Task::Whole(index, slot)) => { drop(state); self.send_workers()?; self.decode_here(index, slot)?; self.shared.lock()?.held[index] = Some(slot); } + Some(part @ Task::Part(..)) => { + drop(state); + self.send_workers()?; + self.decode_for_pool(part)?; + } None if running => { let mut state = self .shared @@ -1013,7 +2103,7 @@ where impl<'p, 'c, W, D> SubBandRows for PoolRows<'_, 'p, 'c, '_, W, D> where W: TaskScratch + 'p, - D: Fn(usize, &mut [f32], &mut W, &mut Vec) -> Result<()> + Sync, + D: Fn(usize, BandOutput<'_, '_>, &mut W, &mut BlockScratch) -> Result<()> + Sync, 'c: 'p, { fn prepare_row(&mut self, sub_band: usize, y: u32) -> Result<()> { @@ -1075,9 +2165,9 @@ mod tests { use crate::reader::BitReader; use crate::{encode, DecodeSettings, EncodeOptions, Image, DEFAULT_MAX_DECODE_BYTES}; - fn budget_fixture(width: u32, height: u32) -> Vec { + fn budget_fixture(width: u32, height: u32, components: u16, block_exp: u8) -> Vec { let mut state = 0x2545_f491_u32; - let pixels: Vec = (0..width * height) + let pixels: Vec = (0..width * height * u32::from(components)) .map(|_| { state ^= state << 13; state ^= state >> 17; @@ -1089,122 +2179,1227 @@ mod tests { &pixels, width, height, - 1, + components, 8, false, &EncodeOptions { reversible: true, num_decomposition_levels: 4, - code_block_width_exp: 0, - code_block_height_exp: 0, + code_block_width_exp: block_exp, + code_block_height_exp: block_exp, ..EncodeOptions::default() }, ) .expect("encode budget fixture") } - /// The route's plan and clipped scratch stay live beside the executor. - /// Leave enough simulated headroom for either alone, but not both. No - /// large retained allocation is needed to exercise the cap boundary. + struct CountingSink(usize); + + impl ComponentRowSink for CountingSink { + fn write_rows(&mut self, _x: u32, _y: u32, planes: &[crate::ComponentPlane<'_>]) { + self.0 += planes + .iter() + .map(|plane| plane.samples().len()) + .sum::(); + } + } + + /// Every live component plan and the scratch of the threads that serve + /// them must fit beside the executor, including the shared executor that + /// streams RGB rows. Simulated retained memory leaves room for either + /// owner, but not both. #[test] + #[expect( + clippy::too_many_lines, + reason = "one budget matrix checks the plans, the executor and the shared RGB executor together" + )] fn pool_bands_respect_the_combined_plan_and_executor_budget() { let (width, height) = (203, 157); - let bytes = budget_fixture(width, height); - let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse budget fixture"); - rayon::ThreadPoolBuilder::new() - .num_threads(2) - .build() - .expect("build budget test pool") - .install(|| { - let mut context = DecoderContext::default(); - context.set_output_region(Some((0, 0, width, height))); - context.tile_decode_context.line_region_idwt = true; - let tiles = tile::parse(&mut BitReader::new(image.codestream), &image.header, 0) - .expect("parse tile"); - let baseline = context - .reset( - &image.header, + // A component in 16 x 16 blocks plans its route in less room than + // its executor then needs beside it; 4 x 4 blocks plan in more. + for (components, block_exp) in [(1_u16, 2), (3, 0)] { + let bytes = budget_fixture(width, height, components, block_exp); + let image = + Image::new(&bytes, &DecodeSettings::default()).expect("parse budget fixture"); + rayon::ThreadPoolBuilder::new() + .num_threads(2) + .build() + .expect("build budget test pool") + .install(|| { + let mut context = DecoderContext::default(); + context.set_output_region(Some((0, 0, width, height))); + context.tile_decode_context.line_region_idwt = true; + context.tile_decode_context.stream_rows = components > 1; + let tiles = + tile::parse(&mut BitReader::new(image.codestream), &image.header, 0) + .expect("parse tile"); + let baseline = context + .reset( + &image.header, + &tiles[0], + tiles.structural_workspace_bytes(), + 0, + false, + None, + ) + .expect("prepare output planes"); + let (tile_ctx, storage) = + (&mut context.tile_decode_context, &mut context.storage); + let mut sink = CountingSink(0); + // Warm the retained bank and leave parsed block storage + // available at the tile boundary, before applying the cap. + let output = if components == 1 { + super::super::TileOutput::Planes + } else { + super::super::TileOutput::Sink(&mut sink) + }; + super::super::decode_tile( &tiles[0], - tiles.structural_workspace_bytes(), - 0, + &image.header, + tile_ctx, + storage, + &mut None, + CpuDecodeParallelism::Auto, false, - None, + &mut DecodeProfileTimings::default(), + baseline, + output, ) - .expect("prepare output planes"); - let (tile_ctx, storage) = (&mut context.tile_decode_context, &mut context.storage); - // Stop at the tile boundary so its parsed block storage stays - // available. This also prepares the retained worker bank, so - // the admission check below needs no replacement bank. - super::super::decode_tile( - &tiles[0], - &image.header, - tile_ctx, - storage, - &mut None, - CpuDecodeParallelism::Auto, - false, - &mut DecodeProfileTimings::default(), - baseline, - super::super::TileOutput::Planes, - ) - .expect("decode with ample headroom"); - assert_eq!(tile_ctx.debug_counters.pool_band_components, 1); - let (route_bytes, executor) = { - let BandRouting::Route(route) = - plan_band_route(&tiles[0], 0, tile_ctx, storage, &image.header) - .expect("plan control route") - else { - panic!("control fixture must take the pool route"); - }; - // Reuse the existing peak accounting for the route's owned - // tables and clipped scratch, excluding the retained bank. - let retained_tier1 = tile_ctx.tier1_capacity_bytes().expect("bank bytes"); - tile_ctx.debug_counters.workspace_peak = WorkspacePeak::default(); - route - .record(tile_ctx, storage) + .expect("decode with ample headroom"); + assert_eq!( + tile_ctx.debug_counters.pool_band_components, + usize::from(components) + ); + let (route_bytes, executor) = { + let mut routes = Vec::new(); + let mut live = None; + for component in 0..usize::from(components) { + let BandRouting::Route(route) = plan_band_route_beside( + &tiles[0], + component, + tile_ctx, + storage, + &image.header, + live, + components > 1, + ) + .expect("plan control route") else { + panic!("control fixture must take the pool route"); + }; + live = Some(route.budget()); + routes.push(route); + } + // Reuse the existing peak accounting for the routes' + // owned tables and their threads' scratch, excluding + // the retained bank: the workers and one reader. + let threads = routes + .iter() + .map(|route| route.workers() + 1) + .max() + .unwrap_or(0); + let retained = tile_ctx.tier1_capacity_bytes().expect("bank bytes"); + tile_ctx.debug_counters.workspace_peak = WorkspacePeak::default(); + BandRoute::record_live( + routes.iter().map(|route| &**route), + threads, + tile_ctx, + storage, + ) .expect("record route owners"); - let route_bytes = - tile_ctx.debug_counters.workspace_peak.bytes().tier1 - retained_tier1; - let (rings, bands, _) = - plan_rings(&route, storage, &image.header, 0).expect("plan control rings"); - ( - route_bytes, - executor_bytes(&rings, &bands).expect("executor bytes"), + let route_bytes = + tile_ctx.debug_counters.workspace_peak.bytes().tier1 - retained; + let (rings, bands, _, _) = plan_tile_rings(&routes, storage, &image.header) + .expect("plan control rings"); + ( + route_bytes, + executor_bytes(&rings, &bands, (threads - 1, 1)) + .expect("executor bytes"), + ) + }; + let headroom = route_bytes + executor / 2; + assert!(headroom >= executor, "each owner must fit separately"); + let baseline = DecodeAllocationBudget::for_storage(storage) + .expect("control budget") + .live_bytes(); + let structural = storage.structural_workspace_bytes; + storage.structural_workspace_bytes += + DEFAULT_MAX_DECODE_BYTES - baseline - headroom; + sink.0 = 0; + let result = if components == 1 { + // The least room the component's route plans in, + // which leaves none for its executor. + let (mut short, mut room) = (route_bytes, 4 * headroom); + let mut plans = |headroom: usize| { + storage.structural_workspace_bytes = + structural + DEFAULT_MAX_DECODE_BYTES - baseline - headroom; + match plan_band_route( + &tiles[0], + 0, + tile_ctx, + storage, + &image.header, + false, + ) { + Ok(BandRouting::Route(route)) => Some(route), + Ok(BandRouting::Serial | BandRouting::WholeWindows) => None, + Err(error) => panic!("planning within {headroom}: {error:?}"), + } + }; + assert!(plans(short).is_none() && plans(room).is_some()); + while room - short > 1 { + let middle = short + (room - short) / 2; + if plans(middle).is_some() { + room = middle; + } else { + short = middle; + } + } + let route = plans(room).expect("the route plans"); + let declined = decode_route( + &route, + 0, + &image.header, + tile_ctx, + storage, + tiles[0].component_infos[0].wavelet_transform(), + false, + &mut 0, + ); + assert!( + matches!(declined, Ok(None)), + "the executor must decline before decoding: {declined:?}" + ); + drop(route); + // Where the executor does not fit, the serial bands + // decode the component as the pool does with room. + let mut decode = |headroom: usize| { + storage.structural_workspace_bytes = + structural + DEFAULT_MAX_DECODE_BYTES - baseline - headroom; + tile_ctx.channel_data[0].container.fill(f32::NAN); + let pooled = tile_ctx.debug_counters.pool_band_components; + let decoded = decode_component( + &tiles[0], + 0, + &image.header, + tile_ctx, + storage, + tiles[0].component_infos[0].wavelet_transform(), + false, + &mut DecodeProfileTimings::default(), + ); + assert!( + matches!(decoded, Ok(true)), + "the component decodes within {headroom}: {decoded:?}" + ); + let bits: Vec = tile_ctx.channel_data[0] + .container + .iter() + .map(|sample| sample.to_bits()) + .collect(); + (tile_ctx.debug_counters.pool_band_components - pooled, bits) + }; + let (pooled, with_room) = decode(4 * headroom); + assert_eq!(pooled, 1, "the pool decodes the component with room"); + let (pooled, serial) = decode(room); + assert_eq!(pooled, 0, "the serial bands decode it within {room}"); + assert!( + serial == with_room, + "the serial bands decode it bit for bit" + ); + return; + } else { + decode_tile_rows( + &tiles[0], + &image.header, + tile_ctx, + storage, + &mut sink, + false, + &mut DecodeProfileTimings::default(), + ) + }; + match result { + Err(error) if super::super::reuse::is_capacity_error(&error) => {} + Ok(false) => {} + other => panic!( + "pool route must decline or reject combined live owners: \ + components={components}, headroom={headroom}, route={route_bytes}, \ + executor={executor}, excess={}, result={other:?}", + route_bytes + executor - headroom, + ), + } + assert_eq!( + sink.0, 0, + "budget rejection must precede streaming any rows" + ); + }); + } + } + + /// Collects the sample bits a sink receives, by component, and how often + /// each sample arrives. + struct Collected { + width: usize, + planes: Vec>, + arrivals: Vec, + } + + impl Collected { + fn new(width: u32, height: u32) -> Self { + Self { + width: width as usize, + planes: Vec::new(), + arrivals: alloc::vec![0; width as usize * height as usize], + } + } + } + + impl ComponentRowSink for Collected { + fn write_rows(&mut self, x: u32, y: u32, planes: &[crate::ComponentPlane<'_>]) { + if self.planes.is_empty() { + self.planes = alloc::vec![alloc::vec![0; self.arrivals.len()]; planes.len()]; + } + let (width, height) = planes[0].dimensions(); + for (component, plane) in planes.iter().enumerate() { + for row in 0..height as usize { + for column in 0..width as usize { + let at = (y as usize + row) * self.width + x as usize + column; + self.planes[component][at] = + plane.samples()[row * width as usize + column].to_bits(); + if component == 0 { + self.arrivals[at] += 1; + } + } + } + } + } + } + + /// The bytes a streamed tile's pool bands charge beyond the decode's + /// baseline, in the order they charge them: the list of routes, every + /// route's plan, the rings and bands, with the most planning them holds + /// beside them, the executor's records and the threads' scratch. + #[derive(Clone, Copy, Debug)] + struct Steps { + routes: usize, + plans: usize, + rings: usize, + planning: usize, + /// The streamed IDWT's plan, ring length and reconstruction of each + /// component. + lines: usize, + executor: usize, + scratch: usize, + /// The tile's readers, beside its workers. + readers: usize, + } + + /// How a streamed tile decoded: on the pool's bands or, declined there, + /// on serial bands, and to what rows or capacity error. + struct Streamed { + pooled: bool, + rows: Result, + } + + /// Decode the one tile of `image` on a fresh context, streaming its rows + /// to a sink: first with room to spare, which warms the workspace bank + /// and gives the rows and the [`Steps`], measured from what the routes, + /// records and scratch hold; then again as `decode_tile` does, on the + /// pool's bands or, when they decline the tile, on serial bands, with all + /// but `headroom` bytes beyond the baseline taken as if retained, which + /// every budget of the decode sees. + #[expect( + clippy::too_many_lines, + reason = "the warm pass measures the steps the budgeted pass is then checked against, on one context" + )] + fn decode_streamed(image: &Image<'_>, headroom: usize) -> (Steps, Collected, Streamed) { + let (width, height) = (image.width(), image.height()); + let mut context = DecoderContext::default(); + context.set_output_region(Some((0, 0, width, height))); + context.tile_decode_context.line_region_idwt = true; + context.tile_decode_context.stream_rows = true; + let tiles = tile::parse(&mut BitReader::new(image.codestream), &image.header, 0) + .expect("parse tile"); + let baseline = context + .reset( + &image.header, + &tiles[0], + tiles.structural_workspace_bytes(), + 0, + false, + None, + ) + .expect("prepare output planes"); + let (tile_ctx, storage) = (&mut context.tile_decode_context, &mut context.storage); + let mut warm = Collected::new(width, height); + super::super::decode_tile( + &tiles[0], + &image.header, + tile_ctx, + storage, + &mut None, + CpuDecodeParallelism::Auto, + false, + &mut DecodeProfileTimings::default(), + baseline, + super::super::TileOutput::Sink(&mut warm), + ) + .expect("decode with room to spare"); + let components = tiles[0].component_infos.len(); + assert_eq!(tile_ctx.debug_counters.pool_band_components, components); + assert!(warm.arrivals.iter().all(|&count| count == 1)); + let start = DecodeAllocationBudget::for_storage(storage) + .expect("baseline") + .live_bytes(); + let routes_bytes = components * size_of::>>(); + let mut live = DecodeAllocationBudget::for_storage(storage).expect("baseline"); + live.include_bytes(routes_bytes).expect("routes list"); + let mut routes = Vec::new(); + for component in 0..components { + let BandRouting::Route(route) = plan_band_route_beside( + &tiles[0], + component, + tile_ctx, + storage, + &image.header, + Some(live), + true, + ) + .expect("plan route") else { + panic!("the tile takes the pool's bands"); + }; + live = route.budget(); + routes.push(route); + } + let plans = live.live_bytes() - start - routes_bytes; + // With the bank warm, planning charges no workspace; it charges the + // plans' tables, and a task per strip. + let recorded = tile_ctx.debug_counters.pool_band_plan_bytes; + BandRoute::record_live(routes.iter().map(|route| &**route), 0, tile_ctx, storage) + .expect("record plans"); + let tables = tile_ctx.debug_counters.pool_band_plan_bytes - recorded; + assert!( + plans >= tables, + "planning charged {plans} for {tables} of tables" + ); + let (_, planning) = + tile_ring_plan_bytes(&routes, storage, &image.header).expect("planning bytes"); + let (rings, mut bands, strip_starts, active_levels) = + plan_tile_rings(&routes, storage, &image.header).expect("plan rings"); + let ring_bytes = rings.capacity() * size_of::() + + bands.capacity() * size_of::() + + (strip_starts.capacity() + active_levels.capacity()) * size_of::(); + // The threads the tile streamed on: the most workers its routes + // planned that the bank holds beside its readers, which split its + // bands. + let readers = tile_ctx.debug_counters.pool_band_readers; + let route_workers = routes + .iter() + .map(|route| route.workers()) + .max() + .unwrap_or(0); + let workers = route_workers.min(tile_ctx.classic_task_workspaces.len() - readers); + split_bands( + tile_ctx, + storage.shared_pool_band_rings, + &rings, + &mut bands, + workers + readers, + |band, parts| { + let (route, strip) = route_strip(&routes, &strip_starts, band.strip)?; + route.strip_parts(strip, parts) + }, + ) + .expect("split bands"); + let steps = Steps { + routes: routes_bytes, + plans, + rings: ring_bytes, + planning, + lines: components + * (size_of::() + + size_of::() + + size_of::>()), + executor: executor_records(&rings, &bands, workers, readers), + scratch: tile_ctx.debug_counters.thread_scratch_held, + readers, + }; + drop((routes, rings, bands, strip_starts, active_levels)); + storage.structural_workspace_bytes += DEFAULT_MAX_DECODE_BYTES - start - headroom; + let mut sink = Collected::new(width, height); + let pooled = decode_tile_rows( + &tiles[0], + &image.header, + tile_ctx, + storage, + &mut sink, + false, + &mut DecodeProfileTimings::default(), + ) + .unwrap_or_else(|error| panic!("the pool's bands decline instead: {error:?}")); + if pooled { + return ( + steps, + warm, + Streamed { + pooled, + rows: Ok(sink), + }, + ); + } + assert!( + sink.arrivals.iter().all(|&count| count == 0), + "no row before the pool declines" + ); + storage.parallel_bands = false; + storage.banded_tier1 = true; + let rows = banded_tier1::decode_tile_rows( + &tiles[0], + &image.header, + tile_ctx, + storage, + &mut sink, + false, + &mut DecodeProfileTimings::default(), + ) + .map(|()| sink); + (steps, warm, Streamed { pooled, rows }) + } + + /// A tile of blocks of several codeword segments streams its rows from + /// pool bands, with one reader or one per component, under exactly the + /// bytes they charge: the list of routes, the plans, the executor's + /// records and the threads' scratch, as their sizes give them, the + /// records' from their types and counts. A byte short of the list, of the + /// plans, of the records or of the scratch, the pool declines the tile, + /// before any row reaches the sink, and under the same retained bytes + /// serial bands stream the same rows when they fit and fail with a + /// capacity error when they don't. + #[test] + fn streamed_pool_bands_take_a_segmented_tile_at_exactly_their_bytes() { + let bytes = include_bytes!("../../../fixtures/multi_segment/bypass_termall_rgb_53.j2k"); + let image = Image::new(bytes, &DecodeSettings::default()).expect("parse fixture"); + // One reader for the tile on 4 threads, one per component on 12. + for (threads, expected_readers) in [(4, 1), (12, 3)] { + rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool") + .install(|| take_at_exactly_their_bytes(&image, expected_readers)); + } + } + + fn take_at_exactly_their_bytes(image: &Image<'_>, expected_readers: usize) { + let (steps, _, _) = decode_streamed(image, DEFAULT_MAX_DECODE_BYTES / 2); + let Steps { + routes, + plans, + rings, + planning, + lines, + executor, + scratch, + readers, + } = steps; + assert_eq!(readers, expected_readers, "{steps:?}"); + assert!( + routes > 0 && plans > 0 && rings > 0 && executor > 0 && scratch > 0, + "{steps:?}" + ); + // Planning the rings holds more beside them only until the executor + // and the scratch take its place. + let all = routes + plans + rings + planning.max(lines + executor + scratch); + let (_, warm, exact) = decode_streamed(image, all); + assert!(exact.pooled, "pool bands within {all} bytes: {steps:?}"); + let rows = exact.rows.expect("pool bands decode"); + assert!(rows.arrivals.iter().all(|&count| count == 1)); + assert!(rows.planes == warm.planes, "pool bands' rows"); + let mut serial = Vec::new(); + for headroom in [ + routes - 1, + routes + plans - 1, + routes + plans + rings - 1, + routes + plans + rings + lines - 1, + routes + plans + rings + lines + executor - 1, + all - 1, + ] { + let (_, _, short) = decode_streamed(image, headroom); + assert!( + !short.pooled, + "pool bands within {headroom} bytes: {steps:?}" + ); + match short.rows { + Ok(rows) => { + assert!(rows.arrivals.iter().all(|&count| count == 1)); + assert!(rows.planes == warm.planes, "serial rows at {headroom}"); + serial.push(true); + } + Err(error) => { + assert!( + super::super::reuse::is_capacity_error(&error), + "serial bands at {headroom}: {error:?}" + ); + serial.push(false); + } + } + } + // Serial bands need less than the pool's bands, and more than + // their list of routes. + assert_eq!(serial.first(), Some(&false), "{steps:?}"); + assert_eq!(serial.last(), Some(&true), "{steps:?}"); + assert!( + serial.windows(2).all(|pair| pair[0] <= pair[1]), + "serial bands fit more as the headroom grows: {serial:?}" + ); + } + + /// The bytes of the records the band executor allocates while `workers` + /// and `readers` threads decode `bands` of `rings`, from their types: per + /// band its status, held and owned slot, part counters, decoded entry + /// and shared output; a queue entry per part; per ring its readiness and + /// freed counters, slot list and slots, and each reader's band being read + /// and next band; per thread its workspace pair; per reader its view and + /// its entry in the readers' list; and the pool's job per worker and per + /// pool thread that may retire one, and what the locks of bands in parts + /// and of the shared state allocate, which + /// `band_executor_allocates_within_its_charge` measures. It decodes in a + /// pool, so its workers take no block of the pool's shared queue. + fn executor_records(rings: &[Ring], bands: &[Band], workers: usize, readers: usize) -> usize { + let per_band = size_of::() + + 2 * size_of::>() + + 3 * size_of::() + + size_of::>(); + let entries: usize = bands.iter().map(|band| band.parts).sum(); + let per_ring = 2 * size_of::() + + size_of::>() + + readers * (size_of::>() + size_of::()); + let slots: usize = rings.iter().map(|ring| ring.slots).sum(); + let per_thread = size_of::<(&mut (), &mut BlockScratch)>(); + let per_reader = size_of::>() + + size_of::<&mut (dyn crate::j2c::idwt::SubBandRows + Send)>(); + let locks = bands.iter().filter(|band| band.parts > 1).count() + 2; + bands.len() * per_band + + entries * size_of::, usize)>>() + + rings.len() * per_ring + + slots * size_of::<&mut [f32]>() + + (workers + readers) * per_thread + + readers * per_reader + + (workers + rayon::current_num_threads()) * SENT_WORKER_BYTES + + locks * LOCK_BYTES + } + + /// What the band executor allocated for a tile, measured under the + /// counting allocator, beside what the decode charges for it. + #[derive(Debug)] + struct Measured { + /// The most planning the routes held at once, and what they kept, + /// against what their budgets charged. + routing: (usize, usize), + routing_charge: usize, + /// The most planning the rings and bands held at once, and what it + /// kept, against the bytes charged for them and for the most beside + /// them. + planning: (usize, usize), + planning_charge: (usize, usize), + /// The most the executor held at once while the bands decoded, + /// against its charge. + running: usize, + running_charge: usize, + readers: usize, + parted: usize, + bands: usize, + deep: bool, + shallow: bool, + shared: bool, + /// The deepest finest-level and coarser-level rings that hold no + /// whole window. + slots: (usize, usize), + } + + /// Read every row of the rings that `ring_readers` gives to `readers`, + /// the first of which is reader `first`: each reader on a thread of its + /// own in a pool, and one after another outside one, where handing them + /// to the pool would allocate. + fn read_rings( + rings: &[Ring], + ring_readers: &[usize], + readers: &mut [&mut (dyn SubBandRows + Send)], + first: usize, + ) -> Result<()> { + let Some((reader, rest)) = readers.split_first_mut() else { + return Ok(()); + }; + let mut read = || -> Result<()> { + for (ring, _) in rings + .iter() + .zip(ring_readers) + .filter(|(_, reader)| **reader == first) + { + for y in ring.stored.y0..ring.stored.y1 { + reader.prepare_row(ring.sub_band, y)?; + } + } + Ok(()) + }; + if rest.is_empty() { + return read(); + } + let mut next = || read_rings(rings, ring_readers, rest, first + 1); + if rayon::current_thread_index().is_none() { + return read().and_then(|()| next()); + } + let (mine, theirs) = rayon::join(read, next); + mine.and(theirs) + } + + /// Decode the tile of `bytes`, or its `roi`, streaming its rows on the + /// current pool, with its classic bands in `parts` parts when given: + /// once to warm the bank, then, under the counting allocator, planning + /// its routes, rings and bands and running the band executor as the + /// streamed tile drives it, reading every row of every ring, each + /// component's through its own reader when it has one. + #[expect( + clippy::too_many_lines, + reason = "the planning and executor phases are measured in order under the counting allocator, on one warm bank" + )] + fn measure_executor( + bytes: &[u8], + roi: Option<(u32, u32, u32, u32)>, + parts: Option, + shared: bool, + ) -> Measured { + let image = Image::new(bytes, &DecodeSettings::default()).expect("parse fixture"); + let roi = roi.unwrap_or((0, 0, image.width(), image.height())); + let mut context = DecoderContext::default(); + context.set_shared_pool(shared); + context.set_output_region(Some(roi)); + context.tile_decode_context.line_region_idwt = true; + context.tile_decode_context.stream_rows = true; + context.tile_decode_context.band_parts = parts; + let tiles = tile::parse(&mut BitReader::new(image.codestream), &image.header, 0) + .expect("parse tile"); + let baseline = context + .reset( + &image.header, + &tiles[0], + tiles.structural_workspace_bytes(), + 0, + false, + None, + ) + .expect("prepare output planes"); + let (tile_ctx, storage) = (&mut context.tile_decode_context, &mut context.storage); + super::super::decode_tile( + &tiles[0], + &image.header, + tile_ctx, + storage, + &mut None, + CpuDecodeParallelism::Auto, + false, + &mut DecodeProfileTimings::default(), + baseline, + super::super::TileOutput::Sink(&mut CountingSink(0)), + ) + .expect("decode with room to spare"); + let components = tiles[0].component_infos.len(); + assert_eq!(tile_ctx.debug_counters.pool_band_components, components); + let mut routes = Vec::with_capacity(components); + let mut live = DecodeAllocationBudget::for_storage(storage).expect("baseline"); + let start = live.live_bytes(); + settle(); + let base = j2k_alloc_probe::live_bytes(); + j2k_alloc_probe::reset_peak(); + for component in 0..components { + let BandRouting::Route(route) = plan_band_route_beside( + &tiles[0], + component, + tile_ctx, + storage, + &image.header, + Some(live), + true, + ) + .expect("plan route") else { + panic!("the tile takes the pool's bands"); + }; + live = route.budget(); + routes.push(route); + } + let routing = ( + j2k_alloc_probe::peak_bytes() - base, + j2k_alloc_probe::live_bytes() - base, + ); + let routing_charge = live.live_bytes() - start; + let planning_charge = + tile_ring_plan_bytes(&routes, storage, &image.header).expect("planning bytes"); + let base = j2k_alloc_probe::live_bytes(); + j2k_alloc_probe::reset_peak(); + let (rings, mut bands, strip_starts, active_levels) = + plan_tile_rings(&routes, storage, &image.header).expect("plan rings"); + let planning = ( + j2k_alloc_probe::peak_bytes() - base, + j2k_alloc_probe::live_bytes() - base, + ); + let line_plans = banded_tier1::plan_tile_lines( + &tiles[0], + &image.header, + tile_ctx, + storage, + active_levels.iter().copied().map(Ok), + ) + .expect("plan lines"); + let copy = line_plans + .first() + .map_or(IntRect::from_xywh(0, 0, 0, 0), |plan| plan.copy); + let coding = routes[0].coding(); + let bank = match coding { + CodeBlockCoding::Classic => tile_ctx.classic_task_workspaces.len(), + CodeBlockCoding::HighThroughput => tile_ctx.ht_task_workspaces.len(), + }; + let route_workers = routes + .iter() + .map(|route| route.workers()) + .max() + .unwrap_or(0); + let samples = copy.width() as usize * copy.height() as usize; + let readers = if parallel_synthesis(components, coding, bank, route_workers, samples) { + components + } else { + 1 + }; + let workers = route_workers.min(bank - readers); + if coding == CodeBlockCoding::Classic { + split_bands( + tile_ctx, + storage.shared_pool_band_rings, + &rings, + &mut bands, + workers + readers, + |band, parts| { + let (route, strip) = route_strip(&routes, &strip_starts, band.strip)?; + route.strip_parts(strip, parts) + }, + ) + .expect("split bands"); + } + let running_charge = + executor_bytes(&rings, &bands, (workers, readers)).expect("executor bytes"); + let mut scratches = BlockScratch::for_threads( + workers + readers, + BandRoute::shared_scratch(routes.iter().map(|route| &**route)).expect("sizes"), + &mut DecodeAllocationBudget::for_storage(storage).expect("budget"), + ) + .expect("thread scratch"); + let helps = readers == 1 && coding == CodeBlockCoding::Classic; + let mut coefficients = core::mem::take(&mut storage.coefficients); + // Each component's rings read through its own reader, when it has + // one. + let mut ring_readers = Vec::new(); + for (component_idx, route) in routes.iter().enumerate() { + let (component_rings, _) = + ring_counts(route, storage, &image.header, component_idx).expect("rings"); + let reader = if readers == components { + component_idx + } else { + 0 + }; + ring_readers.extend(core::iter::repeat_n(reader, component_rings)); + } + let read = |readers: &mut [&mut (dyn SubBandRows + Send)]| -> Result<()> { + read_rings(&rings, &ring_readers, readers, 0) + }; + let base = j2k_alloc_probe::live_bytes(); + j2k_alloc_probe::reset_peak(); + let slots = split_slots(&rings, &bands, storage, &mut coefficients).expect("slots"); + let source = PayloadSource::new(storage); + let decoded = match coding { + CodeBlockCoding::Classic => run_bands( + (workers, readers, helps), + &rings, + &bands, + slots, + (&mut tile_ctx.classic_task_workspaces, &mut scratches), + |strip, output, workspace, scratch| { + let (route, strip) = route_strip(&routes, &strip_starts, strip)?; + route.decode_classic(strip, output, workspace, (scratch, source)) + }, + read, + false, + &mut 0, + None, + ), + CodeBlockCoding::HighThroughput => run_bands( + (workers, readers, helps), + &rings, + &bands, + slots, + (&mut tile_ctx.ht_task_workspaces, &mut scratches), + |strip, output, workspace, scratch| { + let (route, strip) = route_strip(&routes, &strip_starts, strip)?; + route.decode_ht(strip, output, workspace, (scratch, source)) + }, + read, + false, + &mut 0, + None, + ), + }; + let running = j2k_alloc_probe::peak_bytes() - base; + assert_eq!(decoded.ok(), Some(true), "the executor decodes every band"); + storage.coefficients = coefficients; + Measured { + routing, + routing_charge, + planning, + planning_charge, + running, + running_charge, + readers, + parted: bands.iter().filter(|band| band.parts > 1).count(), + bands: bands.len(), + deep: storage.deep_band_rings, + shallow: storage.shallow_band_rings, + shared: storage.shared_pool_band_rings, + slots: deepest_rings(&rings), + } + } + + /// The most slots of a finest-level ring, and of a coarser-level one, + /// that holds no whole window. + fn deepest_rings(rings: &[Ring]) -> (usize, usize) { + let deepest = |finest: bool| { + rings + .iter() + .filter(|ring| ring.finest == finest && !ring.whole_window) + .map(|ring| ring.slots) + .max() + .unwrap_or(0) + }; + (deepest(true), deepest(false)) + } + + /// Whether this process is the one that measures the test `name`: the + /// allocator's high-water mark is the process's, so when it is not, + /// this runs the test alone in a process of its own and checks that it + /// passed there. + fn measures_alone(name: &str) -> bool { + const PROBE: &str = "J2K_NATIVE_BAND_EXECUTOR_PROBE"; + if std::env::var_os(PROBE).is_some() { + return true; + } + let module = module_path!() + .split_once("::") + .map(|(_, path)| path) + .expect("module path"); + let output = std::process::Command::new(std::env::current_exe().expect("test binary")) + .args([ + &format!("{module}::{name}"), + "--exact", + "--test-threads=1", + "--nocapture", + ]) + .env(PROBE, "1") + .output() + .expect("run the measuring process"); + let report = format!( + "{}{}", + String::from_utf8_lossy(&output.stdout), + String::from_utf8_lossy(&output.stderr) + ); + assert!(output.status.success(), "{report}"); + assert!(report.contains("1 passed"), "{report}"); + false + } + + /// Under the counting allocator, the whole-window tile route, entered + /// from a thread outside the pool that never entered it before, holds + /// at most what it charges for its plan, tasks and threads' scratch, + /// the locks it shares and what the pool takes for it. + #[test] + fn tile_route_allocates_within_its_charge() { + use core::sync::atomic::{AtomicBool, Ordering}; + + static ROUTED: AtomicBool = AtomicBool::new(false); + if !measures_alone("tile_route_allocates_within_its_charge") { + return; + } + rayon::ThreadPoolBuilder::new() + .num_threads(4) + .build_global() + .expect("the measuring process's first use of the global pool"); + rayon::broadcast(|_| ()); + let bytes = budget_fixture(203, 157, 1, 2); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse fixture"); + let mut context = DecoderContext::default(); + let tiles = tile::parse(&mut BitReader::new(image.codestream), &image.header, 0) + .expect("parse tile"); + let baseline = context + .reset( + &image.header, + &tiles[0], + tiles.structural_workspace_bytes(), + 0, + false, + None, + ) + .expect("prepare output planes"); + let (tile_ctx, storage) = (&mut context.tile_decode_context, &mut context.storage); + // Warm the bank on this thread, so the route reuses it. + super::super::decode_tile( + &tiles[0], + &image.header, + tile_ctx, + storage, + &mut None, + CpuDecodeParallelism::Auto, + false, + &mut DecodeProfileTimings::default(), + baseline, + super::super::TileOutput::Planes, + ) + .expect("decode with room to spare"); + tile_ctx.tile_route_post_decode = Some(|_| { + ROUTED.store(true, Ordering::Relaxed); + Ok(()) + }); + let retained = tile_ctx.tier1_capacity_bytes().expect("bank bytes"); + tile_ctx.debug_counters.workspace_peak = WorkspacePeak::default(); + let held = std::thread::scope(|scope| { + scope + .spawn(|| { + settle(); + let base = j2k_alloc_probe::live_bytes(); + j2k_alloc_probe::reset_peak(); + super::super::subband::decode_component_tile_bit_planes( + &tiles[0], + 0..1, + tile_ctx, + storage, + &image.header, + &mut None, + CpuDecodeParallelism::Auto, + false, ) - }; - let headroom = route_bytes + executor / 2; - assert!(headroom >= executor, "each owner must fit separately"); - let baseline = DecodeAllocationBudget::for_storage(storage) - .expect("control budget") - .live_bytes(); - storage.structural_workspace_bytes += - DEFAULT_MAX_DECODE_BYTES - baseline - headroom; - let route = match plan_band_route(&tiles[0], 0, tile_ctx, storage, &image.header) { - Ok(BandRouting::Route(route)) => route, - Ok(BandRouting::Serial | BandRouting::WholeWindows) => return, - Err(error) if super::super::reuse::is_capacity_error(&error) => return, - Err(error) => panic!("unexpected planning error: {error:?}"), - }; - let result = decode_route( - &route, - 0, - &image.header, - tile_ctx, - storage, - tiles[0].component_infos[0].wavelet_transform(), - false, - &mut 0, + .expect("decode the tile"); + j2k_alloc_probe::peak_bytes() - base + }) + .join() + .expect("the measuring thread") + }); + assert!(ROUTED.load(Ordering::Relaxed), "the tile route decodes it"); + let charged = tile_ctx.debug_counters.workspace_peak.bytes().tier1 - retained; + assert!( + held <= charged, + "the tile route held {held} of {charged} bytes" + ); + } + + /// Wait until the process has held the same bytes for 100 ms, so the + /// pools' threads have gone idle: a thread allocates its queue's state + /// and the locks it sleeps on when it first idles. + fn settle() { + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(30); + let (mut last, mut still) = (j2k_alloc_probe::live_bytes(), 0); + while still < 20 { + assert!( + std::time::Instant::now() < deadline, + "the process's allocations never settled" + ); + std::thread::sleep(std::time::Duration::from_millis(5)); + let now = j2k_alloc_probe::live_bytes(); + still = if now == last { still + 1 } else { 0 }; + last = now; + } + } + + /// Pools of `threads` threads each, every thread of which has run a + /// job, so none allocates its first job's state inside a measurement. + fn warmed_pools(threads: &[usize]) -> Vec { + threads + .iter() + .map(|&threads| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + pool.broadcast(|_| ()); + pool + }) + .collect() + } + + /// Under the counting allocator, the band executor holds at most what + /// the decode charges for it, planning its routes, rings and bands and + /// while they decode: for classic and HTJ2K tiles, whole and in a + /// region, with bands in parts, on pools of 2, 4, 10, 11 and 12 threads + /// and from outside a pool, with one reader and with one per component, + /// with shallow rings and deep ones, with the shared-pool mark and + /// without. + #[test] + fn band_executor_allocates_within_its_charge() { + if !measures_alone("band_executor_allocates_within_its_charge") { + return; + } + // The decode from outside a pool streams on the global pool, of + // enough threads whatever the machine. + rayon::ThreadPoolBuilder::new() + .num_threads(12) + .build_global() + .expect("the measuring process's first use of the global pool"); + let classic = include_bytes!("../../../fixtures/multi_segment/bypass_termall_rgb_53.j2k"); + let ht = ht_fixture(); + let wide = shallow_fixture(); + let (mut readers, mut parted, mut clipped, mut deep) = (Vec::new(), 0, false, false); + let (mut shallow, mut shared) = (Vec::new(), Vec::new()); + let mut check = |case: &str, measured: Measured, whole: &Measured| { + assert!( + measured.routing.0 <= measured.routing_charge, + "{case}: {measured:?}" + ); + let (peak, kept) = measured.planning; + let (planned, beside) = measured.planning_charge; + assert!( + kept <= planned && peak <= planned + beside, + "{case}: {measured:?}" + ); + assert!( + measured.running <= measured.running_charge, + "{case}: {measured:?}" + ); + if measured.shared { + assert!( + measured.slots.0 <= 2 && measured.slots.1 <= 2, + "{case}: shared rings" ); - match result { - Err(error) if super::super::reuse::is_capacity_error(&error) => {} - Ok(false) => {} - other => panic!( - "pool route must decline or reject combined live owners: \ - headroom={headroom}, route={route_bytes}, executor={executor}, \ - excess={}, result={other:?}", - route_bytes + executor - headroom, - ), + shared.push((case.to_owned(), measured.slots, measured.deep)); + } else if measured.shallow { + assert_eq!(measured.slots.0, 2, "{case}: shallow finest rings"); + shallow.push((case.to_owned(), measured.slots, measured.parted)); + } + readers.push(measured.readers); + parted += measured.parted; + clipped |= measured.bands < whole.bands; + deep |= measured.deep; + }; + // The pools, and the global one, stay up until every measurement + // ends, so no pool's teardown frees bytes inside another's window. + // Every thread of each runs a job before any measurement, and every + // measurement waits until they idle (see `settle`). + let pools = warmed_pools(&[2, 4, 10, 11, 12]); + rayon::broadcast(|_| ()); + for pool in &pools { + let threads = pool.current_num_threads(); + pool.install(|| { + for (label, bytes, splits) in [ + ("classic", &classic[..], &[None, Some(4)][..]), + ("HTJ2K", &ht[..], &[None][..]), + ("wide classic", &wide[..], &[None, Some(4)][..]), + ] { + let whole = measure_executor(bytes, None, None, false); + for &roi in &[None, Some((48, 48, 16, 16))] { + for &parts in splits { + for shared in [false, true] { + let case = format!( + "{label} {roi:?} parts {parts:?} shared {shared} on {threads}" + ); + check(&case, measure_executor(bytes, roi, parts, shared), &whole); + } + } + } } }); + } + // Called from outside a pool, the executor sends its workers through + // the global pool's shared queue. + let whole = measure_executor(classic, None, None, false); + check( + "classic outside the pool", + measure_executor(classic, None, Some(4), false), + &whole, + ); + drop(pools); + assert!(readers.contains(&1) && readers.contains(&3), "{readers:?}"); + assert!( + parted > 0 && clipped && deep, + "parts {parted}, region {clipped}, deep {deep}" + ); + // The wide tile at full resolution on 10 and 11 threads holds the + // shallow finest rings, and its bands split into parts by themselves. + assert!( + shallow + .iter() + .any(|(case, slots, parted)| case.contains("parts None") + && slots.1 == 2 + && *parted > 0), + "{shallow:?}" + ); + // Marked decodes hold two bands in every ring below 12 threads, and + // never with the deep rings, which hold from 12. + assert!( + shared.iter().any(|(_, slots, _)| *slots == (2, 2)) + && shared.iter().all(|(_, _, deep_rings)| !deep_rings), + "{shared:?}" + ); + } + + /// A classic RGB tile of noise wide and tall enough for the shallow + /// finest rings: a reader per component on 10 threads (the unit tests' + /// sample gate), finest-level bands of several blocks, and coarser + /// levels with more bands than slots. + fn shallow_fixture() -> Vec { + let mut state = 0x9e37_79b9_u32; + let pixels: Vec = (0..384 * 192 * 3) + .map(|_| { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + state.to_le_bytes()[3] + }) + .collect(); + encode( + &pixels, + 384, + 192, + 3, + 8, + false, + &EncodeOptions { + num_decomposition_levels: 4, + code_block_height_exp: 2, + ..EncodeOptions::default() + }, + ) + .expect("encode shallow ring fixture") + } + + /// A three-component 64 x 64 HTJ2K image in 16 x 16 code-blocks. + fn ht_fixture() -> Vec { + let mut state = 0x2545_f491_u32; + let pixels: Vec = (0..64 * 64 * 3) + .map(|_| { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + state.to_le_bytes()[3] + }) + .collect(); + encode( + &pixels, + 64, + 64, + 3, + 8, + false, + &EncodeOptions { + reversible: true, + num_decomposition_levels: 2, + code_block_width_exp: 2, + code_block_height_exp: 2, + use_ht_block_coding: true, + ..EncodeOptions::default() + }, + ) + .expect("encode HTJ2K fixture") } } diff --git a/crates/j2k-native/src/j2c/decode/planes_floor.rs b/crates/j2k-native/src/j2c/decode/planes_floor.rs new file mode 100644 index 000000000..27b2b5a90 --- /dev/null +++ b/crates/j2k-native/src/j2c/decode/planes_floor.rs @@ -0,0 +1,233 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! What a full-image decode into component planes charges the decode budget +//! at least, planned from the codestream before anything is decoded. +//! +//! The planes decode charges, in order: the parsed tiles' structural bytes +//! and every component's image-sized plane (`TileDecodeContext::reset`), then +//! for each tile its code-block records, the coefficients of its largest +//! component and that component's IDWT workspace (`build`'s allocation plan, +//! with the component coefficient layout). Each charge here is the same or +//! smaller, so a decode whose floor passes the budget is refused on the +//! planes, serially or on tile workers, and a decode the planes can finish +//! never has a floor past the budget. The floor leaves out what `build` +//! counts by walking precincts (most tag-tree nodes) and every later stage. + +use core::mem::size_of; + +use crate::checked_decode_sample_count; +use crate::error::{Result, ValidationError}; +use crate::j2c::build::{global_code_block_count, IdwtWorkspaceTracker, RecordCounts, SubBandType}; +use crate::j2c::codestream::Header; +use crate::j2c::rect::IntRect; +use crate::j2c::tile::{self, ResolutionTile, Tile}; +use crate::j2c::ComponentData; +use crate::math::{SimdBuffer, SIMD_WIDTH}; +use crate::reader::BitReader; + +/// The bytes a full-image decode of `data` into component planes charges at +/// least, when its full-image region decode could stream its rows instead: +/// the image streams as far as its main header says (the caller checks +/// `Image::region_decode_streams_rows`), and every tile has its components +/// at full sampling and an inverse MCT only over three or more components +/// whose first three share a wavelet (as `streams_rows` requires). `None` +/// when it could not, or when the header alone puts the planes, one tile's +/// coefficients and its IDWT buffers within half of `budget`, which leaves +/// the planes decode far from it unless its metadata is unusually large. +/// +/// `retained_image_bytes` is what the image holds while it decodes, the +/// tile parse's baseline. Errors are the tile parse's, which the planes +/// decode reports too. +pub(crate) fn full_decode_planes_floor( + data: &[u8], + header: &Header<'_>, + retained_image_bytes: usize, + budget: usize, +) -> Result> { + let planes = planes_bytes(header)?; + if header_estimate(header, planes).saturating_mul(2) <= budget { + return Ok(None); + } + let mut reader = BitReader::new(data); + let tiles = tile::parse(&mut reader, header, retained_image_bytes)?; + if tiles.is_empty() { + return Ok(None); + } + let skipped = header.skipped_resolution_levels; + let mut tile_floor = 0_usize; + for tile in tiles.iter() { + if !tile_streams_rows(tile, header.component_infos.len()) { + return Ok(None); + } + tile_floor = tile_floor.max(tile_workspace_floor(tile, skipped)); + } + Ok(Some( + tiles + .structural_workspace_bytes() + .saturating_add(planes) + .saturating_add(tile_floor), + )) +} + +/// What `reset` charges for every component's plane of the output image. +fn planes_bytes(header: &Header<'_>) -> Result { + region_planes_bytes( + header, + header.size_data.image_width(), + header.size_data.image_height(), + ) +} + +/// What `reset` charges for every component's plane of a `width` x +/// `height` output. +pub(super) fn region_planes_bytes(header: &Header<'_>, width: u32, height: u32) -> Result { + let samples = checked_decode_sample_count(width, height)?; + let padded = + SimdBuffer::::padded_len(samples).ok_or(ValidationError::ImageTooLarge)?; + let components = header.component_infos.len(); + Ok(padded + .saturating_mul(size_of::()) + .saturating_mul(components) + .saturating_add(components.saturating_mul(size_of::()))) +} + +/// The planes, plus the coefficients and twice the output-resolution +/// samples of one SIZ tile in `f32`: about what the planes decode charges +/// for its largest tile, the IDWT's output, next-level scratch and band +/// scratch included, without records or metadata. +fn header_estimate(header: &Header<'_>, planes: usize) -> usize { + let size = &header.size_data; + let full = |tile: u32, extent: u32, offset: u32| tile.min(extent.saturating_sub(offset)); + let width = full( + size.tile_width, + size.reference_grid_width, + size.tile_x_offset, + ); + let height = full( + size.tile_height, + size.reference_grid_height, + size.tile_y_offset, + ); + let reduced = |length: u32| { + u32::try_from(u64::from(length) >> header.skipped_resolution_levels) + .unwrap_or(u32::MAX) + .saturating_add(2) + }; + let full_samples = (width as usize).saturating_mul(height as usize); + let output_samples = (reduced(width) as usize).saturating_mul(reduced(height) as usize); + planes.saturating_add( + full_samples + .saturating_add(output_samples.saturating_mul(2)) + .saturating_mul(size_of::()), + ) +} + +/// The tile-level conditions of `streams_rows` that `tile`'s header +/// settles. +fn tile_streams_rows(tile: &Tile<'_>, components: usize) -> bool { + let infos = &tile.component_infos; + infos.len() == components + && infos.iter().all(|component| { + component.size_info.horizontal_resolution == 1 + && component.size_info.vertical_resolution == 1 + && !component.requires_exact_integer_decode() + }) + && (!tile.mct + || (components >= 3 + && infos[1..3].iter().all(|component| { + component.wavelet_transform() == infos[0].wavelet_transform() + }))) +} + +/// What `build` charges for `tile` at least beside the planes: its records, +/// its largest component's `f32` coefficients at full resolution, and the +/// largest component IDWT workspace at the output resolution. +fn tile_workspace_floor(tile: &Tile<'_>, skipped: u8) -> usize { + let mut records = RecordCounts::default(); + let mut coefficients = 0_usize; + let mut idwt = 0_usize; + for component_tile in tile.component_tiles() { + coefficients = coefficients.max(rect_samples(component_tile.rect)); + records.tile_decompositions = records.tile_decompositions.saturating_add(1); + let active = usize::from( + component_tile + .component_info + .num_decomposition_levels() + .saturating_sub(skipped), + ); + let mut idwt_tracker = IdwtWorkspaceTracker::new( + Some(active), + component_tile.component_info.wavelet_transform(), + ); + let mut ll_rect = IntRect::from_xywh(0, 0, 0, 0); + for resolution_tile in component_tile.resolution_tiles() { + if resolution_tile.resolution == 0 { + ll_rect = resolution_tile.sub_band_rect(SubBandType::LowLow); + add_sub_band_records( + &mut records, + &resolution_tile, + SubBandType::LowLow, + tile.num_layers, + ); + continue; + } + records.decompositions = records.decompositions.saturating_add(1); + idwt_tracker.observe(resolution_tile.rect); + for sub_band_type in [ + SubBandType::HighLow, + SubBandType::LowHigh, + SubBandType::HighHigh, + ] { + add_sub_band_records( + &mut records, + &resolution_tile, + sub_band_type, + tile.num_layers, + ); + } + } + idwt = idwt.max( + idwt_tracker + .finish(ll_rect) + .ok() + .flatten() + .unwrap_or(usize::MAX), + ); + } + let records_bytes = records.checked_bytes().unwrap_or(usize::MAX); + records_bytes.saturating_add( + coefficients + .saturating_add(idwt) + .saturating_mul(size_of::()), + ) +} + +fn rect_samples(rect: IntRect) -> usize { + (rect.width() as usize).saturating_mul(rect.height() as usize) +} + +/// Add one sub-band's records to the floor, counting tag-tree nodes only as +/// the two trees of each precinct need at least: a root and one leaf per +/// code-block. +fn add_sub_band_records( + records: &mut RecordCounts, + resolution_tile: &ResolutionTile<'_>, + sub_band_type: SubBandType, + layers: u8, +) { + let precincts = usize::try_from(resolution_tile.num_precincts()).unwrap_or(usize::MAX); + let code_blocks = global_code_block_count( + resolution_tile, + resolution_tile.sub_band_rect(sub_band_type), + ) + .unwrap_or(usize::MAX); + records.sub_bands = records.sub_bands.saturating_add(1); + records.precincts = records.precincts.saturating_add(precincts); + records.code_blocks = records.code_blocks.saturating_add(code_blocks); + records.layers = records + .layers + .saturating_add(code_blocks.saturating_mul(usize::from(layers))); + records.tag_tree_nodes = records + .tag_tree_nodes + .saturating_add(precincts.max(code_blocks).saturating_mul(2)); +} diff --git a/crates/j2k-native/src/j2c/decode/reuse.rs b/crates/j2k-native/src/j2c/decode/reuse.rs index ea05e8cba..6effc87ab 100644 --- a/crates/j2k-native/src/j2c/decode/reuse.rs +++ b/crates/j2k-native/src/j2c/decode/reuse.rs @@ -159,6 +159,17 @@ impl DecoderContext<'_> { } impl TileDecodeContext { + /// The decode budget the component planes are charged against. + pub(super) fn planes_budget(&self) -> usize { + #[cfg(test)] + return self.planes_budget.unwrap_or(DEFAULT_MAX_DECODE_BYTES); + #[cfg(not(test))] + { + let _ = self; + DEFAULT_MAX_DECODE_BYTES + } + } + pub(crate) fn retained_channel_bytes(&self) -> Result { retained_channel_bytes_with_cap(&self.channel_data, DEFAULT_MAX_DECODE_BYTES) } @@ -245,7 +256,8 @@ impl TileDecodeContext { let fresh_baseline = retained_baseline_bytes .checked_sub(retained_channel_bytes) .ok_or_else(allocation_overflow)?; - let mut budget = ContextCapacityBudget::from_live_bytes(retained_baseline_bytes)?; + let cap = self.planes_budget(); + let mut budget = ContextCapacityBudget::with_cap(retained_baseline_bytes, cap)?; let reused = reset_channel_data( &mut self.channel_data, &initial_tile.component_infos, @@ -258,7 +270,7 @@ impl TileDecodeContext { Ok(deferred) => (deferred, budget.bytes()), Err(error) if retained_channel_bytes != 0 && is_capacity_error(&error) => { self.release_channel_allocations(); - let mut fresh_budget = ContextCapacityBudget::from_live_bytes(fresh_baseline)?; + let mut fresh_budget = ContextCapacityBudget::with_cap(fresh_baseline, cap)?; let deferred = reset_channel_data( &mut self.channel_data, &initial_tile.component_infos, @@ -514,10 +526,6 @@ struct ContextCapacityBudget { } impl ContextCapacityBudget { - fn from_live_bytes(bytes: usize) -> Result { - Self::with_cap(bytes, DEFAULT_MAX_DECODE_BYTES) - } - const fn with_cap(bytes: usize, cap: usize) -> Result { if bytes > cap { return Err(DecodeError::AllocationTooLarge { diff --git a/crates/j2k-native/src/j2c/decode/state.rs b/crates/j2k-native/src/j2c/decode/state.rs index 948d376d1..c05947f36 100644 --- a/crates/j2k-native/src/j2c/decode/state.rs +++ b/crates/j2k-native/src/j2c/decode/state.rs @@ -121,8 +121,28 @@ pub(crate) struct DecompositionStorage<'a> { /// IDWT on the worker pool (see [`parallel_bands`]). Each sub-band's /// `coefficients` then hold a ring of bands. pub(crate) parallel_bands: bool, - /// Whether every component's IDWT rings, and a scratch row each, are - /// live at once, because the tile streams its rows to a sink. + /// Whether the band rings of `parallel_bands` hold one band more at + /// every level, for a classic tile that streams whole rows at full + /// resolution from a reader per component (see + /// [`parallel_bands::deep_rings`] and [`parallel_bands::reads_whole_rows`]). + pub(crate) deep_band_rings: bool, + /// Whether the finest level's band rings of `parallel_bands` hold one + /// band fewer, for a classic tile that streams full-width rows from a + /// reader per component without deep rings (see + /// [`parallel_bands::shallow_rings`]). + pub(crate) shallow_band_rings: bool, + /// Whether the band rings of `parallel_bands` hold two bands at every + /// level and their bands stay whole, for a decode whose pool other images + /// share, on a pool too small for deep rings (see + /// [`parallel_bands::shared_rings`]). + pub(crate) shared_pool_band_rings: bool, + /// Whether a tile streaming its rows from `parallel_bands` reconstructs + /// them through one reader whatever its pool's workers (see + /// [`parallel_bands::one_reader`]), which finishes each row in the IDWT's + /// rings and holds no batch of rows. + pub(crate) one_synthesis_reader: bool, + /// Whether every component's IDWT rings are live at once, because the + /// tile streams its rows to a sink, finishing them in the rings. pub(crate) lockstep_lines: bool, /// The f32 samples of IDWT workspace the ROI plan charged for this tile. pub(crate) roi_idwt_charge_elements: usize, @@ -162,6 +182,10 @@ impl DecompositionStorage<'_> { self.line_region_idwt = false; self.banded_tier1 = false; self.parallel_bands = false; + self.deep_band_rings = false; + self.shallow_band_rings = false; + self.shared_pool_band_rings = false; + self.one_synthesis_reader = false; self.lockstep_lines = false; self.roi_idwt_charge_elements = 0; self.tier1_band = None; @@ -230,6 +254,10 @@ pub(crate) struct TileDecodeContext { pub(crate) bit_plane_decode_buffers: BitPlaneDecodeBuffers, /// A reusable context for decoding HTJ2K code blocks. pub(crate) ht_block_decode_context: HtBlockDecodeContext, + /// Scratch an HTJ2K code-block that the sub-band stores only part of + /// decodes into before its stored part is copied out, sized with the + /// serial Tier-1 workspace. + pub(crate) ht_clipped_block_scratch: Vec, #[cfg(feature = "parallel")] pub(in crate::j2c::decode) ht_task_workspaces: Vec, /// One classic workspace per worker of the tile-wide route. @@ -258,6 +286,18 @@ pub(crate) struct TileDecodeContext { /// band is left running with no worker to finish it. #[cfg(all(test, feature = "parallel"))] pub(crate) strand_pool_band: bool, + /// Test-only: with `strand_pool_band`, the band task started after this + /// many others is the one abandoned. + #[cfg(all(test, feature = "parallel"))] + pub(crate) strand_pool_band_after: usize, + /// Test-only: a streamed tile's band decode after this many fails as a + /// corrupt code-block's would, on serial and pool bands. + #[cfg(test)] + pub(crate) fail_band_after: Option, + /// Test-only: each pool band splits into this many parts, as far as its + /// blocks allow, whatever the pool's size. + #[cfg(all(test, feature = "parallel"))] + pub(crate) band_parts: Option, /// Test-only: the average work per band below which a component's /// bands decode over whole windows instead of on the pool; zero, so /// unit tests' tiny fixtures take the pool, unless a test sets it. @@ -285,6 +325,12 @@ pub(crate) struct TileDecodeContext { /// their per-thread counters see the decode. #[cfg(test)] pub(crate) enter_pool: bool, + /// Test-only decode budget for the component planes, in place of the + /// default: `reset` charges them against it, and a region decode + /// compares them with it when it chooses whether its rows stream from + /// serial bands (see `choose_region_idwt`). + #[cfg(test)] + pub(crate) planes_budget: Option, /// The raw, decoded samples for each channel. pub(crate) channel_data: Vec, /// Optional output window for region-local decode storage. @@ -294,6 +340,9 @@ pub(crate) struct TileDecodeContext { /// Whether the inverse MCT is left to the caller's RGB8 pack, set /// around a decode like `round_irreversible_output`. pub(crate) defer_mct: bool, + /// Whether this context's decodes share their pool with other images + /// decoding at the same time (see `DecoderContext::set_shared_pool`). + pub(crate) shared_pool: bool, /// The inverse MCT this decode left to the caller. pub(crate) deferred_mct: Option, /// The last IDWT level this decode leaves to the RGB8 pack with the @@ -350,6 +399,8 @@ impl TileDecodeContext { let serial_small_decodes = self.serial_small_decodes; #[cfg(test)] let enter_pool = self.enter_pool; + #[cfg(test)] + let planes_budget = self.planes_budget; *self = Self::default(); self.output_region = output_region; self.round_irreversible_output = round_irreversible_output; @@ -369,6 +420,7 @@ impl TileDecodeContext { self.staged_rgb8_pack = staged_rgb8_pack; self.serial_small_decodes = serial_small_decodes; self.enter_pool = enter_pool; + self.planes_budget = planes_budget; } } @@ -381,6 +433,7 @@ impl TileDecodeContext { self.bit_plane_decode_context = BitPlaneDecodeContext::default(); self.bit_plane_decode_buffers = BitPlaneDecodeBuffers::default(); self.ht_block_decode_context = HtBlockDecodeContext::default(); + self.ht_clipped_block_scratch = Vec::new(); #[cfg(feature = "parallel")] { self.ht_task_workspaces = Vec::new(); @@ -393,9 +446,15 @@ impl TileDecodeContext { let classic_bytes = self.bit_plane_decode_context.allocated_bytes()?; let buffer_bytes = self.bit_plane_decode_buffers.allocated_bytes()?; let ht_bytes = self.ht_block_decode_context.allocated_bytes()?; + let clipped_bytes = self + .ht_clipped_block_scratch + .capacity() + .checked_mul(size_of::()) + .ok_or(ValidationError::ImageTooLarge)?; classic_bytes .checked_add(buffer_bytes) .and_then(|bytes| bytes.checked_add(ht_bytes)) + .and_then(|bytes| bytes.checked_add(clipped_bytes)) .ok_or(ValidationError::ImageTooLarge.into()) } diff --git a/crates/j2k-native/src/j2c/decode/subband.rs b/crates/j2k-native/src/j2c/decode/subband.rs index a8405a236..5c37ca1d1 100644 --- a/crates/j2k-native/src/j2c/decode/subband.rs +++ b/crates/j2k-native/src/j2c/decode/subband.rs @@ -44,7 +44,8 @@ pub(crate) use self::placement::BlockPlacement; pub(crate) use self::tile_parallel::MIN_BAND_WORK; #[cfg(feature = "parallel")] pub(in crate::j2c::decode) use self::tile_parallel::{ - plan_band_route, routes_tile, BandRoute, BandRouting, + plan_band_route, plan_band_route_beside, pool_entry_bytes, routes_tile, BandOutput, BandRoute, + BandRouting, BlockScratch, PayloadSource, LOCK_BYTES, }; /// Decode the code-blocks of the tile's `components` into their sub-bands. @@ -233,7 +234,7 @@ pub(super) fn decode_sub_band_bitplanes( } /// Where `code_block` lands in `sub_band`'s stored rectangle. -fn stored_placement(code_block: &CodeBlock, sub_band: &SubBand) -> BlockPlacement { +pub(super) fn stored_placement(code_block: &CodeBlock, sub_band: &SubBand) -> BlockPlacement { BlockPlacement::new(code_block.rect, sub_band.storage_rect) } diff --git a/crates/j2k-native/src/j2c/decode/subband/classic.rs b/crates/j2k-native/src/j2c/decode/subband/classic.rs index 0cdf67373..0ea09f25f 100644 --- a/crates/j2k-native/src/j2c/decode/subband/classic.rs +++ b/crates/j2k-native/src/j2c/decode/subband/classic.rs @@ -85,10 +85,6 @@ impl ClassicJobParameters { } } -#[expect( - clippy::cast_precision_loss, - reason = "the codec float domain intentionally receives bounded integer samples at this rounding boundary" -)] #[expect( clippy::too_many_arguments, reason = "the Tier-1 boundary keeps geometry, state buffers, and validated options explicit" @@ -332,31 +328,17 @@ pub(super) fn decode_sub_band_classic_blocks( code_block.number_of_coding_passes, component_info.roi_shift, ); - let exact = !irreversible_midpoint - && component_info.roi_shift == 0 - && midpoint.reversible_is_exact(); placement.store_rows( tile_ctx.bit_plane_decode_context.coefficient_rows(), base_store, output_stride, |coefficients, out_row| { - if irreversible_midpoint { - midpoint.dequantize_row(coefficients, out_row, dequantization_step); - } else if exact { - bitplane::MidpointReconstructor::scale_exact_reversible_row( - coefficients, - out_row, - dequantization_step, - ); - } else { - for (output, coefficient) in out_row.iter_mut().zip(coefficients) { - *output = apply_roi_maxshift_inverse_i64( - midpoint.reconstruct_reversible(*coefficient), - component_info.roi_shift, - ) as f32 - * dequantization_step; - } - } + midpoint.reconstruct_row( + coefficients, + out_row, + dequantization_step, + irreversible_midpoint, + ); }, )?; } diff --git a/crates/j2k-native/src/j2c/decode/subband/ht.rs b/crates/j2k-native/src/j2c/decode/subband/ht.rs index c652f3671..dbf1ac738 100644 --- a/crates/j2k-native/src/j2c/decode/subband/ht.rs +++ b/crates/j2k-native/src/j2c/decode/subband/ht.rs @@ -289,8 +289,6 @@ pub(super) fn decode_sub_band_ht_blocks( } } - #[cfg(feature = "simd")] - let mut clipped_block_scratch = Vec::new(); for precinct in sub_band .precincts .clone() @@ -334,7 +332,7 @@ pub(super) fn decode_sub_band_ht_blocks( let mut budget = DecodeAllocationBudget::for_storage(storage)?; ( super::placement::zeroed_block_scratch( - &mut clipped_block_scratch, + &mut tile_ctx.ht_clipped_block_scratch, code_block.rect.width(), code_block.rect.height(), &mut budget, @@ -369,7 +367,7 @@ pub(super) fn decode_sub_band_ht_blocks( } if !whole { placement.copy_from_block( - &clipped_block_scratch, + &tile_ctx.ht_clipped_block_scratch, code_block.rect.width(), &mut storage.coefficients[sub_band.coefficients.clone()], stored_stride as usize, diff --git a/crates/j2k-native/src/j2c/decode/subband/tile_parallel.rs b/crates/j2k-native/src/j2c/decode/subband/tile_parallel.rs index 0d3b8f90c..d0ee87264 100644 --- a/crates/j2k-native/src/j2c/decode/subband/tile_parallel.rs +++ b/crates/j2k-native/src/j2c/decode/subband/tile_parallel.rs @@ -8,9 +8,12 @@ //! blocks owns a contiguous range of its sub-band's coefficients, so tasks //! write their output in place through disjoint slices and Rayon balances //! them with work stealing. HT cleanup-only and single-segment classic -//! payloads are borrowed from the codestream; only payloads that span several -//! segments are gathered. Classic blocks take this route when each decodes as -//! one arithmetic segment: no selective bypass or per-pass termination. +//! payloads are borrowed from the codestream; a payload that spans several +//! segments is gathered from the block's segment records into its worker's +//! scratch just before the block decodes. A classic block of several +//! codeword segments (selective bypass, per-pass termination) collects their +//! descriptors from the same records into its worker's scratch as its +//! payload is borrowed or gathered. use alloc::boxed::Box; use alloc::vec::Vec; @@ -28,16 +31,20 @@ use super::{ J2kCodeBlockDecodeWorkspace, Result, Tile, TileDecodeContext, MAX_BITPLANE_COUNT, }; use crate::error::{DecodingError, ValidationError}; -use crate::j2c::build::{CodeBlock, CodeBlockCoding}; +use crate::j2c::build::{CodeBlock, CodeBlockCoding, Layer, Segment}; use crate::j2c::codestream::CodeBlockStyle; -use crate::j2c::decode::direct_plan::classic_allocation_counts; +use crate::j2c::decode::direct_plan::{ + classic_allocation_counts, extend_reserved, push_reserved, walk_classic_segments, +}; use crate::j2c::decode::peak::WorkspaceBytes; use crate::j2c::decode::workspace::{task_workspace_bytes, TaskScratch, TaskWorkspace}; use crate::j2c::ht_block_decode::selected_code_block_segment_lengths; use crate::j2c::tile::{ComponentTile, ResolutionTile}; -use crate::scalar::decode_ht_code_block_scalar_with_workspace_midpoint; +use crate::scalar::{ + decode_ht_code_block_scalar_with_workspace_midpoint, internal_j2k_code_block_style, +}; use crate::{ - decode_ht_code_block_scalar_with_workspace, try_resize_decode_elements, + decode_ht_code_block_scalar_with_workspace, try_reserve_decode_elements, HtCodeBlockDecodeWorkspace, J2kCodeBlockSegment, }; @@ -126,8 +133,226 @@ struct BandParameters { enum Payload<'a> { /// A single segment borrowed from the codestream. Borrowed(&'a [u8]), - /// Gathered segments starting at this offset of the payload arena. - Gathered(usize), + /// Segments gathered from the block's records into the decoding + /// worker's scratch. + Gathered, +} + +/// The records a gathered payload is read from while blocks decode: every +/// code-block, layer and segment of the tile, borrowed from its storage. +#[derive(Clone, Copy)] +pub(in crate::j2c::decode) struct PayloadSource<'s, 'a> { + code_blocks: &'s [CodeBlock], + layers: &'s [Layer], + segments: &'s [Segment<'a>], +} + +impl<'s, 'a> PayloadSource<'s, 'a> { + pub(in crate::j2c::decode) fn new(storage: &'s DecompositionStorage<'a>) -> Self { + Self { + code_blocks: &storage.code_blocks, + layers: &storage.layers, + segments: &storage.segments, + } + } +} + +/// A block's payload, and the codeword segments of a classic block of +/// several, which other blocks leave empty. +type BlockPayload<'p> = (&'p [u8], &'p [J2kCodeBlockSegment]); + +/// A worker's scratch for the blocks it decodes: a block it decodes apart +/// from its sub-band, a payload gathered from several segments, and the +/// codeword segments of a classic block of several. +pub(in crate::j2c::decode) struct BlockScratch { + clipped: Vec, + payload: Vec, + segments: Vec, +} + +/// What each thread's [`BlockScratch`] holds: the bytes of the largest +/// payload it gathers, the codeword segments of the classic block of the +/// most, and the samples of the largest block it decodes apart from its +/// sub-band. +#[derive(Clone, Copy)] +pub(in crate::j2c::decode) struct ScratchSizes { + payload_len: usize, + segments: usize, + samples: usize, +} + +impl ScratchSizes { + /// Bytes of a thread's scratch of these sizes, besides its entry in the + /// threads' list. + fn bytes(self) -> usize { + [ + self.payload_len, + self.segments + .saturating_mul(size_of::()), + self.samples.saturating_mul(size_of::()), + ] + .into_iter() + .fold(0, usize::saturating_add) + } + + /// Bytes of a thread's scratch of these sizes with its entry in the + /// threads' list, as the workspace peak records it. + fn thread_bytes(self) -> usize { + self.bytes().saturating_add(size_of::()) + } + + /// The larger of each size of `self` and `other`. + fn max(self, other: Self) -> Self { + Self { + payload_len: self.payload_len.max(other.payload_len), + segments: self.segments.max(other.segments), + samples: self.samples.max(other.samples), + } + } + + /// Charge `budget` for the scratch of `threads` threads, besides their + /// list, which [`BlockScratch::for_threads`] charges. + fn include(self, budget: &mut DecodeAllocationBudget, threads: usize) -> Result<()> { + budget.include_bytes( + threads + .checked_mul(self.bytes()) + .ok_or(ValidationError::ImageTooLarge)?, + ) + } +} + +#[cfg(test)] +std::thread_local! { + /// Test-only: the pool threads' scratches this thread holds. + static LIVE_SCRATCHES: core::cell::Cell = const { core::cell::Cell::new(0) }; +} + +impl BlockScratch { + fn new() -> Self { + #[cfg(test)] + LIVE_SCRATCHES.with(|live| live.set(live.get() + 1)); + Self { + clipped: Vec::new(), + payload: Vec::new(), + segments: Vec::new(), + } + } + + /// Scratch for each of `count` threads, reserved before they start for + /// `sizes`, which the route charged per thread: decoding never grows + /// it. Charges `budget` for the list, and for capacity the allocator + /// gave beyond `sizes`. + pub(in crate::j2c::decode) fn for_threads( + count: usize, + sizes: ScratchSizes, + budget: &mut DecodeAllocationBudget, + ) -> Result> { + let mut scratches = Vec::new(); + budget.reserve_new(&mut scratches, count)?; + for _ in 0..count { + let mut scratch = Self::new(); + try_reserve_decode_elements(&mut scratch.clipped, sizes.samples)?; + try_reserve_decode_elements(&mut scratch.payload, sizes.payload_len)?; + try_reserve_decode_elements(&mut scratch.segments, sizes.segments)?; + budget.include_capacity_overage::(sizes.samples, scratch.clipped.capacity())?; + budget.include_capacity_overage::(sizes.payload_len, scratch.payload.capacity())?; + budget.include_capacity_overage::( + sizes.segments, + scratch.segments.capacity(), + )?; + scratches.push(scratch); + } + Ok(scratches) + } + + /// Scratch for each of `count` threads, as [`Self::for_threads`] + /// reserves it, when nothing charged it yet: charges `budget` for all of + /// it. Returns `None` when it does not fit. + pub(in crate::j2c::decode) fn charged_for_threads( + count: usize, + sizes: ScratchSizes, + budget: &mut DecodeAllocationBudget, + ) -> Result>> { + match sizes + .include(budget, count) + .and_then(|()| Self::for_threads(count, sizes, budget)) + { + Ok(scratches) => Ok(Some(scratches)), + Err(error) if crate::j2c::decode::reuse::is_capacity_error(&error) => Ok(None), + Err(error) => Err(error), + } + } + + /// Test-only: how many pool threads' scratches this thread holds. + #[cfg(test)] + pub(in crate::j2c::decode) fn live() -> usize { + LIVE_SCRATCHES.with(core::cell::Cell::get) + } + + /// Test-only: the most block-scratch samples one of `scratches` holds. + #[cfg(test)] + pub(in crate::j2c::decode) fn block_samples(scratches: &[Self]) -> usize { + scratches + .iter() + .map(|scratch| scratch.clipped.capacity()) + .max() + .unwrap_or(0) + } + + /// Test-only: the bytes `scratches` hold, their list included. + #[cfg(test)] + pub(in crate::j2c::decode) fn held_bytes(scratches: &[Self]) -> usize { + scratches + .iter() + .fold(size_of_val(scratches), |bytes, scratch| { + [ + scratch.clipped.capacity() * size_of::(), + scratch.payload.capacity(), + scratch.segments.capacity() * size_of::(), + ] + .into_iter() + .fold(bytes, usize::saturating_add) + }) + } +} + +#[cfg(test)] +impl Drop for BlockScratch { + fn drop(&mut self) { + LIVE_SCRATCHES.with(|live| live.set(live.get() - 1)); + } +} + +/// Bytes a lock or a condition variable may allocate when first used: none +/// where the platform's locks are futexes, a boxed `pthread` mutex of 64 +/// bytes or condition variable of 48 where they are not. +pub(in crate::j2c::decode) const LOCK_BYTES: usize = 64; + +/// Bytes the pool may take for a parallel decode entered from outside it: +/// a block of its shared queue for the jobs the decode hands it, 63 of two +/// words with their states and a link, and the lock and condition variable +/// the calling thread waits on, whose lock the waiter and the worker that +/// wakes it may each allocate when they first use it at once. +pub(in crate::j2c::decode) fn pool_entry_bytes() -> usize { + if rayon::current_thread_index().is_some() { + 0 + } else { + 64 * 3 * size_of::() + 3 * LOCK_BYTES + } +} + +/// The first `samples` of a thread's block scratch, within the capacity +/// [`BlockScratch::for_threads`] reserved: it never grows. +fn block_scratch(scratch: &mut Vec, samples: usize) -> Result<&mut [f32]> { + if samples > scratch.capacity() { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + if scratch.len() < samples { + scratch.resize(samples, 0.0); + } + scratch + .get_mut(..samples) + .ok_or_else(|| DecodingError::CodeBlockDecodeFailure.into()) } struct BlockJob<'a> { @@ -163,9 +388,12 @@ struct TilePlan<'a> { strict: bool, jobs: Vec>, strips: Vec, - arena: Vec, - /// Bytes of the payloads to gather into `arena` once the route is taken. - arena_len: usize, + /// Bytes of the largest payload gathered from several segments. Each + /// worker gathers such payloads into scratch of this size. + max_gathered_len: usize, + /// Codeword segments of the classic block of several with the most. + /// Each worker collects their descriptors into scratch of this size. + max_segments: usize, skipped: usize, coded_bytes: usize, max_width: u32, @@ -177,6 +405,18 @@ struct TilePlan<'a> { band_sub_bands: Vec, } +impl TilePlan<'_> { + /// What each thread's scratch holds for the plan, with block scratch of + /// `samples`. + fn scratch_sizes(&self, samples: usize) -> ScratchSizes { + ScratchSizes { + payload_len: self.max_gathered_len, + segments: self.max_segments, + samples, + } + } +} + /// How [`plan_tile_route`] wants the tile's code blocks decoded. #[cfg_attr(test, derive(Debug, PartialEq, Eq))] pub(in crate::j2c::decode) enum TileRoute { @@ -189,9 +429,9 @@ pub(in crate::j2c::decode) enum TileRoute { } /// Decode every code block of the tile in parallel when all of its -/// components use HT block coding only, or classic coding with one arithmetic -/// segment per block, and the tile carries enough coded data to amortize a -/// fork-join. Returns without side effects for the non-`Decoded` routes. +/// components use HT block coding only, or classic coding only, and the tile +/// carries enough coded data to amortize a fork-join. Returns without side +/// effects for the non-`Decoded` routes. pub(in crate::j2c::decode) fn plan_tile_route<'a>( tile: &Tile<'a>, components: Range, @@ -228,7 +468,7 @@ pub(in crate::j2c::decode) fn plan_tile_route<'a>( return Ok(route); } let mut budget = DecodeAllocationBudget::for_storage(storage)?; - let mut plan = match plan_tile( + let plan = match plan_tile( tile, components, storage, @@ -259,15 +499,18 @@ pub(in crate::j2c::decode) fn plan_tile_route<'a>( let workers = rayon::current_num_threads().min(plan.strips.len()); let counts = TaskCounts::new(&plan, workers, storage.stale_coefficients)?; let route_budget = (|| -> Result<()> { - // Payloads are gathered only once the tile takes the route. - gather_payloads(&mut plan, storage, &mut budget)?; + // Each worker holds scratch of its own, reserved before it starts. + plan.scratch_sizes(counts.scratch_samples) + .include(&mut budget, workers)?; // `plan_tile` charged one task per strip. budget.include_elements::>(counts.tasks - plan.strips.len())?; budget.include_elements::>(counts.shared)?; - // Each worker allocates its block scratch on first use. - budget.include_elements::( - workers - .checked_mul(counts.scratch_samples) + // The shared outputs' locks and the queue's, where locking first + // allocates, and what the pool takes when entered from outside it. + budget.include_bytes( + (counts.shared + 1) + .checked_mul(LOCK_BYTES) + .and_then(|locks| locks.checked_add(pool_entry_bytes())) .ok_or(ValidationError::ImageTooLarge)?, )?; if has_later_components { @@ -317,8 +560,8 @@ fn decode_with_retained_bank<'a>( &mut tile_ctx.classic_task_workspaces, storage, budget, - |job, output, stride, workspace| { - decode_classic_strip_block(plan, job, output, stride, workspace) + |job, payload, output, stride, workspace| { + decode_classic_strip_block(plan, job, payload, output, stride, workspace) }, ), CodeBlockCoding::HighThroughput => decode_with_bank( @@ -328,8 +571,8 @@ fn decode_with_retained_bank<'a>( &mut tile_ctx.ht_task_workspaces, storage, budget, - |job, output, stride, workspace| { - decode_strip_block(plan, job, output, stride, workspace) + |job, payload, output, stride, workspace| { + decode_strip_block(plan, job, payload, output, stride, workspace) }, ), } @@ -370,8 +613,8 @@ fn count_decoded_blocks( } /// Record the decode's workspace peak while the plan, its tasks and each -/// worker's clipped-block scratch, as budgeted, are live next to the retained -/// workspace bank. +/// worker's scratch, as budgeted, are live next to the retained workspace +/// bank. fn record_route_peak( tile_ctx: &mut TileDecodeContext, storage: &DecompositionStorage<'_>, @@ -380,17 +623,12 @@ fn record_route_peak( counts: TaskCounts, ) -> Result<()> { let route_bytes = [ - plan.bands.capacity() * size_of::(), - plan.classic_bands.capacity() * size_of::(), - plan.jobs.capacity() * size_of::>(), - plan.strips.capacity() * size_of::(), - plan.band_sub_bands.capacity() * size_of::(), - plan.arena.capacity(), + plan_bytes(plan), + workers.saturating_mul(plan.scratch_sizes(counts.scratch_samples).thread_bytes()), counts.tasks * size_of::>(), counts.shared * size_of::>(), - workers - .saturating_mul(counts.scratch_samples) - .saturating_mul(size_of::()), + (counts.shared + 1) * LOCK_BYTES, + pool_entry_bytes(), ] .into_iter() .fold(0, usize::saturating_add); @@ -399,6 +637,19 @@ fn record_route_peak( Ok(()) } +/// Bytes the plan's own tables hold. +fn plan_bytes(plan: &TilePlan<'_>) -> usize { + [ + plan.bands.capacity() * size_of::(), + plan.classic_bands.capacity() * size_of::(), + plan.jobs.capacity() * size_of::>(), + plan.strips.capacity() * size_of::(), + plan.band_sub_bands.capacity() * size_of::(), + ] + .into_iter() + .fold(0, usize::saturating_add) +} + /// Prepare one workspace per worker in the retained `bank`, then decode the /// plan's strips into `storage`'s coefficients in place through /// `decode_block`, which leaves a stale buffer current. Returns the bank's @@ -410,7 +661,8 @@ fn decode_with_bank<'a, W: TaskScratch>( bank: &mut Vec>, storage: &mut DecompositionStorage<'a>, budget: &mut DecodeAllocationBudget, - decode_block: impl Fn(&BlockJob<'_>, &mut [f32], usize, &mut W) -> Result<()> + Sync, + decode_block: impl Fn(&BlockJob<'_>, BlockPayload<'_>, &mut [f32], usize, &mut W) -> Result<()> + + Sync, ) -> Result> { let Ok(growths) = prepare_uniform_task_workspaces( bank, @@ -422,15 +674,22 @@ fn decode_with_bank<'a, W: TaskScratch>( ) else { return Ok(None); }; + let sizes = plan.scratch_sizes(counts.scratch_samples); + let Ok(mut scratches) = BlockScratch::for_threads(workers, sizes, budget) else { + return Ok(None); + }; let mut coefficients = core::mem::take(&mut storage.coefficients); let stale = storage.stale_coefficients; let result = decode_strips( plan, &mut coefficients, stale, - &mut bank[..workers], + (&mut bank[..workers], &mut scratches), counts, - |job, output, stride, slot| decode_block(job, output, stride, &mut slot.workspace), + PayloadSource::new(storage), + |job, payload, output, stride, slot| { + decode_block(job, payload, output, stride, &mut slot.workspace) + }, ); storage.coefficients = coefficients; result?; @@ -445,7 +704,7 @@ pub(in crate::j2c::decode) fn routes_tile(tile: &Tile<'_>) -> bool { } /// The block coding of every component in `components` when the route can -/// decode it: HT only, or classic with one arithmetic segment per block. +/// decode it: HT only, or classic only. fn tile_route_coding(tile: &Tile<'_>, components: Range) -> Option { let mut styles = tile .component_infos @@ -459,9 +718,7 @@ fn tile_route_coding(tile: &Tile<'_>, components: Range) -> Option( strict: header.strict, jobs: Vec::new(), strips: Vec::new(), - arena: Vec::new(), - arena_len: 0, + max_gathered_len: 0, + max_segments: 0, skipped: 0, coded_bytes: 0, max_width: 0, @@ -507,8 +764,6 @@ fn plan_tile<'a>( } else { budget.reserve_new(&mut plan.bands, storage.sub_bands.len())?; } - let mut arena_len = 0usize; - let component_infos = tile .component_infos .get(components.clone()) @@ -586,15 +841,19 @@ fn plan_tile<'a>( { continue; } - let job = plan_block( + let (job, segments) = plan_block( code_block, code_block_idx, stored_placement(code_block, sub_band), band, storage, - &mut arena_len, classic_style, )?; + plan.max_segments = plan.max_segments.max(segments); + if matches!(job.payload, Payload::Gathered) { + plan.max_gathered_len = + plan.max_gathered_len.max(job.payload_length as usize); + } if !job.placement.is_whole(job.width, job.height) { let samples = (job.width as usize) .checked_mul(job.height as usize) @@ -634,7 +893,14 @@ fn plan_tile<'a>( if !banded && !strips_are_disjoint(&mut plan.strips, storage.coefficients.len()) { return Ok(None); } - plan.arena_len = arena_len; + // Strips were reserved one per block; keep room for the rows planned. + let reserved = plan.strips.capacity(); + plan.strips.shrink_to_fit(); + budget.release_bytes( + reserved + .saturating_sub(plan.strips.capacity()) + .saturating_mul(size_of::()), + )?; budget.include_elements::>(plan.strips.len())?; Ok(Some(plan)) } @@ -695,7 +961,7 @@ fn required_code_blocks( count += 1; segment_bytes = storage.layers[code_block.layers.clone()] .iter() - .filter_map(|layer| layer.segments.clone()) + .filter_map(|layer| layer.segments()) .flat_map(|range| &storage.segments[range]) .fold(segment_bytes, |bytes, segment| { bytes.saturating_add(segment.data.len()) @@ -708,31 +974,26 @@ fn required_code_blocks( Ok((count, segment_bytes, rows)) } -/// A block's job. A payload in one codestream segment is borrowed; others -/// get room in the arena. +/// A block's job, and the codeword segments whose descriptors it collects +/// when it decodes: those of a classic block of several, or none. A payload +/// in one codestream segment is borrowed; others are gathered when the +/// block decodes. fn plan_block<'a>( code_block: &CodeBlock, code_block_idx: usize, placement: BlockPlacement, band: usize, storage: &DecompositionStorage<'a>, - arena_len: &mut usize, classic_style: Option, -) -> Result> { - let (payload_length, refinement_length, single) = match classic_style { - Some(style) => classic_payload(code_block, style, storage)?, - None => ht_payload(code_block, storage)?, - }; - let payload = if let Some(data) = single { - Payload::Borrowed(data) +) -> Result<(BlockJob<'a>, usize)> { + let (payload_length, refinement_length, single, segments) = if let Some(style) = classic_style { + classic_payload(code_block, style, storage)? } else { - let start = *arena_len; - *arena_len = start - .checked_add(payload_length) - .ok_or(ValidationError::ImageTooLarge)?; - Payload::Gathered(start) + let (payload_length, refinement_length, single) = ht_payload(code_block, storage)?; + (payload_length, refinement_length, single, 0) }; - Ok(BlockJob { + let payload = single.map_or(Payload::Gathered, Payload::Borrowed); + let job = BlockJob { payload, code_block: code_block_idx, payload_length: u32::try_from(payload_length) @@ -745,7 +1006,8 @@ fn plan_block<'a>( missing_bit_planes: code_block.missing_bit_planes, number_of_coding_passes: code_block.number_of_coding_passes, band, - }) + }; + Ok((job, segments)) } /// An HT block's payload and refinement lengths, and its cleanup segment when @@ -764,13 +1026,14 @@ fn ht_payload<'a>( } /// A classic block's payload length, checked as -/// `collect_classic_code_block_data` checks it, and its segment data when at -/// most one segment holds any. +/// `collect_classic_code_block_data` checks it, its segment data when at +/// most one segment holds any, and its codeword segments when it has +/// several. fn classic_payload<'a>( code_block: &CodeBlock, style: CodeBlockStyle, storage: &DecompositionStorage<'a>, -) -> Result<(usize, usize, Option<&'a [u8]>)> { +) -> Result<(usize, usize, Option<&'a [u8]>, usize)> { let counts = classic_allocation_counts(code_block, style, storage)?; if counts.coding_passes != code_block.number_of_coding_passes { return Err(DecodingError::CodeBlockDecodeFailure.into()); @@ -778,13 +1041,18 @@ fn classic_payload<'a>( let single = (counts.fragments <= 1).then(|| { storage.layers[code_block.layers.clone()] .iter() - .filter_map(|layer| layer.segments.clone()) + .filter_map(|layer| layer.segments()) .flat_map(|range| &storage.segments[range]) .map(|segment| segment.data) .find(|data| !data.is_empty()) .unwrap_or_default() }); - Ok((counts.data_bytes, 0, single)) + let segments = if style.single_codeword_segment() { + 0 + } else { + counts.segments + }; + Ok((counts.data_bytes, 0, single, segments)) } /// The block's cleanup segment when it is the only selected segment. @@ -800,7 +1068,7 @@ fn single_cleanup_segment<'a>( .ok_or(DecodingError::CodeBlockDecodeFailure)?; let mut cleanup = None; for layer in &storage.layers[code_block.layers.clone()] { - let Some(range) = layer.segments.clone() else { + let Some(range) = layer.segments() else { continue; }; for segment in &storage.segments[range] { @@ -903,56 +1171,62 @@ fn strips_are_disjoint(strips: &mut [Strip], coefficient_len: usize) -> bool { previous_end <= coefficient_len } -fn gather_payloads<'a>( - plan: &mut TilePlan<'a>, - storage: &DecompositionStorage<'a>, - budget: &mut DecodeAllocationBudget, -) -> Result<()> { - if plan.arena_len == 0 { - return Ok(()); +/// Gather the payload of `job`, a block whose payload spans several +/// segments, into `scratch`: for a classic block every segment, for an HT +/// block the selected set's cleanup and refinement segments. +fn gather_payload<'s>( + coding: CodeBlockCoding, + job: &BlockJob<'_>, + source: PayloadSource<'_, '_>, + scratch: &'s mut Vec, +) -> Result<&'s [u8]> { + let expected = job.payload_length as usize; + scratch.clear(); + // The thread's scratch holds the plan's largest gathered payload. + if expected > scratch.capacity() { + return Err(DecodingError::CodeBlockDecodeFailure.into()); } - budget.reserve_new(&mut plan.arena, plan.arena_len)?; - for job in &mut plan.jobs { - let Payload::Gathered(_) = job.payload else { + let code_block = source + .code_blocks + .get(job.code_block) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let cleanup_index = if coding == CodeBlockCoding::Classic { + None + } else { + let selected_set = code_block + .ht_selected_set + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + Some( + selected_set + .checked_mul(2) + .ok_or(DecodingError::CodeBlockDecodeFailure)?, + ) + }; + let layers = source + .layers + .get(code_block.layers.clone()) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + for layer in layers { + let Some(range) = layer.segments() else { continue; }; - // Row sorting can change precinct traversal order. Assign the offset - // when the payload is appended in its final execution order. - let start = plan.arena.len(); - job.payload = Payload::Gathered(start); - let code_block = &storage.code_blocks[job.code_block]; - // A classic payload is every segment; an HT payload the selected - // set's cleanup and refinement segments. - let cleanup_index = if plan.coding == CodeBlockCoding::Classic { - None - } else { - let selected_set = code_block - .ht_selected_set - .ok_or(DecodingError::CodeBlockDecodeFailure)?; - Some( - selected_set - .checked_mul(2) - .ok_or(DecodingError::CodeBlockDecodeFailure)?, - ) - }; - for layer in &storage.layers[code_block.layers.clone()] { - let Some(range) = layer.segments.clone() else { - continue; - }; - for segment in &storage.segments[range] { - if cleanup_index - .is_none_or(|index| segment.idx == index || segment.idx == index + 1) - { - plan.arena.extend_from_slice(segment.data); + let segments = source + .segments + .get(range) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + for segment in segments { + if cleanup_index.is_none_or(|index| segment.idx == index || segment.idx == index + 1) { + if scratch.len() + segment.data.len() > expected { + return Err(DecodingError::CodeBlockDecodeFailure.into()); } + scratch.extend_from_slice(segment.data); } } - let expected = job.payload_length as usize; - if plan.arena.len() - start != expected { - return Err(DecodingError::CodeBlockDecodeFailure.into()); - } } - Ok(()) + if scratch.len() != expected { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + Ok(scratch) } fn charge_actual_workspace( @@ -1172,15 +1446,17 @@ fn push_task(tasks: &mut Vec, task: T) -> Result<()> { Ok(()) } -/// Decode the strips on one worker per workspace, each block through -/// `decode_block`. +/// Decode the strips on one worker per workspace, with its own scratch, +/// each block through `decode_block`. fn decode_strips<'plan, 'coefficients, W: Send>( plan: &'plan TilePlan<'_>, coefficients: &'coefficients mut [f32], stale: bool, - workspaces: &mut [W], + (workspaces, scratches): (&mut [W], &mut [BlockScratch]), counts: TaskCounts, - decode_block: impl Fn(&BlockJob<'_>, &mut [f32], usize, &mut W) -> Result<()> + Sync, + source: PayloadSource<'_, '_>, + decode_block: impl Fn(&BlockJob<'_>, BlockPayload<'_>, &mut [f32], usize, &mut W) -> Result<()> + + Sync, ) -> Result<()> { let mut shared: Vec> = Vec::new(); shared @@ -1233,15 +1509,20 @@ fn decode_strips<'plan, 'coefficients, W: Send>( // Largest tasks first so stragglers are small; zeroing tasks last. tasks.sort_unstable_by_key(|task| core::cmp::Reverse(task.work())); let queue = Mutex::new(tasks.into_iter()); + // Lock each lock once here, where no worker races to allocate it. + for lock in &shared { + drop(lock.lock()); + } + drop(queue.lock()); let failed = AtomicBool::new(false); // Each worker owns one already-budgeted workspace for the whole tile. // A single queue pop per task keeps work balanced across unequal cores // without forking a Rayon job and borrowing/returning scratch for each row. workspaces .par_iter_mut() + .zip(scratches) .with_max_len(1) - .try_for_each(|workspace| -> Result<()> { - let mut scratch = Vec::new(); + .try_for_each(|(workspace, scratch)| -> Result<()> { while !failed.load(Ordering::Relaxed) { let next = queue .lock() @@ -1259,7 +1540,14 @@ fn decode_strips<'plan, 'coefficients, W: Send>( if stale { output.fill(0.0); } - decode_strip(plan, strip, output, workspace, &mut scratch, &decode_block) + decode_strip( + plan, + strip, + output, + workspace, + (&mut *scratch, source), + &decode_block, + ) } StripTask::Part(output, strip, jobs, _) => decode_part( plan, @@ -1267,9 +1555,8 @@ fn decode_strips<'plan, 'coefficients, W: Send>( jobs, output, stale, - counts.scratch_samples, workspace, - &mut scratch, + (&mut *scratch, source), &decode_block, ), }; @@ -1295,11 +1582,15 @@ fn decode_part( jobs: Range, output: &Mutex<&mut [f32]>, stale: bool, - scratch_samples: usize, workspace: &mut W, - scratch: &mut Vec, - decode_block: &impl Fn(&BlockJob<'_>, &mut [f32], usize, &mut W) -> Result<()>, + (scratch, source): (&mut BlockScratch, PayloadSource<'_, '_>), + decode_block: &impl Fn(&BlockJob<'_>, BlockPayload<'_>, &mut [f32], usize, &mut W) -> Result<()>, ) -> Result<()> { + let BlockScratch { + clipped: scratch, + payload: payload_scratch, + segments: segment_scratch, + } = scratch; let lock = || { output .lock() @@ -1326,9 +1617,6 @@ fn decode_part( .fill(0.0); } } - if scratch.len() < scratch_samples { - try_resize_decode_elements(scratch, scratch_samples, 0.0)?; - } for job in plan .jobs .get(jobs) @@ -1339,13 +1627,12 @@ fn decode_part( target_y: 0, ..job.placement }; - let block = scratch - .get_mut(..job.width as usize * job.height as usize) - .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let block = block_scratch(scratch, job.width as usize * job.height as usize)?; if !placement.is_whole(job.width, job.height) { block.fill(0.0); } - decode_block(job, block, job.width as usize, workspace)?; + let payload = job_payload(plan, job, source, (payload_scratch, segment_scratch))?; + decode_block(job, payload, block, job.width as usize, workspace)?; placement.copy_from_block(block, job.width, &mut lock()?, strip.stride)?; } Ok(()) @@ -1356,9 +1643,14 @@ fn decode_strip( strip: &Strip, output: &mut [f32], workspace: &mut W, - clipped_scratch: &mut Vec, - decode_block: &impl Fn(&BlockJob<'_>, &mut [f32], usize, &mut W) -> Result<()>, + (scratch, source): (&mut BlockScratch, PayloadSource<'_, '_>), + decode_block: &impl Fn(&BlockJob<'_>, BlockPayload<'_>, &mut [f32], usize, &mut W) -> Result<()>, ) -> Result<()> { + let BlockScratch { + clipped: clipped_scratch, + payload: payload_scratch, + segments: segment_scratch, + } = scratch; for job in &plan.jobs[strip.jobs.clone()] { // The strip starts at the block row's first stored row. let placement = BlockPlacement { @@ -1371,16 +1663,12 @@ fn decode_strip( let (block_output, output_stride) = if whole { (&mut output[placement.target_x as usize..], strip.stride) } else { - if clipped_scratch.is_empty() { - try_resize_decode_elements(clipped_scratch, plan.max_clipped_samples, 0.0)?; - } - let block = clipped_scratch - .get_mut(..job.width as usize * job.height as usize) - .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let block = block_scratch(clipped_scratch, job.width as usize * job.height as usize)?; block.fill(0.0); (block, job.width as usize) }; - decode_block(job, block_output, output_stride, workspace)?; + let payload = job_payload(plan, job, source, (payload_scratch, segment_scratch))?; + decode_block(job, payload, block_output, output_stride, workspace)?; if !whole { placement.copy_from_block(clipped_scratch, job.width, output, strip.stride)?; } @@ -1391,13 +1679,14 @@ fn decode_strip( fn decode_strip_block( plan: &TilePlan<'_>, job: &BlockJob<'_>, + (payload, _): BlockPayload<'_>, output: &mut [f32], output_stride: usize, workspace: &mut HtCodeBlockDecodeWorkspace, ) -> Result<()> { let band = plan.bands[job.band]; let decode_job = HtCodeBlockDecodeJob { - data: job_payload(plan, job)?, + data: payload, cleanup_length: job.payload_length - job.refinement_length, refinement_length: job.refinement_length, width: job.width, @@ -1431,21 +1720,28 @@ fn decode_strip_block( fn decode_classic_strip_block( plan: &TilePlan<'_>, job: &BlockJob<'_>, + (payload, segments): BlockPayload<'_>, output: &mut [f32], output_stride: usize, workspace: &mut J2kCodeBlockDecodeWorkspace, ) -> Result<()> { let band = plan.classic_bands[job.band]; - let segments = [J2kCodeBlockSegment { + // A block of one codeword segment is one arithmetic segment. + let single = [J2kCodeBlockSegment { data_offset: 0, data_length: job.payload_length, start_coding_pass: 0, end_coding_pass: job.number_of_coding_passes, use_arithmetic: true, }]; + let segments = if band_style(plan, job.band).single_codeword_segment() { + &single[..] + } else { + segments + }; let decode_job = band.job( - job_payload(plan, job)?, - &segments, + payload, + segments, job.width, job.height, output_stride, @@ -1455,15 +1751,83 @@ fn decode_classic_strip_block( band.decode(decode_job, output, workspace) } -/// The job's payload, borrowed from the codestream or from the arena. -fn job_payload<'p>(plan: &'p TilePlan<'_>, job: &BlockJob<'p>) -> Result<&'p [u8]> { - match job.payload { - Payload::Borrowed(data) => Ok(data), - Payload::Gathered(start) => plan - .arena - .get(start..start + job.payload_length as usize) - .ok_or_else(|| DecodingError::CodeBlockDecodeFailure.into()), +/// The job's payload, borrowed from the codestream or gathered into +/// `payload`, and for a classic block of several codeword segments, their +/// descriptors, collected into `segments`. +fn job_payload<'p>( + plan: &TilePlan<'_>, + job: &BlockJob<'p>, + source: PayloadSource<'_, '_>, + (payload, segments): (&'p mut Vec, &'p mut Vec), +) -> Result> { + if plan.coding == CodeBlockCoding::Classic { + let style = band_style(plan, job.band); + if !style.single_codeword_segment() { + return collect_segments(style, job, source, (payload, segments)); + } } + let data = match job.payload { + Payload::Borrowed(data) => data, + Payload::Gathered => gather_payload(plan.coding, job, source, payload)?, + }; + Ok((data, &[])) +} + +/// The payload and codeword segments of `job`, a classic block of several: +/// one walk over its segment records collects the segments into +/// `segments` and, when its payload spans several fragments, gathers them +/// into `payload`. The thread's scratch holds both for the plan's largest. +fn collect_segments<'p>( + style: CodeBlockStyle, + job: &BlockJob<'p>, + source: PayloadSource<'_, '_>, + (payload, segments): (&'p mut Vec, &'p mut Vec), +) -> Result> { + let code_block = source + .code_blocks + .get(job.code_block) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let gathered = matches!(job.payload, Payload::Gathered); + payload.clear(); + segments.clear(); + let (data_bytes, _, coding_passes) = walk_classic_segments( + code_block, + style, + (source.layers, source.segments), + |fragment| { + if gathered { + extend_reserved(payload, fragment)?; + } + Ok(()) + }, + |segment| push_reserved(segments, segment), + )?; + if data_bytes != job.payload_length as usize || coding_passes != job.number_of_coding_passes { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + let data = match job.payload { + Payload::Borrowed(data) => data, + Payload::Gathered => payload, + }; + Ok((data, segments)) +} + +/// The code-block style of band `band`'s blocks in a classic plan. +fn band_style(plan: &TilePlan<'_>, band: usize) -> CodeBlockStyle { + internal_j2k_code_block_style(plan.classic_bands[band].style) +} + +/// Where a band decodes: into its slot, or as run `part` of its blocks split +/// for `parts` parts (see [`BandRoute::strip_parts`]), into the slot its +/// parts share. A part zeroes its own columns of the slot, from its first +/// block to the next part's, and copies each block in under the lock. +pub(in crate::j2c::decode) enum BandOutput<'o, 'c> { + Whole(&'o mut [f32]), + Part { + part: usize, + parts: usize, + slot: &'o Mutex<&'c mut [f32]>, + }, } /// One code-block row of a sub-band, as [`plan_band_route`] plans it: the @@ -1484,9 +1848,10 @@ pub(in crate::j2c::decode) struct BandRoute<'a> { plan: TilePlan<'a>, workers: usize, growths: usize, - /// The decode's budget with the plan, the payload arena and every - /// thread's clipped-block scratch charged, which stay live while the - /// bands decode. + /// The decode's budget with the plan, every thread's payload and block + /// scratch and the bank's growth charged, which stay live while the + /// bands decode, beside whatever the budget the route was planned in + /// held already. budget: DecodeAllocationBudget, } @@ -1506,12 +1871,47 @@ pub(in crate::j2c::decode) enum BandRouting<'a> { /// bytes and its rows of code-blocks. With a plan, the coding's retained bank /// holds a workspace for each of `workers` pool threads and one more for the /// calling thread, charged to the structural workspace. +/// +/// A `lockstep` decode, whose rows stream to a sink, has no per-sub-band +/// path: a classic component with fewer rows of code-blocks than threads +/// stays on the route, whose bands split into parts for the idle threads, +/// rather than decode its bands serially. A 512 x 512 RGB image at 6.6 bits +/// per pixel decoded 4.5 times as long on 18 threads as on 12 that way. Its +/// executor runs every component's routes on the same threads, and charges +/// their scratch once for all of them (see +/// [`BandRoute::shared_scratch`]); a route charges its threads' scratch +/// otherwise. pub(in crate::j2c::decode) fn plan_band_route<'a>( tile: &Tile<'a>, component_idx: usize, tile_ctx: &mut TileDecodeContext, storage: &mut DecompositionStorage<'a>, header: &Header<'_>, + lockstep: bool, +) -> Result> { + plan_band_route_beside( + tile, + component_idx, + tile_ctx, + storage, + header, + None, + lockstep, + ) +} + +/// [`plan_band_route`], charging the route to `live` when other routes stay +/// live beside it: the budget of the route planned last for the same +/// executor, which holds every earlier route's charges. Without it, the +/// route is charged to the decode's storage budget. +pub(in crate::j2c::decode) fn plan_band_route_beside<'a>( + tile: &Tile<'a>, + component_idx: usize, + tile_ctx: &mut TileDecodeContext, + storage: &mut DecompositionStorage<'a>, + header: &Header<'_>, + live: Option, + lockstep: bool, ) -> Result> { let components = component_idx..component_idx + 1; let Some(coding) = tile_route_coding(tile, components.clone()) else { @@ -1524,17 +1924,24 @@ pub(in crate::j2c::decode) fn plan_band_route<'a>( }; let (required, segment_bytes, rows) = required_code_blocks(tile, components.clone(), storage, header, coding)?; - if let Some(route) = route_fallback( - coding, - rows, - segment_bytes, - threads, - MIN_PARALLEL_CODED_BYTES, - ) { + let fallback = |strips, coded_bytes| { + route_fallback( + coding, + strips, + coded_bytes, + threads, + MIN_PARALLEL_CODED_BYTES, + ) + .filter(|route| !(lockstep && matches!(route, TileRoute::PerSubBand))) + }; + if let Some(route) = fallback(rows, segment_bytes) { return Ok(declined(route)); } - let mut budget = DecodeAllocationBudget::for_storage(storage)?; - let mut plan = match plan_tile( + let mut budget = match live { + Some(live) => live, + None => DecodeAllocationBudget::for_storage(storage)?, + }; + let plan = match plan_tile( tile, components, storage, @@ -1552,13 +1959,7 @@ pub(in crate::j2c::decode) fn plan_band_route<'a>( } Err(error) => return Err(error), }; - if let Some(route) = route_fallback( - coding, - plan.strips.len(), - plan.coded_bytes, - threads, - MIN_PARALLEL_CODED_BYTES, - ) { + if let Some(route) = fallback(plan.strips.len(), plan.coded_bytes) { return Ok(declined(route)); } let work = plan @@ -1576,13 +1977,12 @@ pub(in crate::j2c::decode) fn plan_band_route<'a>( } let workers = threads.min(plan.strips.len()); let prepared = (|| -> Result { - gather_payloads(&mut plan, storage, &mut budget)?; - // Each thread allocates its clipped-block scratch on first use. - budget.include_elements::( - (workers + 1) - .checked_mul(plan.max_clipped_samples) - .ok_or(ValidationError::ImageTooLarge)?, - )?; + // The route itself, boxed beside the others of its tile. + budget.include_bytes(size_of::>())?; + // Each thread holds scratch of its own, reserved before it starts. + if !lockstep { + band_scratch_sizes(&plan)?.include(&mut budget, workers + 1)?; + } let (width, height) = (plan.max_width, plan.max_height); let structural = &mut storage.structural_workspace_bytes; match coding { @@ -1618,6 +2018,22 @@ pub(in crate::j2c::decode) fn plan_band_route<'a>( } } +/// What each thread of a band route holds for `plan`: block scratch for +/// the largest block its sub-band stores only part of and, in a classic +/// plan, whose bands may decode in parts that copy every block of their run +/// in through it, for the largest block. HTJ2K bands decode whole, their +/// whole blocks straight into their slots. +fn band_scratch_sizes(plan: &TilePlan<'_>) -> Result { + let samples = match plan.coding { + CodeBlockCoding::Classic => (plan.max_width as usize) + .checked_mul(plan.max_height as usize) + .ok_or(ValidationError::ImageTooLarge)? + .max(plan.max_clipped_samples), + CodeBlockCoding::HighThroughput => plan.max_clipped_samples, + }; + Ok(plan.scratch_sizes(samples)) +} + impl BandRoute<'_> { pub(in crate::j2c::decode) fn coding(&self) -> CodeBlockCoding { self.plan.coding @@ -1634,6 +2050,30 @@ impl BandRoute<'_> { self.workers } + /// What each thread's scratch holds for this route, as its planning + /// charged it. + pub(in crate::j2c::decode) fn scratch_sizes(&self) -> Result { + band_scratch_sizes(&self.plan) + } + + /// What each thread's scratch holds to serve all of `routes` at once: + /// the largest of each size over them. + pub(in crate::j2c::decode) fn shared_scratch<'r>( + mut routes: impl Iterator, + ) -> Result + where + Self: 'r, + { + routes.try_fold( + ScratchSizes { + payload_len: 0, + segments: 0, + samples: 0, + }, + |sizes, route| Ok(sizes.max(band_scratch_sizes(&route.plan)?)), + ) + } + pub(in crate::j2c::decode) fn strips(&self) -> impl Iterator + '_ { self.plan.strips.iter().map(|strip| { let band = self.plan.jobs[strip.jobs.start].band; @@ -1646,66 +2086,143 @@ impl BandRoute<'_> { }) } - /// Decode strip `index` of a classic plan into `output`, which holds the - /// band's rows from its first. + /// The runs of blocks strip `index` splits into when it aims at + /// `parts` parts of about equal work: `parts` at most, fewer when a few + /// blocks hold most of its work. + pub(in crate::j2c::decode) fn strip_parts(&self, index: usize, parts: usize) -> Result { + let strip = self + .plan + .strips + .get(index) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + let mut runs = 0usize; + split_strip(&self.plan, strip, parts, |_, _| { + runs += 1; + Ok(()) + })?; + Ok(runs.max(1)) + } + + /// Decode strip `index` of a classic plan into `output`, whose slot + /// holds the band's rows from its first. pub(in crate::j2c::decode) fn decode_classic( &self, index: usize, - output: &mut [f32], + output: BandOutput<'_, '_>, workspace: &mut J2kCodeBlockDecodeWorkspace, - clipped_scratch: &mut Vec, + scratch: (&mut BlockScratch, PayloadSource<'_, '_>), ) -> Result<()> { let plan = &self.plan; - decode_strip( - plan, - &plan.strips[index], + self.decode_band( + index, output, workspace, - clipped_scratch, - &|job, output, stride, workspace| { - decode_classic_strip_block(plan, job, output, stride, workspace) + scratch, + &|job, payload, output, stride, workspace| { + decode_classic_strip_block(plan, job, payload, output, stride, workspace) }, ) } - /// Decode strip `index` of an HT plan into `output`, which holds the - /// band's rows from its first. + /// Decode strip `index` of an HT plan into `output`, whose slot holds + /// the band's rows from its first. pub(in crate::j2c::decode) fn decode_ht( &self, index: usize, - output: &mut [f32], + output: BandOutput<'_, '_>, workspace: &mut HtCodeBlockDecodeWorkspace, - clipped_scratch: &mut Vec, + scratch: (&mut BlockScratch, PayloadSource<'_, '_>), ) -> Result<()> { let plan = &self.plan; - decode_strip( - plan, - &plan.strips[index], + self.decode_band( + index, output, workspace, - clipped_scratch, - &|job, output, stride, workspace| { - decode_strip_block(plan, job, output, stride, workspace) + scratch, + &|job, payload, output, stride, workspace| { + decode_strip_block(plan, job, payload, output, stride, workspace) }, ) } + fn decode_band( + &self, + index: usize, + output: BandOutput<'_, '_>, + workspace: &mut W, + scratch: (&mut BlockScratch, PayloadSource<'_, '_>), + decode_block: &impl Fn(&BlockJob<'_>, BlockPayload<'_>, &mut [f32], usize, &mut W) -> Result<()>, + ) -> Result<()> { + let plan = &self.plan; + let strip = plan + .strips + .get(index) + .ok_or(DecodingError::CodeBlockDecodeFailure)?; + match output { + BandOutput::Whole(slot) => { + decode_strip(plan, strip, slot, workspace, scratch, decode_block) + } + BandOutput::Part { part, parts, slot } => { + let mut jobs = None; + let mut run = 0usize; + split_strip(plan, strip, parts, |range, _| { + if run == part { + jobs = Some(range); + } + run += 1; + Ok(()) + })?; + let jobs = jobs.ok_or(DecodingError::CodeBlockDecodeFailure)?; + decode_part( + plan, + strip, + jobs, + slot, + true, + workspace, + scratch, + decode_block, + ) + } + } + } + /// Count the plan's blocks and record the decode's peak with the plan - /// and every thread's clipped-block scratch live. + /// and every thread's scratch live. pub(in crate::j2c::decode) fn record( &self, tile_ctx: &mut TileDecodeContext, storage: &DecompositionStorage<'_>, ) -> Result<()> { - // Band slots, not route tasks, hold this plan's outputs. - let counts = TaskCounts { - tasks: 0, - shared: 0, - scratch_samples: self.plan.max_clipped_samples, - part_work: None, - }; - record_route_peak(tile_ctx, storage, &self.plan, self.workers + 1, counts)?; - count_decoded_blocks(tile_ctx, &self.plan, self.workers + 1, self.growths); + Self::record_live(core::iter::once(self), self.workers + 1, tile_ctx, storage) + } + + /// Count every route's blocks and record the decode's peak with all + /// their plans live, beside the scratch of the `threads` threads that + /// serve them all. + pub(in crate::j2c::decode) fn record_live<'r>( + routes: impl Iterator + Clone, + threads: usize, + tile_ctx: &mut TileDecodeContext, + storage: &DecompositionStorage<'_>, + ) -> Result<()> + where + Self: 'r, + { + let plans = routes.clone().fold(0usize, |plans, route| { + plans.saturating_add(plan_bytes(&route.plan)) + }); + let scratch = Self::shared_scratch(routes.clone())?.thread_bytes(); + let bytes = plans.saturating_add(threads.saturating_mul(scratch)); + let sample = WorkspaceBytes::sample(tile_ctx, storage)?.with_tier1(bytes); + tile_ctx.debug_counters.workspace_peak.record(sample); + #[cfg(test)] + { + tile_ctx.debug_counters.pool_band_plan_bytes += plans; + } + for route in routes { + count_decoded_blocks(tile_ctx, &route.plan, route.workers + 1, route.growths); + } Ok(()) } } @@ -1715,9 +2232,209 @@ mod tests { use alloc::vec::Vec; use super::{ - route_fallback, split_runs, tile_route_fallback, CodeBlockCoding, Strip, TaskCounts, - TileRoute, + collect_segments, gather_payload, plan_block, route_fallback, split_runs, + tile_route_fallback, BlockJob, BlockPlacement, BlockScratch, CodeBlock, CodeBlockCoding, + CodeBlockStyle, DecodeAllocationBudget, DecompositionStorage, Layer, Payload, + PayloadSource, ScratchSizes, Segment, Strip, TaskCounts, TileRoute, }; + use crate::j2c::rect::IntRect; + use crate::J2kCodeBlockSegment; + + /// A thread gathers payloads into scratch reserved for the plan's + /// largest before it starts, as charged, and gathering a payload larger + /// than the last never grows it: 1000 bytes in two fragments, then 1500. + #[test] + fn gathered_payloads_stay_in_the_scratch_charged_for_them() { + let data = [7_u8; 1000]; + let segment = |len: usize| Segment { + idx: 0, + coding_pases: 1, + data_length: u32::try_from(len).expect("test segment lengths fit u32"), + data: &data[..len], + }; + let segments = [segment(500), segment(500), segment(1000), segment(500)]; + let layers = [0..1, 1..2, 2..3, 3..4].map(|range| Layer::with_segments(range).unwrap()); + let block = |layers| CodeBlock { + rect: IntRect::from_xywh(0, 0, 4, 4), + x_idx: 0, + y_idx: 0, + layers, + has_been_included: true, + missing_bit_planes: 0, + number_of_coding_passes: 2, + ht_total_coding_passes: 0, + ht_first_cleanup_pass: None, + ht_selected_set: None, + coding: Some(CodeBlockCoding::Classic), + l_block: 3, + non_empty_layer_count: 2, + }; + let code_blocks = [block(0..2), block(2..4)]; + let source = PayloadSource { + code_blocks: &code_blocks, + layers: &layers, + segments: &segments, + }; + let job = |code_block, payload_length| BlockJob { + payload: Payload::Gathered, + code_block, + payload_length, + refinement_length: 0, + placement: BlockPlacement::new( + IntRect::from_xywh(0, 0, 4, 4), + IntRect::from_xywh(0, 0, 4, 4), + ), + width: 4, + height: 4, + missing_bit_planes: 0, + number_of_coding_passes: 2, + band: 0, + }; + let mut budget = DecodeAllocationBudget::from_live_bytes(0).unwrap(); + let mut scratches = BlockScratch::charged_for_threads(1, sizes(1500, 0, 0), &mut budget) + .unwrap() + .expect("scratch within the budget"); + assert_eq!(budget.live_bytes(), 1500 + size_of::()); + let scratch = &mut scratches[0].payload; + for (code_block, len) in [(0, 1000), (1, 1500)] { + let gathered = gather_payload( + CodeBlockCoding::Classic, + &job(code_block, len), + source, + scratch, + ) + .unwrap() + .len(); + assert_eq!(gathered, len as usize); + assert_eq!(scratch.capacity(), 1500); + } + // A payload past the reserved scratch is a planning error, not growth. + let mut short = BlockScratch::for_threads(1, sizes(1000, 0, 0), &mut budget).unwrap(); + assert!(gather_payload( + CodeBlockCoding::Classic, + &job(1, 1500), + source, + &mut short[0].payload + ) + .is_err()); + } + + fn sizes(payload_len: usize, segments: usize, samples: usize) -> ScratchSizes { + ScratchSizes { + payload_len, + segments, + samples, + } + } + + /// Scratch for three threads, each with a 1500-byte payload, seven + /// segment descriptors and a 64 x 64 block, fits a budget of exactly its + /// bytes and their list, which it then holds, and not one byte less. + #[test] + fn thread_scratch_fits_exactly_the_bytes_it_holds() { + let held = 3 + * (1500 + + 7 * size_of::() + + 64 * 64 * size_of::() + + size_of::()); + let at = |cap: usize| { + let mut budget = DecodeAllocationBudget::from_live_bytes_with_cap(100, cap).unwrap(); + let scratches = + BlockScratch::charged_for_threads(3, sizes(1500, 7, 64 * 64), &mut budget).unwrap(); + (scratches, budget.live_bytes()) + }; + let (scratches, live) = at(100 + held); + let scratches = scratches.expect("scratch within its exact budget"); + assert_eq!(live, 100 + held); + assert_eq!(BlockScratch::held_bytes(&scratches), held); + assert!(at(100 + held - 1).0.is_none(), "one byte short"); + } + + /// A classic block of several codeword segments whose bytes are one + /// piece of the codestream, the rest of its segments empty, borrows its + /// payload and collects every segment's descriptor: ten arithmetic + /// passes, then a raw segment of two passes and no bytes. + #[test] + fn a_segmented_block_in_one_piece_borrows_its_payload() { + let data = [7_u8; 100]; + let mut storage = DecompositionStorage::default(); + storage.segments.extend([ + Segment { + idx: 0, + coding_pases: 10, + data_length: 100, + data: &data, + }, + Segment { + idx: 1, + coding_pases: 2, + data_length: 0, + data: &[], + }, + ]); + storage + .layers + .extend([0..1, 1..2].map(|range| Layer::with_segments(range).unwrap())); + let rect = IntRect::from_xywh(0, 0, 4, 4); + storage.code_blocks.push(CodeBlock { + rect, + x_idx: 0, + y_idx: 0, + layers: 0..2, + has_been_included: true, + missing_bit_planes: 0, + number_of_coding_passes: 12, + ht_total_coding_passes: 0, + ht_first_cleanup_pass: None, + ht_selected_set: None, + coding: Some(CodeBlockCoding::Classic), + l_block: 3, + non_empty_layer_count: 1, + }); + let style = CodeBlockStyle { + selective_arithmetic_coding_bypass: true, + ..CodeBlockStyle::default() + }; + let placement = BlockPlacement::new(rect, rect); + let (job, segments) = plan_block( + &storage.code_blocks[0], + 0, + placement, + 0, + &storage, + Some(style), + ) + .unwrap(); + assert!(matches!(job.payload, Payload::Borrowed(payload) if core::ptr::eq(payload, &data))); + assert_eq!(segments, 2); + let mut budget = DecodeAllocationBudget::from_live_bytes(0).unwrap(); + let mut scratches = BlockScratch::for_threads(1, sizes(0, 2, 0), &mut budget).unwrap(); + let BlockScratch { + payload, + segments: descriptors, + .. + } = &mut scratches[0]; + let (payload, descriptors) = collect_segments( + style, + &job, + PayloadSource::new(&storage), + (payload, descriptors), + ) + .unwrap(); + assert!(core::ptr::eq(payload, &data)); + let segment = + |data_offset, data_length, start_coding_pass, end_coding_pass| J2kCodeBlockSegment { + data_offset, + data_length, + start_coding_pass, + end_coding_pass, + use_arithmetic: start_coding_pass == 0, + }; + assert_eq!( + descriptors, + [segment(0, 100, 0, 10), segment(100, 0, 10, 12)] + ); + } #[test] fn split_runs_cover_the_strip_in_balanced_parts() { diff --git a/crates/j2k-native/src/j2c/decode/tier1.rs b/crates/j2k-native/src/j2c/decode/tier1.rs index f5db2b15a..ec388de7f 100644 --- a/crates/j2k-native/src/j2c/decode/tier1.rs +++ b/crates/j2k-native/src/j2c/decode/tier1.rs @@ -2,15 +2,16 @@ //! One active-tile budget for serial Tier-1 contexts and copied segment scratch. -use super::subband::code_block_required_by_index; +use super::subband::{code_block_required_by_index, stored_placement}; use super::{ - CodeBlock, DecodeAllocationBudget, DecompositionStorage, Header, Result, Tile, + CodeBlock, DecodeAllocationBudget, DecompositionStorage, Header, Result, SubBand, Tile, TileDecodeContext, }; use crate::error::{DecodingError, ValidationError}; use crate::j2c::bitplane::classic_decode_workspace_bytes; use crate::j2c::build::CodeBlockCoding; use crate::j2c::ht_block_decode::ht_decode_workspace_bytes; +use crate::try_resize_decode_elements; use core::mem::size_of; use core::ops::Range; @@ -27,6 +28,7 @@ struct Tier1Requirements { ht_width: u32, ht_height: u32, ht_data_bytes: usize, + ht_clipped_samples: usize, has_classic: bool, has_ht: bool, } @@ -48,7 +50,7 @@ impl Tier1Requirements { .get(code_block.layers.clone()) .ok_or(DecodingError::CodeBlockDecodeFailure)?; for layer in layers { - let Some(segment_range) = layer.segments.clone() else { + let Some(segment_range) = layer.segments() else { continue; }; let segments = storage @@ -75,11 +77,24 @@ impl Tier1Requirements { fn observe_ht( &mut self, code_block: &CodeBlock, + sub_band: &SubBand, storage: &DecompositionStorage<'_>, ) -> Result<()> { self.has_ht = true; - self.ht_width = self.ht_width.max(code_block.rect.width()); - self.ht_height = self.ht_height.max(code_block.rect.height()); + let (width, height) = (code_block.rect.width(), code_block.rect.height()); + self.ht_width = self.ht_width.max(width); + self.ht_height = self.ht_height.max(height); + // A block decoding straight to samples writes all of them, so one + // the sub-band stores only part of needs scratch. + if cfg!(feature = "simd") + && code_block.number_of_coding_passes != 0 + && !stored_placement(code_block, sub_band).is_whole(width, height) + { + let samples = (width as usize) + .checked_mul(height as usize) + .ok_or(ValidationError::ImageTooLarge)?; + self.ht_clipped_samples = self.ht_clipped_samples.max(samples); + } if code_block.number_of_coding_passes != 0 { let (cleanup_bytes, refinement_bytes) = crate::j2c::ht_block_decode::selected_code_block_segment_lengths( @@ -112,6 +127,7 @@ impl Tier1Requirements { .checked_add(ht_decode_workspace_bytes(self.ht_width, self.ht_height)?) .ok_or(ValidationError::ImageTooLarge)?; include_elements::(&mut bytes, self.ht_data_bytes)?; + include_elements::(&mut bytes, self.ht_clipped_samples)?; } Ok(bytes) } @@ -146,6 +162,13 @@ pub(super) fn prepare_tier1_workspace( requirements.ht_height, requirements.ht_data_bytes, )?; + // Sized now, so a band decoding after rows went out never grows + // it past the budget. + let clipped = &mut tile_ctx.ht_clipped_block_scratch; + if clipped.len() < requirements.ht_clipped_samples { + clipped.clear(); + try_resize_decode_elements(clipped, requirements.ht_clipped_samples, 0.0)?; + } } let actual_bytes = tile_ctx.serial_tier1_capacity_bytes()?; @@ -243,7 +266,7 @@ fn observe_sub_band( requirements.observe_classic(code_block, storage)?; } Some(CodeBlockCoding::HighThroughput) => { - requirements.observe_ht(code_block, storage)?; + requirements.observe_ht(code_block, sub_band, storage)?; } None if code_block.number_of_coding_passes == 0 => {} None => return Err(DecodingError::CodeBlockDecodeFailure.into()), diff --git a/crates/j2k-native/src/j2c/decode/workspace.rs b/crates/j2k-native/src/j2c/decode/workspace.rs index 7d8e49163..791047908 100644 --- a/crates/j2k-native/src/j2c/decode/workspace.rs +++ b/crates/j2k-native/src/j2c/decode/workspace.rs @@ -207,7 +207,8 @@ impl Default for DecoderContext<'_> { impl DecoderContext<'_> { /// Create a borrowing decoder context from a lifetime-free reusable workspace. #[must_use] - pub fn from_workspace(workspace: DecoderWorkspace) -> Self { + pub fn from_workspace(mut workspace: DecoderWorkspace) -> Self { + workspace.tile_decode_context.shared_pool = false; Self { tile_decode_context: workspace.tile_decode_context, storage: DecompositionStorage::default(), @@ -223,6 +224,7 @@ impl DecoderContext<'_> { #[must_use] pub fn into_workspace(mut self) -> DecoderWorkspace { self.storage.release_all_allocations(); + self.tile_decode_context.shared_pool = false; DecoderWorkspace { tile_decode_context: self.tile_decode_context, cpu_decode_parallelism: self.cpu_decode_parallelism, @@ -316,6 +318,19 @@ impl DecoderContext<'_> { self.cpu_decode_parallelism = parallelism; } + /// Mark this context's decodes as sharing their worker pool with other + /// images that decode on it at the same time, as the images of a batch do + /// that plans as many at once as its pool has threads. + /// Pool threads with no band of one image to decode take another image's + /// bands, so on a pool of fewer than 12 threads each image's band rings + /// hold two bands at every level and its bands stay whole; larger pools + /// keep the usual rings. Off by default; recovering the context's + /// workspace turns it off. + #[doc(hidden)] + pub fn set_shared_pool(&mut self, shared: bool) { + self.tile_decode_context.shared_pool = shared; + } + /// Turn the row-streamed region IDWT on or off; it is on by default and /// lowers region-decode memory without changing output. #[cfg(test)] diff --git a/crates/j2k-native/src/j2c/encode/single_tile/tests.rs b/crates/j2k-native/src/j2c/encode/single_tile/tests.rs index fe6ec8527..2bf0e747d 100644 --- a/crates/j2k-native/src/j2c/encode/single_tile/tests.rs +++ b/crates/j2k-native/src/j2c/encode/single_tile/tests.rs @@ -337,7 +337,7 @@ fn contribution_final_bytes(codestream: &[u8]) -> Vec { segment::parse(tile, &parsed.header, &mut storage).expect("parse encoded packets"); for code_block in &storage.code_blocks { for layer in &storage.layers[code_block.layers.clone()] { - let Some(segments) = layer.segments.clone() else { + let Some(segments) = layer.segments() else { continue; }; if let Some(&last) = storage.segments[segments] diff --git a/crates/j2k-native/src/j2c/ht_block_decode/magnitude.rs b/crates/j2k-native/src/j2c/ht_block_decode/magnitude.rs index ac89adb3c..c6158e726 100644 --- a/crates/j2k-native/src/j2c/ht_block_decode/magnitude.rs +++ b/crates/j2k-native/src/j2c/ht_block_decode/magnitude.rs @@ -24,6 +24,26 @@ pub(super) fn supports_cleanup_to_f32(missing_msbs: u32, width: u32) -> bool { width != 0 && width.is_multiple_of(4) && uses_simd_rows(missing_msbs) } +/// Validate cleanup scratch and clear the preceding quad row's magnitudes. +fn prepare_cleanup_scratch( + width: u32, + height: u32, + sstr: usize, + scratch: &[u16], + v_n_scratch: &mut [u32], +) -> Option<()> { + let quad_rows = height.div_ceil(2) as usize; + if scratch.len() < sstr * (quad_rows + 1) { + return None; + } + let v_n_width = width.div_ceil(2) as usize + 2; + if v_n_scratch.len() < v_n_width { + return None; + } + v_n_scratch[..v_n_width].fill(0); + Some(()) +} + fn sample_mask(bit: u32) -> u32 { 1 << (4 + bit) } @@ -354,15 +374,7 @@ pub(super) fn decode_cleanup_and_magnitude_sign_phase( vlc_buffer: &mut Vec, observer: &mut impl HtDecodeObserver, ) -> Option<()> { - let quad_rows = height.div_ceil(2) as usize; - if scratch.len() < sstr * (quad_rows + 1) { - return None; - } - let v_n_width = width.div_ceil(2) as usize + 2; - if v_n_scratch.len() < v_n_width { - return None; - } - v_n_scratch[..v_n_width].fill(0); + prepare_cleanup_scratch(width, height, sstr, scratch, v_n_scratch)?; let mut mel = MelDecoder::new(coded_data, lcup, scup); let consumable = (width as usize) @@ -465,15 +477,7 @@ pub(super) fn decode_cleanup_and_magnitude_sign_to_f32 Result { let previous_row_len = (width.div_ceil(4) as usize) .checked_add(8) .ok_or(ValidationError::ImageTooLarge)?; + #[cfg(feature = "simd")] + let vlc_bytes = flat_vlc_capacity(width, height); + #[cfg(not(feature = "simd"))] + let vlc_bytes = 0; Ok(HtWorkspaceLayout { coefficients: coefficient_len, cleanup_symbols: cleanup_len, @@ -50,7 +56,7 @@ fn workspace_layout(width: u32, height: u32) -> Result { sigma_symbols: sigma_len, previous_row_symbols: previous_row_len, magsgn_bytes: flat_magsgn_capacity(coefficient_len)?, - vlc_bytes: flat_vlc_capacity(width, height), + vlc_bytes, }) } diff --git a/crates/j2k-native/src/j2c/ht_block_decode/readers.rs b/crates/j2k-native/src/j2c/ht_block_decode/readers.rs index 9cc51fd39..6f93505bd 100644 --- a/crates/j2k-native/src/j2c/ht_block_decode/readers.rs +++ b/crates/j2k-native/src/j2c/ht_block_decode/readers.rs @@ -590,6 +590,10 @@ pub(super) fn read_u32_pair(values: &[u16], index: usize) -> u32 { mod tests; /// Largest cleanup suffix (MEL + VLC) in bytes, Clause 7.1.4. +#[cfg_attr( + not(any(test, feature = "simd")), + expect(dead_code, reason = "only SIMD HT decoding sizes flat VLC bits") +)] pub(super) const MAX_CLEANUP_SUFFIX: usize = 4079; /// Padding after the unstuffed VLC bytes: one top-up window past the data, @@ -599,6 +603,10 @@ const FLAT_VLC_PADDING: usize = 16; /// Bytes of VLC data a `width` x `height` block can consume: a quad pair takes /// at most 30 bits (two seven-bit codewords and a 16-bit U-VLC pair), and the /// pairs of a quad row cover four columns. +#[cfg_attr( + not(any(test, feature = "simd")), + expect(dead_code, reason = "only SIMD HT decoding sizes flat VLC bits") +)] pub(super) fn flat_vlc_consumable(width: u32, height: u32) -> usize { let pairs = (width.div_ceil(4) as usize).saturating_mul(height.div_ceil(2) as usize); pairs @@ -609,6 +617,10 @@ pub(super) fn flat_vlc_consumable(width: u32, height: u32) -> usize { } /// Capacity of the flat VLC buffer of a `width` x `height` block. +#[cfg_attr( + not(any(test, feature = "simd")), + expect(dead_code, reason = "only SIMD HT decoding reserves flat VLC bits") +)] pub(super) fn flat_vlc_capacity(width: u32, height: u32) -> usize { flat_vlc_consumable(width, height) + FLAT_VLC_PADDING } diff --git a/crates/j2k-native/src/j2c/ht_block_decode/segments.rs b/crates/j2k-native/src/j2c/ht_block_decode/segments.rs index 8b7c40981..8a64b1a2c 100644 --- a/crates/j2k-native/src/j2c/ht_block_decode/segments.rs +++ b/crates/j2k-native/src/j2c/ht_block_decode/segments.rs @@ -71,7 +71,7 @@ pub(crate) fn visit_code_block_segments<'a>( let mut cleanup_seen = false; for layer in &storage.layers[code_block.layers.start..code_block.layers.end] { - let Some(range) = layer.segments.clone() else { + let Some(range) = layer.segments() else { continue; }; diff --git a/crates/j2k-native/src/j2c/ht_block_decode/segments/owned.rs b/crates/j2k-native/src/j2c/ht_block_decode/segments/owned.rs index 00aae36f5..5f93250c7 100644 --- a/crates/j2k-native/src/j2c/ht_block_decode/segments/owned.rs +++ b/crates/j2k-native/src/j2c/ht_block_decode/segments/owned.rs @@ -17,7 +17,7 @@ pub(crate) fn selected_code_block_segment_lengths( let mut refinement_length = 0usize; for layer in &storage.layers[code_block.layers.clone()] { - let Some(range) = layer.segments.clone() else { + let Some(range) = layer.segments() else { continue; }; @@ -64,7 +64,7 @@ pub(crate) fn collect_code_block_data_into( data.clear(); for layer in &storage.layers[code_block.layers.clone()] { - let Some(range) = layer.segments.clone() else { + let Some(range) = layer.segments() else { continue; }; for segment in &storage.segments[range] { diff --git a/crates/j2k-native/src/j2c/ht_block_decode/segments/tests.rs b/crates/j2k-native/src/j2c/ht_block_decode/segments/tests.rs index 944115358..8d1459520 100644 --- a/crates/j2k-native/src/j2c/ht_block_decode/segments/tests.rs +++ b/crates/j2k-native/src/j2c/ht_block_decode/segments/tests.rs @@ -23,9 +23,9 @@ fn collector_uses_the_selected_non_empty_ht_set() { non_empty_layer_count: 1, }; let mut storage = DecompositionStorage::default(); - storage.layers.push(Layer { - segments: Some(0..4), - }); + storage + .layers + .push(Layer::with_segments(0..4).expect("test segment range")); storage.segments.extend([ Segment { idx: 0, @@ -88,12 +88,8 @@ fn referenced_collector_visits_refinement_fragments_in_packet_order() { }; let mut storage = DecompositionStorage::default(); storage.layers.extend([ - Layer { - segments: Some(0..2), - }, - Layer { - segments: Some(2..3), - }, + Layer::with_segments(0..2).expect("test segment range"), + Layer::with_segments(2..3).expect("test segment range"), ]); storage.segments.extend([ Segment { @@ -152,12 +148,8 @@ fn owned_collector_concatenates_refinement_fragments_in_packet_order() { }; let mut storage = DecompositionStorage::default(); storage.layers.extend([ - Layer { - segments: Some(0..2), - }, - Layer { - segments: Some(2..3), - }, + Layer::with_segments(0..2).expect("test segment range"), + Layer::with_segments(2..3).expect("test segment range"), ]); storage.segments.extend([ Segment { diff --git a/crates/j2k-native/src/j2c/ht_block_decode/tests.rs b/crates/j2k-native/src/j2c/ht_block_decode/tests.rs index b884451eb..1f5464f22 100644 --- a/crates/j2k-native/src/j2c/ht_block_decode/tests.rs +++ b/crates/j2k-native/src/j2c/ht_block_decode/tests.rs @@ -597,6 +597,10 @@ fn decode_combined_validated_with_scratch_reuses_zeroed_buffers() { let first_capacities = scratch.capacities_for_test(); assert!(first_capacities.cleanup > 0); assert!(first_capacities.v_n > 0); + #[cfg(feature = "simd")] + assert!(first_capacities.vlc > 0); + #[cfg(not(feature = "simd"))] + assert_eq!(first_capacities.vlc, 0); scratch.poison_for_test(); decoded.fill(0); diff --git a/crates/j2k-native/src/j2c/idwt.rs b/crates/j2k-native/src/j2c/idwt.rs index 5e57feace..c820c116f 100644 --- a/crates/j2k-native/src/j2c/idwt.rs +++ b/crates/j2k-native/src/j2c/idwt.rs @@ -2,7 +2,7 @@ use alloc::vec::Vec; -use crate::{try_resize_decode_elements, Result}; +use crate::{try_zero_fill_decode_elements, Result}; mod banded; mod direct; @@ -27,6 +27,8 @@ pub(crate) use lines::{LineReconstruction, SubBandRows}; pub(crate) use model::idwt_buffer_size; pub(crate) use model::{IDWTOutput, IdwtPlacement}; pub(crate) use orchestrate::{apply, planned_workspace_elements}; +#[cfg(feature = "parallel")] +pub(crate) use roi::streamed_batch_rows; pub(crate) use roi::{last_level_fits_channels, roi_idwt_peak_elements, ComponentLines, LinePlan}; // Decoded high-pass sub-bands include JPEG 2000's orientation gain. OpenJPEG's @@ -63,7 +65,12 @@ fn try_grow_level_buffer(buffer: &mut Vec, len: usize) -> Result<()> { } #[cfg(test)] ZERO_INITIALIZED_SAMPLES.set(ZERO_INITIALIZED_SAMPLES.get() + len - buffer.len()); - try_resize_decode_elements(buffer, len, 0.0) + if buffer.capacity() < len { + // Nothing left behind is read, so a buffer that must grow is replaced + // by a zeroed one rather than copied into a larger allocation. + buffer.clear(); + } + try_zero_fill_decode_elements(buffer, len) } #[cfg(test)] diff --git a/crates/j2k-native/src/j2c/idwt/banded.rs b/crates/j2k-native/src/j2c/idwt/banded.rs index afa8993e8..177ed7c13 100644 --- a/crates/j2k-native/src/j2c/idwt/banded.rs +++ b/crates/j2k-native/src/j2c/idwt/banded.rs @@ -28,6 +28,7 @@ use super::horizontal::filter_row; use super::interleave::interleave_row; use super::model::{IDWTInput, IDWTTempOutput}; use super::try_grow_level_buffer; +use super::vertical::irreversible_row_scale; #[cfg(feature = "parallel")] use crate::error::{bail, DecodingError}; use crate::math::{self, dispatch, f32x8, Level, Simd, SIMD_WIDTH}; @@ -58,7 +59,7 @@ const RING_ROWS: usize = MAX_HALO_ROWS + 2; const MIN_STRIP_ROWS: usize = 64; /// Scratch elements for the two horizontally lifted bands of one row. -fn horizontal_scratch_elements(width: usize) -> Result { +pub(super) fn horizontal_scratch_elements(width: usize) -> Result { if width < MIN_SIMD_ROW_WIDTH { return Ok(0); } @@ -305,6 +306,23 @@ impl<'a> Level2d<'a> { } } + fn lifting(&self) -> Lifting { + Lifting { + transform: self.transform, + y0: self.y0, + height: self.height, + } + } + + fn row_geometry(&self) -> RowGeometry { + RowGeometry { + width: self.width, + x0: self.x0, + transform: self.transform, + high_pass: self.high_pass, + } + } + /// This level with its LL band rows `first_row..` taken from `rows`. #[cfg(feature = "parallel")] fn with_ll_rows(mut self, rows: &'a [f32], first_row: usize) -> Self { @@ -319,6 +337,26 @@ impl<'a> Level2d<'a> { } } +/// What the vertical lifting of a level needs from it: its transform, its +/// first row, whose parity says which rows each step updates, and its height. +#[derive(Clone, Copy)] +pub(super) struct Lifting { + pub(super) transform: WaveletTransform, + pub(super) y0: usize, + pub(super) height: usize, +} + +/// What the horizontal synthesis of one row needs from its level: the row's +/// width, its first column, whose parity says whether it starts low-pass, the +/// transform, and the 9-7I high-pass scale. +#[derive(Clone, Copy)] +pub(super) struct RowGeometry { + pub(super) width: usize, + pub(super) x0: usize, + pub(super) transform: WaveletTransform, + pub(super) high_pass: f32, +} + pub(super) fn filter_2d_banded( input: IDWTInput<'_>, decomposition: &Decomposition, @@ -407,11 +445,15 @@ pub(super) fn filter_2d_banded_from_bands( try_grow_level_buffer(band_scratch, scratch_len)?; dispatch!(Level::new(), simd => run_strips( simd, - level, + level.width, target, parallel_layout.strip_len, &mut band_scratch[..scratch_len], per_worker, + #[inline(always)] + |simd, first_row, output, worker_scratch| { + stream_strip(simd, level, first_row, output, worker_scratch); + }, )); } Ok(()) @@ -421,11 +463,12 @@ pub(super) fn filter_2d_banded_from_bands( #[inline(always)] fn run_strips( simd: S, - level: Level2d<'_>, + width: usize, target: &mut [f32], strip_len: usize, scratch: &mut [f32], per_worker: usize, + strip_kernel: impl Fn(S, usize, &mut [f32], &mut [f32]) + Sync, ) { use rayon::prelude::*; use std::sync::Mutex; @@ -446,15 +489,15 @@ fn run_strips( #[inline(always)] || { while let Some((strip, output)) = next() { - let first_row = strip * strip_len / level.width; - stream_strip(simd, level, first_row, output, worker_scratch); + let first_row = strip * strip_len / width; + strip_kernel(simd, first_row, output, worker_scratch); } }, ); #[cfg(not(feature = "simd"))] while let Some((strip, output)) = next() { - let first_row = strip * strip_len / level.width; - stream_strip(simd, level, first_row, output, worker_scratch); + let first_row = strip * strip_len / width; + strip_kernel(simd, first_row, output, worker_scratch); } }); } @@ -532,13 +575,24 @@ fn filter_levels_pair( let steps = lifting_steps(outer_level.transform); let scratch = PairScratch::new(inner_level.width, width, layout, steps)?; try_grow_level_buffer(band_scratch, scratch.len)?; - dispatch!(Level::new(), simd => run_pair_strips( + dispatch!(Level::new(), simd => run_strips( simd, - [inner_level, outer_level], + outer_level.width, target, layout.strip_len, &mut band_scratch[..scratch.len], - [scratch.per_worker, scratch.rows_len], + scratch.per_worker, + #[inline(always)] + |simd, first_row, output, worker_scratch| { + pair_strip( + simd, + [inner_level, outer_level], + first_row, + output, + worker_scratch, + scratch.rows_len, + ); + }, )); Ok(()) } @@ -592,45 +646,6 @@ const fn pair_ll_rows(strip_rows: usize, count: usize) -> usize { strip_rows.div_ceil(2) + count + 3 } -#[cfg(feature = "parallel")] -#[inline(always)] -fn run_pair_strips( - simd: S, - levels: [Level2d<'_>; 2], - target: &mut [f32], - strip_len: usize, - scratch: &mut [f32], - [per_worker, rows_len]: [usize; 2], -) { - use rayon::prelude::*; - use std::sync::Mutex; - - let strips = Mutex::new(target.chunks_mut(strip_len).enumerate()); - let next = || strips.lock().ok().and_then(|mut strips| strips.next()); - let width = levels[1].width; - scratch - .par_chunks_mut(per_worker) - .with_max_len(1) - .for_each(|worker_scratch| { - // As in `run_strips`. - #[cfg(feature = "simd")] - simd.vectorize( - #[inline(always)] - || { - while let Some((strip, output)) = next() { - let first_row = strip * strip_len / width; - pair_strip(simd, levels, first_row, output, worker_scratch, rows_len); - } - }, - ); - #[cfg(not(feature = "simd"))] - while let Some((strip, output)) = next() { - let first_row = strip * strip_len / width; - pair_strip(simd, levels, first_row, output, worker_scratch, rows_len); - } - }); -} - /// Reconstruct rows `first_row..` of `outer` into `output`, after the rows /// of `inner` they read. #[cfg(feature = "parallel")] @@ -647,7 +662,7 @@ fn pair_strip( worker_scratch: &mut [f32], rows_len: usize, ) { - let (_, count) = vertical_steps(outer); + let (_, count) = vertical_steps(outer.lifting()); let end = first_row + output.len() / outer.width; // The arrivals of `stream_strip` and the LL band rows they read. let lo = first_row.saturating_sub(count); @@ -680,7 +695,7 @@ enum VerticalStep { /// The vertical lifting steps of a level in order, each with the parity of /// the rows it updates, and how many there are. -fn vertical_steps(level: Level2d<'_>) -> ([(usize, VerticalStep); MAX_HALO_ROWS], usize) { +fn vertical_steps(level: Lifting) -> ([(usize, VerticalStep); MAX_HALO_ROWS], usize) { let low = level.y0 % 2; let high = 1 - low; match level.transform { @@ -736,31 +751,11 @@ fn synthesize_level_row( low: &mut [f32], high: &mut [f32], ) { - let row_scale = if level.transform == WaveletTransform::Irreversible97 { - if index % 2 == level.y0 % 2 { - dwt::DWT97_KAPPA_F32 - } else { - level.high_pass - } - } else { - 1.0 - }; - let source = if index % 2 == level.even.start_row % 2 { - level.even.row(index) - } else { - level.odd.row(index) - }; - match source { - Some((first, second)) => { - synthesize_row(simd, level, first, second, row, low, high, row_scale); - } - // Lifting maps an all-zero row to zeros. - None => row.fill(0.0), - } + synthesize_level_columns(simd, level, index, row, 0..level.width, low, high); } -/// [`synthesize_level_row`] restricted to `columns`, one of the level's -/// [`column_strips`], into `out`. +/// Synthesize level row `index` over `columns` (the whole row, or one of the +/// level's [`column_strips`]) into `out`, with 9-7I row scaling. #[expect( clippy::inline_always, reason = "specialized with the streaming kernels" @@ -775,17 +770,11 @@ fn synthesize_level_columns( low: &mut [f32], high: &mut [f32], ) { - if columns.len() == level.width { - synthesize_level_row(simd, level, index, out, low, high); - return; - } - // Strips are only made of even-origin levels (`column_strips`). let row_scale = if level.transform == WaveletTransform::Reversible53 { 1.0 - } else if index % 2 == level.y0 % 2 { - dwt::DWT97_KAPPA_F32 } else { - level.high_pass + let y0_parity = u32::from(!level.y0.is_multiple_of(2)); + irreversible_row_scale(y0_parity, index.is_multiple_of(2), level.high_pass) }; let source = if index % 2 == level.even.start_row % 2 { level.even.row(index) @@ -793,17 +782,34 @@ fn synthesize_level_columns( level.odd.row(index) }; match source { - Some((low_source, high_source)) => synthesize_row_columns( - simd, - level, - low_source, - high_source, - out, - columns, - low, - high, - row_scale, - ), + Some((low_source, high_source)) => { + if columns.len() == level.width { + synthesize_row( + simd, + level.row_geometry(), + low_source, + high_source, + out, + low, + high, + row_scale, + ); + } else { + // Strips are only made of even-origin levels (`column_strips`). + synthesize_row_columns( + simd, + level, + low_source, + high_source, + out, + columns, + low, + high, + row_scale, + ); + } + } + // Lifting maps an all-zero row to zeros. None => out.fill(0.0), } } @@ -832,7 +838,7 @@ fn stream_level( ) { let width = level.width; let height = level.height; - let (_, count) = vertical_steps(level); + let (_, count) = vertical_steps(level.lifting()); let (low, high) = split_horizontal_scratch(level, horizontal_scratch); let mut rows = RingRows { rows: target, @@ -845,7 +851,7 @@ fn stream_level( for arrival in arrivals.start..arrivals.end.min(height) { synthesize_level_row(simd, level, arrival, rows.row_mut(arrival), low, high); } - lift_arrivals(simd, level, arrivals, 0..height, &mut rows); + lift_arrivals(simd, level.lifting(), arrivals, 0..height, &mut rows); } } @@ -870,7 +876,7 @@ fn stream_strip( ) { let width = level.width; let height = level.height; - let (_, count) = vertical_steps(level); + let (_, count) = vertical_steps(level.lifting()); let end = first_row + output.len() / width; let lo = first_row.saturating_sub(count); let hi = (end + count).min(height); @@ -889,7 +895,7 @@ fn stream_strip( for arrival in arrivals.start..arrivals.end.min(hi) { synthesize_level_row(simd, level, arrival, rows.row_mut(arrival), low, high); } - lift_arrivals(simd, level, arrivals, lo..hi, &mut rows); + lift_arrivals(simd, level.lifting(), arrivals, lo..hi, &mut rows); } } @@ -1081,6 +1087,33 @@ trait LevelRows { fn window(&mut self, first: usize) -> Option<[&mut [f32]; ROWS]>; } +/// Run the vertical lifting steps of `arrivals` (see [`lift_arrivals`]) on +/// every row of a level `width` samples wide whose rows live in `ring`, row +/// `n` in slot `n % (ring.len() / width)`, at the CPU's SIMD level, for +/// callers outside a dispatch. The ring must hold every row the arrivals' +/// steps read (see [`lift_arrivals`]). +pub(super) fn lift_ring_arrivals( + level: Lifting, + width: usize, + ring: &mut [f32], + arrivals: Range, +) { + let slots = ring.len().checked_div(width).unwrap_or(0); + let mut rows = RingRows { + rows: ring, + width, + stride: width, + slots, + }; + dispatch!(Level::new(), simd => lift_arrivals( + simd, + level, + arrivals, + 0..level.height, + &mut rows, + )); +} + /// Arrivals [`stream_level`] and [`stream_strip`] synthesize before they run /// those arrivals' vertical steps, so one sweep of the columns runs the /// steps of several wavefronts. @@ -1107,7 +1140,7 @@ const SWEEP_WAVEFRONTS: usize = 2; #[inline(always)] fn lift_arrivals( simd: S, - level: Level2d<'_>, + level: Lifting, arrivals: Range, live: Range, rows: &mut impl LevelRows, @@ -1498,7 +1531,39 @@ pub(in crate::j2c) fn lift_rows( } /// Rows narrower than this use the scalar interleaved `1D_SR`. -const MIN_SIMD_ROW_WIDTH: usize = 2 * SIMD_WIDTH; +pub(super) const MIN_SIMD_ROW_WIDTH: usize = 2 * SIMD_WIDTH; + +/// [`synthesize_row`] at the CPU's SIMD level, for callers outside a +/// dispatch: the horizontal synthesis of one row of `geometry` from its two +/// band rows, `first` holding the samples of column `x0`'s parity, written +/// interleaved into `row` and, for 9-7I, scaled by `row_scale`. `scratch` +/// holds at least [`horizontal_scratch_elements`] samples. +pub(super) fn synthesize_band_row( + geometry: RowGeometry, + first: &[f32], + second: &[f32], + row: &mut [f32], + scratch: &mut [f32], + row_scale: f32, +) { + let (low_len, high_len) = if geometry.x0.is_multiple_of(2) { + (first.len(), second.len()) + } else { + (second.len(), first.len()) + }; + let (low, rest) = scratch.split_at_mut(low_len + 2); + let high = &mut rest[..high_len + 2]; + dispatch!(Level::new(), simd => synthesize_row( + simd, + geometry, + first, + second, + row, + low, + high, + row_scale, + )); +} /// Horizontal `1D_SR` (F.3.6) of one row from its two band rows, written /// interleaved into `row`. @@ -1518,7 +1583,7 @@ const MIN_SIMD_ROW_WIDTH: usize = 2 * SIMD_WIDTH; #[inline(always)] fn synthesize_row( simd: S, - level: Level2d<'_>, + level: RowGeometry, first: &[f32], second: &[f32], row: &mut [f32], @@ -2304,7 +2369,7 @@ mod tests { for (width, height) in [(1, 9), (7, 12), (8, 13), (9, 40), (17, 31), (33, 70)] { let rect = IntRect::from_xywh(3, y0, width, height); let level = Level2d::new(rect, [&[]; 4], transform, 1.25); - let (steps, count) = vertical_steps(level); + let (steps, count) = vertical_steps(level.lifting()); let height = height as usize; let width = width as usize; // The whole level, and strip-like windows with halos. @@ -2340,7 +2405,7 @@ mod tests { let arrivals = first..(first + batch).min(live.end + count); dispatch!(Level::new(), simd => lift_arrivals( simd, - level, + level.lifting(), arrivals.clone(), live.clone(), &mut rows, diff --git a/crates/j2k-native/src/j2c/idwt/banded/last_level.rs b/crates/j2k-native/src/j2c/idwt/banded/last_level.rs index 175669936..27fd87f9b 100644 --- a/crates/j2k-native/src/j2c/idwt/banded/last_level.rs +++ b/crates/j2k-native/src/j2c/idwt/banded/last_level.rs @@ -314,7 +314,7 @@ fn stream_rows( let height = levels[0].height; let row_bytes = 3 * width; let end = first_row + output.len() / row_bytes; - let (_, count) = vertical_steps(levels[0]); + let (_, count) = vertical_steps(levels[0].lifting()); let lo = first_row.saturating_sub(count); let hi = (end + count).min(height); let ring_rows = ring_rows(batch); @@ -378,7 +378,7 @@ fn stream_rows( high, ); } - lift_arrivals(simd, *level, ahead, lo..hi, &mut rows); + lift_arrivals(simd, level.lifting(), ahead, lo..hi, &mut rows); } } emit_rows(rings, index, emitted..unread.max(emitted)); diff --git a/crates/j2k-native/src/j2c/idwt/lines.rs b/crates/j2k-native/src/j2c/idwt/lines.rs index d3f30b4a5..fe12c13cb 100644 --- a/crates/j2k-native/src/j2c/idwt/lines.rs +++ b/crates/j2k-native/src/j2c/idwt/lines.rs @@ -8,46 +8,66 @@ //! window's rows top to bottom, pulling the LL rows it needs from the level //! below, and the caller receives the last level's rows one at a time. //! -//! Vertical lifting is streamed by keeping, for each step, how many rows the -//! step has finished. A step updates the rows of one parity from the rows -//! above and below them as the previous step left them. It can finish a row -//! once the previous step has finished every row it reads, and the next step -//! reaches a row only after this one has finished both of its neighbours, so -//! every read sees the value the whole-window filter would see. +//! Vertical lifting is streamed with the schedule of the banded IDWT (see +//! `banded::lift_arrivals`): when a level's row `n` arrives, step `k` +//! (1-based) runs on row `n - k` if that row has the step's parity. Each +//! step then reads its row's neighbours exactly as the previous step left +//! them, so every read sees the value the whole-window filter would see, and +//! a row is final as soon as the inputs its last step needs have arrived. use alloc::vec::Vec; use super::super::codestream::WaveletTransform; use super::super::decode::DecompositionStorage; use super::super::rect::IntRect; -use super::filter_common::periodic_symmetric_extension_left; +use super::banded::{ + horizontal_scratch_elements, lift_ring_arrivals, synthesize_band_row, Lifting, RowGeometry, + MIN_SIMD_ROW_WIDTH, +}; use super::horizontal::filter_row; use super::model::CoefficientSource; use super::roi::interleave_samples_roi; -use super::vertical::{ - irreversible_row_scale, lift_one_row, scale_one_row, split_rows, vertical_step_count, - vertical_steps, VerticalStep, -}; +use super::vertical::{irreversible_row_scale, scale_one_row, vertical_step_count, vertical_steps}; use crate::error::{DecodingError, ValidationError}; use crate::Result; -/// Rows each level keeps: one per lifting step, the row above the oldest -/// unfinished one, and the newest row. +/// Rows each level keeps: the rows an arrival's steps read, one per lifting +/// step above the arrival, its own row and the row above the deepest step's +/// (see `banded::lift_arrivals`). pub(crate) const fn ring_rows(transform: WaveletTransform) -> usize { vertical_step_count(transform) + 2 } -/// Samples the rings of one component's levels hold together. +/// Samples the rings of one component's levels hold together, with the +/// scratch their rows' horizontal synthesis shares. pub(crate) fn ring_samples( windows: impl IntoIterator, transform: WaveletTransform, ) -> Result { - windows.into_iter().try_fold(0usize, |total, window| { - (window.width() as usize) - .checked_mul(ring_rows(transform)) - .and_then(|samples| total.checked_add(samples)) - .ok_or(ValidationError::ImageTooLarge.into()) - }) + let mut widest = 0; + let rings = windows.into_iter().try_fold(0usize, |total, window| { + widest = widest.max(window.width() as usize); + level_ring_samples(window, transform).and_then(|samples| { + total + .checked_add(samples) + .ok_or(ValidationError::ImageTooLarge.into()) + }) + })?; + rings + .checked_add(synthesis_scratch_samples(widest)?) + .ok_or(ValidationError::ImageTooLarge.into()) +} + +/// Samples the ring of a level with this window holds. +fn level_ring_samples(window: IntRect, transform: WaveletTransform) -> Result { + (window.width() as usize) + .checked_mul(ring_rows(transform)) + .ok_or(ValidationError::ImageTooLarge.into()) +} + +/// Scratch for the horizontal synthesis of a row of the widest level. +fn synthesis_scratch_samples(widest: usize) -> Result { + horizontal_scratch_elements(widest) } /// Where the streamed IDWT reads the rows of a component's stored sub-bands. @@ -96,8 +116,9 @@ struct LevelRows<'r> { ring: &'r mut [f32], /// Rows interleaved, filtered horizontally and scaled. fed: usize, - /// Rows each vertical step has finished. - finished: [usize; 4], + /// Arrivals whose vertical steps have run: one per fed row, then the + /// arrivals past the last row that finish the rows above it. + arrived: usize, } impl LevelRows<'_> { @@ -118,21 +139,76 @@ impl LevelRows<'_> { &self.ring[self.slot(row) * width..][..width] } - /// Rows that are final. + fn row_mut(&mut self, row: usize) -> &mut [f32] { + let width = self.width(); + let slot = self.slot(row); + &mut self.ring[slot * width..][..width] + } + + /// Rows that are final. Row `r` takes its last step at arrival `r + k`, + /// where `k` is that step: the last of the transform's steps for the rows + /// that step updates, the one before it for the others. fn done(&self, transform: WaveletTransform) -> usize { - if self.height() == 1 { - self.fed + let height = self.height(); + if height == 1 { + return self.fed; + } + let Some(last_arrival) = self.arrived.checked_sub(1) else { + return 0; + }; + let steps = vertical_step_count(transform); + let (_, last_step_even) = vertical_steps(transform, self.window.y0)[steps - 1]; + let final_row = |row: usize| { + let last_step = if row.is_multiple_of(2) == last_step_even { + steps + } else { + steps - 1 + }; + row + last_step <= last_arrival + }; + // Every row up to `steps` above the last arrival is final, and the + // next one when its last step is the one before the transform's last. + let mut done = (last_arrival + 1).saturating_sub(steps).min(height); + while done < height && final_row(done) { + done += 1; + } + done + } + + /// Rows that no vertical step reads any more: row `r` is last read by + /// the last step of row `r + 1` (row `r - 1`'s comes earlier), so it is + /// unread once the row below it is final, and the last row once every + /// row is. + fn unread(&self, transform: WaveletTransform) -> usize { + let done = self.done(transform); + if done == self.height() { + done } else { - self.finished[vertical_step_count(transform) - 1] + done.saturating_sub(1) } } + + /// The vertical steps of this level's next arrival, which must not have + /// to wait for a row: every row is fed, or the arrival's row is. + fn lift_next_arrival(&mut self, transform: WaveletTransform) { + let (width, arrival) = (self.width(), self.arrived); + let level = Lifting { + transform, + y0: self.window.y0 as usize, + height: self.height(), + }; + lift_ring_arrivals(level, width, self.ring, arrival..arrival + 1); + self.arrived += 1; + } } -/// What every level reads besides its own LL input. +/// What every level reads besides its own LL input, and the scratch its +/// rows' horizontal synthesis shares. struct Inputs<'r> { rows: &'r mut dyn SubBandRows, ll_sub_band: usize, transform: WaveletTransform, + scratch: &'r mut [f32], } /// Reconstruct a component's active levels from its stored sub-bands, read @@ -174,6 +250,8 @@ pub(super) fn reconstruct_rows( /// advance together. pub(crate) struct LineReconstruction<'r> { levels: Vec>, + /// Scratch for the horizontal synthesis of one row at a time. + scratch: &'r mut [f32], ll_sub_band: usize, transform: WaveletTransform, /// The rows of the last level still to return. @@ -200,7 +278,7 @@ impl<'r> LineReconstruction<'r> { .map_err(|_| DecodingError::HostAllocationFailed)?; let mut rest = ring; for (window, sub_bands) in windows.iter().zip(sub_bands) { - let len = ring_samples([*window], transform)?; + let len = level_ring_samples(*window, transform)?; if window.is_empty() || rest.len() < len { return Err(DecodingError::CodeBlockDecodeFailure.into()); } @@ -211,9 +289,15 @@ impl<'r> LineReconstruction<'r> { sub_bands: *sub_bands, ring: level_ring, fed: 0, - finished: [0; 4], + arrived: 0, }); } + let widest = levels.iter().map(LevelRows::width).max().unwrap_or(0); + let scratch_len = synthesis_scratch_samples(widest)?; + if rest.len() < scratch_len { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + let (scratch, _) = core::mem::take(&mut rest).split_at_mut(scratch_len); let window = levels[levels.len() - 1].window; let (rows, columns) = if copy.is_empty() { @@ -232,6 +316,7 @@ impl<'r> LineReconstruction<'r> { }; Ok(Self { levels, + scratch, ll_sub_band, transform, rows, @@ -246,15 +331,59 @@ impl<'r> LineReconstruction<'r> { let Some(y) = self.rows.next() else { return Ok(None); }; + let row = self.advance(rows, y, Ready::Final)?; + let last = self.levels.len() - 1; + Ok(Some((y, &self.levels[last].row(row)[self.columns.clone()]))) + } + + /// [`Self::next_row`], handed over once no vertical step reads the row + /// any more, so that the caller may transform it in place. That takes + /// the arrival of one more row than [`Self::next_row`] waits for, which + /// the ring holds; the last row needs none, since nothing reconstructs + /// after it, so the bands a decode reads stay those final rows read. + pub(crate) fn next_row_mut( + &mut self, + rows: &mut dyn SubBandRows, + ) -> Result> { + let Some(y) = self.rows.next() else { + return Ok(None); + }; + let ready = if self.rows.is_empty() { + Ready::Final + } else { + Ready::Unread + }; + let row = self.advance(rows, y, ready)?; + let last = self.levels.len() - 1; + let columns = self.columns.clone(); + Ok(Some((y, &mut self.levels[last].row_mut(row)[columns]))) + } + + /// Advance the last level until its next row is final, as + /// [`Self::next_row`] does, without handing the row over. A caller that + /// takes a row of several reconstructions with [`Self::next_row_mut`] + /// takes them all here first: their bands then decode in the order that + /// handing the rows over final reads them. + pub(crate) fn finish_next_row(&mut self, rows: &mut dyn SubBandRows) -> Result<()> { + self.rows + .clone() + .next() + .map_or(Ok(()), |y| self.advance(rows, y, Ready::Final).map(drop)) + } + + /// Advance the last level until its absolute row `y` is `ready`, + /// returning its row in the level's window. + fn advance(&mut self, rows: &mut dyn SubBandRows, y: u32, ready: Ready) -> Result { let last = self.levels.len() - 1; let row = (y - self.levels[last].window.y0) as usize; let mut inputs = Inputs { rows, ll_sub_band: self.ll_sub_band, transform: self.transform, + scratch: &mut *self.scratch, }; - finish_row(&mut self.levels, last, row, &mut inputs)?; - Ok(Some((y, &self.levels[last].row(row)[self.columns.clone()]))) + advance_until(&mut self.levels, last, row, &mut inputs, ready)?; + Ok(row) } /// The samples reconstructed so far, over every level. @@ -265,6 +394,15 @@ impl<'r> LineReconstruction<'r> { } } +/// How far a level's row must be along before it is handed over. +#[derive(Clone, Copy)] +enum Ready { + /// Its last vertical step has run (see [`LevelRows::done`]). + Final, + /// No vertical step reads it any more (see [`LevelRows::unread`]). + Unread, +} + /// Advance `levels[level]` until `row` of its window is final. Rows must be /// requested in increasing order. fn finish_row( @@ -272,20 +410,44 @@ fn finish_row( level: usize, row: usize, inputs: &mut Inputs<'_>, +) -> Result<()> { + advance_until(levels, level, row, inputs, Ready::Final) +} + +/// Advance `levels[level]` until `row` of its window is `ready`. Rows must +/// be requested in increasing order. +fn advance_until( + levels: &mut [LevelRows<'_>], + level: usize, + row: usize, + inputs: &mut Inputs<'_>, + ready: Ready, ) -> Result<()> { let ring_rows = ring_rows(inputs.transform); - while levels[level].done(inputs.transform) <= row { + let steps = vertical_step_count(inputs.transform); + let transform = inputs.transform; + let ready_rows = |level: &LevelRows<'_>| match ready { + Ready::Final => level.done(transform), + Ready::Unread => level.unread(transform), + }; + while ready_rows(&levels[level]) <= row { let current = &levels[level]; - // Refilling a slot drops the row `ring_rows` above the new one, - // which no unfinished step may still read. - let oldest_read = current.done(inputs.transform).saturating_sub(1); - if current.fed >= current.height() - || (current.fed >= ring_rows && current.fed - ring_rows >= oldest_read) - { + let height = current.height(); + if current.fed < height { + // Refilling a slot drops the row `ring_rows` above the new one, + // which the previous arrival's deepest step read last: the ring + // holds a row per step and two more. + if current.arrived != current.fed { + return Err(DecodingError::CodeBlockDecodeFailure.into()); + } + feed_row(levels, level, inputs)?; + if height == 1 { + continue; + } + } else if height == 1 || current.arrived >= height + steps { return Err(DecodingError::CodeBlockDecodeFailure.into()); } - feed_row(levels, level, inputs)?; - advance_steps(&mut levels[level], inputs.transform); + levels[level].lift_next_arrival(inputs.transform); } if levels[level].fed > row + ring_rows { // The requested row has already left the ring. @@ -295,6 +457,10 @@ fn finish_row( } /// Interleave, filter horizontally and scale the next row of `levels[level]`. +#[expect( + clippy::too_many_lines, + reason = "one row's band reads, interleave, horizontal filter and scaling stay together for its whole-row and windowed paths" +)] fn feed_row(levels: &mut [LevelRows<'_>], level: usize, inputs: &mut Inputs<'_>) -> Result<()> { let (lower, upper) = levels.split_at_mut(level); let current = &mut upper[0]; @@ -340,6 +506,54 @@ fn feed_row(levels: &mut [LevelRows<'_>], level: usize, inputs: &mut Inputs<'_>) let width = current.width(); let slot = current.slot(local); let target = &mut current.ring[slot * width..][..width]; + // The low band holds the even columns, the high band the odd ones. + let (low, high) = if y.is_multiple_of(2) { + (ll, band(sub_bands[0])) + } else { + (band(sub_bands[1]), band(sub_bands[2])) + }; + let low_len = (window.x1.div_ceil(2) - window.x0.div_ceil(2)) as usize; + // Rows of one sample keep their own scaling, and narrow rows and band + // rows their rectangles don't hold whole take the windowed path below. + let band_rows = (height > 1 && width >= MIN_SIMD_ROW_WIDTH) + .then(|| { + Some(( + stored_row(low, window.x0.div_ceil(2), low_len, band_y)?, + stored_row(high, window.x0 / 2, width - low_len, band_y)?, + )) + }) + .flatten(); + if let Some((low_row, high_row)) = band_rows { + let (first, second) = if window.x0.is_multiple_of(2) { + (low_row, high_row) + } else { + (high_row, low_row) + }; + let row_scale = if inputs.transform == WaveletTransform::Irreversible97 { + irreversible_row_scale( + window.y0, + local.is_multiple_of(2), + super::OPENJPEG_NORMALIZED_HIGH_PASS_F32, + ) + } else { + 1.0 + }; + synthesize_band_row( + RowGeometry { + width, + x0: window.x0 as usize, + transform: inputs.transform, + high_pass: super::OPENJPEG_NORMALIZED_HIGH_PASS_F32, + }, + first, + second, + target, + inputs.scratch, + row_scale, + ); + current.fed += 1; + return Ok(()); + } interleave_samples_roi( ll, band(sub_bands[0]), @@ -373,53 +587,213 @@ fn feed_row(levels: &mut [LevelRows<'_>], level: usize, inputs: &mut Inputs<'_>) Ok(()) } -/// Let every vertical step finish as many rows as the rows fed so far allow. -fn advance_steps(level: &mut LevelRows<'_>, transform: WaveletTransform) { - let height = level.height(); - if height == 1 { - return; - } - let steps = vertical_steps(transform, level.window.y0); - for (index, &(step, even_offsets)) in steps - .iter() - .take(vertical_step_count(transform)) - .enumerate() +/// Zero samples for band rows that lie wholly outside their stored +/// rectangles, as sub-bands without code-blocks in a row do: as wide as the +/// band rows of a 32768-wide level. A wider zero row takes the scalar path, +/// which made the empty 9000-wide band rows of an 18000-wide image's chroma +/// 1.6 times slower to reconstruct than its luma. +static ZERO_ROW: [f32; 16384] = [0.0; 16384]; + +/// The `len` samples of row `band_y` of `source` from band column `band_x0`, +/// which read zero outside its rectangle as [`CoefficientSource::get`] does: +/// read in place when the rectangle holds them all, or zeros when it holds +/// none. `None` when it holds only some. +fn stored_row( + source: CoefficientSource<'_>, + band_x0: u32, + len: usize, + band_y: u32, +) -> Option<&[f32]> { + let rect = source.rect; + let band_x1 = u64::from(band_x0) + len as u64; + if band_y < rect.y0 || band_y >= rect.y1 || band_x1 <= u64::from(rect.x0) || band_x0 >= rect.x1 { - let available = if index == 0 { - level.fed - } else { - level.finished[index - 1] - }; - while level.finished[index] < height { - let row = level.finished[index]; - if row.is_multiple_of(2) == even_offsets { - let above = periodic_symmetric_extension_left(row, 1); - let below = if row + 1 < height { - row + 1 + return ZERO_ROW.get(..len); + } + if band_x0 < rect.x0 || band_x1 > u64::from(rect.x1) { + return None; + } + let start = (band_y - rect.y0) as usize * source.stride as usize + (band_x0 - rect.x0) as usize; + source.coefficients.get(start..start.checked_add(len)?) +} + +#[cfg(test)] +mod tests { + use super::super::filter_common::periodic_symmetric_extension_left; + use super::*; + + /// The rows final once `fed` rows of a level have arrived when every + /// step finishes, in order, each row whose inputs the previous step has + /// finished: the earliest a row can be final, which the arrival schedule + /// must match so that no row reaches the output later than that. + fn earliest_final_rows( + transform: WaveletTransform, + y0: u32, + height: usize, + fed: usize, + ) -> usize { + if height == 1 { + return fed; + } + let steps = vertical_step_count(transform); + let mut finished = [0; 4]; + for (index, &(_, even_offsets)) in vertical_steps(transform, y0)[..steps].iter().enumerate() + { + let available = if index == 0 { fed } else { finished[index - 1] }; + while finished[index] < height { + let row = finished[index]; + let ready = if row.is_multiple_of(2) == even_offsets { + let above = periodic_symmetric_extension_left(row, 1); + let below = if row + 1 < height { + row + 1 + } else { + height - 2 + }; + row.max(above).max(below) < available } else { - height - 2 + row < available }; - if row.max(above).max(below) >= available { + if !ready { break; } - lift_level_row(level, step, row, above, below); - } else if row >= available { - break; + finished[index] += 1; } - level.finished[index] += 1; } + finished[steps - 1] } -} -fn lift_level_row( - level: &mut LevelRows<'_>, - step: VerticalStep, - row: usize, - above: usize, - below: usize, -) { - let width = level.width(); - let (target, above, below) = (level.slot(row), level.slot(above), level.slot(below)); - let (row_samples, above, below) = split_rows(level.ring, width, target, above, below); - lift_one_row(step, row_samples, above, below); + /// The arrivals after which row `row` of a level is final, and after + /// which no vertical step reads it: the arrival of the last step that + /// updates it, and of the last step that updates a row whose mirrored + /// neighbours include it. + fn last_update_and_read( + transform: WaveletTransform, + y0: u32, + height: usize, + row: usize, + ) -> (usize, usize) { + let steps = vertical_step_count(transform); + let (mut update, mut read) = (0, 0); + for (index, &(_, even_offsets)) in vertical_steps(transform, y0)[..steps].iter().enumerate() + { + let arrival = |target: usize| target + index + 1; + for target in 0..height { + if target.is_multiple_of(2) != even_offsets { + continue; + } + let above = periodic_symmetric_extension_left(target, 1); + let below = if target + 1 < height { + target + 1 + } else { + height - 2 + }; + if target == row { + update = update.max(arrival(target)); + } + if above == row || below == row { + read = read.max(arrival(target)); + } + } + } + (update, read.max(update)) + } + + /// After every arrival, a level's final rows are the rows whose last + /// update has run, and its unread rows those no step will read again, + /// each counted from the first row while every row so far qualifies. + #[test] + fn final_and_unread_rows_follow_the_arrivals() { + for transform in [ + WaveletTransform::Reversible53, + WaveletTransform::Irreversible97, + ] { + let steps = vertical_step_count(transform); + for y0 in [0, 1, 4, 9] { + for height in 2..=20 { + let rows: Vec<(usize, usize)> = (0..height) + .map(|row| last_update_and_read(transform, y0, height, row)) + .collect(); + let mut ring = []; + let mut level = LevelRows { + window: IntRect::from_xywh( + 0, + y0, + 1, + u32::try_from(height).expect("height"), + ), + sub_bands: [0; 3], + ring: &mut ring, + fed: 0, + arrived: 0, + }; + let mut handed_over = 0; + for arrived in 0..=height + steps { + level.fed = arrived.min(height); + level.arrived = arrived; + let prefix = |past: fn(&(usize, usize)) -> usize| { + rows.iter().take_while(|row| past(row) < arrived).count() + }; + let label = + format!("{transform:?} y0={y0} height={height} arrived={arrived}"); + assert_eq!(level.done(transform), prefix(|row| row.0), "final {label}"); + assert_eq!( + level.unread(transform), + prefix(|row| row.1), + "unread {label}" + ); + // Every row this arrival makes unread is still in + // the ring, which holds the last `ring_rows` fed. + assert!( + handed_over + ring_rows(transform) >= level.fed, + "ring {label}" + ); + handed_over = level.unread(transform); + } + } + } + } + } + + /// A level hands each row over after exactly the rows the earliest + /// schedule needs: after `fed` rows, or once every row has arrived and + /// the arrivals past the last one ran, its final rows are those. + #[test] + fn rows_are_final_after_the_rows_their_steps_need() { + for transform in [ + WaveletTransform::Reversible53, + WaveletTransform::Irreversible97, + ] { + let steps = vertical_step_count(transform); + for y0 in [0, 1, 2, 7] { + for height in 1..=24 { + let mut ring = []; + let mut level = LevelRows { + window: IntRect::from_xywh( + 0, + y0, + 1, + u32::try_from(height).expect("height"), + ), + sub_bands: [0; 3], + ring: &mut ring, + fed: 0, + arrived: 0, + }; + for fed in 0..=height { + level.fed = fed; + level.arrived = if fed == height && height > 1 { + height + steps + } else { + fed + }; + assert_eq!( + level.done(transform), + earliest_final_rows(transform, y0, height, fed), + "{transform:?} y0={y0} height={height} fed={fed}" + ); + } + } + } + } + } } diff --git a/crates/j2k-native/src/j2c/idwt/roi.rs b/crates/j2k-native/src/j2c/idwt/roi.rs index ba19b4511..24b62d503 100644 --- a/crates/j2k-native/src/j2c/idwt/roi.rs +++ b/crates/j2k-native/src/j2c/idwt/roi.rs @@ -15,7 +15,50 @@ use crate::error::{DecodingError, ValidationError}; use crate::j2c::roi::output_region_rect; use crate::j2c::Header; use crate::math::{SimdBuffer, SIMD_WIDTH}; -use crate::{checked_decode_usize_product2, try_resize_decode_elements, Result}; +use crate::{ + checked_decode_usize_product2, try_resize_decode_elements, try_zero_fill_decode_elements, + Result, +}; + +/// Samples of each component's rows a tile streaming from pool bands +/// reconstructs at most at once, before they are transformed and handed over. +const STREAMED_BATCH_SAMPLES: usize = 2 * 1024 * 1024; + +/// Batches a tile streaming from pool bands splits its rows into at least, +/// so the two batches it holds at once stay at most half of the rows' +/// samples. +const STREAMED_MIN_BATCHES: usize = 4; + +/// Rows a batch of a tile holding one batch keeps at least when it is +/// halved: halving the 116-row batches of an 18000-wide tile cost 2 to 5%. +const STREAMED_DEEP_MIN_ROWS: usize = 128; + +/// Rows a batch of a tile holding two batches keeps at most. HTJ2K tiles +/// pipeline two: on 10 to 18 threads, 256-row batches of a 4096-wide tile +/// decode as fast as 512-row ones (0.99-1.01 of their time) and hold two +/// thirds of the memory (48 vs 72 MiB); 128-row ones hold half (36 MiB) but +/// take 3-10% longer, and 1024 x 1024 regions 6-15% longer. +const STREAMED_PIPELINED_MAX_ROWS: usize = 256; + +/// Rows of each component a tile streaming from pool bands through a reader +/// per component reconstructs at once, for `height` rows of `width` samples. +/// A tile that holds `one_batch` (a classic tile; HTJ2K tiles pipeline two) +/// transforms it before the next one reconstructs, so its batches take half +/// the rows while that leaves at least `STREAMED_DEEP_MIN_ROWS`: as fast, +/// with half the rows held. Two batches keep to +/// `STREAMED_PIPELINED_MAX_ROWS` each. A tile with one reader holds no +/// batch: it finishes each row in the IDWT's ring. +pub(crate) fn streamed_batch_rows(width: usize, height: usize, one_batch: bool) -> usize { + let rows = STREAMED_BATCH_SAMPLES / width.max(1); + let rows = if !one_batch { + rows.min(STREAMED_PIPELINED_MAX_ROWS) + } else if rows / 2 >= STREAMED_DEEP_MIN_ROWS { + rows / 2 + } else { + rows + }; + rows.min(height / STREAMED_MIN_BATCHES).clamp(1, 512) +} /// The most f32 samples [`apply_roi`] holds at once for any one component. /// @@ -29,8 +72,9 @@ use crate::{checked_decode_usize_product2, try_resize_decode_elements, Result}; /// at the 9/7 ring depth whatever the component's transform. With no active /// level it holds nothing, since it copies the LL window straight into the /// plane. When the tile streams its rows to a sink, every component's rings -/// are live at once, with a scratch row each as wide as its final window, so -/// the peak is their sum. +/// are live at once, the rows finishing in place in them, with one or two +/// batches of rows as wide as each final window from pool bands (see +/// [`streamed_batch_rows`]), so the peak is their sum. pub(crate) fn roi_idwt_peak_elements( storage: &DecompositionStorage<'_>, skipped_resolution_levels: u8, @@ -62,15 +106,34 @@ pub(crate) fn roi_idwt_peak_elements( .decompositions .clone() .take(active_len) + .skip_while(|decomposition_idx| { + roi_plan.idwt_window(*decomposition_idx).is_none() + && storage.decompositions[*decomposition_idx].rect.is_empty() + }) .map_while(|decomposition_idx| roi_plan.idwt_window(decomposition_idx)); let rings = ring_samples(windows, WaveletTransform::Irreversible97)?; if storage.lockstep_lines { - let scratch_row = roi_plan - .final_window(component_idx) - .map_or(0, |window| window.width() as usize); + let batch_rows = roi_plan.final_window(component_idx).map_or(0, |window| { + let (width, height) = (window.width() as usize, window.height() as usize); + if storage.parallel_bands && !storage.one_synthesis_reader { + // Two batches, when each component reconstructs + // on its own reader, unless the band rings hold + // more bands instead; one reader holds none. + let batches = if storage.deep_band_rings { 1 } else { 2 }; + width + .saturating_mul(streamed_batch_rows( + width, + height, + storage.deep_band_rings, + )) + .saturating_mul(batches) + } else { + 0 + } + }); lockstep = lockstep .checked_add(rings) - .and_then(|total| total.checked_add(scratch_row)) + .and_then(|total| total.checked_add(batch_rows)) .ok_or(ValidationError::ImageTooLarge)?; } else { peak = peak.max(rings); @@ -199,6 +262,10 @@ pub(super) fn apply_roi( for (local_idx, decomposition) in decompositions.iter().take(active_len).enumerate() { let decomposition_idx = tile_decompositions.decompositions.start + local_idx; let Some(output_window) = roi_plan.idwt_window(decomposition_idx) else { + if !have_current && decomposition.rect.is_empty() { + // A coarse level with no samples: the next level's LL input is empty. + continue; + } output.coefficients.clear(); output.rect = IntRect::from_xywh(0, 0, 0, 0); return Ok(()); @@ -346,6 +413,10 @@ impl LinePlan { // Without a window the level has nothing to reconstruct, as in the // whole-window path. let Some(window) = roi_plan.idwt_window(decomposition_idx) else { + if windows.is_empty() && storage.decompositions[decomposition_idx].rect.is_empty() { + // A coarse level with no samples: the next level's LL input is empty. + continue; + } return Ok(None); }; windows.push(window); @@ -413,7 +484,7 @@ impl ComponentLines { let mut ring = core::mem::take(&mut tile_ctx.idwt_line_rings); let ring_len = plan.ring_len()?; if ring.len() < ring_len { - try_resize_decode_elements(&mut ring, ring_len, 0.0)?; + try_zero_fill_decode_elements(&mut ring, ring_len)?; } tile_ctx.idwt_line_rings = ring; record_workspace_now(tile_ctx, storage)?; diff --git a/crates/j2k-native/src/j2c/idwt/tests.rs b/crates/j2k-native/src/j2c/idwt/tests.rs index 696ab5fff..8ca683e06 100644 --- a/crates/j2k-native/src/j2c/idwt/tests.rs +++ b/crates/j2k-native/src/j2c/idwt/tests.rs @@ -627,3 +627,499 @@ fn banded_2d_matches_plain_2d_bit_for_bit() { } } } + +/// Sub-bands for the streamed IDWT, each stored in its own rectangle. +struct StoredBands { + bands: Vec<(Vec, IntRect)>, +} + +impl super::lines::SubBandRows for StoredBands { + fn prepare_row(&mut self, _sub_band: usize, _y: u32) -> Result<()> { + Ok(()) + } + + fn samples(&self, sub_band: usize) -> (&[f32], IntRect, u32) { + let (samples, rect) = &self.bands[sub_band]; + (samples, *rect, rect.width()) + } +} + +/// `rect` grown by `[left, top, right, bottom]` samples; negative values +/// shrink it. +#[expect( + clippy::cast_sign_loss, + clippy::cast_possible_truncation, + reason = "the fixture rectangles stay at small non-negative coordinates" +)] +fn adjusted(rect: IntRect, [left, top, right, bottom]: [i64; 4]) -> IntRect { + let at = |value: u32, delta: i64| (i64::from(value) + delta).max(0) as u32; + IntRect::from_ltrb( + at(rect.x0, -left), + at(rect.y0, -top), + at(rect.x1, right).max(at(rect.x0, -left)), + at(rect.y1, bottom).max(at(rect.y0, -top)), + ) +} + +/// The rows `copy` of the streamed IDWT of `windows`, each taken through +/// `LineReconstruction::next_row_mut` and overwritten with NaN as soon as it +/// is read, so a row a later step still read would poison the rows after it. +fn streamed_rows_overwritten( + rows: &mut StoredBands, + sub_bands: &[[usize; 3]], + windows: &[IntRect], + transform: WaveletTransform, + copy: IntRect, +) -> Vec<(u32, Vec)> { + let mut ring = vec![ + f32::NAN; + super::lines::ring_samples(windows.iter().copied(), transform) + .expect("ring") + ]; + let mut lines = + super::lines::LineReconstruction::new(sub_bands, windows, 0, transform, &mut ring, copy) + .expect("plan the streamed rows"); + let mut out = Vec::new(); + while let Some((y, row)) = lines.next_row_mut(rows).expect("streamed row") { + out.push((y, bits(row))); + row.fill(f32::NAN); + } + out +} + +/// Band rows a reconstruction of one component asks for, logged in the +/// order every component asks. +struct LoggedBands<'a> { + bands: &'a StoredBands, + component: usize, + log: &'a core::cell::RefCell>, +} + +impl super::lines::SubBandRows for LoggedBands<'_> { + fn prepare_row(&mut self, sub_band: usize, y: u32) -> Result<()> { + self.log.borrow_mut().push((self.component, sub_band, y)); + Ok(()) + } + + fn samples(&self, sub_band: usize) -> (&[f32], IntRect, u32) { + let (samples, rect) = &self.bands.bands[sub_band]; + (samples, *rect, rect.width()) + } +} + +/// A streamed tile finishes each row of its components in place, which +/// needs one arrival more than a final row; taking every component's row +/// final first must keep the band rows they ask for, and so the bands they +/// decode, in the order that handing the rows over final asks for them: +/// over three components, both transforms, two levels, up to four skipped +/// first rows, and the window's bottom edge or rows short of it. +#[test] +fn rows_finished_in_place_ask_for_bands_as_final_rows_do() { + use super::banded::packed_band_rects; + use super::lines::{ring_samples, LineReconstruction}; + + for transform in [ + WaveletTransform::Reversible53, + WaveletTransform::Irreversible97, + ] { + for (x0, y0, width, height) in [(0, 0, 9, 7), (1, 1, 20, 13), (3, 2, 33, 21)] { + let first = IntRect::from_xywh(x0, y0, width, height); + let next = IntRect::from_xywh(2 * x0 + 1, 2 * y0, 2 * width - 1, 2 * height); + let mut seed = 11_u64; + let bands = StoredBands { + bands: packed_band_rects(first) + .into_iter() + .chain(packed_band_rects(next).into_iter().skip(1)) + .map(|rect| { + seed += 1; + let len = (rect.width() * rect.height()) as usize; + (pseudo_random_coefficients(len, seed, false), rect) + }) + .collect(), + }; + let windows = [first, next]; + for (skipped, cut) in [(0, 0), (1, 0), (2, 1), (3, 2), (4, 3), (0, 5), (1, 6)] { + let copy = IntRect::from_ltrb(next.x0, next.y0 + skipped, next.x1, next.y1 - cut); + let order = |in_place: bool| { + let log = core::cell::RefCell::new(Vec::new()); + let ring = ring_samples(windows, transform).expect("ring"); + let mut rings = vec![vec![f32::NAN; ring]; 3]; + let mut lines: Vec<_> = rings + .iter_mut() + .map(|ring| { + LineReconstruction::new( + &[[1, 2, 3], [4, 5, 6]], + &windows, + 0, + transform, + ring, + copy, + ) + .expect("plan the streamed rows") + }) + .collect(); + let mut sources: Vec<_> = (0..3) + .map(|component| LoggedBands { + bands: &bands, + component, + log: &log, + }) + .collect(); + for y in copy.y0..copy.y1 { + if in_place { + for (line, source) in lines.iter_mut().zip(sources.iter_mut()) { + line.finish_next_row(source).expect("final row"); + } + for (line, source) in lines.iter_mut().zip(sources.iter_mut()) { + let (row_y, _) = + line.next_row_mut(source).expect("row").expect("a row"); + assert_eq!(row_y, y); + } + } else { + for (line, source) in lines.iter_mut().zip(sources.iter_mut()) { + let (row_y, _) = + line.next_row(source).expect("row").expect("a row"); + assert_eq!(row_y, y); + } + } + } + log.into_inner() + }; + let label = format!("{transform:?} {first:?} skipped {skipped} cut {cut}"); + let final_order = order(false); + assert!(!final_order.is_empty(), "{label}"); + assert_eq!(order(true), final_order, "{label}"); + } + } + } +} + +/// The whole-window region IDWT of one level, as `roi::fill_level_roi` runs it. +fn whole_window_level( + sources: [CoefficientSource<'_>; 4], + window: IntRect, + transform: WaveletTransform, +) -> Vec { + let mut output = vec![0.0; (window.width() * window.height()) as usize]; + interleave_samples_roi( + sources[0], + sources[1], + sources[2], + sources[3], + &mut output, + window, + ); + super::horizontal::filter_horizontal_codestream(&mut output, window, transform); + super::vertical::filter_vertical_codestream(&mut output, window, transform); + output +} + +/// The streamed IDWT synthesizes a row from its two band rows in place with +/// the banded SIMD kernel when their stored rectangles hold them whole, and +/// otherwise through the windowed interleave. Every row must match the +/// whole-window region IDWT, which reads every sample through +/// `CoefficientSource::get` and filters with the scalar `filter_row`, bit for +/// bit: both transforms, both column and row parities, every height up to a +/// few wraps of the level's ring, stored rectangles wider than the window's +/// bands and clipped on either side, sub-bands with nothing stored, and a +/// second level reading the first one's rows. +#[test] +#[expect(clippy::too_many_lines, reason = "one table-driven equivalence sweep")] +fn streamed_rows_match_the_whole_window_levels_bit_for_bit() { + use super::banded::packed_band_rects; + use super::lines::{reconstruct_rows, ring_samples}; + + let mut seed = 7_u64; + for transform in [ + WaveletTransform::Reversible53, + WaveletTransform::Irreversible97, + ] { + let integral = transform == WaveletTransform::Reversible53; + for (x0, y0) in [(0, 0), (1, 0), (0, 1), (1, 1), (5, 2), (8, 3)] { + for (width, height) in [ + (16, 9), + (17, 20), + (31, 2), + (40, 33), + (64, 12), + (161, 40), + // Every height up to a few ring wraps, around the SIMD width. + (15, 1), + (16, 3), + (17, 4), + (24, 5), + (32, 6), + (33, 7), + (8, 8), + (47, 10), + (49, 11), + ] { + for clip in [ + [0, 0, 0, 0], + [2, 1, 3, 2], + [-2, 0, 0, 0], + [0, 0, -3, 0], + [-1, -1, -2, -1], + [3, 0, -1, 1], + [1, 0, 1, 0], + ] { + let label = format!( + "{transform:?} origin=({x0},{y0}) size={width}x{height} clip={clip:?}" + ); + let window = IntRect::from_xywh(x0, y0, width, height); + // A sub-band without code-blocks stores nothing and + // reads as zero: here HH, or LH with a clipped HL. + let empty = match clip { + [1, 0, 1, 0] => Some(3), + [-2, 0, 0, 0] => Some(2), + _ => None, + }; + let mut bands = Vec::new(); + for (index, rect) in packed_band_rects(window).into_iter().enumerate() { + let stored = if empty == Some(index) { + IntRect::from_xywh(0, 0, 0, 0) + } else { + adjusted(rect, clip) + }; + seed += 1; + let samples = pseudo_random_coefficients( + (stored.width() * stored.height()) as usize, + seed, + integral, + ); + bands.push((samples, stored)); + } + let sources: [CoefficientSource<'_>; 4] = core::array::from_fn(|index| { + let (samples, rect) = &bands[index]; + CoefficientSource::new(samples, *rect, rect.width()) + }); + let expected = whole_window_level(sources, window, transform); + + let mut rows = StoredBands { + bands: bands.clone(), + }; + let mut ring = vec![f32::NAN; ring_samples([window], transform).expect("ring")]; + let mut streamed = vec![f32::NAN; expected.len()]; + let mut output_samples = 0; + reconstruct_rows( + &mut rows, + &[[1, 2, 3]], + &[window], + 0, + transform, + &mut ring, + window, + &mut |y, samples| { + let start = ((y - window.y0) * width) as usize; + streamed[start..][..samples.len()].copy_from_slice(samples); + Ok(()) + }, + &mut output_samples, + ) + .expect("streamed level"); + assert_eq!(bits(&streamed), bits(&expected), "one level {label}"); + let overwritten = streamed_rows_overwritten( + &mut StoredBands { + bands: bands.clone(), + }, + &[[1, 2, 3]], + &[window], + transform, + window, + ); + assert_eq!(overwritten.len(), height as usize, "rows {label}"); + for (y, row) in overwritten { + let start = ((y - window.y0) * width) as usize; + assert_eq!( + row, + bits(&expected[start..][..width as usize]), + "one level, rows overwritten as read, {label} row {y}" + ); + } + + // A second level whose LL is the first level's window, + // clipped or grown the same way. + let next = IntRect::from_xywh(2 * x0 + 1, 2 * y0, 2 * width - 3, 2 * height); + let next_bands = packed_band_rects(next); + let first = adjusted(next_bands[0], clip); + if first.width() < 2 || first.height() < 2 { + continue; + } + let stored = packed_band_rects(first) + .into_iter() + .chain(next_bands[1..].iter().map(|rect| adjusted(*rect, clip))); + let levels: Vec<(Vec, IntRect)> = stored + .map(|rect| { + seed += 1; + let len = (rect.width() * rect.height()) as usize; + (pseudo_random_coefficients(len, seed, integral), rect) + }) + .collect(); + let source = |index: usize| { + let (samples, rect) = &levels[index]; + CoefficientSource::new(samples, *rect, rect.width()) + }; + let coarse = whole_window_level( + [source(0), source(1), source(2), source(3)], + first, + transform, + ); + let expected = whole_window_level( + [ + CoefficientSource::new(&coarse, first, first.width()), + source(4), + source(5), + source(6), + ], + next, + transform, + ); + let mut rows = StoredBands { + bands: levels.clone(), + }; + let windows = [first, next]; + let mut ring = + vec![f32::NAN; ring_samples(windows, transform).expect("two-level ring")]; + let copy = adjusted(next, [-1, -2, -2, -1]); + let mut streamed = vec![f32::NAN; expected.len()]; + reconstruct_rows( + &mut rows, + &[[1, 2, 3], [4, 5, 6]], + &windows, + 0, + transform, + &mut ring, + copy, + &mut |y, samples| { + let start = ((y - next.y0) * next.width() + copy.x0 - next.x0) as usize; + streamed[start..][..samples.len()].copy_from_slice(samples); + Ok(()) + }, + &mut output_samples, + ) + .expect("streamed levels"); + for y in copy.y0..copy.y1 { + let start = ((y - next.y0) * next.width() + copy.x0 - next.x0) as usize; + let range = start..start + copy.width() as usize; + assert_eq!( + bits(&streamed[range.clone()]), + bits(&expected[range]), + "two levels {label} row {y}" + ); + } + let overwritten = streamed_rows_overwritten( + &mut StoredBands { bands: levels }, + &[[1, 2, 3], [4, 5, 6]], + &windows, + transform, + copy, + ); + assert_eq!(overwritten.len(), copy.height() as usize, "rows {label}"); + for (y, row) in overwritten { + let start = ((y - next.y0) * next.width() + copy.x0 - next.x0) as usize; + assert_eq!( + row, + bits(&expected[start..][..copy.width() as usize]), + "two levels, rows overwritten as read, {label} row {y}" + ); + } + } + } + } + } +} + +/// Band rows wider than 4096 samples that hold no code-block, as in the +/// chroma of an 18000-wide image, read as zero rows through the banded SIMD +/// kernel like narrower ones; every row still matches the whole-window region +/// IDWT bit for bit, with either high band empty. +#[test] +fn wide_empty_band_rows_match_the_whole_window_level_bit_for_bit() { + use super::lines::{reconstruct_rows, ring_samples}; + + for transform in [ + WaveletTransform::Reversible53, + WaveletTransform::Irreversible97, + ] { + let integral = transform == WaveletTransform::Reversible53; + for (x0, empty) in [(0, 3), (1, 2), (1, 3)] { + let window = IntRect::from_xywh(x0, 2, 16_400, 6); + let mut seed = 31_u64; + let bands: Vec<(Vec, IntRect)> = super::banded::packed_band_rects(window) + .into_iter() + .enumerate() + .map(|(index, rect)| { + let stored = if index == empty { + IntRect::from_xywh(0, 0, 0, 0) + } else { + rect + }; + seed += 1; + let len = (stored.width() * stored.height()) as usize; + (pseudo_random_coefficients(len, seed, integral), stored) + }) + .collect(); + let sources: [CoefficientSource<'_>; 4] = core::array::from_fn(|index| { + let (samples, rect) = &bands[index]; + CoefficientSource::new(samples, *rect, rect.width()) + }); + let expected = whole_window_level(sources, window, transform); + let mut rows = StoredBands { bands }; + let mut ring = vec![f32::NAN; ring_samples([window], transform).expect("ring")]; + let mut streamed = vec![f32::NAN; expected.len()]; + let mut output_samples = 0; + reconstruct_rows( + &mut rows, + &[[1, 2, 3]], + &[window], + 0, + transform, + &mut ring, + window, + &mut |y, samples| { + let start = ((y - window.y0) * window.width()) as usize; + streamed[start..][..samples.len()].copy_from_slice(samples); + Ok(()) + }, + &mut output_samples, + ) + .expect("streamed level"); + assert_eq!( + bits(&streamed), + bits(&expected), + "{transform:?} origin x {x0}, empty band {empty}" + ); + } + } +} + +/// A tile streaming from pool bands with deep band rings holds one batch of +/// rows instead of two, so its batch takes half the rows unless that leaves +/// fewer than 128; two pipelined batches keep to 256 rows each; tiles too +/// short for full batches still split their rows into at least four +/// batches, and a row wider than a batch is a batch of its own. +#[test] +fn deep_band_rings_halve_the_streamed_row_batch() { + use super::roi::streamed_batch_rows; + + // 5661 x 8287 (one tile): 256 rows of each component in each of two + // pipelined batches (370 fit their samples), 185 rows in the one batch + // of deep rings. + assert_eq!(streamed_batch_rows(5661, 8287, false), 256); + assert_eq!(streamed_batch_rows(5661, 8287, true), 185); + // 4096 wide: 256 rows either way (512 fit, or half of them). + assert_eq!(streamed_batch_rows(4096, 4096, false), 256); + assert_eq!(streamed_batch_rows(4096, 4096, true), 256); + // 1024 wide: 256 rows in each of two batches, 512 in one. + assert_eq!(streamed_batch_rows(1024, 1 << 16, false), 256); + assert_eq!(streamed_batch_rows(1024, 1 << 16, true), 512); + for deep_band_rings in [false, true] { + // 18000 wide: 116 rows, too few to halve. + assert_eq!(streamed_batch_rows(18000, 12000, deep_band_rings), 116); + // 8200 wide: 255 rows, too few to halve. + assert_eq!(streamed_batch_rows(8200, 12000, deep_band_rings), 255); + assert_eq!(streamed_batch_rows(256, 200, deep_band_rings), 50); + assert_eq!(streamed_batch_rows(256, 3, deep_band_rings), 1); + assert_eq!(streamed_batch_rows(3 << 20, 4096, deep_band_rings), 1); + } +} diff --git a/crates/j2k-native/src/j2c/idwt/vertical.rs b/crates/j2k-native/src/j2c/idwt/vertical.rs index e881c176f..3dd32d6ad 100644 --- a/crates/j2k-native/src/j2c/idwt/vertical.rs +++ b/crates/j2k-native/src/j2c/idwt/vertical.rs @@ -72,11 +72,6 @@ pub(super) fn irreversible_row_scale(y0: u32, even_offset: bool, high_pass: f32) } } -/// [`lift_row`] at the CPU's SIMD level, for callers outside a dispatch. -pub(super) fn lift_one_row(step: VerticalStep, row: &mut [f32], above: &[f32], below: &[f32]) { - dispatch!(Level::new(), simd => lift_row(simd, step, row, above, below)); -} - /// [`scale_row`] at the CPU's SIMD level, for callers outside a dispatch. pub(super) fn scale_one_row(row: &mut [f32], factor: f32) { dispatch!(Level::new(), simd => scale_row(simd, row, factor)); diff --git a/crates/j2k-native/src/j2c/mct.rs b/crates/j2k-native/src/j2c/mct.rs index 960e93f72..9adcac074 100644 --- a/crates/j2k-native/src/j2c/mct.rs +++ b/crates/j2k-native/src/j2c/mct.rs @@ -121,17 +121,7 @@ pub(crate) fn apply_inverse_rows( round_irreversible_output: bool, rows: (&mut [f32], &mut [f32], &mut [f32]), ) { - let transform = component_infos[0].wavelet_transform(); - let addends = [ - unsigned_level_shift(&component_infos[0]), - unsigned_level_shift(&component_infos[1]), - unsigned_level_shift(&component_infos[2]), - ]; - let centered_output = - round_irreversible_output && transform == WaveletTransform::Irreversible97; - let transform_addends = if centered_output { [0.0; 3] } else { addends }; - let rounding = centered_output.then_some(addends); - transform_planes(transform, rows, transform_addends, rounding, false); + DeferredMct::for_rows(component_infos, round_irreversible_output).transform_chunk(rows); } /// Samples per plane in one chunk: three planes' worth stays in cache @@ -166,6 +156,90 @@ const PACK_CHUNK: usize = 1 << 10; const PACK_TASK_CHUNKS: usize = 64; impl DeferredMct { + /// The transform [`apply_inverse_rows`] applies to rows of + /// `component_infos`, for a streamed decode to pack its rows with. + pub(crate) fn for_rows( + component_infos: &[ComponentInfo], + round_irreversible_output: bool, + ) -> Self { + let transform = component_infos[0].wavelet_transform(); + let addends = [ + unsigned_level_shift(&component_infos[0]), + unsigned_level_shift(&component_infos[1]), + unsigned_level_shift(&component_infos[2]), + ]; + let centered_output = + round_irreversible_output && transform == WaveletTransform::Irreversible97; + Self { + transform, + addends: if centered_output { [0.0; 3] } else { addends }, + rounding: centered_output.then_some(addends), + last_level: None, + } + } + + /// Whether [`Self::pack_ict_rgb8_rows`] packs rows: this build fuses the + /// transform into the RGB8 pack for 9/7 samples (see [`Self::pack_rgb8`]). + pub(crate) fn fuses_rgb8_pack(self) -> bool { + self.transform == WaveletTransform::Irreversible97 + && cfg!(all( + feature = "simd", + any( + target_arch = "x86_64", + all(feature = "std", target_arch = "aarch64") + ) + )) + } + + /// Transform `rows` and pack them as RGB8 into `output` in one pass, + /// reading the rows only, as the fused pack of [`Self::pack_rgb8`] + /// does; with the same expressions as [`Self::transform_chunk`] and the + /// standard RGB8 pack, so the bytes are theirs. Only where + /// [`Self::fuses_rgb8_pack`]; otherwise nothing is written. + pub(crate) fn pack_ict_rgb8_rows(self, rows: [&[f32]; 3], output: &mut [u8]) { + if !self.fuses_rgb8_pack() { + return; + } + #[cfg(all(feature = "simd", target_arch = "x86_64"))] + { + self.pack_ict_rgb8_x86(rows, output); + } + #[cfg(not(all(feature = "simd", target_arch = "x86_64")))] + { + if self.rounding.is_some() { + self.pack_ict_rgb8::(rows, output); + } else { + self.pack_ict_rgb8::(rows, output); + } + } + } + + /// Transform one row of each of the three components and pack it as + /// RGB8 into `output`: through the fused ICT pack where this build has + /// one (see [`Self::fuses_rgb8_pack`]), which only reads the rows, + /// otherwise transformed in place and packed by `pack`, which must be + /// the standard RGB8 pack where the fused one exists. Both routes finish + /// their rows here: the planes' last level as it reconstructs them (see + /// [`Self::pack_rgb8`]) and the streamed IDWT's rows. + #[expect( + clippy::inline_always, + reason = "the row finish inlines into the planes' dispatched row streamer" + )] + #[inline(always)] + pub(crate) fn pack_rgb8_row( + self, + [r, g, b]: [&mut [f32]; 3], + output: &mut [u8], + pack: impl Fn([&[f32]; 3], &mut [u8]), + ) { + if self.fuses_rgb8_pack() { + self.pack_ict_rgb8_rows([r, g, b], output); + } else { + self.transform_chunk((&mut *r, &mut *g, &mut *b)); + pack([r, g, b], output); + } + } + /// Transform the planes in place, as [`apply_inverse`] would have. Planes /// that hold a last level's sub-bands are left to [`Self::pack_rgb8`]. pub(crate) fn apply( @@ -214,28 +288,7 @@ impl DeferredMct { scratch, output, parallel, - |[r, g, b], output| { - #[cfg(all(feature = "simd", target_arch = "x86_64"))] - if self.transform == WaveletTransform::Irreversible97 { - self.pack_ict_rgb8_x86([r, g, b], output); - return; - } - if cfg!(all( - feature = "std", - feature = "simd", - target_arch = "aarch64" - )) && self.transform == WaveletTransform::Irreversible97 - { - if self.rounding.is_some() { - self.pack_ict_rgb8::([r, g, b], output); - } else { - self.pack_ict_rgb8::([r, g, b], output); - } - } else { - self.transform_chunk((&mut *r, &mut *g, &mut *b)); - pack([r, g, b], output); - } - }, + |rows, output| self.pack_rgb8_row(rows, output, &pack), ); } #[cfg(feature = "parallel")] @@ -387,7 +440,7 @@ impl DeferredMct { ); } - fn transform_chunk(self, planes: (&mut [f32], &mut [f32], &mut [f32])) { + pub(crate) fn transform_chunk(self, planes: (&mut [f32], &mut [f32], &mut [f32])) { transform_planes(self.transform, planes, self.addends, self.rounding, false); } } diff --git a/crates/j2k-native/src/j2c/mod.rs b/crates/j2k-native/src/j2c/mod.rs index 5f8c71d87..97a335640 100644 --- a/crates/j2k-native/src/j2c/mod.rs +++ b/crates/j2k-native/src/j2c/mod.rs @@ -59,14 +59,16 @@ pub(crate) use decode::{ build_referenced_classic_rgba_plan, build_referenced_htj2k_color_plan, build_referenced_htj2k_grayscale_plan, build_referenced_htj2k_rgba_plan, decode_preparsed_with_capacity_retry as decode_preparsed, - decode_rows_with_capacity_retry as decode_rows, prepare_region_tiles, + decode_rows_with_capacity_retry as decode_rows, full_decode_planes_floor, image_streams_rows, + prepare_region_tiles, }; pub use decode::{CpuDecodeParallelism, DecoderContext, DecoderWorkspace, DecoderWorkspaceStats}; pub use recode::Reversible53CoefficientImage; +pub(crate) use roi::output_grid_offset; #[cfg(feature = "std")] pub(crate) use segment::directory::PacketDirectory; pub(crate) use segment::MAX_BITPLANE_COUNT; -pub(crate) use tile::ParsedTiles; +pub(crate) use tile::{tile_rect, ParsedTiles}; pub(crate) struct ParsedCodestream<'a> { pub(crate) header: Header<'a>, diff --git a/crates/j2k-native/src/j2c/segment.rs b/crates/j2k-native/src/j2c/segment.rs index e24fc3025..e0435b58d 100644 --- a/crates/j2k-native/src/j2c/segment.rs +++ b/crates/j2k-native/src/j2c/segment.rs @@ -612,7 +612,7 @@ fn parse_inner<'a>( let layer = &mut storage.layers[code_block.layers.clone()] [progression_data.layer_num as usize]; - if let Some(segments) = layer.segments.clone() { + if let Some(segments) = layer.segments() { let segments = &mut storage.segments[segments.clone()]; for segment in segments { diff --git a/crates/j2k-native/src/j2c/segment/classic.rs b/crates/j2k-native/src/j2c/segment/classic.rs index 6fb191c88..177b68f69 100644 --- a/crates/j2k-native/src/j2c/segment/classic.rs +++ b/crates/j2k-native/src/j2c/segment/classic.rs @@ -6,7 +6,7 @@ use super::{ code_block_is_read, decode_num_coding_passes, read_lblock_increment, read_segment_length, resolve_code_block_inclusion, PacketResult, PacketSkips, SegmentSink, MAX_BITPLANE_COUNT, }; -use crate::j2c::build::{CodeBlock, CodeBlockCoding, Segment}; +use crate::j2c::build::{CodeBlock, CodeBlockCoding, Layer, Segment}; use crate::j2c::codestream::{CodeBlockStyle, ComponentInfo}; use crate::j2c::decode::DecompositionStorage; use crate::j2c::progression::ProgressionData; @@ -92,7 +92,8 @@ pub(super) fn resolve_segments( packet_workspace_error: &mut storage.packet_workspace_error, }, )?; - layer.segments = Some(start..storage.segments.len()); + *layer = Layer::with_segments(start..storage.segments.len()) + .ok_or("classic segment records exceed the layer record range")?; } else { let mut bytes = 0; parse_contribution_lengths( diff --git a/crates/j2k-native/src/j2c/segment/ht.rs b/crates/j2k-native/src/j2c/segment/ht.rs index 03ec2f073..16c194acf 100644 --- a/crates/j2k-native/src/j2c/segment/ht.rs +++ b/crates/j2k-native/src/j2c/segment/ht.rs @@ -7,7 +7,7 @@ use super::{ read_segment_length, resolve_code_block_inclusion, PacketResult, PacketSkips, SegmentSink, MAX_BITPLANE_COUNT, }; -use crate::j2c::build::{CodeBlock, CodeBlockCoding, Segment}; +use crate::j2c::build::{CodeBlock, CodeBlockCoding, Layer, Segment}; use crate::j2c::codestream::{CodeBlockStyle, ComponentInfo}; use crate::j2c::decode::DecompositionStorage; use crate::j2c::progression::ProgressionData; @@ -69,7 +69,7 @@ pub(super) fn resolve_segments( let layer = storage.layers[code_block.layers.clone()] .get_mut(layer_num) .ok_or("HT packet references a missing quality layer")?; - if layer.segments.is_some() { + if layer.segments().is_some() { return Err("HT quality layer already owns segment metadata"); } Some(layer) @@ -112,7 +112,8 @@ pub(super) fn resolve_segments( packet_workspace_error: &mut storage.packet_workspace_error, }, )?; - layer.segments = Some(start..storage.segments.len()); + *layer = Layer::with_segments(start..storage.segments.len()) + .ok_or("HT segment records exceed the layer record range")?; } else { let mut bytes = 0; parse_segment_lengths( diff --git a/crates/j2k-native/src/j2c/tile.rs b/crates/j2k-native/src/j2c/tile.rs index 0aed2a516..d97d2d71e 100644 --- a/crates/j2k-native/src/j2c/tile.rs +++ b/crates/j2k-native/src/j2c/tile.rs @@ -3,7 +3,9 @@ use alloc::vec::Vec; use super::build::{PrecinctData, SubBandType}; -use super::codestream::{markers, ComponentInfo, Header, ProgressionChange, ProgressionOrder}; +use super::codestream::{ + markers, ComponentInfo, Header, ProgressionChange, ProgressionOrder, SizeData, +}; use super::rect::IntRect; use crate::error::{bail, MarkerError, Result, TileError, ValidationError}; use crate::reader::BitReader; @@ -159,48 +161,51 @@ impl TilePart<'_> { } } +/// Reference-grid bounds of a tile, clipped to the image area. +pub(crate) fn tile_rect(size_data: &SizeData, x_coord: u32, y_coord: u32) -> IntRect { + // See B-7, B-8, B-9 and B-10. Saturating arithmetic: the results + // are clamped against the reference grid anyway, and a saturated + // intermediate already exceeds every in-grid bound, so the clamp + // still produces the mathematically correct edge for crafted + // headers whose products overflow u32. + let x0 = u32::max( + size_data + .tile_x_offset + .saturating_add(x_coord.saturating_mul(size_data.tile_width)), + size_data.image_area_x_offset, + ); + let y0 = u32::max( + size_data + .tile_y_offset + .saturating_add(y_coord.saturating_mul(size_data.tile_height)), + size_data.image_area_y_offset, + ); + + // Note that `x1` and `y1` are exclusive. + let x1 = u32::min( + size_data.tile_x_offset.saturating_add( + x_coord + .saturating_add(1) + .saturating_mul(size_data.tile_width), + ), + size_data.reference_grid_width, + ); + let y1 = u32::min( + size_data.tile_y_offset.saturating_add( + y_coord + .saturating_add(1) + .saturating_mul(size_data.tile_height), + ), + size_data.reference_grid_height, + ); + + IntRect::from_ltrb(x0, y0, x1, y1) +} + impl Tile<'_> { fn new(idx: u32, header: &Header<'_>) -> Self { - let rect = { - let size_data = &header.size_data; - - let x_coord = size_data.tile_x_coord(idx); - let y_coord = size_data.tile_y_coord(idx); - - // See B-7, B-8, B-9 and B-10. Saturating arithmetic: the results - // are clamped against the reference grid anyway, and a saturated - // intermediate already exceeds every in-grid bound, so the clamp - // still produces the mathematically correct edge for crafted - // headers whose products overflow u32. - let x0 = u32::max( - size_data - .tile_x_offset - .saturating_add(x_coord.saturating_mul(size_data.tile_width)), - size_data.image_area_x_offset, - ); - let y0 = u32::max( - size_data - .tile_y_offset - .saturating_add(y_coord.saturating_mul(size_data.tile_height)), - size_data.image_area_y_offset, - ); - - // Note that `x1` and `y1` are exclusive. - let x1 = u32::min( - size_data - .tile_x_offset - .saturating_add((x_coord + 1).saturating_mul(size_data.tile_width)), - size_data.reference_grid_width, - ); - let y1 = u32::min( - size_data - .tile_y_offset - .saturating_add((y_coord + 1).saturating_mul(size_data.tile_height)), - size_data.reference_grid_height, - ); - - IntRect::from_ltrb(x0, y0, x1, y1) - }; + let size = &header.size_data; + let rect = tile_rect(size, size.tile_x_coord(idx), size.tile_y_coord(idx)); Tile { #[cfg(feature = "std")] diff --git a/crates/j2k-native/src/lib.rs b/crates/j2k-native/src/lib.rs index 92a2bb40e..5d00c464a 100644 --- a/crates/j2k-native/src/lib.rs +++ b/crates/j2k-native/src/lib.rs @@ -164,6 +164,8 @@ pub(crate) mod writer; use crate::math::{dispatch, Level}; #[cfg(test)] use color::cielab_to_rgb; +#[doc(hidden)] +pub use color::pack_rgb8; pub(crate) use color::{ convert_color_space, interleave_and_convert_after_mct, interleave_and_convert_region, resolve_alpha_and_color_space, resolve_palette_indices, validate_and_reorder_channels, @@ -307,6 +309,17 @@ pub(crate) const MAX_J2K_TILE_COUNT: u64 = u16::MAX as u64 + 1; #[doc(hidden)] pub use j2k_types::{DEFAULT_MAX_CODEC_BYTES, DEFAULT_MAX_DECODE_BYTES}; +/// The threads a [`CpuDecodeParallelism::Auto`] decode runs on: the global +/// worker pool's size with the `parallel` feature, else one. +#[doc(hidden)] +#[must_use] +pub fn cpu_decode_pool_threads() -> usize { + #[cfg(feature = "parallel")] + return rayon::current_num_threads(); + #[cfg(not(feature = "parallel"))] + 1 +} + #[inline] pub(crate) fn checked_decode_usize_product2(left: usize, right: usize) -> Result { left.checked_mul(right) @@ -366,6 +379,52 @@ pub(crate) fn try_resize_decode_elements( Ok(()) } +/// [`try_resize_decode_elements`] to zeros. On a pool of at most +/// [`MAX_POOL_THREADS_FOR_ZEROED_BUFFERS`] threads, an empty buffer whose +/// capacity is short of `target_len` is replaced by a zeroed allocation +/// instead of being zeroed here: memory fresh from the system is already +/// zero, so the threads that fill the buffer take its page faults, instead +/// of this thread taking every one of them first while the pool waits. +pub(crate) fn try_zero_fill_decode_elements( + values: &mut Vec, + target_len: usize, +) -> Result<()> { + zero_fill_decode_elements( + values, + target_len, + cpu_decode_pool_threads() <= MAX_POOL_THREADS_FOR_ZEROED_BUFFERS, + ) +} + +/// Most pool threads for which fresh decode buffers come zeroed from the +/// allocator. First-touch page faults on one allocation from many threads +/// at once contend in the kernel: on 4 to 10 threads, spreading them over +/// the pool cut a process's first decode of a 4096 x 2048 RGB image by 10 to +/// 15%, but on 18 threads the faults took about four times the system time +/// and HTJ2K images, which decode each page quickly, lost 3 to 7%. +const MAX_POOL_THREADS_FOR_ZEROED_BUFFERS: usize = 10; + +/// [`try_zero_fill_decode_elements`] with the choice made. An empty buffer +/// short of capacity holds nothing that is read, so it is released rather +/// than copied into a larger allocation; the new one comes zeroed from the +/// allocator when `zeroed_allocation`, else it is zeroed here. +fn zero_fill_decode_elements( + values: &mut Vec, + target_len: usize, + zeroed_allocation: bool, +) -> Result<()> { + checked_decode_byte_len2(target_len, core::mem::size_of::())?; + if values.is_empty() && values.capacity() < target_len { + *values = Vec::new(); + if zeroed_allocation { + *values = bytemuck::allocation::try_zeroed_vec(target_len) + .map_err(|()| DecodingError::HostAllocationFailed)?; + return Ok(()); + } + } + try_resize_decode_elements(values, target_len, T::default()) +} + #[inline] pub(crate) fn checked_decode_sample_count(width: u32, height: u32) -> Result { #[cfg(target_pointer_width = "64")] diff --git a/crates/j2k-native/src/math.rs b/crates/j2k-native/src/math.rs index e5de1cefd..da54e976b 100644 --- a/crates/j2k-native/src/math.rs +++ b/crates/j2k-native/src/math.rs @@ -922,9 +922,9 @@ impl SimdBuffer { let capacity = Self::padded_len(capacity_len.max(original_len)).ok_or(SimdBufferAllocationError)?; let mut data = Vec::new(); - data.try_reserve_exact(capacity) + crate::try_zero_fill_decode_elements(&mut data, capacity) .map_err(|_| SimdBufferAllocationError)?; - data.resize(padded_len, 0.0); + data.truncate(padded_len); Ok(Self { data, original_len }) } diff --git a/crates/j2k-native/src/scalar.rs b/crates/j2k-native/src/scalar.rs index 851491b64..4087aa804 100644 --- a/crates/j2k-native/src/scalar.rs +++ b/crates/j2k-native/src/scalar.rs @@ -4,13 +4,12 @@ use alloc::vec::Vec; use crate::error::bail; use crate::{ - add_roi_shift_to_bitplanes, apply_roi_maxshift_inverse_i64, checked_decode_byte_len3, j2c, - profile, DecodingError, EncodedHtJ2kCodeBlock, EncodedJ2kCodeBlock, - HtCleanupEncodeDistribution, HtCodeBlockDecodeJob, HtCodeBlockDecodePhaseLimit, - J2kCodeBlockDecodeJob, J2kCodeBlockSegment, J2kCodeBlockStyle, J2kForwardDwt53Level, - J2kForwardDwt53Output, J2kForwardDwt97Level, J2kForwardDwt97Output, J2kPacketizationEncodeJob, - J2kSubBandDecodeJob, J2kSubBandType, J2kTier1TokenSegment, Result, ValidationError, - MAX_CLASSIC_DECODE_BITPLANES, MAX_DEINTERLEAVE_REFERENCE_BIT_DEPTH, + add_roi_shift_to_bitplanes, checked_decode_byte_len3, j2c, profile, DecodingError, + EncodedHtJ2kCodeBlock, EncodedJ2kCodeBlock, HtCleanupEncodeDistribution, HtCodeBlockDecodeJob, + HtCodeBlockDecodePhaseLimit, J2kCodeBlockDecodeJob, J2kCodeBlockSegment, J2kCodeBlockStyle, + J2kForwardDwt53Level, J2kForwardDwt53Output, J2kForwardDwt97Level, J2kForwardDwt97Output, + J2kPacketizationEncodeJob, J2kSubBandDecodeJob, J2kSubBandType, J2kTier1TokenSegment, Result, + ValidationError, MAX_CLASSIC_DECODE_BITPLANES, MAX_DEINTERLEAVE_REFERENCE_BIT_DEPTH, }; mod classic_decode; @@ -55,7 +54,9 @@ fn internal_j2k_sub_band_type(sub_band_type: J2kSubBandType) -> j2c::build::SubB } } -fn internal_j2k_code_block_style(style: J2kCodeBlockStyle) -> j2c::codestream::CodeBlockStyle { +pub(crate) fn internal_j2k_code_block_style( + style: J2kCodeBlockStyle, +) -> j2c::codestream::CodeBlockStyle { j2c::codestream::CodeBlockStyle { selective_arithmetic_coding_bypass: style.selective_arithmetic_coding_bypass, reset_context_probabilities: style.reset_context_probabilities, diff --git a/crates/j2k-native/src/scalar/classic_decode.rs b/crates/j2k-native/src/scalar/classic_decode.rs index 4e5d7ed1c..7b8f36734 100644 --- a/crates/j2k-native/src/scalar/classic_decode.rs +++ b/crates/j2k-native/src/scalar/classic_decode.rs @@ -1,9 +1,9 @@ // SPDX-License-Identifier: MIT OR Apache-2.0 use super::{ - add_roi_shift_to_bitplanes, apply_roi_maxshift_inverse_i64, bail, - internal_j2k_code_block_style, internal_j2k_sub_band_type, j2c, profile, DecodingError, - J2kCodeBlockDecodeJob, J2kSubBandDecodeJob, Result, MAX_CLASSIC_DECODE_BITPLANES, + add_roi_shift_to_bitplanes, bail, internal_j2k_code_block_style, internal_j2k_sub_band_type, + j2c, profile, DecodingError, J2kCodeBlockDecodeJob, J2kSubBandDecodeJob, Result, + MAX_CLASSIC_DECODE_BITPLANES, }; /// Adapter scalar classic J2K decoder helper for backend experimentation. @@ -135,10 +135,6 @@ pub(super) fn checked_code_block_output_layout( Ok(CodeBlockOutputLayout { stride }) } -#[expect( - clippy::cast_precision_loss, - reason = "the public scalar adapter intentionally emits f32 coefficients" -)] fn write_j2k_code_block_output( decode_context: &j2c::bitplane::BitPlaneDecodeContext, job: J2kCodeBlockDecodeJob<'_>, @@ -148,7 +144,6 @@ fn write_j2k_code_block_output( ) { let midpoint = decode_context.midpoint_reconstructor(job.number_of_coding_passes, job.roi_shift); - let exact = !irreversible_midpoint && job.roi_shift == 0 && midpoint.reversible_is_exact(); for (row_idx, coeff_row) in decode_context .coefficient_rows() .enumerate() @@ -156,25 +151,12 @@ fn write_j2k_code_block_output( { let row_start = row_idx * job.output_stride; let output_row = &mut output[row_start..row_start + layout.stride]; - if irreversible_midpoint { - midpoint.dequantize_row(coeff_row, output_row, job.dequantization_step); - continue; - } - if exact { - j2c::bitplane::MidpointReconstructor::scale_exact_reversible_row( - coeff_row, - output_row, - job.dequantization_step, - ); - continue; - } - for (coefficient, sample) in coeff_row.iter().zip(output_row.iter_mut()) { - *sample = apply_roi_maxshift_inverse_i64( - midpoint.reconstruct_reversible(*coefficient), - job.roi_shift, - ) as f32 - * job.dequantization_step; - } + midpoint.reconstruct_row( + coeff_row, + output_row, + job.dequantization_step, + irreversible_midpoint, + ); } } diff --git a/crates/j2k-native/src/tests.rs b/crates/j2k-native/src/tests.rs index 3c5e10a36..8544577e3 100644 --- a/crates/j2k-native/src/tests.rs +++ b/crates/j2k-native/src/tests.rs @@ -11,12 +11,15 @@ mod ht_refinement; mod idwt_into_channels; mod idwt_window_storage; mod line_region_idwt; +#[cfg(feature = "parallel")] +mod multi_segment; #[cfg(feature = "std")] mod packet_directory; #[cfg(feature = "parallel")] mod parallel_bands; mod progression_matrix; mod row_sink; +mod streamed_publication; mod workspace_peak; mod workspace_reuse; @@ -1407,9 +1410,9 @@ fn decode_region_rejects_code_block_layer_metadata_amplification() { }; let mut bytes = encode(&pixels, 8, 8, 1, 8, false, &options).expect("encode many-layer classic fixture"); - // About a million 4x4 code-blocks with 32 layers each: their layer + // About four million 4x4 code-blocks with 32 layers each: their layer // records alone would exceed the decode budget. - rewrite_siz_to_single_large_tile(&mut bytes, 4_096); + rewrite_siz_to_single_large_tile(&mut bytes, 8_192); let image = Image::new(&bytes, &DecodeSettings::default()).expect("large SIZ parses"); // A tiny region builds layer records only for the code-blocks it reads, @@ -2979,3 +2982,49 @@ fn decode_settings_constructors_make_strictness_explicit() { assert!(!DecodeSettings::strict().lenient_tolerance_enabled()); assert!(DecodeSettings::strict().strict); } + +#[test] +fn zero_filled_decode_elements_start_as_zeros_whatever_the_buffer_held() { + for zeroed_allocation in [true, false] { + // An empty buffer without room gets a fresh zeroed buffer. + let mut fresh = Vec::::new(); + zero_fill_decode_elements(&mut fresh, 4099, zeroed_allocation).expect("fresh buffer"); + assert_eq!(fresh.len(), 4099); + assert!(fresh.iter().all(|&sample| sample == 0.0)); + + // An emptied buffer with room keeps its allocation, and none of the + // samples it held before shows through. + let mut reused = vec![7.5_f32; 64]; + let capacity = reused.capacity(); + reused.clear(); + zero_fill_decode_elements(&mut reused, 48, zeroed_allocation).expect("reused buffer"); + assert_eq!(reused.capacity(), capacity); + assert_eq!(reused, vec![0.0; 48]); + + // A buffer that already holds samples keeps them, as + // `try_resize_decode_elements` does, and grows with zeros. + let mut grown = vec![3_i64; 3]; + zero_fill_decode_elements(&mut grown, 5, zeroed_allocation).expect("grown buffer"); + assert_eq!(grown, [3, 3, 3, 0, 0]); + } +} + +#[test] +fn zero_filled_decode_elements_reject_lengths_past_the_decode_cap() { + for zeroed_allocation in [true, false] { + let mut values = Vec::::new(); + assert_eq!( + zero_fill_decode_elements( + &mut values, + DEFAULT_MAX_DECODE_BYTES / 4 + 1, + zeroed_allocation + ), + Err(DecodeError::Validation(ValidationError::ImageTooLarge)) + ); + assert_eq!( + zero_fill_decode_elements(&mut values, usize::MAX, zeroed_allocation), + Err(DecodeError::Validation(ValidationError::ImageTooLarge)) + ); + assert_eq!(values.capacity(), 0); + } +} diff --git a/crates/j2k-native/src/tests/fused_last_level.rs b/crates/j2k-native/src/tests/fused_last_level.rs index d5e608a6a..a1f3fbd75 100644 --- a/crates/j2k-native/src/tests/fused_last_level.rs +++ b/crates/j2k-native/src/tests/fused_last_level.rs @@ -220,3 +220,133 @@ fn fused_region_pack_matches_staged_samples() { } } } + +/// The rows `roi` of an RGB8 image of `width` samples per row. +fn crop(rgb: &[u8], width: u32, (x, y, w, h): (u32, u32, u32, u32)) -> Vec { + (y..y + h) + .flat_map(|row| { + let start = ((row * width + x) * 3) as usize; + rgb[start..start + (w * 3) as usize].iter().copied() + }) + .collect() +} + +/// The streamed rows of `bytes`, packed into an RGB8 destination or handed +/// over as rows, must be the staged decode's bytes, on one thread and on +/// pools whose components reconstruct through one reader or a reader each, +/// for the whole image, a reduced one, and regions. +fn assert_streamed_rows_match_staged(bytes: &[u8], label: &str) { + #[cfg(feature = "parallel")] + let pools: Vec<(usize, rayon::ThreadPool)> = [1, 2, 4, 12] + .into_iter() + .map(|threads| { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + (threads, pool) + }) + .collect(); + for reduction in [0, 1] { + let image = Image::new_with_reduction(bytes, &DecodeSettings::default(), reduction) + .expect("parse streamed fixture"); + let (w, h) = (image.width(), image.height()); + let mut staged = DecoderContext::default(); + staged.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); + staged.tile_decode_context.staged_rgb8_pack = true; + let expected = image + .decode_with_context(&mut staged) + .expect("staged decode") + .data; + for roi in [ + (0, 0, w, h), + (1, 1, w - 1, h - 1), + (w / 3, h / 3, w / 3, h / 4), + ] { + let expected = crop(&expected, w, roi); + let rows = |parallelism, offers_destination| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(parallelism); + let mut sink = super::row_sink::Rgb8Sink { + width: roi.2 as usize, + rgb: vec![0; (roi.2 * roi.3 * 3) as usize], + offers_destination, + rectangles: 0, + }; + image + .decode_region_rows_for_integer_output_with_context( + roi, + &mut context, + &mut sink, + ) + .expect("decode streamed rows"); + sink.rgb + }; + let label = format!("{label} reduction={reduction} roi={roi:?}"); + for offers_destination in [true, false] { + assert_eq!( + rows(CpuDecodeParallelism::Serial, offers_destination), + expected, + "{label} serial destination={offers_destination}" + ); + #[cfg(feature = "parallel")] + for (threads, pool) in &pools { + assert_eq!( + pool.install(|| rows(CpuDecodeParallelism::Auto, offers_destination)), + expected, + "{label} threads={threads} destination={offers_destination}" + ); + } + } + } + } +} + +/// The streamed route finishes each row of the last level in the IDWT's +/// ring, as the planes' fused last level finishes its rows in its own: at +/// the fused level's sizes and origins, and at the staged ones. +#[test] +fn streamed_rows_match_the_fused_last_level() { + for case in CASES + .iter() + .filter(|case| case.components == 3 && case.bit_depth == 8) + { + let (width, height) = case.size; + for reversible in [true, false] { + let samples = noisy_samples(width * height * 3); + let bytes = encode( + &samples, + width, + height, + 3, + 8, + false, + &EncodeOptions { + reversible, + num_decomposition_levels: case.levels, + tile_size: case.tile, + ..EncodeOptions::default() + }, + ) + .expect("encode streamed fixture"); + let label = format!( + "{width}x{height} levels={} tile={:?} reversible={reversible}", + case.levels, case.tile + ); + assert_streamed_rows_match_staged(&bytes, &label); + } + } + let fixtures: [(&str, &[u8]); 2] = [ + ( + "5/3 75x56 at (5, 3)", + include_bytes!("../../fixtures/image_origin/openjpeg_rgb_75x56_origin5x3_53.j2k"), + ), + ( + "9/7 72x59 at (7, 1)", + include_bytes!("../../fixtures/image_origin/openjpeg_rgb_72x59_origin7x1_97.j2k"), + ), + ]; + for (name, bytes) in fixtures { + assert_streamed_rows_match_staged(bytes, name); + } +} diff --git a/crates/j2k-native/src/tests/multi_segment.rs b/crates/j2k-native/src/tests/multi_segment.rs new file mode 100644 index 000000000..171672639 --- /dev/null +++ b/crates/j2k-native/src/tests/multi_segment.rs @@ -0,0 +1,663 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! Classic code-blocks of several codeword segments (selective bypass, +//! termination on each pass, or both) and the other code-block styles +//! (context reset, segmentation symbols, vertically causal context, +//! predictable termination) decode on the worker pool as on one thread: in +//! place on the tile route, and band by band ahead of the streamed IDWT, +//! whose rows then need no component planes. +//! +//! The fixtures in `fixtures/multi_segment/` were encoded with `OpenJPEG` +//! 2.5.4's `opj_compress` from synthetic images of 64 x 64 and 128 x 128 +//! samples. The unsigned 8-bit red, green and blue samples at `(x, y)` are +//! +//! ```text +//! r = (7x² + 13y² + 5xy + 31x) mod 256 +//! g = (3x² + 11y² + 9xy + 29y + 64) mod 256 +//! b = (13x² + 5y² + 3xy + 7(x xor y)) mod 256 +//! ``` +//! +//! and the gray sample is `(r + 2g + b) / 4`, rounded down. Written as a +//! binary PPM (`P6`) and PGM (`P5`) of maximum 255, `rgb64.ppm`, +//! `rgb128.ppm`, `gray64.pgm` and `gray128.pgm`, they encode with two +//! decomposition levels, 16 x 16 code-blocks, RPCL order and three or four +//! quality layers, the last of most at a low compression ratio, so that +//! blocks code bit-planes far below their first four: +//! +//! ```sh +//! c="-n 3 -b 16,16 -p RPCL" +//! opj_compress -i gray128.pgm -o bypass_gray_53.j2k $c -M 1 -r 4,2,1 +//! opj_compress -i rgb64.ppm -o bypass_rgb_97.j2k $c -I -M 1 -r 8,4,2,1 +//! opj_compress -i gray128.pgm -o termall_gray_97.j2k $c -I -M 4 -r 16,8,3 +//! opj_compress -i rgb128.ppm -o termall_rgb_53.j2k $c -M 4 -r 32,16,8 +//! opj_compress -i rgb128.ppm -o reset_rgb_97.j2k $c -I -M 2 -r 32,16,8 +//! opj_compress -i gray128.pgm -o segmark_gray_53.j2k $c -M 32 -r 16,8,3 +//! opj_compress -i gray64.pgm -o bypass_termall_gray_97.j2k $c -I -M 5 -r 4,2,1 +//! opj_compress -i rgb64.ppm -o bypass_termall_rgb_53.j2k $c -M 5 -r 8,4,1 +//! opj_compress -i gray64.pgm -o all_modes_gray_97.j2k $c -I -M 63 -r 4,2,1 +//! ``` +//! +//! As archival images are often coded, one has selective bypass, four +//! decomposition levels, 64 x 64 code-blocks, precincts, packet lengths and +//! seven layers: +//! +//! ```sh +//! opj_compress -i rgb128.ppm -o bypass_rgb_97_64x64.j2k -n 5 -b 64,64 -p RPCL -PLT -I -M 1 \ +//! -c "[128,128],[128,128],[64,64]" -r 24,16,10,6,3,2,1.5 +//! ``` +//! +//! And one codes its first component in 32 x 64 code-blocks and the others +//! in 64 x 64 ones. Two encodes differ only in their code-block size: +//! +//! ```sh +//! c="-n 2 -p RPCL -PLT -mct 0 -M 1 -r 6,3,1.6" +//! opj_compress -i rgb128.ppm -o first.j2k $c -b 32,64 +//! opj_compress -i rgb128.ppm -o rest.j2k $c -b 64,64 +//! ``` +//! +//! `bypass_rgb_53_blocks_per_component.j2k` is the main header of +//! `rest.j2k`, without its comment and with a COC marker after its COD that +//! gives component 0 the first encode's code-block size (`Scoc` 0, then +//! `SPcoc` as in its COD); then one tile-part without packet lengths, whose +//! packets, in their RPCL order, are the first encode's for component 0 +//! and the second's for the others, as their packet lengths split them. +//! `opj_decompress` decodes its first component as the first encode's and +//! the others as the second's. +//! +//! `-M` adds 1 for selective bypass, 2 for context reset, 4 for termination +//! on each pass, 8 for the vertically causal context, 16 for predictable +//! termination and 32 for segmentation symbols. The samples contain no +//! externally sourced image content. + +use alloc::string::String; +use alloc::vec::Vec; +use core::cell::Cell; + +use crate::{ + ComponentPlane, ComponentRowSink, CpuDecodeParallelism, DecodeSettings, DecoderContext, Image, +}; + +const FIXTURES: [(&str, &[u8]); 11] = [ + ( + "bypass gray 5/3", + include_bytes!("../../fixtures/multi_segment/bypass_gray_53.j2k"), + ), + ( + "bypass rgb 9/7", + include_bytes!("../../fixtures/multi_segment/bypass_rgb_97.j2k"), + ), + ( + "termall gray 9/7", + include_bytes!("../../fixtures/multi_segment/termall_gray_97.j2k"), + ), + ( + "termall rgb 5/3", + include_bytes!("../../fixtures/multi_segment/termall_rgb_53.j2k"), + ), + ( + "reset rgb 9/7", + include_bytes!("../../fixtures/multi_segment/reset_rgb_97.j2k"), + ), + ( + "segmark gray 5/3", + include_bytes!("../../fixtures/multi_segment/segmark_gray_53.j2k"), + ), + ( + "bypass and termall gray 9/7", + include_bytes!("../../fixtures/multi_segment/bypass_termall_gray_97.j2k"), + ), + ( + "bypass and termall rgb 5/3", + include_bytes!("../../fixtures/multi_segment/bypass_termall_rgb_53.j2k"), + ), + ( + "all modes gray 9/7", + include_bytes!("../../fixtures/multi_segment/all_modes_gray_97.j2k"), + ), + ( + "bypass rgb 9/7, 64 x 64 blocks", + include_bytes!("../../fixtures/multi_segment/bypass_rgb_97_64x64.j2k"), + ), + ( + "bypass rgb 5/3, code-block size per component", + include_bytes!("../../fixtures/multi_segment/bypass_rgb_53_blocks_per_component.j2k"), + ), +]; + +/// Collects the sample bits a sink receives into region-sized planes, and +/// checks that each sample arrives once. +struct Rows { + width: usize, + planes: Vec>, + arrivals: Vec, +} + +impl ComponentRowSink for Rows { + fn write_rows(&mut self, x: u32, y: u32, planes: &[ComponentPlane<'_>]) { + if self.planes.is_empty() { + self.planes = alloc::vec![alloc::vec![0; self.arrivals.len()]; planes.len()]; + } + let (width, height) = planes[0].dimensions(); + for (component, plane) in planes.iter().enumerate() { + for row in 0..height as usize { + for column in 0..width as usize { + let at = (y as usize + row) * self.width + x as usize + column; + self.planes[component][at] = + plane.samples()[row * width as usize + column].to_bits(); + if component == 0 { + self.arrivals[at] += 1; + } + } + } + } + } +} + +/// What a decode wrote, and where it ran. +struct Decoded { + samples: Vec>, + streamed: bool, + pool_components: usize, + parted_bands: usize, + uncharged_line_samples: usize, + channel_bytes: usize, + /// The bytes the decode's budget held after `reset`. + baseline_bytes: usize, + /// The bytes that budget charged for its threads' scratch, and the + /// bytes the scratch held once the bands decoded. + thread_scratch: (usize, usize), +} + +fn context(parallelism: CpuDecodeParallelism) -> DecoderContext<'static> { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(parallelism); + context +} + +/// The region's rows, streamed to a sink where the decode can. +fn rows(image: &Image<'_>, roi: (u32, u32, u32, u32)) -> Decoded { + rows_with(image, roi, |_| {}) +} + +/// [`rows`] from a context `configure` sets up. +fn rows_with( + image: &Image<'_>, + roi: (u32, u32, u32, u32), + configure: impl FnOnce(&mut DecoderContext<'_>), +) -> Decoded { + let mut context = context(CpuDecodeParallelism::Auto); + configure(&mut context); + let mut sink = Rows { + width: roi.2 as usize, + planes: Vec::new(), + arrivals: alloc::vec![0; (roi.2 * roi.3) as usize], + }; + image + .decode_region_rows_for_integer_output_with_context(roi, &mut context, &mut sink) + .expect("decode region rows"); + assert!( + sink.arrivals.iter().all(|&count| count == 1), + "every sample arrives once" + ); + let counters = context.tile_decode_context.debug_counters; + Decoded { + samples: sink.planes, + streamed: context.tile_decode_context.stream_rows, + pool_components: counters.pool_band_components, + parted_bands: counters.pool_band_parts, + uncharged_line_samples: counters.uncharged_line_samples, + channel_bytes: counters.workspace_peak.bytes().channels, + baseline_bytes: counters.decode_baseline_bytes, + thread_scratch: ( + counters.thread_scratch_charged, + counters.thread_scratch_held, + ), + } +} + +/// The region's component planes from a serial decode. +fn planes(image: &Image<'_>, roi: (u32, u32, u32, u32)) -> Vec> { + let mut context = context(CpuDecodeParallelism::Serial); + let components = image + .decode_region_components_for_integer_output_with_context(roi, &mut context) + .expect("decode region planes"); + components + .planes() + .iter() + .map(|plane| { + plane.samples()[..(roi.2 * roi.3) as usize] + .iter() + .map(|sample| sample.to_bits()) + .collect() + }) + .collect() +} + +/// The image's native samples, whole or of `roi`, with Auto parallelism. +fn native(image: &Image<'_>, roi: Option<(u32, u32, u32, u32)>) -> Vec { + let mut context = context(CpuDecodeParallelism::Auto); + match roi { + Some(roi) => image.decode_native_region_with_context(roi, &mut context), + None => image.decode_native_with_context(&mut context), + } + .expect("decode native samples") + .data +} + +fn pool(threads: usize) -> rayon::ThreadPool { + rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool") +} + +fn regions(image: &Image<'_>) -> [(u32, u32, u32, u32); 3] { + let (width, height) = (image.width(), image.height()); + [ + (0, 0, width, height), + (width / 5, height / 7, width / 2, height / 3), + (width / 3, 1, width / 4, height - 2), + ] +} + +#[test] +fn multi_segment_blocks_decode_on_a_pool_as_on_one_thread() { + let one = pool(1); + for (label, bytes) in FIXTURES { + for reduction in [0_u8, 1] { + let image = Image::new_with_reduction(bytes, &DecodeSettings::default(), reduction) + .expect("parse fixture"); + let regions = regions(&image); + let expected_full = one.install(|| native(&image, None)); + let expected: Vec<_> = regions + .iter() + .map(|&roi| one.install(|| (native(&image, Some(roi)), rows(&image, roi)))) + .collect(); + for (&roi, (_, rows)) in regions.iter().zip(&expected) { + assert_eq!( + rows.samples, + planes(&image, roi), + "{label} 1/{} {roi:?}: rows on one thread against the planes", + 1 << reduction + ); + } + for threads in [2, 4, 12] { + pool(threads).install(|| { + assert!( + native(&image, None) == expected_full, + "{label} 1/{} on {threads} threads: full image", + 1 << reduction + ); + for (&roi, (expected_native, expected_rows)) in regions.iter().zip(&expected) { + let context = format!("{label} 1/{} {roi:?}", 1 << reduction); + assert!( + native(&image, Some(roi)) == *expected_native, + "{context} on {threads} threads: region" + ); + assert_eq!( + rows(&image, roi).samples, + expected_rows.samples, + "{context} on {threads} threads: rows" + ); + } + }); + } + } + } +} + +/// On a pool, these tiles decode on the pool's own routes: the tile route +/// for the planes, and pool bands for streamed rows, which then hold no +/// planes and leave no line-buffer samples uncharged. +#[test] +fn multi_segment_tiles_take_the_pool_routes() { + let mut parted = 0; + for threads in [4, 12] { + pool(threads).install(|| { + for (label, bytes) in FIXTURES { + let image = Image::new(bytes, &DecodeSettings::default()).expect("parse fixture"); + let mut context = context(CpuDecodeParallelism::Auto); + image + .decode_native_with_context(&mut context) + .expect("decode fixture"); + assert!( + context + .tile_decode_context + .debug_counters + .classic_task_workspace_growths + > 0, + "{label} on {threads} threads: the planes decode on the tile route" + ); + for roi in regions(&image) { + let decoded = rows(&image, roi); + let components = decoded.samples.len(); + assert!( + decoded.streamed + && decoded.pool_components == components + && decoded.channel_bytes <= 256, + "{label} {roi:?} on {threads} threads: rows from pool bands, no \ + planes: streamed {}, pool components {} of {components}, plane bytes {}", + decoded.streamed, + decoded.pool_components, + decoded.channel_bytes, + ); + assert_eq!(decoded.uncharged_line_samples, 0, "{label} {roi:?}"); + parted += decoded.parted_bands; + } + } + }); + } + assert!(parted > 0, "some pool bands decode in parts"); +} + +/// A full-image rows decode on a pool whose tiles only serial bands can +/// stream (here the pool's routes are switched off, as for a tile they +/// can't decode) streams from serial bands when its planes would pass the +/// decode budget, rather than fail on them, and keeps the planes within it. +/// With the pool's routes on, its bands stream these tiles within the +/// budget. +#[test] +fn rows_stream_from_serial_bands_when_the_planes_pass_the_budget() { + for (label, bytes) in FIXTURES { + let image = Image::new(bytes, &DecodeSettings::default()).expect("parse fixture"); + let roi = (0, 0, image.width(), image.height()); + let expected = pool(1).install(|| rows(&image, roi)); + pool(4).install(|| { + let planes = rows_with(&image, roi, |context| { + context.tile_decode_context.force_per_sub_band = true; + }); + assert!(!planes.streamed, "{label}: planes within the budget"); + assert_eq!(planes.samples, expected.samples, "{label}: planes"); + // `reset` charged the structural bytes and the planes: the + // planes pass this budget, which leaves room for a packet cache + // of up to half their size, which a decode past the budget + // drops to try again. + let budget = planes.baseline_bytes - planes.channel_bytes / 2; + let streamed = rows_with(&image, roi, |context| { + context.tile_decode_context.force_per_sub_band = true; + context.tile_decode_context.planes_budget = Some(budget); + }); + assert!( + streamed.streamed && streamed.pool_components == 0 && streamed.channel_bytes <= 256, + "{label}: serial bands past the budget: streamed {}, pool components {}, plane \ + bytes {}", + streamed.streamed, + streamed.pool_components, + streamed.channel_bytes, + ); + assert_eq!(streamed.samples, expected.samples, "{label}: serial bands"); + // The pool's bands take these tiles, within the budget too. + let pooled = rows_with(&image, roi, |context| { + context.tile_decode_context.planes_budget = Some(budget); + }); + let components = pooled.samples.len(); + assert!( + pooled.streamed && pooled.pool_components == components, + "{label}: pool bands within the budget" + ); + assert_eq!(pooled.samples, expected.samples, "{label}: pool bands"); + }); + } +} + +/// What the codeword segments of an image's classic code-blocks hold, as +/// its packets give them. +#[derive(Clone, Copy, Debug, Default)] +struct Segments { + /// Blocks of several segments. + segmented: usize, + /// Raw segments, and raw segments an arithmetic one follows. + raw: usize, + raw_then_arithmetic: usize, + /// Segments that hold coding passes and no bytes. + empty: usize, +} + +std::thread_local! { + /// Whether the image [`segments`] reads codes with selective bypass, and + /// in one codeword segment per block, and what it found in its tile. + static STYLE: Cell<(bool, bool)> = const { Cell::new((false, true)) }; + static FOUND: Cell> = const { Cell::new(None) }; +} + +/// The codeword segments of `image`'s classic code-blocks, read from its +/// tile's records once the tile route on a pool has decoded them. +fn segments(image: &Image<'_>) -> Segments { + let style = image + .header + .global_coding_style + .component_parameters + .parameters + .code_block_style; + pool(4).install(|| { + STYLE.with(|cell| { + cell.set(( + style.selective_arithmetic_coding_bypass, + style.single_codeword_segment(), + )); + }); + let mut context = context(CpuDecodeParallelism::Auto); + context.tile_decode_context.tile_route_post_decode = Some(|storage| { + // Past the first ten passes, the passes but the cleanup ones + // are raw. A block of one segment numbers its layers' + // contributions instead. + let (bypass, single) = STYLE.with(Cell::get); + let raw = |pass: u8| bypass && pass >= 10 && !pass.is_multiple_of(3); + let mut found = Segments::default(); + for block in &storage.code_blocks { + // Each segment's first pass, passes and bytes. + let mut segments: Vec<(u8, u8, usize)> = Vec::new(); + let (mut index, mut passes) = (None, 0_u8); + for layer in &storage.layers[block.layers.clone()] { + let Some(range) = layer.segments() else { + continue; + }; + for record in &storage.segments[range] { + if index != Some(record.idx) && !(single && index.is_some()) { + index = Some(record.idx); + segments.push((passes, 0, 0)); + } + if let Some(segment) = segments.last_mut() { + segment.1 += record.coding_pases; + segment.2 += record.data.len(); + } + passes += record.coding_pases; + } + } + found.segmented += usize::from(segments.len() > 1); + for (at, &(first, passes, bytes)) in segments.iter().enumerate() { + if raw(first) { + found.raw += 1; + found.raw_then_arithmetic += usize::from( + segments.get(at + 1).is_some_and(|&(next, _, _)| !raw(next)), + ); + } + found.empty += usize::from(passes > 0 && bytes == 0); + } + } + FOUND.with(|cell| cell.set(Some(found))); + Ok(()) + }); + image + .decode_native_with_context(&mut context) + .expect("decode fixture"); + FOUND + .with(Cell::take) + .expect("the tile route decoded the tile") + }) +} + +/// The fixtures' blocks hold every kind of codeword segment: blocks of +/// several in every fixture whose style makes them, raw segments and an +/// arithmetic segment after them in every one with selective bypass, and +/// some segments with coding passes and no bytes. +#[test] +fn fixtures_hold_every_kind_of_codeword_segment() { + let mut empty = 0; + for (label, bytes) in FIXTURES { + let image = Image::new(bytes, &DecodeSettings::default()).expect("parse fixture"); + let style = image + .header + .global_coding_style + .component_parameters + .parameters + .code_block_style; + let found = segments(&image); + assert_eq!( + found.segmented > 0, + !style.single_codeword_segment(), + "{label}: {found:?}" + ); + if style.selective_arithmetic_coding_bypass { + assert!( + found.raw > 0 && found.raw_then_arithmetic > 0, + "{label}: {found:?}" + ); + } + empty += found.empty; + } + assert!(empty > 0, "some segments hold coding passes and no bytes"); +} + +/// How a decode of an image ended: the planes' samples and the streamed +/// rows' samples, by component, or each one's error. +struct Outcome { + planes: Result, String>, + rows: Result>, String>, + /// Tiles whose failed pool bands decoded again over whole windows, and + /// the pool threads' scratches still held when they did. + reruns: usize, + live_scratches: usize, +} + +/// A strict or lenient Auto decode of `bytes` on the current pool, of the +/// planes and of the rows streamed from the whole image; rows that stream +/// must reach the sink once each. +fn outcome(bytes: &[u8], strict: bool, case: &str) -> Outcome { + let settings = DecodeSettings { + strict, + ..DecodeSettings::default() + }; + let image = match Image::new(bytes, &settings) { + Ok(image) => image, + Err(error) => { + return Outcome { + planes: Err(format!("parse {error:?}")), + rows: Err(format!("parse {error:?}")), + reruns: 0, + live_scratches: 0, + } + } + }; + let planes = image + .decode_native_with_context(&mut context(CpuDecodeParallelism::Auto)) + .map(|decoded| decoded.data) + .map_err(|error| format!("{error:?}")); + let roi = (0, 0, image.width(), image.height()); + let mut context = context(CpuDecodeParallelism::Auto); + let mut sink = Rows { + width: roi.2 as usize, + planes: Vec::new(), + arrivals: alloc::vec![0; (roi.2 * roi.3) as usize], + }; + let rows = image + .decode_region_rows_for_integer_output_with_context(roi, &mut context, &mut sink) + .map(|()| { + assert!( + sink.arrivals.iter().all(|&count| count == 1), + "{case}: every sample arrives once" + ); + sink.planes + }) + .map_err(|error| format!("{error:?}")); + let counters = context.tile_decode_context.debug_counters; + Outcome { + planes, + rows, + reruns: counters.whole_window_reruns, + live_scratches: counters.scratches_live_in_reruns, + } +} + +/// Corrupt packets of these blocks decode on a pool as on one thread, to +/// the same samples or the same error, strict or lenient, as planes and as +/// streamed rows. A tile whose pool bands fail decodes them again over +/// whole windows to find the error, without the pool threads' scratch. +#[test] +fn corrupt_multi_segment_blocks_decode_on_a_pool_as_on_one_thread() { + let mut state = 0x9e37_79b9_7f4a_7c15_u64; + let mut next = move || { + state ^= state << 13; + state ^= state >> 7; + state ^= state << 17; + state + }; + let (one, four) = (pool(1), pool(4)); + let (mut failures, mut reruns, mut live_scratches) = (0, 0, 0); + for (label, bytes) in FIXTURES { + // Corrupt only the tile's packets, after its start of data. + let data = bytes + .windows(2) + .position(|pair| pair == [0xff, 0x93]) + .expect("start of data") + + 2; + for iteration in 0..16 { + let mut mutant = Vec::new(); + mutant.extend_from_slice(bytes); + for _ in 0..=next() % 3 { + let span = u64::try_from(mutant.len() - data).expect("fixture length"); + let at = data + usize::try_from(next() % span).expect("offset in the fixture"); + mutant[at] ^= 1 << (next() % 8); + } + for strict in [false, true] { + let case = format!("{label} iteration {iteration} strict {strict}"); + let expected = one.install(|| outcome(&mutant, strict, &case)); + let pooled = four.install(|| outcome(&mutant, strict, &case)); + // Whole sample vectors, compared without printing them. + assert!( + pooled.planes == expected.planes, + "{case}: planes {:?}, on one thread {:?}", + pooled.planes.as_ref().err(), + expected.planes.as_ref().err() + ); + assert!( + pooled.rows == expected.rows, + "{case}: rows {:?}, on one thread {:?}", + pooled.rows.as_ref().err(), + expected.rows.as_ref().err() + ); + failures += usize::from(expected.rows.is_err()); + reruns += pooled.reruns; + live_scratches += pooled.live_scratches; + } + } + } + assert!( + failures > 20 && reruns > 0, + "{failures} failures, {reruns} whole-window reruns" + ); + assert_eq!( + live_scratches, 0, + "pool threads' scratches held while failed bands decoded again" + ); +} + +/// A tile streaming its rows from pool bands holds the scratch of every +/// thread, its workers' and its readers', as charged before they start, +/// though its components' blocks may differ in size. +#[test] +fn streamed_pool_band_threads_hold_the_scratch_charged_for_them() { + for threads in [4, 12] { + pool(threads).install(|| { + for (label, bytes) in FIXTURES { + let image = Image::new(bytes, &DecodeSettings::default()).expect("parse fixture"); + let decoded = rows(&image, (0, 0, image.width(), image.height())); + let (charged, held) = decoded.thread_scratch; + assert!( + decoded.pool_components == decoded.samples.len() && held > 0 && charged == held, + "{label} on {threads} threads: scratch charged {charged}, held {held}" + ); + } + }); + } +} diff --git a/crates/j2k-native/src/tests/parallel_bands.rs b/crates/j2k-native/src/tests/parallel_bands.rs index b74ce4e59..bdfc3037a 100644 --- a/crates/j2k-native/src/tests/parallel_bands.rs +++ b/crates/j2k-native/src/tests/parallel_bands.rs @@ -17,13 +17,25 @@ struct RegionDecode { decoded_blocks: usize, skipped_blocks: usize, pool_components: usize, + parted_bands: usize, peak_coefficients: usize, } /// Decode `roi` with Auto parallelism on the current pool, in bands on the /// pool unless `whole_windows`. fn decode(image: &Image<'_>, roi: (u32, u32, u32, u32), whole_windows: bool) -> RegionDecode { - decode_with_cutoff(image, roi, whole_windows, 0) + decode_with(image, roi, whole_windows, None, 0) +} + +/// [`decode`], with every pool band split into `parts` parts as far as its +/// blocks allow. +fn decode_in_parts( + image: &Image<'_>, + roi: (u32, u32, u32, u32), + whole_windows: bool, + parts: Option, +) -> RegionDecode { + decode_with(image, roi, whole_windows, parts, 0) } /// [`decode`] with bands averaging less than `band_work_cutoff` decoding over @@ -33,11 +45,23 @@ fn decode_with_cutoff( roi: (u32, u32, u32, u32), whole_windows: bool, band_work_cutoff: usize, +) -> RegionDecode { + decode_with(image, roi, whole_windows, None, band_work_cutoff) +} + +/// [`decode_in_parts`] and [`decode_with_cutoff`] at once. +fn decode_with( + image: &Image<'_>, + roi: (u32, u32, u32, u32), + whole_windows: bool, + parts: Option, + band_work_cutoff: usize, ) -> RegionDecode { let mut context = DecoderContext::default(); context.tile_decode_context.band_work_cutoff = band_work_cutoff; context.set_cpu_decode_parallelism(CpuDecodeParallelism::Auto); context.tile_decode_context.whole_window_tier1 = whole_windows; + context.tile_decode_context.band_parts = parts; let native = image .decode_native_region_with_context(roi, &mut context) .expect("decode region") @@ -48,6 +72,7 @@ fn decode_with_cutoff( decoded_blocks: counters.decoded_code_blocks, skipped_blocks: counters.skipped_code_blocks, pool_components: counters.pool_band_components, + parted_bands: counters.pool_band_parts, peak_coefficients: counters.workspace_peak.bytes().coefficients, } } @@ -127,22 +152,80 @@ fn pool_bands_match_whole_window_tier1_bit_for_bit() { } } } - // The route decodes HT blocks and classic blocks of one - // arithmetic segment. The encoder terminates every pass when it - // writes several layers, so those classic blocks keep whole - // windows. Classic tiles with fewer rows of code-blocks than - // threads do too, so only two threads reach every other case. - let routed = case.options.use_ht_block_coding || case.options.num_layers <= 1; - if !routed { - assert_eq!(pool_components, 0, "{}", case.label); - } else if threads == 2 { + // The route decodes HT and classic blocks, those the encoder + // terminates at every pass when it writes several layers + // included. Classic tiles with fewer rows of code-blocks than + // threads keep whole windows, so only two threads reach every + // case. + if threads == 2 { assert!(pool_components > 0, "{} never used the pool", case.label); } } } } +/// Threads may decode a classic band in parts, runs of its blocks, at once: +/// each part zeroes its own columns of the band's slot and copies its blocks +/// in under the slot's lock. Every split must match whole-window Tier-1 bit +/// for bit and decode and skip the same code-blocks. +#[test] +fn pool_band_parts_match_whole_window_tier1_bit_for_bit() { + let mut parted = 0; + for threads in [2, 5] { + let pool = pool(threads); + for case in cases() + .into_iter() + .filter(|case| !case.options.use_ht_block_coding && case.options.num_layers <= 1) + { + let pixels = noisy_pixels(case.components); + let bytes = encode( + &pixels, + WIDTH, + HEIGHT, + u16::from(case.components), + 8, + false, + &case.options, + ) + .expect("encode band part fixture"); + for reduction in [0_u8, 1] { + let image = + Image::new_with_reduction(&bytes, &DecodeSettings::default(), reduction) + .expect("parse band part fixture"); + let (width, height) = (image.width(), image.height()); + let tile = TILE >> reduction; + for roi in [ + (0, 0, width, height), + (width / 3, height / 3, width / 7, height / 9), + (tile - 5, tile - 3, 11, 7), + (width - 1, height - 1, 1, 1), + ] { + let whole = pool.install(|| decode(&image, roi, true)); + for parts in [2, 3, 7] { + let label = format!( + "{} threads={threads} reduction={reduction} roi={roi:?} parts={parts}", + case.label + ); + let banded = + pool.install(|| decode_in_parts(&image, roi, false, Some(parts))); + assert_eq!(banded.native, whole.native, "{label}"); + assert_eq!(banded.decoded_blocks, whole.decoded_blocks, "{label}"); + assert_eq!(banded.skipped_blocks, whole.skipped_blocks, "{label}"); + parted += banded.parted_bands; + } + } + } + } + } + assert!(parted > 0, "no band decoded in parts"); +} + fn rgb_fixture() -> Vec { + rgb_fixture_coded(false) +} + +/// The noisy RGB image in 16 x 16 code-blocks, HTJ2K ones when `ht`. +fn rgb_fixture_coded(use_ht_block_coding: bool) -> Vec { let pixels = noisy_pixels(3); encode( &pixels, @@ -156,12 +239,105 @@ fn rgb_fixture() -> Vec { num_decomposition_levels: 4, code_block_width_exp: 2, code_block_height_exp: 2, + use_ht_block_coding, ..EncodeOptions::default() }, ) .expect("encode rgb fixture") } +/// Discards the rows a decode streams. +struct DiscardRows; + +impl crate::ComponentRowSink for DiscardRows { + fn write_rows(&mut self, _x: u32, _y: u32, _planes: &[crate::ComponentPlane<'_>]) {} +} + +/// A tile whose rows stream from pool bands holds every component's plan at +/// once, beside the scratch of the threads that serve them all, and its +/// workspace peak counts them together: the three plans add up to the ones +/// a region decode into planes holds one after another, and the peak holds +/// all of them. +#[test] +fn streamed_pool_bands_record_every_live_plan() { + let bytes = rgb_fixture(); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse rgb fixture"); + let roi = (0, 0, WIDTH, HEIGHT); + pool(4).install(|| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(CpuDecodeParallelism::Auto); + image + .decode_native_region_with_context(roi, &mut context) + .expect("decode region planes"); + let one_at_a_time = context.tile_decode_context.debug_counters; + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(CpuDecodeParallelism::Auto); + image + .decode_region_rows_for_integer_output_with_context(roi, &mut context, &mut DiscardRows) + .expect("decode region rows"); + assert!(context.tile_decode_context.stream_rows); + let together = context.tile_decode_context.debug_counters; + assert_eq!(one_at_a_time.pool_band_components, 3); + assert_eq!(together.pool_band_components, 3); + assert_eq!( + together.pool_band_plan_bytes, + one_at_a_time.pool_band_plan_bytes + ); + let bank = context + .tile_decode_context + .tier1_capacity_bytes() + .expect("bank bytes"); + let tier1 = together.workspace_peak.bytes().tier1; + assert!( + tier1 >= bank + together.pool_band_plan_bytes, + "peak Tier-1 bytes {tier1}: the bank's {bank} and the plans' {}", + together.pool_band_plan_bytes + ); + }); +} + +/// A tile streaming its rows from pool bands reserves each thread block +/// scratch only for the blocks it decodes apart from their band's slot: an +/// HTJ2K tile, whose bands never split into parts, at most for the blocks a +/// region clips, and none for the whole image; a classic tile, whose bands +/// may decode in parts, for a whole 16 x 16 block. +#[test] +fn streamed_pool_band_threads_reserve_block_scratch_as_their_bands_need_it() { + const BLOCK: usize = 16 * 16; + pool(4).install(|| { + for ht in [true, false] { + let bytes = rgb_fixture_coded(ht); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse rgb fixture"); + let held = |roi| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(CpuDecodeParallelism::Auto); + image + .decode_region_rows_for_integer_output_with_context( + roi, + &mut context, + &mut DiscardRows, + ) + .expect("decode region rows"); + let counters = context.tile_decode_context.debug_counters; + assert_eq!(counters.pool_band_components, 3, "ht={ht} {roi:?}"); + assert_eq!( + counters.thread_scratch_held, counters.thread_scratch_charged, + "ht={ht} {roi:?}" + ); + counters.thread_block_samples + }; + let whole = held((0, 0, WIDTH, HEIGHT)); + let clipped = held((10, 12, 170, 130)); + if ht { + assert_eq!(whole, 0, "HTJ2K, whole image"); + assert!(clipped <= BLOCK, "HTJ2K region: {clipped}"); + } else { + assert_eq!((whole, clipped), (BLOCK, BLOCK), "classic"); + } + } + }); +} + /// With the pool's other thread held by unrelated work, workers can't run /// until the decode is over, so the IDWT must decode every band itself. #[test] @@ -452,3 +628,58 @@ fn cheap_bands_decode_over_whole_windows() { assert_eq!(cheap.native, pixels); assert_eq!(cheap.native, unlimited.native); } + +/// A region can need a sub-band's rows only from a code-block row it clips, +/// when the rows of code-blocks below it code nothing: the bands then hold +/// fewer rows than the ring was laid out for, and the ring cuts more slots +/// from its rows than it was laid out with. Each slot must still hold whole +/// rows, so that the bands, split into parts, decode as on one thread. +#[test] +fn pool_band_parts_decode_where_only_clipped_code_block_rows_code() { + let (width, height) = (256, 256); + // Noise in the top rows, flat below: of the finest sub-bands' rows of + // 32 x 32 code-blocks, only the first codes anything. + let mut state = 0x2545_f491_u32; + let pixels: Vec = (0..width * height) + .map(|at| { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + if at / width < 48 { + state.to_le_bytes()[3] + } else { + 128 + } + }) + .collect(); + let bytes = encode( + &pixels, + width, + height, + 1, + 8, + false, + &EncodeOptions { + reversible: true, + num_decomposition_levels: 2, + code_block_width_exp: 3, + code_block_height_exp: 3, + ..EncodeOptions::default() + }, + ) + .expect("encode clipped-rows fixture"); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse clipped-rows fixture"); + // The regions clip the first row of code-blocks of every sub-band, to + // 20 down to 12 of the finest sub-bands' 32 rows: 96 rows laid out for + // three slots then hold four to eight bands of the clipped rows, five + // and seven of which leave rows over. + for top in (40..=56).step_by(2) { + let roi = (0, top, width, height - top); + let serial = pool(1).install(|| decode(&image, roi, false)); + for parts in [2, 4] { + let pooled = pool(4).install(|| decode_in_parts(&image, roi, false, Some(parts))); + assert!(pooled.parted_bands > 0, "{roi:?} in {parts} parts"); + assert_eq!(pooled.native, serial.native, "{roi:?} in {parts} parts"); + } + } +} diff --git a/crates/j2k-native/src/tests/row_sink.rs b/crates/j2k-native/src/tests/row_sink.rs index 60bf21c9e..6ac13c0cb 100644 --- a/crates/j2k-native/src/tests/row_sink.rs +++ b/crates/j2k-native/src/tests/row_sink.rs @@ -1,9 +1,10 @@ // SPDX-License-Identifier: MIT OR Apache-2.0 -//! A serial region decode on the streamed IDWT hands its rows to a row sink -//! as they are finished, with every component decoded together and no -//! component planes. Its samples must match the planes bit for bit, every -//! sample must arrive once, and it must hold less. +//! A region decode on the streamed IDWT hands its rows to a row sink as they +//! are finished, with every component decoded together and no component +//! planes, its bands decoded serially or on the worker pool. Its samples must +//! match the planes bit for bit, every sample must arrive once, and it must +//! hold less. use alloc::vec::Vec; @@ -63,12 +64,40 @@ impl ComponentRowSink for CollectingSink { } } +#[expect( + clippy::struct_field_names, + reason = "decoded_blocks and skipped_blocks mirror the decoder's decoded and skipped code-block counters" +)] struct Decoded { planes: Vec>, depths: Vec<(u8, bool)>, streamed: bool, channel_bytes: usize, peak_bytes: usize, + #[cfg_attr( + not(feature = "parallel"), + expect(dead_code, reason = "only the pool band tests compare the counters") + )] + pool_components: usize, + #[cfg_attr( + not(feature = "parallel"), + expect(dead_code, reason = "only the pool band tests compare the counters") + )] + decoded_blocks: usize, + #[cfg_attr( + not(feature = "parallel"), + expect(dead_code, reason = "only the pool band tests compare the counters") + )] + skipped_blocks: usize, + #[cfg_attr( + not(feature = "parallel"), + expect(dead_code, reason = "only the pool band tests compare the counters") + )] + parted_bands: usize, + /// The deepest finest-level and coarser-level rings of the pool bands + /// that held no whole window. + #[cfg(feature = "parallel")] + ring_slots: (usize, usize), } fn decode_planes(image: &Image<'_>, roi: (u32, u32, u32, u32)) -> Decoded { @@ -101,12 +130,22 @@ fn decode_planes(image: &Image<'_>, roi: (u32, u32, u32, u32)) -> Decoded { if std::env::var_os("ROW_SINK_DEBUG").is_some() { std::eprintln!("planes {roi:?} {peak:?}"); } + let counters = context.tile_decode_context.debug_counters; Decoded { planes, depths, streamed: false, channel_bytes: peak.channels, peak_bytes: peak.total(), + pool_components: counters.pool_band_components, + decoded_blocks: counters.decoded_code_blocks, + skipped_blocks: counters.skipped_code_blocks, + parted_bands: counters.pool_band_parts, + #[cfg(feature = "parallel")] + ring_slots: ( + counters.pool_band_finest_slots, + counters.pool_band_other_slots, + ), } } @@ -114,9 +153,19 @@ fn decode_rows( image: &Image<'_>, roi: (u32, u32, u32, u32), parallelism: CpuDecodeParallelism, +) -> Decoded { + decode_rows_with(image, roi, parallelism, |_| {}) +} + +fn decode_rows_with( + image: &Image<'_>, + roi: (u32, u32, u32, u32), + parallelism: CpuDecodeParallelism, + configure: impl FnOnce(&mut DecoderContext), ) -> Decoded { let mut context = DecoderContext::default(); context.set_cpu_decode_parallelism(parallelism); + configure(&mut context); let mut sink = CollectingSink::new(roi.2, roi.3); image .decode_region_rows_for_integer_output_with_context(roi, &mut context, &mut sink) @@ -133,12 +182,22 @@ fn decode_rows( if std::env::var_os("ROW_SINK_DEBUG").is_some() { std::eprintln!("rows {roi:?} {peak:?}"); } + let counters = context.tile_decode_context.debug_counters; Decoded { planes: sink.planes, depths: sink.depths, streamed: context.tile_decode_context.stream_rows, channel_bytes: peak.channels, peak_bytes: peak.total(), + pool_components: counters.pool_band_components, + decoded_blocks: counters.decoded_code_blocks, + skipped_blocks: counters.skipped_code_blocks, + parted_bands: counters.pool_band_parts, + #[cfg(feature = "parallel")] + ring_slots: ( + counters.pool_band_finest_slots, + counters.pool_band_other_slots, + ), } } @@ -235,22 +294,409 @@ fn row_sink_matches_component_planes_bit_for_bit() { rows.peak_bytes, planes.peak_bytes ); - // A parallel decode keeps its planes and hands them over at - // the end. + // A parallel decode streams its rows when its bands can + // decode on the pool, and otherwise hands over its planes at + // the end (see `pool_band_rows_match_component_planes_bit_for_bit`). let parallel = decode_rows(&image, roi, CpuDecodeParallelism::Auto); - #[cfg(feature = "parallel")] - if rayon::current_num_threads() > 1 { - assert!(!parallel.streamed, "{label}: parallel planes"); - } assert_eq!(parallel.planes, planes.planes, "{label}: parallel"); } } } } -/// Outcome of a serial region decode, through the planes or the row sink: -/// a hash of the samples, or the error. -fn outcome(bytes: &[u8], strict: bool, roi: (u32, u32, u32, u32), rows: bool) -> String { +/// With two threads or more, a region decode whose bands the route can +/// decode streams its rows too, with every component's bands on the pool at +/// once. Its rows must match the serial planes bit for bit, decode and skip +/// the same code-blocks, and hold no planes. +#[cfg(feature = "parallel")] +#[test] +fn pool_band_rows_match_component_planes_bit_for_bit() { + for threads in [2, 4, 12] { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + for case in row_sink_cases() { + let pixels = noisy_pixels(case.components); + let bytes = encode( + &pixels, + WIDTH, + HEIGHT, + u16::from(case.components), + 8, + false, + &case.options, + ) + .expect("encode pool row fixture"); + let mut pool_components = 0; + for reduction in [0_u8, 1, 2] { + let image = + Image::new_with_reduction(&bytes, &DecodeSettings::default(), reduction) + .expect("parse pool row fixture"); + let (width, height) = (image.width(), image.height()); + let tile = TILE >> reduction; + for roi in [ + (0, 0, width, height), + ( + width / 20, + height / 20, + width - width / 10, + height - height / 10, + ), + (width / 3, height / 3, width / 7, height / 9), + (tile - 5, tile - 3, 11, 7), + (0, height - 9, width, 9), + (width - 1, height - 1, 1, 1), + ] { + let label = format!( + "{} threads={threads} reduction={reduction} roi={roi:?}", + case.label + ); + let planes = decode_planes(&image, roi); + let rows = + pool.install(|| decode_rows(&image, roi, CpuDecodeParallelism::Auto)); + assert_eq!(rows.planes, planes.planes, "{label}"); + assert_eq!(rows.depths, planes.depths, "{label}"); + if rows.pool_components > 0 { + assert!(rows.streamed, "{label}: streamed"); + assert!(rows.channel_bytes <= 256, "{label}: no planes"); + assert_eq!(rows.decoded_blocks, planes.decoded_blocks, "{label}"); + assert_eq!(rows.skipped_blocks, planes.skipped_blocks, "{label}"); + } + pool_components += rows.pool_components; + } + } + // As in `pool_bands_match_whole_window_tier1_bit_for_bit`: the + // route takes HT and classic blocks, and two threads reach every + // case. + if threads == 2 { + assert!(pool_components > 0, "{} never used the pool", case.label); + } + } + } +} + +/// A streamed decode whose classic bands split into parts, which threads +/// decode at once into each band's slot, must give the serial planes' rows +/// bit for bit, decode and skip the same code-blocks, and hold no planes. +#[cfg(feature = "parallel")] +#[test] +fn pool_band_part_rows_match_component_planes_bit_for_bit() { + let mut parted = 0; + for threads in [2, 13] { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + for case in row_sink_cases() + .into_iter() + .filter(|case| !case.options.use_ht_block_coding && case.options.num_layers <= 1) + { + let pixels = noisy_pixels(case.components); + let bytes = encode( + &pixels, + WIDTH, + HEIGHT, + u16::from(case.components), + 8, + false, + &case.options, + ) + .expect("encode band part row fixture"); + let image = Image::new_with_reduction(&bytes, &DecodeSettings::default(), 0) + .expect("parse band part row fixture"); + let (width, height) = (image.width(), image.height()); + for roi in [ + (0, 0, width, height), + (width / 3, height / 3, width / 7, height / 9), + (0, height - 9, width, 9), + ] { + let planes = decode_planes(&image, roi); + for parts in [None, Some(2), Some(5)] { + let label = format!( + "{} threads={threads} roi={roi:?} parts={parts:?}", + case.label + ); + let rows = pool.install(|| { + decode_rows_with(&image, roi, CpuDecodeParallelism::Auto, |context| { + context.tile_decode_context.band_parts = parts; + }) + }); + assert_eq!(rows.planes, planes.planes, "{label}"); + if rows.pool_components > 0 { + assert!(rows.streamed, "{label}: streamed"); + assert!(rows.channel_bytes <= 256, "{label}: no planes"); + assert_eq!(rows.decoded_blocks, planes.decoded_blocks, "{label}"); + assert_eq!(rows.skipped_blocks, planes.skipped_blocks, "{label}"); + } + parted += rows.parted_bands; + } + } + } + } + assert!(parted > 0, "no band decoded in parts"); +} + +/// Each component's route plans its own clipped code-blocks, while a pool +/// worker or reader decodes the strips of every component with one scratch. +/// Components with detail only along the right edge code only the narrow +/// last blocks of each band, and a noisy one full-size blocks, so a worker +/// that served an edge component first must still fit the noisy component's +/// clipped blocks. +#[cfg(feature = "parallel")] +#[test] +fn pool_band_rows_fit_every_components_clipped_blocks() { + let noise = noisy_pixels(1); + let mut pixels = Vec::with_capacity(noise.len() * 3); + for (at, &sample) in noise.iter().enumerate() { + let edge = at % WIDTH as usize >= WIDTH as usize - 6; + let detail = if edge { sample } else { 128 }; + pixels.extend_from_slice(&[sample, detail, detail ^ 0x55]); + } + for (label, use_ht_block_coding) in [("classic", false), ("ht", true)] { + let bytes = encode( + &pixels, + WIDTH, + HEIGHT, + 3, + 8, + false, + &EncodeOptions { + num_decomposition_levels: 4, + code_block_width_exp: 4, + code_block_height_exp: 0, + use_ht_block_coding, + use_mct: false, + ..EncodeOptions::default() + }, + ) + .expect("encode clipped block fixture"); + let image = Image::new_with_reduction(&bytes, &DecodeSettings::default(), 0) + .expect("parse clipped block fixture"); + for threads in [2, 3, 6, 12] { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + for roi in [(100, 17, 103, 120), (37, 3, 166, 9), (60, 41, 143, 101)] { + let planes = decode_planes(&image, roi); + for attempt in 0..4 { + let rows = + pool.install(|| decode_rows(&image, roi, CpuDecodeParallelism::Auto)); + assert_eq!( + rows.planes, planes.planes, + "{label} threads={threads} roi={roi:?} attempt={attempt}" + ); + } + } + } + } +} + +/// A sink with an RGB8 destination, which [`ComponentRowSink::write_rows`] +/// fills the way the destination's contract says: each 8-bit sample rounded +/// and saturated. +pub(super) struct Rgb8Sink { + pub(super) width: usize, + pub(super) rgb: Vec, + pub(super) offers_destination: bool, + pub(super) rectangles: usize, +} + +impl ComponentRowSink for Rgb8Sink { + #[expect( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + reason = "samples are rounded and saturated to 8 bits, as the destination contract says" + )] + fn write_rows(&mut self, x: u32, y: u32, planes: &[ComponentPlane<'_>]) { + self.rectangles += 1; + let (width, height) = planes[0].dimensions(); + for row in 0..height as usize { + for column in 0..width as usize { + let pixel = (y as usize + row) * self.width + x as usize + column; + for (channel, plane) in planes.iter().enumerate() { + let sample = plane.samples()[row * width as usize + column]; + self.rgb[pixel * 3 + channel] = sample.round().clamp(0.0, 255.0) as u8; + } + } + } + } + + fn rgb8_region(&mut self, width: u32, height: u32) -> Option<&mut [u8]> { + let len = (width as usize * 3).checked_mul(height as usize)?; + self.offers_destination.then(|| &mut self.rgb[..len]) + } +} + +/// A streamed decode may pack three 8-bit components straight into a sink's +/// RGB8 destination. The bytes must be those the sink packs from the rows, +/// on one thread and on the pool, where each component's reconstruction runs +/// as its own task when the pool is large enough. +#[cfg(feature = "parallel")] +#[test] +fn rgb8_destination_matches_the_sink_pack_byte_for_byte() { + let pools: Vec = [2, 4, 12] + .into_iter() + .map(|threads| { + rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool") + }) + .collect(); + for case in row_sink_cases() + .into_iter() + .filter(|case| case.components == 3) + { + let pixels = noisy_pixels(case.components); + let bytes = encode(&pixels, WIDTH, HEIGHT, 3, 8, false, &case.options) + .expect("encode RGB8 destination fixture"); + let mut packed_into_destination = 0; + for reduction in [0_u8, 1] { + let image = Image::new_with_reduction(&bytes, &DecodeSettings::default(), reduction) + .expect("parse RGB8 destination fixture"); + let (width, height) = (image.width(), image.height()); + for roi in [ + (0, 0, width, height), + ( + width / 20, + height / 20, + width - width / 10, + height - height / 10, + ), + (width / 3, height / 3, width / 7, height / 9), + ] { + let decode = |parallelism, offers_destination| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(parallelism); + let mut sink = Rgb8Sink { + width: roi.2 as usize, + rgb: alloc::vec![0; (roi.2 * roi.3 * 3) as usize], + offers_destination, + rectangles: 0, + }; + image + .decode_region_rows_for_integer_output_with_context( + roi, + &mut context, + &mut sink, + ) + .expect("decode RGB8 destination rows"); + sink + }; + let label = format!("{} reduction={reduction} roi={roi:?}", case.label); + let expected = decode(CpuDecodeParallelism::Serial, false); + for (pool, threads) in pools.iter().zip([2, 4, 12]) { + let packed = pool.install(|| decode(CpuDecodeParallelism::Auto, true)); + assert_eq!(packed.rgb, expected.rgb, "{label} threads={threads}"); + packed_into_destination += usize::from(packed.rectangles == 0); + let rows = pool.install(|| decode(CpuDecodeParallelism::Auto, false)); + assert_eq!(rows.rgb, expected.rgb, "{label} threads={threads} rows"); + } + } + } + assert!( + packed_into_destination > 0, + "{}: never packed into the destination", + case.label + ); + } +} + +/// The serial bands pack three 8-bit components into a sink's RGB8 +/// destination too: a 9/7 image with the MCT straight from the IDWT's rows +/// through the fused pack, others after transforming a copy of the rows. The +/// bytes must be those the sink packs from the rows. +#[test] +fn serial_rgb8_destination_matches_the_sink_pack_byte_for_byte() { + for case in row_sink_cases() + .into_iter() + .filter(|case| case.components == 3) + { + let pixels = noisy_pixels(case.components); + let bytes = encode(&pixels, WIDTH, HEIGHT, 3, 8, false, &case.options) + .expect("encode serial RGB8 destination fixture"); + let mut packed_into_destination = 0; + for reduction in [0_u8, 1] { + let image = Image::new_with_reduction(&bytes, &DecodeSettings::default(), reduction) + .expect("parse serial RGB8 destination fixture"); + let (width, height) = (image.width(), image.height()); + for roi in [ + (0, 0, width, height), + (width / 3, height / 3, width / 7, height / 9), + (0, height - 9, width, 9), + ] { + let decode = |offers_destination| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); + let mut sink = Rgb8Sink { + width: roi.2 as usize, + rgb: alloc::vec![0; (roi.2 * roi.3 * 3) as usize], + offers_destination, + rectangles: 0, + }; + image + .decode_region_rows_for_integer_output_with_context( + roi, + &mut context, + &mut sink, + ) + .expect("decode serial RGB8 destination rows"); + sink + }; + let label = format!("{} reduction={reduction} roi={roi:?}", case.label); + let expected = decode(false); + let packed = decode(true); + assert_eq!(packed.rgb, expected.rgb, "{label}"); + packed_into_destination += usize::from(packed.rectangles == 0); + } + } + assert!( + packed_into_destination > 0, + "{}: never packed into the destination", + case.label + ); + } +} + +/// A streamed decode has no per-sub-band path, so a classic component with +/// fewer rows of code-blocks than the pool has threads must still decode its +/// bands on the pool, in parts, rather than serially, with the planes' rows +/// bit for bit. +#[cfg(feature = "parallel")] +#[test] +fn pool_band_rows_use_a_pool_larger_than_the_code_block_rows() { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(48) + .build() + .expect("build pool"); + for case in row_sink_cases().into_iter().filter(|case| { + case.components == 3 && !case.options.use_ht_block_coding && case.options.num_layers <= 1 + }) { + let pixels = noisy_pixels(case.components); + let bytes = encode(&pixels, WIDTH, HEIGHT, 3, 8, false, &case.options) + .expect("encode large pool fixture"); + let image = Image::new_with_reduction(&bytes, &DecodeSettings::default(), 0) + .expect("parse large pool fixture"); + let roi = (0, 0, image.width(), image.height()); + let planes = decode_planes(&image, roi); + let rows = pool.install(|| decode_rows(&image, roi, CpuDecodeParallelism::Auto)); + assert_eq!(rows.planes, planes.planes, "{}", case.label); + assert!(rows.streamed, "{}: streamed", case.label); + assert!(rows.pool_components > 0, "{}: decoded serially", case.label); + } +} + +/// Outcome of a region decode, through the planes or the row sink: a hash +/// of the samples, or the error. +fn outcome( + bytes: &[u8], + strict: bool, + roi: (u32, u32, u32, u32), + rows: bool, + parallelism: CpuDecodeParallelism, +) -> String { let settings = DecodeSettings { strict, ..DecodeSettings::default() @@ -260,7 +706,7 @@ fn outcome(bytes: &[u8], strict: bool, roi: (u32, u32, u32, u32), rows: bool) -> Err(error) => return format!("parse {error:?}"), }; let mut context = DecoderContext::default(); - context.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); + context.set_cpu_decode_parallelism(parallelism); let hash = |planes: &[Vec]| { planes .iter() @@ -300,8 +746,14 @@ fn outcome(bytes: &[u8], strict: bool, roi: (u32, u32, u32, u32), rows: bool) -> fn row_sink_rejects_corrupt_code_blocks_as_component_planes_do() { // Small code-blocks give each component many bands, and the components // decode together, so two corrupt blocks often fail in a different order - // than the planes path decodes them. + // than the planes path decodes them. On the pool they also fail in any + // order across components. const SIZE: u32 = 48; + #[cfg(feature = "parallel")] + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(4) + .build() + .expect("build pool"); let mut state = 0x9e37_79b9_7f4a_7c15_u64; let mut next = move || { state ^= state << 13; @@ -340,12 +792,19 @@ fn row_sink_rejects_corrupt_code_blocks_as_component_planes_do() { for strict in [false, true] { let roi = [(0, 0, SIZE, SIZE), (30, 30, 18, 18), (5, 9, 13, 7)] [usize::try_from(next() % 3).expect("region index")]; - let planes = outcome(&mutant, strict, roi, false); - let rows = outcome(&mutant, strict, roi, true); - assert_eq!( - rows, planes, + let label = format!( "ht={use_ht_block_coding} iteration={iteration} strict={strict} roi={roi:?}" ); + let serial = CpuDecodeParallelism::Serial; + let planes = outcome(&mutant, strict, roi, false, serial); + let rows = outcome(&mutant, strict, roi, true, serial); + assert_eq!(rows, planes, "{label}"); + #[cfg(feature = "parallel")] + { + let auto = CpuDecodeParallelism::Auto; + let pool_rows = pool.install(|| outcome(&mutant, strict, roi, true, auto)); + assert_eq!(pool_rows, planes, "pool {label}"); + } failures += usize::from(planes.starts_with("decode")); } } @@ -454,6 +913,245 @@ fn streamed_rows_charge_every_components_rings() { } } +/// A classic tile of several components reconstructs each on a reader of +/// its own from 12 pool threads, and on 10 and 11 only from a size; the +/// readers then hold a batch of rows each, where one reader finishes its +/// rows in the IDWT's rings. The ROI plan must charge that batch on every +/// such pool: the +/// line buffer each streamed tile takes stays within the IDWT charge, and +/// the rows match the serial planes. The unit tests' size gate lies between +/// this full image and its half, so both sides of it run on 10 and 11 +/// threads. +#[cfg(feature = "parallel")] +#[test] +fn classic_streamed_rows_charge_the_batches_of_their_readers() { + let pixels = noisy_pixels(3); + let bytes = encode( + &pixels, + WIDTH, + HEIGHT, + 3, + 8, + false, + &EncodeOptions { + num_decomposition_levels: 4, + // 16-row blocks give every component more rows of blocks than + // the pool threads, so its bands decode on the pool. + code_block_height_exp: 2, + ..EncodeOptions::default() + }, + ) + .expect("encode reader charge fixture"); + for threads in [9, 10, 11, 12] { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + for reduction in [0_u8, 1] { + let image = Image::new_with_reduction(&bytes, &DecodeSettings::default(), reduction) + .expect("parse reader charge fixture"); + let roi = (0, 0, image.width(), image.height()); + let label = format!("threads={threads} reduction={reduction}"); + let planes = decode_planes(&image, roi); + let (rows, counters) = pool.install(|| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(CpuDecodeParallelism::Auto); + let mut sink = CollectingSink::new(roi.2, roi.3); + image + .decode_region_rows_for_integer_output_with_context( + roi, + &mut context, + &mut sink, + ) + .expect("decode region rows"); + assert!(context.tile_decode_context.stream_rows, "{label}"); + assert!(sink.arrivals.iter().all(|&count| count == 1), "{label}"); + (sink.planes, context.tile_decode_context.debug_counters) + }); + assert_eq!(rows, planes.planes, "{label}"); + assert_eq!(counters.pool_band_components, 3, "{label}: pool bands"); + let readers = if threads >= 12 || (threads >= 10 && reduction == 0) { + 3 + } else { + 1 + }; + assert_eq!(counters.pool_band_readers, readers, "{label}: readers"); + assert_eq!( + counters.uncharged_line_samples, 0, + "{label}: line buffer beyond the charge" + ); + // One reader finishes its rows in the IDWT's rings; a reader per + // component holds a batch of them. + assert_eq!( + counters.line_batch_samples == 0, + readers == 1, + "{label}: row batch {}", + counters.line_batch_samples + ); + } + } +} + +/// The fixture's noise repeated `across` times side by side and `down` +/// times one under the other, with `components` components, encoded with +/// four levels and 16-row code-blocks: every component has more rows of +/// blocks than the pool threads, so its bands decode on the pool. +#[cfg(feature = "parallel")] +fn pool_band_fixture(components: u8, across: usize, down: usize) -> Vec { + let row = usize::from(components) * WIDTH as usize; + let rows: Vec = noisy_pixels(components) + .chunks_exact(row) + .flat_map(|samples| core::iter::repeat_n(samples, across).flatten()) + .copied() + .collect(); + let width = WIDTH * u32::try_from(across).expect("fixture width"); + let height = HEIGHT * u32::try_from(down).expect("fixture height"); + encode( + &rows.repeat(down), + width, + height, + u16::from(components), + 8, + false, + &EncodeOptions { + num_decomposition_levels: 4, + code_block_height_exp: 2, + ..EncodeOptions::default() + }, + ) + .expect("encode pool band fixture") +} + +/// A classic tile that reconstructs each component on its own reader without +/// deep band rings holds two bands in each finest ring when its pool has +/// more threads than the tile has finest rings, and its bands then split +/// into parts so every thread has one: three components on 10 and 11 +/// threads and at half resolution from 12, four components at half +/// resolution on 13 threads but not on 12. One reader keeps three bands, +/// and the deep rings of full resolution from 12 threads four, both in +/// whole bands; the coarser levels keep theirs. The rows match the serial +/// planes on every pool. +#[cfg(feature = "parallel")] +#[test] +fn shallow_finest_rings_split_their_bands_into_parts() { + // Three fixture widths side by side give finest-level bands of several + // code-blocks at both resolutions, each above the unit tests' reader + // size gate, and two heights give the coarser levels rings that hold no + // whole window. + for (components, threads) in [(3_u8, &[9_usize, 10, 11, 12][..]), (4, &[12, 13][..])] { + let bytes = pool_band_fixture(components, 3, 2); + for reduction in [0_u8, 1] { + let image = Image::new_with_reduction(&bytes, &DecodeSettings::default(), reduction) + .expect("parse shallow ring fixture"); + let roi = (0, 0, image.width(), image.height()); + let planes = decode_planes(&image, roi); + for &threads in threads { + let label = + format!("components={components} threads={threads} reduction={reduction}"); + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + let rows = pool.install(|| decode_rows(&image, roi, CpuDecodeParallelism::Auto)); + assert_eq!(rows.planes, planes.planes, "{label}"); + assert_eq!( + rows.pool_components, + usize::from(components), + "{label}: pool bands" + ); + let deep = threads >= 12 && reduction == 0; + let shallow = threads >= 10 && !deep && 3 * usize::from(components) < threads; + let slots = if deep { + (4, 3) + } else if shallow { + (2, 2) + } else { + (3, 2) + }; + assert_eq!(rows.ring_slots, slots, "{label}: ring slots"); + assert_eq!(rows.parted_bands > 0, shallow, "{label}: parted bands"); + } + } + } +} + +/// A decode that shares its pool with other images in flight, as the images +/// of an explicit-worker batch that fill its pool do, holds two bands in +/// every ring below 12 threads, in place of three at the finest level, and +/// keeps its bands whole, where on 10 threads a lone decode's two-band rings +/// split them; with fewer bands per ring it peaks lower than a lone decode. +/// From 12 threads it holds the lone decode's deep rings. Its rows match the serial planes. A context +/// recovered from the workspace of a marked decode decodes as a fresh one +/// does. +#[cfg(feature = "parallel")] +#[test] +fn shared_pool_rows_hold_two_bands_per_ring() { + // Two fixture heights give the coarser levels rings that hold no whole + // window, deep ones too. + let bytes = pool_band_fixture(3, 1, 2); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse shared pool fixture"); + let roi = (0, 0, image.width(), image.height()); + let planes = decode_planes(&image, roi); + for (threads, lone_slots, shared_slots) in [ + (4, (3, 2), (2, 2)), + (10, (2, 2), (2, 2)), + (12, (4, 3), (4, 3)), + ] { + let label = format!("threads={threads}"); + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + let (lone, shared, recovered) = pool.install(|| { + let lone = decode_rows(&image, roi, CpuDecodeParallelism::Auto); + let shared = decode_rows_with(&image, roi, CpuDecodeParallelism::Auto, |context| { + context.set_shared_pool(true); + }); + let mut marked = DecoderContext::default(); + marked.set_shared_pool(true); + let mut sink = CollectingSink::new(roi.2, roi.3); + image + .decode_region_rows_for_integer_output_with_context(roi, &mut marked, &mut sink) + .expect("decode marked rows"); + let workspace = marked.into_workspace(); + let recovered = decode_rows_with(&image, roi, CpuDecodeParallelism::Auto, |context| { + *context = DecoderContext::from_workspace(workspace); + }); + (lone, shared, recovered) + }); + for (decoded, name) in [ + (&lone, "lone"), + (&shared, "shared"), + (&recovered, "recovered"), + ] { + assert_eq!(decoded.planes, planes.planes, "{label} {name}"); + assert_eq!(decoded.pool_components, 3, "{label} {name}: pool bands"); + } + assert_eq!(lone.ring_slots, lone_slots, "{label}: lone ring slots"); + assert_eq!( + shared.ring_slots, shared_slots, + "{label}: shared ring slots" + ); + assert_eq!( + recovered.ring_slots, lone_slots, + "{label}: recovered ring slots" + ); + assert_eq!( + (lone.parted_bands > 0, shared.parted_bands), + (threads == 10, 0), + "{label}: bands in parts" + ); + if shared_slots != lone_slots { + assert!( + shared.peak_bytes < lone.peak_bytes, + "{label}: shared {} lone {}", + shared.peak_bytes, + lone.peak_bytes + ); + } + } +} + /// A streamed decode that holds the image's packet directory and runs out of /// budget gives the directory up and reruns only while no row has reached the /// sink, since a rerun after that would send those rows again. This decode @@ -530,3 +1228,216 @@ fn streamed_rows_evict_the_packet_directory_before_the_first_row() { metadata ); } + +/// A classic tile whose components reconstruct on readers of their own +/// holds one batch of rows, which the ROI plan charges by the tile's width. +/// In an 8,200-wide image of 8,192-wide tiles, a batch sized by the image's +/// width keeps its 255 rows whole (half would be under 128), where the +/// tile's 8,192-wide rows are charged 128 (half of 256): the batch must keep +/// to the tile's charge, or the first tile holds twice it. +#[cfg(feature = "parallel")] +#[test] +fn reader_batches_of_tiles_narrower_than_the_image_stay_within_their_charge() { + let (width, height) = (8200_u32, 1024_u32); + // Smooth content keeps the large fixture's Tier-1 cheap. + let pixels: Vec = (0..height) + .flat_map(|y| { + (0..width).flat_map(move |x| { + [ + x.to_le_bytes()[0], + y.to_le_bytes()[0], + (x + y).to_le_bytes()[0], + ] + }) + }) + .collect(); + let bytes = encode( + &pixels, + width, + height, + 3, + 8, + false, + &EncodeOptions { + num_decomposition_levels: 2, + // 16-row blocks give each component more rows of blocks than + // the pool threads, so its bands decode on the pool. + code_block_height_exp: 2, + tile_size: Some((8192, height)), + ..EncodeOptions::default() + }, + ) + .expect("encode wide tile fixture"); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse wide tile fixture"); + let roi = (0, 0, width, height); + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(12) + .build() + .expect("build pool"); + let counters = pool.install(|| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(CpuDecodeParallelism::Auto); + let mut sink = CollectingSink::new(roi.2, roi.3); + image + .decode_region_rows_for_integer_output_with_context(roi, &mut context, &mut sink) + .expect("decode region rows"); + assert!(context.tile_decode_context.stream_rows); + assert!(sink.arrivals.iter().all(|&count| count == 1)); + context.tile_decode_context.debug_counters + }); + assert_eq!(counters.pool_band_readers, 3, "a reader per component"); + assert_eq!( + counters.line_batch_samples, + 3 * 8192 * 128, + "one batch of 128 rows of each component of the wide tile" + ); + assert_eq!( + counters.uncharged_line_samples, 0, + "line buffer beyond the charge" + ); +} + +/// A serial region decode that runs out of budget must do so before its +/// first row, whatever the budget, so that the retries without retained +/// scratch or the packet directory can still rerun it. A region that clips +/// the code-blocks of its last rows decodes them only after its first rows +/// went out, so the scratch an HTJ2K block clipped this way decodes into must +/// be held, and charged, from the start. Budgets just past the largest one +/// that fits can fit again where they leave out the optional row index, so +/// each one either decodes every sample once or fails before any. +#[test] +fn streamed_rows_run_out_of_budget_before_their_first_row() { + for use_ht_block_coding in [false, true] { + let pixels = noisy_pixels(3); + let bytes = encode( + &pixels, + WIDTH, + HEIGHT, + 3, + 8, + false, + &EncodeOptions { + use_ht_block_coding, + num_decomposition_levels: 3, + code_block_height_exp: 1, + ..EncodeOptions::default() + }, + ) + .expect("encode budget fixture"); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse fixture"); + for roi in [ + (37, 21, 130, 101), + (0, 0, WIDTH, 61), + (0, 0, WIDTH, HEIGHT - 11), + ] { + let label = format!("ht={use_ht_block_coding} {roi:?}"); + let decode = |baseline: usize| { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); + let mut sink = CollectingSink::new(roi.2, roi.3); + let result = image.run_decode_rows( + &mut context, + Some(roi), + None, + true, + baseline, + Some(&mut sink), + ); + (result, sink) + }; + let (mut fits, mut fails) = (0, crate::DEFAULT_MAX_DECODE_BYTES); + while fails - fits > 1 { + let candidate = fits + (fails - fits) / 2; + match decode(candidate).0 { + Ok(_) => fits = candidate, + Err(_) => fails = candidate, + } + } + let mut failures = 0; + for over in [ + 0, + 1, + 2, + 3, + 5, + 9, + 17, + 65, + 257, + 1 << 10, + 1 << 12, + 1 << 14, + 1 << 16, + ] { + let (result, sink) = decode(fits + over); + match result { + Ok(_) => assert!( + sink.arrivals.iter().all(|&count| count == 1), + "{label} +{over}: every sample once" + ), + Err(error) => { + assert!( + crate::j2c::is_capacity_error(&error), + "{label} +{over}: {error:?}" + ); + assert!( + sink.arrivals.iter().all(|&count| count == 0), + "{label} +{over}: rows before the failure" + ); + failures += 1; + } + } + } + assert!(failures > 0, "{label}: some budget fails"); + } + } +} + +/// Stitched windows of a reduced resolution, as a tiled viewer or an IIIF +/// server requests them, hold few rows of code-blocks per component: fewer +/// than a pool of 18 threads has workers. Such a window's streamed rows must +/// still decode its classic bands on the pool, in parts, and not fall back to +/// one serial thread for the whole tile, with the planes' rows bit for bit. +#[cfg(feature = "parallel")] +#[test] +fn reduced_windows_stream_on_a_pool_with_more_threads_than_their_rows() { + for threads in [18, 48] { + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool"); + for case in row_sink_cases().into_iter().filter(|case| { + case.components == 3 + && !case.options.use_ht_block_coding + && case.options.num_layers <= 1 + && case.options.tile_size.is_none() + }) { + let pixels = noisy_pixels(case.components); + let bytes = encode(&pixels, WIDTH, HEIGHT, 3, 8, false, &case.options) + .expect("encode reduced window fixture"); + for reduction in [1_u8, 2] { + let image = + Image::new_with_reduction(&bytes, &DecodeSettings::default(), reduction) + .expect("parse reduced window fixture"); + let (width, height) = (image.width(), image.height()); + let (half_width, half_height) = (width.div_ceil(2), height.div_ceil(2)); + for roi in [ + (0, 0, half_width, half_height), + (half_width, 0, width - half_width, half_height), + (0, height - 3, half_width, 3), + ] { + let label = format!( + "{} threads={threads} reduction={reduction} roi={roi:?}", + case.label + ); + let planes = decode_planes(&image, roi); + let rows = + pool.install(|| decode_rows(&image, roi, CpuDecodeParallelism::Auto)); + assert_eq!(rows.planes, planes.planes, "{label}"); + assert!(rows.streamed, "{label}: streamed"); + assert_eq!(rows.pool_components, 3, "{label}: decoded serially"); + } + } + } + } +} diff --git a/crates/j2k-native/src/tests/streamed_publication.rs b/crates/j2k-native/src/tests/streamed_publication.rs new file mode 100644 index 000000000..92d23d30a --- /dev/null +++ b/crates/j2k-native/src/tests/streamed_publication.rs @@ -0,0 +1,545 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! A streamed decode hands its rows to the sink as it finishes them, so a +//! decode that fails partway leaves some rows written. Which rows that is +//! depends on the route: rows one at a time on serial bands and with one +//! pool reader, whole batches with a reader per component. Whatever the +//! route, every row the sink receives must be the row a clean decode +//! produces, received once and in order from the top, with nothing after a +//! row that is missing; the error must be the one the failure raises; and a +//! decode that runs out of budget must fail before its first row, so that +//! the retries that rerun it (without the retained scratch, or without the +//! image's packet directory) still can. + +use alloc::vec; +use alloc::vec::Vec; + +use super::banded_tier1::{noisy_pixels, HEIGHT, WIDTH}; +use crate::error::{DecodeError, DecodingError}; +use crate::{ + encode, ComponentPlane, ComponentRowSink, CpuDecodeParallelism, DecodeSettings, DecoderContext, + EncodeOptions, Image, +}; + +/// What an RGB8 destination holds where nothing was written. +const UNWRITTEN: u8 = 0xa5; + +/// The rows a sink received, as sample bits per component, with the number +/// of times each pixel arrived; or, with a destination, the RGB8 bytes the +/// decoder packed into it. Each time the decoder asks for the destination, +/// the rows written since the last time are counted, moved out and the +/// destination blanked again, so a row written twice counts twice even with +/// the same bytes. +struct Published { + width: usize, + height: usize, + planes: Vec>, + arrivals: Vec, + rgb: Option>, + received: Vec, +} + +impl Published { + fn new((width, height): (u32, u32), destination: bool) -> Self { + let samples = (width * height) as usize; + Self { + width: width as usize, + height: height as usize, + planes: vec![vec![0; samples]; 3], + arrivals: vec![0; samples], + rgb: destination.then(|| vec![UNWRITTEN; samples * 3]), + received: vec![UNWRITTEN; if destination { samples * 3 } else { 0 }], + } + } + + /// Count and move out the destination's pixels written since the last + /// call: of every row whose first or last pixel was written (the + /// decoder writes whole rows of a tile this wide), or with `all`, of + /// every row. + fn collect(&mut self, all: bool) { + let Some(rgb) = self.rgb.as_mut() else { + return; + }; + let blank = |pixel: &[u8]| pixel.iter().all(|&byte| byte == UNWRITTEN); + for (y, row) in rgb.chunks_exact_mut(self.width * 3).enumerate() { + if !all && blank(&row[..3]) && blank(&row[row.len() - 3..]) { + continue; + } + for (x, pixel) in row.as_chunks_mut::<3>().0.iter_mut().enumerate() { + if !blank(pixel) { + let at = y * self.width + x; + self.arrivals[at] += 1; + self.received[at * 3..][..3].copy_from_slice(pixel); + pixel.fill(UNWRITTEN); + } + } + } + } + + /// The rows from the top that hold the clean decode's samples, checking + /// that every other row holds nothing and that no pixel arrived twice. + fn rows(&self, clean: &Self, label: &str) -> usize { + let mut published = 0; + for y in 0..self.height { + let label = format!("{label}: row {y}"); + if self.row_written(clean, y, &label) { + assert_eq!(published, y, "{label} after a missing row"); + published += 1; + } + } + published + } + + /// Whether row `y` arrived, checking that all of it arrived at most once + /// and that, if it arrived, it holds the clean row's samples or bytes. + fn row_written(&self, clean: &Self, y: usize, label: &str) -> bool { + let row = y * self.width..(y + 1) * self.width; + let arrivals = &self.arrivals[row.clone()]; + assert!( + arrivals.iter().all(|&count| count == arrivals[0]) && arrivals[0] <= 1, + "{label} arrived {arrivals:?}" + ); + let written = arrivals[0] == 1; + if written && self.rgb.is_some() { + let bytes = row.start * 3..row.end * 3; + assert_eq!( + self.received[bytes.clone()], + clean.received[bytes], + "{label}" + ); + } else if written { + for (plane, clean) in self.planes.iter().zip(&clean.planes) { + assert_eq!(plane[row.clone()], clean[row.clone()], "{label}"); + } + } + written + } + + fn untouched(&self) -> bool { + self.arrivals.iter().all(|&count| count == 0) + } +} + +impl ComponentRowSink for Published { + fn write_rows(&mut self, x: u32, y: u32, planes: &[ComponentPlane<'_>]) { + let (width, height) = planes[0].dimensions(); + for row in 0..height as usize { + for column in 0..width as usize { + let at = (y as usize + row) * self.width + x as usize + column; + for (component, plane) in planes.iter().enumerate() { + self.planes[component][at] = + plane.samples()[row * width as usize + column].to_bits(); + } + self.arrivals[at] += 1; + } + } + } + + fn rgb8_region(&mut self, width: u32, height: u32) -> Option<&mut [u8]> { + self.collect(false); + let len = (width as usize * 3).checked_mul(height as usize)?; + self.rgb.as_mut().map(|rgb| &mut rgb[..len]) + } +} + +/// A fixture with many bands per component: 8-row code-blocks over three +/// levels of three noisy components. +fn fixture(reversible: bool, use_ht_block_coding: bool, rpcl: bool) -> Vec { + encode( + &noisy_pixels(3), + WIDTH, + HEIGHT, + 3, + 8, + false, + &EncodeOptions { + reversible, + use_ht_block_coding, + num_decomposition_levels: 3, + code_block_height_exp: 1, + progression_order: if rpcl { + crate::EncodeProgressionOrder::Rpcl + } else { + EncodeOptions::default().progression_order + }, + ..EncodeOptions::default() + }, + ) + .expect("encode publication fixture") +} + +/// How a streamed decode runs: on serial bands, or on a pool of this many +/// threads (one reader below 12 for classic tiles, a reader per component +/// from 12 or for HTJ2K ones). +#[derive(Clone, Copy, Debug)] +enum Route { + Serial, + #[cfg(feature = "parallel")] + Pool(usize), +} + +fn routes() -> Vec { + #[cfg(feature = "parallel")] + return vec![Route::Serial, Route::Pool(4), Route::Pool(12)]; + #[cfg(not(feature = "parallel"))] + return vec![Route::Serial]; +} + +struct Outcome { + result: crate::Result<()>, + sink: Published, + streamed_samples: usize, + #[cfg_attr(not(feature = "parallel"), expect(dead_code, reason = "pool counters"))] + readers: usize, + #[cfg_attr(not(feature = "parallel"), expect(dead_code, reason = "pool counters"))] + fallbacks: usize, +} + +/// Stream the whole image along `route`, into a destination or as rows, on +/// a context `configure` sets up, from `baseline` bytes. +fn stream<'a>( + image: &Image<'a>, + route: Route, + destination: bool, + baseline: usize, + configure: impl FnOnce(&mut DecoderContext<'a>) + Send, +) -> Outcome { + stream_region(image, route, FULL, destination, baseline, configure) +} + +/// The whole image, as a region. +const FULL: (u32, u32, u32, u32) = (0, 0, WIDTH, HEIGHT); + +/// Stream `region` of the image along `route`, as [`stream`] does. +fn stream_region<'a>( + image: &Image<'a>, + route: Route, + region: (u32, u32, u32, u32), + destination: bool, + baseline: usize, + configure: impl FnOnce(&mut DecoderContext<'a>) + Send, +) -> Outcome { + let run = || { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(match route { + Route::Serial => CpuDecodeParallelism::Serial, + #[cfg(feature = "parallel")] + Route::Pool(_) => CpuDecodeParallelism::Auto, + }); + configure(&mut context); + let mut sink = Published::new((region.2, region.3), destination); + let result = image + .run_decode_rows( + &mut context, + Some(region), + None, + true, + baseline, + Some(&mut sink), + ) + .map(drop); + sink.collect(true); + let tile_ctx = &context.tile_decode_context; + // A decode past its budget may fail before it plans its rows. + assert!( + tile_ctx.stream_rows || result.is_err(), + "{route:?}: the decode streams" + ); + #[cfg(feature = "parallel")] + let (readers, fallbacks) = ( + tile_ctx.debug_counters.pool_band_readers, + tile_ctx.debug_counters.pool_band_fallbacks, + ); + #[cfg(not(feature = "parallel"))] + let (readers, fallbacks) = (0, 0); + Outcome { + result, + sink, + streamed_samples: tile_ctx.streamed_samples, + readers, + fallbacks, + } + }; + match route { + Route::Serial => run(), + #[cfg(feature = "parallel")] + Route::Pool(threads) => rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("build pool") + .install(run), + } +} + +fn is_band_failure(result: &crate::Result<()>) -> bool { + matches!( + result, + Err(DecodeError::Decoding(DecodingError::CodeBlockDecodeFailure)) + ) +} + +/// A band decode that fails partway through a streamed decode leaves only +/// clean rows, once and from the top, and the failure's error: none before +/// the first band, rows one at a time on serial bands and with one pool +/// reader, whole batches with a reader per component. On serial bands a +/// later failure leaves no fewer rows; on the pool, which band fails is the +/// one a worker happens to take at that count. +#[test] +fn a_failing_band_leaves_only_clean_rows_from_the_top() { + for (reversible, ht) in [(false, false), (true, false), (false, true)] { + let bytes = fixture(reversible, ht, false); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse fixture"); + for route in routes() { + for destination in [false, true] { + let clean = stream(&image, route, destination, 0, |_| {}); + clean.result.expect("clean decode"); + assert_eq!(clean.sink.rows(&clean.sink, "clean"), HEIGHT as usize); + #[cfg(feature = "parallel")] + if let Route::Pool(threads) = route { + let readers = if ht || threads >= 12 { 3 } else { 1 }; + assert_eq!(clean.readers, readers, "{route:?} ht={ht}: readers"); + } + let mut published = Vec::new(); + for after in [0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 10_000] { + let label = format!( + "reversible={reversible} ht={ht} {route:?} destination={destination} \ + failing band {after}" + ); + let outcome = stream(&image, route, destination, 0, |context| { + context.tile_decode_context.fail_band_after = Some(after); + }); + let rows = outcome.sink.rows(&clean.sink, &label); + if outcome.result.is_ok() { + // Past the tile's last band nothing fails. + assert_eq!(rows, HEIGHT as usize, "{label}"); + continue; + } + assert!( + is_band_failure(&outcome.result), + "{label}: {:?}", + outcome.result + ); + published.push(rows); + } + let label = format!("reversible={reversible} ht={ht} {route:?} {destination}"); + assert_eq!( + published.first(), + Some(&0), + "{label}: the first band fails first" + ); + if matches!(route, Route::Serial) { + assert!( + published.windows(2).all(|pair| pair[0] <= pair[1]), + "{label}: later failures leave no fewer rows: {published:?}" + ); + assert!( + published + .iter() + .any(|&rows| rows > 0 && rows < HEIGHT as usize), + "{label}: some failure comes partway: {published:?}" + ); + } + } + } + } +} + +/// A band the pool leaves running with no worker to finish it is a +/// scheduler fault. Before any row reaches the sink, the tile decodes again +/// on serial bands, as fast decodes do; after, the rows can't be taken +/// back, so the decode fails, having handed over only clean rows, once. +/// Which rows precede a strand depends on how far the workers ran ahead, so +/// each kind of tile must show both outcomes over its pools and outputs. +#[cfg(feature = "parallel")] +#[test] +fn a_band_stranded_after_published_rows_ends_the_decode() { + for ht in [false, true] { + let bytes = fixture(false, ht, false); + let image = Image::new(&bytes, &DecodeSettings::default()).expect("parse fixture"); + let (mut fell_back, mut failed_after_rows) = (0, 0); + for threads in [4, 12] { + let route = Route::Pool(threads); + for destination in [false, true] { + let clean = stream(&image, route, destination, 0, |_| {}); + clean.result.expect("clean decode"); + for after in [ + 0, 1, 2, 3, 4, 6, 8, 12, 16, 24, 32, 48, 64, 96, 128, 192, 256, + ] { + let label = format!( + "ht={ht} threads={threads} destination={destination} stranded task {after}" + ); + let outcome = stream(&image, route, destination, 0, |context| { + context.tile_decode_context.strand_pool_band = true; + context.tile_decode_context.strand_pool_band_after = after; + }); + let rows = outcome.sink.rows(&clean.sink, &label); + if outcome.result.is_ok() { + assert_eq!(rows, HEIGHT as usize, "{label}"); + fell_back += usize::from(outcome.fallbacks == 1); + } else { + assert!(is_band_failure(&outcome.result), "{label}"); + assert_eq!(outcome.fallbacks, 1, "{label}"); + assert!(rows > 0, "{label}: stranded before any row, yet failed"); + failed_after_rows += 1; + } + } + } + } + assert!( + fell_back > 0, + "ht={ht}: a strand before the first row falls back" + ); + assert!( + failed_after_rows > 0, + "ht={ht}: a strand after published rows fails" + ); + } +} + +/// The largest baseline at which a fresh decode of `region` along `route` +/// fits its budget. +fn fitting_baseline( + image: &Image<'_>, + route: Route, + region: (u32, u32, u32, u32), + destination: bool, +) -> usize { + let (mut fits, mut fails) = (0, crate::DEFAULT_MAX_DECODE_BYTES); + while fails - fits > 1 { + let candidate = fits + (fails - fits) / 2; + let outcome = stream_region(image, route, region, destination, candidate, |_| {}); + match outcome.result { + Ok(()) => fits = candidate, + Err(_) => fails = candidate, + } + } + fits +} + +/// A streamed decode charges everything it holds before its first row, so +/// whatever budget it starts from, a capacity failure comes before any row +/// reaches the sink: the retained-scratch and packet-directory retries, +/// which rerun only a decode that has handed nothing over, stay open on +/// every route. A decode that succeeds through either retry still does, +/// with every row once. Besides the whole image, a region that clips the +/// code-blocks of its last rows, which decode only after its first rows +/// went out. +#[test] +fn capacity_failures_come_before_the_first_row() { + for ht in [false, true] { + let bytes = fixture(false, ht, true); + for region in [FULL, (0, 0, WIDTH, HEIGHT - 11), (0, 0, WIDTH, 61)] { + for route in routes() { + for destination in [false, true] { + let label = format!("ht={ht} {region:?} {route:?} destination={destination}"); + capacity_failures_leave_the_sink_untouched( + &bytes, + route, + region, + destination, + &label, + ); + } + } + } + } +} + +fn capacity_failures_leave_the_sink_untouched( + bytes: &[u8], + route: Route, + region: (u32, u32, u32, u32), + destination: bool, + label: &str, +) { + let rows = region.3 as usize; + let fresh = Image::new(bytes, &DecodeSettings::default()).expect("parse fixture"); + let clean = stream_region(&fresh, route, region, destination, 0, |_| {}); + clean.result.expect("clean decode"); + let fits = fitting_baseline(&fresh, route, region, destination); + // Past the largest budget that fits, one that leaves out the optional + // row index can fit again: each decodes every row once or fails first. + let mut failures = 0; + for over in [ + 1, + 2, + 3, + 5, + 9, + 17, + 65, + 257, + 1 << 10, + 1 << 12, + 1 << 14, + 1 << 16, + 1 << 18, + ] { + let outcome = stream_region(&fresh, route, region, destination, fits + over, |_| {}); + let label = format!("{label} +{over}"); + match outcome.result { + Ok(()) => assert_eq!(outcome.sink.rows(&clean.sink, &label), rows), + Err(error) => { + assert!(crate::j2c::is_capacity_error(&error), "{label}: {error:?}"); + assert_eq!(outcome.streamed_samples, 0, "{label}"); + assert!(outcome.sink.untouched(), "{label}: rows before the failure"); + failures += 1; + } + } + } + assert!(failures > 0, "{label}: some budget fails"); + for under in [0, 1, 1 << 10, 1 << 16] { + let outcome = stream_region(&fresh, route, region, destination, fits - under, |_| {}); + outcome.result.expect("within the budget"); + assert_eq!(outcome.sink.rows(&clean.sink, label), rows); + } + + // A context holding the scratch of a planes decode starts over the + // budget; the retry without it fits. + let mut reused = DecoderContext::default(); + drop( + fresh + .decode_region_components_for_integer_output_with_context(region, &mut reused) + .expect("planes decode"), + ); + assert!(reused.tile_decode_context.retained_scratch_bytes().unwrap() > 0); + let outcome = stream_region(&fresh, route, region, destination, fits, |context| { + core::mem::swap(context, &mut reused); + context.set_cpu_decode_parallelism(match route { + Route::Serial => CpuDecodeParallelism::Serial, + #[cfg(feature = "parallel")] + Route::Pool(_) => CpuDecodeParallelism::Auto, + }); + }); + outcome + .result + .expect("a reused context fits as a fresh one"); + assert_eq!(outcome.sink.rows(&clean.sink, label), rows); + + // An image whose repeated regions built a packet directory gives it up + // for a request that fits without it. + #[cfg(feature = "std")] + { + let image = Image::new(bytes, &DecodeSettings::default()).expect("parse fixture"); + let metadata = image.retained_allocation_bytes().expect("metadata bytes"); + for _ in 0..2 { + stream_region(&image, route, region, destination, metadata, |_| {}) + .result + .expect("decode with the directory"); + } + assert!( + image.retained_allocation_bytes().expect("retained bytes") > metadata, + "{label}: repeated decodes build a directory" + ); + let outcome = stream_region(&image, route, region, destination, fits, |_| {}); + outcome + .result + .expect("evicting the directory fits the request"); + assert_eq!(outcome.sink.rows(&clean.sink, label), rows); + assert_eq!( + image.retained_allocation_bytes().expect("retained bytes"), + metadata, + "{label}: the directory is evicted" + ); + } +} diff --git a/crates/j2k-native/src/tests/workspace_reuse.rs b/crates/j2k-native/src/tests/workspace_reuse.rs index e75effd51..b48e55542 100644 --- a/crates/j2k-native/src/tests/workspace_reuse.rs +++ b/crates/j2k-native/src/tests/workspace_reuse.rs @@ -574,8 +574,8 @@ fn classic_tile_route_matches_serial_decode() { // (case, codestream, decoded in place on the tile route). The later // components of a colour image decode into the reused coefficient buffer, // most blocks of the three-layer OpenJPEG fixture gather their payloads, - // and the crate's own layered classic encode terminates every pass, which - // keeps it on the per-sub-band path. + // and the crate's own layered classic encode terminates every pass, so + // its blocks collect their codeword segments. for (case, bytes, in_place) in [ ("lossless colour", gradient(3, 1), true), ( @@ -592,7 +592,7 @@ fn classic_tile_route_matches_serial_decode() { .to_vec(), true, ), - ("pass termination", gradient(1, 3), false), + ("pass termination", gradient(1, 3), true), ] { let image = Image::new(&bytes, &DecodeSettings::default()).expect("classic image"); let mut serial = DecoderContext::default(); diff --git a/crates/j2k-test-support/fixtures/empty_coarse_levels/README.md b/crates/j2k-test-support/fixtures/empty_coarse_levels/README.md new file mode 100644 index 000000000..7d6f564e5 --- /dev/null +++ b/crates/j2k-test-support/fixtures/empty_coarse_levels/README.md @@ -0,0 +1,31 @@ +# Empty coarse levels fixtures + +Two small images with two decomposition levels whose last tile row (and, in +the RGB image, column) is one sample at an odd position. That tile's two +coarse resolutions are empty (rows `ceil(5/2)` to `ceil(6/2)` and `ceil(5/4)` +to `ceil(6/4)`) while its full resolution is not. A region decode of the tile +must reconstruct it from the first level that has samples, as the full-image +decode does. + +The unsigned 8-bit sample of component `c` at `(x, y)` is: + +```text +(13*x + 29*y + 37*c + 7) mod 251 +``` + +Create a binary PGM (`P5`, 8 x 6, component 0) and PPM (`P6`, 6 x 6, +components 0 to 2) with those row-major samples, then run OpenJPEG 2.5.4: + +```sh +opj_compress -i gray8x6.pgm -o openjpeg_gray_8x6_tile8x5_53.j2k -n 3 -t 8,5 +opj_compress -i rgb6x6.ppm -o openjpeg_rgb_6x6_tile5x5_97.j2k -n 3 -t 5,5 -I +``` + +| File | Contents | SHA-256 | +| --- | --- | --- | +| `openjpeg_gray_8x6_tile8x5_53.j2k` | Gray, 8 x 5 tiles, reversible 5/3; decodes to the samples exactly. | `09b737d7300ea4ddfc81a29740083105594e77c9029a2635c77bd05c98fb4d85` | +| `openjpeg_rgb_6x6_tile5x5_97.j2k` | RGB, 5 x 5 tiles, irreversible 9/7 with the colour transform. | `ea6d998d5d77cc63f86dbda5f9924177848eac1ea25c17cb5ea162ba687561e7` | + +The samples are generated for j2k and contain no externally sourced image +content. OpenJPEG is an independent fixture encoder, not a production codec +dependency. diff --git a/crates/j2k-test-support/fixtures/empty_coarse_levels/openjpeg_gray_8x6_tile8x5_53.j2k b/crates/j2k-test-support/fixtures/empty_coarse_levels/openjpeg_gray_8x6_tile8x5_53.j2k new file mode 100644 index 000000000..3c8ab14e5 Binary files /dev/null and b/crates/j2k-test-support/fixtures/empty_coarse_levels/openjpeg_gray_8x6_tile8x5_53.j2k differ diff --git a/crates/j2k-test-support/fixtures/empty_coarse_levels/openjpeg_rgb_6x6_tile5x5_97.j2k b/crates/j2k-test-support/fixtures/empty_coarse_levels/openjpeg_rgb_6x6_tile5x5_97.j2k new file mode 100644 index 000000000..9c2fc3775 Binary files /dev/null and b/crates/j2k-test-support/fixtures/empty_coarse_levels/openjpeg_rgb_6x6_tile5x5_97.j2k differ diff --git a/crates/j2k/Cargo.toml b/crates/j2k/Cargo.toml index 6905aac67..3f324f80d 100644 --- a/crates/j2k/Cargo.toml +++ b/crates/j2k/Cargo.toml @@ -24,12 +24,12 @@ j2k-core = { path = "../j2k-core", version = "=0.13.2" } j2k-native = { path = "../j2k-native", version = "=0.13.2" } j2k-types = { path = "../j2k-types", version = "=0.13.2" } moxcms = { workspace = true } +rayon = { workspace = true } thiserror = { workspace = true } [dev-dependencies] proptest = { workspace = true } criterion = { workspace = true } -rayon = { workspace = true } j2k-native = { path = "../j2k-native", version = "=0.13.2" } j2k-test-support = { path = "../j2k-test-support", features = ["j2k-native-fixtures"] } diff --git a/crates/j2k/src/batch/scheduler.rs b/crates/j2k/src/batch/scheduler.rs index 09c239f6e..b69844406 100644 --- a/crates/j2k/src/batch/scheduler.rs +++ b/crates/j2k/src/batch/scheduler.rs @@ -25,7 +25,7 @@ use crate::CpuDecodeParallelism; mod retained; pub(crate) use self::retained::{ - decode_retained_tiles, retained_inner_parallelism, run_retained_chunks, + decode_retained_tiles, retained_inner_parallelism, retained_worker_count, run_retained_chunks, }; pub(super) type ScopedWorkerHandle<'scope> = ( diff --git a/crates/j2k/src/batch/scheduler/retained.rs b/crates/j2k/src/batch/scheduler/retained.rs index 427157ba9..8c2d7b036 100644 --- a/crates/j2k/src/batch/scheduler/retained.rs +++ b/crates/j2k/src/batch/scheduler/retained.rs @@ -26,6 +26,15 @@ pub(crate) fn retained_inner_parallelism( Ok(plan_batch(job_count, options)?.inner_parallelism) } +/// Workers that [`run_retained_chunks`] runs at once for a batch of +/// `job_count` jobs. +pub(crate) fn retained_worker_count( + job_count: usize, + options: TileBatchOptions, +) -> Result { + Ok(plan_batch(job_count, options)?.worker_count) +} + /// Run disjoint jobs without destroying worker-owned scratch after each call. pub(crate) fn run_retained_chunks( workers: &mut [BatchWorker], diff --git a/crates/j2k/src/batch/worker.rs b/crates/j2k/src/batch/worker.rs index 437404c4e..8e8073e69 100644 --- a/crates/j2k/src/batch/worker.rs +++ b/crates/j2k/src/batch/worker.rs @@ -35,6 +35,9 @@ pub(crate) struct BatchWorker { /// Parallelism one owned-batch image decode may use; set per submission /// by the retained scheduler. owned_inner_parallelism: CpuDecodeParallelism, + /// Test-only count of offset-plan executions with nested parallelism. + #[cfg(test)] + parallel_plan_decode_calls: u64, } impl BatchWorker { @@ -56,6 +59,8 @@ impl BatchWorker { prepared_plan_decode_calls: 0, allocation_budget, owned_inner_parallelism: CpuDecodeParallelism::Serial, + #[cfg(test)] + parallel_plan_decode_calls: 0, } } diff --git a/crates/j2k/src/batch/worker/owned.rs b/crates/j2k/src/batch/worker/owned.rs index 3f239838a..59fdc3066 100644 --- a/crates/j2k/src/batch/worker/owned.rs +++ b/crates/j2k/src/batch/worker/owned.rs @@ -119,8 +119,7 @@ impl BatchWorker { signed: bool, ) -> j2k_native::Result> { self.prepared_plan_decode_calls = self.prepared_plan_decode_calls.saturating_add(1); - self.prepared_plan_scratch - .set_parallel_decode(self.owned_inner_parallelism == CpuDecodeParallelism::Auto); + self.prepare_plan_parallelism(); j2k_native::execute_referenced_htj2k_plan( plan, encoded_input, @@ -136,8 +135,7 @@ impl BatchWorker { signed: bool, ) -> j2k_native::Result> { self.prepared_plan_decode_calls = self.prepared_plan_decode_calls.saturating_add(1); - self.prepared_plan_scratch - .set_parallel_decode(self.owned_inner_parallelism == CpuDecodeParallelism::Auto); + self.prepare_plan_parallelism(); j2k_native::execute_referenced_classic_plan( plan, encoded_input, @@ -146,6 +144,23 @@ impl BatchWorker { ) } + /// Nested parallelism of the next offset-plan execution: only where the + /// owned scheduler allowed it for this worker's decodes. + fn prepare_plan_parallelism(&mut self) { + let parallel = self.owned_inner_parallelism == CpuDecodeParallelism::Auto; + #[cfg(test)] + { + self.parallel_plan_decode_calls += u64::from(parallel); + } + self.prepared_plan_scratch.set_parallel_decode(parallel); + } + + /// Offset-plan executions with nested parallelism so far. + #[cfg(test)] + pub(crate) const fn parallel_plan_decode_calls(&self) -> u64 { + self.parallel_plan_decode_calls + } + pub(crate) const fn prepared_plan_decode_calls(&self) -> u64 { self.prepared_plan_decode_calls } diff --git a/crates/j2k/src/decode.rs b/crates/j2k/src/decode.rs index bb7aeeb40..49531480e 100644 --- a/crates/j2k/src/decode.rs +++ b/crates/j2k/src/decode.rs @@ -12,15 +12,28 @@ pub use component_handoff::{ J2kNativeComponentPlane, }; mod output; +pub(crate) use output::{ + decode_native_components_with_native_context, native_components_route, + native_region_components_route, NativeRoute, +}; +mod route; mod settings; mod srgb8; +mod stripes; use output::{ - can_decode_u8_directly, expand_rgb8_to_rgba8, write_components_u16_output, - write_components_u8_output, write_u16_output, write_u8_output, U8RowSink, + can_decode_u8_directly, expand_rgb8_to_rgba8, region_rows_preferred_u16, rows_match_native_u16, + write_components_u16_output, write_components_u8_output, write_u16_output, write_u8_output, + U16RowSink, U8RowSink, }; +pub(crate) use route::{full_image_route, FullImageRoute, PlanesBudget, RouteRequest}; pub use settings::DecodeSettings; pub(crate) use srgb8::decode_image_srgb8; pub use srgb8::{J2kSrgb8Image, J2kSrgb8Layout}; +pub(crate) use stripes::{ + decode_image_rows_in_stripes, rows_stream_whole, DeliverRow, StreamedRowsError, +}; +#[cfg(test)] +pub(crate) use stripes::{InjectedFailure, INJECTED_FAILURE, STREAMED_ROW_DECODES}; /// Non-fatal JPEG 2000 decode warning surfaced through decode outcomes. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -62,6 +75,70 @@ pub(crate) fn decode_warnings_for_image(image: &Image<'_>) -> Vec( + image: &Image<'a>, + native_context: &mut j2k_native::DecoderContext<'a>, + out: &mut [u8], + stride: usize, + fmt: PixelFormat, + route: &FullImageRoute, +) -> Result<(), J2kError> { + match route { + FullImageRoute::Planes => { + decode_image_into_with_native_context(image, native_context, out, stride, fmt) + } + FullImageRoute::StreamedRegion => decode_image_region_rows_into_with_native_context( + image, + native_context, + out, + stride, + fmt, + Rect::full((image.width(), image.height())), + ), + FullImageRoute::StreamedTiles { tiles, workers } => { + route::tiles::decode_tiles_into(image, tiles, *workers, (out, stride, fmt)) + } + } +} + +/// Decode the full `image` into `out` on `route`, which [`full_image_route`] +/// chose for `request`. A stream the budget refuses runs on the planes +/// instead, from a fresh context, when they fit the budget (see +/// `route::planes_fit`); the planes write every row, including rows the +/// stream wrote before it was refused. +pub(crate) fn decode_full_image_on_route<'a>( + image: &Image<'a>, + native_context: &mut j2k_native::DecoderContext<'a>, + out: &mut [u8], + stride: usize, + request: RouteRequest, + route: &FullImageRoute, +) -> Result<(), J2kError> { + let fmt = request.fmt; + match decode_full_image_into_with_native_context(image, native_context, out, stride, fmt, route) + { + Err(J2kError::ResourceLimit { .. }) + if route.streams() && route::planes_fit(image, request) => + { + let parallelism = native_context.cpu_decode_parallelism(); + *native_context = j2k_native::DecoderContext::default(); + native_context.set_cpu_decode_parallelism(parallelism); + let planes = FullImageRoute::Planes; + decode_full_image_into_with_native_context( + image, + native_context, + out, + stride, + fmt, + &planes, + ) + } + decoded => decoded, + } +} + pub(crate) fn decode_image_into_with_native_context<'a>( image: &Image<'a>, native_context: &mut j2k_native::DecoderContext<'a>, @@ -140,18 +217,26 @@ pub(crate) fn decode_image_region_into_with_native_context<'a>( ) -> Result<(), J2kError> { match fmt { PixelFormat::Rgb8 | PixelFormat::Rgba8 | PixelFormat::Gray8 => { - // A serial decode writes rows as they are finished instead of - // holding the region's component planes. A decode that fails - // may have written some of them. - let mut sink = U8RowSink::new(image.color_space(), image.has_alpha(), out, stride, fmt); - image - .decode_region_rows_for_integer_output_with_context( - (roi.x, roi.y, roi.w, roi.h), - native_context, - &mut sink, - ) - .map_err(J2kError::from_native_decode_error)?; - sink.finish() + decode_image_region_rows_into_with_native_context( + image, + native_context, + out, + stride, + fmt, + roi, + ) + } + PixelFormat::Rgb16 | PixelFormat::Rgba16 | PixelFormat::Gray16 + if region_rows_preferred_u16(image, fmt, roi) => + { + decode_image_region_rows_into_with_native_context( + image, + native_context, + out, + stride, + fmt, + roi, + ) } PixelFormat::Rgb16 | PixelFormat::Rgba16 | PixelFormat::Gray16 => { let raw = image @@ -173,6 +258,45 @@ pub(crate) fn decode_image_region_into_with_native_context<'a>( } } +/// Decode `roi` of `image` into `out` through a row sink, which writes +/// rows as they are finished instead of holding the region's component +/// planes: always at 8 bits, and at 16 bits when the rows carry the native +/// samples (see `rows_match_native_u16`), whose conversion the sink then +/// matches. A decode that fails may have written some rows. +pub(crate) fn decode_image_region_rows_into_with_native_context<'a>( + image: &Image<'a>, + native_context: &mut j2k_native::DecoderContext<'a>, + out: &mut [u8], + stride: usize, + fmt: PixelFormat, + roi: Rect, +) -> Result<(), J2kError> { + let roi = (roi.x, roi.y, roi.w, roi.h); + match fmt { + PixelFormat::Rgb8 | PixelFormat::Rgba8 | PixelFormat::Gray8 => { + let mut sink = U8RowSink::new(image.color_space(), image.has_alpha(), out, stride, fmt); + image + .decode_region_rows_for_integer_output_with_context(roi, native_context, &mut sink) + .map_err(J2kError::from_native_decode_error)?; + sink.finish() + } + PixelFormat::Rgb16 | PixelFormat::Rgba16 | PixelFormat::Gray16 + if rows_match_native_u16(image, fmt) => + { + let mut sink = + U16RowSink::new(image.color_space(), image.has_alpha(), out, stride, fmt); + image + .decode_region_rows_for_integer_output_with_context(roi, native_context, &mut sink) + .map_err(J2kError::from_native_decode_error)?; + sink.finish() + } + _ => Err(Unsupported { + what: "pixel format cannot stream its rows for this image", + } + .into()), + } +} + pub(crate) fn decode_prepared_image_region_into( decoder: &mut j2k_native::PreparedRegionDecoder<'_, '_, '_>, out: &mut [u8], diff --git a/crates/j2k/src/decode/output.rs b/crates/j2k/src/decode/output.rs index fae0c8b9d..00bdf0455 100644 --- a/crates/j2k/src/decode/output.rs +++ b/crates/j2k/src/decode/output.rs @@ -2,11 +2,126 @@ //! Pixel-layout conversion for decoded native output. +mod native; mod u16; mod u8; -pub(super) use u16::{write_components_u16_output, write_u16_output}; +use crate::backend::ColorSpace; +use crate::J2kError; +use j2k_core::PixelFormat; + +pub(crate) use native::{ + decode_native_components_with_native_context, native_components_route, + native_region_components_route, NativeRoute, +}; +use native::{planes_pack_floor, region_rows_preferred}; + +pub(super) use u16::{ + region_rows_preferred_u16, rows_match_native_u16, u16_rows_pack_floor, + write_components_u16_output, write_plane_rows_u16, write_u16_output, +}; pub(super) use u8::{ - can_decode_u8_directly, expand_rgb8_to_rgba8, write_components_u8_output, write_u8_output, - U8RowSink, + can_decode_u8_directly, expand_rgb8_to_rgba8, write_components_u8_output, write_plane_rows_u8, + write_u8_output, }; + +/// Shared destination and error plumbing for the 8- and 16-bit component +/// row sinks. Its pixel format selects the matching plane writer. The 16-bit +/// path matches [`write_u16_output`] through [`rows_match_native_u16`], and +/// the first write error prevents every later write. +pub(in crate::decode) struct ComponentRowOutput<'o> { + color_space: &'o ColorSpace, + has_alpha: bool, + out: &'o mut [u8], + stride: usize, + fmt: PixelFormat, + error: Option, +} + +impl<'o> ComponentRowOutput<'o> { + /// `out` must already hold the region at `stride` in `fmt`. + pub(in crate::decode) fn new( + color_space: &'o ColorSpace, + has_alpha: bool, + out: &'o mut [u8], + stride: usize, + fmt: PixelFormat, + ) -> Self { + Self { + color_space, + has_alpha, + out, + stride, + fmt, + error: None, + } + } + + pub(in crate::decode) fn finish(self) -> Result<(), J2kError> { + self.error.map_or(Ok(()), Err) + } +} + +pub(in crate::decode) type U8RowSink<'o> = ComponentRowOutput<'o>; +pub(in crate::decode) type U16RowSink<'o> = ComponentRowOutput<'o>; + +impl j2k_native::ComponentRowSink for ComponentRowOutput<'_> { + fn write_rows(&mut self, x: u32, y: u32, planes: &[j2k_native::ComponentPlane<'_>]) { + if self.error.is_some() { + return; + } + let dims = planes + .first() + .map_or((0, 0), j2k_native::ComponentPlane::dimensions); + let offset = (y as usize) + .checked_mul(self.stride) + .and_then(|row| row.checked_add((x as usize).checked_mul(self.fmt.bytes_per_pixel())?)); + let result = match offset.and_then(|offset| self.out.get_mut(offset..)) { + Some(out) => match self.fmt { + PixelFormat::Gray8 | PixelFormat::Rgb8 | PixelFormat::Rgba8 => u8::write_planes_u8( + self.color_space, + self.has_alpha, + planes, + dims, + out, + self.stride, + self.fmt, + ), + PixelFormat::Gray16 | PixelFormat::Rgb16 | PixelFormat::Rgba16 => { + u16::write_planes_u16( + self.color_space, + self.has_alpha, + planes, + dims, + out, + self.stride, + self.fmt, + ) + } + _ => Err(J2kError::internal_backend( + "component row sink has unsupported pixel format", + )), + }, + None => Err(J2kError::DimensionOverflow { + width: x, + height: y, + }), + }; + self.error = result.err(); + } + + fn rgb8_region(&mut self, width: u32, height: u32) -> Option<&mut [u8]> { + // `write_rows` packs three 8-bit planes as `write_rgb_component_rows_u8` + // does, so a tight RGB8 buffer can take them packed by the decoder. + let row_bytes = (width as usize).checked_mul(3)?; + let tight_rgb8 = self.error.is_none() + && matches!(self.color_space, ColorSpace::RGB) + && !self.has_alpha + && self.fmt == PixelFormat::Rgb8 + && self.stride == row_bytes; + if !tight_rgb8 { + return None; + } + self.out.get_mut(..row_bytes.checked_mul(height as usize)?) + } +} diff --git a/crates/j2k/src/decode/output/native.rs b/crates/j2k/src/decode/output/native.rs new file mode 100644 index 000000000..fd0395fea --- /dev/null +++ b/crates/j2k/src/decode/output/native.rs @@ -0,0 +1,128 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! Plan-time choice of how a full-image decode into owned native-bit-depth +//! component planes runs, and the planes route's charge for packing them. + +use crate::backend::{ColorSpace, Image}; +use crate::J2kError; +use j2k_core::Rect; + +/// How a decode into owned native component planes runs. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum NativeRoute { + /// Component planes of the image or region, packed once decoded. + Planes, + /// A region decode that packs its rows into the planes as they are + /// finished and holds no component planes. + Rows, +} + +/// The bytes the planes route holds at least when it packs `image`'s output +/// from its component planes at `bytes_per_sample` native bytes per sample: +/// every `f32` plane and the packed samples, both at once, after the +/// decode's other buffers are released. A 16-bit decode's packed samples +/// are working memory, which the decode budget charges and refuses past it; +/// native component planes are the decode's output, which it does not +/// charge. Counts the components of `image`'s color space and alpha only, +/// so it is at most the real amount. +pub(in crate::decode) fn planes_pack_floor(image: &Image<'_>, bytes_per_sample: usize) -> usize { + region_pack_floor( + image, + Rect::full((image.width(), image.height())), + bytes_per_sample, + ) +} + +/// [`planes_pack_floor`] for the planes of `roi` of `image`. +fn region_pack_floor(image: &Image<'_>, roi: Rect, bytes_per_sample: usize) -> usize { + let components = usize::from(image.color_space().num_channels()) + .saturating_add(usize::from(image.has_alpha())); + (roi.w as usize) + .saturating_mul(roi.h as usize) + .saturating_mul(components) + .saturating_mul(bytes_per_sample.saturating_add(size_of::())) +} + +/// How a full-image decode of `image` into owned native component planes +/// runs when its planes may charge `planes_budget` bytes: as a region decode +/// of the whole image that packs its rows into the planes as they are +/// finished, when its rows carry the native samples (see +/// `Image::streamed_rows_native_precision`) and the planes route would be +/// refused by its decode (see `Image::full_decode_planes_floor`) or would +/// hold more than the budget at once with its packed output (see +/// [`planes_pack_floor`]), which the rows route never holds. +/// +/// A planning error keeps the planes route, which meets the same error and +/// reports it. +pub(crate) fn native_components_route(image: &Image<'_>, planes_budget: usize) -> NativeRoute { + let Some(precision) = image.streamed_rows_native_precision() else { + return NativeRoute::Planes; + }; + let pack_floor = planes_pack_floor(image, usize::from(precision).div_ceil(8)); + match image.full_decode_planes_floor(planes_budget) { + Ok(Some(floor)) if floor.max(pack_floor) > planes_budget => NativeRoute::Rows, + _ => NativeRoute::Planes, + } +} + +/// Whether a region decode of `roi` of `image`, whose rows carry its native +/// samples of `bytes_per_sample` bytes, streams them rather than packing +/// region-sized planes: always for an RGB image, whose streamed rows finish +/// first; for others when the planes and their packed samples (see +/// [`planes_pack_floor`]) would pass the native decode budget. One component +/// gives the streamed rows' Tier-1 too little work per row on a large pool, +/// so a gray region within the budget keeps the planes. +pub(in crate::decode) fn region_rows_preferred( + image: &Image<'_>, + roi: Rect, + bytes_per_sample: usize, +) -> bool { + matches!(image.color_space(), ColorSpace::RGB) + || region_pack_floor(image, roi, bytes_per_sample) > j2k_native::DEFAULT_MAX_DECODE_BYTES +} + +/// How a region decode of `roi` of `image` into owned native component +/// planes runs: as one that packs its rows into the planes as they are +/// finished, so that it holds no region-sized `f32` planes, when its rows +/// carry the native samples, the region decode streams them (see +/// `Image::region_decode_streams_rows`, which the full-image floor checks +/// too) and [`region_rows_preferred`]. An image whose region decode would +/// fall back to component planes keeps the planes route, which decodes and +/// charges them the same way. +pub(crate) fn native_region_components_route(image: &Image<'_>, roi: Rect) -> NativeRoute { + match image.streamed_rows_native_precision() { + Some(precision) + if image.region_decode_streams_rows() + && region_rows_preferred(image, roi, usize::from(precision).div_ceil(8)) => + { + NativeRoute::Rows + } + _ => NativeRoute::Planes, + } +} + +/// Decode `roi` of `image`, or the whole image, into owned native component +/// planes on `route`, which [`native_components_route`] or +/// [`native_region_components_route`] chose for it. +pub(crate) fn decode_native_components_with_native_context<'a>( + image: &Image<'a>, + native_context: &mut j2k_native::DecoderContext<'a>, + roi: Option, + route: NativeRoute, +) -> Result { + match (roi, route) { + (None, NativeRoute::Planes) => image.decode_native_components_with_context(native_context), + (Some(roi), NativeRoute::Planes) => image.decode_native_region_components_with_context( + (roi.x, roi.y, roi.w, roi.h), + native_context, + ), + (roi, NativeRoute::Rows) => { + let roi = roi.unwrap_or_else(|| Rect::full((image.width(), image.height()))); + image.decode_native_region_components_rows_with_context( + (roi.x, roi.y, roi.w, roi.h), + native_context, + ) + } + } + .map_err(J2kError::from_native_decode_error) +} diff --git a/crates/j2k/src/decode/output/u16.rs b/crates/j2k/src/decode/output/u16.rs index f76306fe5..cfe580d9c 100644 --- a/crates/j2k/src/decode/output/u16.rs +++ b/crates/j2k/src/decode/output/u16.rs @@ -2,8 +2,9 @@ //! Sixteen-bit channel-layout conversion for native decoded samples. -use crate::backend::{ColorSpace, DecodedComponents as NativeDecodedComponents, RawBitmap}; +use crate::backend::{ColorSpace, DecodedComponents as NativeDecodedComponents, Image, RawBitmap}; use crate::J2kError; +use core::ops::Range; use j2k_core::{PixelFormat, Unsupported}; use super::u8::{component_sample_count, output_rows, validate_component_planes}; @@ -14,36 +15,116 @@ pub(in crate::decode) fn write_components_u16_output( stride: usize, fmt: PixelFormat, ) -> Result<(), J2kError> { - let dims = components.dimensions(); - let expected_samples = component_sample_count(dims)?; - let width = dims.0 as usize; - let height = dims.1 as usize; - let planes = components.planes(); - match ( + write_planes_u16( components.color_space(), components.has_alpha(), - planes.len(), + components.planes(), + components.dimensions(), + out, + stride, fmt, - ) { + ) +} + +/// Whether a region decode of `image` to `fmt` may stream its rows through +/// [`super::U16RowSink`]: they carry the samples `decode_native_region` packs (see +/// `Image::streamed_rows_native_precision`: every component has one +/// precision of at most 24 bits), which `U16Conversion` converts to the +/// bytes [`write_u16_output`] writes from those packed samples, and +/// `fmt` is a layout the image's color space maps to. Images that fail any +/// of these keep the native-bit-depth planes, with their output or error. +pub(in crate::decode) fn rows_match_native_u16(image: &Image<'_>, fmt: PixelFormat) -> bool { + image.streamed_rows_native_precision().is_some() + && matches!( + (image.color_space(), image.has_alpha(), fmt), + (ColorSpace::Gray, false, PixelFormat::Gray16) + | (ColorSpace::RGB, _, PixelFormat::Rgb16 | PixelFormat::Rgba16) + ) +} + +/// The bytes the planes route charges at least when it packs `image`'s +/// native samples for a 16-bit `fmt` (see [`super::planes_pack_floor`]), +/// when a region decode of `image` may stream its rows to `fmt` instead +/// (see [`rows_match_native_u16`]). +pub(in crate::decode) fn u16_rows_pack_floor(image: &Image<'_>, fmt: PixelFormat) -> Option { + let precision = image.streamed_rows_native_precision()?; + rows_match_native_u16(image, fmt) + .then(|| super::planes_pack_floor(image, usize::from(precision).div_ceil(8))) +} + +/// Whether a 16-bit region decode of `roi` of `image` to `fmt` streams its +/// rows (see [`rows_match_native_u16`] and [`super::region_rows_preferred`]) +/// rather than packing the region's native samples. +pub(in crate::decode) fn region_rows_preferred_u16( + image: &Image<'_>, + fmt: PixelFormat, + roi: j2k_core::Rect, +) -> bool { + image + .streamed_rows_native_precision() + .is_some_and(|precision| { + rows_match_native_u16(image, fmt) + && super::region_rows_preferred(image, roi, usize::from(precision).div_ceil(8)) + }) +} + +/// Write `planes`, each `dims` in size, to `out` from its first byte. +pub(super) fn write_planes_u16( + color_space: &ColorSpace, + has_alpha: bool, + planes: &[j2k_native::ComponentPlane<'_>], + dims: (u32, u32), + out: &mut [u8], + stride: usize, + fmt: PixelFormat, +) -> Result<(), J2kError> { + let rows = 0..dims.1 as usize; + write_plane_rows_u16(color_space, has_alpha, planes, dims, rows, out, stride, fmt) +} + +/// Write rows `rows` of `planes`, each `dims` in size, to `out` from its +/// first byte, as [`write_planes_u16`] writes those rows of whole planes. +#[expect( + clippy::too_many_arguments, + reason = "the row-range writer keeps the planes' layout, the rows and the output layout explicit" +)] +pub(in crate::decode) fn write_plane_rows_u16( + color_space: &ColorSpace, + has_alpha: bool, + planes: &[j2k_native::ComponentPlane<'_>], + dims: (u32, u32), + rows: Range, + out: &mut [u8], + stride: usize, + fmt: PixelFormat, +) -> Result<(), J2kError> { + let expected_samples = component_sample_count(dims)?; + let width = dims.0 as usize; + if rows.start > rows.end || rows.end > dims.1 as usize { + return Err(J2kError::internal_backend( + "component rows outside their planes", + )); + } + match (color_space, has_alpha, planes.len(), fmt) { (ColorSpace::Gray, false, 1, PixelFormat::Gray16) => { validate_component_planes(&planes[..1], expected_samples)?; - write_component_rows_u16(&planes[0], out, stride, width, height); + write_component_rows_u16(&planes[0], out, stride, width, rows); Ok(()) } (ColorSpace::RGB, false, 3, PixelFormat::Rgb16) | (ColorSpace::RGB, true, 4, PixelFormat::Rgb16) => { validate_component_planes(&planes[..3], expected_samples)?; - write_rgb_component_rows_u16(planes, out, stride, width, height); + write_rgb_component_rows_u16(planes, out, stride, width, rows); Ok(()) } (ColorSpace::RGB, false, 3, PixelFormat::Rgba16) => { validate_component_planes(&planes[..3], expected_samples)?; - write_rgba_component_rows_u16(planes, out, stride, width, height, true); + write_rgba_component_rows_u16(planes, out, stride, width, rows, true); Ok(()) } (ColorSpace::RGB, true, 4, PixelFormat::Rgba16) => { validate_component_planes(&planes[..4], expected_samples)?; - write_rgba_component_rows_u16(planes, out, stride, width, height, false); + write_rgba_component_rows_u16(planes, out, stride, width, rows, false); Ok(()) } _ => Err(Unsupported { @@ -53,20 +134,23 @@ pub(in crate::decode) fn write_components_u16_output( } } +/// Rows `rows` of `plane`, `width` samples each, written from the first row +/// of `out` on. fn write_component_rows_u16( plane: &j2k_native::ComponentPlane<'_>, out: &mut [u8], stride: usize, width: usize, - height: usize, + rows: Range, ) { - for y in 0..height { - let src = &plane.samples()[y * width..(y + 1) * width]; - let dst = &mut out[y * stride..y * stride + width * 2]; - for (sample, destination) in src.iter().zip(dst.as_chunks_mut::<2>().0) { - *destination = - component_sample_as_u16(*sample, plane.bit_depth(), plane.signed()).to_le_bytes(); - } + let conversion = U16Conversion::new(plane.bit_depth(), plane.signed()); + for (index, y) in rows.enumerate() { + conversion.visit(ChannelWriter { + src: &plane.samples()[y * width..(y + 1) * width], + dst: &mut out[index * stride..index * stride + width * 2], + channels: 1, + channel: 0, + }); } } @@ -75,9 +159,9 @@ fn write_rgb_component_rows_u16( out: &mut [u8], stride: usize, width: usize, - height: usize, + rows: Range, ) { - write_color_component_rows_u16(planes, out, stride, width, height, 3, false); + write_color_component_rows_u16(planes, out, stride, width, rows, 3, false); } fn write_rgba_component_rows_u16( @@ -85,10 +169,10 @@ fn write_rgba_component_rows_u16( out: &mut [u8], stride: usize, width: usize, - height: usize, + rows: Range, synthesize_alpha: bool, ) { - write_color_component_rows_u16(planes, out, stride, width, height, 4, synthesize_alpha); + write_color_component_rows_u16(planes, out, stride, width, rows, 4, synthesize_alpha); } fn write_color_component_rows_u16( @@ -96,69 +180,247 @@ fn write_color_component_rows_u16( out: &mut [u8], stride: usize, width: usize, - height: usize, + rows: Range, output_channels: usize, synthesize_alpha: bool, ) { - for y in 0..height { - let row = y * width; - let destination = &mut out[y * stride..y * stride + width * output_channels * 2]; - for x in 0..width { - let pixel = &mut destination[x * output_channels * 2..(x + 1) * output_channels * 2]; - for channel in 0..3 { - let sample = component_sample_as_u16( - planes[channel].samples()[row + x], - planes[channel].bit_depth(), - planes[channel].signed(), - ); - pixel[channel * 2..channel * 2 + 2].copy_from_slice(&sample.to_le_bytes()); + let color = [0, 1, 2] + .map(|channel| U16Conversion::new(planes[channel].bit_depth(), planes[channel].signed())); + let fused = color[1] == color[0] && color[2] == color[0]; + let alpha = (output_channels == 4 && !synthesize_alpha) + .then(|| U16Conversion::new(planes[3].bit_depth(), planes[3].signed())); + let opaque = opaque_alpha_u16( + usize::from(planes[0].bit_depth()).div_ceil(8), + planes[0].bit_depth(), + ) + .to_le_bytes(); + for (index, y) in rows.enumerate() { + let row = y * width..(y + 1) * width; + let destination = &mut out[index * stride..index * stride + width * output_channels * 2]; + if fused { + color[0].visit(PixelWriter { + src: [0, 1, 2].map(|channel| &planes[channel].samples()[row.clone()]), + dst: destination, + channels: output_channels, + }); + } else { + for (channel, conversion) in color.iter().enumerate() { + conversion.visit(ChannelWriter { + src: &planes[channel].samples()[row.clone()], + dst: destination, + channels: output_channels, + channel, + }); } - if output_channels == 4 { - let alpha = if synthesize_alpha { - opaque_alpha_u16( - usize::from(planes[0].bit_depth()).div_ceil(8), - planes[0].bit_depth(), - ) - } else { - component_sample_as_u16( - planes[3].samples()[row + x], - planes[3].bit_depth(), - planes[3].signed(), - ) - }; - pixel[6..8].copy_from_slice(&alpha.to_le_bytes()); + } + match &alpha { + Some(conversion) => conversion.visit(ChannelWriter { + src: &planes[3].samples()[row], + dst: destination, + channels: 4, + channel: 3, + }), + None if output_channels == 4 => { + for pixel in destination.as_chunks_mut::<8>().0 { + pixel[6..8].copy_from_slice(&opaque); + } + } + None => {} + } + } +} + +/// How [`U16Conversion`] turns a rounded sample into its output. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum U16Kind { + /// Unsigned 8 bits: the byte times 257, which is its widened value. + Byte, + /// Fewer unsigned bits: the byte's widened value from the table. + ByteTable, + /// Signed, at most 8 bits: the two's-complement byte's widened value. + SignedByteTable, + /// Unsigned, 9 bits or more: the sample clamped to 16 bits. + Unsigned, + /// Signed, 9 bits or more: the two's complement of the sample clamped + /// to 16 bits. + Signed, +} + +/// Converts the decoded samples of one component of `bit_depth` bits to +/// 16-bit output: rounded, clamped to the component's signed or unsigned +/// range, and to 16 bits past 16, and, for at most 8 bits, clamped to a +/// byte and widened across the 16-bit range as [`write_u16_output`] widens +/// native 8-bit samples. The choices are made once per plane. +#[derive(Debug, PartialEq)] +struct U16Conversion { + kind: U16Kind, + /// The component's range, or the byte's for at most 8 unsigned bits. + bounds: (f32, f32), + /// The widened value of each byte for the table kinds. + widened: [u16; 256], +} + +impl U16Conversion { + fn new(bit_depth: u8, signed: bool) -> Self { + if bit_depth <= 8 { + let (kind, bounds) = if signed { + let magnitude = 1_i16 << bit_depth.saturating_sub(1); + ( + U16Kind::SignedByteTable, + (f32::from(-magnitude), f32::from(magnitude - 1)), + ) + } else if bit_depth == 8 { + (U16Kind::Byte, (0.0, f32::from(u8::MAX))) + } else { + (U16Kind::ByteTable, (0.0, f32::from(u8::MAX))) + }; + return Self { + kind, + bounds, + widened: widened_u8_samples(bit_depth), + }; + } + let (kind, bounds) = if signed { + let magnitude = 1_i32 << (bit_depth.min(16) - 1); + ( + U16Kind::Signed, + ( + f32::from(i16::try_from(-magnitude).unwrap_or(i16::MIN)), + f32::from(i16::try_from(magnitude - 1).unwrap_or(i16::MAX)), + ), + ) + } else { + let max = (1_u32 << bit_depth.min(16)) - 1; + ( + U16Kind::Unsigned, + (0.0, f32::from(u16::try_from(max).unwrap_or(u16::MAX))), + ) + }; + Self { + kind, + bounds, + widened: [0; 256], + } + } + + /// Hand `writer` this conversion as a closure of its kind. + #[expect( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + clippy::inline_always, + reason = "decoded samples are rounded and clamped to the component's declared integer representation; the chosen conversion is inlined into the writer so each kind compiles to its own fixed-size pixel loop" + )] + #[inline(always)] + fn visit(&self, writer: impl U16Writer) { + let (min, max) = self.bounds; + let widened = &self.widened; + match self.kind { + U16Kind::Byte => { + writer.write(|sample: f32| u16::from(sample.round().clamp(min, max) as u8) * 257); } + U16Kind::ByteTable => writer + .write(|sample: f32| widened[usize::from(sample.round().clamp(min, max) as u8)]), + U16Kind::SignedByteTable => writer.write(|sample: f32| { + widened[usize::from((sample.round().clamp(min, max) as i8) as u8)] + }), + U16Kind::Unsigned => writer.write(|sample: f32| sample.round().clamp(min, max) as u16), + U16Kind::Signed => { + writer.write(|sample: f32| (sample.round().clamp(min, max) as i16) as u16); + } + } + } +} + +/// Writes rows of samples with a [`U16Conversion`]'s closure. +trait U16Writer { + fn write(self, convert: impl Fn(f32) -> u16); +} + +/// One component's samples into one channel of interleaved pixels. +struct ChannelWriter<'s, 'd> { + src: &'s [f32], + dst: &'d mut [u8], + channels: usize, + channel: usize, +} + +impl U16Writer for ChannelWriter<'_, '_> { + #[expect( + clippy::inline_always, + reason = "the conversion closure is inlined into the fixed-size pixel loop it runs in" + )] + #[inline(always)] + fn write(self, convert: impl Fn(f32) -> u16) { + let Self { + src, + dst, + channels, + channel, + } = self; + // Whole pixels of a constant size, which the compiler vectorizes. + match channels { + 1 => write_channel::<2>(src, dst, channel, convert), + 3 => write_channel::<6>(src, dst, channel, convert), + _ => write_channel::<8>(src, dst, channel, convert), } } } #[expect( - clippy::cast_possible_truncation, - clippy::cast_sign_loss, - reason = "decoded samples are rounded and clamped to the component's declared integer representation" + clippy::inline_always, + reason = "the conversion closure is inlined into the fixed-size pixel loop it runs in" )] -fn component_sample_as_u16(sample: f32, bit_depth: u8, signed: bool) -> u16 { - let rounded = sample.round(); - if signed { - if bit_depth <= 8 { - let magnitude_bits = u32::from(bit_depth.saturating_sub(1)); - let min = -(1_i16 << magnitude_bits); - let max = (1_i16 << magnitude_bits) - 1; - return widen_u8_sample_to_u16( - (rounded.clamp(f32::from(min), f32::from(max)) as i8) as u8, - bit_depth, - ); +#[inline(always)] +fn write_channel( + src: &[f32], + dst: &mut [u8], + channel: usize, + convert: impl Fn(f32) -> u16, +) { + for (pixel, &sample) in dst.as_chunks_mut::().0.iter_mut().zip(src) { + pixel[channel * 2..channel * 2 + 2].copy_from_slice(&convert(sample).to_le_bytes()); + } +} + +/// Three components' samples into the first three channels of interleaved +/// pixels, one pixel at a time. +struct PixelWriter<'s, 'd> { + src: [&'s [f32]; 3], + dst: &'d mut [u8], + channels: usize, +} + +impl U16Writer for PixelWriter<'_, '_> { + #[expect( + clippy::inline_always, + reason = "the conversion closure is inlined into the fixed-size pixel loop it runs in" + )] + #[inline(always)] + fn write(self, convert: impl Fn(f32) -> u16) { + let Self { src, dst, channels } = self; + match channels { + 3 => write_pixels::<6>(src, dst, convert), + _ => write_pixels::<8>(src, dst, convert), } - let magnitude_bits = u32::from(bit_depth.min(16).saturating_sub(1)); - let min = i16::try_from(-(1_i32 << magnitude_bits)).unwrap_or(i16::MIN); - let max = i16::try_from((1_i32 << magnitude_bits) - 1).unwrap_or(i16::MAX); - return (rounded.clamp(f32::from(min), f32::from(max)) as i16) as u16; } - if bit_depth <= 8 { - return widen_u8_sample_to_u16(rounded.clamp(0.0, f32::from(u8::MAX)) as u8, bit_depth); +} + +#[expect( + clippy::inline_always, + reason = "the conversion closure is inlined into the fixed-size pixel loop it runs in" +)] +#[inline(always)] +fn write_pixels( + [red, green, blue]: [&[f32]; 3], + dst: &mut [u8], + convert: impl Fn(f32) -> u16, +) { + let samples = red.iter().zip(green).zip(blue); + for (pixel, ((&red, &green), &blue)) in dst.as_chunks_mut::().0.iter_mut().zip(samples) { + pixel[..2].copy_from_slice(&convert(red).to_le_bytes()); + pixel[2..4].copy_from_slice(&convert(green).to_le_bytes()); + pixel[4..6].copy_from_slice(&convert(blue).to_le_bytes()); } - let max = u16::try_from((1_u32 << u32::from(bit_depth.min(16))) - 1).unwrap_or(u16::MAX); - rounded.clamp(0.0, f32::from(max)) as u16 } pub(in crate::decode) fn write_u16_output( @@ -171,13 +433,14 @@ pub(in crate::decode) fn write_u16_output( ) -> Result<(), J2kError> { let width = raw.width as usize; let height = raw.height as usize; + let signed = channel_signedness(raw); match (color_space, has_alpha, raw.num_components, fmt) { (ColorSpace::RGB, false, 3, PixelFormat::Rgb16) => { convert_or_copy_u16( &raw.data, raw.bytes_per_sample, raw.bit_depth, - 3, + &signed[..3], out, stride, (width, height), @@ -189,6 +452,7 @@ pub(in crate::decode) fn write_u16_output( src: &raw.data, bytes_per_sample: raw.bytes_per_sample, bit_depth: raw.bit_depth, + signed, source_channels: 4, layout: U16ChannelLayout::Drop, out, @@ -202,6 +466,7 @@ pub(in crate::decode) fn write_u16_output( src: &raw.data, bytes_per_sample: raw.bytes_per_sample, bit_depth: raw.bit_depth, + signed, source_channels: 3, layout: U16ChannelLayout::Synthesize, out, @@ -215,6 +480,7 @@ pub(in crate::decode) fn write_u16_output( src: &raw.data, bytes_per_sample: raw.bytes_per_sample, bit_depth: raw.bit_depth, + signed, source_channels: 4, layout: U16ChannelLayout::Preserve, out, @@ -228,7 +494,7 @@ pub(in crate::decode) fn write_u16_output( &raw.data, raw.bytes_per_sample, raw.bit_depth, - 1, + &signed[..1], out, stride, (width, height), @@ -253,6 +519,8 @@ struct U16ChannelRows<'src, 'out> { src: &'src [u8], bytes_per_sample: u8, bit_depth: u8, + /// Whether each source channel is signed. + signed: [bool; 4], source_channels: usize, layout: U16ChannelLayout, out: &'out mut [u8], @@ -265,6 +533,7 @@ fn write_u16_channel_rows(job: U16ChannelRows<'_, '_>) { src, bytes_per_sample, bit_depth, + signed, source_channels, layout, out, @@ -280,6 +549,7 @@ fn write_u16_channel_rows(job: U16ChannelRows<'_, '_>) { let src_row_bytes = width * source_channels * bytes_per_sample; let dst_row_bytes = width * dst_channels * 2; let alpha = opaque_alpha_u16(bytes_per_sample, bit_depth); + let widened = widened_u8_samples(bit_depth); let dst_rows = output_rows(out, stride, dst_row_bytes, height); for (src_row, dst_row) in src.chunks_exact(src_row_bytes).zip(dst_rows) { @@ -287,7 +557,13 @@ fn write_u16_channel_rows(job: U16ChannelRows<'_, '_>) { let src_pixel = &src_row[x * source_channels * bytes_per_sample..]; let dst_pixel = &mut dst_row[x * dst_channels * 2..(x + 1) * dst_channels * 2]; for channel in 0..3 { - let sample = output_u16_sample(src_pixel, channel, bytes_per_sample, bit_depth); + let sample = output_u16_sample( + src_pixel, + channel, + bytes_per_sample, + signed[channel], + &widened, + ); dst_pixel[channel * 2..channel * 2 + 2].copy_from_slice(&sample.to_le_bytes()); } match layout { @@ -296,7 +572,8 @@ fn write_u16_channel_rows(job: U16ChannelRows<'_, '_>) { dst_pixel[6..8].copy_from_slice(&alpha.to_le_bytes()); } U16ChannelLayout::Preserve => { - let sample = output_u16_sample(src_pixel, 3, bytes_per_sample, bit_depth); + let sample = + output_u16_sample(src_pixel, 3, bytes_per_sample, signed[3], &widened); dst_pixel[6..8].copy_from_slice(&sample.to_le_bytes()); } } @@ -312,17 +589,49 @@ fn opaque_alpha_u16(bytes_per_sample: usize, bit_depth: u8) -> u16 { } } +/// Whether each channel of `raw`'s pixels is signed (the first four). +fn channel_signedness(raw: &RawBitmap) -> [bool; 4] { + core::array::from_fn(|channel| { + raw.component_signed + .get(channel) + .copied() + .unwrap_or(raw.signed) + }) +} + +/// `widened` is [`widened_u8_samples`] of the samples' bit depth. fn output_u16_sample( src_pixel: &[u8], channel: usize, bytes_per_sample: usize, - bit_depth: u8, + signed: bool, + widened: &[u16; 256], ) -> u16 { let offset = channel * bytes_per_sample; - if bytes_per_sample == 2 { - return u16::from_le_bytes([src_pixel[offset], src_pixel[offset + 1]]); + match bytes_per_sample { + 1 => widened[usize::from(src_pixel[offset])], + 2 => u16::from_le_bytes([src_pixel[offset], src_pixel[offset + 1]]), + _ => clamped_wide_sample(&src_pixel[offset..offset + bytes_per_sample], signed), } - widen_u8_sample_to_u16(src_pixel[offset], bit_depth) +} + +/// The 16-bit output of a native sample of more than two little-endian +/// bytes, from an image deeper than 16 bits: its value clamped to the +/// 16-bit range, as two's complement when `signed`. This is what +/// [`U16Conversion`] writes for the sample's decoded value, and what scaled +/// decodes, which convert component planes, always wrote. +fn clamped_wide_sample(bytes: &[u8], signed: bool) -> u16 { + let len = bytes.len().min(8); + let mut le = [0_u8; 8]; + le[..len].copy_from_slice(&bytes[..len]); + if signed { + // Sign-extend from the sample's top byte. + let unused = 64 - 8 * len; + let value = (i64::from_le_bytes(le) << unused) >> unused; + let clamped = value.clamp(i64::from(i16::MIN), i64::from(i16::MAX)); + return u16::from_le_bytes(i16::try_from(clamped).unwrap_or(i16::MAX).to_le_bytes()); + } + u16::try_from(u64::from_le_bytes(le)).unwrap_or(u16::MAX) } fn widen_u8_sample_to_u16(sample: u8, bit_depth: u8) -> u16 { @@ -331,33 +640,218 @@ fn widen_u8_sample_to_u16(sample: u8, bit_depth: u8) -> u16 { .unwrap_or(u16::MAX) } +/// [`widen_u8_sample_to_u16`] of every byte at `bit_depth`, so that a +/// conversion looks each sample up instead of dividing. +fn widened_u8_samples(bit_depth: u8) -> [u16; 256] { + core::array::from_fn(|byte| { + widen_u8_sample_to_u16(u8::try_from(byte).unwrap_or(u8::MAX), bit_depth) + }) +} + +/// `signed` holds whether each of the pixels' channels is signed. fn convert_or_copy_u16( src: &[u8], bytes_per_sample: u8, bit_depth: u8, - channels: usize, + signed: &[bool], out: &mut [u8], stride: usize, dims: (usize, usize), ) { let (width, height) = dims; + let channels = signed.len(); let dst_row_bytes = width * channels * 2; let src_row_bytes = width * channels * usize::from(bytes_per_sample); + let widened = widened_u8_samples(bit_depth); let dst_rows = output_rows(out, stride, dst_row_bytes, height); for (src_row, dst_row) in src.chunks_exact(src_row_bytes).zip(dst_rows) { if bytes_per_sample == 2 { dst_row.copy_from_slice(src_row); continue; } + if bytes_per_sample > 2 { + let samples = src_row.chunks_exact(usize::from(bytes_per_sample)); + for (index, (sample, dst_sample)) in + samples.zip(dst_row.as_chunks_mut::<2>().0).enumerate() + { + *dst_sample = clamped_wide_sample(sample, signed[index % channels]).to_le_bytes(); + } + continue; + } for (sample, dst_sample) in src_row.iter().zip(dst_row.as_chunks_mut::<2>().0) { - *dst_sample = widen_u8_sample_to_u16(*sample, bit_depth).to_le_bytes(); + *dst_sample = widened[usize::from(*sample)].to_le_bytes(); } } } #[cfg(test)] mod tests { - use super::{opaque_alpha_u16, widen_u8_sample_to_u16}; + use super::{ + clamped_wide_sample, convert_or_copy_u16, opaque_alpha_u16, widen_u8_sample_to_u16, + ChannelWriter, PixelWriter, U16Conversion, + }; + + /// The per-sample conversion the component-plane writers used before + /// they converted a plane at a time. + #[expect( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + reason = "decoded samples are rounded and clamped to the component's declared integer representation" + )] + fn component_sample_as_u16(sample: f32, bit_depth: u8, signed: bool) -> u16 { + let rounded = sample.round(); + if signed { + if bit_depth <= 8 { + let magnitude_bits = u32::from(bit_depth.saturating_sub(1)); + let min = -(1_i16 << magnitude_bits); + let max = (1_i16 << magnitude_bits) - 1; + return widen_u8_sample_to_u16( + (rounded.clamp(f32::from(min), f32::from(max)) as i8) as u8, + bit_depth, + ); + } + let magnitude_bits = u32::from(bit_depth.min(16).saturating_sub(1)); + let min = i16::try_from(-(1_i32 << magnitude_bits)).unwrap_or(i16::MIN); + let max = i16::try_from((1_i32 << magnitude_bits) - 1).unwrap_or(i16::MAX); + return (rounded.clamp(f32::from(min), f32::from(max)) as i16) as u16; + } + if bit_depth <= 8 { + return widen_u8_sample_to_u16(rounded.clamp(0.0, f32::from(u8::MAX)) as u8, bit_depth); + } + let max = u16::try_from((1_u32 << u32::from(bit_depth.min(16))) - 1).unwrap_or(u16::MAX); + rounded.clamp(0.0, f32::from(max)) as u16 + } + + #[test] + #[expect(clippy::cast_precision_loss, reason = "test values fit f32 exactly")] + fn plane_conversion_matches_the_sample_conversion() { + let mut samples = vec![ + f32::NAN, + f32::INFINITY, + f32::NEG_INFINITY, + -0.0, + 1.0e9, + -1.0e9, + ]; + for value in -70_000_i32..70_000 { + for offset in [-0.5, 0.0, 0.25, 0.5] { + samples.push(value as f32 + offset); + } + } + for bit_depth in 1..=16 { + for signed in [false, true] { + let conversion = U16Conversion::new(bit_depth, signed); + let mut out = vec![0; samples.len() * 2]; + conversion.visit(ChannelWriter { + src: &samples, + dst: &mut out, + channels: 1, + channel: 0, + }); + let mut pixels = vec![0; samples.len() * 8]; + conversion.visit(PixelWriter { + src: [&samples; 3], + dst: &mut pixels, + channels: 4, + }); + for (pixel, sample) in pixels.as_chunks::<8>().0.iter().zip(out.as_chunks::<2>().0) + { + assert_eq!( + pixel[..6], + [*sample; 3].concat(), + "{bit_depth} bits, signed {signed}" + ); + } + for (sample, bytes) in samples.iter().zip(out.as_chunks::<2>().0) { + assert_eq!( + u16::from_le_bytes(*bytes), + component_sample_as_u16(*sample, bit_depth, signed), + "{sample} at {bit_depth} bits, signed {signed}" + ); + } + } + } + } + + #[test] + #[expect(clippy::cast_precision_loss, reason = "test values fit f32 exactly")] + fn deep_native_samples_convert_as_their_decoded_values_do() { + for bit_depth in 17..=24_u8 { + for signed in [false, true] { + let (min, max) = if signed { + (-(1_i64 << (bit_depth - 1)), (1_i64 << (bit_depth - 1)) - 1) + } else { + (0, (1_i64 << bit_depth) - 1) + }; + let edges = [-32_769, -32_768, -1, 0, 1, 32_767, 32_768, 65_535, 65_536]; + let values: Vec = (min..=max) + .step_by(997) + .chain([min, max]) + .chain( + edges + .into_iter() + .filter(|value| (min..=max).contains(value)), + ) + .collect(); + let samples: Vec = values.iter().map(|value| *value as f32).collect(); + let mut expected = vec![0; samples.len() * 2]; + U16Conversion::new(bit_depth, signed).visit(ChannelWriter { + src: &samples, + dst: &mut expected, + channels: 1, + channel: 0, + }); + // Native samples as `decode_native` packs them: three + // little-endian bytes, two's complement when signed. + let native: Vec = values + .iter() + .flat_map(|value| { + let bytes = value.to_le_bytes(); + [bytes[0], bytes[1], bytes[2]] + }) + .collect(); + let stride = samples.len() * 2; + let mut out = vec![0; stride]; + convert_or_copy_u16( + &native, + 3, + bit_depth, + &[signed], + &mut out, + stride, + (samples.len(), 1), + ); + assert_eq!(out, expected, "{bit_depth} bits, signed {signed}"); + for (sample, bytes) in samples.iter().zip(out.as_chunks::<2>().0) { + assert_eq!( + u16::from_le_bytes(*bytes), + component_sample_as_u16(*sample, bit_depth, signed), + "{sample} at {bit_depth} bits, signed {signed}" + ); + } + } + } + } + + #[test] + fn samples_wider_than_three_bytes_clamp_to_sixteen_bits() { + assert_eq!(clamped_wide_sample(&(-5_i32).to_le_bytes(), true), 0xfffb); + assert_eq!( + clamped_wide_sample(&(-70_000_i32).to_le_bytes(), true), + 0x8000 + ); + assert_eq!(clamped_wide_sample(&70_000_i32.to_le_bytes(), true), 0x7fff); + assert_eq!(clamped_wide_sample(&1234_u32.to_le_bytes(), false), 1234); + assert_eq!( + clamped_wide_sample(&(1_u64 << 36).to_le_bytes()[..5], false), + u16::MAX + ); + assert_eq!(clamped_wide_sample(&i64::MIN.to_le_bytes(), true), 0x8000); + assert_eq!( + clamped_wide_sample(&u64::MAX.to_le_bytes(), false), + u16::MAX + ); + } #[test] fn eight_bit_samples_widen_across_the_complete_u16_domain() { @@ -370,5 +864,6 @@ mod tests { assert_eq!(opaque_alpha_u16(1, 8), u16::MAX); assert_eq!(opaque_alpha_u16(2, 12), 0x0fff); assert_eq!(opaque_alpha_u16(2, 16), u16::MAX); + assert_eq!(opaque_alpha_u16(3, 20), u16::MAX); } } diff --git a/crates/j2k/src/decode/output/u8.rs b/crates/j2k/src/decode/output/u8.rs index 0797d3aa0..6364d1b99 100644 --- a/crates/j2k/src/decode/output/u8.rs +++ b/crates/j2k/src/decode/output/u8.rs @@ -4,6 +4,7 @@ use crate::backend::{ColorSpace, DecodedComponents as NativeDecodedComponents}; use crate::J2kError; +use core::ops::Range; use j2k_core::{PixelFormat, Unsupported}; pub(in crate::decode) fn can_decode_u8_directly( @@ -40,7 +41,7 @@ pub(in crate::decode) fn write_components_u8_output( } /// Write `planes`, each `dims` in size, to `out` from its first byte. -fn write_planes_u8( +pub(super) fn write_planes_u8( color_space: &ColorSpace, has_alpha: bool, planes: &[j2k_native::ComponentPlane<'_>], @@ -48,30 +49,54 @@ fn write_planes_u8( out: &mut [u8], stride: usize, fmt: PixelFormat, +) -> Result<(), J2kError> { + let rows = 0..dims.1 as usize; + write_plane_rows_u8(color_space, has_alpha, planes, dims, rows, out, stride, fmt) +} + +/// Write rows `rows` of `planes`, each `dims` in size, to `out` from its +/// first byte, as [`write_planes_u8`] writes those rows of whole planes. +#[expect( + clippy::too_many_arguments, + reason = "the row-range writer keeps the planes' layout, the rows and the output layout explicit" +)] +pub(in crate::decode) fn write_plane_rows_u8( + color_space: &ColorSpace, + has_alpha: bool, + planes: &[j2k_native::ComponentPlane<'_>], + dims: (u32, u32), + rows: Range, + out: &mut [u8], + stride: usize, + fmt: PixelFormat, ) -> Result<(), J2kError> { let expected_samples = component_sample_count(dims)?; let width = dims.0 as usize; - let height = dims.1 as usize; + if rows.start > rows.end || rows.end > dims.1 as usize { + return Err(J2kError::internal_backend( + "component rows outside their planes", + )); + } match (color_space, has_alpha, planes.len(), fmt) { (ColorSpace::Gray, false, 1, PixelFormat::Gray8) => { validate_component_planes(&planes[..1], expected_samples)?; - write_component_rows_u8(&planes[0], out, stride, width, height); + write_component_rows_u8(&planes[0], out, stride, width, rows); Ok(()) } (ColorSpace::RGB, false, 3, PixelFormat::Rgb8) | (ColorSpace::RGB, true, 4, PixelFormat::Rgb8) => { validate_component_planes(&planes[..3], expected_samples)?; - write_rgb_component_rows_u8(planes, out, stride, width, height); + write_rgb_component_rows_u8(planes, out, stride, width, rows); Ok(()) } (ColorSpace::RGB, false, 3, PixelFormat::Rgba8) => { validate_component_planes(&planes[..3], expected_samples)?; - write_rgba_component_rows_u8(planes, out, stride, width, height, true); + write_rgba_component_rows_u8(planes, out, stride, width, rows, true); Ok(()) } (ColorSpace::RGB, true, 4, PixelFormat::Rgba8) => { validate_component_planes(&planes[..4], expected_samples)?; - write_rgba_component_rows_u8(planes, out, stride, width, height, false); + write_rgba_component_rows_u8(planes, out, stride, width, rows, false); Ok(()) } _ => Err(Unsupported { @@ -81,72 +106,6 @@ fn write_planes_u8( } } -/// Writes a region decode's rectangles of component rows to an 8-bit -/// output as [`write_components_u8_output`] writes whole planes. The first -/// error stops every later write and is returned by [`Self::finish`]. -pub(in crate::decode) struct U8RowSink<'o> { - color_space: &'o ColorSpace, - has_alpha: bool, - out: &'o mut [u8], - stride: usize, - fmt: PixelFormat, - error: Option, -} - -impl<'o> U8RowSink<'o> { - /// `out` must already hold the region at `stride` in `fmt`. - pub(in crate::decode) fn new( - color_space: &'o ColorSpace, - has_alpha: bool, - out: &'o mut [u8], - stride: usize, - fmt: PixelFormat, - ) -> Self { - Self { - color_space, - has_alpha, - out, - stride, - fmt, - error: None, - } - } - - pub(in crate::decode) fn finish(self) -> Result<(), J2kError> { - self.error.map_or(Ok(()), Err) - } -} - -impl j2k_native::ComponentRowSink for U8RowSink<'_> { - fn write_rows(&mut self, x: u32, y: u32, planes: &[j2k_native::ComponentPlane<'_>]) { - if self.error.is_some() { - return; - } - let dims = planes - .first() - .map_or((0, 0), j2k_native::ComponentPlane::dimensions); - let offset = (y as usize) - .checked_mul(self.stride) - .and_then(|row| row.checked_add((x as usize).checked_mul(self.fmt.bytes_per_pixel())?)); - let result = match offset.and_then(|offset| self.out.get_mut(offset..)) { - Some(out) => write_planes_u8( - self.color_space, - self.has_alpha, - planes, - dims, - out, - self.stride, - self.fmt, - ), - None => Err(J2kError::DimensionOverflow { - width: x, - height: y, - }), - }; - self.error = result.err(); - } -} - pub(super) fn component_sample_count(dims: (u32, u32)) -> Result { (dims.0 as usize) .checked_mul(dims.1 as usize) @@ -188,16 +147,18 @@ pub(super) fn output_rows( .map(move |row| &mut row[..row_bytes]) } +/// Rows `rows` of `plane`, `width` samples each, written from the first row +/// of `out` on. fn write_component_rows_u8( plane: &j2k_native::ComponentPlane<'_>, out: &mut [u8], stride: usize, width: usize, - height: usize, + rows: Range, ) { - for y in 0..height { + for (index, y) in rows.enumerate() { let src = &plane.samples()[y * width..(y + 1) * width]; - let dst = &mut out[y * stride..y * stride + width]; + let dst = &mut out[index * stride..index * stride + width]; write_samples_as_u8(src, plane.bit_depth(), dst); } } @@ -207,11 +168,23 @@ fn write_rgb_component_rows_u8( out: &mut [u8], stride: usize, width: usize, - height: usize, + rows: Range, ) { - for y in 0..height { + if planes[..3].iter().all(|plane| plane.bit_depth() == 8) { + // `sample_as_u8` at 8 bits is the RGB8 pack's rounding and saturation, + // which j2k-native runs at the CPU's SIMD level. + for (index, y) in rows.enumerate() { + let row = y * width..(y + 1) * width; + j2k_native::pack_rgb8( + [0, 1, 2].map(|channel| &planes[channel].samples()[row.clone()]), + &mut out[index * stride..index * stride + width * 3], + ); + } + return; + } + for (index, y) in rows.enumerate() { let row = y * width; - let dst = &mut out[y * stride..y * stride + width * 3]; + let dst = &mut out[index * stride..index * stride + width * 3]; for x in 0..width { let dst = &mut dst[x * 3..x * 3 + 3]; for channel in 0..3 { @@ -229,12 +202,12 @@ fn write_rgba_component_rows_u8( out: &mut [u8], stride: usize, width: usize, - height: usize, + rows: Range, synthesize_alpha: bool, ) { - for y in 0..height { + for (index, y) in rows.enumerate() { let row = y * width; - let dst = &mut out[y * stride..y * stride + width * 4]; + let dst = &mut out[index * stride..index * stride + width * 4]; for x in 0..width { let dst = &mut dst[x * 4..x * 4 + 4]; for channel in 0..3 { @@ -258,15 +231,32 @@ fn write_samples_as_u8(src: &[f32], bit_depth: u8, dst: &mut [u8]) { } } +/// [`sample_as_u8`] for an 8-bit sample. +#[expect( + clippy::cast_possible_truncation, + clippy::cast_sign_loss, + reason = "samples are rounded and clamped to the unsigned 8-bit output domain before conversion" +)] +fn sample_as_u8_depth8(sample: f32) -> u8 { + sample.round().clamp(0.0, f32::from(u8::MAX)) as u8 +} + #[expect( clippy::cast_possible_truncation, clippy::cast_sign_loss, + clippy::cast_precision_loss, reason = "samples are rounded and clamped to the unsigned 8-bit output domain before conversion" )] fn sample_as_u8(sample: f32, bit_depth: u8) -> u8 { let rounded = sample.round(); if bit_depth == 8 { - return rounded.clamp(0.0, f32::from(u8::MAX)) as u8; + return sample_as_u8_depth8(sample); + } + if bit_depth > 16 { + // The sample over the component's range, as the 8-bit planes of a + // full-image decode quantize it. + let source_max = ((1_u64 << bit_depth.min(63)) - 1) as f32; + return ((sample / source_max) * f32::from(u8::MAX)).round() as u8; } let max_value = if bit_depth >= 16 { f32::from(u16::MAX) diff --git a/crates/j2k/src/decode/route.rs b/crates/j2k/src/decode/route.rs new file mode 100644 index 000000000..c86d212f4 --- /dev/null +++ b/crates/j2k/src/decode/route.rs @@ -0,0 +1,489 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! Plan-time choice of how a full-image 8- or 16-bit decode runs. +//! +//! The planes route decodes every component into an image-sized plane of +//! `f32` samples and packs the output from them. Its working memory grows +//! with the image, and the native decoder refuses a decode whose planes and +//! largest tile's workspace pass its budget. The streamed route runs the +//! decode as a region decode of the whole image, which hands finished rows +//! to the output and keeps no planes (see +//! `Image::decode_region_rows_for_integer_output_with_context`); an image of +//! several tiles runs it on several tiles at once (see [`tiles`]). The +//! native decoder plans the planes route's charge and whether the region +//! decode streams from the codestream before anything is decoded (see +//! `Image::full_decode_planes_floor`), so the choice is made once, before +//! the decode. Only a stream chosen by preference whose own working memory +//! the budget refuses runs again, on the planes, and only when they fit (see +//! `planes_fit`): streaming by preference never fails a decode the planes +//! finish. + +use j2k_core::{Downscale, PixelFormat, Rect}; +use j2k_native::CpuDecodeParallelism; + +use super::output::u16_rows_pack_floor; +use crate::backend::Image; + +pub(crate) mod tiles; + +/// How a full-image decode runs. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum FullImageRoute { + /// Component planes for the whole image. + Planes, + /// A region decode of the whole image that streams its rows. + StreamedRegion, + /// Streamed region decodes of one tile each, `workers` at once, each on + /// its worker's thread alone. + StreamedTiles { tiles: Vec, workers: usize }, +} + +impl FullImageRoute { + /// Whether the decode streams, holding no component planes. + pub(crate) fn streams(&self) -> bool { + !matches!(self, Self::Planes) + } +} + +/// What a full-image decode's component planes may charge. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum PlanesBudget { + /// The native decode budget; an image within it still streams where + /// streaming is the better route (see [`streams_by_default`]). + Default, + /// These bytes and no preference: the planes route unless the planes + /// would pass them. A test hook that forces either route. + Fixed(usize), +} + +/// The facts of a decode request the route depends on besides the image. +#[derive(Debug, Clone, Copy)] +pub(crate) struct RouteRequest { + pub(crate) fmt: PixelFormat, + pub(crate) budget: PlanesBudget, + pub(crate) parallelism: CpuDecodeParallelism, + /// The image's components. + pub(crate) components: u16, + /// The bytes of the coded image, its codestream and container. + pub(crate) coded_bytes: usize, + /// The resolution the image is decoded at. + pub(crate) scale: Downscale, +} + +/// How a full-image decode of `image` runs for `request`: streamed when the +/// output has 8 bits per sample, or 16 bits whose rows carry the native +/// samples the planes route packs (see `u16_rows_pack_floor`), the region +/// decode streams, and either the planes (or their 16-bit pack) would pass +/// the budget or the streamed route is the better one for this image. +/// +/// A planning error keeps the planes route, which meets the same error and +/// reports it. +pub(crate) fn full_image_route(image: &Image<'_>, request: RouteRequest) -> FullImageRoute { + let Some(pack_floor) = pack_floor(image, request.fmt) else { + return FullImageRoute::Planes; + }; + let planes_budget = match request.budget { + PlanesBudget::Default => j2k_native::DEFAULT_MAX_DECODE_BYTES, + PlanesBudget::Fixed(bytes) => bytes, + }; + // A budget of zero skips the planner's header-only shortcut, so the + // floor is known whenever the region decode would stream; the shortcut + // saves the tile parse when only the budget can choose streaming. + let streams = if request.budget == PlanesBudget::Default && streams_by_default(image, request) { + matches!(image.full_decode_planes_floor(0), Ok(Some(_))) + } else { + matches!( + image.full_decode_planes_floor(planes_budget), + Ok(Some(floor)) if floor.max(pack_floor) > planes_budget + ) + }; + if streams { + streamed_route(image, request) + } else { + FullImageRoute::Planes + } +} + +/// What the 16-bit pack of `image` into `fmt` holds besides the planes (see +/// `u16_rows_pack_floor`), nothing for an 8-bit output, or `None` when the +/// streamed rows cannot write `fmt`. +fn pack_floor(image: &Image<'_>, fmt: PixelFormat) -> Option { + match fmt { + PixelFormat::Rgb8 | PixelFormat::Rgba8 | PixelFormat::Gray8 => Some(0), + PixelFormat::Rgb16 | PixelFormat::Rgba16 | PixelFormat::Gray16 => { + u16_rows_pack_floor(image, fmt) + } + _ => None, + } +} + +/// Whether a full-image decode of `image` for `request` that streams by +/// preference may run on the planes instead when the budget refuses its +/// stream: at the default budget, when the planes and their 16-bit pack fit +/// it. The planes grow with an image's area and the streamed rows' line +/// buffers with its width, so a wide and shallow image can fit the one and +/// not the other: an RGB image 8,388,608 samples wide and a row or two deep +/// fits its planes, but not its reconstruction rows. +pub(super) fn planes_fit(image: &Image<'_>, request: RouteRequest) -> bool { + let budget = j2k_native::DEFAULT_MAX_DECODE_BYTES; + request.budget == PlanesBudget::Default + && pack_floor(image, request.fmt).is_some_and(|pack_floor| { + !matches!( + image.full_decode_planes_floor(budget), + Ok(Some(floor)) if floor.max(pack_floor) > budget + ) + }) +} + +/// Pool threads from which a classic tile of several components +/// reconstructs each component on its own pool reader (`parallel_synthesis` +/// in the native decoder's pool bands). From there, a full-resolution +/// untiled image decodes 3-6% faster on the planes. +const CLASSIC_SYNTHESIS_THREADS: usize = 12; + +/// Samples from which a full-resolution classic image streams on 1 to 11 +/// pool threads (a 512 x 512 image at 6.6 coded bits per pixel and 768 x 512 +/// ones at 9.6 decode 5-10% faster streamed on 6 threads, 1% slower on one, +/// in a third of the memory); smaller images spend more of their time +/// setting up the pool bands, and on one thread the line IDWT's rows. +const MIN_STREAMED_SAMPLES: u64 = 1 << 18; + +/// Samples below which a full-resolution classic image also streams from +/// [`CLASSIC_SYNTHESIS_THREADS`] pool threads: the 512 x 512 and 768 x 512 +/// images decode 3-14% faster streamed on 18 threads, a 4096 x 1024 one +/// 6-7% slower. +const MAX_STREAMED_SAMPLES_ON_A_LARGE_POOL: u64 = 1 << 19; + +/// Samples a tile needs to stream one after another with the pool behind +/// it (measured on 1024 x 1024 tiles only). +const MIN_STREAMED_TILE_SAMPLES: u64 = 1 << 20; + +/// Samples a classic tile needs to stream one after another on one thread: +/// 4096 x 4096 and 2048 x 2048 RGB images in 256 x 256 tiles decode 1% +/// slower streamed, in a fiftieth to a two-hundredth of the memory. An HTJ2K +/// tile needs [`MIN_STREAMED_SAMPLES`] (512 x 512 tiles 0.93-0.99 of the +/// planes), the streamed IDWT's rows costing more on narrower tiles. +const MIN_STREAMED_TILE_SAMPLES_ON_ONE_THREAD: u64 = 1 << 16; + +/// Output samples from which a reduced-resolution classic decode streams on +/// any pool (a 1024 x 512 output and larger: 6-53% faster on 18 threads), +/// and an HTJ2K one on two threads or more or on one at 1/4 and below. On 12 +/// pool threads or more a smaller classic output gains nothing from +/// streaming (a 384 x 256 output of a 768 x 512 image 2-3% slower on 12 +/// threads, 2% faster on 18), and smaller HTJ2K outputs are up to 59% slower +/// streamed on any pool. +const MIN_REDUCED_STREAMED_SAMPLES: u64 = 1 << 19; + +/// Output samples from which a smaller reduced classic decode streams on 2 +/// to 11 pool threads: 256 x 256 and 384 x 256 outputs decode 3-12% faster +/// streamed on 6 threads, a 192 x 128 one 1-3% slower. +const MIN_SMALL_REDUCED_STREAMED_SAMPLES: u64 = 1 << 16; + +/// Output samples from which a smaller reduced classic decode streams on one +/// thread: 512 x 256 to 758 x 532 outputs of 4096 x 2048 to 6061 x 4252 +/// images decode 3-11% faster streamed, in a thirtieth to a sixtieth of the +/// memory (the planes route holds the full-resolution coefficients), while +/// 256 x 256 and 384 x 256 outputs of small images are 0-6% slower. +const MIN_SMALL_REDUCED_STREAMED_SAMPLES_ON_ONE_THREAD: u64 = 1 << 17; + +/// Samples from which an untiled one-component classic image streams at +/// full resolution on 2 to 11 pool threads: the smallest gray image +/// measured. With the pool bands decoding a band in parts when the rings +/// hold too few bands, 2896 x 2896 to 5792 x 5792 gray images decode 12-15% +/// faster streamed on 6 threads, in a fifth to a seventh of the memory +/// (5792 x 5792: 459 vs 68 MB). +const MIN_STREAMED_GRAY_SAMPLES: u64 = 2896 * 2896; + +/// Samples from which a one-component classic image streams at full +/// resolution on one thread: 1024 x 1024 to 4096 x 1024 gray images at 1.5 +/// bits per pixel to lossless decode 0-3% slower streamed, in a fifteenth to +/// a thirtieth of the memory. +const MIN_STREAMED_GRAY_SAMPLES_ON_ONE_THREAD: u64 = 1 << 20; + +/// Samples from which a gray image streams on larger pools too: 5792 x 5792 +/// decodes as fast streamed on 18 threads (0.96-1.02), 2896 x 2896 and +/// 4096 x 4096 are 3-14% slower. +const MIN_STREAMED_GRAY_SAMPLES_ON_A_LARGE_POOL: u64 = 5792 * 5792; + +/// Coded bits per sample from which a full-resolution untiled classic image +/// streams as [`full_resolution_streams`] says: below it Tier-1 is light, the +/// decode is bound by the inverse transforms and the pack, and the planes' +/// fused last level does those faster on small images. +const MIN_STREAMED_CODED_BITS_PER_SAMPLE: u64 = 4; + +/// Samples from which a full-resolution classic image below +/// [`MIN_STREAMED_CODED_BITS_PER_SAMPLE`] streams on a pool of up to +/// [`MAX_LOW_RATE_SHARED_READER_THREADS`] threads or of +/// [`CLASSIC_SYNTHESIS_THREADS`] or more: with its rows packed through the +/// fused RGB8 pack, a 4096 x 2048 image at 2 bits per pixel decodes 4-7% +/// faster streamed on 4 to 6 threads and 5% faster on 18, in less than half +/// the memory; on one thread 4096 x 2048 to 5428 x 4460 images at 1.5 to 2 +/// bits per pixel decode 2-3% slower streamed, in a tenth to a twentieth of +/// the memory (5428 x 4460: 309 vs 16 MiB), and 2048 x 2048 to 2896 x 2896 +/// ones 4% slower; 512 x 512 ones at 2 bits per pixel are 22% slower on 6 +/// threads. +const MIN_STREAMED_LOW_RATE_SAMPLES: u64 = 1 << 23; + +/// Pool threads up to which a large image below +/// [`MIN_STREAMED_CODED_BITS_PER_SAMPLE`] streams while one pool reader +/// reconstructs all of its components (below [`CLASSIC_SYNTHESIS_THREADS`]): +/// on more, Tier-1 of so few coded bits outruns that reader, and the +/// 4096 x 2048 image decodes 16-19% slower streamed on 10 threads. +const MAX_LOW_RATE_SHARED_READER_THREADS: usize = 6; + +/// Whether an image the planes route can decode streams anyway, because +/// streaming was measured at least as fast for its kind, or on one thread +/// nearly as fast in much less memory; for three components: +/// - a full-resolution tiled image whose tiles decode at once (see +/// [`tile_workers`]); +/// - an untiled reduced image of [`MIN_REDUCED_STREAMED_SAMPLES`] output +/// samples or more: classic at any thread count, HTJ2K on two threads or +/// more and on one from 1/4 (1/4 outputs of 2896 x 2896 to 4096 x 4096 +/// images at 1 to 12 bits per pixel 6-13% faster streamed, in a fifteenth +/// of the memory; 1/2 outputs 1-9% slower); a smaller classic one of +/// [`MIN_SMALL_REDUCED_STREAMED_SAMPLES`] or more on 2 to 11 pool threads +/// and of [`MIN_SMALL_REDUCED_STREAMED_SAMPLES_ON_ONE_THREAD`] or more on +/// one; on one thread a tiled one too (4096 x 4096 RGB in 1024 x 1024 +/// tiles at 1/2 and 1/4 1-4% faster, in a fiftieth of the memory); +/// - a full-resolution classic image of at least 4 coded bits per pixel, or a +/// large one at fewer on most pools and on one thread (see +/// [`full_resolution_streams`]); a tiled one whose tiles do not decode at +/// once streams them one after another, so each of its tiles must meet +/// that as an image of its own, of [`MIN_STREAMED_TILE_SAMPLES`] or more +/// (4096 x 4096 RGB in 16 tiles of 1024 x 1024: 116.9 vs 124.9 ms on the +/// planes with 6 threads); on one thread any classic tile of +/// [`MIN_STREAMED_TILE_SAMPLES_ON_ONE_THREAD`] or more streams, and any +/// HTJ2K tile of [`MIN_STREAMED_SAMPLES`] or more. +/// +/// A one-component classic image streams as [`gray_streams`] says, or on +/// one thread at full resolution in tiles of +/// [`MIN_STREAMED_TILE_SAMPLES_ON_ONE_THREAD`] or more, and a 16-bit output +/// on one thread as [`sixteen_bit_streams_on_one_thread`] says. Otherwise +/// untiled HTJ2K images at full resolution (whose streamed IDWT and pack +/// cost more per sample than the planes' fused last level: 5-28% slower on +/// one thread), HTJ2K gray images, small images and, on pools, reduced tiled +/// images (one tile after another was 1-13% slower on 6 threads, 1.5-10 +/// times on 18) keep the planes. +fn streams_by_default(image: &Image<'_>, request: RouteRequest) -> bool { + let threads = pool_threads(request.parallelism); + let htj2k = image.uses_high_throughput_block_coding(); + let tiled = image.tile_count() > 1; + let mut samples = u64::from(image.width()) * u64::from(image.height()); + if threads == 1 + && matches!(request.components, 1 | 3) + && matches!( + request.fmt, + PixelFormat::Rgb16 | PixelFormat::Rgba16 | PixelFormat::Gray16 + ) + && sixteen_bit_streams_on_one_thread(image, samples) + { + return true; + } + if request.components == 1 { + let reduced = request.scale != Downscale::None; + if htj2k { + return false; + } + if tiled && !reduced && threads == 1 { + return average_tile_samples(image, samples) + .is_some_and(|tile| tile >= MIN_STREAMED_TILE_SAMPLES_ON_ONE_THREAD); + } + return gray_streams(samples, reduced, tiled, threads); + } + if request.components != 3 { + return false; + } + if request.scale != Downscale::None { + return (!tiled || threads == 1) && reduced_streams(samples, request.scale, htj2k, threads); + } + let mut coded_bytes = request.coded_bytes; + if tiled { + let Ok(rects) = image.output_tile_rects() else { + return false; + }; + if tile_workers(&rects, threads).is_some() { + return true; + } + let tile_count = rects.len().max(1); + samples /= u64::try_from(tile_count).unwrap_or(u64::MAX); + coded_bytes /= tile_count; + if threads == 1 { + return samples + >= if htj2k { + MIN_STREAMED_SAMPLES + } else { + MIN_STREAMED_TILE_SAMPLES_ON_ONE_THREAD + }; + } + if samples < MIN_STREAMED_TILE_SAMPLES { + return false; + } + } + !htj2k && full_resolution_streams(samples, coded_bytes, threads) +} + +/// Whether a 16-bit decode of `image`, of `samples` output samples, streams +/// on one thread whatever its coding: from [`MIN_STREAMED_SAMPLES`], untiled +/// or in tiles of [`MIN_STREAMED_TILE_SAMPLES_ON_ONE_THREAD`] or more. The +/// planes route packs such outputs from planes of native samples it fills +/// first, which the streamed rows skip: 4096 x 2048 HTJ2K images decode +/// 21-25% faster streamed, 2896 x 2896 to 5428 x 4460 classic images at 1.5 +/// bits per pixel 6-9% faster and 512 x 512 to 768 x 768 ones 1% faster, +/// while smaller images and tiles are 4-5% slower. +fn sixteen_bit_streams_on_one_thread(image: &Image<'_>, samples: u64) -> bool { + samples >= MIN_STREAMED_SAMPLES + && (image.tile_count() <= 1 + || average_tile_samples(image, samples) + .is_some_and(|tile| tile >= MIN_STREAMED_TILE_SAMPLES_ON_ONE_THREAD)) +} + +/// The output samples of an average tile of `image`, of `samples` output +/// samples in all, or `None` when its tiles cannot be listed. +fn average_tile_samples(image: &Image<'_>, samples: u64) -> Option { + let tiles = image.output_tile_rects().ok()?.len().max(1); + Some(samples / u64::try_from(tiles).unwrap_or(u64::MAX)) +} + +/// Whether a one-component classic image of `samples` output samples, +/// `tiled` or not, streams on `threads` pool threads: on one thread reduced +/// to [`MIN_SMALL_REDUCED_STREAMED_SAMPLES_ON_ONE_THREAD`] or more, tiled or +/// not (5100 x 6600 and 5792 x 5792 images at 1/2 to 1/8 0-3% faster, in a +/// fiftieth to a hundredth of the memory), or untiled at full resolution +/// from [`MIN_STREAMED_GRAY_SAMPLES_ON_ONE_THREAD`]; on more threads untiled +/// at full resolution from [`MIN_STREAMED_GRAY_SAMPLES`], and from 12 threads +/// only from [`MIN_STREAMED_GRAY_SAMPLES_ON_A_LARGE_POOL`]. A tiled gray image +/// at full resolution streams on one thread in tiles of +/// [`MIN_STREAMED_TILE_SAMPLES_ON_ONE_THREAD`] or more (2048 x 2048 in +/// 512 x 512 tiles to 5100 x 6600 in 1024 x 1024 ones 1-3% slower, in a +/// fortieth to a hundredth of the memory). +fn gray_streams(samples: u64, reduced: bool, tiled: bool, threads: usize) -> bool { + if threads == 1 { + return if reduced { + samples >= MIN_SMALL_REDUCED_STREAMED_SAMPLES_ON_ONE_THREAD + } else { + !tiled && samples >= MIN_STREAMED_GRAY_SAMPLES_ON_ONE_THREAD + }; + } + !reduced + && !tiled + && samples >= MIN_STREAMED_GRAY_SAMPLES + && (threads < CLASSIC_SYNTHESIS_THREADS + || samples >= MIN_STREAMED_GRAY_SAMPLES_ON_A_LARGE_POOL) +} + +/// Whether a reduced-resolution decode of a three-component image at +/// `scale` to `samples` output samples streams on `threads` pool threads. +fn reduced_streams(samples: u64, scale: Downscale, htj2k: bool, threads: usize) -> bool { + if samples >= MIN_REDUCED_STREAMED_SAMPLES { + !htj2k || threads >= 2 || (threads == 1 && scale.denominator() >= 4) + } else if threads == 1 { + !htj2k && samples >= MIN_SMALL_REDUCED_STREAMED_SAMPLES_ON_ONE_THREAD + } else { + !htj2k + && samples >= MIN_SMALL_REDUCED_STREAMED_SAMPLES + && (2..CLASSIC_SYNTHESIS_THREADS).contains(&threads) + } +} + +/// Whether a full-resolution three-component classic image (or tile) of +/// `samples` pixels coded in `coded_bytes` streams on `threads` pool threads: +/// from [`MIN_STREAMED_SAMPLES`] and [`MIN_STREAMED_CODED_BITS_PER_SAMPLE`] +/// below [`CLASSIC_SYNTHESIS_THREADS`] pool threads, and on more when it is +/// below [`MAX_STREAMED_SAMPLES_ON_A_LARGE_POOL`]; at fewer coded bits, from +/// [`MIN_STREAMED_LOW_RATE_SAMPLES`] on up to +/// [`MAX_LOW_RATE_SHARED_READER_THREADS`] pool threads and from +/// [`CLASSIC_SYNTHESIS_THREADS`]. +fn full_resolution_streams(samples: u64, coded_bytes: usize, threads: usize) -> bool { + let coded_bits = u64::try_from(coded_bytes) + .unwrap_or(u64::MAX) + .saturating_mul(8); + if coded_bits < samples.saturating_mul(MIN_STREAMED_CODED_BITS_PER_SAMPLE) { + return samples >= MIN_STREAMED_LOW_RATE_SAMPLES + && (matches!(threads, 1..=MAX_LOW_RATE_SHARED_READER_THREADS) + || threads >= CLASSIC_SYNTHESIS_THREADS); + } + samples >= MIN_STREAMED_SAMPLES + && match threads { + 0 => samples >= MIN_STREAMED_TILE_SAMPLES, + 1..CLASSIC_SYNTHESIS_THREADS => true, + _ => samples < MAX_STREAMED_SAMPLES_ON_A_LARGE_POOL, + } +} + +/// The threads a decode with `parallelism` runs on. +fn pool_threads(parallelism: CpuDecodeParallelism) -> usize { + match parallelism { + CpuDecodeParallelism::Auto => j2k_native::cpu_decode_pool_threads(), + CpuDecodeParallelism::Serial => 1, + } +} + +/// Tiles above which an image decodes one tile after another rather than at +/// once: each tile's region decode parses the tile-part headers of the whole +/// image again, a cost that grows with the square of the tiles. +const MAX_TILES_AT_ONCE: usize = 1024; + +/// Output samples a tile needs on average for its own region decode to pay +/// for setting it up. +const MIN_SAMPLES_PER_TILE: usize = 256 * 256; + +/// Workers that decode the tiles at `rects` at once on `threads` pool +/// threads: the calling thread and one per pool thread, the threads a +/// streamed region decode runs on. Each worker decodes its tile alone, so +/// the image needs a tile for every worker, and its tiles must keep at least +/// 80% of the workers busy over the rounds they take: 16 tiles on 7 workers +/// take three rounds with about a quarter of them idle, and decode no faster +/// than the planes. Other images decode one tile after another with the pool +/// behind each. +fn tile_workers(rects: &[(u32, u32, u32, u32)], threads: usize) -> Option { + let tiles = rects.len(); + let samples: usize = rects + .iter() + .map(|&(_, _, w, h)| w as usize * h as usize) + .sum(); + let workers = threads.checked_add(1)?; + let rounds = tiles.div_ceil(workers); + (threads >= 2 + && (workers..=MAX_TILES_AT_ONCE).contains(&tiles) + && samples >= tiles.saturating_mul(MIN_SAMPLES_PER_TILE) + && tiles.saturating_mul(5) >= rounds.saturating_mul(workers).saturating_mul(4)) + .then_some(workers) +} + +/// Bytes a tile worker needs per sample of its tile beside the tile's +/// output rows: a streamed decode of one tile on one thread. +const EST_STREAMED_BYTES_PER_TILE_SAMPLE: usize = 16; + +/// The streamed route for `image`: its tiles at once when it has a tile for +/// every worker and the workers' tiles fit in the native decode budget, else +/// one region decode of the whole image. +fn streamed_route(image: &Image<'_>, request: RouteRequest) -> FullImageRoute { + let Ok(rects) = image.output_tile_rects() else { + return FullImageRoute::StreamedRegion; + }; + let Some(workers) = tile_workers(&rects, pool_threads(request.parallelism)) else { + return FullImageRoute::StreamedRegion; + }; + let largest = rects + .iter() + .map(|&(_, _, w, h)| w as usize * h as usize) + .max() + .unwrap_or(0); + let per_worker = largest + .saturating_mul(EST_STREAMED_BYTES_PER_TILE_SAMPLE + request.fmt.bytes_per_pixel()) + .max(1); + let workers = workers.min(j2k_native::DEFAULT_MAX_DECODE_BYTES / per_worker); + if workers < 2 { + return FullImageRoute::StreamedRegion; + } + FullImageRoute::StreamedTiles { + tiles: rects + .into_iter() + .map(|(x, y, w, h)| Rect { x, y, w, h }) + .collect(), + workers, + } +} + +#[cfg(test)] +mod tests; diff --git a/crates/j2k/src/decode/route/tests.rs b/crates/j2k/src/decode/route/tests.rs new file mode 100644 index 000000000..1e917c0bc --- /dev/null +++ b/crates/j2k/src/decode/route/tests.rs @@ -0,0 +1,1305 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +use super::{full_image_route, FullImageRoute, PlanesBudget, RouteRequest}; +use crate::backend::{image_at_scale, Image}; +use crate::decode::{ + decode_full_image_into_with_native_context, decode_full_image_on_route, + decode_native_components_with_native_context, native_components_route, + native_region_components_route, NativeRoute, +}; +use crate::{DecodeSettings, J2kDecoder, J2kError, J2kScratchPool}; +use j2k_core::{Downscale, PixelFormat, Rect}; +use j2k_native::{encode, CpuDecodeParallelism, EncodeOptions}; +use j2k_native::{DecodeError, DecoderContext, DecodingError}; + +fn codestream(components: u16, options: &EncodeOptions) -> Vec { + let (width, height) = (96_u32, 80_u32); + let pixels: Vec = (0..width * height * u32::from(components)) + .map(|index| (index.wrapping_mul(2_654_435_761) >> 24) as u8) + .collect(); + encode(&pixels, width, height, components, 8, false, options).expect("encode fixture") +} + +fn route(bytes: &[u8], scale: Downscale, fmt: PixelFormat, budget: usize) -> FullImageRoute { + let image = image_at_scale(bytes, DecodeSettings::default(), scale).expect("image"); + full_image_route(&image, request(fmt, budget, CpuDecodeParallelism::Serial)) +} + +fn request(fmt: PixelFormat, budget: usize, parallelism: CpuDecodeParallelism) -> RouteRequest { + RouteRequest { + fmt, + budget: PlanesBudget::Fixed(budget), + parallelism, + components: 3, + coded_bytes: 0, + scale: Downscale::None, + } +} + +/// The floor the native decoder plans for the planes of `bytes` at `scale`. +fn floor(bytes: &[u8], scale: Downscale) -> usize { + let image = image_at_scale(bytes, DecodeSettings::default(), scale).expect("image"); + image + .full_decode_planes_floor(0) + .expect("plan") + .expect("a streamable image has a floor") +} + +#[test] +fn an_image_streams_only_when_its_planes_pass_the_budget() { + let fixtures = [ + codestream( + 3, + &EncodeOptions { + num_decomposition_levels: 3, + reversible: false, + ..EncodeOptions::default() + }, + ), + codestream( + 1, + &EncodeOptions { + num_decomposition_levels: 3, + tile_size: Some((32, 32)), + use_ht_block_coding: true, + ..EncodeOptions::default() + }, + ), + ]; + for bytes in &fixtures { + for scale in [Downscale::None, Downscale::Half, Downscale::Quarter] { + let floor = floor(bytes, scale); + // The planes alone of the smallest output are more than this. + assert!(floor > 20 * 24 * 4, "{scale:?}: {floor}"); + for fmt in [PixelFormat::Rgb8, PixelFormat::Rgba8, PixelFormat::Gray8] { + assert_eq!( + route(bytes, scale, fmt, floor - 1), + FullImageRoute::StreamedRegion + ); + assert_eq!(route(bytes, scale, fmt, floor), FullImageRoute::Planes); + assert_eq!( + route(bytes, scale, fmt, j2k_native::DEFAULT_MAX_DECODE_BYTES), + FullImageRoute::Planes + ); + } + } + } +} + +#[test] +fn reduced_decodes_plan_full_resolution_coefficients() { + let bytes = codestream( + 1, + &EncodeOptions { + num_decomposition_levels: 3, + ..EncodeOptions::default() + }, + ); + // A quarter of the planes and IDWT buffers, but every coefficient. + let full = floor(&bytes, Downscale::None); + let half = floor(&bytes, Downscale::Half); + assert!(half < full, "{half} < {full}"); + assert!(half > 96 * 80 * 4, "{half}"); +} + +/// A 16-bit decode streams when the planes decode or the pack of its native +/// samples, which holds every `f32` plane besides them, would pass the +/// budget. With small tiles and 16-bit samples the pack charges more than +/// the decode. +#[test] +fn sixteen_bit_output_streams_when_its_planes_or_their_pack_pass_the_budget() { + let (width, height) = (96_u32, 80_u32); + let samples: Vec = (0..width * height * 3 * 2) + .map(|index| (index.wrapping_mul(2_654_435_761) >> 24) as u8) + .collect(); + let options = EncodeOptions { + num_decomposition_levels: 2, + tile_size: Some((16, 16)), + ..EncodeOptions::default() + }; + let bytes = encode(&samples, width, height, 3, 16, false, &options).expect("encode fixture"); + let floor = floor(&bytes, Downscale::None); + let pack = 96 * 80 * 3 * (4 + 2); + assert!(pack > floor, "{pack} > {floor}"); + for fmt in [PixelFormat::Rgb16, PixelFormat::Rgba16] { + assert_eq!( + route(&bytes, Downscale::None, fmt, pack - 1), + FullImageRoute::StreamedRegion + ); + assert_eq!( + route(&bytes, Downscale::None, fmt, pack), + FullImageRoute::Planes + ); + } + // The 8-bit planes write the output directly, without a pack. + assert_eq!( + route(&bytes, Downscale::None, PixelFormat::Rgb8, pack - 1), + FullImageRoute::Planes + ); + // An RGB image has no 16-bit gray layout; the planes report that. + assert_eq!( + route(&bytes, Downscale::None, PixelFormat::Gray16, 0), + FullImageRoute::Planes + ); +} + +/// A native decoder context for `parallelism`. +fn context<'a>(parallelism: CpuDecodeParallelism) -> DecoderContext<'a> { + let mut context = DecoderContext::default(); + context.set_cpu_decode_parallelism(parallelism); + context +} + +/// `image` decoded on `route` into a tight `fmt` buffer. +fn decode_on( + image: &Image<'_>, + parallelism: CpuDecodeParallelism, + fmt: PixelFormat, + route: &FullImageRoute, +) -> Vec { + let stride = image.width() as usize * fmt.bytes_per_pixel(); + let mut out = vec![0; stride * image.height() as usize]; + decode_full_image_into_with_native_context( + image, + &mut context(parallelism), + &mut out, + stride, + fmt, + route, + ) + .unwrap_or_else(|error| panic!("{fmt:?} {parallelism:?} {route:?}: {error:?}")); + out +} + +/// A tile whose coarse resolutions are empty while finer ones are not (a +/// one-sample tile row or column at an odd position, with two levels) +/// streams like any other, as the region IDWTs reconstruct it from its first +/// level with samples: the planes floor plans it, a budget of zero puts +/// every output on the streamed route, and that route writes what the planes +/// route writes under a raised budget. (These images are too small for any +/// budget but zero to stream them: their header estimate stays far below.) +#[test] +fn tiles_with_empty_coarse_levels_stream_past_the_budget() { + let gray: &[u8] = include_bytes!( + "../../../../j2k-test-support/fixtures/empty_coarse_levels/openjpeg_gray_8x6_tile8x5_53.j2k" + ); + let rgb: &[u8] = include_bytes!( + "../../../../j2k-test-support/fixtures/empty_coarse_levels/openjpeg_rgb_6x6_tile5x5_97.j2k" + ); + for (bytes, formats) in [ + (gray, [PixelFormat::Gray8, PixelFormat::Gray16]), + (rgb, [PixelFormat::Rgb8, PixelFormat::Rgb16]), + ] { + for scale in [Downscale::None, Downscale::Half] { + let image = image_at_scale(bytes, DecodeSettings::default(), scale).expect("image"); + assert!(floor(bytes, scale) > 0, "{scale:?}"); + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + let label = format!("{:?} {scale:?} {parallelism:?}", image.color_space()); + for fmt in formats { + let streamed = full_image_route(&image, request(fmt, 0, parallelism)); + assert_eq!(streamed, FullImageRoute::StreamedRegion, "{label} {fmt:?}"); + let planes = full_image_route(&image, request(fmt, usize::MAX, parallelism)); + assert_eq!(planes, FullImageRoute::Planes, "{label} {fmt:?}"); + assert_eq!( + decode_on(&image, parallelism, fmt, &streamed), + decode_on(&image, parallelism, fmt, &planes), + "{label} {fmt:?}" + ); + } + let native = |route| { + let decoded = decode_native_components_with_native_context( + &image, + &mut context(parallelism), + None, + route, + ) + .unwrap_or_else(|error| panic!("{label} {route:?}: {error:?}")); + let planes = decoded.into_parts().planes; + planes + .into_iter() + .map(|plane| { + let plane = plane.into_parts(); + ( + plane.data, + plane.dimensions, + plane.bit_depth, + plane.signed, + plane.bytes_per_sample, + ) + }) + .collect::>() + }; + assert_eq!( + native_components_route(&image, 0), + NativeRoute::Rows, + "{label}" + ); + assert_eq!( + native_components_route(&image, usize::MAX), + NativeRoute::Planes, + "{label}" + ); + assert_eq!( + native(NativeRoute::Rows), + native(NativeRoute::Planes), + "{label}" + ); + } + } + } +} + +/// The output rectangles of an image's tiles cover it once, at every scale. +#[test] +fn tile_rects_cover_the_output_once() { + let bytes = codestream( + 3, + &EncodeOptions { + num_decomposition_levels: 3, + tile_size: Some((40, 24)), + ..EncodeOptions::default() + }, + ); + for scale in [ + Downscale::None, + Downscale::Half, + Downscale::Quarter, + Downscale::Eighth, + ] { + let image = image_at_scale(&bytes, DecodeSettings::default(), scale).expect("image"); + let (width, height) = (image.width(), image.height()); + let mut covered = vec![0_u8; width as usize * height as usize]; + let rects = image.output_tile_rects().expect("tile rects"); + assert!(rects.len() > 1, "{scale:?}"); + for (x, y, w, h) in rects { + for row in y..y + h { + for column in x..x + w { + covered[(row * width + column) as usize] += 1; + } + } + } + assert!(covered.iter().all(|&count| count == 1), "{scale:?}"); + } +} + +/// A tiled image that streams decodes its tiles on several workers when the +/// decode may use the pool and has a busy tile for each worker, and as one +/// region otherwise. +#[test] +fn tiled_images_stream_tiles_at_once_on_the_pool() { + let bytes = codestream( + 3, + &EncodeOptions { + num_decomposition_levels: 2, + tile_size: Some((8, 8)), + ..EncodeOptions::default() + }, + ); + let image = image_at_scale(&bytes, DecodeSettings::default(), Downscale::None).expect("image"); + let rects = image.output_tile_rects().expect("tile rects"); + assert_eq!(rects.len(), 120); + assert_eq!(rects[0], (0, 0, 8, 8)); + assert_eq!(rects[119], (88, 72, 8, 8)); + let serial = full_image_route( + &image, + request(PixelFormat::Rgb8, 0, CpuDecodeParallelism::Serial), + ); + assert_eq!(serial, FullImageRoute::StreamedRegion); + // 8 x 8 tiles are too small to decode on their own. + let auto = full_image_route( + &image, + request(PixelFormat::Rgb8, 0, CpuDecodeParallelism::Auto), + ); + assert_eq!(auto, FullImageRoute::StreamedRegion); +} + +/// Tiles at once take the calling thread and every pool thread, and need a +/// tile for each that keeps them busy, of a size worth its own decode. +#[test] +fn tiles_at_once_need_a_busy_tile_for_every_worker() { + let grid = |tiles: usize, side: u32| vec![(0, 0, side, side); tiles]; + let workers = |tiles, threads| super::tile_workers(&grid(tiles, 1024), threads); + assert_eq!(workers(32, 18), Some(19)); + assert_eq!(workers(256, 18), Some(19)); + assert_eq!(workers(32, 6), Some(7)); + // Fewer tiles than workers, or rounds that leave a fifth of them idle. + assert_eq!(workers(16, 18), None); + assert_eq!(workers(16, 6), None); + assert_eq!(workers(20, 18), None); + // One thread decodes tiles one after another. + assert_eq!(workers(100, 1), None); + // Small tiles, or so many that reparsing the tile headers dominates. + assert_eq!(super::tile_workers(&grid(32, 128), 6), None); + assert_eq!(super::tile_workers(&grid(32, 256), 6), Some(7)); + assert_eq!(workers(2048, 6), None); +} + +/// Without a fixed budget, tiny reduced, gray and tiled images and small +/// full-resolution images keep the planes. +#[test] +fn the_default_streams_where_streaming_measured_faster() { + let options = |use_ht_block_coding| EncodeOptions { + num_decomposition_levels: 3, + reversible: false, + use_ht_block_coding, + ..EncodeOptions::default() + }; + let classic = codestream(3, &options(false)); + let ht = codestream(3, &options(true)); + let gray = codestream(1, &options(false)); + let tiled = codestream( + 3, + &EncodeOptions { + tile_size: Some((48, 40)), + ..options(false) + }, + ); + let default = |bytes: &[u8], scale, fmt, components, parallelism| { + let image = image_at_scale(bytes, DecodeSettings::default(), scale).expect("image"); + full_image_route( + &image, + RouteRequest { + fmt, + budget: PlanesBudget::Default, + parallelism, + components, + coded_bytes: bytes.len(), + scale, + }, + ) + }; + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + for scale in [Downscale::Half, Downscale::Quarter] { + for fmt in [PixelFormat::Rgb8, PixelFormat::Rgba8, PixelFormat::Gray8] { + // 48 x 40 and 24 x 20 outputs are below 256 x 256. + assert_eq!( + default(&classic, scale, fmt, 3, parallelism), + FullImageRoute::Planes, + "{scale:?} {fmt:?} {parallelism:?}" + ); + assert_eq!( + default(&ht, scale, fmt, 3, parallelism), + FullImageRoute::Planes, + "{scale:?} {fmt:?} {parallelism:?}" + ); + } + assert_eq!( + default(&gray, scale, PixelFormat::Gray8, 1, parallelism), + FullImageRoute::Planes + ); + assert_eq!( + default(&tiled, scale, PixelFormat::Rgb8, 3, parallelism), + FullImageRoute::Planes + ); + } + // 96 x 80 samples are far below a megapixel, tiled or not, and below + // the smallest gray image that streams. + for bytes in [classic.as_slice(), ht.as_slice(), tiled.as_slice()] { + assert_eq!( + default(bytes, Downscale::None, PixelFormat::Rgb8, 3, parallelism), + FullImageRoute::Planes + ); + } + assert_eq!( + default(&gray, Downscale::None, PixelFormat::Gray8, 1, parallelism), + FullImageRoute::Planes + ); + assert_eq!( + default( + &classic, + Downscale::Half, + PixelFormat::Rgb16, + 3, + parallelism + ), + FullImageRoute::Planes + ); + } +} + +/// Decoding the tiles at once on several workers writes the bytes the planes +/// write, at every scale, into padded rows, and only the image's bytes. +#[test] +fn tiles_at_once_write_the_planes_bytes() { + let bytes = codestream( + 3, + &EncodeOptions { + num_decomposition_levels: 3, + reversible: false, + tile_size: Some((24, 16)), + ..EncodeOptions::default() + }, + ); + for scale in [Downscale::None, Downscale::Half, Downscale::Quarter] { + let image = image_at_scale(&bytes, DecodeSettings::default(), scale).expect("image"); + let tiles: Vec = image + .output_tile_rects() + .expect("tile rects") + .into_iter() + .map(|(x, y, w, h)| Rect { x, y, w, h }) + .collect(); + for fmt in [PixelFormat::Rgb8, PixelFormat::Rgba8] { + let row = image.width() as usize * fmt.bytes_per_pixel(); + let stride = row + 3; + let len = stride * image.height() as usize; + let mut planes = vec![0xab; len]; + crate::decode::decode_full_image_into_with_native_context( + &image, + &mut DecoderContext::default(), + &mut planes, + stride, + fmt, + &FullImageRoute::Planes, + ) + .expect("planes"); + for workers in [2, 3, 7] { + let mut streamed = vec![0xab; len]; + super::tiles::decode_tiles_into( + &image, + &tiles, + workers, + (&mut streamed, stride, fmt), + ) + .expect("tiles at once"); + assert!(streamed == planes, "{scale:?} {fmt:?} {workers}"); + } + } + } +} + +/// A codestream cut short fails the same way however many workers decode +/// its tiles: with the error of the first tile that fails. +#[test] +fn tiles_at_once_report_the_first_failing_tile() { + let bytes = codestream( + 3, + &EncodeOptions { + num_decomposition_levels: 2, + tile_size: Some((32, 32)), + ..EncodeOptions::default() + }, + ); + let truncated = &bytes[..bytes.len() * 3 / 4]; + let Ok(image) = image_at_scale(truncated, DecodeSettings::default(), Downscale::None) else { + return; + }; + let tiles: Vec = image + .output_tile_rects() + .expect("tile rects") + .into_iter() + .map(|(x, y, w, h)| Rect { x, y, w, h }) + .collect(); + let stride = image.width() as usize * 3; + let mut out = vec![0; stride * image.height() as usize]; + let one = + super::tiles::decode_tiles_into(&image, &tiles, 1, (&mut out, stride, PixelFormat::Rgb8)) + .map_err(|error| error.to_string()); + for workers in [2, 5] { + let many = super::tiles::decode_tiles_into( + &image, + &tiles, + workers, + (&mut out, stride, PixelFormat::Rgb8), + ) + .map_err(|error| error.to_string()); + assert_eq!(one, many, "{workers}"); + } +} + +/// Full-resolution classic images stream from 4 coded bits per pixel on 1 +/// to 11 pool threads from 512 x 512 samples and on larger pools too below +/// 2^19 samples; at fewer bits per pixel from 4096 x 2048 samples on pools +/// of up to 6 threads and of 12 or more. +#[test] +fn full_resolution_images_stream_when_tier1_bound() { + use super::full_resolution_streams; + let mosaic = 4096 * 2048; + // 11 MB lossless and 2.2 MB at 2 bits per pixel. + assert!(full_resolution_streams(mosaic, 11_104_128, 6)); + // At 2 bits per pixel on one thread and on small and large pools, but + // not on the 7 to 11 threads where one pool reader holds the decode back. + for threads in [1, 2, 6, 12, 18] { + assert!( + full_resolution_streams(mosaic, 2_246_803, threads), + "{threads}" + ); + } + for threads in [7, 10, 11] { + assert!( + !full_resolution_streams(mosaic, 2_246_803, threads), + "{threads}" + ); + } + for threads in [12, 18] { + assert!( + !full_resolution_streams(mosaic, 11_104_128, threads), + "{threads}" + ); + } + // 5428 x 4460 at 1.5 bits per pixel streams on one thread, 2896 x 2896 + // at 1.5 keeps the planes on any pool. + assert!(full_resolution_streams(5428 * 4460, 4_553_929, 1)); + for threads in [1, 6, 18] { + assert!( + !full_resolution_streams(2896 * 2896, 1_579_087, threads), + "{threads}" + ); + } + for threads in [1, 2, 11] { + assert!( + full_resolution_streams(mosaic, 11_104_128, threads), + "{threads}" + ); + } + // 768 x 512 at 9.6 and 512 x 512 at 6.6 bits per pixel stream on any + // pool, 4096 x 1024 at 5.9 only below 12 threads. + for threads in [1, 2, 6, 12, 18] { + assert!( + full_resolution_streams(768 * 512, 472_804, threads), + "{threads}" + ); + assert!( + full_resolution_streams(512 * 512, 217_619, threads), + "{threads}" + ); + } + assert!(full_resolution_streams(4096 * 1024, 3_114_650, 1)); + assert!(full_resolution_streams(4096 * 1024, 3_114_650, 6)); + assert!(!full_resolution_streams(4096 * 1024, 3_114_650, 12)); + // No pool threads at all decide as before: from a megapixel at 4 bits per + // pixel or more, never at fewer. + assert!(full_resolution_streams(mosaic, 11_104_128, 0)); + assert!(!full_resolution_streams(768 * 512, 472_804, 0)); + assert!(!full_resolution_streams(5428 * 4460, 4_553_929, 0)); + // Below 512 x 512, and 512 x 512 tiles at 2 bits per pixel. + for threads in [1, 6] { + assert!(!full_resolution_streams(500 * 500, 250_000, threads)); + assert!(!full_resolution_streams(512 * 384, 118_125, threads)); + assert!(!full_resolution_streams(512 * 512, 64_947, threads)); + } +} + +/// Reduced decodes stream from a 1024 x 512 output on any pool, HTJ2K on two +/// threads or more and on one from 1/4; smaller classic outputs from +/// 256 x 256 on 2 to 11 pool threads and from 512 x 256 on one. +#[test] +fn reduced_decodes_stream_when_large_or_on_a_mid_sized_pool() { + use super::reduced_streams; + let (large, small) = (1024 * 512, 384 * 256); + let (half, quarter) = (Downscale::Half, Downscale::Quarter); + for threads in [1, 6, 18] { + for scale in [half, quarter, Downscale::Eighth] { + assert!(reduced_streams(large, scale, false, threads), "{threads}"); + assert!(!reduced_streams(small, scale, true, threads), "{threads}"); + } + } + // On one thread HTJ2K streams at 1/4 and 1/8, not at 1/2. + assert!(!reduced_streams(4 * large, half, true, 1)); + assert!(reduced_streams(large, quarter, true, 1)); + assert!(reduced_streams(large, Downscale::Eighth, true, 1)); + for threads in [2, 6, 11] { + assert!(reduced_streams(4 * large, half, true, threads), "{threads}"); + assert!(reduced_streams(small, half, false, threads), "{threads}"); + } + for threads in [12, 18] { + assert!(!reduced_streams(small, half, false, threads), "{threads}"); + } + // 256 x 256 streams on 6 threads, 192 x 128 keeps the planes; on one + // thread from 512 x 256. + assert!(reduced_streams(256 * 256, half, false, 6)); + assert!(!reduced_streams(192 * 128, half, false, 6)); + assert!(reduced_streams(512 * 256, Downscale::Eighth, false, 1)); + assert!(!reduced_streams(small, half, false, 1)); + assert!(!reduced_streams(256 * 256, half, false, 1)); + // No pool threads at all decide as before. + assert!(reduced_streams(large, half, false, 0)); + assert!(!reduced_streams(large, quarter, true, 0)); + assert!(!reduced_streams(512 * 256, Downscale::Eighth, false, 0)); +} + +/// Gray images stream at full resolution from 2896 x 2896 on 2 to 11 pool +/// threads and on larger pools from 5792 x 5792, untiled at full resolution +/// only; on one thread untiled from 1024 x 1024 and reduced from 512 x 256, +/// tiled or not. +#[test] +fn large_gray_images_stream() { + use super::gray_streams; + for threads in [0, 1, 6, 11] { + assert!( + gray_streams(2896 * 2896, false, false, threads), + "{threads}" + ); + } + for threads in [0, 6, 11] { + assert!( + !gray_streams(2048 * 2048, false, false, threads), + "{threads}" + ); + assert!( + !gray_streams(2896 * 2896, true, false, threads), + "{threads}" + ); + assert!( + !gray_streams(5792 * 5792, false, true, threads), + "{threads}" + ); + } + for threads in [12, 18] { + assert!( + gray_streams(5792 * 5792, false, false, threads), + "{threads}" + ); + assert!( + !gray_streams(4096 * 4096, false, false, threads), + "{threads}" + ); + } + // One thread: 1024 x 1024 and larger, and reduced outputs from 512 x 256. + assert!(gray_streams(1024 * 1024, false, false, 1)); + assert!(!gray_streams(1024 * 512, false, false, 1)); + for tiled in [false, true] { + assert!(gray_streams(1275 * 1650, true, tiled, 1)); + assert!(gray_streams(512 * 256, true, tiled, 1)); + assert!(!gray_streams(256 * 256, true, tiled, 1)); + } +} + +/// `bytes`, a one-tile codestream, claiming a `width` x `height` image and +/// tile. Headers only: the route reads them and plans the planes from them. +fn claiming(bytes: &[u8], (width, height): (u32, u32)) -> Vec { + let bytes = rewrite_siz(bytes, &[6, 22], |_| width); + rewrite_siz(&bytes, &[10, 26], |_| height) +} + +/// The default route of `bytes`, an image of `components`, at `scale` into +/// RGB8 or Gray8 on one thread, as if the codestream were `coded_bytes` long. +fn one_thread_route( + bytes: &[u8], + components: u16, + scale: Downscale, + coded_bytes: usize, +) -> FullImageRoute { + let fmt = if components == 1 { + PixelFormat::Gray8 + } else { + PixelFormat::Rgb8 + }; + one_thread_route_to(bytes, components, scale, coded_bytes, fmt) +} + +/// [`one_thread_route`] into `fmt`: the same with +/// `CpuDecodeParallelism::Serial` and on a pool of one thread. +fn one_thread_route_to( + bytes: &[u8], + components: u16, + scale: Downscale, + coded_bytes: usize, + fmt: PixelFormat, +) -> FullImageRoute { + let image = image_at_scale(bytes, DecodeSettings::default(), scale).expect("image"); + let route = |parallelism| { + full_image_route( + &image, + RouteRequest { + fmt, + budget: PlanesBudget::Default, + parallelism, + components, + coded_bytes, + scale, + }, + ) + }; + let serial = route(CpuDecodeParallelism::Serial); + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(1) + .build() + .expect("one-thread pool"); + let one_thread_pool = pool.install(|| route(CpuDecodeParallelism::Auto)); + assert_eq!(serial, one_thread_pool, "{fmt:?} {scale:?}"); + serial +} + +/// On one thread the default streams large low-rate and mid-sized classic +/// RGB images, gray images from a megapixel, reduced and tiled images and +/// HTJ2K images from 1/4, and keeps HTJ2K images at full resolution and 1/2, +/// HTJ2K gray images and small images on the planes. +#[test] +fn one_thread_streams_wherever_streaming_is_nearly_as_fast() { + use FullImageRoute::{Planes, StreamedRegion}; + let options = |use_ht_block_coding| EncodeOptions { + num_decomposition_levels: 3, + reversible: false, + use_ht_block_coding, + ..EncodeOptions::default() + }; + let (classic, ht) = ( + codestream(3, &options(false)), + codestream(3, &options(true)), + ); + let (gray, gray_ht) = ( + codestream(1, &options(false)), + codestream(1, &options(true)), + ); + let full = Downscale::None; + // RGB at 2 and 1.5 bits per pixel, and at 6. + let mosaic = claiming(&classic, (4096, 2048)); + assert_eq!( + one_thread_route_to(&mosaic, 3, full, 2_246_803, PixelFormat::Rgba8), + StreamedRegion + ); + assert_eq!( + one_thread_route(&mosaic, 3, full, 2_246_803), + StreamedRegion + ); + let square = claiming(&classic, (2896, 2896)); + assert_eq!(one_thread_route(&square, 3, full, 1_579_087), Planes); + let mid = claiming(&classic, (1024, 768)); + assert_eq!(one_thread_route(&mid, 3, full, 589_824), StreamedRegion); + let small = claiming(&classic, (512, 384)); + assert_eq!(one_thread_route(&small, 3, full, 147_456), Planes); + // HTJ2K keeps the planes at full resolution and at 1/2, and streams + // from 1/4. + let ht_mosaic = claiming(&ht, (4096, 2048)); + assert_eq!(one_thread_route(&ht_mosaic, 3, full, 12_236_468), Planes); + assert_eq!( + one_thread_route(&ht_mosaic, 3, Downscale::Half, 12_236_468), + Planes + ); + assert_eq!( + one_thread_route(&ht_mosaic, 3, Downscale::Quarter, 2_249_731), + StreamedRegion + ); + // Reduced classic outputs from 512 x 256. + for scale in [Downscale::Quarter, Downscale::Eighth] { + assert_eq!( + one_thread_route(&mosaic, 3, scale, 2_246_803), + StreamedRegion, + "{scale:?}" + ); + } + assert_eq!( + one_thread_route(&mid, 3, Downscale::Eighth, 589_824), + Planes + ); + // Gray: classic from a megapixel and reduced, HTJ2K never. + let gray_square = claiming(&gray, (1024, 1024)); + assert_eq!( + one_thread_route(&gray_square, 1, full, 196_608), + StreamedRegion + ); + let gray_page = claiming(&gray, (5100, 6600)); + assert_eq!( + one_thread_route(&gray_page, 1, Downscale::Quarter, 6_319_456), + StreamedRegion + ); + let gray_strip = claiming(&gray, (1024, 512)); + assert_eq!(one_thread_route(&gray_strip, 1, full, 98_304), Planes); + let gray_ht_square = claiming(&gray_ht, (2048, 2048)); + assert_eq!( + one_thread_route(&gray_ht_square, 1, full, 1_114_149), + Planes + ); +} + +/// On one thread a tiled classic image streams its tiles one after another +/// at full resolution from 256 x 256 tiles, gray or RGB, and an HTJ2K one +/// from 512 x 512 tiles, or from 256 x 256 into 16 bits. A reduced classic +/// decode streams by its output's size whatever the tiles. +#[test] +fn one_thread_streams_tiles_from_256_x_256() { + let encoded = |components: u16, (width, height): (u32, u32), tile: u32, ht: bool| { + let pixels = vec![0x80_u8; (width * height) as usize * usize::from(components)]; + let options = EncodeOptions { + num_decomposition_levels: 3, + tile_size: Some((tile, tile)), + use_ht_block_coding: ht, + ..EncodeOptions::default() + }; + encode(&pixels, width, height, components, 8, false, &options).expect("encode fixture") + }; + let flat = |components, size, tile| encoded(components, size, tile, false); + for components in [3, 1] { + let bytes = flat(components, (1024, 512), 256); + let coded = bytes.len(); + for scale in [Downscale::None, Downscale::Half] { + assert_eq!( + one_thread_route(&bytes, components, scale, coded), + FullImageRoute::StreamedRegion, + "{components} {scale:?}" + ); + } + let bytes = flat(components, (512, 256), 128); + assert_eq!( + one_thread_route(&bytes, components, Downscale::None, bytes.len()), + FullImageRoute::Planes, + "{components}" + ); + } + let ht = encoded(3, (1024, 512), 256, true); + assert_eq!( + one_thread_route(&ht, 3, Downscale::None, ht.len()), + FullImageRoute::Planes + ); + let ht = encoded(3, (1024, 512), 512, true); + assert_eq!( + one_thread_route(&ht, 3, Downscale::None, ht.len()), + FullImageRoute::StreamedRegion + ); + // 16-bit outputs: HTJ2K tiles from 256 x 256. + let ht = encoded(3, (1024, 512), 256, true); + for fmt in [PixelFormat::Rgb16, PixelFormat::Rgba16] { + assert_eq!( + one_thread_route_to(&ht, 3, Downscale::None, ht.len(), fmt), + FullImageRoute::StreamedRegion, + "{fmt:?}" + ); + } + let ht = encoded(3, (1024, 512), 128, true); + assert_eq!( + one_thread_route_to(&ht, 3, Downscale::None, ht.len(), PixelFormat::Rgb16), + FullImageRoute::Planes + ); + // A 512 x 256 output of 128 x 128 tiles streams, a full-resolution + // decode of them keeps the planes. + let bytes = flat(3, (1024, 512), 128); + assert_eq!( + one_thread_route(&bytes, 3, Downscale::Half, bytes.len()), + FullImageRoute::StreamedRegion + ); + assert_eq!( + one_thread_route(&bytes, 3, Downscale::None, bytes.len()), + FullImageRoute::Planes + ); +} + +/// Images of more than 8 bits and signed images stream at 16 bits and into +/// native component planes once their planes pass the budget. +#[test] +fn deep_and_signed_images_stream_past_the_budget() { + let (width, height) = (64_u32, 48_u32); + for (components, bit_depth, signed, reversible) in [ + (3, 12, true, false), + (3, 16, false, true), + (1, 12, false, true), + (1, 16, true, false), + (1, 4, false, true), + (1, 20, false, true), + (3, 24, true, false), + ] { + let bytes_per_sample = u32::from(bit_depth).div_ceil(8); + let samples: Vec = (0..width * height * u32::from(components) * bytes_per_sample) + .map(|index| (index.wrapping_mul(2_654_435_761) >> 24) as u8) + .collect(); + let options = EncodeOptions { + num_decomposition_levels: 3, + reversible, + ..EncodeOptions::default() + }; + let bytes = encode( + &samples, width, height, components, bit_depth, signed, &options, + ) + .expect("encode fixture"); + let image = + image_at_scale(&bytes, DecodeSettings::default(), Downscale::None).expect("image"); + assert_eq!(image.streamed_rows_native_precision(), Some(bit_depth)); + let fmt = if components == 1 { + PixelFormat::Gray16 + } else { + PixelFormat::Rgb16 + }; + let label = format!("{components} x {bit_depth} bits, signed {signed}"); + assert_eq!( + full_image_route(&image, request(fmt, 0, CpuDecodeParallelism::Serial)), + FullImageRoute::StreamedRegion, + "{label}" + ); + assert_eq!( + native_components_route(&image, 0), + NativeRoute::Rows, + "{label}" + ); + let budget = j2k_native::DEFAULT_MAX_DECODE_BYTES; + assert_eq!( + full_image_route(&image, request(fmt, budget, CpuDecodeParallelism::Serial)), + FullImageRoute::Planes + ); + assert_eq!(native_components_route(&image, budget), NativeRoute::Planes); + } +} + +/// `bytes` with the SIZ fields at `fields` (byte offsets from the marker) +/// rewritten by `rewrite`. +fn rewrite_siz(bytes: &[u8], fields: &[usize], rewrite: impl Fn(u32) -> u32) -> Vec { + let mut bytes = bytes.to_vec(); + let siz = bytes + .windows(2) + .position(|marker| marker == [0xff, 0x51]) + .expect("SIZ marker"); + for &field in fields { + let field = &mut bytes[siz + field..siz + field + 4]; + let value = u32::from_be_bytes(field.try_into().expect("four-byte SIZ field")); + field.copy_from_slice(&rewrite(value).to_be_bytes()); + } + bytes +} + +/// A region decode into native component planes packs its rows into planes +/// it allocates before decoding only when the region decode streams them, +/// by the check the full-image floor makes: an image of more than four +/// components, or with image and tile offsets, keeps the planes route +/// however large it is, whose decode meets the budget before it allocates +/// image-sized buffers, and the rows decode refuses it. +#[test] +fn native_region_rows_only_for_images_whose_region_decodes_stream() { + let options = EncodeOptions { + num_decomposition_levels: 2, + ..EncodeOptions::default() + }; + let rgb = codestream(3, &options); + // The whole image and its tile one tile further on. + let translated = rewrite_siz(&rgb, &[6, 10, 14, 18, 30, 34], |value| value + 128); + // One 8192 x 8192 tile, whose native planes alone pass the budget: an + // image of these components prefers its streamed rows. Headers only; + // nothing here decodes its packets. + let claimed = |components| { + let bytes = codestream(components, &options); + rewrite_siz(&bytes, &[6, 10, 22, 26], |_| 8192) + }; + for (label, bytes, streams) in [ + ("rgb", rgb.clone(), true), + ("rgb with offsets", translated, false), + ("four components", claimed(4), true), + ("five components", claimed(5), false), + ] { + let image = + image_at_scale(&bytes, DecodeSettings::default(), Downscale::None).expect("image"); + let (width, height) = (image.width(), image.height()); + assert_eq!(image.streamed_rows_native_precision(), Some(8), "{label}"); + assert_eq!(image.region_decode_streams_rows(), streams, "{label}"); + assert_eq!( + image.full_decode_planes_floor(0).expect("plan").is_some(), + streams, + "{label}" + ); + let expected = if streams { + NativeRoute::Rows + } else { + NativeRoute::Planes + }; + assert_eq!( + native_region_components_route(&image, Rect::full((width, height))), + expected, + "{label}" + ); + if !streams { + let Err(error) = image.decode_native_region_components_rows_with_context( + (0, 0, width, height), + &mut DecoderContext::default(), + ) else { + panic!("{label}: the rows decode allocated its planes and decoded"); + }; + assert!( + matches!( + error, + DecodeError::Decoding(DecodingError::UnsupportedFeature(_)) + ), + "{label}: {error:?}" + ); + } + } +} + +/// On one thread a 16-bit output streams from 512 x 512 samples whatever the +/// coding, untiled or in tiles of 256 x 256 or more, where an 8-bit one of +/// a full-resolution HTJ2K image keeps the planes. +#[test] +fn one_thread_streams_sixteen_bit_outputs_from_512_x_512() { + use FullImageRoute::{Planes, StreamedRegion}; + let options = |use_ht_block_coding| EncodeOptions { + num_decomposition_levels: 3, + reversible: false, + use_ht_block_coding, + ..EncodeOptions::default() + }; + let (classic, ht) = ( + codestream(3, &options(false)), + codestream(3, &options(true)), + ); + let gray_ht = codestream(1, &options(true)); + let full = Downscale::None; + let ht_mosaic = claiming(&ht, (4096, 2048)); + for fmt in [PixelFormat::Rgb16, PixelFormat::Rgba16] { + assert_eq!( + one_thread_route_to(&ht_mosaic, 3, full, 2_249_731, fmt), + StreamedRegion, + "{fmt:?}" + ); + } + assert_eq!(one_thread_route(&ht_mosaic, 3, full, 2_249_731), Planes); + // 1.5 bits per pixel below 4096 x 2048, and 512 x 512 at 6. + let square = claiming(&classic, (2896, 2896)); + assert_eq!( + one_thread_route_to(&square, 3, full, 1_579_087, PixelFormat::Rgb16), + StreamedRegion + ); + let tile = claiming(&classic, (512, 512)); + assert_eq!( + one_thread_route_to(&tile, 3, full, 196_608, PixelFormat::Rgb16), + StreamedRegion + ); + let small = claiming(&classic, (512, 384)); + assert_eq!( + one_thread_route_to(&small, 3, full, 147_456, PixelFormat::Rgb16), + Planes + ); + let gray_ht_square = claiming(&gray_ht, (2048, 2048)); + assert_eq!( + one_thread_route_to(&gray_ht_square, 1, full, 1_114_149, PixelFormat::Gray16), + StreamedRegion + ); + // The reduced output counts: 1/2 of 1024 x 512 is below 512 x 512. + let strip = claiming(&ht, (1024, 512)); + assert_eq!( + one_thread_route_to(&strip, 3, Downscale::Half, 1_000_000, PixelFormat::Rgb16), + Planes + ); +} + +/// `bytes`, a one-tile codestream of one coding style with the largest +/// precincts, as a `width` x `height` image whose packets are all empty: its +/// main header with the image and tile resized, then one tile-part of a zero +/// byte per packet. It decodes to mid-level samples whatever its size, +/// without a code-block to decode. +fn with_empty_packets(bytes: &[u8], (width, height): (u32, u32)) -> Vec { + let (mut end, mut levels, mut components) = (2, None, None); + while bytes[end..end + 2] != [0xff, 0x90] { + let length = usize::from(u16::from_be_bytes([bytes[end + 2], bytes[end + 3]])); + match bytes[end + 1] { + 0x51 => components = Some(u16::from_be_bytes([bytes[end + 38], bytes[end + 39]])), + 0x52 => { + assert_eq!(bytes[end + 4] & 1, 0, "the largest precincts"); + levels = Some(u32::from(bytes[end + 9])); + } + 0x53 => panic!("one coding style"), + _ => {} + } + end += 2 + length; + } + let (levels, components) = (levels.expect("COD"), components.expect("SIZ")); + let precincts = |extent: u32| u64::from(extent.div_ceil(1 << 15)); + let per_component: u64 = (0..=levels) + .map(|shift| precincts(width.div_ceil(1 << shift)) * precincts(height.div_ceil(1 << shift))) + .sum(); + let packets = usize::try_from(per_component * u64::from(components)).expect("packets"); + let mut out = claiming(&bytes[..end], (width, height)); + let tile_part = u32::try_from(14 + packets).expect("tile-part length"); + out.extend_from_slice(&[0xff, 0x90, 0, 10, 0, 0]); + out.extend_from_slice(&tile_part.to_be_bytes()); + out.extend_from_slice(&[0, 1, 0xff, 0x93]); + out.resize(out.len() + packets, 0); + out.extend_from_slice(&[0xff, 0xd9]); + out +} + +/// The default request for a full-resolution decode of `bytes`, an image of +/// `components`, into `fmt` with `parallelism`. +fn default_request( + bytes: &[u8], + components: u16, + fmt: PixelFormat, + parallelism: CpuDecodeParallelism, +) -> RouteRequest { + RouteRequest { + fmt, + budget: PlanesBudget::Default, + parallelism, + components, + coded_bytes: bytes.len(), + scale: Downscale::None, + } +} + +/// A wide, shallow RGB image of one 5/3 level in 1024 x 4 code-blocks +/// without a multiple-component transform, of `width` x `height`. +fn wide_and_shallow(width: u32, height: u32) -> Vec { + let options = EncodeOptions { + num_decomposition_levels: 1, + code_block_width_exp: 8, + code_block_height_exp: 0, + use_mct: false, + ..EncodeOptions::default() + }; + with_empty_packets(&codestream(3, &options), (width, height)) +} + +/// A wide, shallow image whose planes fit the decode budget while its +/// streamed rows' line buffers do not (8,388,608 x 1 RGB) is preferred for +/// streaming on one thread and on a pool of six threads. Its stream is +/// refused by the budget before it decodes, and the decode then runs on the +/// planes and writes their bytes. +#[test] +fn a_preferred_stream_past_the_budget_decodes_on_the_planes() { + use CpuDecodeParallelism::{Auto, Serial}; + let width = 1_u32 << 23; + let bytes = wide_and_shallow(width, 1); + let image = image_at_scale(&bytes, DecodeSettings::default(), Downscale::None).expect("image"); + let planes_floor = image.full_decode_planes_floor(0).expect("plan"); + assert!( + planes_floor.is_some_and(|floor| floor <= j2k_native::DEFAULT_MAX_DECODE_BYTES), + "{planes_floor:?}" + ); + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(6) + .build() + .expect("pool"); + for fmt in [PixelFormat::Rgb8, PixelFormat::Rgb16] { + let request = default_request(&bytes, 3, fmt, Serial); + let route = full_image_route(&image, request); + assert_eq!(route, FullImageRoute::StreamedRegion, "{fmt:?}"); + let on_the_pool = + pool.install(|| full_image_route(&image, default_request(&bytes, 3, fmt, Auto))); + assert_eq!(on_the_pool, FullImageRoute::StreamedRegion, "{fmt:?}"); + let stride = width as usize * fmt.bytes_per_pixel(); + let mut out = vec![0; stride]; + let refused = decode_full_image_into_with_native_context( + &image, + &mut context(Serial), + &mut out, + stride, + fmt, + &route, + ); + assert!( + matches!(refused, Err(J2kError::ResourceLimit { .. })), + "{fmt:?}: {refused:?}" + ); + let planes = decode_on(&image, Serial, fmt, &FullImageRoute::Planes); + decode_full_image_on_route( + &image, + &mut context(Serial), + &mut out, + stride, + request, + &route, + ) + .unwrap_or_else(|error| panic!("{fmt:?}: {error:?}")); + assert!(out == planes, "{fmt:?}"); + // The decoder's own full-image decode, on one thread and on the pool. + for parallelism in [Some(crate::CpuDecodeParallelism::Serial), None] { + let mut decoder = J2kDecoder::new(&bytes).expect("decoder"); + if let Some(parallelism) = parallelism { + decoder.set_cpu_decode_parallelism(parallelism); + } + let mut out = vec![0; stride]; + pool.install(|| decoder.decode_into(&mut out, stride, fmt)) + .unwrap_or_else(|error| panic!("{fmt:?} {parallelism:?}: {error:?}")); + assert!(out == planes, "{fmt:?} {parallelism:?}"); + } + } +} + +/// A wide, shallow image whose streamed rows fit the budget keeps its +/// preferred stream on one thread, and the stream writes the planes' bytes: +/// 65,536 x 4 RGB into 16 bits, and 1,048,576 x 8 RGB at a low rate. +#[test] +fn a_wide_image_whose_stream_fits_streams() { + use CpuDecodeParallelism::Serial; + for ((width, height), fmt) in [ + ((1 << 16, 4), PixelFormat::Rgb16), + ((1 << 20, 8), PixelFormat::Rgb8), + ] { + let bytes = wide_and_shallow(width, height); + let image = + image_at_scale(&bytes, DecodeSettings::default(), Downscale::None).expect("image"); + let route = full_image_route(&image, default_request(&bytes, 3, fmt, Serial)); + assert_eq!(route, FullImageRoute::StreamedRegion, "{width} {fmt:?}"); + let streamed = decode_on(&image, Serial, fmt, &route); + let planes = decode_on(&image, Serial, fmt, &FullImageRoute::Planes); + assert!(streamed == planes, "{width} {fmt:?}"); + } +} + +/// Rows of `components` 8-bit samples of a `width` x `height` image with +/// edges and texture. +#[expect( + clippy::cast_possible_truncation, + reason = "each sample keeps the low byte of its pattern on purpose" +)] +fn textured(components: u16, (width, height): (u32, u32)) -> Vec { + let components = u32::from(components); + (0..width * height * components) + .map(|index| { + let (pixel, component) = (index / components, index % components); + let (x, y) = (pixel % width, pixel / width); + ((x * 3 + y * 5) ^ ((x * y) >> 6)).wrapping_add(component * 85) as u8 + }) + .collect() +} + +/// On one thread the decoder's default full-image and reduced decodes of +/// each kind of image that streams there write the planes' bytes: mid-sized +/// RGB, gray from a megapixel, RGB and gray in tiles of 256 x 256, reduced +/// RGB untiled and in tiles, reduced gray, HTJ2K RGB at 1/4, and HTJ2K RGB +/// into 16 bits untiled and in tiles of 256 x 256 (large low-rate RGB in +/// `a_wide_image_whose_stream_fits_streams`). +#[test] +fn one_thread_default_streams_write_the_planes_bytes() { + let encoded = |components: u16, size: (u32, u32), tile: Option, ht: bool| { + let options = EncodeOptions { + num_decomposition_levels: 3, + reversible: false, + tile_size: tile.map(|tile| (tile, tile)), + use_ht_block_coding: ht, + ..EncodeOptions::default() + }; + let pixels = textured(components, size); + encode(&pixels, size.0, size.1, components, 8, false, &options).expect("encode fixture") + }; + let rgb = |size, tile, ht| (encoded(3, size, tile, ht), 3); + let gray = |size, tile| (encoded(1, size, tile, false), 1); + let ht_mosaic = with_empty_packets(&encoded(3, (96, 80), None, true), (4096, 2048)); + let (full, half, quarter) = (Downscale::None, Downscale::Half, Downscale::Quarter); + let (rgb8, rgb16, gray8) = (PixelFormat::Rgb8, PixelFormat::Rgb16, PixelFormat::Gray8); + let cases = [ + ("RGB", rgb((512, 512), None, false), full, rgb8), + ("gray", gray((1024, 1024), None), full, gray8), + ("RGB tiles", rgb((512, 512), Some(256), false), full, rgb8), + ("gray tiles", gray((512, 512), Some(256)), full, gray8), + ("reduced RGB", rgb((1024, 512), None, false), half, rgb8), + ("1/2 tiles", rgb((1024, 512), Some(128), false), half, rgb8), + ("reduced gray", gray((1024, 512), None), half, gray8), + ("HTJ2K at 1/4", (ht_mosaic, 3), quarter, rgb8), + ("HTJ2K 16-bit", rgb((512, 512), None, true), full, rgb16), + ( + "16-bit tiles", + rgb((1024, 512), Some(256), true), + full, + rgb16, + ), + ]; + for (label, (bytes, components), scale, fmt) in cases { + assert_eq!( + one_thread_route_to(&bytes, components, scale, bytes.len(), fmt), + FullImageRoute::StreamedRegion, + "{label}" + ); + let decode = |planes_budget: Option| { + let mut decoder = J2kDecoder::new(&bytes).expect("decoder"); + decoder.set_cpu_decode_parallelism(crate::CpuDecodeParallelism::Serial); + if let Some(bytes) = planes_budget { + decoder.set_full_decode_planes_budget(bytes); + } + let (width, height) = decoder.info().dimensions; + let denominator = scale.denominator(); + let (width, height) = (width.div_ceil(denominator), height.div_ceil(denominator)); + let stride = width as usize * fmt.bytes_per_pixel(); + let mut out = vec![0; stride * height as usize]; + decoder + .decode_scaled_into(&mut J2kScratchPool::new(), &mut out, stride, fmt, scale) + .unwrap_or_else(|error| panic!("{label}: {error:?}")); + out + }; + assert!(decode(None) == decode(Some(usize::MAX)), "{label}"); + } +} diff --git a/crates/j2k/src/decode/route/tiles.rs b/crates/j2k/src/decode/route/tiles.rs new file mode 100644 index 000000000..427ccbc89 --- /dev/null +++ b/crates/j2k/src/decode/route/tiles.rs @@ -0,0 +1,138 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! The streamed full-image route for images of several tiles: workers +//! decode one tile each at a time, as a region decode of that tile into a +//! tile-sized buffer, and copy it into the output. +//! +//! A region decode of one tile decodes that tile alone, and tiles do not +//! share samples, so each output sample comes from exactly one tile decode +//! whichever worker runs it and whenever it finishes. A worker holds one +//! tile's decode buffers and its output rows, so the decode's memory grows +//! with the workers and the tile size, never with the image. + +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::Mutex; + +use j2k_core::{PixelFormat, Rect}; +use j2k_native::CpuDecodeParallelism; + +use crate::backend::Image; +use crate::decode::decode_image_region_into_with_native_context; +use crate::J2kError; + +/// Decode the tiles at `tiles`, output rectangles that cover `image` once, +/// on `workers` threads into `out`: the calling thread and `workers - 1` +/// scoped ones. Each tile decodes serially on its worker's thread. +/// +/// The error is the one of the first tile, in raster order, that failed: +/// tiles are claimed in that order and every claimed tile finishes, so the +/// first failing tile is always among them. +pub(crate) fn decode_tiles_into( + image: &Image<'_>, + tiles: &[Rect], + workers: usize, + (out, stride, fmt): (&mut [u8], usize, PixelFormat), +) -> Result<(), J2kError> { + let next = AtomicUsize::new(0); + let failed = AtomicBool::new(false); + let first_error: Mutex> = Mutex::new(None); + let out = Mutex::new(out); + let work = || { + let mut native_context = j2k_native::DecoderContext::default(); + native_context.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); + let mut buffer = Vec::new(); + while !failed.load(Ordering::Relaxed) { + let index = next.fetch_add(1, Ordering::Relaxed); + let Some(&tile) = tiles.get(index) else { + break; + }; + let decoded = decode_tile(image, &mut native_context, &mut buffer, tile, fmt).and_then( + |tile_stride| { + let mut out = out + .lock() + .map_err(|_| J2kError::internal_backend("tile output lock poisoned"))?; + copy_tile( + &buffer[..tile_stride * tile.h as usize], + tile_stride, + &mut out, + stride, + tile, + fmt, + ); + Ok(()) + }, + ); + if let Err(error) = decoded { + failed.store(true, Ordering::Relaxed); + if let Ok(mut first) = first_error.lock() { + if first.as_ref().is_none_or(|(at, _)| index < *at) { + *first = Some((index, error)); + } + } + } + } + }; + std::thread::scope(|scope| { + for _ in 1..workers { + // A thread the system refuses only leaves fewer workers: the + // calling thread claims every tile the others do not. + if std::thread::Builder::new() + .spawn_scoped(scope, work) + .is_err() + { + break; + } + } + work(); + }); + match first_error.into_inner() { + Ok(None) => Ok(()), + Ok(Some((_, error))) => Err(error), + Err(_) => Err(J2kError::internal_backend("tile error lock poisoned")), + } +} + +/// Region-decode `tile` into the start of `buffer`, grown to fit it, and +/// return its stride. The decode writes every byte it covers, so what an +/// earlier tile left there is overwritten. +fn decode_tile<'a>( + image: &Image<'a>, + native_context: &mut j2k_native::DecoderContext<'a>, + buffer: &mut Vec, + tile: Rect, + fmt: PixelFormat, +) -> Result { + let tile_stride = tile.w as usize * fmt.bytes_per_pixel(); + let len = tile_stride * tile.h as usize; + if buffer.len() < len { + buffer + .try_reserve_exact(len - buffer.len()) + .map_err(|_| J2kError::internal_backend("tile output buffer allocation failed"))?; + buffer.resize(len, 0); + } + decode_image_region_into_with_native_context( + image, + native_context, + &mut buffer[..len], + tile_stride, + fmt, + tile, + )?; + Ok(tile_stride) +} + +/// Copy a decoded tile's rows from `buffer` into `out` at the tile. +fn copy_tile( + buffer: &[u8], + tile_stride: usize, + out: &mut [u8], + stride: usize, + tile: Rect, + fmt: PixelFormat, +) { + let x = tile.x as usize * fmt.bytes_per_pixel(); + for (row, source) in buffer.chunks_exact(tile_stride).enumerate() { + let start = (tile.y as usize + row) * stride + x; + out[start..start + tile_stride].copy_from_slice(source); + } +} diff --git a/crates/j2k/src/decode/stripes.rs b/crates/j2k/src/decode/stripes.rs new file mode 100644 index 000000000..da2298ab3 --- /dev/null +++ b/crates/j2k/src/decode/stripes.rs @@ -0,0 +1,263 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! Bounded row decodes of a whole image in one streamed region decode. +//! +//! A region decode of the whole image hands its rows to a sink as they are +//! finished, top to bottom when every tile spans the image's width. The +//! sink here packs them into a stripe buffer of at most the caller's +//! stripe height and hands each full stripe to the caller, so every +//! code-block and wavelet row is decoded once however short the stripes +//! are, while the packed output held stays one stripe. + +use core::ops::Range; + +use crate::backend::{ColorSpace, Image}; +use crate::J2kError; +use j2k_core::{DecodeRowsError, PixelFormat}; + +use super::output::{write_plane_rows_u16, write_plane_rows_u8}; + +#[cfg(test)] +std::thread_local! { + /// Test-only count of the streamed bounded row decodes this thread + /// started. + pub(crate) static STREAMED_ROW_DECODES: core::cell::Cell = + const { core::cell::Cell::new(0) }; + /// Test-only: the next streamed bounded row decode on this thread ends + /// with a decode error, in place of its region decode or after it. + pub(crate) static INJECTED_FAILURE: core::cell::Cell> = + const { core::cell::Cell::new(None) }; +} + +/// Where a test makes a streamed bounded row decode fail. +#[cfg(test)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) enum InjectedFailure { + /// The region decode fails before it hands over any row. + BeforeRows, + /// The region decode fails after it handed over every row. + AfterRows, +} + +/// Whether a bounded row decode of `image`, whose main header codes +/// `resolution_levels` resolutions, can run as one streamed region decode +/// of the whole image: the main header codes at least one decomposition +/// level, every tile spans the image's width, so its rows arrive whole and +/// in order, and the region decode streams its rows as far as the main and +/// tile headers say (see `Image::full_decode_planes_floor`, which parses +/// the tiles and so comes last). +/// +/// A planning error keeps the stripe-by-stripe decode, which meets the same +/// error and reports it. +pub(crate) fn rows_stream_whole(image: &Image<'_>, resolution_levels: u8) -> bool { + resolution_levels > 1 + && image.region_decode_streams_rows() + && (image.tile_count() == 1 + || image.output_tile_rects().is_ok_and(|rects| { + rects + .iter() + .all(|&(x, _, width, _)| x == 0 && width == image.width()) + })) + && matches!(image.full_decode_planes_floor(0), Ok(Some(_))) +} + +/// Why a streamed bounded row decode did not hand over every row. +pub(crate) enum StreamedRowsError { + /// The decode failed before any row reached the caller, who can decode + /// the image a stripe at a time instead and meet that decode's rows and + /// errors, as for an image that cannot stream. + BeforeFirstRow, + /// The caller's sink failed, or the decode failed after rows reached it. + Rows(DecodeRowsError), +} + +/// Hands one packed row, at its image row, to the caller. +pub(crate) type DeliverRow<'d, E> = dyn FnMut(u32, &[u8]) -> Result<(), E> + 'd; + +/// Writes rows `rows` of a rectangle's component planes to packed output. +type PackRows = for<'p> fn( + &ColorSpace, + bool, + &[j2k_native::ComponentPlane<'p>], + (u32, u32), + Range, + &mut [u8], + usize, + PixelFormat, +) -> Result<(), J2kError>; + +/// Decode the whole `image` in one streamed region decode, packing its rows +/// in `fmt` into `stripe`, which holds whole rows of `row_bytes` bytes, and +/// handing each full stripe, then the last, to `deliver` a row at a time. +/// +/// `image` must stream its rows in order (see [`rows_stream_whole`]); a +/// decode that falls back to component planes hands them over at the end, +/// and they are packed a stripe at a time too. The first error stops the +/// rows reaching `deliver`: a caller error is returned as such even though +/// the decode runs to its end, and a decode error after rows reached the +/// caller leaves the rows of every stripe before it delivered and the +/// stripe it failed in held back. +pub(crate) fn decode_image_rows_in_stripes<'a, E: core::error::Error + 'static>( + image: &Image<'a>, + native_context: &mut j2k_native::DecoderContext<'a>, + fmt: PixelFormat, + (stripe, row_bytes): (&mut [u8], usize), + deliver: &mut DeliverRow<'_, E>, +) -> Result<(), StreamedRowsError> { + let internal = + |what| StreamedRowsError::Rows(DecodeRowsError::Decode(J2kError::internal_backend(what))); + let pack: PackRows = match fmt { + PixelFormat::Rgb8 | PixelFormat::Rgba8 | PixelFormat::Gray8 => write_plane_rows_u8, + PixelFormat::Rgb16 | PixelFormat::Rgba16 | PixelFormat::Gray16 => write_plane_rows_u16, + _ => return Err(internal("bounded row decode format without a row writer")), + }; + let stripe_rows = stripe.len().checked_div(row_bytes).unwrap_or(0); + if stripe_rows == 0 { + return Err(internal("bounded row decode stripe holds no row")); + } + #[cfg(test)] + STREAMED_ROW_DECODES.with(|count| count.set(count.get() + 1)); + #[cfg(test)] + let injected = INJECTED_FAILURE.with(core::cell::Cell::take); + #[cfg(not(test))] + let skip_decode = false; + #[cfg(test)] + let skip_decode = injected == Some(InjectedFailure::BeforeRows); + let mut sink = StripeSink { + color_space: image.color_space(), + has_alpha: image.has_alpha(), + fmt, + pack, + width: image.width(), + stripe, + row_bytes, + stripe_rows, + first: 0, + held: 0, + deliver, + delivered: false, + error: None, + }; + let decoded = if skip_decode { + Ok(()) + } else { + image.decode_region_rows_for_integer_output_with_context( + (0, 0, image.width(), image.height()), + native_context, + &mut sink, + ) + }; + #[cfg(test)] + let decoded = match injected { + Some(_) => decoded.and(Err(j2k_native::DecodingError::CodeBlockDecodeFailure.into())), + None => decoded, + }; + if let Some(error) = sink.error.take() { + return Err(StreamedRowsError::Rows(error)); + } + if let Err(error) = decoded { + return Err(if sink.delivered { + StreamedRowsError::Rows(DecodeRowsError::Decode(J2kError::from_native_decode_error( + error, + ))) + } else { + StreamedRowsError::BeforeFirstRow + }); + } + sink.flush().map_err(StreamedRowsError::Rows)?; + if sink.first != image.height() { + return Err(internal("streamed region decode left rows undelivered")); + } + Ok(()) +} + +/// Packs a streamed region decode's rows into a stripe buffer and hands +/// each full stripe over. The first error stops every later row. +struct StripeSink<'s, E: core::error::Error + 'static> { + color_space: &'s ColorSpace, + has_alpha: bool, + fmt: PixelFormat, + pack: PackRows, + width: u32, + stripe: &'s mut [u8], + row_bytes: usize, + stripe_rows: usize, + /// The image row the stripe's first row holds. + first: u32, + /// The rows the stripe holds. + held: usize, + deliver: &'s mut DeliverRow<'s, E>, + /// Whether any row was handed to `deliver`. + delivered: bool, + error: Option>, +} + +impl StripeSink<'_, E> { + /// Pack a rectangle of whole rows from row `y`, which must be the next + /// row, delivering every stripe it fills. + fn take_rows( + &mut self, + x: u32, + y: u32, + planes: &[j2k_native::ComponentPlane<'_>], + ) -> Result<(), DecodeRowsError> { + let dims = planes + .first() + .map_or((0, 0), j2k_native::ComponentPlane::dimensions); + let next = u32::try_from(self.held) + .ok() + .and_then(|held| self.first.checked_add(held)); + if x != 0 || dims.0 != self.width || Some(y) != next { + return Err(DecodeRowsError::Decode(J2kError::internal_backend( + "streamed rows arrived out of order", + ))); + } + let rows = dims.1 as usize; + let mut row = 0; + while row < rows { + let take = (self.stripe_rows - self.held).min(rows - row); + let out = &mut self.stripe[self.held * self.row_bytes..]; + (self.pack)( + self.color_space, + self.has_alpha, + planes, + dims, + row..row + take, + out, + self.row_bytes, + self.fmt, + ) + .map_err(DecodeRowsError::Decode)?; + self.held += take; + row += take; + if self.held == self.stripe_rows { + self.flush()?; + } + } + Ok(()) + } + + /// Hand the rows the stripe holds over, a row at a time. + fn flush(&mut self) -> Result<(), DecodeRowsError> { + let mut y = self.first; + for row in self.stripe.chunks_exact(self.row_bytes).take(self.held) { + self.delivered = true; + (self.deliver)(y, row).map_err(DecodeRowsError::Sink)?; + y += 1; + } + self.first = y; + self.held = 0; + Ok(()) + } +} + +impl j2k_native::ComponentRowSink for StripeSink<'_, E> { + fn write_rows(&mut self, x: u32, y: u32, planes: &[j2k_native::ComponentPlane<'_>]) { + if self.error.is_some() { + return; + } + if let Err(error) = self.take_rows(x, y, planes) { + self.error = Some(error); + } + } +} diff --git a/crates/j2k/src/owned_batch.rs b/crates/j2k/src/owned_batch.rs index 593c02bf4..2e875afeb 100644 --- a/crates/j2k/src/owned_batch.rs +++ b/crates/j2k/src/owned_batch.rs @@ -14,7 +14,7 @@ use crate::{ backend, batch::{ allocation::{generic_batch_worker_limit, J2K_BATCH_METADATA_ALLOWANCE_BYTES}, - scheduler::{retained_inner_parallelism, run_retained_chunks}, + scheduler::{retained_inner_parallelism, retained_worker_count, run_retained_chunks}, worker::BatchWorker, }, decode::decode_warnings_for_recovery, @@ -55,7 +55,7 @@ use self::cpu_fast::CpuGroupFastWorkspace; mod cpu_staged; use self::cpu_staged::{CpuEntropyOutcome, CpuStagedWorkspace, CPU_ENTROPY_IMAGES_PER_WINDOW}; mod cpu_execute; -use self::cpu_execute::decode_cpu_group; +use self::cpu_execute::{decode_cpu_group, CpuStreamPool}; mod cpu; mod cpu_materialize; mod cpu_staged_execute; diff --git a/crates/j2k/src/owned_batch/cpu.rs b/crates/j2k/src/owned_batch/cpu.rs index 81115c491..bc36a6d66 100644 --- a/crates/j2k/src/owned_batch/cpu.rs +++ b/crates/j2k/src/owned_batch/cpu.rs @@ -6,8 +6,9 @@ use super::{ decode_cpu_group, fmt, generic_batch_worker_limit, prepare_batch_from_images_for, prepare_batch_with_workers, BatchDecodeOptions, BatchDecoder, BatchGroupInfo, BatchInfrastructureError, BatchWorker, CpuBatchWorkspaceStats, CpuGroupFastWorkspace, - CpuStagedWorkspace, EncodedImage, IndexedBatchError, J2kDecodeWarning, Mutex, NativeSampleType, - NonZeroUsize, PrepareFor, PreparedBatch, PreparedImage, Rect, Vec, + CpuStagedWorkspace, CpuStreamPool, EncodedImage, IndexedBatchError, J2kDecodeWarning, Mutex, + NativeSampleType, NonZeroUsize, PrepareFor, PreparedBatch, PreparedBatchGroup, PreparedImage, + Rect, Vec, }; use crate::batch::admission::BatchAllocationBudget; use crate::batch::allocation::try_vec_with_capacity; @@ -174,6 +175,7 @@ pub struct CpuBatchDecoder { workers: Mutex>, fast_workspace: CpuGroupFastWorkspace, staged_workspace: CpuStagedWorkspace, + stream_pool: Mutex, } impl fmt::Debug for CpuBatchDecoder { @@ -202,6 +204,7 @@ impl CpuBatchDecoder { workers: Mutex::new(workers), fast_workspace: CpuGroupFastWorkspace::default(), staged_workspace: CpuStagedWorkspace::default(), + stream_pool: Mutex::default(), } } @@ -211,6 +214,27 @@ impl CpuBatchDecoder { self.options } + /// Threads of the session's streamed-group pool, once a group used it. + #[cfg(test)] + pub(super) fn stream_pool_threads(&mut self) -> Option { + self.stream_pool + .get_mut() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .threads() + } + + /// Offset-plan executions of this session's workers with nested + /// parallelism. + #[cfg(test)] + pub(super) fn parallel_plan_decodes(&mut self) -> u64 { + self.workers + .get_mut() + .unwrap_or_else(std::sync::PoisonError::into_inner) + .iter() + .map(BatchWorker::parallel_plan_decode_calls) + .sum() + } + /// Number of reusable worker context/scratch owners retained by this session. #[must_use] pub fn retained_worker_count(&self) -> usize { @@ -323,8 +347,8 @@ impl CpuBatchDecoder { /// /// Inputs whose output allocation or planned execution exceeds a memory /// limit are recorded as indexed preparation errors in the returned batch. - /// Preparation checks individual images; a group's combined output - /// allocation is checked during decoding. + /// Preparation checks individual images. During decoding, a group too + /// large for one output allocation is split into smaller result groups. pub fn prepare( &self, inputs: Vec, @@ -373,6 +397,10 @@ impl CpuBatchDecoder { .workers .get_mut() .map_err(|_| BatchInfrastructureError::SchedulerPoisoned)?; + let stream_pool = self + .stream_pool + .get_mut() + .map_err(|_| BatchInfrastructureError::SchedulerPoisoned)?; let input_count = prepared .groups() .iter() @@ -392,17 +420,59 @@ impl CpuBatchDecoder { // plan, under the same aggregate cap as the tile-batch facade. let admission = BatchAllocationBudget::with_baseline(0)?; for group in prepared.groups() { - if let Some(decoded) = decode_cpu_group( - workers, - &mut self.fast_workspace, - &mut self.staged_workspace, - group, - prepared.options, - self.options.workers, - &admission, - &mut errors, - )? { - groups.push(decoded); + let images_per_group = output_images_per_group(group); + if images_per_group >= group.images.len() { + if let Some(decoded) = decode_cpu_group( + workers, + &mut self.fast_workspace, + &mut self.staged_workspace, + stream_pool, + group, + prepared.options, + self.options.workers, + &admission, + &mut errors, + )? { + push_group(&mut groups, decoded)?; + } + continue; + } + // One dense buffer for the whole group would exceed the + // per-allocation cap: decode consecutive images as several + // groups with the same info, each buffer within the cap. + for (images, source_indices) in group + .images + .chunks(images_per_group) + .zip(group.source_indices.chunks(images_per_group)) + { + let mut part_images = + try_vec_with_capacity(images.len(), "J2K owned batch output group images")?; + part_images.extend(images.iter().cloned()); + let mut part_indices = try_vec_with_capacity( + source_indices.len(), + "J2K owned batch output group source indices", + )?; + part_indices.extend_from_slice(source_indices); + let part = PreparedBatchGroup { + info: group.info.clone(), + options: group.options, + execution_shape: group.execution_shape, + images: part_images, + source_indices: part_indices, + }; + if let Some(decoded) = decode_cpu_group( + workers, + &mut self.fast_workspace, + &mut self.staged_workspace, + stream_pool, + &part, + prepared.options, + self.options.workers, + &admission, + &mut errors, + )? { + push_group(&mut groups, decoded)?; + } } } errors.sort_by_key(|error| error.index); @@ -410,6 +480,44 @@ impl CpuBatchDecoder { } } +/// Images whose dense outputs fit one per-allocation-capped buffer together +/// (at least one). An image whose own output exceeds the cap still fails in +/// its own group, as before. +fn output_images_per_group(group: &PreparedBatchGroup) -> usize { + let sample_bytes = match group.info.sample_type { + NativeSampleType::U8 => 1, + NativeSampleType::U16 | NativeSampleType::I16 => 2, + _ => return group.images.len(), + }; + group + .info + .samples_per_image() + .and_then(|samples| samples.checked_mul(sample_bytes)) + .map_or(group.images.len(), |image_bytes| { + images_within_cap(image_bytes, j2k_core::DEFAULT_MAX_HOST_ALLOCATION_BYTES) + }) +} + +/// How many images of `image_bytes` one buffer of at most `cap` bytes holds, +/// never fewer than one. +fn images_within_cap(image_bytes: usize, cap: usize) -> usize { + cap.checked_div(image_bytes).unwrap_or(usize::MAX).max(1) +} + +fn push_group( + groups: &mut Vec, + group: CpuBatchGroup, +) -> Result<(), BatchInfrastructureError> { + groups + .try_reserve(1) + .map_err(|_| BatchInfrastructureError::HostAllocationFailed { + what: "J2K owned batch result groups", + bytes: core::mem::size_of::(), + })?; + groups.push(group); + Ok(()) +} + impl BatchDecoder for CpuBatchDecoder { type Output = CpuBatchDecodeResult; type Error = BatchInfrastructureError; @@ -433,3 +541,20 @@ impl BatchDecoder for CpuBatchDecoder { Self::decode_prepared(self, prepared) } } + +#[cfg(test)] +mod output_group_tests { + use super::images_within_cap; + + #[test] + fn groups_above_the_cap_split_into_buffers_within_it() { + const MIB: usize = 1024 * 1024; + // Eight 5792² RGB8 images (96 MiB each) under a 512 MiB cap. + assert_eq!(images_within_cap(96 * MIB, 512 * MIB), 5); + assert_eq!(images_within_cap(512 * MIB, 512 * MIB), 1); + // An image above the cap keeps a group of its own. + assert_eq!(images_within_cap(768 * MIB, 512 * MIB), 1); + assert_eq!(images_within_cap(3 * MIB, 512 * MIB), 170); + assert_eq!(images_within_cap(0, 512 * MIB), usize::MAX); + } +} diff --git a/crates/j2k/src/owned_batch/cpu_execute.rs b/crates/j2k/src/owned_batch/cpu_execute.rs index 9674a0b1d..7b87c7967 100644 --- a/crates/j2k/src/owned_batch/cpu_execute.rs +++ b/crates/j2k/src/owned_batch/cpu_execute.rs @@ -2,17 +2,19 @@ //! CPU homogeneous-group execution and final materialization. +use super::cpu_fast::classic_staging_fits; use super::cpu_materialize::{ convert_u16, convert_u8, decode_image_i16, decode_image_u16, decode_image_u8, - ensure_batch_output_within_cap, round_signed, try_zeroed_vec, + decode_lone_gray_u8, ensure_batch_output_within_cap, round_signed, try_zeroed_vec, }; use super::cpu_staged_execute::run_staged_typed_group; use super::{ - decode_warnings_for_recovery, retained_inner_parallelism, run_retained_chunks, Arc, - BatchDecodeOptions, BatchErrorStage, BatchGroupInfo, BatchInfrastructureError, BatchItemError, - BatchWorker, CpuBatchGroup, CpuBatchSamples, CpuDecodeParallelism, CpuGroupFastWorkspace, - CpuStagedWorkspace, IndexedBatchError, J2kError, NativeSampleType, NonZeroUsize, - PreparedBatchGroup, PreparedImage, TileBatchOptions, Vec, + decode_warnings_for_recovery, retained_inner_parallelism, retained_worker_count, + run_retained_chunks, Arc, BatchCodecRoute, BatchColor, BatchDecodeOptions, BatchErrorStage, + BatchGroupInfo, BatchInfrastructureError, BatchItemError, BatchWorker, CpuBatchGroup, + CpuBatchSamples, CpuDecodeParallelism, CpuGroupFastWorkspace, CpuStagedWorkspace, + IndexedBatchError, J2kError, NativeSampleType, NonZeroUsize, PreparedBatchGroup, PreparedImage, + TileBatchOptions, Vec, }; use crate::batch::admission::{hold_at_least, BatchAllocationBudget}; use crate::batch::allocation::try_vec_with_capacity; @@ -26,6 +28,7 @@ pub(super) fn decode_cpu_group( workers: &mut [BatchWorker], fast_workspace: &mut CpuGroupFastWorkspace, staged_workspace: &mut CpuStagedWorkspace, + stream_pool: &mut CpuStreamPool, group: &PreparedBatchGroup, options: BatchDecodeOptions, workers_option: Option, @@ -36,10 +39,30 @@ pub(super) fn decode_cpu_group( // The staged route flattens entropy work across images for a few serial // workers. When workers may parallelize inside one image, each prepared // plan decodes faster on its own: its code-block rows and wavelet bands - // spread over every core, still without reparsing. - let has_flattened_payloads = fast_workspace.prepare_group(group)? + // spread over every core, still without reparsing. A group with an image + // whose rows stream never stages: streamed images hold a band of rows + // instead of whole-image scratch, at every worker count. + let streams_rows = group + .images + .iter() + .any(|image| image.cpu_execution().streams()); + // Payloads are flattened only for the staged route, and only within the + // flattening and staged-window caps (see `classic_staging_fits`): a + // classic group beyond them decodes its plans one image at a time + // instead of failing. + let staged = !streams_rows && retained_inner_parallelism(group.images.len(), tile_options)? - == CpuDecodeParallelism::Serial; + == CpuDecodeParallelism::Serial + && (group.info.route != BatchCodecRoute::Classic + || classic_staging_fits(group.images.iter())?) + && fast_workspace.prepare_group(group)?; + let (pooled, image_parallelism) = + image_policy(group.images.len(), streams_rows, workers_option); + let stream_pool = if pooled { + Some(stream_pool.get(workers.len())?) + } else { + None + }; let samples_per_image = group .info @@ -68,10 +91,17 @@ pub(super) fn decode_cpu_group( tile_options, samples_per_image, &mut values, - has_flattened_payloads.then_some(&mut *fast_workspace), + staged.then_some(&mut *fast_workspace), staged_workspace, + stream_pool, + image_parallelism, admission, - decode_image_u8, + // The only image of a Gray8 group is the whole in-flight set. + if group.images.len() == 1 && group.info.color == BatchColor::Gray { + decode_lone_gray_u8 + } else { + decode_image_u8 + }, convert_u8, errors, )?; @@ -90,8 +120,10 @@ pub(super) fn decode_cpu_group( tile_options, samples_per_image, &mut values, - has_flattened_payloads.then_some(&mut *fast_workspace), + staged.then_some(&mut *fast_workspace), staged_workspace, + stream_pool, + image_parallelism, admission, decode_image_u16, convert_u16, @@ -112,8 +144,10 @@ pub(super) fn decode_cpu_group( tile_options, samples_per_image, &mut values, - has_flattened_payloads.then_some(&mut *fast_workspace), + staged.then_some(&mut *fast_workspace), staged_workspace, + stream_pool, + image_parallelism, admission, decode_image_i16, round_signed, @@ -157,6 +191,89 @@ pub(super) fn decode_cpu_group( ))) } +/// How the images of a group that decode one at a time run: in the stream +/// pool, and with which inner parallelism otherwise (`None`: the +/// scheduler's own). With an explicit worker count and several images, +/// streamed images decode in the pool of that many threads (two or more) and +/// plans serially, also on the worker itself, where offset-plan executions +/// read it; a single worker decodes serially. +pub(super) fn image_policy( + image_count: usize, + streamed: bool, + workers: Option, +) -> (bool, Option) { + let explicit_group = image_count > 1 && workers.is_some(); + let pooled = explicit_group && streamed && workers.is_some_and(|count| count.get() > 1); + ( + pooled, + explicit_group.then_some(CpuDecodeParallelism::Serial), + ) +} + +/// Whether the images of a group of `job_count` that decode on a stream pool +/// of `pool_threads` share it (see `DecoderContext::set_shared_pool`): the +/// batch plans at least as many workers at once as the pool has threads, so +/// each thread starts with an image of its own and an idle one takes the +/// bands of other images still in flight. The images that finish last keep +/// those rings with fewer images beside them. With fewer workers planned +/// than threads, each image holds the rings of a decode alone on the pool. +pub(super) fn shares_stream_pool( + job_count: usize, + tile_options: TileBatchOptions, + pool_threads: usize, +) -> Result { + Ok(retained_worker_count(job_count, tile_options)? >= pool_threads) +} + +/// The encoded bytes up to which an image of a group that shares its stream +/// pool holds the shared rings, from the encoded lengths of the group's +/// images: one and a half times their mean. Tier-1 work follows the coded +/// bytes, so a larger image is the likeliest to decode last with few other +/// images beside it, and keeps the rings of a decode alone on the pool. +pub(super) fn shared_ring_bytes(lengths: impl ExactSizeIterator) -> usize { + let count = lengths.len().max(1); + let mean = lengths.fold(0_usize, usize::saturating_add) / count; + mean.saturating_add(mean / 2) +} + +/// The threads a group's streamed images decode on when the caller set the +/// worker count, created on first use and kept by the session. +/// +/// A streamed image hands its Tier-1 bands to the rayon pool it decodes in. +/// An explicit worker count used to keep each image serial on its worker +/// thread instead, so `M` images on `N` workers kept +/// `ceil(M / ceil(M / N))` threads busy (8 images on 6 workers: 4) and the +/// last images alone at the end. In a pool of exactly `N` threads idle +/// threads take bands of the images still in flight, so the group keeps all +/// `N` busy and never runs more: the worker threads only wait for their +/// images, and the images in flight stay one per worker. +#[derive(Default)] +pub(super) struct CpuStreamPool { + pool: Option, +} + +impl CpuStreamPool { + fn get(&mut self, threads: usize) -> Result<&rayon::ThreadPool, BatchInfrastructureError> { + let pool = match self.pool.take() { + Some(pool) if pool.current_num_threads() == threads => pool, + _ => rayon::ThreadPoolBuilder::new() + .num_threads(threads) + .thread_name(|index| format!("j2k-batch-stream-{index}")) + .build() + .map_err(|_| BatchInfrastructureError::WorkerSpawnFailed { worker: 0 })?, + }; + Ok(self.pool.insert(pool)) + } + + /// Threads of the pool, once a group has used it. + #[cfg(test)] + pub(super) fn threads(&self) -> Option { + self.pool + .as_ref() + .map(rayon::ThreadPool::current_num_threads) + } +} + pub(super) struct CpuTypedDecodeJob<'image, 'output, T> { pub(super) slot: usize, pub(super) image: &'image PreparedImage, @@ -187,6 +304,8 @@ fn run_typed_group( output: &mut Vec, flattened_payloads: Option<&mut CpuGroupFastWorkspace>, staged_workspace: &mut CpuStagedWorkspace, + stream_pool: Option<&rayon::ThreadPool>, + image_parallelism: Option, admission: &Arc, decode: CpuDecodeFn, convert: fn(f32, u8) -> T, @@ -220,31 +339,56 @@ fn run_typed_group( convert, )?; } else { + let shared_pool = match stream_pool { + Some(pool) => shares_stream_pool(jobs.len(), tile_options, pool.current_num_threads())?, + None => false, + }; + let shared_bytes = shared_ring_bytes(group.images.iter().map(|image| image.bytes().len())); run_retained_chunks( workers, &mut jobs, &mut results, tile_options, |worker, jobs, results| { + // On the worker itself, where offset-plan executions read it. + if let Some(parallelism) = image_parallelism { + worker.set_owned_inner_parallelism(parallelism); + } let parallelism = worker.prepare_owned_decode(); let workspace = worker.take_native_workspace(); let mut context = j2k_native::DecoderContext::from_workspace(workspace); let mut claim = None; for (job, slot) in jobs.iter_mut().zip(results) { + context.set_shared_pool(shared_pool && job.image.bytes().len() <= shared_bytes); hold_at_least( admission, &mut claim, job.image.cpu_execution().claim_bytes(), )?; - *slot = Some(decode( - job.image, - options, - &group.info, - parallelism, - &mut context, - worker, - job.output, - )); + *slot = Some(match stream_pool { + // This thread waits while a pool thread decodes the + // image and idle ones take its bands. + Some(pool) => pool.install(|| { + decode( + job.image, + options, + &group.info, + CpuDecodeParallelism::Auto, + &mut context, + worker, + job.output, + ) + }), + None => decode( + job.image, + options, + &group.info, + parallelism, + &mut context, + worker, + job.output, + ), + }); } worker.restore_native_workspace(context.into_workspace()); Ok(()) diff --git a/crates/j2k/src/owned_batch/cpu_fast.rs b/crates/j2k/src/owned_batch/cpu_fast.rs index 9b344ee91..b57861270 100644 --- a/crates/j2k/src/owned_batch/cpu_fast.rs +++ b/crates/j2k/src/owned_batch/cpu_fast.rs @@ -10,14 +10,89 @@ use j2k_native::{ J2kDirectCodeBlockIndex, }; +use super::cpu_staged::{staged_window_fits, CPU_ENTROPY_IMAGES_PER_WINDOW}; use super::{BatchCodecRoute, PreparedBatchGroup, PreparedImage}; -use crate::batch::allocation::J2K_BATCH_METADATA_ALLOWANCE_BYTES; +use crate::batch::allocation::{try_vec_with_capacity, J2K_BATCH_METADATA_ALLOWANCE_BYTES}; mod classic; mod ht; mod plan; use self::plan::{checked_metadata_bytes, reserve_reused}; +#[cfg(test)] +std::thread_local! { + /// Test-only cap on a group's flattened metadata, on this thread. + pub(super) static FLATTENED_METADATA_CAP_OVERRIDE: core::cell::Cell> = + const { core::cell::Cell::new(None) }; +} + +fn flattened_metadata_cap() -> usize { + #[cfg(test)] + if let Some(cap) = FLATTENED_METADATA_CAP_OVERRIDE.with(core::cell::Cell::get) { + return cap; + } + J2K_BATCH_METADATA_ALLOWANCE_BYTES +} + +/// The flattened metadata bytes of a classic group of `image_count` images +/// and `payload_count` payloads, as `prepare_group` charges them. +#[cfg(test)] +pub(super) fn classic_flattened_metadata_bytes(image_count: usize, payload_count: usize) -> usize { + checked_metadata_bytes::<(J2kClassicCodeBlockPayload, J2kCodestreamRange)>( + image_count, + payload_count, + ) + .unwrap_or(usize::MAX) +} + +/// Whether flattening `images`, a classic group for the staged route in its +/// order, stays within the caps `CpuGroupFastWorkspace::prepare_group` and +/// the staged windows enforce: the flattened metadata and compressed arena +/// of the whole group, and the entropy jobs of its widest window, counted as +/// if a window's code-blocks shared one tile. A group that does not fit +/// decodes its plans one image at a time instead, which needs none of them. +/// +/// # Errors +/// Only when the per-image counts cannot be allocated. +pub(super) fn classic_staging_fits<'a>( + images: impl ExactSizeIterator, +) -> Result { + let image_count = images.len(); + let mut counts = try_vec_with_capacity(image_count, "J2K CPU staged payload counts")?; + let mut payload_bytes = 0usize; + for image in images { + let Some(plan) = image.classic_plan() else { + return Ok(false); + }; + counts.push(plan.payload_count()); + for payload in plan.cpu_plan().payloads() { + payload_bytes = payload_bytes.saturating_add(payload.combined_length); + } + } + let payload_count = counts + .iter() + .fold(0usize, |total, &count| total.saturating_add(count)); + let window = CPU_ENTROPY_IMAGES_PER_WINDOW.min(image_count); + let widest_window = counts + .windows(window.max(1)) + .map(|counts| { + counts + .iter() + .fold(0usize, |total, &count| total.saturating_add(count)) + }) + .max() + .unwrap_or(0); + Ok( + checked_metadata_bytes::<(J2kClassicCodeBlockPayload, J2kCodestreamRange)>( + image_count, + payload_count, + ) + .is_ok_and(|bytes| bytes <= flattened_metadata_cap()) + && payload_bytes <= DEFAULT_MAX_HOST_ALLOCATION_BYTES + && staged_window_fits(window, widest_window), + ) +} + #[derive(Debug, Clone, Copy, PartialEq, Eq)] enum CpuPayloadBucket { Cleanup, @@ -183,11 +258,11 @@ impl CpuGroupFastWorkspace { payload_bytes: usize, ) -> Result<(), BatchInfrastructureError> { let metadata_bytes = checked_metadata_bytes::(image_count, payload_count)?; - if metadata_bytes > J2K_BATCH_METADATA_ALLOWANCE_BYTES { + if metadata_bytes > flattened_metadata_cap() { return Err(BatchInfrastructureError::AllocationTooLarge { what: "J2K CPU flattened group metadata", requested: metadata_bytes, - cap: J2K_BATCH_METADATA_ALLOWANCE_BYTES, + cap: flattened_metadata_cap(), }); } if payload_bytes > DEFAULT_MAX_HOST_ALLOCATION_BYTES { diff --git a/crates/j2k/src/owned_batch/cpu_materialize.rs b/crates/j2k/src/owned_batch/cpu_materialize.rs index f9dda639d..2b64f1e8e 100644 --- a/crates/j2k/src/owned_batch/cpu_materialize.rs +++ b/crates/j2k/src/owned_batch/cpu_materialize.rs @@ -91,9 +91,48 @@ pub(super) fn decode_image_u8<'image>( worker: &mut BatchWorker, out: &mut [u8], ) -> Result<(), J2kError> { - if image.cpu_execution() == CpuExecution::ExceedsDecodeBound { - return Err(CpuExecution::decode_bound_error()); + match image.cpu_execution() { + CpuExecution::ExceedsDecodeBound => Err(CpuExecution::decode_bound_error()), + CpuExecution::StreamedRows | CpuExecution::StreamedRowsOrPlan { .. } => { + decode_streamed_samples(image, options, info, parallelism, context, out, convert_u8) + } + CpuExecution::OffsetPlan { .. } | CpuExecution::ComponentPlanes => { + decode_planned_u8(image, options, info, parallelism, context, worker, out) + } + } +} + +/// [`decode_image_u8`] for the only image of a Gray8 group: an image that may +/// stream or decode its plan decodes its plan, which spreads one component's +/// code-block rows over every core while nothing else is in flight. +pub(super) fn decode_lone_gray_u8<'image>( + image: &'image PreparedImage, + options: BatchDecodeOptions, + info: &BatchGroupInfo, + parallelism: CpuDecodeParallelism, + context: &mut j2k_native::DecoderContext<'image>, + worker: &mut BatchWorker, + out: &mut [u8], +) -> Result<(), J2kError> { + if matches!( + image.cpu_execution(), + CpuExecution::StreamedRowsOrPlan { .. } + ) { + decode_planned_u8(image, options, info, parallelism, context, worker, out) + } else { + decode_image_u8(image, options, info, parallelism, context, worker, out) } +} + +fn decode_planned_u8<'image>( + image: &'image PreparedImage, + options: BatchDecodeOptions, + info: &BatchGroupInfo, + parallelism: CpuDecodeParallelism, + context: &mut j2k_native::DecoderContext<'image>, + worker: &mut BatchWorker, + out: &mut [u8], +) -> Result<(), J2kError> { if decode_prepared_plan_samples(image, info, worker, out, |sample, precision| { u8::try_from(round_unsigned(sample, precision)).unwrap_or(u8::MAX) })? { @@ -153,6 +192,141 @@ pub(super) fn decode_image_i16<'image>( Ok(()) } +/// Decode `image` through the bounded region decode, which hands over its +/// rows as they are finished, writing each sample exactly as +/// [`write_samples`] writes whole planes. Working memory stays bounded by the +/// streamed decode instead of growing with the image. +fn decode_streamed_samples<'image, T: Copy>( + image: &'image PreparedImage, + options: BatchDecodeOptions, + info: &BatchGroupInfo, + parallelism: CpuDecodeParallelism, + context: &mut j2k_native::DecoderContext<'image>, + out: &mut [T], + convert: fn(f32, u8) -> T, +) -> Result<(), J2kError> { + context.set_cpu_decode_parallelism(parallelism.to_native()); + let plan = image.plan(); + let decoded = backend::image_at_scale(image.bytes(), options.settings, plan.scale())?; + let output_rect = plan.output_rect(); + if (output_rect.w, output_rect.h) != info.dimensions { + return Err(J2kError::internal_backend( + "prepared batch output metadata changed during decode", + )); + } + let mut sink = SampleRowSink::new(info, out, convert)?; + decoded + .decode_region_rows_for_integer_output_with_context( + (output_rect.x, output_rect.y, output_rect.w, output_rect.h), + context, + &mut sink, + ) + .map_err(J2kError::from_native_decode_error)?; + sink.finish() +} + +/// Writes streamed region rectangles into one image's dense batch samples. +struct SampleRowSink<'out, 'info, T> { + info: &'info BatchGroupInfo, + out: &'out mut [T], + convert: fn(f32, u8) -> T, + pixels: usize, + written_pixels: usize, + malformed: bool, +} + +impl<'out, 'info, T: Copy> SampleRowSink<'out, 'info, T> { + fn new( + info: &'info BatchGroupInfo, + out: &'out mut [T], + convert: fn(f32, u8) -> T, + ) -> Result { + let pixels = (info.dimensions.0 as usize) + .checked_mul(info.dimensions.1 as usize) + .ok_or_else(|| J2kError::internal_backend("prepared batch sample count overflow"))?; + if pixels.checked_mul(info.color.channels()) != Some(out.len()) { + return Err(J2kError::internal_backend( + "prepared batch output slot does not match its image", + )); + } + Ok(Self { + info, + out, + convert, + pixels, + written_pixels: 0, + malformed: false, + }) + } + + /// Every pixel arrived exactly once (rectangles never overlap). + fn finish(self) -> Result<(), J2kError> { + if self.malformed || self.written_pixels != self.pixels { + return Err(J2kError::internal_backend( + "streamed batch decode did not cover its output exactly", + )); + } + Ok(()) + } + + fn write_rectangle(&mut self, x: u32, y: u32, planes: &[j2k_native::ComponentPlane<'_>]) { + let channels = self.info.color.channels(); + let width = self.info.dimensions.0 as usize; + let Some(rect_dimensions) = planes.first().map(j2k_native::ComponentPlane::dimensions) + else { + self.malformed = true; + return; + }; + let (x, y, rect_width, rect_height) = ( + x as usize, + y as usize, + rect_dimensions.0 as usize, + rect_dimensions.1 as usize, + ); + let rect_pixels = rect_width * rect_height; + if planes.len() != channels + || x + rect_width > width + || y + rect_height > self.info.dimensions.1 as usize + || planes.iter().any(|plane| { + plane.dimensions() != rect_dimensions || plane.samples().len() < rect_pixels + }) + { + self.malformed = true; + return; + } + let precision = self.info.precision; + let planar = self.info.layout == BatchLayout::Nchw || channels == 1; + for (channel, plane) in planes.iter().enumerate() { + for (row, samples) in plane.samples()[..rect_pixels] + .chunks_exact(rect_width) + .enumerate() + { + let pixel = (y + row) * width + x; + if planar { + let destination = &mut self.out[channel * self.pixels + pixel..][..rect_width]; + for (destination, &sample) in destination.iter_mut().zip(samples) { + *destination = (self.convert)(sample, precision); + } + } else { + let destination = &mut self.out[pixel * channels..][..rect_width * channels]; + for (pixel, &sample) in destination.chunks_exact_mut(channels).zip(samples) { + pixel[channel] = (self.convert)(sample, precision); + } + } + } + } + self.written_pixels += rect_pixels; + } +} + +impl j2k_native::ComponentRowSink for SampleRowSink<'_, '_, T> { + fn write_rows(&mut self, x: u32, y: u32, planes: &[j2k_native::ComponentPlane<'_>]) { + if !self.malformed { + self.write_rectangle(x, y, planes); + } + } +} + fn validate_decoded_components( components: &j2k_native::DecodedComponents<'_>, info: &BatchGroupInfo, diff --git a/crates/j2k/src/owned_batch/cpu_staged.rs b/crates/j2k/src/owned_batch/cpu_staged.rs index 940c966dc..6f408032f 100644 --- a/crates/j2k/src/owned_batch/cpu_staged.rs +++ b/crates/j2k/src/owned_batch/cpu_staged.rs @@ -130,6 +130,12 @@ impl CpuStagedWorkspace { } } +/// Whether a staged window of `image_count` images and `entropy_count` +/// entropy jobs in one tile fits the staged metadata cap. +pub(super) fn staged_window_fits(image_count: usize, entropy_count: usize) -> bool { + ensure_metadata_capacity(image_count, entropy_count).is_ok() +} + fn ensure_metadata_capacity( image_count: usize, entropy_count: usize, diff --git a/crates/j2k/src/owned_batch/cpu_staged_execute.rs b/crates/j2k/src/owned_batch/cpu_staged_execute.rs index a280aaab3..8d0d02072 100644 --- a/crates/j2k/src/owned_batch/cpu_staged_execute.rs +++ b/crates/j2k/src/owned_batch/cpu_staged_execute.rs @@ -47,9 +47,12 @@ pub(super) fn run_staged_typed_group( let mut window_start = 0; while window_start < image_jobs.len() { let (window_end, window_bytes) = staged_window_len( - image_jobs[window_start..] - .iter() - .map(|job| job.image.cpu_execution().claim_bytes()), + image_jobs[window_start..].iter().map(|job| { + let execution = job.image.cpu_execution(); + execution + .plan_bytes() + .unwrap_or_else(|| execution.claim_bytes()) + }), window_budget, ); let window_end = window_start + window_end; diff --git a/crates/j2k/src/owned_batch/prepare/cpu_preflight_tests.rs b/crates/j2k/src/owned_batch/prepare/cpu_preflight_tests.rs index 0395bb08d..a0aef92e1 100644 --- a/crates/j2k/src/owned_batch/prepare/cpu_preflight_tests.rs +++ b/crates/j2k/src/owned_batch/prepare/cpu_preflight_tests.rs @@ -4,26 +4,66 @@ use alloc::{sync::Arc, vec, vec::Vec}; use core::num::NonZeroUsize; use super::super::{ - prepare_batch, BatchDecodeOptions, BatchErrorStage, BatchItemError, CpuBatchDecoder, - CpuBatchSamples, CpuExecution, EncodedImage, IndexedBatchError, PreparedBatch, + prepare_batch, BatchDecodeOptions, BatchErrorStage, BatchItemError, BatchLayout, + CpuBatchDecoder, CpuBatchSamples, CpuDecodeParallelism, CpuExecution, DecodeRequest, + EncodedImage, IndexedBatchError, PreparedBatch, }; -use super::image::PLANNED_DECODE_BOUND; +use super::image::{PLANNED_DECODE_BOUND, STREAM_ABOVE_OVERRIDE, STREAM_ABOVE_PLANNED_BYTES}; +use j2k_core::Downscale; -fn classic_rgb8(width: u32, height: u32) -> Arc<[u8]> { - let samples = (0..width * height * 3) - .map(|index| u8::try_from((index * 7 + index / 97) & 0xff).expect("masked sample")) - .collect::>(); +fn classic_fixture(width: u32, height: u32, components: u16, precision: u8) -> Arc<[u8]> { + fixture(width, height, components, precision, false) +} + +fn fixture(width: u32, height: u32, components: u16, precision: u8, ht: bool) -> Arc<[u8]> { + seeded_fixture(width, height, components, precision, ht, 0) +} + +/// A fixture whose samples differ for every `seed`: the same size, other +/// coded bytes. +fn seeded_fixture( + width: u32, + height: u32, + components: u16, + precision: u8, + ht: bool, + seed: u32, +) -> Arc<[u8]> { + let count = width * height * u32::from(components); + let samples = if precision <= 8 { + (0..count) + .map(|index| { + u8::try_from((index * 7 + index / 97 + seed * index / 13) & 0xff) + .expect("masked sample") + }) + .collect::>() + } else { + (0..count) + .flat_map(|index| { + u16::try_from((index * 977 + 31) % (1 << precision)) + .expect("masked sample") + .to_le_bytes() + }) + .collect() + }; let options = j2k_native::EncodeOptions { reversible: true, num_decomposition_levels: 3, + use_ht_block_coding: ht, ..j2k_native::EncodeOptions::default() }; Arc::from( - j2k_native::encode(&samples, width, height, 3, 8, false, &options) - .expect("encode classic RGB8 fixture"), + j2k_native::encode( + &samples, width, height, components, precision, false, &options, + ) + .expect("encode classic fixture"), ) } +fn classic_rgb8(width: u32, height: u32) -> Arc<[u8]> { + classic_fixture(width, height, 3, 8) +} + /// One worker keeps preparation and decode on this thread, where the /// planned-bound override applies. fn single_worker_options() -> BatchDecodeOptions { @@ -48,6 +88,21 @@ impl Drop for PlannedBound { } } +struct StreamAbove; + +impl StreamAbove { + fn hold(bytes: usize) -> Self { + STREAM_ABOVE_OVERRIDE.with(|threshold| threshold.set(Some(bytes))); + Self + } +} + +impl Drop for StreamAbove { + fn drop(&mut self) { + STREAM_ABOVE_OVERRIDE.with(|threshold| threshold.set(None)); + } +} + fn planned_bytes(prepared: &PreparedBatch) -> Vec<(usize, CpuExecution)> { let mut planned = prepared .groups() @@ -79,25 +134,87 @@ fn is_decode_bound_error(error: &IndexedBatchError, index: usize, stage: BatchEr ) } +fn single_worker_options_with(layout: BatchLayout) -> BatchDecodeOptions { + let mut options = single_worker_options(); + options.layout = layout; + options +} + +fn all_u8_samples(result: &super::super::CpuBatchDecodeResult) -> Vec<(Vec, Vec)> { + assert!(result.errors().is_empty(), "{:?}", result.errors()); + let mut groups = result + .groups() + .iter() + .map(|group| { + let CpuBatchSamples::U8(samples) = group.samples() else { + panic!("8-bit decodes to u8 samples"); + }; + (group.source_indices().to_vec(), samples.clone()) + }) + .collect::>(); + groups.sort(); + groups +} + #[test] -fn cpu_prepare_rejects_plans_over_the_decode_bound_before_decoding() { +fn cpu_batch_streams_rgb8_and_gray8_plans_over_the_decode_bound() { let small = classic_rgb8(16, 16); let large = classic_rgb8(96, 64); + let gray = classic_fixture(80, 72, 1, 8); let inputs = || { vec![ EncodedImage::full(Arc::clone(&small)), EncodedImage::full(Arc::clone(&large)), + EncodedImage::full(Arc::clone(&gray)), + EncodedImage::new( + Arc::clone(&large), + DecodeRequest::Reduced { + scale: Downscale::Half, + }, + ), + ] + }; + for layout in [BatchLayout::Nchw, BatchLayout::Nhwc] { + let mut decoder = CpuBatchDecoder::new(single_worker_options_with(layout)); + let unbounded = decoder.prepare(inputs()).expect("prepare without a bound"); + let planned = planned_bytes(&unbounded); + let CpuExecution::OffsetPlan { bytes: small_bytes } = planned[0].1 else { + panic!("the small image keeps its offset plan: {planned:?}"); + }; + let reference = all_u8_samples( + &decoder + .decode_prepared(&unbounded) + .expect("decode without a bound"), + ); + + let _bound = PlannedBound::hold(small_bytes); + let bounded = decoder.prepare(inputs()).expect("prepare under the bound"); + assert!(bounded.errors().is_empty(), "{:?}", bounded.errors()); + let planned = planned_bytes(&bounded); + assert!(matches!(planned[0].1, CpuExecution::OffsetPlan { .. })); + for (index, execution) in &planned[1..] { + assert_eq!(*execution, CpuExecution::StreamedRows, "input {index}"); + } + let streamed = all_u8_samples(&decoder.decode_prepared(&bounded).expect("decode")); + assert_eq!(streamed, reference, "{layout:?}"); + } +} + +#[test] +fn cpu_prepare_rejects_plans_over_the_bound_without_a_streamed_route() { + let small = classic_rgb8(16, 16); + let gray12 = classic_fixture(96, 64, 1, 12); + let inputs = || { + vec![ + EncodedImage::full(Arc::clone(&small)), + EncodedImage::full(Arc::clone(&gray12)), ] }; let mut decoder = CpuBatchDecoder::new(single_worker_options()); let unbounded = decoder.prepare(inputs()).expect("prepare without a bound"); - let planned = planned_bytes(&unbounded); - let [(0, CpuExecution::OffsetPlan { bytes: small_bytes }), (1, CpuExecution::OffsetPlan { bytes: large_bytes })] = - planned[..] - else { - panic!("both images keep offset plans: {planned:?}"); + let CpuExecution::OffsetPlan { bytes: small_bytes } = planned_bytes(&unbounded)[0].1 else { + panic!("the small image keeps its offset plan"); }; - assert!(small_bytes < large_bytes, "{small_bytes} < {large_bytes}"); let reference = decoder .decode_prepared(&unbounded) .expect("decode without a bound"); @@ -111,7 +228,6 @@ fn cpu_prepare_rejects_plans_over_the_decode_bound_before_decoding() { 1, BatchErrorStage::Prepare )); - assert_eq!(planned_bytes(&rejected).len(), 1); let output = decoder .decode_prepared(&rejected) .expect("decode the admitted image"); @@ -137,8 +253,587 @@ fn cpu_prepare_rejects_plans_over_the_decode_bound_before_decoding() { 1, BatchErrorStage::Decode )); - assert_eq!( - u8_samples(output.groups(), 0), - u8_samples(reference.groups(), 0) +} + +/// `codestream` as JP2 in `color`, with `channel_definitions` in a cdef box. +fn wrap_jp2( + codestream: &[u8], + color: j2k_core::Colorspace, + channel_definitions: &[crate::J2kChannelDefinition], +) -> Arc<[u8]> { + Arc::from( + crate::wrap_j2k_codestream( + codestream, + crate::J2kFileWrapOptions::jp2() + .with_color(crate::J2kFileColorSpec::Enumerated(color)) + .with_metadata(crate::J2kFileBoxMetadata { + palette: None, + component_mappings: &[], + channel_definitions, + }), + ) + .expect("wrap JP2"), + ) +} + +fn identity_rgb_channel_definitions() -> [crate::J2kChannelDefinition; 3] { + [1, 2, 3].map(|color| crate::J2kChannelDefinition { + channel_index: color - 1, + channel_type: crate::J2kChannelType::Color, + association: crate::J2kChannelAssociation::Color { index: color }, + }) +} + +/// `codestream` with the image area and tile grid moved to `origin` on +/// both axes. A multiple of every code-block and DWT alignment keeps the +/// same coded samples. +fn with_siz_origin(codestream: &[u8], origin: u32) -> Arc<[u8]> { + let mut bytes = codestream.to_vec(); + assert_eq!(&bytes[..4], &[0xff, 0x4f, 0xff, 0x51], "SOC then SIZ"); + let mut add = |offset: usize| { + let field = <[u8; 4]>::try_from(&bytes[offset..offset + 4]).expect("SIZ field"); + let value = u32::from_be_bytes(field) + origin; + bytes[offset..offset + 4].copy_from_slice(&value.to_be_bytes()); + }; + // Xsiz, Ysiz, XOsiz, YOsiz, then XTOsiz, YTOsiz. + for offset in [8, 12, 16, 20, 32, 36] { + add(offset); + } + Arc::from(bytes) +} + +#[test] +fn gray8_and_rgb8_images_whose_region_decode_cannot_stream_keep_the_plan_route() { + let rgb = classic_rgb8(96, 64); + let gray = classic_fixture(80, 72, 1, 8); + let streams = [ + Arc::clone(&rgb), + Arc::clone(&gray), + wrap_jp2(&gray, j2k_core::Colorspace::SGray, &[]), + ]; + // A channel definition, or offsets, keep the region decode on whole + // component planes, which it hands over at the end. + let planes = [ + wrap_jp2( + &rgb, + j2k_core::Colorspace::SRgb, + &identity_rgb_channel_definitions(), + ), + with_siz_origin(&rgb, 4096), + with_siz_origin(&gray, 4096), + ]; + let inputs = || { + streams + .iter() + .chain(&planes) + .map(|image| EncodedImage::full(Arc::clone(image))) + .collect::>() + }; + let mut decoder = CpuBatchDecoder::new(single_worker_options()); + let reference = all_u8_samples( + &decoder + .decode_prepared(&decoder.prepare(inputs()).expect("prepare plans")) + .expect("decode plans"), ); + // Moved, wrapped and plain images decode the same samples. + let samples = |index: usize| { + reference + .iter() + .find(|(indices, _)| indices.contains(&index)) + .map(|(indices, samples)| { + let per_image = samples.len() / indices.len(); + let slot = indices.iter().position(|&i| i == index).expect("slot"); + samples[slot * per_image..(slot + 1) * per_image].to_vec() + }) + .expect("decoded image") + }; + for (index, plain) in [(2, 1), (3, 0), (4, 0), (5, 1)] { + assert_eq!(samples(index), samples(plain), "input {index}"); + } + + { + let _threshold = StreamAbove::hold(0); + let prepared = decoder + .prepare(inputs()) + .expect("prepare over the threshold"); + let planned = planned_bytes(&prepared); + for (index, execution) in &planned { + assert_eq!( + matches!(execution, CpuExecution::StreamedRowsOrPlan { .. }), + *index < streams.len(), + "input {index}: {execution:?}" + ); + assert_eq!( + matches!(execution, CpuExecution::OffsetPlan { .. }), + *index >= streams.len(), + "input {index}: {execution:?}" + ); + } + let streamed = all_u8_samples(&decoder.decode_prepared(&prepared).expect("decode")); + assert_eq!(streamed, reference); + } + + // Over the bound, only the images that stream their rows are admitted. + let _bound = PlannedBound::hold(0); + let bounded = decoder.prepare(inputs()).expect("prepare under the bound"); + assert!(planned_bytes(&bounded) + .iter() + .all(|(_, execution)| *execution == CpuExecution::StreamedRows)); + let rejected = bounded.errors(); + assert_eq!(rejected.len(), planes.len(), "{rejected:?}"); + for (error, index) in rejected.iter().zip(streams.len()..) { + assert!( + is_decode_bound_error(error, index, BatchErrorStage::Prepare), + "{error:?}" + ); + } +} + +#[test] +fn cpu_batch_streams_large_rgb8_and_gray8_plans_below_the_decode_bound() { + let small = classic_rgb8(16, 16); + let large = classic_rgb8(96, 64); + let gray = classic_fixture(80, 72, 1, 8); + let gray12 = classic_fixture(96, 64, 1, 12); + let inputs = || { + vec![ + EncodedImage::full(Arc::clone(&small)), + EncodedImage::full(Arc::clone(&large)), + EncodedImage::full(Arc::clone(&gray)), + EncodedImage::new( + Arc::clone(&large), + DecodeRequest::Region { + roi: j2k_core::Rect { + x: 13, + y: 7, + w: 61, + h: 50, + }, + }, + ), + // Two Gray8 images in flight stream; a lone one decodes its plan. + EncodedImage::full(Arc::clone(&gray)), + EncodedImage::full(Arc::clone(&gray12)), + ] + }; + for layout in [BatchLayout::Nchw, BatchLayout::Nhwc] { + let mut decoder = CpuBatchDecoder::new(single_worker_options_with(layout)); + let planned_route = decoder.prepare(inputs()).expect("prepare small plans"); + let planned = planned_bytes(&planned_route); + assert!( + planned + .iter() + .all(|(_, execution)| matches!(execution, CpuExecution::OffsetPlan { .. })), + "fixtures this small keep their plans: {planned:?}" + ); + let CpuExecution::OffsetPlan { bytes: small_bytes } = planned[0].1 else { + unreachable!("checked above"); + }; + let reference = decoder + .decode_prepared(&planned_route) + .expect("decode the plans"); + + let _threshold = StreamAbove::hold(small_bytes); + let streamed_route = decoder + .prepare(inputs()) + .expect("prepare under the threshold"); + assert_eq!(streamed_route.errors(), &[] as &[IndexedBatchError]); + let streamed = planned_bytes(&streamed_route); + assert!(matches!(streamed[0].1, CpuExecution::OffsetPlan { .. })); + for (index, execution) in &streamed[1..5] { + assert!( + matches!(execution, CpuExecution::StreamedRowsOrPlan { .. }), + "input {index}: {execution:?}" + ); + } + // 12-bit output has no streamed route and stays within the bound. + assert_eq!(streamed[5], planned[5], "{layout:?}"); + let output = decoder + .decode_prepared(&streamed_route) + .expect("decode streamed items"); + assert_eq!(output.errors(), &[] as &[IndexedBatchError]); + assert_eq!(output.groups(), reference.groups(), "{layout:?}"); + } +} + +#[test] +fn cpu_batch_streams_rgb8_images_whose_plans_pass_the_default_threshold() { + // About 14 bytes of plan per 8-bit sample: 1.4 MiB of RGB8 output + // plans several MiB, over the default threshold. + let rgb = classic_rgb8(704, 704); + let rgb12 = classic_fixture(704, 704, 3, 12); + let inputs = || { + vec![ + EncodedImage::full(Arc::clone(&rgb)), + EncodedImage::full(Arc::clone(&rgb12)), + ] + }; + let mut decoder = CpuBatchDecoder::new(single_worker_options()); + let prepared = decoder.prepare(inputs()).expect("prepare"); + let planned = planned_bytes(&prepared); + assert!( + matches!(planned[0].1, CpuExecution::StreamedRowsOrPlan { .. }), + "{planned:?}" + ); + let CpuExecution::OffsetPlan { bytes } = planned[1].1 else { + panic!("12-bit output keeps its plan: {planned:?}"); + }; + assert!(bytes > STREAM_ABOVE_PLANNED_BYTES, "{bytes}"); + let output = decoder.decode_prepared(&prepared).expect("decode"); + assert_eq!(output.errors(), &[] as &[IndexedBatchError]); + + // The streamed samples equal the plan route's. + let _threshold = StreamAbove::hold(usize::MAX); + let planned_route = decoder.prepare(inputs()).expect("prepare plans"); + assert!(matches!( + planned_bytes(&planned_route)[0].1, + CpuExecution::OffsetPlan { .. } + )); + let reference = decoder + .decode_prepared(&planned_route) + .expect("decode plans"); + assert_eq!(output.groups(), reference.groups()); +} + +#[test] +fn a_lone_gray8_image_decodes_its_plan_and_gray8_images_in_flight_stream() { + let gray = classic_fixture(80, 72, 1, 8); + let rgb = classic_rgb8(80, 72); + let decode = |copies: usize, image: &Arc<[u8]>| { + let inputs = vec![EncodedImage::full(Arc::clone(image)); copies]; + let mut decoder = CpuBatchDecoder::new(single_worker_options()); + let reference = decoder + .decode_prepared(&decoder.prepare(inputs.clone()).expect("prepare plans")) + .expect("decode plans"); + let plan_calls = decoder.workspace_stats().prepared_plan_decode_calls(); + let _threshold = StreamAbove::hold(0); + let prepared = decoder.prepare(inputs).expect("prepare"); + assert!(planned_bytes(&prepared) + .iter() + .all(|(_, execution)| matches!(execution, CpuExecution::StreamedRowsOrPlan { .. }))); + let output = decoder.decode_prepared(&prepared).expect("decode"); + assert_eq!(output.groups(), reference.groups(), "{copies} copies"); + decoder.workspace_stats().prepared_plan_decode_calls() - plan_calls + }; + assert_eq!(decode(1, &gray), 1, "a lone Gray8 image decodes its plan"); + assert_eq!(decode(2, &gray), 0, "Gray8 images in flight stream"); + assert_eq!(decode(1, &rgb), 0, "a lone RGB8 image streams"); +} + +#[derive(Debug, Clone, Copy)] +enum Pattern { + /// Copies of one image. + Copies(usize), + /// Four different images. + Distinct, + /// One image twice and two others: `[a, b, a, c]`. + Mixed, + /// One image twice around another: `[a, b, a]`. + AroundOne, +} + +#[test] +fn groups_whose_images_stream_never_stage_and_others_stage_at_one_worker() { + let limit = crate::batch::allocation::generic_batch_worker_limit(); + for is_gray in [false, true] { + let (width, height, components) = if is_gray { (80, 72, 1) } else { (96, 64, 3) }; + let images = (0..4) + .map(|seed| seeded_fixture(width, height, components, 8, false, seed)) + .collect::>(); + let inputs = |pattern: Pattern| -> Vec { + let picks: Vec = match pattern { + Pattern::Copies(copies) => vec![0; copies], + Pattern::Distinct => vec![0, 1, 2, 3], + Pattern::Mixed => vec![0, 1, 0, 2], + Pattern::AroundOne => vec![0, 1, 0], + }; + picks + .into_iter() + .map(|pick| EncodedImage::full(Arc::clone(&images[pick]))) + .collect() + }; + // Prepared on this thread, where the threshold override applies. + let batch_preparer = CpuBatchDecoder::new(single_worker_options()); + // Copies, distinct and mixed images stream at every worker count, on + // a pool of that many threads when there are several; the codec's + // own count and a lone image keep their threads. + for (workers, pattern, pool) in [ + (Some(1), Pattern::Copies(4), None), + (Some(1), Pattern::Distinct, None), + (Some(1), Pattern::Mixed, None), + (Some(1), Pattern::AroundOne, None), + (Some(2), Pattern::Copies(3), Some(2)), + (Some(2), Pattern::Mixed, Some(2)), + (Some(2), Pattern::AroundOne, Some(2)), + (Some(3), Pattern::Distinct, Some(3)), + (Some(4), Pattern::Mixed, Some(4)), + (Some(6), Pattern::Copies(3), Some(6)), + (Some(8), Pattern::Mixed, Some(8)), + (Some(3), Pattern::Copies(1), None), + (None, Pattern::Copies(4), None), + ] { + let label = format!("gray {is_gray}, {workers:?} workers, {pattern:?}"); + // Plans that do not stream stage at one worker, as before. + let mut plan_decoder = CpuBatchDecoder::new(single_worker_options()); + let planned = batch_preparer + .prepare(inputs(pattern)) + .expect("prepare plans"); + assert!(planned_bytes(&planned) + .iter() + .all(|(_, execution)| matches!(execution, CpuExecution::OffsetPlan { .. }))); + let reference = plan_decoder + .decode_prepared(&planned) + .expect("decode plans"); + let payloads = planned.groups()[0] + .images() + .iter() + .map(|image| image.classic_plan().expect("classic plan").payload_count() as u64) + .sum::(); + let staged = planned.groups()[0].images().len() > 1; + assert_eq!( + plan_decoder.workspace_stats().flattened_payload_jobs(), + if staged { payloads } else { 0 }, + "{label}" + ); + let _threshold = StreamAbove::hold(0); + let prepared = batch_preparer.prepare(inputs(pattern)).expect("prepare"); + assert!(planned_bytes(&prepared) + .iter() + .all(|(_, execution)| matches!( + execution, + CpuExecution::StreamedRowsOrPlan { .. } + ))); + let mut decoder = CpuBatchDecoder::new(BatchDecodeOptions { + workers: workers.and_then(NonZeroUsize::new), + ..BatchDecodeOptions::default() + }); + let output = decoder.decode_prepared(&prepared).expect("decode"); + assert_eq!(output.errors(), &[] as &[IndexedBatchError], "{label}"); + assert_eq!(output.groups(), reference.groups(), "{label}"); + let stats = decoder.workspace_stats(); + assert_eq!(stats.flattened_payload_jobs(), 0, "{label}"); + assert_eq!(stats.cross_image_entropy_windows(), 0, "{label}"); + assert_eq!( + decoder.stream_pool_threads(), + pool.map(|threads: usize| threads.min(limit)), + "{label}" + ); + } + } +} + +#[test] +fn htj2k_groups_stream_with_one_or_two_workers() { + let ht = fixture(96, 64, 3, 8, true); + let inputs = || vec![EncodedImage::full(Arc::clone(&ht)); 3]; + let mut batch_preparer = CpuBatchDecoder::new(single_worker_options()); + let reference = batch_preparer + .decode_prepared(&batch_preparer.prepare(inputs()).expect("prepare plans")) + .expect("decode plans"); + let _threshold = StreamAbove::hold(0); + let prepared = batch_preparer.prepare(inputs()).expect("prepare"); + assert!(planned_bytes(&prepared) + .iter() + .all(|(_, execution)| matches!(execution, CpuExecution::StreamedRowsOrPlan { .. }))); + // HTJ2K gains nothing from the staged route's back-to-back code-blocks. + for (workers, pool) in [(1, None), (2, Some(2))] { + let mut decoder = CpuBatchDecoder::new(BatchDecodeOptions { + workers: NonZeroUsize::new(workers), + ..BatchDecodeOptions::default() + }); + let output = decoder.decode_prepared(&prepared).expect("decode"); + assert_eq!(output.errors(), &[] as &[IndexedBatchError], "{workers}"); + assert_eq!(output.groups(), reference.groups(), "{workers}"); + assert_eq!( + decoder.workspace_stats().cross_image_entropy_windows(), + 0, + "{workers}" + ); + assert_eq!(decoder.stream_pool_threads(), pool, "{workers}"); + } +} + +#[test] +fn explicit_worker_counts_set_the_policy_of_images_decoded_one_at_a_time() { + use super::super::cpu_execute::image_policy; + let workers = NonZeroUsize::new; + let serial = Some(CpuDecodeParallelism::Serial); + // Several images: streamed ones in the pool of two or more threads, and + // plans serial; one worker decodes serially. + assert_eq!(image_policy(3, true, workers(1)), (false, serial)); + assert_eq!(image_policy(3, true, workers(2)), (true, serial)); + assert_eq!(image_policy(3, false, workers(2)), (false, serial)); + // A lone image and automatic workers keep the scheduler's choice. + assert_eq!(image_policy(1, true, workers(2)), (false, None)); + assert_eq!(image_policy(3, true, None), (false, None)); +} + +#[test] +fn batch_images_share_the_stream_pool_only_when_they_fill_it() { + use super::super::cpu_execute::shares_stream_pool; + let limit = crate::batch::allocation::generic_batch_worker_limit(); + // The session's workers, and so its stream pool, stop at the limit. + let cases = [ + // Eight images on 2, 3, 4 or 8 workers, and three on 2, keep an image + // in flight for every thread of the pool. + (8, 2, true), + (8, 3, true), + (8, 4, true), + (8, 8, true), + (3, 2, true), + // Eight on 6 run four at a time, five on 4 three, two on 18 two: + // fewer images in flight than threads. + (8, 6, false), + (5, 4, false), + (2, 18, false), + ]; + for (images, workers, shared) in cases.into_iter().filter(|case| case.1 <= limit) { + let options = crate::TileBatchOptions::new(NonZeroUsize::new(workers)); + assert_eq!( + shares_stream_pool(images, options, workers).expect("plan"), + shared, + "{images} on {workers}" + ); + } +} + +#[test] +fn images_far_above_their_group_mean_keep_the_lone_rings() { + use super::super::cpu_execute::shared_ring_bytes; + // Seven crops of about the same coded size share the rings; one noise + // image of the same shape with 3.7 times their bytes keeps its own. + let crops = [ + 3_585_095, 3_622_220, 3_720_453, 3_745_469, 3_787_355, 3_807_463, 3_648_167, + ]; + let mut group = crops.to_vec(); + group.push(13_840_854); + let limit = shared_ring_bytes(group.into_iter()); + assert!(crops.iter().all(|&bytes| bytes <= limit), "{limit}"); + assert!(13_840_854 > limit, "{limit}"); + // A uniform group shares them all, half again above the mean at most. + assert_eq!(shared_ring_bytes([100, 100, 100].into_iter()), 150); + assert_eq!(shared_ring_bytes([90, 100, 110, 160].into_iter()), 172); + assert_eq!(shared_ring_bytes(core::iter::empty()), 0); +} + +#[test] +fn plans_beyond_the_flattening_cap_decode_one_image_at_a_time() { + use super::super::cpu_fast::{ + classic_flattened_metadata_bytes, FLATTENED_METADATA_CAP_OVERRIDE, + }; + struct FlattenedCap; + impl FlattenedCap { + fn hold(bytes: usize) -> Self { + FLATTENED_METADATA_CAP_OVERRIDE.with(|cap| cap.set(Some(bytes))); + Self + } + } + impl Drop for FlattenedCap { + fn drop(&mut self) { + FLATTENED_METADATA_CAP_OVERRIDE.with(|cap| cap.set(None)); + } + } + let images = (0..3) + .map(|seed| seeded_fixture(96, 64, 3, 8, false, seed)) + .collect::>(); + let inputs = || { + images + .iter() + .map(|bytes| EncodedImage::full(Arc::clone(bytes))) + .collect::>() + }; + // Small plans that do not stream: one worker stages them. + let decoder = CpuBatchDecoder::new(single_worker_options()); + let prepared = decoder.prepare(inputs()).expect("prepare plans"); + assert!(planned_bytes(&prepared) + .iter() + .all(|(_, execution)| matches!(execution, CpuExecution::OffsetPlan { .. }))); + let payloads = prepared.groups()[0] + .images() + .iter() + .map(|image| image.classic_plan().expect("classic plan").payload_count()) + .sum::(); + let needed = classic_flattened_metadata_bytes(3, payloads); + let mut reference = None; + // At the cap the plans stage; one byte below it each decodes on its + // own, where flattening them would have failed the batch. + for (cap, staged) in [(needed, true), (needed - 1, false), (0, false)] { + let _cap = FlattenedCap::hold(cap); + let mut decoder = CpuBatchDecoder::new(single_worker_options()); + let (groups, errors) = decoder + .decode_prepared(&prepared) + .expect("decode") + .into_parts(); + assert_eq!(errors, [], "cap {cap}"); + match &reference { + Some(reference) => assert_eq!(&groups, reference, "cap {cap}"), + None => reference = Some(groups), + } + assert_eq!( + decoder.workspace_stats().flattened_payload_jobs(), + if staged { payloads as u64 } else { 0 }, + "cap {cap}" + ); + assert_eq!(decoder.parallel_plan_decodes(), 0, "cap {cap}"); + } +} + +#[test] +fn planned_images_among_streamed_ones_keep_the_group_worker_policy() { + let limit = crate::batch::allocation::generic_batch_worker_limit(); + let rgb = classic_rgb8(96, 64); + let gray = classic_fixture(80, 72, 1, 8); + // Copies `a` stream; `b` keeps its offset plan: for RGB8 the same JP2 + // with identity channel definitions, for Gray8 offsets. + let srgb = j2k_core::Colorspace::SRgb; + let cases = [ + ( + wrap_jp2(&rgb, srgb, &[]), + wrap_jp2(&rgb, srgb, &identity_rgb_channel_definitions()), + false, + ), + (Arc::clone(&gray), with_siz_origin(&gray, 4096), true), + ]; + for (copy, planned, is_gray) in cases { + let inputs = || { + [©, &planned, ©] + .map(|bytes| EncodedImage::full(Arc::clone(bytes))) + .to_vec() + }; + let mut batch_preparer = CpuBatchDecoder::new(single_worker_options()); + let reference = batch_preparer + .decode_prepared(&batch_preparer.prepare(inputs()).expect("prepare plans")) + .expect("decode plans"); + let _threshold = StreamAbove::hold(0); + let prepared = batch_preparer.prepare(inputs()).expect("prepare"); + assert_eq!(prepared.groups().len(), 1, "gray {is_gray}"); + let streams = planned_bytes(&prepared) + .iter() + .map(|(_, execution)| execution.streams()) + .collect::>(); + assert_eq!(streams, [true, false, true], "gray {is_gray}"); + // `[a, b, a]`: `b` decodes its plan without nested parallelism, as + // the group's other plans would, and in the pool of two threads. + for workers in [1, 2] { + let label = format!("gray {is_gray}, {workers} workers"); + let mut decoder = CpuBatchDecoder::new(BatchDecodeOptions { + workers: NonZeroUsize::new(workers), + ..BatchDecodeOptions::default() + }); + let output = decoder.decode_prepared(&prepared).expect("decode"); + assert_eq!(output.errors(), &[] as &[IndexedBatchError], "{label}"); + assert_eq!(output.groups(), reference.groups(), "{label}"); + assert_eq!( + decoder.workspace_stats().flattened_payload_jobs(), + 0, + "{label}" + ); + assert_eq!(decoder.parallel_plan_decodes(), 0, "{label}"); + assert_eq!( + decoder.stream_pool_threads(), + (workers > 1).then_some(workers.min(limit)), + "{label}" + ); + } + } } diff --git a/crates/j2k/src/owned_batch/prepare/image.rs b/crates/j2k/src/owned_batch/prepare/image.rs index 3bb05b393..70b484580 100644 --- a/crates/j2k/src/owned_batch/prepare/image.rs +++ b/crates/j2k/src/owned_batch/prepare/image.rs @@ -54,7 +54,11 @@ pub(super) fn prepare_image( .map_or(PreparedCodecPlan::MetadataOnly, PreparedCodecPlan::Htj2k), }; reconcile_codec_plan_metadata(&mut info, &codec_plan)?; - let cpu_execution = choose_cpu_execution(&codec_plan); + let cpu_execution = choose_cpu_execution( + &codec_plan, + &info, + native_image.region_decode_streams_rows(), + ); let preparation_depth = codec_plan.preparation_depth(); let execution_shape = batch_execution_shape(&support, plan, preparation_depth); let image = PreparedImage { @@ -76,10 +80,25 @@ pub(super) fn prepare_image( /// Choose the CPU execution from the planned working memory, before any /// decode work, with the same accounting the offset plan's execution -/// enforces. Only an image whose serial execution already exceeds the -/// per-decode bound is [`CpuExecution::ExceedsDecodeBound`], so the choice -/// rejects nothing execution could decode; the claim is the parallel peak. -pub(super) fn choose_cpu_execution(codec_plan: &PreparedCodecPlan) -> CpuExecution { +/// enforces. +/// +/// A Gray8 or RGB8 image whose planned execution exceeds +/// [`STREAM_ABOVE_PLANNED_BYTES`] streams its rows when the region decode +/// can stream them (`region_streams_rows`): the offset plan holds +/// whole-image f32 planes and coefficients (about 14 times the 8-bit +/// output), which every image in flight kept until its decode ended, while +/// the streamed route holds a band of rows. While its plan fits the +/// per-decode bound, the image keeps it for decoding alone in a Gray8 group +/// ([`CpuExecution::StreamedRowsOrPlan`]). Other outputs keep their offset +/// plan unless its serial execution exceeds the per-decode bound, which +/// rejects them, as it does images whose region decode would hand over +/// whole component planes instead of rows. The claim of an offset plan is +/// its parallel peak. +pub(super) fn choose_cpu_execution( + codec_plan: &PreparedCodecPlan, + info: &BatchGroupInfo, + region_streams_rows: bool, +) -> CpuExecution { let planned = |parallel| match codec_plan { PreparedCodecPlan::Classic(plan) => { j2k_native::referenced_classic_plan_execution_bytes(plan.cpu_plan(), parallel) @@ -92,26 +111,63 @@ pub(super) fn choose_cpu_execution(codec_plan: &PreparedCodecPlan) -> CpuExecuti if matches!(codec_plan, PreparedCodecPlan::MetadataOnly) { return CpuExecution::ComponentPlanes; } + let streams_rows = region_streams_rows && has_streamed_output(info); + let exceeds_bound = if streams_rows { + CpuExecution::StreamedRows + } else { + CpuExecution::ExceedsDecodeBound + }; match (planned(false), planned(true)) { ( Err(j2k_native::DecodeError::Validation(j2k_native::ValidationError::ImageTooLarge)), _, - ) => CpuExecution::ExceedsDecodeBound, - (Ok(serial), _) if serial > planned_decode_bound() => CpuExecution::ExceedsDecodeBound, - (Ok(_), Ok(bytes)) => CpuExecution::OffsetPlan { bytes }, - // A plan that fits only serially, or a malformed plan, keeps the - // worst-case claim and reports any error from execution, as before. + ) => exceeds_bound, + (Ok(serial), _) if serial > planned_decode_bound() => exceeds_bound, + (Ok(_), Ok(bytes)) if !streams_rows || bytes <= stream_above_planned_bytes() => { + CpuExecution::OffsetPlan { bytes } + } + (Ok(_), Ok(bytes)) => CpuExecution::StreamedRowsOrPlan { bytes }, + // An 8-bit plan that fits only serially streams. + (Ok(_), _) if streams_rows => CpuExecution::StreamedRows, + // Any other plan that fits only serially, or a malformed plan, keeps + // the worst-case claim and reports any error from execution. _ => CpuExecution::OffsetPlan { bytes: crate::batch::allocation::GENERIC_WORKER_CLAIM_BYTES, }, } } +/// Gray8 and RGB8 outputs have a streamed route: the region decode hands +/// over finished rows of 8-bit components where the image lets it (see +/// `j2k_native::Image::region_decode_streams_rows`). Other outputs keep the +/// bound. +fn has_streamed_output(info: &BatchGroupInfo) -> bool { + info.sample_type == NativeSampleType::U8 + && matches!(info.color, BatchColor::Gray | BatchColor::Rgb) +} + +/// Planned offset-plan execution above which Gray8/RGB8 images stream +/// their rows (see [`choose_cpu_execution`]). Smaller images keep their +/// parse-free plan, whose working memory is negligible at that size. +pub(super) const STREAM_ABOVE_PLANNED_BYTES: usize = 4 << 20; + #[cfg(test)] std::thread_local! { /// Test-only lower bound for [`choose_cpu_execution`] on this thread. pub(super) static PLANNED_DECODE_BOUND: core::cell::Cell> = const { core::cell::Cell::new(None) }; + /// Test-only streaming threshold for [`choose_cpu_execution`] on this + /// thread. + pub(super) static STREAM_ABOVE_OVERRIDE: core::cell::Cell> = + const { core::cell::Cell::new(None) }; +} + +fn stream_above_planned_bytes() -> usize { + #[cfg(test)] + if let Some(bytes) = STREAM_ABOVE_OVERRIDE.with(core::cell::Cell::get) { + return bytes; + } + STREAM_ABOVE_PLANNED_BYTES } /// The per-decode bound planned executions are held to. Native planning diff --git a/crates/j2k/src/owned_batch/prepared.rs b/crates/j2k/src/owned_batch/prepared.rs index cb97967f2..f3b38d1b5 100644 --- a/crates/j2k/src/owned_batch/prepared.rs +++ b/crates/j2k/src/owned_batch/prepared.rs @@ -43,8 +43,19 @@ pub(super) enum CpuExecution { /// Reparse and decode whole component planes; no offset plan exists. ComponentPlanes, /// The planned offset-plan execution exceeds the per-decode memory - /// bound, so the CPU decoder rejects the image before any decode work. + /// bound and the output has no streamed route (not Gray8 or RGB8), so the + /// CPU decoder rejects the image before any decode work. ExceedsDecodeBound, + /// A Gray8/RGB8 image whose offset plan exceeds the per-decode bound + /// streams rows from the full-output region decode, whose working memory + /// does not grow with the image. + StreamedRows, + /// A Gray8/RGB8 image whose offset plan fits the bound (`bytes` is its + /// parallel peak) but exceeds the streaming threshold. It streams rows + /// like [`Self::StreamedRows`], except as the only image of a Gray8 + /// group: alone in flight, its plan spreads one component's code-block + /// rows over every core, which the streamed route does not match. + StreamedRowsOrPlan { bytes: usize }, } impl CpuExecution { @@ -53,11 +64,33 @@ impl CpuExecution { pub(super) const fn claim_bytes(self) -> usize { match self { Self::OffsetPlan { bytes } => bytes, - Self::ComponentPlanes => crate::batch::allocation::GENERIC_WORKER_CLAIM_BYTES, + // The streamed route has no planned peak yet; it is claimed as one + // worst-case native decode (measured: 95 MiB for a 5661 x 8287 + // RGB8 scan at default threads). + // An optional plan fits the bound, so its peak is within this + // claim too. + Self::ComponentPlanes | Self::StreamedRows | Self::StreamedRowsOrPlan { .. } => { + crate::batch::allocation::GENERIC_WORKER_CLAIM_BYTES + } Self::ExceedsDecodeBound => 0, } } + /// The planned peak of the offset plan this image may decode, if any: + /// what the staged route, which decodes plans only, claims for it. + pub(super) const fn plan_bytes(self) -> Option { + match self { + Self::OffsetPlan { bytes } | Self::StreamedRowsOrPlan { bytes } => Some(bytes), + Self::ComponentPlanes | Self::ExceedsDecodeBound | Self::StreamedRows => None, + } + } + + /// Whether the CPU decoder may stream this image's rows; a group holding + /// such an image never takes the staged route. + pub(super) const fn streams(self) -> bool { + matches!(self, Self::StreamedRows | Self::StreamedRowsOrPlan { .. }) + } + /// The error the offset-plan execution would report, raised up front. pub(super) fn decode_bound_error() -> crate::J2kError { crate::J2kError::from_native_decode_error(j2k_native::ValidationError::ImageTooLarge.into()) diff --git a/crates/j2k/src/view.rs b/crates/j2k/src/view.rs index 99777626e..4e8cd5095 100644 --- a/crates/j2k/src/view.rs +++ b/crates/j2k/src/view.rs @@ -7,9 +7,11 @@ use crate::{ inspect_info_from_image, Image, }, decode::{ - decode_image_into_with_native_context, decode_image_region_into_with_native_context, - decode_warnings_for_image, validate_buffer, validate_region, J2kDecodeOutcome, - J2kDecodedComponents, J2kDecodedNativeComponents, + decode_full_image_on_route, decode_image_region_into_with_native_context, + decode_native_components_with_native_context, decode_warnings_for_image, full_image_route, + native_components_route, native_region_components_route, validate_buffer, validate_region, + J2kDecodeOutcome, J2kDecodedComponents, J2kDecodedNativeComponents, NativeRoute, + PlanesBudget, RouteRequest, }, parse::{parse_image_info, parse_info}, scratch::J2kScratchPool, @@ -123,6 +125,9 @@ pub struct J2kDecoder<'a> { image: Option>, settings: DecodeSettings, native_context: j2k_native::DecoderContext<'a>, + /// Set by [`Self::set_full_decode_planes_budget`]; see + /// [`Self::full_decode_planes_budget`]. + full_decode_planes_budget: Option, } /// Options for bounded J2K row decoding. @@ -190,6 +195,7 @@ impl<'a> J2kDecoder<'a> { image: view.image, settings: view.settings, native_context: j2k_native::DecoderContext::default(), + full_decode_planes_budget: None, }) } @@ -244,19 +250,30 @@ impl<'a> J2kDecoder<'a> { /// /// This preserves per-component bit depth, signedness, sampling, and byte /// width for callers that cannot use a single interleaved packed bitmap. + /// A decode whose `f32` component planes would pass the native decode + /// budget runs as a region decode of the whole image, which packs rows + /// into the returned planes as they are finished. /// /// # Errors /// Returns [`J2kError`] when the codestream cannot be decoded. pub fn decode_native_components(&mut self) -> Result { self.ensure_image()?; + let planes_budget = self.full_decode_planes_budget(); let (Some(image), native_context) = (self.image.as_ref(), &mut self.native_context) else { return Err(J2kError::internal_backend("internal image cache missing")); }; - let decoded = image - .decode_native_components_with_context(native_context) - .map_err(J2kError::from_native_decode_error)?; - let retained_image_bytes = component_handoff_image_bytes(image)?; - J2kDecodedNativeComponents::try_from_native(decoded, retained_image_bytes) + let planes_budget = match planes_budget { + PlanesBudget::Default => j2k_native::DEFAULT_MAX_DECODE_BYTES, + PlanesBudget::Fixed(bytes) => bytes, + }; + let route = native_components_route(image, planes_budget); + let decoded = + decode_native_components_with_native_context(image, native_context, None, route); + let retained_image_bytes = component_handoff_image_bytes(image); + if route == NativeRoute::Rows { + self.native_context = self.fresh_native_context(); + } + J2kDecodedNativeComponents::try_from_native(decoded?, retained_image_bytes?) } /// Extract selected codestream wavelet bands on the CPU, without inverse @@ -281,6 +298,13 @@ impl<'a> J2kDecoder<'a> { /// Decode a source-coordinate region into owned native-bit-depth component /// planes. /// + /// Like 16-bit region decodes, it packs rows into the returned planes as + /// they are finished, without region-sized `f32` planes, for RGB images + /// without palette, channel definition, sYCC, `CIELab` or image and tile + /// offsets whose components share one precision, and for other such + /// images of up to four components when the planes would pass the + /// native decode budget. + /// /// # Errors /// Returns [`J2kError`] when the region is invalid or the codestream cannot /// be decoded. @@ -293,14 +317,14 @@ impl<'a> J2kDecoder<'a> { let (Some(image), native_context) = (self.image.as_ref(), &mut self.native_context) else { return Err(J2kError::internal_backend("internal image cache missing")); }; - let decoded = image - .decode_native_region_components_with_context( - (roi.x, roi.y, roi.w, roi.h), - native_context, - ) - .map_err(J2kError::from_native_decode_error)?; - let retained_image_bytes = component_handoff_image_bytes(image)?; - J2kDecodedNativeComponents::try_from_native(decoded, retained_image_bytes) + let route = native_region_components_route(image, roi); + let decoded = + decode_native_components_with_native_context(image, native_context, Some(roi), route); + let retained_image_bytes = component_handoff_image_bytes(image); + if route == NativeRoute::Rows { + self.native_context = self.fresh_native_context(); + } + J2kDecodedNativeComponents::try_from_native(decoded?, retained_image_bytes?) } /// Return the CPU decode parallelism policy for this decoder. @@ -316,9 +340,19 @@ impl<'a> J2kDecoder<'a> { /// Decode the full image into `out` using `stride` bytes per output row. /// + /// A decode whose component planes would pass the native decode budget + /// runs as a region decode of the whole image, which writes rows as they + /// are finished instead of holding the planes: at 8 bits, and at 16 bits + /// for images whose components share one precision of at most 24 bits. + /// A 16-bit output holds each sample's value clamped to the 16-bit range + /// (two's complement when signed); samples of at most 8 bits are widened + /// across it. + /// /// # Errors /// Returns [`J2kError`] when the format is unsupported, the output buffer - /// is too small, or the codestream fails during decode. + /// is too small, or the codestream fails during decode. A decode that + /// runs as a region decode and fails partway may already have written + /// some rows of `out`. pub fn decode_into( &mut self, out: &mut [u8], @@ -374,13 +408,31 @@ impl<'a> J2kDecoder<'a> { ) -> Result { validate_buffer(self.info.dimensions, out.len(), stride, fmt)?; self.ensure_image()?; + let budget = self.full_decode_planes_budget(); let (Some(image), native_context) = (self.image.as_ref(), &mut self.native_context) else { return Err(J2kError::internal_backend("internal image cache missing")); }; - decode_image_into_with_native_context(image, native_context, out, stride, fmt)?; + let request = RouteRequest { + fmt, + budget, + parallelism: native_context.cpu_decode_parallelism(), + components: self.info.components, + coded_bytes: self.bytes.len(), + scale: Downscale::None, + }; + let route = full_image_route(image, request); + let decoded = + decode_full_image_on_route(image, native_context, out, stride, request, &route); + let warnings = decode_warnings_for_image(image); + if route.streams() { + // As after a region decode, release the decode buffers rather + // than hold them while the decoder is idle. + self.native_context = self.fresh_native_context(); + } + decoded?; Ok(j2k_core::DecodeOutcome::new( Rect::full(self.info.dimensions), - decode_warnings_for_image(image), + warnings, )) } @@ -440,12 +492,16 @@ impl<'a> J2kDecoder<'a> { /// Decode the full image at a reduced resolution. /// /// `scale` uses the shared [`Downscale`] contract; `Downscale::None` - /// delegates to full-resolution decode. + /// delegates to full-resolution decode. As with [`Self::decode_into`], + /// a decode whose component planes would pass the native decode budget + /// runs as a region decode of the whole reduced image. /// /// # Errors /// Returns [`J2kError`] when the format or scale request is unsupported, /// including a scale deeper than the codestream's resolution ladder, the - /// output buffer is too small, or decode fails. + /// output buffer is too small, or decode fails. A decode that runs as a + /// region decode and fails partway may already have written some rows of + /// `out`. pub fn decode_scaled_into( &mut self, pool: &mut J2kScratchPool, @@ -462,7 +518,16 @@ impl<'a> J2kDecoder<'a> { let image_dims = (image.width(), image.height()); validate_buffer(image_dims, out.len(), stride, fmt)?; let mut native_context = self.fresh_native_context(); - decode_image_into_with_native_context(&image, &mut native_context, out, stride, fmt)?; + let request = RouteRequest { + fmt, + budget: self.full_decode_planes_budget(), + parallelism: native_context.cpu_decode_parallelism(), + components: self.info.components, + coded_bytes: self.bytes.len(), + scale, + }; + let route = full_image_route(&image, request); + decode_full_image_on_route(&image, &mut native_context, out, stride, request, &route)?; Ok(j2k_core::DecodeOutcome::new( Rect::full(image_dims), warnings, @@ -529,6 +594,26 @@ impl<'a> J2kDecoder<'a> { .ok_or_else(|| J2kError::internal_backend("internal image cache missing")) } + /// What a full-image decode's component planes may charge: the native + /// decode budget, past which the planes decode is refused, with + /// streaming wherever it was measured faster, or on one thread nearly as + /// fast in far less memory, unless a test set a fixed budget. + fn full_decode_planes_budget(&self) -> PlanesBudget { + self.full_decode_planes_budget + .map_or(PlanesBudget::Default, PlanesBudget::Fixed) + } + + /// Test hook: hold full-image decodes' component planes to `bytes` and + /// drop the preference for streaming, so that a decode takes the + /// streamed region route exactly when its planes would pass `bytes`: 0 + /// streams every decode that can stream, `usize::MAX` keeps every decode + /// the planes can finish on the planes. The native decode + /// budget itself is unchanged. + #[doc(hidden)] + pub fn set_full_decode_planes_budget(&mut self, bytes: usize) { + self.full_decode_planes_budget = Some(bytes); + } + fn fresh_native_context(&self) -> j2k_native::DecoderContext<'a> { let mut native_context = j2k_native::DecoderContext::default(); native_context.set_cpu_decode_parallelism(self.native_context.cpu_decode_parallelism()); diff --git a/crates/j2k/src/view/rows.rs b/crates/j2k/src/view/rows.rs index 752f10382..cbd965936 100644 --- a/crates/j2k/src/view/rows.rs +++ b/crates/j2k/src/view/rows.rs @@ -4,7 +4,10 @@ use super::{J2kDecoder, J2kRowDecodeOptions}; use crate::{ - decode::{decode_prepared_image_region_into, J2kDecodeOutcome}, + decode::{ + decode_image_rows_in_stripes, decode_prepared_image_region_into, rows_stream_whole, + DeliverRow, J2kDecodeOutcome, StreamedRowsError, + }, scratch::J2kScratchPool, J2kError, }; @@ -67,9 +70,35 @@ impl J2kDecoder<'_> { /// Decode rows into a `u8` row sink while bounding host output scratch to /// at most `options.max_rows_per_stripe()` rows. /// + /// Whether the image decodes once is planned from its main and tile + /// headers before anything is decoded: an image whose tiles span its + /// width and whose region decode streams its rows as far as those headers + /// say decodes as one region decode of the whole image, which hands + /// finished rows over a stripe at a time; other images decode each stripe + /// as a region of its own. Coding the plan does not see, such as a + /// tile-level coding style without decomposition levels or HTJ2K + /// code-blocks mixed with classic ones, can keep that region decode from + /// streaming: it then decodes into component planes of the whole image + /// and hands their rows over a stripe at a time once they are complete. + /// + /// # Memory + /// `max_rows_per_stripe` and `max_stripe_bytes` bound the packed output + /// stripe. These limits do not bound decoder working memory or total + /// process memory. Decoder memory depends on the image and parallelism + /// and may include the working set of a whole-image decode, including + /// component planes when native streaming is unavailable. Smaller stripes + /// do not necessarily reduce decoder memory. + /// /// # Errors /// Returns a decode error for unsupported formats or malformed input, and /// forwards sink errors without converting them to successful decodes. + /// An error may leave a prefix of rows delivered. A sink error stops + /// further sink calls and is returned as [`DecodeRowsError::Sink`], even + /// if decoding subsequently fails; the underlying decode may continue + /// before this method returns. A decode failure is retried stripe by + /// stripe only if no row has been handed to the sink. After delivery + /// starts, decode failures return [`DecodeRowsError::Decode`] without + /// replaying rows. pub fn decode_rows_u8_bounded>( &mut self, sink: &mut R, @@ -87,6 +116,24 @@ impl J2kDecoder<'_> { let (stripe_rows, max_stripe_len) = bounded_row_stripe_layout(row_bytes, height, options) .map_err(DecodeRowsError::Decode)?; self.ensure_image().map_err(DecodeRowsError::Decode)?; + let mut pool = J2kScratchPool::new(); + if self.rows_stream_whole() { + let stripe = pool + .packed_bytes(max_stripe_len) + .map_err(|error| DecodeRowsError::Decode(J2kError::Buffer(error)))?; + match self.decode_rows_streamed(fmt, (stripe, row_bytes), &mut |y, row| { + sink.write_row(y, row) + }) { + Ok(()) => { + return Ok(j2k_core::DecodeOutcome::new( + Rect::full(self.info.dimensions), + Vec::new(), + )) + } + Err(StreamedRowsError::Rows(error)) => return Err(error), + Err(StreamedRowsError::BeforeFirstRow) => {} + } + } let (Some(image), native_context) = (self.image.as_ref(), &mut self.native_context) else { return Err(DecodeRowsError::Decode(J2kError::internal_backend( "internal image cache missing", @@ -95,7 +142,6 @@ impl J2kDecoder<'_> { let mut decoder = image .prepare_region_decoder_with_context(native_context) .map_err(|error| DecodeRowsError::Decode(J2kError::from_native_decode_error(error)))?; - let mut pool = J2kScratchPool::new(); let mut y = 0_u32; while y < height { let rows = stripe_rows.min(height - y); @@ -136,9 +182,29 @@ impl J2kDecoder<'_> { /// Decode rows into a `u16` row sink while bounding host output scratch to /// at most `options.max_rows_per_stripe()` rows. /// + /// Images decode once or a stripe at a time as planned for + /// [`Self::decode_rows_u8_bounded`]; images deeper than 24 bits decode a + /// stripe at a time. + /// + /// # Memory + /// `max_rows_per_stripe` and `max_stripe_bytes` bound the packed output + /// stripe, beside which this method uses one row of `u16` conversion + /// scratch. These limits do not bound decoder working memory or total + /// process memory. Decoder memory depends on the image and parallelism + /// and may include the working set of a whole-image decode, including + /// component planes when native streaming is unavailable. Smaller stripes + /// do not necessarily reduce decoder memory. + /// /// # Errors /// Returns a decode error for unsupported formats or malformed input, and /// forwards sink errors without converting them to successful decodes. + /// An error may leave a prefix of rows delivered. A sink error stops + /// further sink calls and is returned as [`DecodeRowsError::Sink`], even + /// if decoding subsequently fails; the underlying decode may continue + /// before this method returns. A decode failure is retried stripe by + /// stripe only if no row has been handed to the sink. After delivery + /// starts, decode failures return [`DecodeRowsError::Decode`] without + /// replaying rows. pub fn decode_rows_u16_bounded>( &mut self, sink: &mut R, @@ -180,6 +246,25 @@ impl J2kDecoder<'_> { return self.decode_rows_u16_exact_reparsed(sink, fmt, layout); } self.ensure_image().map_err(DecodeRowsError::Decode)?; + if self.rows_stream_whole() { + let mut pool = J2kScratchPool::new(); + let (packed, row) = pool + .packed_bytes_and_row_u16(layout.max_stripe_len, layout.samples_per_row) + .map_err(|error| DecodeRowsError::Decode(J2kError::Buffer(error)))?; + match self.decode_rows_streamed(fmt, (packed, row_bytes), &mut |y, packed_row| { + widen_row_u16(packed_row, row); + sink.write_row(y, row) + }) { + Ok(()) => { + return Ok(j2k_core::DecodeOutcome::new( + Rect::full((width, height)), + Vec::new(), + )) + } + Err(StreamedRowsError::Rows(error)) => return Err(error), + Err(StreamedRowsError::BeforeFirstRow) => {} + } + } let (Some(image), native_context) = (self.image.as_ref(), &mut self.native_context) else { return Err(DecodeRowsError::Decode(J2kError::internal_backend( "internal image cache missing", @@ -193,6 +278,34 @@ impl J2kDecoder<'_> { }) } + /// Whether this decoder's image decodes its rows in one streamed region + /// decode (see [`rows_stream_whole`]). + fn rows_stream_whole(&self) -> bool { + self.cached_image() + .is_ok_and(|image| rows_stream_whole(image, self.info.resolution_levels)) + } + + /// Decode the whole image in one streamed region decode, handing its + /// rows to `deliver` a stripe of `stripe` at a time (see + /// [`decode_image_rows_in_stripes`]). + fn decode_rows_streamed( + &mut self, + fmt: PixelFormat, + stripe: (&mut [u8], usize), + deliver: &mut DeliverRow<'_, E>, + ) -> Result<(), StreamedRowsError> { + let (Some(image), native_context) = (self.image.as_ref(), &mut self.native_context) else { + return Err(StreamedRowsError::Rows(DecodeRowsError::Decode( + J2kError::internal_backend("internal image cache missing"), + ))); + }; + let decoded = decode_image_rows_in_stripes(image, native_context, fmt, stripe, deliver); + // As after a region decode, release the decode buffers rather than + // hold them while the decoder is idle. + self.native_context = self.fresh_native_context(); + decoded + } + fn decode_rows_u16_exact_reparsed>( &mut self, sink: &mut R, @@ -248,10 +361,7 @@ fn pump_u16_rows>( .map_err(DecodeRowsError::Decode)?; for row_index in 0..rows { let start = row_index as usize * layout.row_bytes; - let packed_row = &packed[start..start + layout.row_bytes]; - for (dst, src) in row.iter_mut().zip(packed_row.as_chunks::<2>().0) { - *dst = u16::from_le_bytes(*src); - } + widen_row_u16(&packed[start..start + layout.row_bytes], row); sink.write_row(y + row_index, row) .map_err(DecodeRowsError::Sink)?; } @@ -263,6 +373,13 @@ fn pump_u16_rows>( )) } +/// The 16-bit samples of a packed little-endian row. +fn widen_row_u16(packed: &[u8], row: &mut [u16]) { + for (dst, src) in row.iter_mut().zip(packed.as_chunks::<2>().0) { + *dst = u16::from_le_bytes(*src); + } +} + fn row_format_u8(info: &j2k_core::Info) -> Result { match info.components { 1 => Ok(PixelFormat::Gray8), @@ -337,6 +454,191 @@ fn row_samples_for(info: &j2k_core::Info, fmt: PixelFormat) -> Result) -> core::fmt::Result { + write!(f, "row {} refused", self.0) + } + } + + impl core::error::Error for Refused {} + + /// Rows as little-endian bytes and every row index handed over, in + /// order, refusing row `refuse` when it is set. + #[derive(Default)] + struct Collect { + bytes: Vec, + order: Vec, + refuse: Option, + } + + impl Collect { + fn take(&mut self, y: u32) -> Result<(), Refused> { + self.order.push(y); + if self.refuse == Some(y) { + return Err(Refused(y)); + } + Ok(()) + } + } + + impl RowSink for Collect { + type Error = Refused; + + fn write_row(&mut self, y: u32, row: &[u8]) -> Result<(), Refused> { + self.take(y)?; + self.bytes.extend_from_slice(row); + Ok(()) + } + } + + impl RowSink for Collect { + type Error = Refused; + + fn write_row(&mut self, y: u32, row: &[u16]) -> Result<(), Refused> { + self.take(y)?; + self.bytes + .extend(row.iter().flat_map(|sample| sample.to_le_bytes())); + Ok(()) + } + } + + fn rgb_codestream(tile_size: Option<(u32, u32)>, num_decomposition_levels: u8) -> Vec { + let pixels: Vec = (0..64 * 48 * 3_u32) + .map(|index| u8::try_from(index * 7 % 251).expect("a byte")) + .collect(); + let options = EncodeOptions { + tile_size, + num_decomposition_levels, + ..EncodeOptions::default() + }; + j2k_native::encode(&pixels, 64, 48, 3, 8, false, &options).expect("encode") + } + + /// The `decode_into` output in `fmt`. + fn full_decode(bytes: &[u8], fmt: PixelFormat) -> Vec { + let mut decoder = J2kDecoder::new(bytes).expect("decoder"); + let stride = 64 * fmt.bytes_per_pixel(); + let mut full = vec![0; stride * 48]; + decoder + .decode_into(&mut full, stride, fmt) + .expect("full decode"); + full + } + + /// A bounded decode in 7-row stripes into `sink`, with `injected` set + /// for it, and the streamed decodes it started. + fn decode_rows( + bytes: &[u8], + fmt: PixelFormat, + injected: Option, + sink: &mut Collect, + ) -> (Result<(), DecodeRowsError>, usize) { + let streamed = || STREAMED_ROW_DECODES.with(core::cell::Cell::get); + let mut decoder = J2kDecoder::new(bytes).expect("decoder"); + let before = streamed(); + INJECTED_FAILURE.with(|failure| failure.set(injected)); + let options = J2kRowDecodeOptions::new(7); + let result = if fmt == PixelFormat::Rgb16 { + decoder.decode_rows_u16_bounded(sink, options) + } else { + decoder.decode_rows_u8_bounded(sink, options) + }; + assert_eq!(INJECTED_FAILURE.with(core::cell::Cell::get), None); + (result.map(drop), streamed() - before) + } + + #[test] + fn rows_decode_in_one_streamed_decode_only_when_they_arrive_whole_and_in_order() { + for (label, bytes, streams) in [ + ("untiled", rgb_codestream(None, 3), true), + ( + "tiles spanning the width", + rgb_codestream(Some((64, 16)), 3), + true, + ), + ( + "tiles side by side", + rgb_codestream(Some((32, 16)), 3), + false, + ), + ("no decomposition levels", rgb_codestream(None, 0), false), + ] { + for fmt in [PixelFormat::Rgb8, PixelFormat::Rgb16] { + let mut sink = Collect::default(); + let (result, streamed) = decode_rows(&bytes, fmt, None, &mut sink); + result.expect("rows"); + assert_eq!(streamed, usize::from(streams), "{label} {fmt:?}"); + assert_eq!(sink.order, (0..48).collect::>(), "{label} {fmt:?}"); + assert!( + sink.bytes == full_decode(&bytes, fmt), + "{label} {fmt:?}: rows differ from decode_into" + ); + } + } + } + + #[test] + fn a_streamed_decode_failing_before_its_first_row_decodes_a_stripe_at_a_time() { + let bytes = rgb_codestream(None, 3); + for fmt in [PixelFormat::Rgb8, PixelFormat::Rgb16] { + let full = full_decode(&bytes, fmt); + let mut sink = Collect::default(); + let (result, streamed) = + decode_rows(&bytes, fmt, Some(InjectedFailure::BeforeRows), &mut sink); + result.expect("rows a stripe at a time"); + assert_eq!(streamed, 1, "{fmt:?}"); + assert_eq!(sink.order, (0..48).collect::>(), "{fmt:?}"); + assert!(sink.bytes == full, "{fmt:?}: rows differ from decode_into"); + // The stripe-at-a-time decode meets the caller's sink as it is. + let mut sink = Collect { + refuse: Some(10), + ..Collect::default() + }; + let (result, _) = + decode_rows(&bytes, fmt, Some(InjectedFailure::BeforeRows), &mut sink); + assert!( + matches!(result, Err(DecodeRowsError::Sink(Refused(10)))), + "{fmt:?}" + ); + assert_eq!(sink.order, (0..=10).collect::>(), "{fmt:?}"); + } + } + + #[test] + fn a_streamed_decode_failing_after_rows_reached_the_sink_reports_it_without_a_rerun() { + let bytes = rgb_codestream(None, 3); + for fmt in [PixelFormat::Rgb8, PixelFormat::Rgb16] { + let full = full_decode(&bytes, fmt); + let row_bytes = 64 * fmt.bytes_per_pixel(); + let mut sink = Collect::default(); + let (result, streamed) = + decode_rows(&bytes, fmt, Some(InjectedFailure::AfterRows), &mut sink); + assert!(matches!(result, Err(DecodeRowsError::Decode(_))), "{fmt:?}"); + assert_eq!(streamed, 1, "{fmt:?}"); + // Six whole 7-row stripes; the last one, which the failure + // ended, is held back, and nothing is handed over twice. + assert_eq!(sink.order, (0..42).collect::>(), "{fmt:?}"); + assert!(sink.bytes == full[..42 * row_bytes], "{fmt:?}"); + // A sink error comes first. + let mut sink = Collect { + refuse: Some(10), + ..Collect::default() + }; + let (result, _) = decode_rows(&bytes, fmt, Some(InjectedFailure::AfterRows), &mut sink); + assert!( + matches!(result, Err(DecodeRowsError::Sink(Refused(10)))), + "{fmt:?}" + ); + assert_eq!(sink.order, (0..=10).collect::>(), "{fmt:?}"); + } + } #[test] fn bounded_row_stripe_layout_clamps_rows_to_byte_cap() { diff --git a/crates/j2k/src/view/tests.rs b/crates/j2k/src/view/tests.rs index 2e136f936..784614758 100644 --- a/crates/j2k/src/view/tests.rs +++ b/crates/j2k/src/view/tests.rs @@ -29,6 +29,7 @@ fn fresh_native_context_preserves_configured_parallelism() { image: None, settings: DecodeSettings::strict(), native_context: j2k_native::DecoderContext::default(), + full_decode_planes_budget: None, }; decoder.set_cpu_decode_parallelism(CpuDecodeParallelism::Serial); diff --git a/crates/j2k/tests/bounded_rows.rs b/crates/j2k/tests/bounded_rows.rs new file mode 100644 index 000000000..34716d110 --- /dev/null +++ b/crates/j2k/tests/bounded_rows.rs @@ -0,0 +1,409 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! A bounded row decode of an image whose tiles span its width runs as one +//! region decode of the whole image that streams its rows and hands them +//! over a stripe at a time; other images decode each stripe as a region of +//! its own. Either way the rows must be the bytes `decode_into` writes, in +//! order and once each, for every stripe height, pool and sample depth, and +//! a sink that fails must stop the rows at the row it refused and be +//! reported as the sink's error. + +use core::fmt; + +use j2k::{CpuDecodeParallelism, DecodeRowsError, J2kDecoder, J2kRowDecodeOptions, RowSink}; +use j2k_core::PixelFormat; +use j2k_native::{encode, EncodeOptions}; +use j2k_test_support::wrap_jp2_rgba_codestream; +use rayon::ThreadPoolBuilder; + +const WIDTH: u32 = 203; +const HEIGHT: u32 = 157; + +fn noisy_samples(components: u32, bit_depth: u8) -> Vec { + let mut state = 0x2545_f491_u32; + let mut next = move || { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + state + }; + let samples = WIDTH * HEIGHT * components; + if bit_depth <= 8 { + return (0..samples).map(|_| (next() >> 24) as u8).collect(); + } + let max = (1_u32 << bit_depth) - 1; + (0..samples) + .flat_map(|_| { + u16::try_from(next() % (max + 1)) + .expect("sample") + .to_le_bytes() + }) + .collect() +} + +fn codestream(components: u16, bit_depth: u8, options: &EncodeOptions) -> Vec { + encoded(components, bit_depth, false, options) +} + +fn encoded(components: u16, bit_depth: u8, signed: bool, options: &EncodeOptions) -> Vec { + let samples = noisy_samples(u32::from(components), bit_depth); + encode( + &samples, WIDTH, HEIGHT, components, bit_depth, signed, options, + ) + .expect("encode fixture") +} + +/// A sink error carrying the row the sink refused. +#[derive(Debug, PartialEq, Eq)] +struct Refused(u32); + +impl fmt::Display for Refused { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "row {} refused", self.0) + } +} + +impl std::error::Error for Refused {} + +/// The rows a sink took, as little-endian bytes, and every row index it +/// was given, refusing the row `refuse` when it is set. +#[derive(Default)] +struct Rows { + bytes: Vec, + order: Vec, + refuse: Option, +} + +impl Rows { + fn take(&mut self, y: u32) -> Result<(), Refused> { + self.order.push(y); + if self.refuse == Some(y) { + return Err(Refused(y)); + } + Ok(()) + } +} + +impl RowSink for Rows { + type Error = Refused; + + fn write_row(&mut self, y: u32, row: &[u8]) -> Result<(), Refused> { + self.take(y)?; + self.bytes.extend_from_slice(row); + Ok(()) + } +} + +impl RowSink for Rows { + type Error = Refused; + + fn write_row(&mut self, y: u32, row: &[u16]) -> Result<(), Refused> { + self.take(y)?; + self.bytes + .extend(row.iter().flat_map(|sample| sample.to_le_bytes())); + Ok(()) + } +} + +fn fmt_for(components: u16, wide: bool) -> PixelFormat { + match (components, wide) { + (1, false) => PixelFormat::Gray8, + (1, true) => PixelFormat::Gray16, + (4, false) => PixelFormat::Rgba8, + (4, true) => PixelFormat::Rgba16, + (_, false) => PixelFormat::Rgb8, + (_, true) => PixelFormat::Rgb16, + } +} + +fn decode_into(bytes: &[u8], parallelism: CpuDecodeParallelism, fmt: PixelFormat) -> Vec { + let mut decoder = J2kDecoder::new(bytes).expect("decoder"); + decoder.set_cpu_decode_parallelism(parallelism); + let stride = WIDTH as usize * fmt.bytes_per_pixel(); + let mut out = vec![0_u8; stride * HEIGHT as usize]; + decoder + .decode_into(&mut out, stride, fmt) + .expect("full decode"); + out +} + +fn decode_rows( + bytes: &[u8], + parallelism: CpuDecodeParallelism, + wide: bool, + stripe_rows: u32, + sink: &mut Rows, +) -> Result<(), DecodeRowsError> { + let mut decoder = J2kDecoder::new(bytes).expect("decoder"); + decoder.set_cpu_decode_parallelism(parallelism); + decode_rows_with(&mut decoder, wide, stripe_rows, sink) +} + +fn decode_rows_with( + decoder: &mut J2kDecoder<'_>, + wide: bool, + stripe_rows: u32, + sink: &mut Rows, +) -> Result<(), DecodeRowsError> { + let options = J2kRowDecodeOptions::new(stripe_rows); + let outcome = if wide { + decoder.decode_rows_u16_bounded(sink, options) + } else { + decoder.decode_rows_u8_bounded(sink, options) + }?; + assert_eq!(outcome.decoded, j2k_core::Rect::full((WIDTH, HEIGHT))); + Ok(()) +} + +struct Fixture { + name: &'static str, + bytes: Vec, + components: u16, + /// Whether 16-bit rows are compared too. + wide: bool, + /// Whether the rows decode in one streamed region decode, rather than + /// a stripe at a time. + streams: bool, +} + +fn fixtures() -> Vec { + let base = EncodeOptions { + num_decomposition_levels: 4, + ..EncodeOptions::default() + }; + let lossy = EncodeOptions { + reversible: false, + ..base.clone() + }; + let rgba = codestream(4, 8, &base); + vec![ + Fixture { + name: "rgb 9/7", + bytes: codestream(3, 8, &lossy), + components: 3, + wide: true, + streams: true, + }, + Fixture { + name: "rgb 5/3 HT", + bytes: codestream( + 3, + 8, + &EncodeOptions { + use_ht_block_coding: true, + ..base.clone() + }, + ), + components: 3, + wide: true, + streams: true, + }, + Fixture { + name: "gray 12-bit 9/7", + bytes: codestream(1, 12, &lossy), + components: 1, + wide: true, + streams: true, + }, + Fixture { + name: "gray signed 12-bit 5/3", + bytes: encoded(1, 12, true, &base), + components: 1, + wide: true, + streams: true, + }, + // Tiles as wide as the image: one region decode, a tile after another. + Fixture { + name: "rgb tiles spanning the width", + bytes: codestream( + 3, + 8, + &EncodeOptions { + tile_size: Some((WIDTH, 48)), + ..lossy.clone() + }, + ), + components: 3, + wide: true, + streams: true, + }, + // Rows of one tile arrive before the next tile's: a stripe at a time. + Fixture { + name: "rgb tiles side by side", + bytes: codestream( + 3, + 8, + &EncodeOptions { + tile_size: Some((64, 64)), + ..lossy.clone() + }, + ), + components: 3, + wide: true, + streams: false, + }, + // A channel definition keeps region decodes on component planes. + Fixture { + name: "rgba with channel definition", + bytes: wrap_jp2_rgba_codestream(&rgba, WIDTH, HEIGHT, 8), + components: 4, + wide: true, + streams: false, + }, + // No wavelet levels to stream through. + Fixture { + name: "rgb without decomposition levels", + bytes: codestream( + 3, + 8, + &EncodeOptions { + num_decomposition_levels: 0, + ..base.clone() + }, + ), + components: 3, + wide: false, + streams: false, + }, + ] +} + +fn parallelisms() -> [(&'static str, CpuDecodeParallelism, usize); 4] { + [ + ("serial", CpuDecodeParallelism::Serial, 1), + ("auto on 1 thread", CpuDecodeParallelism::Auto, 1), + ("auto on 4 threads", CpuDecodeParallelism::Auto, 4), + ("auto on 11 threads", CpuDecodeParallelism::Auto, 11), + ] +} + +#[test] +fn bounded_rows_match_full_decode_at_every_stripe_height_pool_and_depth() { + let fixtures = fixtures(); + for (pool_name, parallelism, threads) in parallelisms() { + let pool = ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("pool"); + pool.install(|| { + for fixture in &fixtures { + for wide in [false, true].into_iter().filter(|&wide| !wide || fixture.wide) { + let fmt = fmt_for(fixture.components, wide); + let expected = decode_into(&fixture.bytes, parallelism, fmt); + // One-row stripes of a stripe-at-a-time decode take a + // region decode per row: serially only. + let one_row = fixture.streams || parallelism == CpuDecodeParallelism::Serial; + let heights = [1, 7, 64, HEIGHT, 4 * HEIGHT]; + for stripe_rows in heights.into_iter().filter(|&rows| rows > 1 || one_row) { + let mut sink = Rows::default(); + decode_rows(&fixture.bytes, parallelism, wide, stripe_rows, &mut sink) + .unwrap_or_else(|error| { + panic!( + "{} {fmt:?} {stripe_rows} rows, {pool_name}: {error}", + fixture.name + ) + }); + assert_eq!( + sink.order, + (0..HEIGHT).collect::>(), + "{} {fmt:?} {stripe_rows} rows, {pool_name}: rows in order, once", + fixture.name + ); + assert!( + sink.bytes == expected, + "{} {fmt:?} {stripe_rows} rows, {pool_name}: rows differ from decode_into", + fixture.name + ); + } + } + } + }); + } +} + +#[test] +fn a_refused_row_stops_the_rows_and_is_reported_as_the_sinks_error() { + let fixtures = fixtures(); + for (pool_name, parallelism, threads) in parallelisms() { + let pool = ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("pool"); + pool.install(|| { + for fixture in &fixtures { + for wide in [false, true].into_iter().filter(|&wide| !wide || fixture.wide) { + let fmt = fmt_for(fixture.components, wide); + let expected = decode_into(&fixture.bytes, parallelism, fmt); + let row_bytes = WIDTH as usize * fmt.bytes_per_pixel(); + for (stripe_rows, refuse) in [(1, 0), (7, 50), (64, 100), (HEIGHT, HEIGHT - 1)] + { + let mut decoder = J2kDecoder::new(&fixture.bytes).expect("decoder"); + decoder.set_cpu_decode_parallelism(parallelism); + let mut sink = Rows { + refuse: Some(refuse), + ..Rows::default() + }; + let result = decode_rows_with(&mut decoder, wide, stripe_rows, &mut sink); + let context = + format!("{} {fmt:?} {stripe_rows} rows, {pool_name}", fixture.name); + assert!( + matches!(result, Err(DecodeRowsError::Sink(Refused(row))) if row == refuse), + "{context}: {result:?}" + ); + // The rows before the refused one, then the refused + // one, and nothing after it. + assert_eq!( + sink.order, + (0..=refuse).collect::>(), + "{context}: rows handed over" + ); + assert!( + sink.bytes == expected[..refuse as usize * row_bytes], + "{context}: rows before the refused one" + ); + // The decoder decodes every row again afterwards. + if stripe_rows == 7 { + let mut again = Rows::default(); + decode_rows_with(&mut decoder, wide, stripe_rows, &mut again) + .unwrap_or_else(|error| panic!("{context}: again: {error}")); + assert!(again.bytes == expected, "{context}: rows after a refusal"); + } + } + } + } + }); + } +} + +#[test] +fn a_codestream_that_fails_before_its_rows_hands_over_none_and_reports_the_decode_error() { + let mut bytes = codestream( + 3, + 8, + &EncodeOptions { + num_decomposition_levels: 4, + reversible: false, + ..EncodeOptions::default() + }, + ); + // Without its end-of-codestream marker the tile data is rejected. + assert_eq!(bytes.pop(), Some(0xd9)); + assert_eq!(bytes.pop(), Some(0xff)); + for (pool_name, parallelism, threads) in parallelisms() { + let pool = ThreadPoolBuilder::new() + .num_threads(threads) + .build() + .expect("pool"); + pool.install(|| { + for wide in [false, true] { + let mut sink = Rows::default(); + let result = decode_rows(&bytes, parallelism, wide, 7, &mut sink); + assert!( + matches!(result, Err(DecodeRowsError::Decode(_))), + "{pool_name} wide={wide}: {result:?}" + ); + assert!(sink.order.is_empty(), "{pool_name} wide={wide}: no rows"); + } + }); + } +} diff --git a/crates/j2k/tests/decode.rs b/crates/j2k/tests/decode.rs index eeaac395b..2cf21d67a 100644 --- a/crates/j2k/tests/decode.rs +++ b/crates/j2k/tests/decode.rs @@ -1015,16 +1015,21 @@ enum StridedRoute { /// Decode `bytes` on `route` to `fmt` at a stride three bytes past the /// output rows, into a buffer of exactly [`j2k_core::strided_output_len`] /// bytes when `tight`, which has no padding after its last row, and of one -/// padded row per output row otherwise. Every byte starts as `0xa5`. +/// padded row per output row otherwise. Every byte starts as `0xa5`. A +/// `planes_budget` holds full-image decodes' planes to it: 0 streams every +/// decode that can stream, `usize::MAX` keeps them on the planes. fn decode_to_strided_buffer( bytes: &[u8], - parallelism: CpuDecodeParallelism, + (parallelism, planes_budget): (CpuDecodeParallelism, Option), route: StridedRoute, fmt: PixelFormat, tight: bool, ) -> Vec { let mut decoder = J2kDecoder::new(bytes).expect("decoder"); decoder.set_cpu_decode_parallelism(parallelism); + if let Some(bytes) = planes_budget { + decoder.set_full_decode_planes_budget(bytes); + } let dims = decoder.info().dimensions; let roi = Rect { x: 3, @@ -1062,8 +1067,7 @@ fn decode_to_strided_buffer( #[test] fn decode_writes_the_last_row_of_a_buffer_without_trailing_padding() { - let (width, height) = (45_u32, 31_u32); - let encode_image = |components: u16, bit_depth: u8, reversible: bool| { + let encode_image = |(width, height): (u32, u32), components: u16, bit_depth: u8, tile| { let bytes_per_sample = u32::from(bit_depth).div_ceil(8); let mut pixels: Vec = (0..width * height * u32::from(components) * bytes_per_sample) .map(|i| (i * 37 + i / 11).to_le_bytes()[0]) @@ -1075,8 +1079,9 @@ fn decode_writes_the_last_row_of_a_buffer_without_trailing_padding() { } } let options = EncodeOptions { - reversible, + reversible: components != 3, num_decomposition_levels: 3, + tile_size: tile, ..EncodeOptions::default() }; encode( @@ -1084,11 +1089,14 @@ fn decode_writes_the_last_row_of_a_buffer_without_trailing_padding() { ) .expect("encode") }; - let gray8 = encode_image(1, 8, true); - let gray12 = encode_image(1, 12, false); - let rgb8 = encode_image(3, 8, false); - let rgb12 = encode_image(3, 12, true); - let rgba8_jp2 = wrap_jp2_rgba_codestream(&encode_image(4, 8, true), width, height, 8); + let small = (45, 31); + let gray8 = encode_image(small, 1, 8, None); + let gray12 = encode_image(small, 1, 12, None); + let rgb8 = encode_image(small, 3, 8, None); + let rgb12 = encode_image(small, 3, 12, None); + let rgba8_jp2 = wrap_jp2_rgba_codestream(&encode_image(small, 4, 8, None), 45, 31, 8); + // Six tiles, which a two-thread pool streams three at a time. + let rgb8_tiled = encode_image((768, 512), 3, 8, Some((256, 256))); let gray_formats = [PixelFormat::Gray8, PixelFormat::Gray16]; let color_formats = [ PixelFormat::Rgb8, @@ -1104,24 +1112,38 @@ fn decode_writes_the_last_row_of_a_buffer_without_trailing_padding() { ("rgba8", &rgba8_jp2, &color_formats), ]; let mut failures = Vec::new(); + let mut compare = |name: &str, bytes: &[u8], settings, route, fmt| { + let padded = decode_to_strided_buffer(bytes, settings, route, fmt, false); + let tight = decode_to_strided_buffer(bytes, settings, route, fmt, true); + if tight != padded[..tight.len()] { + failures.push(format!("{name} {fmt:?} {route:?} {settings:?}")); + } + }; for (name, bytes, formats) in cases { for &fmt in formats { - for route in [ - StridedRoute::Full, - StridedRoute::Region, - StridedRoute::Scaled, - StridedRoute::RegionScaled, - ] { - for parallelism in [CpuDecodeParallelism::Auto, CpuDecodeParallelism::Serial] { - let padded = decode_to_strided_buffer(bytes, parallelism, route, fmt, false); - let tight = decode_to_strided_buffer(bytes, parallelism, route, fmt, true); - if tight != padded[..tight.len()] { - failures.push(format!("{name} {fmt:?} {route:?} {parallelism:?}")); + for parallelism in [CpuDecodeParallelism::Auto, CpuDecodeParallelism::Serial] { + // The planes, the streamed rows, and the default route. + for budget in [Some(usize::MAX), Some(0), None] { + for route in [StridedRoute::Full, StridedRoute::Scaled] { + compare(name, bytes, (parallelism, budget), route, fmt); } } + for route in [StridedRoute::Region, StridedRoute::RegionScaled] { + compare(name, bytes, (parallelism, None), route, fmt); + } } } } + let pool = rayon::ThreadPoolBuilder::new() + .num_threads(2) + .build() + .expect("pool"); + pool.install(|| { + for fmt in color_formats { + let settings = (CpuDecodeParallelism::Auto, Some(0)); + compare("rgb8 tiled", &rgb8_tiled, settings, StridedRoute::Full, fmt); + } + }); assert!( failures.is_empty(), "a buffer that ends after the last row's pixels must hold the padded decode's \ diff --git a/crates/j2k/tests/full_image_streaming.rs b/crates/j2k/tests/full_image_streaming.rs new file mode 100644 index 000000000..9a49d7ed5 --- /dev/null +++ b/crates/j2k/tests/full_image_streaming.rs @@ -0,0 +1,232 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! An 8-bit full-image decode whose component planes would pass the decode +//! budget, or for which streaming is the faster route, runs as a region +//! decode of the whole image that streams its rows (several tiles at once +//! when it has them and the pool to run them). A decoder whose planes budget +//! is fixed takes either route on demand, and the streamed route must write +//! the bytes the planes route writes, at every scale, stride and +//! parallelism, and fail where it fails. + +use j2k::{CpuDecodeParallelism, J2kDecoder, J2kScratchPool}; +use j2k_core::{Downscale, PixelFormat}; +use j2k_native::{encode, EncodeOptions}; +use j2k_test_support::wrap_jp2_rgba_codestream; + +const WIDTH: u32 = 203; +const HEIGHT: u32 = 157; +const PADDING: u8 = 0xab; + +fn noisy_pixels(components: u32) -> Vec { + let mut state = 0x2545_f491_u32; + (0..WIDTH * HEIGHT * components) + .map(|_| { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + (state >> 24) as u8 + }) + .collect() +} + +fn codestream(components: u16, options: &EncodeOptions) -> Vec { + let pixels = noisy_pixels(u32::from(components)); + encode(&pixels, WIDTH, HEIGHT, components, 8, false, options).expect("encode fixture") +} + +/// The image at `scale` in a buffer padded past each row, or `None` when the +/// decode fails, from a decoder whose planes budget is `planes_budget` when +/// one is given. +fn decode( + bytes: &[u8], + planes_budget: Option, + parallelism: CpuDecodeParallelism, + fmt: PixelFormat, + scale: Downscale, + padding: usize, +) -> Option> { + let mut decoder = J2kDecoder::new(bytes).expect("decoder"); + decoder.set_cpu_decode_parallelism(parallelism); + if let Some(bytes) = planes_budget { + decoder.set_full_decode_planes_budget(bytes); + } + let (width, height) = decoder.info().dimensions; + let divisor = scale.denominator(); + let (width, height) = (width.div_ceil(divisor), height.div_ceil(divisor)); + let row = width as usize * fmt.bytes_per_pixel(); + let stride = row + padding; + let mut out = vec![PADDING; stride * height as usize]; + let pool = &mut J2kScratchPool::new(); + let result = match scale { + Downscale::None => decoder.decode_into(&mut out, stride, fmt), + _ => decoder.decode_scaled_into(pool, &mut out, stride, fmt, scale), + }; + result.ok()?; + for line in out.chunks(stride) { + assert!( + line[row..].iter().all(|&byte| byte == PADDING), + "stride padding untouched" + ); + } + Some(out) +} + +#[test] +fn streamed_full_image_matches_planes() { + let base = EncodeOptions { + num_decomposition_levels: 4, + ..EncodeOptions::default() + }; + let rgb = codestream( + 3, + &EncodeOptions { + reversible: false, + ..base.clone() + }, + ); + let rgb_tiled_ht = codestream( + 3, + &EncodeOptions { + reversible: true, + tile_size: Some((64, 64)), + use_ht_block_coding: true, + ..base.clone() + }, + ); + // Enough tiles for every worker of a large pool: decoded tiles at once. + let rgb_many_tiles = codestream( + 3, + &EncodeOptions { + reversible: false, + tile_size: Some((16, 16)), + ..base.clone() + }, + ); + let gray = codestream( + 1, + &EncodeOptions { + reversible: true, + ..base.clone() + }, + ); + let gray_tiled = codestream( + 1, + &EncodeOptions { + reversible: false, + tile_size: Some((96, 64)), + ..base.clone() + }, + ); + let rgba = wrap_jp2_rgba_codestream( + &codestream( + 4, + &EncodeOptions { + reversible: false, + ..base + }, + ), + WIDTH, + HEIGHT, + 8, + ); + let (rgb_formats, gray_formats) = ( + &[PixelFormat::Rgb8, PixelFormat::Rgba8, PixelFormat::Gray8][..], + &[PixelFormat::Gray8, PixelFormat::Rgb8][..], + ); + let fixtures = [ + ("rgb", rgb, rgb_formats), + ("rgb tiled HT", rgb_tiled_ht, rgb_formats), + ("rgb many tiles", rgb_many_tiles, rgb_formats), + ("gray", gray, gray_formats), + ("gray tiled", gray_tiled, gray_formats), + ("rgba", rgba, rgb_formats), + ]; + let mut decoded = 0; + for (label, bytes, formats) in &fixtures { + for &fmt in *formats { + for scale in [Downscale::None, Downscale::Half, Downscale::Eighth] { + for (parallelism, padding) in [ + (CpuDecodeParallelism::Serial, 5), + (CpuDecodeParallelism::Auto, 0), + ] { + let planes = decode(bytes, Some(usize::MAX), parallelism, fmt, scale, padding); + let streamed = decode(bytes, Some(0), parallelism, fmt, scale, padding); + let default = decode(bytes, None, parallelism, fmt, scale, padding); + assert!( + planes == streamed && planes == default, + "{label} {fmt:?} {scale:?} {parallelism:?}: planes {} streamed {} default {}", + planes.is_some(), + streamed.is_some(), + default.is_some() + ); + decoded += usize::from(planes.is_some()); + } + } + } + } + // Gray to RGB and RGB to gray are rejected; the rest must decode. + assert!(decoded >= 9 * 3 * 2, "{decoded} decodes succeeded"); +} + +/// Tiles whose coarse resolutions are empty while finer ones are not (a +/// one-sample tile row or column at an odd position, with two levels) +/// stream past a lowered budget like any other, and the streamed route +/// writes what the planes route writes. +#[test] +fn tiles_with_empty_coarse_levels_stream_as_the_planes_decode() { + let gray: &[u8] = include_bytes!( + "../../j2k-test-support/fixtures/empty_coarse_levels/openjpeg_gray_8x6_tile8x5_53.j2k" + ); + let rgb: &[u8] = include_bytes!( + "../../j2k-test-support/fixtures/empty_coarse_levels/openjpeg_rgb_6x6_tile5x5_97.j2k" + ); + for (label, bytes, fmt) in [ + ("gray 5/3", gray, PixelFormat::Gray8), + ("rgb 9/7", rgb, PixelFormat::Rgb8), + ] { + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + for scale in [Downscale::None, Downscale::Half] { + let planes = decode(bytes, Some(usize::MAX), parallelism, fmt, scale, 0); + let lowered = decode(bytes, Some(0), parallelism, fmt, scale, 0); + let default = decode(bytes, None, parallelism, fmt, scale, 0); + assert!(planes.is_some(), "{label} {parallelism:?} {scale:?}"); + assert_eq!(planes, lowered, "{label} {parallelism:?} {scale:?}"); + assert_eq!(planes, default, "{label} {parallelism:?} {scale:?}"); + } + } + } +} + +/// A decoder that streamed one full-image decode decodes the next on the +/// planes, and the other way round, from the context it kept. +#[test] +fn routes_alternate_on_one_decoder() { + let bytes = codestream( + 3, + &EncodeOptions { + num_decomposition_levels: 3, + reversible: false, + ..EncodeOptions::default() + }, + ); + let fmt = PixelFormat::Rgb8; + let stride = WIDTH as usize * 3; + let expected = decode( + &bytes, + None, + CpuDecodeParallelism::Auto, + fmt, + Downscale::None, + 0, + ) + .expect("planes decode"); + let mut decoder = J2kDecoder::new(&bytes).expect("decoder"); + for budget in [usize::MAX, 0, usize::MAX, 0] { + decoder.set_full_decode_planes_budget(budget); + let mut out = vec![0; stride * HEIGHT as usize]; + decoder + .decode_into(&mut out, stride, fmt) + .unwrap_or_else(|error| panic!("budget {budget}: {error:?}")); + assert!(out == expected, "budget {budget}"); + } +} diff --git a/crates/j2k/tests/owned_batch/cpu_reuse.rs b/crates/j2k/tests/owned_batch/cpu_reuse.rs index f0b4b50e3..bcfeb19c8 100644 --- a/crates/j2k/tests/owned_batch/cpu_reuse.rs +++ b/crates/j2k/tests/owned_batch/cpu_reuse.rs @@ -356,8 +356,10 @@ fn cpu_session_retained_workspace_stabilizes_during_thousand_batch_soak() { assert_eq!(soaked.prepared_plan_decode_calls(), 1_001); assert_eq!(warmed.decode_calls(), 0); assert_eq!(soaked.decode_calls(), 0); - assert_eq!(warmed.flattened_group_plans(), 1); - assert_eq!(soaked.flattened_group_plans(), 1_001); + // A lone image decodes its plan with inner parallelism, so its payloads + // are never flattened for the staged route. + assert_eq!(warmed.flattened_group_plans(), 0); + assert_eq!(soaked.flattened_group_plans(), 0); assert!(warmed.retained_prepared_plan_classic_workspace_bytes() > 0); assert_eq!(soaked.scratch_capacity_retries(), 0); assert_eq!( diff --git a/crates/j2k/tests/owned_batch/native_types_and_requests.rs b/crates/j2k/tests/owned_batch/native_types_and_requests.rs index c79697b9e..e3ea4e33a 100644 --- a/crates/j2k/tests/owned_batch/native_types_and_requests.rs +++ b/crates/j2k/tests/owned_batch/native_types_and_requests.rs @@ -86,7 +86,9 @@ fn prepared_htj2k_gray_and_rgb_support_native_types_and_requests_exactly() { let stats = decoder.workspace_stats(); assert_eq!(stats.preparation_calls(), requests.len() as u64); assert_eq!(stats.prepared_plan_decode_calls(), requests.len() as u64); - assert_eq!(stats.flattened_group_plans(), requests.len() as u64); + // Each request is a group of one image, which decodes its + // plan with inner parallelism and flattens no payloads. + assert_eq!(stats.flattened_group_plans(), 0); assert_eq!(stats.output_group_allocations(), requests.len() as u64); assert_eq!(stats.output_compaction_copied_samples(), 0); } diff --git a/crates/j2k/tests/owned_batch_fixtures/ht_matrix/pass_bucket_parity.rs b/crates/j2k/tests/owned_batch_fixtures/ht_matrix/pass_bucket_parity.rs index 533b83599..d4488fc34 100644 --- a/crates/j2k/tests/owned_batch_fixtures/ht_matrix/pass_bucket_parity.rs +++ b/crates/j2k/tests/owned_batch_fixtures/ht_matrix/pass_bucket_parity.rs @@ -1,6 +1,6 @@ // SPDX-License-Identifier: MIT OR Apache-2.0 -use std::{collections::BTreeSet, sync::Arc}; +use std::{collections::BTreeSet, num::NonZeroUsize, sync::Arc}; use j2k::{ prepare_batch, BatchDecodeOptions, BatchLayout, CpuBatchDecoder, CpuBatchSamples, EncodedImage, @@ -87,9 +87,12 @@ fn external_cleanup_magref_and_generated_sigprop_jobs_decode_in_batches() { let (cleanup, cleanup_pixels) = layered_ht_fixture(1); let (sigprop, sigprop_pixels) = layered_ht_fixture(2); let (magref, magref_pixels) = layered_ht_fixture(3); + // One worker decodes a group's images serially, on the staged route that + // flattens their payloads into pass-bucket jobs. let options = { let mut options = BatchDecodeOptions::default(); options.layout = BatchLayout::Nhwc; + options.workers = NonZeroUsize::new(1); options }; let prepared = prepare_batch( diff --git a/crates/j2k/tests/region_rows.rs b/crates/j2k/tests/region_rows.rs index 33fdbd4ef..492fa1855 100644 --- a/crates/j2k/tests/region_rows.rs +++ b/crates/j2k/tests/region_rows.rs @@ -169,3 +169,83 @@ fn serial_region_rows_match_parallel_planes() { // Gray to RGB and RGB to gray are rejected; the rest must decode. assert!(decoded > 50, "{decoded} decodes succeeded"); } + +/// The image decoded into a tight buffer, and its row length. +fn decode_full( + bytes: &[u8], + parallelism: CpuDecodeParallelism, + fmt: PixelFormat, +) -> (Vec, usize) { + let mut decoder = J2kDecoder::new(bytes).expect("decoder"); + decoder.set_cpu_decode_parallelism(parallelism); + let (width, height) = decoder.info().dimensions; + let stride = width as usize * fmt.bytes_per_pixel(); + let mut out = vec![0; stride * height as usize]; + decoder + .decode_into(&mut out, stride, fmt) + .expect("full decode"); + (out, stride) +} + +/// A region of a tile whose coarse resolutions are empty (a one-sample tile +/// row or column at an odd position, with two levels) decodes the full +/// image's pixels, on the streamed rows and on the planes, at both bit depths. +#[test] +fn region_of_tile_with_empty_coarse_levels_matches_full_decode() { + let gray: &[u8] = include_bytes!( + "../../j2k-test-support/fixtures/empty_coarse_levels/openjpeg_gray_8x6_tile8x5_53.j2k" + ); + let rgb: &[u8] = include_bytes!( + "../../j2k-test-support/fixtures/empty_coarse_levels/openjpeg_rgb_6x6_tile5x5_97.j2k" + ); + let fixtures = [ + ( + "gray 5/3", + gray, + [PixelFormat::Gray8, PixelFormat::Gray16], + [(0, 5, 8, 1), (3, 5, 1, 1), (2, 3, 5, 3), (0, 0, 8, 6)], + ), + ( + "rgb 9/7", + rgb, + [PixelFormat::Rgb8, PixelFormat::Rgb16], + [(5, 0, 1, 6), (5, 5, 1, 1), (0, 5, 6, 1), (3, 2, 3, 4)], + ), + ]; + for (label, bytes, formats, regions) in fixtures { + for fmt in formats { + let bytes_per_pixel = fmt.bytes_per_pixel(); + for parallelism in [CpuDecodeParallelism::Auto, CpuDecodeParallelism::Serial] { + let (full, full_stride) = decode_full(bytes, parallelism, fmt); + for (x, y, w, h) in regions { + let roi = Rect { x, y, w, h }; + let stride = w as usize * bytes_per_pixel; + let mut out = vec![0; stride * h as usize]; + let mut decoder = J2kDecoder::new(bytes).expect("decoder"); + decoder.set_cpu_decode_parallelism(parallelism); + decoder + .decode_region_into(&mut J2kScratchPool::new(), &mut out, stride, fmt, roi) + .unwrap_or_else(|error| { + panic!("{label} {fmt:?} {parallelism:?} {roi:?}: {error:?}") + }); + let crop: Vec = full + .chunks(full_stride) + .skip(y as usize) + .take(h as usize) + .flat_map(|row| &row[x as usize * bytes_per_pixel..][..stride]) + .copied() + .collect(); + assert_eq!(out, crop, "{label} {fmt:?} {parallelism:?} {roi:?}"); + } + } + } + } + // The reversible image decodes losslessly. + let expected: Vec = (0..6_u32) + .flat_map(|y| (0..8_u32).map(move |x| ((13 * x + 29 * y + 7) % 251) as u8)) + .collect(); + assert_eq!( + decode_full(gray, CpuDecodeParallelism::Auto, PixelFormat::Gray8).0, + expected + ); +} diff --git a/crates/j2k/tests/resource_limits.rs b/crates/j2k/tests/resource_limits.rs index 3b82a1ebd..9a69c7959 100644 --- a/crates/j2k/tests/resource_limits.rs +++ b/crates/j2k/tests/resource_limits.rs @@ -58,8 +58,18 @@ const TINY_ROI: Rect = Rect { #[test] fn decoder_component_planes_over_the_cap_report_requested_and_cap_bytes() { let codestream = large_siz_codestream(65_537); - let mut decoder = J2kDecoder::new(&codestream).expect("large SIZ parses"); + // Past the budget, native component planes stream as a region decode of + // the whole image, which refuses the claimed tile as too large without + // byte counts. + let error = J2kDecoder::new(&codestream) + .expect("large SIZ parses") + .decode_native_components() + .expect_err("a 65537x65537 tile exceeds the decode limits"); + resource_limit(&error); + // The planes route reports the bytes its planes need and the cap. + let mut decoder = J2kDecoder::new(&codestream).expect("large SIZ parses"); + decoder.set_full_decode_planes_budget(usize::MAX); let error = decoder .decode_native_components() .expect_err("65537x65537 planes exceed the decode cap"); @@ -71,15 +81,19 @@ fn decoder_component_planes_over_the_cap_report_requested_and_cap_bytes() { #[test] fn decode_into_over_the_cap_reports_requested_and_cap_bytes() { // 11600x11600 gray needs 538 MB of f32 planes, over the 512 MiB decode - // cap. The decode is rejected before it writes the caller's buffer. + // cap. The decode is rejected before it writes the caller's buffer. By + // default, 8- and 16-bit decodes past the cap stream as a region of the + // whole image instead (and fail on this codestream's packets), so the + // planes route is pinned. let dimension = 11_600_u32; let codestream = large_siz_codestream(dimension); let mut decoder = J2kDecoder::new(&codestream).expect("large SIZ parses"); - let stride = dimension as usize; - let mut out = vec![0_u8; stride * stride]; + decoder.set_full_decode_planes_budget(usize::MAX); + let stride = dimension as usize * 2; + let mut out = vec![0_u8; stride * dimension as usize]; let error = decoder - .decode_into(&mut out, stride, PixelFormat::Gray8) + .decode_into(&mut out, stride, PixelFormat::Gray16) .expect_err("the planes exceed the decode cap"); let (requested, cap) = resource_limit(&error); diff --git a/crates/j2k/tests/streamed_sixteen_bit_and_native.rs b/crates/j2k/tests/streamed_sixteen_bit_and_native.rs new file mode 100644 index 000000000..8b76ff161 --- /dev/null +++ b/crates/j2k/tests/streamed_sixteen_bit_and_native.rs @@ -0,0 +1,499 @@ +// SPDX-License-Identifier: MIT OR Apache-2.0 + +//! A 16-bit full-image decode and a decode into owned native component +//! planes whose `f32` planes would pass the decode budget run as a region +//! decode of the whole image that streams its rows, as 8-bit decodes do, and +//! 16-bit region decodes stream their rows always. A decoder whose planes +//! budget is lowered takes the streamed route for small images, and must +//! write the bytes the planes route writes for every precision and +//! signedness, at every scale, stride and parallelism, and fail where it +//! fails. + +use j2k::{CpuDecodeParallelism, J2kDecodedNativeComponents, J2kDecoder, J2kScratchPool}; +use j2k_core::{Downscale, PixelFormat, Rect}; +use j2k_native::{encode, EncodeOptions}; +use j2k_test_support::wrap_jp2_rgba_codestream; + +const WIDTH: u32 = 203; +const HEIGHT: u32 = 157; +const PADDING: u8 = 0xab; + +const RGB_FORMATS: &[PixelFormat] = &[ + PixelFormat::Rgb16, + PixelFormat::Rgba16, + PixelFormat::Gray16, + PixelFormat::Rgb8, +]; +const GRAY_FORMATS: &[PixelFormat] = &[PixelFormat::Gray16, PixelFormat::Rgb16, PixelFormat::Gray8]; +const RGBA_FORMATS: &[PixelFormat] = &[PixelFormat::Rgba16, PixelFormat::Rgb16, PixelFormat::Rgba8]; + +/// Noise samples of `bit_depth` bits, one to three little-endian bytes +/// each, which the encoder masks to the precision (and sign-extends when +/// signed). +fn noisy_samples(components: u16, bit_depth: u8) -> Vec { + let bytes = usize::from(bit_depth).div_ceil(8); + let mut state = 0x2545_f491_u32 ^ u32::from(bit_depth); + (0..WIDTH as usize * HEIGHT as usize * usize::from(components) * bytes) + .map(|_| { + state ^= state << 13; + state ^= state >> 17; + state ^= state << 5; + (state >> 24) as u8 + }) + .collect() +} + +fn codestream(components: u16, bit_depth: u8, signed: bool, options: &EncodeOptions) -> Vec { + let samples = noisy_samples(components, bit_depth); + encode( + &samples, WIDTH, HEIGHT, components, bit_depth, signed, options, + ) + .expect("encode fixture") +} + +/// Images of 4, 8, 12, 16, 20 and 24 bits, signed and unsigned, reversible +/// and irreversible, tiled and untiled, classic and HT, with the output +/// formats each is decoded to (8-bit ones included: they stream past the +/// decode budget too). +fn fixtures() -> Vec<(&'static str, Vec, &'static [PixelFormat])> { + let levels = EncodeOptions { + num_decomposition_levels: 4, + ..EncodeOptions::default() + }; + let irreversible = EncodeOptions { + reversible: false, + ..levels.clone() + }; + let reversible = EncodeOptions { + reversible: true, + ..levels.clone() + }; + vec![ + ( + "rgb 8-bit 9/7", + codestream(3, 8, false, &irreversible), + RGB_FORMATS, + ), + ( + "rgb 8-bit tiled HT 5/3", + codestream( + 3, + 8, + false, + &EncodeOptions { + tile_size: Some((64, 64)), + use_ht_block_coding: true, + ..reversible.clone() + }, + ), + RGB_FORMATS, + ), + ( + "rgb 12-bit signed 9/7", + codestream(3, 12, true, &irreversible), + RGB_FORMATS, + ), + ( + "rgb 16-bit 5/3", + codestream(3, 16, false, &reversible), + RGB_FORMATS, + ), + ( + "gray 12-bit 5/3", + codestream(1, 12, false, &reversible), + GRAY_FORMATS, + ), + ( + "gray 16-bit tiled 9/7", + codestream( + 1, + 16, + false, + &EncodeOptions { + tile_size: Some((96, 64)), + ..irreversible.clone() + }, + ), + GRAY_FORMATS, + ), + ( + "gray 16-bit signed 9/7", + codestream(1, 16, true, &irreversible), + GRAY_FORMATS, + ), + ( + "gray 20-bit 5/3", + codestream(1, 20, false, &reversible), + GRAY_FORMATS, + ), + ( + "rgb 24-bit signed 9/7", + codestream(3, 24, true, &irreversible), + RGB_FORMATS, + ), + ( + "gray 8-bit signed 5/3", + codestream(1, 8, true, &reversible), + GRAY_FORMATS, + ), + ( + "gray 4-bit 5/3", + codestream(1, 4, false, &reversible), + GRAY_FORMATS, + ), + ( + "rgba 8-bit 9/7", + wrap_jp2_rgba_codestream(&codestream(4, 8, false, &irreversible), WIDTH, HEIGHT, 8), + RGBA_FORMATS, + ), + ] +} + +fn decoder( + bytes: &[u8], + planes_budget: Option, + parallelism: CpuDecodeParallelism, +) -> J2kDecoder<'_> { + let mut decoder = J2kDecoder::new(bytes).expect("decoder"); + decoder.set_cpu_decode_parallelism(parallelism); + if let Some(bytes) = planes_budget { + decoder.set_full_decode_planes_budget(bytes); + } + decoder +} + +/// The image at `scale` in a buffer padded past each row, or `None` when the +/// decode fails, from a decoder whose planes budget is `planes_budget` when +/// one is given. +fn decode( + bytes: &[u8], + planes_budget: Option, + parallelism: CpuDecodeParallelism, + fmt: PixelFormat, + scale: Downscale, + padding: usize, +) -> Option> { + let mut decoder = decoder(bytes, planes_budget, parallelism); + let divisor = scale.denominator(); + let (width, height) = (WIDTH.div_ceil(divisor), HEIGHT.div_ceil(divisor)); + let row = width as usize * fmt.bytes_per_pixel(); + let stride = row + padding; + let mut out = vec![PADDING; stride * height as usize]; + let pool = &mut J2kScratchPool::new(); + let result = match scale { + Downscale::None => decoder.decode_into(&mut out, stride, fmt), + _ => decoder.decode_scaled_into(pool, &mut out, stride, fmt, scale), + }; + result.ok()?; + for line in out.chunks(stride) { + assert!( + line[row..].iter().all(|&byte| byte == PADDING), + "stride padding untouched" + ); + } + Some(out) +} + +#[test] +fn streamed_full_image_outputs_match_planes() { + let mut decoded = 0; + for (label, bytes, formats) in &fixtures() { + for &fmt in *formats { + for scale in [Downscale::None, Downscale::Half, Downscale::Eighth] { + for (parallelism, padding) in [ + (CpuDecodeParallelism::Serial, 6), + (CpuDecodeParallelism::Auto, 0), + ] { + let planes = decode(bytes, None, parallelism, fmt, scale, padding); + let streamed = decode(bytes, Some(0), parallelism, fmt, scale, padding); + assert!( + planes == streamed, + "{label} {fmt:?} {scale:?} {parallelism:?}: planes {} streamed {}", + planes.is_some(), + streamed.is_some() + ); + decoded += usize::from(planes.is_some()); + } + } + } + } + // Gray to RGB and RGB to gray are rejected; the rest must decode. + assert!( + decoded >= (4 * 3 + 5 * 2 + 3) * 3 * 2, + "{decoded} decodes succeeded" + ); +} + +fn native_components( + bytes: &[u8], + planes_budget: Option, + parallelism: CpuDecodeParallelism, +) -> J2kDecodedNativeComponents { + decoder(bytes, planes_budget, parallelism) + .decode_native_components() + .expect("native components") +} + +#[test] +fn streamed_native_components_match_planes() { + for (label, bytes, _) in &fixtures() { + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + let planes = native_components(bytes, None, parallelism); + let streamed = native_components(bytes, Some(0), parallelism); + assert_eq!(planes, streamed, "{label} {parallelism:?}"); + } + } +} + +/// Native component region decodes stream their rows into the planes, +/// which must hold the full image's planes cropped to the region. +#[test] +fn native_region_components_match_the_full_image() { + for (label, bytes, _) in &fixtures() { + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + let full = native_components(bytes, None, parallelism); + for roi in regions() { + let region = decoder(bytes, None, parallelism) + .decode_native_region_components(roi) + .unwrap_or_else(|error| panic!("{label} {roi:?}: {error:?}")); + assert_eq!(region.dimensions(), (roi.w, roi.h), "{label} {roi:?}"); + assert_eq!(region.color_space(), full.color_space(), "{label}"); + assert_eq!(region.planes().len(), full.planes().len(), "{label}"); + for (plane, whole) in region.planes().iter().zip(full.planes()) { + assert_eq!(plane.bit_depth(), whole.bit_depth(), "{label}"); + assert_eq!(plane.signed(), whole.signed(), "{label}"); + let bytes = usize::from(whole.bytes_per_sample()); + let row = roi.w as usize * bytes; + for (y, line) in plane.data().chunks(row).enumerate() { + let start = + ((roi.y as usize + y) * WIDTH as usize + roi.x as usize) * bytes; + assert!( + line == &whole.data()[start..start + row], + "{label} {parallelism:?} {roi:?} row {y}" + ); + } + } + } + } + } +} + +fn regions() -> [Rect; 4] { + [ + Rect::full((WIDTH, HEIGHT)), + Rect { + x: 17, + y: 9, + w: 64, + h: 50, + }, + Rect { + x: WIDTH - 33, + y: HEIGHT - 21, + w: 33, + h: 21, + }, + Rect { + x: 100, + y: 0, + w: 1, + h: HEIGHT, + }, + ] +} + +/// The 16-bit region decodes stream their rows, which must be the rows of +/// the full image the planes decode. +#[test] +fn sixteen_bit_regions_match_the_full_image() { + let pool = &mut J2kScratchPool::new(); + for (label, bytes, formats) in &fixtures() { + for &fmt in formats.iter().filter(|fmt| fmt.bytes_per_pixel() % 2 == 0) { + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + let Some(full) = decode(bytes, None, parallelism, fmt, Downscale::None, 0) else { + continue; + }; + let pixel = fmt.bytes_per_pixel(); + let full_stride = WIDTH as usize * pixel; + for roi in regions() { + let stride = roi.w as usize * pixel + 2; + let mut out = vec![PADDING; stride * roi.h as usize]; + decoder(bytes, None, parallelism) + .decode_region_into(pool, &mut out, stride, fmt, roi) + .unwrap_or_else(|error| panic!("{label} {fmt:?} {roi:?}: {error:?}")); + for (row, line) in out.chunks(stride).enumerate() { + let start = (roi.y as usize + row) * full_stride + roi.x as usize * pixel; + assert!( + line[..roi.w as usize * pixel] + == full[start..start + roi.w as usize * pixel], + "{label} {fmt:?} {parallelism:?} {roi:?} row {row}" + ); + assert_eq!(line[roi.w as usize * pixel..], [PADDING; 2]); + } + } + } + } + } +} + +/// Samples of `bit_depth` bits, at least a quarter of them within 16 bits +/// (signed: within `i16`). +fn deep_samples(components: u16, bit_depth: u8, signed: bool) -> Vec { + let span = 1_i64 << 18; + let offset = if signed { span / 2 } else { 0 }; + let (min, max) = if signed { + (-(1 << (bit_depth - 1)), (1 << (bit_depth - 1)) - 1) + } else { + (0, (1 << bit_depth) - 1) + }; + let count = WIDTH as usize * HEIGHT as usize * usize::from(components); + (0_i64..) + .take(count) + .map(|index| ((index * 7919 + index / 3) % span - offset).clamp(min, max)) + .collect() +} + +/// A 16-bit output of an image deeper than 16 bits holds each sample's +/// value clamped to the 16-bit range (two's complement when signed), as a +/// scaled decode always wrote it, on the planes route, on the streamed +/// route and in region decodes; an 8-bit output of an unsigned one holds +/// the sample over the component's range, on either route. +#[test] +fn deep_samples_decode_to_their_values_clamped_to_sixteen_bits() { + let reversible = EncodeOptions { + num_decomposition_levels: 3, + reversible: true, + ..EncodeOptions::default() + }; + let pool = &mut J2kScratchPool::new(); + for (components, bit_depth, signed, fmt) in [ + (1_u16, 20_u8, false, PixelFormat::Gray16), + (1, 24, true, PixelFormat::Gray16), + (3, 20, true, PixelFormat::Rgb16), + (3, 17, false, PixelFormat::Rgb16), + ] { + let label = format!("{components} x {bit_depth} bits, signed {signed}"); + let values = deep_samples(components, bit_depth, signed); + let samples: Vec = values + .iter() + .flat_map(|value| { + let bytes = value.to_le_bytes(); + [bytes[0], bytes[1], bytes[2]] + }) + .collect(); + let bytes = encode( + &samples, + WIDTH, + HEIGHT, + components, + bit_depth, + signed, + &reversible, + ) + .expect("encode deep fixture"); + let expected: Vec = values + .iter() + .flat_map(|&value| { + let clamped = if signed { + let value = i16::try_from(value.clamp(-32_768, 32_767)).expect("i16"); + u16::from_le_bytes(value.to_le_bytes()) + } else { + u16::try_from(value.clamp(0, 65_535)).expect("u16") + }; + clamped.to_le_bytes() + }) + .collect(); + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + for budget in [None, Some(0)] { + let decoded = decode(&bytes, budget, parallelism, fmt, Downscale::None, 0) + .unwrap_or_else(|| panic!("{label} {budget:?}: decode failed")); + assert!( + decoded == expected, + "{label} {parallelism:?} planes budget {budget:?}" + ); + } + if !signed { + let fmt = if components == 1 { + PixelFormat::Gray8 + } else { + PixelFormat::Rgb8 + }; + #[expect( + clippy::cast_precision_loss, + reason = "the unsigned fixtures are at most 20 bits deep, exact in f32" + )] + let max = ((1_u64 << bit_depth) - 1) as f32; + #[expect( + clippy::cast_possible_truncation, + clippy::cast_precision_loss, + clippy::cast_sign_loss, + reason = "unsigned samples below 2^20 are exact in f32 and scale into 0..=255" + )] + let expected: Vec = values + .iter() + .map(|&value| ((value as f32 / max) * 255.0).round() as u8) + .collect(); + for budget in [None, Some(0)] { + let decoded = decode(&bytes, budget, parallelism, fmt, Downscale::None, 0); + assert!( + decoded.as_ref() == Some(&expected), + "{label} {fmt:?} {parallelism:?} planes budget {budget:?}" + ); + } + } + let pixel = fmt.bytes_per_pixel(); + for roi in regions() { + let stride = roi.w as usize * pixel; + let mut out = vec![PADDING; stride * roi.h as usize]; + decoder(&bytes, None, parallelism) + .decode_region_into(pool, &mut out, stride, fmt, roi) + .unwrap_or_else(|error| panic!("{label} {roi:?}: {error:?}")); + for (row, line) in out.chunks(stride).enumerate() { + let start = ((roi.y as usize + row) * WIDTH as usize + roi.x as usize) * pixel; + assert!( + line == &expected[start..start + stride], + "{label} {parallelism:?} {roi:?} row {row}" + ); + } + } + } + } +} + +/// Tiles whose coarse resolutions are empty while finer ones are not stream +/// past a lowered budget for 16-bit and native component output too, with +/// the planes route's bytes. +#[test] +fn tiles_with_empty_coarse_levels_stream_as_the_planes_decode() { + let gray: &[u8] = include_bytes!( + "../../j2k-test-support/fixtures/empty_coarse_levels/openjpeg_gray_8x6_tile8x5_53.j2k" + ); + let rgb: &[u8] = include_bytes!( + "../../j2k-test-support/fixtures/empty_coarse_levels/openjpeg_rgb_6x6_tile5x5_97.j2k" + ); + for (label, bytes, fmt) in [ + ("gray 5/3", gray, PixelFormat::Gray16), + ("rgb 9/7", rgb, PixelFormat::Rgb16), + ] { + for parallelism in [CpuDecodeParallelism::Serial, CpuDecodeParallelism::Auto] { + let mut planes = decoder(bytes, None, parallelism); + let mut lowered = decoder(bytes, Some(0), parallelism); + let (width, height) = planes.info().dimensions; + let stride = width as usize * fmt.bytes_per_pixel(); + let mut expected = vec![0; stride * height as usize]; + let mut out = vec![0; stride * height as usize]; + planes + .decode_into(&mut expected, stride, fmt) + .unwrap_or_else(|error| panic!("{label}: {error:?}")); + lowered + .decode_into(&mut out, stride, fmt) + .unwrap_or_else(|error| panic!("{label}: {error:?}")); + assert_eq!(out, expected, "{label} {parallelism:?}"); + assert_eq!( + lowered.decode_native_components().expect("native"), + planes.decode_native_components().expect("native"), + "{label} {parallelism:?}" + ); + } + } +} diff --git a/xtask/api/stable-api-1.0.implementation-public-api.txt b/xtask/api/stable-api-1.0.implementation-public-api.txt index 303139432..c32ff36b2 100644 --- a/xtask/api/stable-api-1.0.implementation-public-api.txt +++ b/xtask/api/stable-api-1.0.implementation-public-api.txt @@ -42,6 +42,7 @@ pub fn j2k::J2kDecoder<'a>::decode_scaled_into(&mut self, &mut Self::Pool, &mut pub fn j2k::J2kDecoder<'a>::from_view(Self::View) -> core::result::Result pub fn j2k::J2kDecoder<'a>::inspect(&'a [u8]) -> core::result::Result pub fn j2k::J2kDecoder<'a>::parse(&'a [u8]) -> core::result::Result +pub fn j2k::J2kDecoder<'a>::set_full_decode_planes_budget(&mut self, usize) pub fn j2k::J2kError::is_buffer_error(&self) -> bool pub fn j2k::J2kError::is_not_implemented(&self) -> bool pub fn j2k::J2kError::is_truncated(&self) -> bool @@ -2463,11 +2464,13 @@ pub enum j2k_native::Jp2ColorSpec pub enum j2k_native::Jp2ComponentMappingType pub enum j2k_native::Jp2FileKind pub fn j2k_native::ComponentPlane<'a>::into_parts(self) -> j2k_native::ComponentPlaneParts<'a> +pub fn j2k_native::ComponentRowSink::rgb8_region(&mut self, u32, u32) -> core::option::Option<&mut [u8]> pub fn j2k_native::ComponentRowSink::write_rows(&mut self, u32, u32, &[j2k_native::ComponentPlane<'_>]) pub fn j2k_native::DecodedComponents<'a>::into_parts(self) -> j2k_native::DecodedComponentsParts<'a> pub fn j2k_native::DecodedComponents<'a>::live_bytes(&self) -> usize pub fn j2k_native::DecodedNativeComponents::allocated_bytes(&self) -> core::option::Option pub fn j2k_native::DecodedNativeComponents::into_parts(self) -> j2k_native::DecodedNativeComponentsParts +pub fn j2k_native::DecoderContext<'_>::set_shared_pool(&mut self, bool) pub fn j2k_native::HtCodeBlockDecodeProfile::new() -> Self pub fn j2k_native::HtCodeBlockDecodeWorkspace::coefficient_capacity(&self) -> usize pub fn j2k_native::HtCodeBlockDecoder::decode_code_block(&mut self, j2k_native::HtCodeBlockDecodeJob<'_>, &mut [f32]) -> j2k_native::Result<()> @@ -2499,6 +2502,7 @@ pub fn j2k_native::Image<'a>::decode_components_for_integer_output_with_context< pub fn j2k_native::Image<'a>::decode_components_for_integer_output_with_ht_decoder<'ctx>(&self, &'ctx mut j2k_native::DecoderContext<'a>, &mut dyn j2k_native::HtCodeBlockDecoder) -> j2k_native::Result> pub fn j2k_native::Image<'a>::decode_components_with_ht_decoder<'ctx>(&self, &'ctx mut j2k_native::DecoderContext<'a>, &mut dyn j2k_native::HtCodeBlockDecoder) -> j2k_native::Result> pub fn j2k_native::Image<'a>::decode_native_components_with_retained_capacity(&self, usize) -> j2k_native::Result +pub fn j2k_native::Image<'a>::decode_native_region_components_rows_with_context(&self, (u32, u32, u32, u32), &mut j2k_native::DecoderContext<'a>) -> j2k_native::Result pub fn j2k_native::Image<'a>::decode_native_with_retained_capacity(&self, usize) -> j2k_native::Result pub fn j2k_native::Image<'a>::decode_region_components_for_integer_output_with_context<'ctx>(&self, (u32, u32, u32, u32), &'ctx mut j2k_native::DecoderContext<'a>) -> j2k_native::Result> pub fn j2k_native::Image<'a>::decode_region_components_for_integer_output_with_ht_decoder<'ctx>(&self, &'ctx mut j2k_native::DecoderContext<'a>, (u32, u32, u32, u32), &mut dyn j2k_native::HtCodeBlockDecoder) -> j2k_native::Result> @@ -2506,11 +2510,15 @@ pub fn j2k_native::Image<'a>::decode_region_components_with_ht_decoder<'ctx>(&se pub fn j2k_native::Image<'a>::decode_region_rows_for_integer_output_with_context(&self, (u32, u32, u32, u32), &mut j2k_native::DecoderContext<'a>, &mut dyn j2k_native::ComponentRowSink) -> j2k_native::Result<()> pub fn j2k_native::Image<'a>::decode_reversible_53_coefficients(&self) -> j2k_native::Result pub fn j2k_native::Image<'a>::decode_reversible_53_coefficients_with_context(&self, &mut j2k_native::DecoderContext<'a>) -> j2k_native::Result +pub fn j2k_native::Image<'a>::full_decode_planes_floor(&self, usize) -> j2k_native::Result> pub fn j2k_native::Image<'a>::new_with_reduction(&'a [u8], &j2k_native::DecodeSettings, u8) -> j2k_native::Result pub fn j2k_native::Image<'a>::new_with_retained_baseline(&'a [u8], &j2k_native::DecodeSettings, usize) -> j2k_native::Result +pub fn j2k_native::Image<'a>::output_tile_rects(&self) -> j2k_native::Result> pub fn j2k_native::Image<'a>::prepare_region_decoder_with_context<'image, 'context>(&'image self, &'context mut j2k_native::DecoderContext<'a>) -> j2k_native::Result> pub fn j2k_native::Image<'a>::primary_icc_profile(&self) -> core::option::Option<&[u8]> +pub fn j2k_native::Image<'a>::region_decode_streams_rows(&self) -> bool pub fn j2k_native::Image<'a>::retained_allocation_bytes(&self) -> j2k_native::Result +pub fn j2k_native::Image<'a>::streamed_rows_native_precision(&self) -> core::option::Option pub fn j2k_native::Image<'a>::supports_direct_device_plane_reuse(&self) -> bool pub fn j2k_native::Image<'a>::tile_count(&self) -> u32 pub fn j2k_native::Image<'a>::uses_high_throughput_block_coding(&self) -> bool @@ -2543,6 +2551,7 @@ pub fn j2k_native::ResidentHtj2kEncodeError::fmt(&self, &mut core::fmt::Formatte pub fn j2k_native::ResidentHtj2kEncodeError::source(&self) -> core::option::Option<&(dyn core::error::Error + 'static)> pub fn j2k_native::Reversible53CoefficientImage::encode_htj2k(&self, &j2k_native::EncodeOptions) -> j2k_native::EncodeResult> pub fn j2k_native::collect_ht_cleanup_encode_distribution(&[i32], u32, u32, u8) -> j2k_native::EncodeResult +pub fn j2k_native::cpu_decode_pool_threads() -> usize pub fn j2k_native::decode_ht_code_block_scalar(j2k_native::HtCodeBlockDecodeJob<'_>, &mut [f32]) -> j2k_native::Result<()> pub fn j2k_native::decode_ht_code_block_scalar_until_phase(j2k_native::HtCodeBlockDecodeJob<'_>, &mut [f32], j2k_native::HtCodeBlockDecodePhaseLimit) -> j2k_native::Result<()> pub fn j2k_native::decode_ht_code_block_scalar_with_workspace(j2k_native::HtCodeBlockDecodeJob<'_>, &mut [f32], &mut j2k_native::HtCodeBlockDecodeWorkspace) -> j2k_native::Result<()> @@ -2628,6 +2637,7 @@ pub fn j2k_native::irreversible_quantization_profile_scale_ceiling(u8, &j2k_nati pub fn j2k_native::irreversible_quantization_scale_bounds(u8, &j2k_native::EncodeOptions) -> (f32, f32) pub fn j2k_native::irreversible_quantization_step_for_subband(u8, u8, f32, j2k_types::transform::quantization::IrreversibleQuantizationSubbandScales, j2k_types::tier1::classic::J2kSubBandType, u8, u8) -> j2k_types::transform::quantization::IrreversibleQuantizationStep pub fn j2k_native::pack_j2k_code_block_scalar_from_tier1_tokens(&[u8], &[j2k_native::J2kTier1TokenSegment], u8, u8) -> j2k_native::EncodeResult +pub fn j2k_native::pack_rgb8([&[f32]; 3], &mut [u8]) pub fn j2k_native::packet_math::HtSegmentLengthError::fmt(&self, &mut core::fmt::Formatter<'_>) -> core::fmt::Result pub fn j2k_native::packet_math::bits_for_ht_cleanup_length(u32, u8) -> u32 pub fn j2k_native::packet_math::bits_for_ht_refinement_only_length(u32, u8) -> u32 diff --git a/xtask/release-evidence/public-api/public-api-review-0.13.2.yml b/xtask/release-evidence/public-api/public-api-review-0.13.2.yml index a5536d871..f256ed652 100644 --- a/xtask/release-evidence/public-api/public-api-review-0.13.2.yml +++ b/xtask/release-evidence/public-api/public-api-review-0.13.2.yml @@ -84,10 +84,10 @@ reviews: j2k-native: removed_fingerprint: "none" added_fingerprint: "fnv1a64:22b943651a07318e" - hidden_count: 586 - hidden_fingerprint: "fnv1a64:733f3aa607999371" + hidden_count: 595 + hidden_fingerprint: "fnv1a64:0d8774f02739bc1e" rationale: "Compared the complete ordinary API inventory against v0.13.1. Adds DecodeErrorClass::ResourceLimit { requested, cap } to an already non-exhaustive enum; no existing item changes and existing variant discriminants are preserved. AllocationTooLarge and ImageTooLarge now classify as ResourceLimit instead of the generic Backend class. This is a compatible refinement of the extensible error taxonomy: invalid-input and allocator-failure classes stay unchanged. Callers that want to handle memory limits separately can match the new variant; existing wildcard matches remain valid." - hidden_rationale: "Adds referenced_classic_plan_execution_bytes and referenced_htj2k_plan_execution_bytes for sibling CPU batch adapters to check planned live execution memory before allocation. Both helpers return Result and do not alter existing signatures. Every other hidden item is unchanged from v0.13.1." + hidden_rationale: "Adds referenced_classic_plan_execution_bytes and referenced_htj2k_plan_execution_bytes for sibling CPU batch adapters to check planned live execution memory before allocation. For the facade's streamed routes, j2k-native also adds decode_native_region_components_rows_with_context, full_decode_planes_floor, output_tile_rects, region_decode_streams_rows and streamed_rows_native_precision; cpu_decode_pool_threads reports the pool thread count for facade routing, DecoderContext::set_shared_pool marks CpuBatchDecoder's shared-pool images, and pack_rgb8 shares the existing pixel conversion. ComponentRowSink::rgb8_region is an optional default method, so existing sink implementations remain valid. These are unsupported implementation helpers; existing signatures are unchanged." j2k-jpeg: removed_fingerprint: "none" added_fingerprint: "none" @@ -105,10 +105,10 @@ reviews: j2k: removed_fingerprint: "none" added_fingerprint: "fnv1a64:78b9622111db3437" - hidden_count: 108 - hidden_fingerprint: "fnv1a64:01e74ad6c3ff416e" - rationale: "Compared the complete ordinary API inventory against v0.13.1. Adds J2kError::ResourceLimit { requested, cap, source } to an already non-exhaustive enum; no existing item changes and existing variant discriminants are preserved. Budget and image-size failures now use this variant instead of NativeDecode, retaining the structured native cause. Callers specifically handling budget failures through NativeDecode should also handle ResourceLimit. Batches prepared by CpuBatchDecoder report items whose output allocation or planned execution cannot fit as indexed preparation errors and exclude those items from the prepared groups. Backend-neutral prepare_batch and prepare_batch_from_images retain whole-call output-cap failures for a single oversized image when subsequently decoded on the CPU. On either preparation path, a group whose combined output exceeds the allocation cap still fails the whole decode call. These correct admission timing within the CPU-owned preparation paths and preserve the existing per-item failure model; successful decode output and signatures are unchanged. Adds BackendErrorKind::Decode at the end of its existing non-exhaustive enum and BackendError::decode for failures during backend decode execution. Existing discriminants and constructors are preserved. The public error documentation and changelog describe the refinements." - hidden_rationale: "Compared the complete hidden API inventory against v0.13.1; it is unchanged." + hidden_count: 109 + hidden_fingerprint: "fnv1a64:0427dd567531c8dd" + rationale: "Compared the complete ordinary API inventory against v0.13.1. Adds J2kError::ResourceLimit { requested, cap, source } to an already non-exhaustive enum; no existing item changes and existing variant discriminants are preserved. Budget and image-size failures now use this variant instead of NativeDecode, retaining the structured native cause. Callers specifically handling budget failures through NativeDecode should also handle ResourceLimit. Batches prepared by CpuBatchDecoder report items whose output allocation or planned execution cannot fit as indexed preparation errors and exclude those items from the prepared groups. Backend-neutral prepare_batch and prepare_batch_from_images retain whole-call output-cap failures for a single oversized image when subsequently decoded on the CPU. On either preparation path, groups whose combined output exceeds the allocation cap are decoded into several result groups, each with a buffer within the cap. Automatic streamed routes also allow requests that previously exceeded working-memory limits to succeed. The documented deep-sample conversion fixes make 16-bit outputs of images deeper than 16 bits hold the native sample value clamped to the 16-bit range (two's complement when signed), as scaled decodes already did, and quantize 8-bit outputs over the full component precision, correcting the prior wider-sample byte interpretation and 16-bit scaling ceiling. These preserve the output contracts and ordinary decode signatures. Adds BackendErrorKind::Decode at the end of its existing non-exhaustive enum and BackendError::decode for failures during backend decode execution. Existing discriminants and constructors are preserved. The public error documentation and changelog describe the refinements." + hidden_rationale: "Adds set_full_decode_planes_budget as an unsupported test hook for isolated integration tests of automatic planes/streamed routing. Existing public decode methods continue choosing their routes automatically; no supported user configuration is added. All other hidden facade items are unchanged." j2k-transcode: removed_fingerprint: "none" added_fingerprint: "none" diff --git a/xtask/release-evidence/public-api/reviewed-public-api-diff-0.13.2.md b/xtask/release-evidence/public-api/reviewed-public-api-diff-0.13.2.md index 2fcd04888..4815642d1 100644 --- a/xtask/release-evidence/public-api/reviewed-public-api-diff-0.13.2.md +++ b/xtask/release-evidence/public-api/reviewed-public-api-diff-0.13.2.md @@ -22,10 +22,10 @@ This report is generated by `cargo xtask semver --write-report`. Normal `cargo x | `j2k-cuda-transcode-engine` | `0.13.1` | `0.13.2` | `minor` | 0 | 0 | `none` | `none` | 84 | `fnv1a64:b2fcaa9b0171aba7` | | `j2k-metal-support` | `0.13.1` | `0.13.2` | `minor` | 0 | 0 | `none` | `none` | 11 | `fnv1a64:02139606f1202938` | | `j2k-mpsgraph-support` | `0.13.1` | `0.13.2` | `minor` | 0 | 0 | `none` | `none` | 0 | `none` | -| `j2k-native` | `0.13.1` | `0.13.2` | `minor` | 3 | 0 | `none` | `fnv1a64:22b943651a07318e` | 586 | `fnv1a64:733f3aa607999371` | +| `j2k-native` | `0.13.1` | `0.13.2` | `minor` | 3 | 0 | `none` | `fnv1a64:22b943651a07318e` | 595 | `fnv1a64:0d8774f02739bc1e` | | `j2k-jpeg` | `0.13.1` | `0.13.2` | `minor` | 0 | 0 | `none` | `none` | 701 | `fnv1a64:bd0cf738a0f96592` | | `j2k-tilecodec` | `0.13.1` | `0.13.2` | `minor` | 0 | 0 | `none` | `none` | 32 | `fnv1a64:eb677842559d9b0e` | -| `j2k` | `0.13.1` | `0.13.2` | `minor` | 6 | 0 | `none` | `fnv1a64:78b9622111db3437` | 108 | `fnv1a64:01e74ad6c3ff416e` | +| `j2k` | `0.13.1` | `0.13.2` | `minor` | 6 | 0 | `none` | `fnv1a64:78b9622111db3437` | 109 | `fnv1a64:0427dd567531c8dd` | | `j2k-transcode` | `0.13.1` | `0.13.2` | `minor` | 0 | 0 | `none` | `none` | 458 | `fnv1a64:449a76476549c1e2` | | `j2k-transcode-cuda` | `0.13.1` | `0.13.2` | `minor` | 0 | 0 | `none` | `none` | 20 | `fnv1a64:be22ca1d5e12b043` | | `j2k-jpeg-metal` | `0.13.1` | `0.13.2` | `minor` | 0 | 0 | `none` | `none` | 83 | `fnv1a64:79397ffc221464c3` | @@ -174,7 +174,7 @@ None. ### `j2k-native` -Baseline items: 422. Candidate items: 425. Computed release type: `minor`. Rustdoc-hidden candidate items: 586. Full hidden-inventory fingerprint: `fnv1a64:733f3aa607999371`. +Baseline items: 422. Candidate items: 425. Computed release type: `minor`. Rustdoc-hidden candidate items: 595. Full hidden-inventory fingerprint: `fnv1a64:0d8774f02739bc1e`. #### Removed or changed baseline API items @@ -214,7 +214,7 @@ None. ### `j2k` -Baseline items: 804. Candidate items: 810. Computed release type: `minor`. Rustdoc-hidden candidate items: 108. Full hidden-inventory fingerprint: `fnv1a64:01e74ad6c3ff416e`. +Baseline items: 804. Candidate items: 810. Computed release type: `minor`. Rustdoc-hidden candidate items: 109. Full hidden-inventory fingerprint: `fnv1a64:0427dd567531c8dd`. #### Removed or changed baseline API items