fix(j2k): align packet validation across decode routes and backends

CHANGELOG.md

@@ -7,6 +7,20 @@ and stale roadmap entries have been removed from the public documentation set.
 
 Staged workspace version: `0.13.2`.
 
+- SOP marker segments a decode parses require `Lsop` 4 and an `Nsop` equal to
+  the tile-wide packet ordinal modulo 65536. The ordinal continues across
+  tile-parts and counts packets skipped without parsing; an invalid SOP in a
+  discarded packet remains unread by a selective decode.
+- A strict CUDA decode of a classic code-block whose ROI maxshift raises its
+  coded bit-plane count beyond the native decoder's limit fails with the
+  native `TooManyBitplanes` error, as a CPU decode does, instead of reporting
+  an unsupported CUDA request. Counts within the native limit but beyond the
+  device's limit remain unsupported.
+- A tile-part COD replaces all of the main-header COD style flags instead
+  of retaining its SOP and EPH flags. Valid codestreams that disable packet
+  markers for a tile now decode on the CPU, Metal and CUDA, at full and
+  reduced resolution, instead of requiring markers that the packets do not
+  contain.
 - `AdapterErrorKind::Decode`, `BackendErrorKind::Decode` and
   `BackendError::decode` classify GPU Tier-1 device decode failures. Metal FAIL
   statuses now use the existing `Error::Decode` and `J2kError::Backend`
@@ -15,6 +29,56 @@ Staged workspace version: `0.13.2`.
   status, detail and batch source index. CUDA keeps its typed source and job
   attribution. Unsupported statuses, raw runtime failures and combined
   cleanup or release failures keep their existing classes.
+- The CPU batch decoder converts the rows a streamed image hands over a
+  row at a time instead of a sample at a time, with the same rounding and
+  saturation, so the same bytes. The decode hands its rows over on the
+  thread that reads them, between the batches it reconstructs, while the
+  pool's band workers decode only as far ahead as their band rings hold,
+  so the conversion held the whole decode back. On an 18-core ARM64
+  machine a 5661 x 8287 RGB image as a one-image batch decodes in 127 ms
+  instead of 196 ms, at half resolution in 65 ms instead of 79 ms, and
+  four and eight copies with automatic workers in 447 and 876 ms instead
+  of 592 and 1065 ms.
+- A strict CUDA decode of a classic code-block whose packet headers claim
+  more missing bit-planes than its sub-band codes, or more coding passes
+  than its bit-planes hold, fails with a native decode error,
+  `InvalidBitplaneCount` or `TooManyCodingPasses`, of the class the CPU
+  decode reports for the same codestream, on the single-image decode, the
+  batch of images and the prepared batch. It was rejected as an unsupported
+  CUDA request, a different error class.
+- Grayscale direct and referenced plans, which GPU decodes and decode
+  batches execute, reject a one-component codestream whose COD asks for a
+  multiple component transform, as decodes of it already did. A 39 x 7
+  one-bit gray image in five tiles with that flag failed every decode and
+  decoded in decode batches on the CPU and on CUDA.
+- A PLT length shorter than any packet, zero or, where EPH markers are
+  required, below three bytes, for a packet that a region or
+  reduced-resolution decode skips by its length is a packet error. Skipping
+  by such a length made the decode read the next packet it needs from the
+  skipped packet's bytes: in a flat three-component image whose last needed
+  packet was found that way, reduced decodes wrote other colours and
+  reported success. Full decodes already rejected these lengths.
+- Decodes below full resolution read the packets the reduced output needs on
+  every route: the packets of the resolutions they keep, the others only
+  where they must be parsed to find a later needed one and their PLT
+  lengths cannot skip them, and none after the last needed one, as region
+  decodes already did. Decodes on the component planes and the direct plans
+  that GPU decodes and the staged batch run read every packet, so a
+  codestream damaged only in packets of the discarded resolutions decoded or
+  failed depending on the route and the pool's size; it now decodes
+  everywhere, as the undamaged codestream does at that resolution. Damage in
+  a packet the output needs, or in one parsed to find a later needed one,
+  remains an error. Full-resolution decodes are unchanged, except that the
+  packets of precincts holding no code-block, as at a one-sample tile
+  corner, are read on every route as region decodes read them.
+- A tile-part that holds data after its tile's last packet, bytes or PLT
+  packet lengths past it, decodes on every route. Decodes on the component
+  planes failed with a packet parse error where decodes that stream their
+  rows succeeded, so the same codestream decoded or failed depending on the
+  route, which can follow the pool's size (a half-resolution decode of a
+  512 x 512 RGB image failed on one thread and decoded on four). The output
+  is that of the codestream without the extra data. PLT lengths left over
+  while the tile's packets continue in a later tile-part remain an error.
 - Row-streamed decodes on serial bands finish directly from the inverse
   transform's own rows. The streamed inverse transform lifts every level
   with the banded transform's kernels, one arrival at a time on each level's
@@ -195,9 +259,8 @@ Staged workspace version: `0.13.2`.
   image's output exceeds the allocation cap. On either preparation path,
   a group whose combined output exceeds that cap is decoded as several
   result groups, each with an output buffer within the cap.
-  Staged windows are admitted by planned execution memory, and an admission
-  claim that cannot fit beside the shared plan fails instead of waiting
-  indefinitely. Successful output is unchanged.
+  Staged windows stay within the budget computed from their planned
+  execution memory. Successful output is unchanged.
 - 16-bit full-image decodes (`J2kDecoder::decode_into`, `decode_scaled_into`
   and the decoders behind `decode_tiles_into` at `Rgb16`, `Rgba16` and
   `Gray16`) and `J2kDecoder::decode_native_components` whose component

crates/j2k-alloc-probe/tests/main.rs

@@ -9,9 +9,10 @@ use j2k_native::{
     decode_ht_code_block_scalar_with_workspace, encode,
     encode_ht_code_block_scalar_with_passes_and_workspace,
     encode_precomputed_htj2k_97_with_accelerator_and_max_host_bytes,
-    CpuOnlyJ2kEncodeStageAccelerator, DecodeSettings, DecoderContext, EncodeError, EncodeOptions,
-    HtCodeBlockDecodeJob, HtCodeBlockDecodeWorkspace, HtCodeBlockEncodeWorkspace, Image,
-    J2kForwardDwt97Output, PrecomputedHtj2k97Component, PrecomputedHtj2k97Image,
+    CpuOnlyJ2kEncodeStageAccelerator, DecodeErrorClass, DecodeSettings, DecoderContext,
+    EncodeError, EncodeOptions, HtCodeBlockDecodeJob, HtCodeBlockDecodeWorkspace,
+    HtCodeBlockEncodeWorkspace, Image, J2kForwardDwt97Output, PrecomputedHtj2k97Component,
+    PrecomputedHtj2k97Image,
 };
 use proptest::{
     collection::vec,
@@ -72,6 +73,10 @@ fn main() {
             "precomputed_encode_obeys_ledger_and_allocator_budgets",
             precomputed_encode_obeys_ledger_and_allocator_budgets,
         ),
+        (
+            "reduced_decodes_reject_a_precinct_grid_past_the_cap_before_walking_it",
+            reduced_decodes_reject_a_precinct_grid_past_the_cap_before_walking_it,
+        ),
     ];
 
     let mut failures = 0_u32;
@@ -367,6 +372,71 @@ fn warmed_decoder_context_has_bounded_transients() {
     assert_eq!(output, pixels);
 }
 
+/// A raw codestream of one 16,383 x 16,383 gray tile with one decomposition
+/// level, one-sample precincts in the LL sub-band and two-sample ones above:
+/// 2^28 sub-band precincts, whose records no decode can hold within the cap,
+/// declared in 105 bytes. Its one tile-part holds one empty packet.
+fn precinct_grid_past_the_cap() -> Vec<u8> {
+    let mut bytes = vec![0xff, 0x4f, 0xff, 0x51, 0, 41, 0, 0];
+    for value in [16_383_u32, 16_383, 0, 0, 16_383, 16_383, 0, 0] {
+        bytes.extend_from_slice(&value.to_be_bytes());
+    }
+    bytes.extend_from_slice(&[0, 1, 7, 1, 1]);
+    // COD: precincts given; LRCP, one layer; one level, 4 x 4 blocks, 5/3;
+    // precincts 1 x 1 at resolution 0 and 2 x 2 at resolution 1.
+    bytes.extend_from_slice(&[0xff, 0x52, 0, 14, 1, 0, 0, 1, 0, 1, 0, 0, 0, 1, 0x00, 0x11]);
+    // QCD: reversible, two guard bits.
+    bytes.extend_from_slice(&[0xff, 0x5c, 0, 7, 0x40, 0x48, 0x48, 0x48, 0x48]);
+    bytes.extend_from_slice(&[0xff, 0x90, 0, 10, 0, 0, 0, 0, 0, 15, 0, 1]);
+    bytes.extend_from_slice(&[0xff, 0x93, 0, 0xff, 0xd9]);
+    bytes
+}
+
+/// A half-resolution direct plan and planes decode of a tile whose precinct
+/// records cannot fit the cap fail as the build would, before choosing the
+/// packets they read, which would walk and record each of the 2^27 precincts
+/// first: the plan holds no more than 4 MiB at once, the planes decode no
+/// more beside the 8,192 x 8,192 coefficient plane it allocates before it
+/// builds the tile.
+fn reduced_decodes_reject_a_precinct_grid_past_the_cap_before_walking_it() {
+    const PEAK: usize = 4 << 20;
+    const PLANE: usize = 8_192 * 8_192 * size_of::<i32>();
+    let codestream = precinct_grid_past_the_cap();
+    let half = DecodeSettings {
+        target_resolution: Some((8_192, 8_192)),
+        ..DecodeSettings::default()
+    };
+    let image = Image::new(&codestream, &half).expect("parse header");
+    let resource_limit = |error: &j2k_native::DecodeError| {
+        matches!(error.classify(), DecodeErrorClass::ResourceLimit { .. })
+    };
+
+    let base = reset_peak();
+    let plan = image.build_direct_grayscale_plan_with_context(&mut DecoderContext::default());
+    let plan_peak = peak_bytes() - base;
+    assert!(
+        plan.as_ref().is_err_and(resource_limit),
+        "direct plan: {:?}",
+        plan.err()
+    );
+    assert!(plan_peak <= PEAK, "direct plan held {plan_peak} bytes");
+
+    let mut output = vec![0_u8; 8_192 * 8_192];
+    let mut context = DecoderContext::default();
+    let base = reset_peak();
+    let decoded = image.decode_into(&mut output, &mut context);
+    let decode_peak = peak_bytes() - base;
+    assert!(
+        decoded.as_ref().is_err_and(resource_limit),
+        "planes decode: {:?}",
+        decoded.err()
+    );
+    assert!(
+        decode_peak <= PLANE + PEAK,
+        "planes decode held {decode_peak} bytes"
+    );
+}
+
 fn profile_row_formatting_is_single_allocation() {
     let fields = [("route", "scalar"), ("result", "success")];
     drop(

crates/j2k-cuda/src/direct_plan/classic.rs

@@ -1,16 +1,18 @@
 // SPDX-License-Identifier: MIT OR Apache-2.0
 
 use j2k_native::{
-    J2kCodeBlockSegment, J2kCodeBlockStyle, J2kOwnedCodeBlockBatchJob, J2kOwnedSubBandPlan,
-    J2kSubBandType,
+    DecodeError, DecodingError, J2kCodeBlockSegment, J2kCodeBlockStyle, J2kOwnedCodeBlockBatchJob,
+    J2kOwnedSubBandPlan, J2kSubBandType,
 };
 
 use super::{
     required_regions::RequiredBandRegions, shared::CudaPlanOwners, CudaClassicCodeBlock,
     CudaClassicSegment, CudaClassicSubband, Error, PLAN_PAYLOAD_TOO_LARGE,
 };
+use crate::error::NativeBackendError;
 
 const CLASSIC_PLAN_INVALID: &str = "strict CUDA classic Tier-1 plan is invalid";
+const NATIVE_CLASSIC_BITPLANE_LIMIT: u8 = 63;
 const STYLE_RESET_CONTEXT_PROBABILITIES: u32 = 1 << 0;
 const STYLE_TERMINATION_ON_EACH_PASS: u32 = 1 << 1;
 const STYLE_VERTICALLY_CAUSAL_CONTEXT: u32 = 1 << 2;
@@ -102,16 +104,27 @@ fn append_classic_job_metadata(
 
 fn validate_classic_job(job: &J2kOwnedCodeBlockBatchJob, payload_len: usize) -> Result<(), Error> {
     let Some(coded_bitplanes) = job.total_bitplanes.checked_add(job.roi_shift) else {
+        if job.strict {
+            return rejected_code_block(DecodingError::TooManyBitplanes);
+        }
         return invalid_classic_plan();
     };
+    if job.strict {
+        if coded_bitplanes > NATIVE_CLASSIC_BITPLANE_LIMIT {
+            return rejected_code_block(DecodingError::TooManyBitplanes);
+        }
+        check_strict_code_block(job, coded_bitplanes)?;
+    }
     if !(1..=64).contains(&job.width)
         || !(1..=64).contains(&job.height)
         || !(1..=31).contains(&job.total_bitplanes)
         || coded_bitplanes > 31
-        || job.missing_bit_planes >= coded_bitplanes
     {
         return invalid_classic_plan();
     }
+    if job.missing_bit_planes >= coded_bitplanes {
+        return invalid_classic_plan();
+    }
     let decoded_bitplanes = coded_bitplanes - job.missing_bit_planes;
     let max_passes = 1 + 3 * (decoded_bitplanes - 1);
     if job.number_of_coding_passes > max_passes {
@@ -141,6 +154,35 @@ fn validate_classic_job(job: &J2kOwnedCodeBlockBatchJob, payload_len: usize) ->
     Ok(())
 }
 
+/// Reject, as the CPU decode does, a code-block whose packet headers claim
+/// more missing bit-planes than its sub-band codes, or more coding passes
+/// than its remaining bit-planes hold: errors in the codestream, which a
+/// strict decode reports with their native class on every backend.
+fn check_strict_code_block(
+    job: &J2kOwnedCodeBlockBatchJob,
+    coded_bitplanes: u8,
+) -> Result<(), Error> {
+    let Some(bitplanes) = coded_bitplanes.checked_sub(job.missing_bit_planes) else {
+        return rejected_code_block(DecodingError::InvalidBitplaneCount);
+    };
+    let max_passes = if bitplanes == 0 {
+        0
+    } else {
+        1 + 3 * (u32::from(bitplanes) - 1)
+    };
+    if u32::from(job.number_of_coding_passes) > max_passes {
+        return rejected_code_block(DecodingError::TooManyCodingPasses);
+    }
+    Ok(())
+}
+
+fn rejected_code_block(error: DecodingError) -> Result<(), Error> {
+    Err(Error::NativeDecode {
+        context: "CUDA decode rejected an invalid code-block",
+        source: NativeBackendError::decode(DecodeError::Decoding(error)),
+    })
+}
+
 fn invalid_classic_plan<T>() -> Result<T, Error> {
     Err(Error::capability_rejected(
         j2k_core::CapabilityRejection::geometry_mismatch(CLASSIC_PLAN_INVALID),
@@ -190,3 +232,6 @@ fn checked_u64(value: usize) -> Result<u64, Error> {
         ))
     })
 }
+
+#[cfg(test)]
+mod tests;