Proposal: line-based region decode to cut peak memory

After #126, #129 and #131, a CPU region decode's peak memory is set by three region-sized f32 buffers that are all live together:

the sub-band coefficients;
the per-component decoded samples;
the IDWT windows.

We could cut most of that, roughly three-quarters of the peak by the estimates below, by never holding those buffers whole: reconstruct with a line-based IDWT, decode code-blocks one row band at a time as it needs them, and apply the color transform and output conversion row by row. Before writing anything large, I'd like to check whether you'd want this direction, and how you'd like the API and budget changes shaped.

Measurements

These were measured with temporary instrumentation in decode_tile (not proposed for merge). The setup was linux/arm64 in Docker, serial CPU decode, Rgb8 output, with #131 applied. The source is an untiled 18000×12000 RGB JP2 using 9/7, RPCL, 5 levels, 64×64 code-blocks, 7 layers and PLT.

1024×1024 region at full resolution	Size
Sub-band coefficients (`storage.coefficients`)	24.3 MB
Decoded component buffers (`channel_data`, 3 × f32)	12.0 MB
Windowed IDWT peak (`roi_idwt_peak_elements`)	6.2 MB
Tag-tree nodes, precincts, layers, segments	7.6 MB
Peak RSS of the whole process	58 MB

A 6000×4000 single-tile image has the same pattern: 24 MB of coefficients, 12 MB of component buffers and 6 MB of IDWT windows, out of a 50 MB peak.

Coefficient storage is about twice the region's 12.6 MB because it's code-block granular. At each level the stored rectangle rounds out to whole 64×64 code-blocks, plus the filter margin.

For comparison, our IIIF server's tile benchmark decodes every 1024 px tile of this image at every scale factor:

Run	j2k peak RSS	Kakadu 8.6.2 peak RSS (`kdu_region_decompressor`)
One thread	64 MB	23 MB
8 concurrent decodes	442 MB	102 MB

Only the 6.2 MB of IDWT windows would go away with a stripe IDWT alone. Most of the rest needs the row-streamed output and the incremental Tier-1 below.

Proposal

Two stages, each a separate PR.

1. Row-streamed reconstruction

Run all components through a line-based IDWT in lockstep: each level keeps a small ring of rows sized to the 5/3 or 9/7 lifting support, and pulls rows from the level below on demand. As each output row completes:

apply the inverse MCT and level shift, then clamp;
convert to the requested pixel format and write straight into the caller's buffer.

That removes the IDWT windows and the f32 channel_data buffers: about 18 MB of the 58 MB above.

The cost is that native region decode would no longer produce channel_data for the facade to convert afterwards. It would need an internal row sink in the requested output format. Whole-image decode and the exact-integer path could keep the current path.

2. Incremental Tier-1

Packet parsing stays as it is. Code-block decoding moves behind the stripe IDWT: when a level needs sub-band rows that aren't decoded yet, it decodes the next row band of that sub-band's required code-blocks into a ring buffer. A band is freed once every level above has consumed it. That replaces most of the 24 MB of coefficient storage with about two code-block rows per sub-band.

Parallel Tier-1 would decode a whole band of code-blocks in parallel, rather than a whole tile.

These figures are estimates from the sizes above, not measurements of an implementation. Together, the two stages should bring this request from about 58 MB to about 15 MB.

GPU code that uses the CPU region path

I checked which GPU code reaches the CPU region pipeline:

Metal region decode to a Metal surface (engine::decode_image_region_to_surface, the fallback behind decode_region_scaled_to_surface). It runs the CPU region pipeline through decode_region_components_for_integer_output_with_ht_decoder, with MetalCodeBlockDecoder as the HtCodeBlockDecoder hook. Inside it:
- decode_j2k_sub_band decodes whole classic sub-bands on the device into one &mut [f32];
- decode_store_component and decode_inverse_mct capture planes on the device, and select_plane_stage builds the surface from those planes and the returned component planes.
Both stages would change what this path relies on. Stage 1 removes the per-component planes and moves the color transform to rows. Stage 2 decodes sub-bands a row band at a time rather than whole. The single-decomposition IDWT hook isn't involved, since region decode's windowed IDWT (apply_roi) already bypasses it.
CPU-staged and Auto-on-CPU APIs in j2k-cuda and j2k-metal, such as decode_region_into, decode_region_scaled_into and the *_to_cpu_staged_* / *_to_cpu_surface routes. These go through the public J2kDecoder facade, so they'd be unaffected as long as the facade's API and output stay the same.
CUDA-resident and Metal direct region decode (build_direct_*_plan). These share only the planning front end: build::build in MetadataOnly mode, RoiPlan::build and segment::parse. They never run CPU Tier-1, the IDWT, store or the color transform, so they're unaffected.
The batch decoders use the CPU decoders only as test oracles, and the output would stay bit-identical.

Questions

Is a line-based region decode something you'd take, given how much of the CPU decode path it touches?
For stage 1, would you accept native region decode writing rows into the requested output format through an internal sink? Or would you rather keep channel_data as the boundary and only shrink it, for example by holding component rows rather than planes?
For the budget, I'd charge each level's row rings and each sub-band's band buffers up front from the ROI plan, before Tier-1. They'd replace the coefficient and IDWT-window charges. Does that fit how you'd like the peak modelled?
Should region decode keep a whole-plane fallback, for example for subsampled components or tiles without an ROI plan? Or should every region decode take the line-based path?
For Metal's hooked region decode, which would you prefer?
- Keep the whole-plane pipeline whenever an HtCodeBlockDecoder hook is supplied.
- Add row-band variants of the sub-band and store hooks.
- Move that fallback onto the direct-plan route that CUDA region decode already uses.
Any parts of the current design you'd want preserved that this would break: debug counters, J2K_PROFILE_STAGES stages, or DecoderContext reuse?

I'm happy to start with stage 1 alone and benchmark it before going further.

Thanks for the careful breakdown, especially tracing the GPU callers! Yes, I’d welcome this direction. Starting with stage 1 in a separate PR and measuring it before taking on incremental Tier-1 sounds right.

One thing worth measuring before or alongside stage 1: doing Tier-1 → IDWT → store one component at a time with a single reused coefficient buffer looks like it saves about 16 MiB on your 1024² case by itself. With two other small changes, the estimate is around 36 MiB for the reported 58 MB case, compared with roughly 40 MiB for stage 1 alone. These are estimates, but the component-at-a-time approach conflicts with keeping all components in lockstep, so it’d be useful to compare them before committing to that design.
An internal row sink for the requested output format seems reasonable. I’d land stage 1 behind an internal opt-in initially, with bit-exact parity tests against the whole-plane path. Please retain the existing path for the Metal hooks, palette, component-definition, sYCC and CIELab post-processing, and the public component-plane APIs, including decode_region_components* and PreparedRegionDecoder.
Charging the row rings and band buffers from the ROI plan fits the budget model, including their peak overlap with retained parsing state and temporary conversion buffers. Please also preserve DecoderContext reuse and keep the counters and profiling useful. I’d keep the final memory target provisional: the retained storage looks closer to 22–23 MiB than 15 MB, pending an implementation and measurements.

Coverage for both transforms, reduced-resolution regions, edge regions and the fallback cases would be helpful. I’m happy to run device validation once the PR is ready. Thanks for offering to take it on!

Thanks, this is really helpful. I took your suggestion and measured component-at-a-time first.

I have two branches ready:

Workspace peak counters (perf(j2k-native): report the peak decode workspace by owner #171). The decoder now records its peak workspace by owner: coefficients, channel_data, IDWT buffers, Tier-1 workspace and structure. It samples at each stage boundary and at every level of the windowed IDWT. The figures are added to the J2K_PROFILE_STAGES rows as peak_*_bytes, so later PRs can be compared with the same counter.
Component-at-a-time ( perf(j2k-native): decode one component at a time into a shared coefficient buffer #172) Tier-1 → IDWT → store, with one coefficient buffer sized to the largest component and cleared between components. The budget charges that one buffer instead of the sum. Decodes with an HtCodeBlockDecoder hook, recode and wavelet export keep the whole-tile layout and order. A parity test checks bit-identical output against the whole-tile layout across gray, RGB with and without MCT, 4:2:0, 5/3 and 9/7, classic and HT, tiled and untiled, full and 1/2 resolution, and whole, interior, edge-flush and 1-pixel regions.

Peak workspace from the new counter, serial decode:

Request	Before	Component-at-a-time
1024² region of 18000×12000 RGB, with PLT	50.0 MiB	33.8 MiB
same, at 1/2 resolution	48.6 MiB	33.2 MiB
1024² region of 18000×12000 RGB, no PLT	88.7 MiB	72.4 MiB
1024² region of 6000×4000 RGB	43.5 MiB	27.7 MiB
256² region of 6000×4000 RGB	7.4 MiB	4.1 MiB

The 1024² case saves 16.2 MiB, which matches your estimate. Its 33.8 MiB breaks down as 8.1 MiB coefficients, 12.0 MiB channel_data, 6.2 MiB IDWT and 7.4 MiB structure. Peak RSS falls by about 17 MB, and decode time is unchanged. The largest untiled RGB image that fits the default 512 MiB cap grows from about 4250² to about 5000².

(The counter leaves out the parsed codestream graph and the caller's output buffer, which probably accounts for most of the gap to the 58 MB I first reported.)

On the conflict with lockstep, I'd like to change the order of the stages:

Component-at-a-time (above).
Line-based IDWT, still one component at a time, behind an internal opt-in. This replaces the IDWT windows with row rings and leaves channel_data as it is.
Incremental Tier-1, so coefficients live in bands of code-block rows instead of whole sub-bands.
Lockstep across components, plus the row sink, so channel_data is never allocated.

Lockstep is only expensive while whole-plane coefficients exist. Once incremental Tier-1 has removed them, running components together costs a set of rings and bands per component instead of a plane per component, and it's what lets the row sink apply the inverse MCT per row. With this order, each step should lower the peak on its own and none of them undoes the previous one. The whole-plane path stays for all the fallbacks you listed, and each stage gets parity tests against it.

A few questions before I go further:

Which two other small changes did you have in mind alongside component-at-a-time? I'd like to include them, or at least measure them, before starting the line IDWT.
Component-at-a-time is bit-exact and doesn't change the public API, so I've made it the default for CPU decodes rather than putting it behind the opt-in. Is that OK, or would you rather it start behind the opt-in too?
Where would you like the internal opt-in to live: a field on DecodeSettings, something on DecoderContext, or somewhere else?
On the image without PLT, structure is 46 MiB of the 72 MiB peak. That's outside this plan, but it may be worth a separate issue if you think it's worth pursuing.

thanks, these measurements are really helpful, and the new counter makes the later stages easy to compare! The new order makes sense to me. Each step should lower the peak on its own, and lockstep only comes in once incremental Tier-1 has removed the whole-plane coefficients. I’ve approved CI on #171 and #172, and the exact heads are running through CUDA and Metal now.

The two other changes I had in mind were trimming coefficient storage to the exact IDWT windows and writing the last IDWT level straight into channel_data. Region storage currently rounds out to whole code-blocks and adds a 40-sample 9/7 margin at each level, so storing only the window the IDWT reads should save around 2.4 MiB on top of component-at-a-time. That needs a copy from each code-block’s decode output on the CPU paths and per-block scratch for HT, and storage should stay code-block granular whenever a hook is set. For the last level, writing into a margin-padded channel_data plane and compacting it in place, rather than finishing in the IDWT buffer and copying in store, looks like about 3 MiB. Together I’d expect somewhere near 28 MiB for your 1024² case, but these are estimates against the original 58 MB figure, so it’d be good to measure them with your counter. Tightening the 40/16-sample margins to the actual filter support could save another 1–4 MiB, but it needs a proof of the support at each level and GPU parity, so I’d leave that for last.

Making component-at-a-time the default is fine with me, since it’s bit-exact, internal, and the parity test is broad. The opt-in was meant for the line IDWT because it replaces a path. Before #172 merges, it would be good to see a whole-image decode timing for a large multi-component HTJ2K file with parallel on. The tile-parallel HT route now plans and forks per component instead of per tile, and the posted timing only covers regions. A 29-bit case in the layout parity test would also be helpful, since that’s the only path that uses coefficients_i64. The 8-bit encoder can’t tile subsampled components but the high-bit path can, so that case could cover tiled 4:2:0 too.

For the opt-in, I’d put it on DecoderContext rather than DecodeSettings. DecodeSettings is a public Copy struct with all-public fields and no #[non_exhaustive], so a new field would break struct-literal callers and land in the stable API. DecoderContext already carries a per-context route knob with set_cpu_decode_parallelism, so a pub(crate) field is enough for the parity tests, with a #[doc(hidden)] setter next to it if benches or the facade need to flip it. I’d avoid an environment variable, since it’s global and one test process couldn’t compare both paths side by side.

The structure cost without PLT seems worth a separate issue. 46 of 72 MiB is significant, but it’s a different problem: without PLT every packet header has to be parsed to find the region’s data, and unselected precincts may still need their tag-tree state across layers. It shouldn’t block this plan.

Thanks again for taking this on!

test(j2k-native): check region decodes across a generated progression…

With #188 merged, region decodes stream the IDWT by default. Before starting the next step, incremental Tier-1, I measured what it would buy and what it would cost. I'd like your view before writing code.

Where the peak is now

Peak workspace for integer-output component planes, after #188:

Region	Total	Coefficients	Output planes	Structure
large-plt 1024²	25.1 MiB	5.5	12.0	7.4
archival 1024²	19.2 MiB	5.5	12.0	1.6
archival 256²	2.6 MiB	0.8	0.8	1.0

Incremental Tier-1 can only reduce the coefficients. A simple model of the IDWT windows (9/7, 5 levels, 40-sample margin, 64×64 code-blocks) reproduces the measured 5.5 MiB. Holding two code-block rows per sub-band instead of the whole window gives:

Region	Coefficients now	Banded	Total peak
1024²	5.5 MiB	~1.95 MiB	25.1 → ~21.6 MiB (−14%); archival 19.2 → ~15.7 (−18%)
512²	1.9 MiB	~1.2 MiB	−0.7 MiB
256²	0.8 MiB	~0.75 MiB	almost nothing

The project I'm using this in serves 1024-pixel tiles by default, so the 1024² row is the one that matters for me.

Parallel cost

Classic parallel Tier-1 currently joins after each sub-band, which is about 100 code-blocks per join at the finest level of a 1024² region. To see what smaller batches would cost, I temporarily split each sub-band's parallel batch into chunks with a join between them. Results on an M3 Pro with 12 cores, two runs of 31 interleaved samples each on a quiet machine:

Case	Today	30-block batches	10-block batches
large-plt 1024²	44.8 / 43.8 ms	+4%	+14%
archival 1024²	36.5 / 35.6 ms	+3–6%	+18%
large-plt 1024² at 1/2	52.0 / 58.4 ms	+7–8%	+23–28%

So a band has to cover one code-block row of all three detail sub-bands of a level, about 30 blocks, not one sub-band's row. HL, LH and HH share code-block grid rows in sub-band coordinates, so that batch falls out naturally. Running the IDWT for band g while band g+1 decodes, with rayon::join, would hide the serial IDWT (about 7 ms of the 44) and should more than cover the 4–8%.

Proposed design

Each sub-band of a streamed tile keeps a ring of code-block rows, slot g % ring_bands. Two rows are enough for 64-row code-blocks. Smaller code-blocks get 1 + ceil(support / height) rows, so 4-row blocks stay correct.
lines.rs reads each sub-band row as a one-row CoefficientSource over its ring slot, the same way it reads LL from the level below. CoefficientSource and interleave_samples_roi stay as they are.
Before feed_row reads a band row, it asks a per-component feeder to make sure that row is decoded. If it isn't, the feeder zeroes the next slot and decodes the next grid row of that level in parallel into it, using the existing pending-block lists and BlockPlacement. Refilling a slot gets the same eviction check the IDWT rings use.
The ROI plan charges the rings in place of the coefficient windows, plus Tier-1 scratch for one batch.
Packet parsing and the code-block structure stay as they are.
Tests: the streamed-vs-whole-window parity harness and the progression matrix would also run with banding off. They'd cover small and non-square code-blocks (4×4, 32×8, 8×64), precincts, layers, odd origins and reductions, plus a pinned coefficient-peak drop.

Questions

Is about 3.5 MiB (14%) on 1024² regions worth a Tier-1 scheduler, given the 12 MiB of output planes and the structure cost that remain?
Can I start with classic code-blocks only? HT goes through the tile-parallel route, which plans a whole component in strips. Banding it means re-planning per band and needs its own measurement, so until then HT tiles would keep whole-window coefficients.
Should the IDWT/Tier-1 overlap be part of this PR, or a follow-up? With the 4–8% above, I'd lean towards including it.

One small unrelated finding: at 1/32 scale, with no wavelet levels left, large-plt's region decode still reports idwt=0.8 MiB. It looks like the LL band is copied into an IDWT buffer. That's 0.8 of 10.8 MiB, and I can look at it separately if you think it's worth it.

Thanks for measuring this before writing code, it makes the decision a lot easier!

Before answering, I should mention some decode work I've been doing that overlaps here. To check it helps your case rather than hurting it, I rebuilt your setup (18000×12000 RGB 9/7, RPCL, 7 layers, 128/256 precincts, with and without PLT) and compared it against main. Output is identical everywhere.

Two fixes to the tile route on main are up as #192 and #193. The plan used to reserve job and strip vectors for every code-block in the tile, which without PLT is up to about 20 MiB of heap and budget for a 1024² region. It now reserves only the blocks the region needs. The route also never ran under J2K_PROFILE_STAGES, so your peak counters were always measuring the per-sub-band path instead of what production runs. It does now. One side effect is that peak_idwt_bytes reads 0 in the largest snapshot, so the numbers will shift a little when you compare.

The bigger piece I haven't put up yet. It moves classic code-blocks onto the same tile-wide route as HT whenever each block is a single arithmetic segment, region decodes included. On your images that makes classic region decodes 23 to 29% faster at 10 threads and lowers the heap a bit. Since it changes what stage 3 would build on, I'd like your thoughts before I open it.

On your questions. I think the 3.5 MiB is worth it, mostly because it's the step that makes stage 4 pay off. If the row sink were built on today's Tier-1, it would have to hold all three components' sub-bands at once, about 16.5 MiB, just to save 12 MiB of planes. On banded Tier-1 the two together save about 14 to 15 MiB, so stage 3 then 4 is the right order. For us though I'd put the #190 structure items first.

On starting with classic only, I'd hold off until we've talked about the tile route change. With it, classic-only banding on the per-sub-band path would only cover bypass and termination streams, so I think banding is better designed once against the tile route for HT and classic together. Your batch-size numbers would also move on that route.

I'd keep the IDWT and Tier-1 overlap as a follow-up, measured on whichever route stage 3 ends up on. And the LL copy at 1/32 sounds worth a small separate PR if it's as simple as it looks.

Cool! That all makes sense, and I'm glad the setup was useful for checking your decode work.

The tile-route change sounds good to me, so please go ahead and open it. A 23–29% speed-up on classic region decodes is well worth having, and I'd much rather design banding once for HT and classic together on the route production runs than build a classic-only version first. I'll wait for it, and for #192 and #193, before starting stage 3, and I'll re-measure the batch sizes and the coefficient model on that route, with the profiler now following it. The IDWT/Tier-1 overlap can be a follow-up measured there.

In the meantime I'll work on #190 in the order you suggested: item 2, then item 3, then item 1. I'll start item 2 now, since it's in packet parsing and doesn't touch #191's tag trees, and keep rejection exactly as it is today. I'll leave items 3 and 1 until #191 has merged.

The LL copy is up as #195.

Now that the tile route and #197 are in, here's how we're thinking about full image memory so we build one pipeline with you instead of two.

Full image decodes still hold whole component planes and sub-band coefficients, so they sit far above a region decode. We want the same banded Tier-1 and row sink you're planning for stages 3 and 4, just driven over the whole image and spread across threads.

The split we'd suggest is that you keep owning stages 3 and 4 as you planned, including build.rs, the allocation planning, segment.rs and banded Tier-1. We'd take the full image side on top of it, meaning the stripe scheduling, running the line IDWT and pack in parallel across columns, and the concurrency checks so nothing gets slower. That only works if a few things are shaped for both callers from the start. A band request in tile component and sub-band coordinates rather than region ones, a full image mode that skips the whole coefficient allocation, finished row views that more than one reader can hold for the filter halo, and a sink that takes disjoint rectangles in any order. decode_into stays the API, so nothing public changes for this.

If stage 3 starts as a single reader feeder that's totally fine, we'd use it for the serial step first and then extend that same provider rather than building a second one. Does that split work for you, and do those interface points fit how you're planning stage 3?

That split works for me, and I'd much rather build one provider than two. I'll keep stages 3 and 4, and shape the interfaces for both callers from the start.

I've had a look at how #201 left the tile route, and banding seems to fit it well. Each strip is already a row of code-blocks writing a contiguous range of its sub-band, so a band can be the strips of code-block row g across a level's sub-bands, written into a ring slot instead of tile coefficient storage. The plan stays one per tile, and strips run in band order.

On your four points:

Band requests in tile-component and sub-band coordinates. That's how I'd planned it anyway: a band is a code-block grid row of a level, in sub-band coordinates. A region only narrows the stored window, so a full-image request is the same request with the window covering the sub-band.
A full-image mode that skips the whole coefficient allocation. That follows from the above. The plan charges the rings, sized to the stored window's width, instead of the coefficients, whatever the window. One thing to keep an eye on: the route's job and strip vectors still cover every block the decode needs, which for a full image is all of them. That's fine to start with, but it could later be planned band by band.
Finished row views several readers can hold. I'll make the rows read-only views into the ring, and release a band only once every reader has moved past it, a low-water mark across readers rather than one reader's position. Stage 3 starts with one reader, but the release rule won't need to change when you add more. The open question is how far apart you expect column stripes to get. If they advance together band by band, two or three bands of ring are enough. If they can drift, the ring has to cover the gap, or the slow reader holds the feeder back. Which do you have in mind?
A sink that takes disjoint rectangles in any order. Fine. The region path will just send full-width rows in order. Since the inverse MCT needs all components of the same rectangle, I'd make a sink call carry every component's rows for that rectangle.

A few things I'd like to agree on before starting:

Errors. The route retries serially on failure so the error matches a serial decode. With banding, the IDWT and the sink may already have consumed earlier bands, and band order hits a different first error than today's sub-band order when two blocks are bad. I'd suggest that on any banded failure we rerun the tile the old way, whole coefficients and serial, purely to report main's error. That only costs on the error path, but in full-image mode it needs the whole coefficient allocation to fit. If it doesn't, we return the banded error. Does that work for you, or would you rather define the banded order as the new error order?
One thread. One-thread decodes don't take the tile route, so I'd band the serial path too, with the same provider and a serial executor. The per-sub-band fallback (bypass and termination streams, too few rows) would keep whole coefficients at first.
Stage 4 and your RGB8 work. The fused RGB8 pack, which reconstructs the last level inside the pack (perf(j2k-native): let the fused RGB8 decode fill the last component plane #222, perf(decode): reconstruct LL bands in dead coefficient prefixes #248), overlaps with what the row sink would do. Should the sink take that over, or should the last level stay in your pack, with the sink feeding it?

The order I have in mind: first the band provider with a single reader, for region decodes, with the serial and tile-route executors and a parity harness against whole-window coefficients. Then lockstep and the sink. Before starting I'll re-measure the coefficient model and batch sizes on the tile route.

That all fits really well with what I had in mind, thanks for the detailed read.

On drift, column stripes advance together band by band. My stripes are output rectangles of 64 rows by 512 or 1024 columns at the finest level, and the scheduler always prefers jobs that unblock the oldest output rows, so a slow stripe holds the feeder back rather than the ring growing. Three bands of ring at the finest level and two at the others should be enough, and a full ring just means the feeder waits.

Rerunning the tile the old way to report main's error sounds right to me, and returning the banded error when the whole allocation doesn't fit is fine. It matches how the tile route already retries serially today.
Banding the serial path with the same provider sounds good. The one thing I'd ask is that the serial executor stays plain loops with no locks or atomics, since one thread decodes can't get any slower. Keeping whole coefficients for the per sub-band fallback at first is fine.
I'd keep the last level in the fused RGB8 pack, since its rounding and FMA order are pinned for identical output, and have the sink feed it each rectangle's component rows and own ordering and the caller's buffer. I can shape the pack's entry point so the sink can call it directly.

Your order sounds great, starting with the single reader band provider for regions and the parity harness.

Heads up, 0.13.0 is out with your sparse precinct work, the profile feature and the wasm32 check in it, thanks again for all of that. Sorry for the flood of PRs, I found some big performance gaps while digging into a few issues and wanted them all in one release. A lot landed on main, including faster classic encode, the CUDA region work and a new strict JPEG mode, so it is worth rebasing stage 3 on the new main before you go much further. The API breaks closest to your work are the new signed field on J2kDirectGrayscalePlan and BatchLayout moving to j2k-core, with the old j2k path still re-exported, and the changelog has the full list with migrations.

Some numbers on your workload from my 12 core M4 Pro laptop, using 1024 regions of the 18000 x 12000 test images. On 0.12.0 every one of those regions failed with image is too large under the default limits, because it sized a whole image buffer even for a region. On 0.13.0 they all decode, and a 1024 region in the middle of the PLT image takes about 30 ms at 12 threads and 131 ms on one thread, about 50 ms without PLT and about 24 ms for HTJ2K, with the heap peaking at 21 to 24 MiB per decode. Most of that comes from your region work, which is sick.

Thanks for picking up the count gate and the comment.

I've started stage 4 on top of #287, for serial region decodes, and I'd like to check the interface with you before I go further.

What it does. A serial region decode on the streamed IDWT now decodes all of a tile's components together, a row at a time, and hands each finished row to a sink instead of filling the channel_data planes. One banded provider covers every component's bands, and each component gets its own stepper (reconstruct_rows is now a loop over LineReconstruction::next_row) and ring. Each row is copied out of the ring, then gets exactly the per-sample steps the planes get today: SampleShift, or transform_planes with the same addends and centred rounding, then the shift of components after the third. So the output is bit-identical. The planes path already runs the MCT in 16 Ki chunks at arbitrary offsets, and irreversible_scalar_tail_matches_simd_body_bits pins the tail, so per-row chunks don't change anything.

One thing I hadn't expected: the region path never goes through your fused RGB8 pack. fused_last_level only applies when !line_region_idwt, so region decodes fill f32 planes and the j2k crate converts them per sample afterwards. So for regions, the sink replaces the planes rather than feeding the pack, and the last level stays in the streamed IDWT.

Interface (native, #[doc(hidden)] like the other integer-output methods):

pub trait ComponentRowSink {
    /// One rectangle of the region at (x, y), one plane per component with
    /// the rectangle's dimensions: rounded for integer output, after the
    /// inverse MCT and level shift. Rectangles don't overlap and cover the
    /// region once.
    fn write_rows(&mut self, x: u32, y: u32, planes: &[ComponentPlane<'_>]);
}

impl Image<'_> {
    pub fn decode_region_rows_for_integer_output_with_context(
        &self,
        roi: (u32, u32, u32, u32),
        decoder_context: &mut DecoderContext<'_>,
        sink: &mut dyn ComponentRowSink,
    ) -> Result<()>;
}

It reuses ComponentPlane, so the j2k crate's 8-bit writers can stay as they are and just write at an offset. Whether a decode streams is decided per image before reset: the streamed IDWT, serial Tier-1, no image or tile offsets, 1 to 4 components, and an MCT that's the same in every tile. Otherwise the decode fills its planes as now and hands them to the sink as one rectangle, so callers have a single path. Inside native, a tile's output is now one parameter, TileOutput { Planes, Rows (tile workers), Sink }, in place of tile_rows. Errors work as in stage 3: on a failure, each component's Tier-1 reruns over whole windows, in planes order, and the decode reports that error.

Results on the 18000 × 12000 images and the 6000 × 4000 archival image, serial, peak heap including the caller's 3 MiB RGB8 buffer:

Region	Planes, then RGB8	Row sink
1024² (12 regions over 3 images, 2 at ½)	16.5–19.0 MiB	6.4–8.8 MiB
512²	4.9–6.7 MiB	3.2–5.0 MiB
256²	1.8–3.6 MiB	1.9–3.6 MiB

Pixels: identical on every region.
Speed: serial time within 1% (100.4 → 99.9, 155.8 → 154.9, 135.4 → 134.4, 90.4 → 90.4 and 50.7 → 50.5 ms).
Tests: a parity test compares the sink with the planes bit for bit on 11 encodings (gray, RGB, RGBA, 5/3, 9/7, classic, HT, tiled, without MCT, several block shapes) at 3 reductions and 8 regions. It also checks that every sample arrives once.
Errors: a corrupt-input test gives the same outcomes both ways, and it fails without the rerun.
Fuzz: a differential fuzz of the two paths in one build matched on all 300,000 mutated decodes, about 180,000 of which failed.

On a 1024² region that leaves the decoder itself at about 3.5 to 6 MiB, mostly coefficients and packet state.

Questions:

Small regions. Decoding the components together holds every component's bands at once, so a small region with 64×64 code-blocks can hold more than the planes it saves (about 194 KB of bands against 65 KB plus 6 KB of planes on a 29×17 RGB region). 256² breaks even. I stream whenever the conditions allow, since that's simpler and the absolute cost is small. Would you rather it stream only when the planes would be larger than the extra bands?
Partial output. A decode that fails partway may already have written rows to the caller's buffer, where today nothing is written. I think that's fine for decode_into-style calls if it's documented. Do you agree?
Full-image mode. This sink gets finished samples, after the MCT, and that suits regions. If full-image stripes feed your pack instead, the pack would want component rows before the MCT and the last level. I'd keep this trait for finished samples and give the pack its own entry point when you shape it. Does that match what you had in mind?
Next step. Next I'd route the j2k crate's 8-bit region output (decode_image_region_into_with_native_context) through the sink, which is what Persimmon calls, and then open the PR. Is that the right place to switch it?

The question at the end of #287, about how the parallel lookahead executor should fit your stripe scheduler, is still open too, whenever you get to it.

Yep, I agree with all four. I'd stream the small regions too for now and document partial output on failure. Keeping this sink for finished samples and giving the pack its own entry point works for me, and go ahead with switching the existing 8-bit region calls over.

That bypass wasn't intentional, I missed the region path when doing the packing optimizations. Thanks for finding it! The small fix is up as #293 now, which I'm planning to merge separately once the checks are green. On its own against main, the native packed-region call is about 4–6% faster at 12 workers, with identical pixels. That fuses MCT and RGB8 packing; the last level still stays in the streamed IDWT, and the facade's component-plane route is unchanged.

After looking through the workflow you described earlier, I've also been working on a draft PR for automatic reuse of packet boundaries on repeated region requests without PLT. That's up as #292 now. I'm leaving it as a draft and not merging it yet. You can build on frames-sg/j2k:perf/repeated-region-packet-directory so we can try it with stage 4. It works through the existing methods, so callers don't need to change anything. Starting with single-tile RPCL files with one tile-part and inline headers.

On the 18000×12000 image, warm 1024² component-plane calls went from about 43 → 25 ms in the middle and 47 → 22 ms at the edge, with 12 workers. Eight concurrent requests went from about 124 → 104 ms and 110 → 85 ms. The directory takes about 0.86 MiB for that image; the PLT control stayed around 26 ms either way. The planes match bit for bit.

The overlap with your branch is Image preparation, passing the directory into packet parsing, and memory accounting. We should keep those shared between the plane and sink routes. One thing to watch with partial output: directory eviction and releasing retained scratch can trigger retries, so once rows have reached the sink, those retries shouldn't send the same rectangles again. The whole-window Tier-1 rerun for the original error should stay output-free too. We haven't tried the two changes together yet.

And yes to the shared queue idea from #287. Oldest output rows first, with the column stripes advancing together in 64-row rectangles, 512 or 1024 columns wide. Release a band once every reader has passed it, and let a full ring hold the feeder back. Three bands at the finest level and two at the others are the starting depths I had in mind, then we can check those against the actual support and scheduling.

Good to hear the bypass was simply missed, and the 4–6% on #293 is a nice free win. I've rebased stage 4 onto current main (after #291), and it merges cleanly with #293, since the j2k crate's 8-bit region calls now go through the sink and the native packed-region call stays yours.

On retries, you were right that there was a gap. run_decode_with_capacity_retry reruns the whole decode without the retained scratch after a capacity error, so on a multi-tile image whose second tile hit the cap, the first tile's rows would have reached the sink twice. Now:

Capacity retry. The decode context counts the samples sent to the sink, including rows written by a tile that then fails, and resets the count before each decode. The retry is skipped once that count is non-zero, so the decode returns the capacity error instead. A decode that fails before any row is written still retries as before. A unit test pins this.
Whole-window rerun. It already writes no output: it reruns Tier-1 bit-plane decoding per component only to find the error, and never reaches the IDWT or the sink.
perf(j2k-native): reuse packet boundaries across region decodes #292. It conflicts with stage 4 in four small hunks in image.rs and j2c/decode.rs, all signatures where both branches add a parameter. with_packet_cache_retry and the eviction retry around the region decode will need the same guard on the row path: no rerun once rows have been streamed. The count above is what they'd check. I'll try the two together on perf/repeated-region-packet-directory once stage 4 is up, and report memory and timings for the combination.

I've also added a line about partial output to the # Errors docs of decode_region_into, decode_region_scaled_into and decode_region_scaled_pow2_into, along with the CHANGELOG entry.

The queue plan for 3b sounds right to me: oldest output rows first, 64-row rectangles across 512- or 1024-column stripes, release once every reader has passed, back-pressure from a full ring, and depths of three bands at the finest level and two at the others. I'll start from that after stage 4.

I'll open the stage 4 PR against main shortly.

Thanks again for #310 and #312! I’m getting ready to put up the CPU improvements, including the BYPASS integration from #311 with your credit and parity tests preserved.

I’ll start with the output last-row fix and the decode-kernel improvements. There are also packet-parsing, allocation and tile-route optimizations; I’ll identify those shared-file changes explicitly in the PRs. That first set leaves the band provider and streamed readers unchanged.

The later changes touch the shared streaming path, including cross-component band execution, row finishing and full-image routing. Before I put those up, are you working on anything in that area, or on further packet-state/allocation changes? I’d like to coordinate the order so we don’t duplicate effort or give you an unnecessary rebase. If you have something underway, send me the branch or a quick outline. I can share the relevant changes on my side too.

One API heads-up: the series adds a ResourceLimit error variant so callers can distinguish memory-limit failures from other decode errors. Existing decode signatures stay unchanged; I’ll list the API additions and error-behavior changes explicitly in the PRs.

I'm not currently working on anything. Distinguishing ResourceErrors from other decode errors sounds like a nice improvement.

Navigation Menu

Repository navigation

Proposal: line-based region decode to cut peak memory #133

Measurements

Proposal

1. Row-streamed reconstruction

2. Incremental Tier-1

GPU code that uses the CPU region path

Questions

Where the peak is now

Parallel cost

Proposed design

Questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Proposal: line-based region decode to cut peak memory #133

Description

Measurements

Proposal

1. Row-streamed reconstruction

2. Incremental Tier-1

GPU code that uses the CPU region path

Questions

Activity

jcwal1516 commented on Oct 19, 2026

jcoyne commented on Oct 19, 2026

jcwal1516 commented on Oct 19, 2026

jcoyne commented on Oct 23, 2026

Where the peak is now

Parallel cost

Proposed design

Questions

jcwal1516 commented on Oct 24, 2026

jcoyne commented on Oct 24, 2026

jcwal1516 commented on Oct 25, 2026

jcoyne commented on Oct 25, 2026

jcwal1516 commented on Oct 25, 2026

jcwal1516 commented on Oct 26, 2026

jcwal1516 commented on Oct 26, 2026

jcoyne commented on Oct 27, 2026

jcwal1516 commented on Oct 27, 2026

jcoyne commented on Oct 27, 2026

jcwal1516 commented on Nov 1, 2026

jcoyne commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions