[ptweezy]

When the main loop of the x86 inflate decoders (decode_huffman_code_block_stateless_01 and _04, built from igzip/igzip_decode_block_stateless.asm) finds an invalid distance symbol, a distance before the start of the output, or an invalid lit/len symbol, it leaves wrong counts in inflate_state. In stateful inflate, it can also hand stale bytes to the caller.

The bugs

1. The main loop moves next_out past the whole copy of a length/distance pair before it decodes and checks the distance, and both distance errors return from there. total_out then counts up to 258 bytes that the copy never wrote, and avail_out is short by as much.
   • isal_inflate_stateless() reports those bytes of the output buffer as output.
   • isal_inflate() copies them to the caller with the error. For the first 64 KB of a stream it decodes into tmp_out_buffer, so apart from the first few bytes, the caller gets whatever that buffer held: earlier output of the same stream, output of an earlier stream that used the same inflate_state, or uninitialized memory (isal_inflate_init() and isal_inflate_reset() do not clear it). Later in a stream, the caller gets the old contents of its own buffer.
2. The main loop stops 8 bytes before the end of the input and 274 before the end of the output by moving end_in and end_out back, and its three error exits do not restore them. avail_in comes out 8 bytes short and avail_out 274 bytes short on top of bug 1. Both can wrap to about 4 GB, and a caller that calls isal_inflate() again after the error then reads past the end of its input.

The exits outside the main loop are correct. The C and aarch64 decoders check the distance before moving next_out and track the real buffer ends, so they are not affected. igzip_rand_test misses these bugs because it checks the inflate state only when inflate succeeds.

Changes

1. Compute avail_in and avail_out at the end of the function from the buffer ends in inflate_state, which the function does not change. The exits that already restored end_in and end_out give master's results.
2. On the main loop's two distance errors, move next_out back by the length of the copy.
3. Add a test to igzip_rand_test that inflates a block of literals followed by an invalid lit/len symbol, a length with an invalid distance symbol, or a distance before the start of the output, with the error near and far from the buffer ends, stateless and stateful. It checks the return code, total_out, the output, and that next_in + avail_in and next_out + avail_out are the ends of the buffers.

Testing

On Linux x86-64 with an AVX2 CPU, where make check runs _04:

• At each commit, make -f Makefile.unx check passes for x86-64 and arch=noarch, and tools/check_format.sh passes.
• Without the fixes, the new test fails. For example, when the first symbol of a stateless inflate is a length with an invalid distance symbol, with 16 bytes of input after the block and 300 bytes of output space, inflate returns total_out 3 and avail_out 23 instead of 0 and 300. With only the first fix, the distance cases still fail. The test passes on master with arch=noarch, and on this PR with _01 dispatched.
• A differential test inflated 100,000 valid and corrupted streams (stateless, and stateful with random buffer sizes) with master's and this PR's x86-64 builds and with the C decoder (arch=noarch). This PR matches the C decoder's return code, total_out, output and stateless avail_out in every case; master's total_out differs in 2,035. avail_in can differ by a few bytes because the decoders buffer different amounts of input, but next_in + avail_in is the end of the input in both, except for stateless inflates whose input ends inside a dynamic block header (a separate bug in the C code of isal_inflate_stateless()).

On Windows 11 (Ryzen 7 9800X3D), the mingw-w64 x86-64 build passes make check natively, including the new test, with _04 and with _01 dispatched.

On aarch64, make check passes on macOS (Apple M4; each commit with arch=aarch64, the tip with arch=noarch) and at the tip on Linux in Docker (GCC 14.2, Makefile.unx and autotools), and the x86 commits leave every aarch64 library object unchanged. Added to master alone, the new test passes with the aarch64 and C decoders, and fails with a copy of the C decoder changed to count a copy whose distance check fails, as the x86 decoder does. In 100,000 inflates on the M4 (stateless and stateful, about a third corrupted), the aarch64 and C decoders give the same return code, total_out, output, avail_out, avail_in and next_in.