[ptweezy]

An incomplete distance code in a dynamic block header can make inflate write past dist_huff_code.long_code_lookup, into the block_state, dict_length and bfinal fields of inflate_state. This PR checks that the long codes fit before writing them, and rejects the header with ISAL_INVALID_BLOCK if they do not. The size and layout of inflate_state do not change.

The bug

long_code_lookup holds the codes longer than the short table's index (10 bits for distances, 12 for lit/len), one part per group of long codes that share their first bits. igzip_lib.h sizes it at 80 entries for distances and 1264 for lit/len, counting the codes, the entries that repeat codes shorter than the longest in their group, and the entries no code uses, which only incomplete codes have. setup_dynamic_header() rejects only oversubscribed codes, so incomplete codes get through, and the bound on unused entries is too small: it allows 21 for distances, but a 15-bit code alone in the last group leaves 31 entries of that group unused.

An exhaustive search over the code length counts of distance codes found incomplete codes that need up to 86 entries (30 codes, 25 repeats and 31 unused entries); no complete code needs more than 48. make_inflate_huff_code_dist() writes the extra entries, always zeros, without a check, up to 12 bytes past the array. The write stays inside inflate_state, in C code that every architecture uses. Only bfinal matters, because block_state is set right after the tables are built and inflate does not read dict_length: on the stream this PR adds to the standard vectors, master's isal_inflate_stateless() returns ISAL_END_INPUT instead of finishing the final block.

zlib rejects incomplete codes, except a single code of length 1 (lit/len or distance) and an empty distance code. None of these has long codes, and every code this PR rejects is incomplete, so zlib also rejects every stream this PR rejects.

Changes

Before writing each group of long codes, make_inflate_huff_code_lit_len() and make_inflate_huff_code_dist() check that it fits in long_code_lookup and return ISAL_INVALID_BLOCK if not; setup_dynamic_header() and the static header fallback return that error. For lit/len, searches over code lengths found no code that needs more than 1,208 of the 1,264 entries, so that check is a safeguard. The commit adds the stream above to the standard vectors of igzip_rand_test, which expects ISAL_INVALID_BLOCK.

Testing

On Linux x86-64:

• make -f Makefile.unx check passes for the x86-64 build and arch=noarch, and tools/check_format.sh passes. Without the fix, the new standard vector fails with both builds: inflate returns ISAL_END_INPUT instead of ISAL_INVALID_BLOCK.
• Distance codes that need exactly 80 entries still decode, and codes that need 82, 84 or 86 are rejected.
• Under AddressSanitizer and UBSan, mutated headers of real blocks give master's results. Random mutations never produce a distance code that needs more than 80 entries, so these runs cover the accepted headers, and the new standard vector covers the rejected ones.

On aarch64:

• make check passes on macOS (Apple M4) with host_cpu=aarch64 arch=aarch64 and with arch=noarch, and on Linux in Docker (GCC 14.2) with Makefile.unx and autotools. Without the fix, the new standard vector fails with both macOS builds, as on x86.
• Of 250,000 streams on the M4 from a generator aimed at incomplete distance codes, all 409 that need 82, 84 or 86 entries are rejected with ISAL_INVALID_BLOCK, stateless and stateful; master returns ISAL_DECOMP_OK for 85 of them. Every other stream, including all 711 that need exactly 80, gives master's results.
• Under AddressSanitizer and UBSan, igzip_rand_test, igzip_inflate_test and about 6.5 million header mutations give no reports. AddressSanitizer does not report the overflow on master, because the write stays inside inflate_state.