Skip to content

Polish repository and fix short-buffer SIMD crash - #8

Open
0kogg wants to merge 6 commits into
mainfrom
polish/repo-hygiene
Open

0kogg wants to merge 6 commits into
mainfrom
polish/repo-hygiene

Conversation

@0kogg

@0kogg 0kogg commented Nov 5, 2026 •

Copy link
Copy Markdown
Owner

Repository polish (docs, metadata, templates, wiki, tracked-artifact cleanup) plus the Ubuntu 26.04 segfault fix.

Crash fix

block_xor_raw and calculate_checksum_raw guarded AVX2 loops with i <= size - 32. For buffers below 32 bytes the unsigned subtraction wraps, so the loop stores a full 32-byte vector into a 24-byte parity chunk. The new loop bound is i + 32 <= size (and i + 16 <= size for NEON).

A gdb run on ubuntu-latest (26.04, Python 3.14.10) captured the fault at src/main.cpp:187, block_xor_raw(size=24).

Regression coverage

  • test_native_short_buffer_kernels_stay_in_bounds: raw kernels across sizes 1-64
  • test_parity_protection_short_payloads: protect/repair round-trips, including corruption of short chunks
  • Also fixes a negative slice in parity repair when a chunk starts past the payload length

Verified locally with the exact toolcache Python 3.14.9 build (122 passed x3) and a forced -mavx2 build; the pre-fix AVX2 binary segfaults on the new kernel test, the fixed one passes the full suite.

0kogg added 6 commits November 5, 2026 10:13
- Document WIM3 as the default container, update the format_version table,
  feature matrix, and compatibility matrix.
- Add a docs index, CONTRIBUTING, SECURITY, a pull request template, and
  .gitattributes.
- Add a WIM3 wiki page and refresh wiki home, sidebar, legacy, and roadmap
  links.
- Point package metadata and user-facing issue links at the current
  repository.
- Stop tracking wimf.egg-info and widen .gitignore for build and test
  artifacts.
block_xor_raw and calculate_checksum_raw guarded their vector loops with
`i <= size - 32`. size_t subtraction wraps when the buffer is smaller than
one vector, so the AVX2 body ran with i == 0 and stored 32 bytes into a
24-byte parity chunk. Ubuntu 26.04 runners compile __AVX2__ in and crashed
in test_file_detection_for_all_container_magics with SIGSEGV.

Compare `i + 32 <= size` instead and apply the same guard to the 16-byte
NEON loop. Add regression coverage for the raw kernels and for parity
protect/repair across sizes 1-64, which also exposed a negative slice when
a repaired chunk starts past the original payload length.

Verified: full 122-test suite passes in scalar and forced -mavx2 builds;
the pre-fix binary segfaults on the new kernel test.
zstd.c is compiled as part of the C++ extensions, so the __asan_* hook
prototypes need extern "C" to match the sanitizer runtime and link in
ASAN builds.
The CPython 3.14 shared object is a build artifact and is already matched
by .gitignore. Keeping it in history makes every source checkout platform
specific and any local rebuild dirty.
@0kogg 0kogg changed the title Polish repository documentation and metadata Polish repository and fix short-buffer SIMD crash Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant