Skip to content

A wider compression window in the browser (DVDs 11–20% faster, byte for byte chdman's CHDs); an exnref build in CI - #47

Draft
PowerBeef wants to merge 10 commits into
claude/engine-robustnessfrom
claude/engine-window
Draft

PowerBeef wants to merge 10 commits into
claude/engine-robustnessfrom
claude/engine-window

Conversation

@PowerBeef

@PowerBeef PowerBeef commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Milestone 8, steps 3 and 4 (docs/chd/fork-plan.md). Stacked on #46: merge that first, then this one's base becomes main.

What the profile showed

DEBUG.profile (new) makes the job worker report where its time went, the helpers' time too. "Faster to create" on the 1 GB benchmark DVD at 4 threads: of 13.2 s, the job worker spent 3.8 s reading the input, 2.5 s in the engine itself, 0.6 s handing out hunks, 0.8 s taking results, and waited the rest; the helpers were busy 21 of the 46 s they had.

  • Reading faster didn't help: reading ahead asynchronously or in 4 MB blocks cut the read time to 1.6 s but made extracting 7–26% slower and creating no faster (medians of three). Not kept.
  • The helpers waited on messages: chdman keeps 256 hunks in flight and reads 128 at a time, so with 4 helpers a batch was 16 hunks, about 1.2 ms of work for zlib,huff.

The change

  • In the browser the compressor keeps up to 1024 hunks of 4 KiB or less in flight (chd_file_compressor::m_window; chdman's 256 for larger hunks, so CDs are unchanged, and with a parent). The page sizes its batches from that window (wasm_par_enabled passes it), up to 64 hunks. Natively nothing changes.
  • chdman's window is part of its output, which this found: a partial last hunk is compressed whole, and past the end of the data it holds what chdman's work buffer held there, the hunk 256 before it (or zeros). The SHA-1s don't cover those bytes, the compressed ones do. With a wider window that tail would be other bytes, so it is copied from that hunk, still in the wider buffer. Recorded in the plan's compatibility contract (§3).
  • chdman now prints the whole CHD's final ratio: 0.289 printed the last progress step's, which in the browser could be 0% (the card shows it while finishing). Only the line changes.

Measured

1 GB benchmark DVD and 300 MB CD, 4 threads, median of three, every CHD byte-identical to native chdman:

before after
DVD, Faster to create 12.99 s 10.36 s (−20%)
DVD, Smallest (default) 50.42 s 44.67 s (−11%)
CD, Faster to create / default 5.35 / 12.34 s 5.17 / 11.66 s (unchanged window: noise)

FLAC: looked at, not changed

  • Exact SIMD for FLAC's loops: not written. Emscripten already vectorizes the two largest (autocorrelation and the LPC residual) without changing a result. What is left mostly scalar is 21% of FLAC's instructions, and FLAC is about 39% of a DVD hunk's default compression, so making it twice as fast would save about 4%.
  • Stopping FLAC after the first channel when it already can't win: not exact. chdman's FLAC searches every stereo pairing and keeps the smallest, and the pairings still to come can be far smaller, so nothing counted after the first channel bounds the frame.

Step 4: the new exception format

  • WASM_EH=exnref ./build.sh builds with exnref (-sWASM_LEGACY_EXCEPTIONS=0: 1,704 try_table, no legacy try). Such a page records no sources hash, so check-dist.sh and a release refuse it.
  • .github/workflows/rebuild.yml's second job builds it and runs the conversion tests on it: chdman's CHDs byte for byte, at the same speed (DVD default 44.40 s against 44.67, CD 11.65 against 11.66). Switching, once every browser the page supports has exnref (Chrome 137, Firefox 131, Safari 18.4), is that flag and EH_TEST in app/ui.js.

Checks

  • tests/ui/window.spec.js: images of fewer than 256 hunks, 256–1024 and over 1024, each with a partial last hunk, through the page with 1 and 4 threads, default and zlib,huff: chdman 0.289's CHDs byte for byte. On a build with the wider window but without the tail it fails, as it should.
  • The fuzz build (browser code, -np 1) makes a seed of 300½ hunks under ASan: no report, and the same CHD as chdman 0.289's.
  • REQUIRE_NATIVE=1 full suite: 464 passed, 13 skipped; engine.spec.js on the sanitized build: 43 passed; scripts/fuzz.sh check all: no findings; check-dist.sh, check-licenses.sh, engine-diff.sh --check: pass.
  • After merging Engine robustness: fuzzing and the faults it found; a no-progress watchdog; Firefox, WebKit and benchmarks in CI #46's Firefox fix: the helper start-up test 5 in 5 in Firefox; window, fallbacks, edge-cases, tuning, verify and advanced specs in Chromium with REQUIRE_NATIVE=1: 115 passed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty

…an's CHDs

Profiling the job worker (DEBUG.profile, now in the page) showed the helpers
idle half the time: chdman keeps 256 hunks in flight, so with 4 helpers a
batch was 16 hunks, 1.2 ms of work for zlib,huff, and each costs a message.
In the browser the compressor now keeps up to 1024 hunks of 4 KiB or less in
flight (m_window; chdman's 256 for larger hunks and with a parent), and the
page sizes batches from it, up to 64 hunks.

chdman's window is part of its output: a partial last hunk is compressed
whole, with what the work buffer held past the end of the data, the hunk 256
before it. So that tail is copied from that hunk, still in the wider buffer
(zeros under 256 hunks), and the CHD is chdman 0.289's byte for byte
(tests/ui/window.spec.js, each case, 1 and 4 threads). On the 1 GB DVD at 4
threads: Faster to create 12.99 -> 10.36 s, the default 50.42 -> 44.67 s.

Also: chdman prints the whole CHD's final ratio (0.289 printed the last
progress step's, as little as 0% in the browser).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
… build

WebAssembly's new exception format (exnref, -sWASM_LEGACY_EXCEPTIONS=0),
with its own objects; like EMSCRIPTEN_ANY's, its page records no sources, so
check-dist.sh and a release refuse it. rebuild.yml's second job builds it
and runs the conversion tests on it: chdman's CHDs byte for byte, at the
same speed. The default build is unchanged (the same wasm and glue from a
clean build; the page's hash moves because build.sh is a hashed source).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
…red and not done

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
@PowerBeef PowerBeef changed the title A wider compression window in the browser: DVDs 11–20% faster, byte for byte chdman's CHDs A wider compression window in the browser (DVDs 11–20% faster, byte for byte chdman's CHDs); an exnref build in CI Nov 12, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants