Repository navigation
Conversation
…an's CHDs Profiling the job worker (DEBUG.profile, now in the page) showed the helpers idle half the time: chdman keeps 256 hunks in flight, so with 4 helpers a batch was 16 hunks, 1.2 ms of work for zlib,huff, and each costs a message. In the browser the compressor now keeps up to 1024 hunks of 4 KiB or less in flight (m_window; chdman's 256 for larger hunks and with a parent), and the page sizes batches from it, up to 64 hunks. chdman's window is part of its output: a partial last hunk is compressed whole, with what the work buffer held past the end of the data, the hunk 256 before it. So that tail is copied from that hunk, still in the wider buffer (zeros under 256 hunks), and the CHD is chdman 0.289's byte for byte (tests/ui/window.spec.js, each case, 1 and 4 threads). On the 1 GB DVD at 4 threads: Faster to create 12.99 -> 10.36 s, the default 50.42 -> 44.67 s. Also: chdman prints the whole CHD's final ratio (0.289 printed the last progress step's, as little as 0% in the browser). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
# Conflicts: # CLAUDE.md
… build WebAssembly's new exception format (exnref, -sWASM_LEGACY_EXCEPTIONS=0), with its own objects; like EMSCRIPTEN_ANY's, its page records no sources, so check-dist.sh and a release refuse it. rebuild.yml's second job builds it and runs the conversion tests on it: chdman's CHDs byte for byte, at the same speed. The default build is unchanged (the same wasm and glue from a clean build; the page's hash moves because build.sh is a hashed source). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
…red and not done Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
# Conflicts: # CLAUDE.md
# Conflicts: # dist/discpress.html
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone 8, steps 3 and 4 (docs/chd/fork-plan.md). Stacked on #46: merge that first, then this one's base becomes
main.What the profile showed
DEBUG.profile(new) makes the job worker report where its time went, the helpers' time too. "Faster to create" on the 1 GB benchmark DVD at 4 threads: of 13.2 s, the job worker spent 3.8 s reading the input, 2.5 s in the engine itself, 0.6 s handing out hunks, 0.8 s taking results, and waited the rest; the helpers were busy 21 of the 46 s they had.zlib,huff.The change
chd_file_compressor::m_window; chdman's 256 for larger hunks, so CDs are unchanged, and with a parent). The page sizes its batches from that window (wasm_par_enabledpasses it), up to 64 hunks. Natively nothing changes.Measured
1 GB benchmark DVD and 300 MB CD, 4 threads, median of three, every CHD byte-identical to native chdman:
FLAC: looked at, not changed
Step 4: the new exception format
WASM_EH=exnref ./build.shbuilds with exnref (-sWASM_LEGACY_EXCEPTIONS=0: 1,704try_table, no legacytry). Such a page records no sources hash, socheck-dist.shand a release refuse it..github/workflows/rebuild.yml's second job builds it and runs the conversion tests on it: chdman's CHDs byte for byte, at the same speed (DVD default 44.40 s against 44.67, CD 11.65 against 11.66). Switching, once every browser the page supports has exnref (Chrome 137, Firefox 131, Safari 18.4), is that flag andEH_TESTinapp/ui.js.Checks
tests/ui/window.spec.js: images of fewer than 256 hunks, 256–1024 and over 1024, each with a partial last hunk, through the page with 1 and 4 threads, default andzlib,huff: chdman 0.289's CHDs byte for byte. On a build with the wider window but without the tail it fails, as it should.-np 1) makes a seed of 300½ hunks under ASan: no report, and the same CHD as chdman 0.289's.REQUIRE_NATIVE=1full suite: 464 passed, 13 skipped;engine.spec.json the sanitized build: 43 passed;scripts/fuzz.sh check all: no findings;check-dist.sh,check-licenses.sh,engine-diff.sh --check: pass.window,fallbacks,edge-cases,tuning,verifyandadvancedspecs in Chromium withREQUIRE_NATIVE=1: 115 passed.🤖 Generated with Claude Code
https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty