Skip to content

Engine robustness: fuzzing and the faults it found; a no-progress watchdog; Firefox, WebKit and benchmarks in CI - #46

Draft
PowerBeef wants to merge 13 commits into
mainfrom
claude/engine-robustness
Draft

PowerBeef wants to merge 13 commits into
mainfrom
claude/engine-robustness

Conversation

@PowerBeef

@PowerBeef PowerBeef commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

Step 1 of milestone 8 (docs/chd/fork-plan.md): make the engine and the page's own file readers safe on damaged and crafted input, and say when a job stops making progress; step 2's measuring in CI. Default CHDs are unchanged: byte for byte the same as chdman 0.289's.

Fuzzing

  • tests/fuzz/ and scripts/fuzz.sh: four libFuzzer targets under ASan and UBSan, with the engine built with __EMSCRIPTEN__ defined so the browser-only code is fuzzed too (the read-ahead of extract and verify, the helpers' decompression, identification's probe functions, the ECM decoder). tests/fuzz/host.cpp does what the page's JavaScript does for the engine: it runs chdman commands as the job worker does and answers the hunks they read ahead in four ways (no helpers, all, failing one in three, reversed), and aborts if chdman pauses with nothing to wait for (a hang in the browser).
    • chd: a CHD file, read as identification does, then info, verify and every extract command.
    • desc: a cue sheet, TOC, GDI, NRG or ISO, parsed, then made into a CHD, verified and extracted again.
    • codec: one compressed hunk for each codec.
    • ecm: ECM chunks rebuilt a window at a time.
  • seeds.cpp makes the seeds with the engine itself (v3 and v4 CHDs by hand); each finding is kept in tests/fuzz/regressions/, which every run replays.
  • .github/workflows/fuzz.yml: weekly 30 minutes per fuzzer, 5 when engine/, wasm/ or the fuzzers change; the corpus is kept in the Actions cache (pruned to what adds coverage), and a finding is uploaded.

What it found, fixed in the engine

All but one are in unmodified chdman 0.289 too; engine.spec.js compares with upstream where a fixture can show it (upstream: SIGSEGV, SIGFPE or "stack smashing detected").

  • A Huffman tree with more codes than one bit has room for: its lookup table written past its end, on the stack for a huff hunk or a v5 map. And the table's entries no code covers were never set.
  • A map size computed in 32 bits (wraps at 358 million hunks; written past its end), maps longer than the file allocated before the failing read (up to 4 GB), map bit widths over 32.
  • A map field of exactly 32 bits (found by this PR's CI, on its first run without a cached corpus): the bit reader shifted its 32-bit buffer by 32, which kept its bits on x86 and in WebAssembly, so the next field read them again. Only a crafted map has one (a real CHD would need 2³¹ hunks); now the buffer empties (chd/map-bits-32).
  • A CD codec's sector data longer than its hunk; LaserDisc frames larger than the hunk, more than 16 channels, raw audio shorter than its samples.
  • CD metadata read with %s into 16-byte buffers; CD metadata missing a track (0-byte sectors, a division by zero in extractcd); old-style CHCD metadata read past its end.
  • A read of 0 bytes reading a whole hunk into the caller's buffer (identification of such a track).
  • Hunks naming a codec or a parent the CHD doesn't have (null pointers); a parent whose units are larger than its hunks.
  • Undefined behavior with unchanged results: the bit writer's 32-bit shift, libFLAC's null pointer plus an offset, element 0 of an empty vector (a map or metadata of 0 bytes, and the read-ahead's own, the one not in 0.289).
  • And, from crafted headers: a hunk or unit size of 0, and more hunks than a map can count.

The page

  • tests/ui/readers.spec.js: damaged CSO/ZSO, PBP, ECM, zip and CloneCD files (cut short, fields set to extremes, bytes changed) each end as a card that says why or a conversion that finishes or fails, never an uncaught error or a card left waiting. Not changed here: a damaged file stored in a zip converts, since stored files are read in place and only deflated ones are checked against the zip's CRC-32.
  • A running conversion the page hears nothing from for 90 s while visible says it may be stuck, until it goes on (Engine.run's watchdog; a line in the Advanced tab).
  • On phones and tablets, the compression hint suggests "Nearly as small, faster" (the default stays chdman's).
  • Found by the Firefox and WebKit tests:
    • Firefox's tab crashes when a worker is terminated while it instantiates its WebAssembly, which happened every time a helper failed to start (the job stops while the other helpers are still starting). Every worker role now says wasm-ready once instantiated, and the page terminates workers only through Engine.stop, which waits for that (or 10 s). The job worker instantiates before copying its inputs, so a cancel during the copy isn't held up.
    • A job could choose where its results go before the check for private storage had finished (Store.ready, which jobs and Advanced tab commands now wait for).
    • Once a reload or close has begun, WebKit fails a job worker's reads of the picked files before the page is gone. The job then ended as an error, which cleared its record, so the next visit didn't say a conversion didn't finish. Once beforeunload has been seen, a job that fails keeps its record, which goes only if the page is still there 3 s later (Recovery.ended). The two tests that reload during a conversion accept WebKit's log of that cut-off read.

CI

  • .github/workflows/browsers.yml: the suite in Firefox and WebKit (EXTRA_BROWSERS), against the native references. In WebKit the tests follow the browser: the engine-failure message is Safari's own advice, DATs aren't kept across a reload and the tests of storage that fails as a run starts skip (Playwright's WebKit has no navigator.storage, as for the other storage tests).
  • .github/workflows/bench.yml: bench.js (now --browser) in Chromium, Firefox and WebKit on Ubuntu, against native chdman; the tables are the run's summary. (A macOS runner for WebKit was cancelled before any runner took it.)
  • First results (GitHub's 4-vCPU runners): with the default presets the page is 0.88–0.98× native in Chromium and 1.04–1.13× in Firefox; "Faster to create" on the 1 GB DVD at 4 threads is 2.2× native in Chromium and 2.8× in Firefox, which is the job worker's serial path (milestone 8, step 3).

Checks

  • REQUIRE_NATIVE=1 full suite (Chromium projects): 463 passed, 13 skipped (the usual skips), against unmodified chdman 0.289 and the native engine
  • Firefox: the helper start-up test, which crashed the tab 5 times in 5, passes 10 in 10; WebKit: the reload tests, which failed 1 in 4 on A wider compression window in the browser (DVDs 11–20% faster, byte for byte chdman's CHDs); an exnref build in CI #47's faster engine, pass 12 in 12 there
  • engine.spec.js on the sanitized build (ASan, LSan, UBSan): 43 passed
  • scripts/fuzz.sh check all on every seed, regression and corpus input: no findings; eight rounds of 20–25 minutes per fuzzer, each finding fixed before the next
  • scripts/check-dist.sh, check-licenses.sh, engine-diff.sh --check: pass

🤖 Generated with Claude Code

https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty

libFuzzer under ASan and UBSan (tests/fuzz, scripts/fuzz.sh, weekly in
.github/workflows/fuzz.yml), with the browser-only code built in. It found,
and the engine fixes (all but one also in chdman 0.289): a Huffman tree's
lookup table written past its end and left partly unset, a map size computed
in 32 bits, maps longer than the file allocated first, map bit widths over
32, CD codec sector data and LaserDisc frames longer than their hunk, CD
metadata read with %s into 16-byte buffers or missing a track, old-style CD
metadata read past its end, a 0-byte read that read a whole hunk, hunks
naming a codec or a parent the CHD doesn't have, and undefined behavior with
unchanged results. Also: a header with a hunk or unit size of 0, or more
hunks than a map counts, is an error. Default CHDs are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
CSO/ZSO, PBP, ECM, zip and CloneCD files cut short, with fields set to
extremes and bytes changed: each ends as a card that says why, or a
conversion that finishes or fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
…eset on phones

A conversion the page hears nothing from for 90 s while visible gets a note
that it may be stuck, until it goes on (DEBUG.freeze and stallMs for tests).
On phones and tablets the compression hint suggests Nearly as small, faster;
the default stays chdman's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
browsers.yml runs the suite with EXTRA_BROWSERS against the native references;
bench.yml times the page in Chromium and Firefox (Ubuntu) and WebKit (an
Apple-silicon Mac). bench.js takes --browser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
In Playwright's WebKit the engine-failure message is Safari's own advice, DAT
files aren't kept across a reload (no navigator.storage), and the tests of
storage that fails as a run starts have nothing to fail, so they skip there as
the other storage tests do. The macOS runner for WebKit's benchmark was
cancelled before any runner took it: Ubuntu instead, against native chdman.
bench.js gave native extracts after the first thread count no throughput.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
…here its results go

The check (Store.init) runs when the page opens and wasn't waited for, so a
job started before it was done kept its results in memory, and could ask the
memory question for nothing. Firefox's check can take longer on a busy
machine than the engine takes to load: CI's Firefox run lost one test to it.
Jobs and Advanced tab commands now wait for it (Store.ready); DEBUG.storeDelay
slows it for the test that shows it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
Firefox's tab crashes when a worker is terminated while it instantiates
the engine. A helper that fails to start stops the job at once, while the
other helpers are still starting: the tab crashed every time in the
Firefox tests (fallbacks.spec.js, "a helper thread that fails to start").

Every role now says 'wasm-ready' once instantiated, and the page
terminates workers only through Engine.stop, which waits for that (or for
'fatal', or 10 s). The job worker instantiates before it copies its
inputs, so a cancel while it copies isn't held up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
CI's fuzzing, its first run without a cached corpus, found it: the bit
reader's remove(32) shifted its 32-bit buffer by 32, which C++ leaves
undefined (UBSan reports it). On x86 and in WebAssembly the shift count
is masked, so the buffer kept its bits and the next field read them
again. Only a crafted map reaches it: a real CHD would need 2^31 hunks or
parent units for a 32-bit field. chdman 0.289 has it.

remove() now empties the buffer for 32 bits; fewer, as every real CHD
reads, is unchanged. The input is tests/fuzz/regressions/chd/map-bits-32.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
…ext visit

Once a reload or close has begun, WebKit fails a worker's reads of the
picked files before the page is gone ("The I/O read operation failed").
The job then ended as an error, which cleared its record, so the next
visit didn't say that a conversion didn't finish (recovery.spec.js in
WebKit, 1 in 4 runs on the wider-window branch, whose faster engine
reads sooner).

Once beforeunload has been seen, a job that fails keeps its record and
the record goes only if the page is still here 3 s later, running
nothing newer (the visitor chose to stay).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
Once a reload has begun, WebKit fails the job worker's reads of the
picked files and logs each, as a console error or an uncaught error
("... due to access control checks."). The two tests that reload during a
conversion now accept exactly that in WebKit; the note they check for is
what the page's fix (the record kept) makes them pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
Just after identification ends, a card can be redrawn (seen 5 ms after
app.settled returned, once in six runs): a click then reaches no button,
and the card stays ready, as tests (firefox) found on 07d3d61 (fallbacks,
the ECM image copied first). app.run clicks again if the card is still
ready 3 s later; a page that ignores the button still fails.

CLAUDE.md: Recovery.ended.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty
Now and then Playwright's Firefox never delivers a folder picked through
the chooser: no change event, and setFiles never returns. A plain page
with only a folder input shows it too (1 run in 200), more under CI's
load, where a folder test then waited until the test timeout (tests
(firefox) on 758a592: edge-cases "a cue sheet added later...").

app.addFolder, now used by every folder test, watches the chooser's input
for its change event and chooses the folder once more if none comes
within 10 s; a late first delivery would be a duplicate, which the page
ignores. Firefox: 135 runs of the folder tests and the whole project
(353) pass; a delivery dropped on purpose is chosen again in Chromium,
Firefox and WebKit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QUSh6AFtA1CAR66DNvjtty

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants