Keep opened files' sizes once DuckDB's heap passes 2 GiB

[jeyabbalas]

Summary

Loading several large Parquet files into one page stopped working after the second one. Every load after that failed with "too small to be a Parquet file" or "Prefetch registered for bytes outside file … file size: 0". The cause is a signed shift in duckdb-wasm's browser runtime, which drops an opened file's size once DuckDB's heap passes 2 GiB. The library now corrects those writes inside DuckDB's worker.

This matters for the manual pass, which loads several large files in one page.

Why: root cause

When DuckDB opens a file, duckdb-wasm's runtime (openFile) allocates 24 bytes and writes three values into them through mod.HEAPF64[(ptr >> 3) + i]: the file's size, a buffer pointer and a modification time.

>> is a signed shift. Once the heap is past 2 GiB, _malloc can return an address above 2^31. For such an address (ptr >> 3) is negative, so a Float64Array silently drops the writes. DuckDB then reads the freshly grown memory, which is zeroed, and sees a file of 0 bytes.

Evidence:

• The runtime's side was correct. I routed DuckDB's worker script to a copy with logging. Every openFile saw the File, with the right size (387,603,633 bytes).
• The failing addresses. The 24-byte blocks of the failing loads were at about 2.6 GB. The heap was 2.86 GB then, and 2.05 GB after the first load.
• Why it was intermittent. It depends on where malloc places those 24 bytes. The 3 MB file never takes the heap near 2 GiB.
• What was ruled out. File names, handle registration, and dropSourceFile are not involved: every open found its handle.
• Why prefetch looked involved. Leaving the prefetch settings alone avoided the failure only because the heap then grew less, and the loads took 40 to 65 s instead of 5 to 20 s.
• Holding a 2.2 GB table in DuckDB makes every load fail at once. The new browser test is built on this.
• Upstream is the same. duckdb-wasm's main branch (packages/duckdb-wasm/src/bindings/runtime_browser.ts) still writes mod.HEAPF64[(result >> 3) + …]. The glue code around it already passes pointers unsigned (t>>>=0), so reads into buffers above 2 GiB work. Only these writes are affected.

What changed

• src/worker/openFileFixScript.js (new): the fix, as plain JavaScript. It wraps the runtime's openFile.
  • For the duration of the call, the module's HEAPF64 is a view that moves a write at a negative index to that index plus 2^29, which is where (ptr >> 3) + i came from.
  • Addresses below 2 GiB are untouched.
  • If the call grows the heap, the view follows the new one, since Emscripten replaces its views on growth.
  • After the call, the module gets back its own HEAPF64 descriptor, accessor or data property, or none if the heap was inherited. A heap the call grew is then assigned as Emscripten would have assigned it.
  • A module whose HEAPF64 cannot be redefined gets duckdb-wasm's own openFile, unchanged, and the worker's console says so once.
  • It covers every branch of openFile (File handles, OPFS, and HTTP, as used for extension downloads), not only File handles.
• When it is installed. duckdb-wasm publishes the runtime as globalThis.DUCKDB_RUNTIME when its WebAssembly module is instantiated, so the fix hooks that assignment. It checks again whenever the runtime is read, for a runtime published empty and filled in afterwards.
• src/worker/openFileFix.ts (new): duckdbWorkerSource. It builds DuckDB's worker bootstrap: importScripts(mainWorker), then the fix's text, imported with ?raw so no bundler rewrites it.
  • The fix runs in a try/catch. A fix that cannot install itself leaves DuckDB as it was and logs a warning in the worker's console.
  • The file's comment names the other signed-shift writes in duckdb-wasm, which the library never reaches: a scalar UDF's result (udf_runtime.ts) and dropFiles' pointer array (bindings_base.ts).
• src/worker/duckdb.ts builds its worker from duckdbWorkerSource, for both the CDN and self-hosted bundles.
• Docs:
  • Troubleshooting FAQ §31 covers the two errors: the cause, and what coi hosts see.
  • The CSP guide's coi caveats say the coi pthread workers go without the fix.
  • docs/dev/memory-envelope.md records the finding.

Tests

• Browser tests:

  • tests/browser/parquet-file-load.spec.ts, "loads a Parquet File again and again once DuckDB's heap is past 2 GiB":
    • it holds a 2.2 GB table of random doubles, which takes about 5 s to build, and loads a 10,000-row Parquet File three times;
    • each load must succeed;
    • it takes 9 s; peak browser memory is about 3 GB;
    • fail-before (main's duckdb.ts): all three loads fail with Invalid Input Error: File '__dt_source_N.parquet' too small to be a Parquet file.
  • tests/browser/duckdb-worker-bootstrap.spec.ts (new), "DuckDB still starts and loads when the openFile fix cannot install itself":
    • DuckDB's worker script is served with one line more, which makes DUCKDB_RUNTIME a property that cannot be redefined;
    • fail-before (cd45424's openFileFix.ts): createDataTable rejects with WorkerInitError: Worker error: Uncaught TypeError: Cannot redefine property: DUCKDB_RUNTIME.

• Unit tests, tests/worker/openFileFix.test.ts, 16 tests.

  • They run the shipped text (openFileFixScript.js?raw) in fresh vm contexts, with a fake runtime that writes as duckdb-wasm does and a heap that drops negative-index writes as a Float64Array does.
  • They check:
    • the bug itself;
    • the fix for runtimes published after install, before it, and published empty then filled in;
    • answers below 2 GiB are unchanged;
    • heap growth during the call;
    • the module's heap is restored afterwards;
    • bound methods;
    • no double wrapping;
    • the fallback to duckdb-wasm's openFile for a heap that cannot be redefined, and for a module that takes no new properties, with one warning;
    • an accessor HEAPF64 put back as it was, with a grown heap handed to its setter, and an inherited heap left inherited;
    • the bootstrap: it installs the fix, and it warns and carries on when the fix cannot install itself.

• Mutations: 15 run on the reworked fix, 13 caught.

  • The two missed were equivalent: a fix call before the hook was installed, and a check for a non-configurable HEAPF64 in front of the try/catch. Each only repeated what the code after it does, so I removed both.
  • That leaves 13 of 13 on the final text. I reran the fallback mutation on that text.

• Big files, headless Chromium, on the fixed branch, all in one page:

  Scenario	main	this PR
  4 × wide-50k.parquet (388 MB), a fresh File each time	loads 3 and 4 fail (seen 3 times; one run failed only load 3)	4 of 4 load, 11 to 23 s each
  4 × the same file by URL (fetched as a Blob)	not measured	4 of 4
  Switching wide-50k ↔ deep-1m (1M × 41), 5 loads	not measured	5 of 5
  Input File and IndexedDB File, alternating, 4 loads	loads 3 and 4 fail	4 of 4
  200K → 50K → 200K → 50K (the 200K file is 1.5 GB)	not measured	4 of 4, the 200K loads in 36 and 67 s

  These were run on cd45424. The fix's text has not changed in substance since.

Gates

All eight pass on 83dff48:

• eslint, prettier, typecheck, build, size and docs.
• Unit: 4,651 passed, 10 skipped.
• Browser: 95 passed.

Review

The review confirmed the diagnosis on 1.33.1-dev57.0 and found one Medium and four Lows, all fixed in b0a2021:

1. Medium: the fix did not fail safe, and what shipped was never tested.
   • A throw while the fix installed made every createDataTable reject with WORKER_CRASHED. The fix now runs in a try/catch, warns, and leaves DuckDB as it was.
   • A module whose HEAPF64 cannot be redefined made every file open throw. The call now goes to duckdb-wasm's openFile unchanged.
   • What shipped was the TypeScript function's toString(), whatever the bundler made of it. The fix is now openFileFixScript.js, shipped as its text with ?raw, and the unit tests run that same text.
   • Tests:
     • a browser test in which the fix cannot install and DuckDB still loads;
     • unit tests for the warning, and for both fallbacks.
2. coi.
   • The CSP guide's caveats and troubleshooting FAQ §31 now cover it.
   • The fix checks again on each read of DUCKDB_RUNTIME, for a runtime published empty and filled in afterwards. That is the coi pthread worker's pattern, but those workers load their own script, so the fix does not reach them.
3. Other signed-shift writes. openFileFix.ts's comment names the scalar UDF result writes (udf_runtime.ts) and dropFiles' pointer array (bindings_base.ts). The library registers no UDFs and calls only dropFile.
4. Troubleshooting entry: FAQ §31.
5. Upstream issue. The report text is drafted and not filed.

The delta review (of b0a2021) found it mergeable, with two small things, fixed in 83dff48:

1. Low: the fallback for a HEAPF64 that cannot be redefined was silent, although FAQ §31 tells users to look for [data-table] DuckDB runs without the fix…. It now logs that warning once.
   • Unit test: two opens, one warning.
   • Fail-before: on b0a2021 it fails.
2. Nit: HEAPF64 became a writable data property after each call. The original descriptor is now restored exactly. Unit tests:
   • an accessor HEAPF64 keeps its getter, setter and attributes, and its setter gets the heap the call grew;
   • an inherited heap stays inherited;
   • both fail on b0a2021.

Mutations for the delta: 5 of 5 caught:

• the warning shown every time;
• no warning;
• the heap restored as a data property;
• a grown heap not handed back;
• an inherited heap shadowed.

Not changed

• The loaders, the dispatcher and LoadPayload. The loader-options PR changes those, so the two don't overlap.
• The coi (pthread) bundle's pthread workers. They load their own script and are not patched. The library's default bundles are mvp and eh, so this only matters for a host that self-hosts coi on a cross-origin-isolated page.
• Upstream. The bug should be reported to duckdb-wasm; the report is drafted. When a fixed release ships, the fix can go.
• Pre-existing: relative bundle paths. The CSP guide's self-hosted example uses root-relative mainWorker paths (/assets/duckdb/…). importScripts in DuckDB's blob-URL worker cannot resolve those. Resolving them against the library worker's self.location.href would fix it. It is left for its own PR, with a test that self-hosts the bundles.

🤖 Generated with Claude Code