Skip to content

fix(capture): fetch the scripts a page discovers while running - #29

Closed
pathscale wants to merge 1 commit into
masterfrom
fix/runtime-script-fetch
Closed

fix(capture): fetch the scripts a page discovers while running#29
pathscale wants to merge 1 commit into
masterfrom
fix/runtime-script-fetch

Conversation

@pathscale

Copy link
Copy Markdown
Owner

What

The prefetch walks the parsed HTML and loads every script it names. A page that builds a <script> from JavaScript asks for a URL nobody knew about until the page was already running, and that request reached DefaultScriptFetcher, which serves file: and data: only. The script was dropped with unsupported URL scheme for script: https — which reads like a policy decision rather than the missing capability it is.

Why it matters

Measured across a 104-site corpus, this is the most common engine defect: 26 sites, a quarter of the corpus.

It hides well. Scripts the parser found load perfectly, so a page fails only in the parts it assembles itself. jQuery and $ come out undefined on 7 and 6 sites respectively, which looks like its own defect until you notice the library was simply never fetched.

How

ScriptFetcher::fetch is synchronous, because classic scripts must execute in document order, so the network call has to block. It runs on the page's own provider, keeping the per-origin connection cap the rest of the page's loads obey, and it is bounded by a 10s timeout — a server that accepts and never answers would otherwise hang the capture, and a missing script is a better outcome than a run that never finishes.

The blocking uses block_in_place on the multi-threaded runtime. The current-thread case is spelled out and refuses rather than deadlocking, instead of an unwrap that would be correct only until someone changed the builder.

Verification

Re-captured the six worst-affected sites. Dropped scripts went 12, 6, 4, 2, 2, 2 → 0 on every one.

Rendering barely moved, which is the honest result. On one site the error count rose 9 → 14 with a new kind — unescape is not defined. The scripts now run and reach the next missing global. This unblocks a layer; the missing web APIs behind it (fetch, XMLHttpRequest, Image, getComputedStyle) are what turn it into pixels.

Test serves a script from a real socket and asserts a synchronous fetch completes a round trip from inside the runtime driving the page. Its accept loop carries a deadline rather than blocking, because restoring the defect means no connection is ever made and a blocking accept would hang the run instead of failing it. Confirmed by restoring the defect: the test fails in 10.02s.

cargo fmt --check, cargo clippy -D warnings, and 34 tests all clean.

Scope

Capture path only. browser.rs carries the same fetcher but is reached from a synchronous poll with no runtime handle in scope, so it needs a stored handle and belongs in its own change.

The prefetch walks the parsed HTML and loads every script it names. A page
that builds a `<script>` from JavaScript asks for a URL nobody knew about
until the page was already running, and that request reached
`DefaultScriptFetcher`, which serves `file:` and `data:` only. The script was
dropped with `unsupported URL scheme for script: https`, which reads like a
policy decision rather than the missing capability it is.

Measured over a hundred-site corpus, this is the single most common engine
defect: **26 sites**, a quarter of the corpus. It hides well, because scripts
the parser found load perfectly, so a page fails only in the parts it
assembles itself, and jQuery going undefined on seven sites looks like its own
bug rather than a consequence of never having been fetched.

`ScriptFetcher::fetch` is synchronous, because classic scripts execute in
document order, so the network call blocks. It runs on the page's own provider
and so keeps the per-origin connection cap the rest of the loads obey, and it
is bounded by a timeout: a server that accepts and never answers would
otherwise hang the capture, and a missing script is better than a run that
never ends.

Verified by re-capturing the six worst-affected sites: dropped scripts went
12, 6, 4, 2, 2, 2 to zero everywhere. Rendering barely moved, which is the
honest result rather than a disappointing one. On one site the error count
rose from 9 to 14 with a new kind, `unescape is not defined`: the scripts now
run and reach the *next* missing global. This unblocks a layer; the missing
web APIs behind it are what turn that into pixels.

The test serves a script from a real socket and asserts a synchronous fetch
completes a round trip from inside the runtime driving the page. Its accept
loop has a deadline rather than blocking: restoring the defect makes no
connection at all, and a blocking accept hangs the run instead of failing it.
Confirmed by restoring the defect, which fails the test in ten seconds.

Only the capture path. `browser.rs` has the same fetcher and is reached from a
synchronous poll with no runtime handle to hand, so it needs a stored handle
and is left for its own change.
@pathscale

Copy link
Copy Markdown
Owner Author

Superseded by #30, which stacks this with the viewport fix and adds the window half of the same defect.

@pathscale pathscale closed this Aug 31, 2026
@pathscale
pathscale deleted the fix/runtime-script-fetch branch September 1, 2026 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant