fix(capture): fetch the scripts a page discovers while running - #29
Closed
pathscale wants to merge 1 commit into
Closed
fix(capture): fetch the scripts a page discovers while running#29pathscale wants to merge 1 commit into
pathscale wants to merge 1 commit into
Conversation
The prefetch walks the parsed HTML and loads every script it names. A page that builds a `<script>` from JavaScript asks for a URL nobody knew about until the page was already running, and that request reached `DefaultScriptFetcher`, which serves `file:` and `data:` only. The script was dropped with `unsupported URL scheme for script: https`, which reads like a policy decision rather than the missing capability it is. Measured over a hundred-site corpus, this is the single most common engine defect: **26 sites**, a quarter of the corpus. It hides well, because scripts the parser found load perfectly, so a page fails only in the parts it assembles itself, and jQuery going undefined on seven sites looks like its own bug rather than a consequence of never having been fetched. `ScriptFetcher::fetch` is synchronous, because classic scripts execute in document order, so the network call blocks. It runs on the page's own provider and so keeps the per-origin connection cap the rest of the loads obey, and it is bounded by a timeout: a server that accepts and never answers would otherwise hang the capture, and a missing script is better than a run that never ends. Verified by re-capturing the six worst-affected sites: dropped scripts went 12, 6, 4, 2, 2, 2 to zero everywhere. Rendering barely moved, which is the honest result rather than a disappointing one. On one site the error count rose from 9 to 14 with a new kind, `unescape is not defined`: the scripts now run and reach the *next* missing global. This unblocks a layer; the missing web APIs behind it are what turn that into pixels. The test serves a script from a real socket and asserts a synchronous fetch completes a round trip from inside the runtime driving the page. Its accept loop has a deadline rather than blocking: restoring the defect makes no connection at all, and a blocking accept hangs the run instead of failing it. Confirmed by restoring the defect, which fails the test in ten seconds. Only the capture path. `browser.rs` has the same fetcher and is reached from a synchronous poll with no runtime handle to hand, so it needs a stored handle and is left for its own change.
Owner
Author
|
Superseded by #30, which stacks this with the viewport fix and adds the window half of the same defect. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The prefetch walks the parsed HTML and loads every script it names. A page that builds a
<script>from JavaScript asks for a URL nobody knew about until the page was already running, and that request reachedDefaultScriptFetcher, which servesfile:anddata:only. The script was dropped withunsupported URL scheme for script: https— which reads like a policy decision rather than the missing capability it is.Why it matters
Measured across a 104-site corpus, this is the most common engine defect: 26 sites, a quarter of the corpus.
It hides well. Scripts the parser found load perfectly, so a page fails only in the parts it assembles itself.
jQueryand$come out undefined on 7 and 6 sites respectively, which looks like its own defect until you notice the library was simply never fetched.How
ScriptFetcher::fetchis synchronous, because classic scripts must execute in document order, so the network call has to block. It runs on the page's own provider, keeping the per-origin connection cap the rest of the page's loads obey, and it is bounded by a 10s timeout — a server that accepts and never answers would otherwise hang the capture, and a missing script is a better outcome than a run that never finishes.The blocking uses
block_in_placeon the multi-threaded runtime. The current-thread case is spelled out and refuses rather than deadlocking, instead of anunwrapthat would be correct only until someone changed the builder.Verification
Re-captured the six worst-affected sites. Dropped scripts went 12, 6, 4, 2, 2, 2 → 0 on every one.
Rendering barely moved, which is the honest result. On one site the error count rose 9 → 14 with a new kind —
unescape is not defined. The scripts now run and reach the next missing global. This unblocks a layer; the missing web APIs behind it (fetch,XMLHttpRequest,Image,getComputedStyle) are what turn it into pixels.Test serves a script from a real socket and asserts a synchronous fetch completes a round trip from inside the runtime driving the page. Its accept loop carries a deadline rather than blocking, because restoring the defect means no connection is ever made and a blocking accept would hang the run instead of failing it. Confirmed by restoring the defect: the test fails in 10.02s.
cargo fmt --check,cargo clippy -D warnings, and 34 tests all clean.Scope
Capture path only.
browser.rscarries the same fetcher but is reached from a synchronous poll with no runtime handle in scope, so it needs a stored handle and belongs in its own change.