Repository navigation
fix(browser): clicks land on a scaled screen, and the bot's browser stops waiting for nothing - #213
Merged
Merged
Conversation
…tops waiting for nothing Measured with the real Edge. On a screen scaled above 100% the page was drawn sharper (deviceScaleFactor) with a Page.captureScreenshot per frame, and the browser then read the bot's and the owner's clicks divided by the factor: 4 of 12 size and scale combinations landed. The page is now always drawn in its own pixels; the panel scales the frame. 16 of 16 land. Also: after an action the daemon waited up to 3 s for a network that never goes quiet on pages with analytics, ads or chat (3.2 s a click); a page that was already busy before the action now waits 0.6 s. While the owner watches, the tool waited 560 ms for the cursor and aimed before the wait, so a page that changed meanwhile took the click at the wrong point: the wait is 300 ms (cursor glide 260 ms) and the element is found again after it. A button under a fixed header was clicked on the header: other scroll alignments and points of the element are tried, and when none is free the page's own click is used and the bot is told. Tests: scaled-screen clicks (bot and owner, 100 to 200%), a never-quiet page, a covered button. Spec 21.3, 21.4, 21.7 and 19 updated.
httpsphl
enabled auto-merge
October 6, 2026 20:35
The old behavior cost at least 3.15 s a click; the new one took 2.2 s on the macOS runner, so the bound is 2.9 s, between the two.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The bot's browser was slow, froze, and its clicks (the bot's and the owner's) did nothing or landed in the wrong place. I measured it with the real Edge (a throwaway probe, not committed) instead of guessing. Four causes.
What was wrong, and what changed
browser.resize; the daemon drew the page sharper (deviceScaleFactor) and took aPage.captureScreenshotper frame. The browser then read click coordinates divided by the factor: the click on a button at (350, 220) landed at (280, 176), (233, 146) or (175, 110). Only 4 of 12 size and scale combinations landed, for the bot and for the owner. Holding a lock between pictures and input events did not fix it (11 of 12, then worse). The page is now always drawn in its own pixels and the panel scales the frame: 16 of 16 land.browser.resizestill acceptsscaleand ignores it. Cost: the live view is slightly softer on a scaled screen.motion.css) and the element is found again after it.Measured, before and after
Tests
cargo test --workspace(550), clippy-D warnings,cargo fmt --check,pnpm check(446). New tests, each failing on the old code: clicks of the bot and of the owner at 100/125/150/200% (browser_panel.rs, replacing the test that asserted the old sharper page), a never-quiet page (browser.rs), a covered button (browser.rs). Spec 21.3, 21.4, 21.7 and 19 updated with the measurements.Not covered
Only my own pages were tested with the real Edge; heavy real sites may still be slow for reasons of their own. If something still hangs, the site and a measurement will say where.