Skip to content

fix(browser): clicks land on a scaled screen, and the bot's browser stops waiting for nothing - #213

Merged
httpsphl merged 2 commits into
mainfrom
fix/browser-clicks-and-speed
Oct 6, 2026
Merged

httpsphl merged 2 commits into
mainfrom
fix/browser-clicks-and-speed

Conversation

@httpsphl

@httpsphl httpsphl commented Oct 6, 2026

Copy link
Copy Markdown
Owner

Why

The bot's browser was slow, froze, and its clicks (the bot's and the owner's) did nothing or landed in the wrong place. I measured it with the real Edge (a throwaway probe, not committed) instead of guessing. Four causes.

What was wrong, and what changed

  1. Lost clicks on a scaled screen. The app sends the screen's scale (Windows at 125%, 150%, 200%) in browser.resize; the daemon drew the page sharper (deviceScaleFactor) and took a Page.captureScreenshot per frame. The browser then read click coordinates divided by the factor: the click on a button at (350, 220) landed at (280, 176), (233, 146) or (175, 110). Only 4 of 12 size and scale combinations landed, for the bot and for the owner. Holding a lock between pictures and input events did not fix it (11 of 12, then worse). The page is now always drawn in its own pixels and the panel scales the frame: 16 of 16 land. browser.resize still accepts scale and ignores it. Cost: the live view is slightly softer on a scaled screen.
  2. 3 s per action on pages that never go quiet. After each action the daemon waited up to 3 s for the network to go quiet; any page with analytics, ads or chat never does (3.2 s a click, against 0.17 s on a quiet page). A page that was already busy before the action now waits 0.6 s.
  3. Half a second more per action while the owner watches, and clicks at a stale point. The tool waited 560 ms for the cursor to glide, and had aimed before that wait, so a page that changed meanwhile (image, banner) took the click at the wrong point. The wait is now 300 ms (glide 260 ms in motion.css) and the element is found again after it.
  4. A button under a fixed header was clicked on the header. Now other scroll alignments and other points of the element are tried; if nothing is free, the page's own click is used and the bot is told.

Measured, before and after

Before After
Click on a page with constant traffic 3.18 s 0.78 s
Click with the panel open 735 ms 468 ms
Click with panel open and constant traffic 3.75 s 1.08 s
Clicks that land at 125%, 150%, 200% 4 of 12 16 of 16
Button under a fixed header clicked the header clicks the button

Tests

cargo test --workspace (550), clippy -D warnings, cargo fmt --check, pnpm check (446). New tests, each failing on the old code: clicks of the bot and of the owner at 100/125/150/200% (browser_panel.rs, replacing the test that asserted the old sharper page), a never-quiet page (browser.rs), a covered button (browser.rs). Spec 21.3, 21.4, 21.7 and 19 updated with the measurements.

Not covered

Only my own pages were tested with the real Edge; heavy real sites may still be slow for reasons of their own. If something still hangs, the site and a measurement will say where.

…tops waiting for nothing

Measured with the real Edge. On a screen scaled above 100% the page was drawn sharper (deviceScaleFactor) with a Page.captureScreenshot per frame, and the browser then read the bot's and the owner's clicks divided by the factor: 4 of 12 size and scale combinations landed. The page is now always drawn in its own pixels; the panel scales the frame. 16 of 16 land.

Also: after an action the daemon waited up to 3 s for a network that never goes quiet on pages with analytics, ads or chat (3.2 s a click); a page that was already busy before the action now waits 0.6 s. While the owner watches, the tool waited 560 ms for the cursor and aimed before the wait, so a page that changed meanwhile took the click at the wrong point: the wait is 300 ms (cursor glide 260 ms) and the element is found again after it. A button under a fixed header was clicked on the header: other scroll alignments and points of the element are tried, and when none is free the page's own click is used and the bot is told.

Tests: scaled-screen clicks (bot and owner, 100 to 200%), a never-quiet page, a covered button. Spec 21.3, 21.4, 21.7 and 19 updated.
@httpsphl
httpsphl enabled auto-merge October 6, 2026 20:35
The old behavior cost at least 3.15 s a click; the new one took 2.2 s on the macOS runner, so the bound is 2.9 s, between the two.
@httpsphl
httpsphl merged commit 0cda981 into main Oct 6, 2026
4 checks passed
@httpsphl httpsphl mentioned this pull request Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant