Repository navigation
feat(desktop): the real mouse and keyboard, stopping when the owner moves - #177
Merged
Merged
Conversation
…oves The platform part of D4 (spec 24.7), not used by the tools yet: - real_click, real_type and real_press with SendInput. The app's window comes to the front first (an empty mouse move makes Windows let the daemon do it); before every step the window in front must be the app's, and a click must land on it, not on a window over it; otherwise nothing is sent. Typing is Unicode, so the keyboard layout does not matter. Clicks go to a control's center, from accessibility's BoundingRectangle (sessions are now per-monitor DPI aware), or to a point of the window's picture. - Low-level mouse and keyboard hooks, only during the action, see input that is not injected: the owner's. The bot stops at once. - Keys by name with Ctrl, Alt and Shift, never the system's own combinations (the Windows key, Alt+Tab, Alt+Esc, Ctrl+Esc, Ctrl+Shift+Esc, Ctrl+Alt+Del), in pure code with tests. - The owner's idle time leaves the bot's own input out. - Readings keep each control's place on the screen. The tests that move the cursor and type run only with BOTLOFT_REAL_INPUT_TESTS=1, which CI sets on its Windows runner, where nobody sits at the desktop; never on a developer's computer.
httpsphl
enabled auto-merge
October 4, 2026 23:02
…sic field takes it
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The platform part of D4 (spec 24.7). Nothing calls it yet; the tools and the per-app option come in D4b.
real_click,real_typeandreal_press, usingSendInput:SetForegroundWindow. A minimized window is restored.WindowFromPoint) must be the app's too. Otherwise nothing is sent, and the tool says the window is not in front or is covered.SendInputsending less than asked, means the screen is locked or takes no input.BoundingRectangle(sessions are now per-monitor DPI aware, so the numbers are real pixels), or a point of the window's picture as fractions of its DWM frame. The move is absolute across all screens.KEYEVENTF_UNICODE), so the keyboard layout does not matter. A newline is Enter.keys.rs, pure code with tests):Enter,Tab,Ctrl+S,Shift+Tab,F5, letters and digits, with Ctrl, Alt and Shift.rect). It is not shown to the bot.Checks
cargo fmtandclippy -D warningspass. Desktop lib tests pass (27 here).BOTLOFT_REAL_INPUT_TESTS=1. CI sets it on the Windows runner, where nobody sits at the desktop. I have not run them locally, because that would move the owner's mouse.