Skip to content

feat(desktop): the real mouse and keyboard, stopping when the owner moves - #177

Merged
httpsphl merged 2 commits into
mainfrom
feat/desktop-real-input
Oct 4, 2026
Merged

httpsphl merged 2 commits into
mainfrom
feat/desktop-real-input

Conversation

@httpsphl

@httpsphl httpsphl commented Oct 4, 2026

Copy link
Copy Markdown
Owner

What

The platform part of D4 (spec 24.7). Nothing calls it yet; the tools and the per-app option come in D4b.

  • real_click, real_type and real_press, using SendInput:
    • Bringing the window to the front. An empty mouse move first makes Windows let the daemon call SetForegroundWindow. A minimized window is restored.
    • Checks before every step. The foreground window, up to its root, must be the app's. For a click, the window under the point (WindowFromPoint) must be the app's too. Otherwise nothing is sent, and the tool says the window is not in front or is covered.
    • Locked screen. No foreground window, or SendInput sending less than asked, means the screen is locked or takes no input.
    • Where clicks land. Either the center of a control, from UI Automation's BoundingRectangle (sessions are now per-monitor DPI aware, so the numbers are real pixels), or a point of the window's picture as fractions of its DWM frame. The move is absolute across all screens.
    • Typing uses Unicode (KEYEVENTF_UNICODE), so the keyboard layout does not matter. A newline is Enter.
  • The owner takes over. Low-level mouse and keyboard hooks run only during the action, on a thread with their own message queue, and spot input not marked as injected, which is the owner's. The bot stops at once.
  • Keys by name (keys.rs, pure code with tests): Enter, Tab, Ctrl+S, Shift+Tab, F5, letters and digits, with Ctrl, Alt and Shift.
    • Refused: the Windows key, Alt+Tab, Alt+Esc, Ctrl+Esc, Ctrl+Shift+Esc and Ctrl+Alt+Del.
    • Alt+F4 is allowed: it closes the app's window, as it would for the owner.
  • Owner idle time (spec 24.8) now leaves the bot's own input out.
  • Readings keep each control's place on the screen (rect). It is not shown to the bot.
  • Spec 24.7 is updated.

Checks

  • cargo fmt and clippy -D warnings pass. Desktop lib tests pass (27 here).
  • Real input tests: these move the cursor and type, so they only run with BOTLOFT_REAL_INPUT_TESTS=1. CI sets it on the Windows runner, where nobody sits at the desktop. I have not run them locally, because that would move the owner's mouse.
    • Click a field, press Ctrl+A, type, read the new value.
    • Click Save and see the title change.
  • The test that every control has its place on the screen runs everywhere.
  • Not testable automatically: the owner's input stopping the bot. Any input a test sends is marked as injected, so it is a manual check, listed in spec 19.

…oves

The platform part of D4 (spec 24.7), not used by the tools yet:

- real_click, real_type and real_press with SendInput. The app's window
  comes to the front first (an empty mouse move makes Windows let the
  daemon do it); before every step the window in front must be the
  app's, and a click must land on it, not on a window over it; otherwise
  nothing is sent. Typing is Unicode, so the keyboard layout does not
  matter. Clicks go to a control's center, from accessibility's
  BoundingRectangle (sessions are now per-monitor DPI aware), or to a
  point of the window's picture.
- Low-level mouse and keyboard hooks, only during the action, see input
  that is not injected: the owner's. The bot stops at once.
- Keys by name with Ctrl, Alt and Shift, never the system's own
  combinations (the Windows key, Alt+Tab, Alt+Esc, Ctrl+Esc,
  Ctrl+Shift+Esc, Ctrl+Alt+Del), in pure code with tests.
- The owner's idle time leaves the bot's own input out.
- Readings keep each control's place on the screen.

The tests that move the cursor and type run only with
BOTLOFT_REAL_INPUT_TESTS=1, which CI sets on its Windows runner, where
nobody sits at the desktop; never on a developer's computer.
@httpsphl
httpsphl enabled auto-merge October 4, 2026 23:02
@httpsphl
httpsphl merged commit 590b2e5 into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant