Skip to content

feat(desktop): the owner turns on the real mouse and keyboard, and sees on screen when a bot has them - #180

Merged
httpsphl merged 4 commits into
mainfrom
feat/desktop-real-app
Oct 4, 2026
Merged

httpsphl merged 4 commits into
mainfrom
feat/desktop-real-app

Conversation

@httpsphl

@httpsphl httpsphl commented Oct 4, 2026

Copy link
Copy Markdown
Owner

What

D4c (spec 24.7, 24.10). It sits on top of #178, so this diff shows it too until it merges. With this, D4 is complete.

  • Switch in the bot's details:
    • each grant to use an app gets a "Real mouse and keyboard" switch, off from the start; grants to only see don't get one;
    • turning it on first opens a dialog that says what changes:
      • the bot moves the cursor and types as the owner would, when accessibility can't;
      • while it acts, the mouse and keyboard are its, and a notice on screen says so;
      • moving the mouse or pressing a key stops it at once, and Ctrl+Alt+Esc stops every bot;
      • it never types in password fields and never presses Windows' own keys;
    • turning it off happens at once, with no dialog;
    • the app calls desktop.setOptions.
  • Notice on the owner's screen while a bot uses the real mouse and keyboard:
    • a thin border in the bot's color around the monitor the app's window is on;
    • a dark pill at the top: "Scout is using your mouse and keyboard. Move the mouse to stop it."
    • It is a layered window of its own, on its own thread: always on top, clicks pass through (WS_EX_TRANSPARENT), it never takes focus, never appears in the window list, and never shows up in another window's picture.
    • It appears before each real action and goes away 3 seconds after the last.
  • Notice language:
    • the words live in the daemon, in the app's three languages, because the app may be closed;
    • the app sends its language with session.hello (client.locale) and with the new session.setLocale when the owner changes it;
    • with no language known, it uses English.
  • Spec 24.7, 24.10 and section 24's status (D4 implemented) are updated.

Checks

  • cargo fmt, clippy -D warnings and pnpm check pass.
  • App tests (realInput.test.tsx):
    • the switch is off and only appears on grants to use;
    • the dialog says what changes, and Cancel leaves it off;
    • Turn on turns it on, and turning it off happens at once;
    • a language change reaches the fake daemon.
  • Daemon tests:
    • the notice text in each language and the bot's color;
    • session.setLocale reaches the daemon.
  • On CI's Windows runner only: a real click goes through the notice and reaches the window.
  • Manual: see the notice on a real screen while a bot uses Notepad with the switch on, then move the mouse.

…do, with the owner's real hands

D4b (spec 24.6, 24.7): the real mouse and keyboard become an option of
each grant, real_input, that only the owner changes, through
desktop.setOptions; the bot never asks for it in the chat, and the tools
that need it tell it to ask the owner.

- desktop_press presses keys by name in a window; the system's own
  combinations are refused before anything is asked of the owner.
- desktop_click_at clicks a point of the bot's last picture of that
  window (desktop_screenshot keeps its size).
- desktop_type takes submit: Enter afterwards, with the real keyboard.
- A click or typing a control does not take through accessibility goes
  through the real mouse and keyboard where they are on: a click at its
  center, or click, select from start to end, type.
- The bot takes the mouse and keyboard only when the owner has left them
  alone for 2 seconds, 10 after the owner took them back in the middle
  of an action; a takeover answers the bot at once.

The panel words pressed keys too.
…es on screen when a bot has them

D4c (spec 24.7, 24.10):

- The bot's details gain, on each grant to use an app, a switch for the
  real mouse and keyboard, off from the start. Turning it on first opens
  a dialog that says what changes and how to stop it; turning it off is
  at once. The app calls desktop.setOptions.
- While a bot uses them, the daemon shows a notice on the screen: a thin
  border in the bot's color around the monitor of the app's window and a
  pill at its top, "<bot> is using your mouse and keyboard. Move the
  mouse to stop it." A layered window of its own on its own thread:
  always on top, clicked through, never taking the focus, never listed
  nor in another window's picture. It shows before each real action and
  goes 3 seconds after the last.
- Its words are the daemon's, in the app's three languages, since the
  app may be closed: the app sends its language with session.hello
  (client.locale) and session.setLocale when the owner changes it.

A test, on CI's Windows runner only, clicks through the notice.
@httpsphl
httpsphl enabled auto-merge October 4, 2026 23:34
@httpsphl
httpsphl merged commit 4020e7a into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant