Repository navigation
feat(desktop): the owner turns on the real mouse and keyboard, and sees on screen when a bot has them - #180
Merged
Merged
Conversation
…do, with the owner's real hands D4b (spec 24.6, 24.7): the real mouse and keyboard become an option of each grant, real_input, that only the owner changes, through desktop.setOptions; the bot never asks for it in the chat, and the tools that need it tell it to ask the owner. - desktop_press presses keys by name in a window; the system's own combinations are refused before anything is asked of the owner. - desktop_click_at clicks a point of the bot's last picture of that window (desktop_screenshot keeps its size). - desktop_type takes submit: Enter afterwards, with the real keyboard. - A click or typing a control does not take through accessibility goes through the real mouse and keyboard where they are on: a click at its center, or click, select from start to end, type. - The bot takes the mouse and keyboard only when the owner has left them alone for 2 seconds, 10 after the owner took them back in the middle of an action; a takeover answers the bot at once. The panel words pressed keys too.
…es on screen when a bot has them D4c (spec 24.7, 24.10): - The bot's details gain, on each grant to use an app, a switch for the real mouse and keyboard, off from the start. Turning it on first opens a dialog that says what changes and how to stop it; turning it off is at once. The app calls desktop.setOptions. - While a bot uses them, the daemon shows a notice on the screen: a thin border in the bot's color around the monitor of the app's window and a pill at its top, "<bot> is using your mouse and keyboard. Move the mouse to stop it." A layered window of its own on its own thread: always on top, clicked through, never taking the focus, never listed nor in another window's picture. It shows before each real action and goes 3 seconds after the last. - Its words are the daemon's, in the app's three languages, since the app may be closed: the app sends its language with session.hello (client.locale) and session.setLocale when the owner changes it. A test, on CI's Windows runner only, clicks through the notice.
httpsphl
enabled auto-merge
October 4, 2026 23:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
D4c (spec 24.7, 24.10). It sits on top of #178, so this diff shows it too until it merges. With this, D4 is complete.
desktop.setOptions.WS_EX_TRANSPARENT), it never takes focus, never appears in the window list, and never shows up in another window's picture.session.hello(client.locale) and with the newsession.setLocalewhen the owner changes it;Checks
cargo fmt,clippy -D warningsandpnpm checkpass.realInput.test.tsx):session.setLocalereaches the daemon.