Repository navigation
feat(desktop): keys, clicks at a point and what accessibility cannot do, with the owner's real hands - #178
Merged
Conversation
…do, with the owner's real hands D4b (spec 24.6, 24.7): the real mouse and keyboard become an option of each grant, real_input, that only the owner changes, through desktop.setOptions; the bot never asks for it in the chat, and the tools that need it tell it to ask the owner. - desktop_press presses keys by name in a window; the system's own combinations are refused before anything is asked of the owner. - desktop_click_at clicks a point of the bot's last picture of that window (desktop_screenshot keeps its size). - desktop_type takes submit: Enter afterwards, with the real keyboard. - A click or typing a control does not take through accessibility goes through the real mouse and keyboard where they are on: a click at its center, or click, select from start to end, type. - The bot takes the mouse and keyboard only when the owner has left them alone for 2 seconds, 10 after the owner took them back in the middle of an action; a takeover answers the bot at once. The panel words pressed keys too.
httpsphl
enabled auto-merge
October 4, 2026 23:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
D4b (spec 24.6, 24.7). It sits on top of #177, so this diff shows it too until it merges. The owner's switch in the app and the on-screen notice come in D4c.
real_input) that only the owner changes, throughdesktop.setOptions {grantId, realInput}. It returns the bot's grants and sendsbot.desktop. The bot never asks for it in the chat; tools that need it tell the bot to ask the owner, who turns it on in the bot's details.desktop_press {window, keys}presses keys by name. The system's own combinations are refused before anything is asked of the owner.desktop_click_at {window, x, y}clicks a point of the bot's last picture of that window.desktop_screenshotnow keeps the picture's size, and without one the tool asks for it.desktop_typetakessubmit: Enter afterwards, with the real keyboard.Ctrl+Home,Ctrl+Shift+Endbecause Ctrl+A does not work in every classic field (CI showed this in feat(desktop): the real mouse and keyboard, stopping when the owner moves #177).Checks
cargo fmt,clippy -D warningsandpnpm checkpass. Desktop lib,desktop,desktop_panelandmcptests pass.tests/desktop_real.rswith a real window:setOptionsturns it on;BOTLOFT_REAL_INPUT_TESTS=1(CI's Windows runner): typing withsubmitgoes through the real keyboard and presses Enter. The equivalent test in feat(desktop): the real mouse and keyboard, stopping when the owner moves #177 ran in CI and passed.