Skip to content

feat(desktop): keys, clicks at a point and what accessibility cannot do, with the owner's real hands - #178

Merged
httpsphl merged 3 commits into
mainfrom
feat/desktop-real-tools
Oct 4, 2026
Merged

httpsphl merged 3 commits into
mainfrom
feat/desktop-real-tools

Conversation

@httpsphl

@httpsphl httpsphl commented Oct 4, 2026

Copy link
Copy Markdown
Owner

What

D4b (spec 24.6, 24.7). It sits on top of #177, so this diff shows it too until it merges. The owner's switch in the app and the on-screen notice come in D4c.

  • Real mouse and keyboard is a per-grant option (real_input) that only the owner changes, through desktop.setOptions {grantId, realInput}. It returns the bot's grants and sends bot.desktop. The bot never asks for it in the chat; tools that need it tell the bot to ask the owner, who turns it on in the bot's details.
  • New tools and inputs:
    • desktop_press {window, keys} presses keys by name. The system's own combinations are refused before anything is asked of the owner.
    • desktop_click_at {window, x, y} clicks a point of the bot's last picture of that window. desktop_screenshot now keeps the picture's size, and without one the tool asks for it.
    • desktop_type takes submit: Enter afterwards, with the real keyboard.
  • Fallback when accessibility cannot do it. A click or typing that a control does not take through UI Automation goes through the real mouse and keyboard, where they are on:
  • Owner's hands free. The bot takes the mouse and keyboard only when the owner has left them alone for 2 seconds, or 10 seconds for ten minutes after the owner took over in the middle of an action. A takeover answers the bot at once and is remembered.
  • Panel. It also words pressed keys ("Apertou Ctrl+S").
  • Spec 24.6 and 24.7 are updated.

Checks

  • cargo fmt, clippy -D warnings and pnpm check pass. Desktop lib, desktop, desktop_panel and mcp tests pass.
  • Unit tests cover the 2 s and 10 s quiet rule, picture sizes per window, and the option in the store.
  • New tests/desktop_real.rs with a real window:
    • Runs everywhere:
      • system keys are refused;
      • the option is off until the owner turns it on, and setOptions turns it on;
      • a click at a point needs a picture first;
      • while the owner is using the computer, the bot waits.
    • Runs only with BOTLOFT_REAL_INPUT_TESTS=1 (CI's Windows runner): typing with submit goes through the real keyboard and presses Enter. The equivalent test in feat(desktop): the real mouse and keyboard, stopping when the owner moves #177 ran in CI and passed.

…do, with the owner's real hands

D4b (spec 24.6, 24.7): the real mouse and keyboard become an option of
each grant, real_input, that only the owner changes, through
desktop.setOptions; the bot never asks for it in the chat, and the tools
that need it tell it to ask the owner.

- desktop_press presses keys by name in a window; the system's own
  combinations are refused before anything is asked of the owner.
- desktop_click_at clicks a point of the bot's last picture of that
  window (desktop_screenshot keeps its size).
- desktop_type takes submit: Enter afterwards, with the real keyboard.
- A click or typing a control does not take through accessibility goes
  through the real mouse and keyboard where they are on: a click at its
  center, or click, select from start to end, type.
- The bot takes the mouse and keyboard only when the owner has left them
  alone for 2 seconds, 10 after the owner took them back in the middle
  of an action; a takeover answers the bot at once.

The panel words pressed keys too.
@httpsphl
httpsphl enabled auto-merge October 4, 2026 23:20
@httpsphl
httpsphl merged commit 88606b8 into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant