Skip to content

feat(desktop): bots list, read and picture the owner's windows, app by app as the owner allows - #171

Merged
httpsphl merged 2 commits into
mainfrom
feat/desktop-tools
Oct 4, 2026
Merged

httpsphl merged 2 commits into
mainfrom
feat/desktop-tools

Conversation

@httpsphl

@httpsphl httpsphl commented Oct 4, 2026

Copy link
Copy Markdown
Owner

What

The daemon part of D1 (spec 24.2, 24.6, 24.8, 24.11). It sits on top of #170 and #169, so this diff shows them too until they merge. The app part (the request card and the Desktop section in the bot's settings) comes next, in D1d.

  • Storage. A new desktop_grants table (migration 20):
    • one grant per bot and app, matched by the executable's path regardless of case;
    • a grant only goes up, from see to act;
    • deleting the bot deletes its grants.
  • Protocol.
    • desktop.grants {botId} lists a bot's grants.
    • desktop.revoke {grantId} takes one away and returns the rest.
    • The bot.desktop notification goes out on every change.
    • Ids use the new dsk_ prefix, which is also added to CLAUDE.md.
  • Tools.
    • desktop_windows lists the windows. Titles of apps not granted yet stay hidden, and what is never available is named with the reason.
    • desktop_look {window, why, from?} reads a window through its accessibility tree.
    • desktop_screenshot {window, why} takes a picture of one window.
    • The first use of an app asks the owner through the approvals path. The request is mcp__botloft__desktop with {app, path, level, why}. When the owner allows it, the grant is saved.
    • The bot's permission mode (auto, bypass_permissions) never opens the desktop.
    • The tools refuse windows on the never-granted list.
    • One desktop action at a time across all bots.
  • Owner present. The tools work only when the app is open (a connection past session.hello) and the computer was used in the last 5 minutes (GetLastInputInfo). Tests replace that with a stand-in owner.
  • Spec. Section 24 is updated: the request card has Allow and Deny, why is required, and the protocol is listed.

Checks

  • cargo fmt, clippy -D warnings, cargo test --workspace and pnpm check pass. FakeBotloft gains desktop.*.
  • Unit tests cover grant coverage and the list hiding titles of apps not granted yet.
  • New end-to-end test in tests/desktop.rs (Windows only), using a real window of the test's own:
    • the title is hidden at first;
    • reading it asks the owner, showing the bot's why;
    • once allowed, the reading has the button and the field but never the password text;
    • the grant is saved and announced with bot.desktop;
    • the title then shows, and the picture comes back as a JPEG;
    • after revoking, it asks again, and a refusal reaches the bot with the owner's note.
  • A second test checks that nothing is seen while the owner is away.

Manual check on the real app (after D1d)

Ask a bot to read a Notepad window. Allow it in the chat, check the reading, then revoke it in the bot's settings.

…y app as the owner allows

The daemon part of D1 (spec 24.2, 24.6, 24.8, 24.11):

- desktop_grants (migration 20): one grant per bot and app, by the
  executable's path case aside, that only goes up from see to act;
  deleting the bot deletes them. desktop.grants and desktop.revoke, and
  the bot.desktop notification; ids with the dsk_ prefix.
- Tools desktop_windows, desktop_look and desktop_screenshot. The list
  hides the titles of apps not granted yet and names what is never
  available. Reading or picturing a window of a new app asks the owner
  with the bot's why (mcp__botloft__desktop); allowed, the grant is kept.
  The bot's permission mode never opens the desktop.
- The owner must be there: the app open (a connection past hello) and
  the computer used in the last 5 minutes (GetLastInputInfo); tests stand
  in for the owner. One desktop action at a time across bots.

Tested end to end on Windows with a real window of the test's own: the
title hidden, then the request, the reading without the password, the
grant kept and told, the picture, revoked and refused; and nothing seen
while the owner is away.
@httpsphl
httpsphl merged commit 180f3ee into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant