[httpsphl]

What

D2 (spec 24.5, 24.6): bots act in the owner's apps through UI Automation, without the owner's mouse or keyboard.

• Tools. desktop_click {ref}, desktop_type {ref, text}, desktop_select {ref, option} and desktop_scroll {ref, to?}. Each also takes why?, which is needed only the first time in an app.
• Refs come from the bot's last reading, which the daemon keeps per bot (from desktop_look, or the reading an action returns). Before acting, the control is found again by its RuntimeId. A control that is gone or disabled is refused with a clear message.
• A separate permission to use. The first action in an app asks the owner whether the bot may use it (level: "act"), showing the bot's why. Allowing it raises the existing see grant. Without a why at that moment, the tool asks for one.
• How each action works:
  • Click: the first pattern the control has among Invoke, Toggle, SelectionItem.Select, and Expand/Collapse.
  • Type: Value.SetValue. It refuses password fields (before asking anything) and read-only fields. Enter after typing comes with the real keyboard in D4.
  • Choose: selects the item by name where it shows. If the item only exists in a combo box's open list, it opens the box and takes the item's default action (LegacyIAccessible.DoDefaultAction). A classic Win32 combo box cancels a choice when its list is closed any other way, the same as pressing Esc. I saw this on Windows 11.
  • Scroll: Scroll or SetScrollPercent; for a list item, ScrollIntoView.
• Five-second limit per call into an app (TransactionTimeout). A click that opens a dialog does not hang the bot; it reads that the app did not answer in time.
• After each action the tool returns the window read again, with its title as it is now and a line on what it did, for example Clicked button "Save".
• One UI Automation session at a time in the process. Two sessions on different threads at once fail with "unspecified error"; I saw this on Windows 11. The code that walks the tree was reorganized into a shared session, and the per-control reading moved to uia_control.rs.
• App:
  • the card reads "Scout wants to use Notepad" and explains what using means (clicks and typing as you would, without your mouse, every action shown in the chat, never in password fields);
  • the four tools have names in en, pt-BR and es.
• Spec 24.5 and 24.6 are updated. Section 24's status now says D1 and D2 are implemented.

Checks

• cargo fmt, clippy -D warnings and pnpm check pass (395 tests).
• cargo test --workspace passes. Two real-browser tests (browser_panel, command_requests) failed while a release build ran on the same machine, and both pass on their own.
• Real-control tests in the tests' own window (act_tests.rs):
  • a click renames the window;
  • typing replaces a field;
  • a check box toggles;
  • a combo box option is chosen, and a missing option is reported;
  • a 50-item list scrolls to its end;
  • typing in a password field is refused;
  • a control that is gone is reported.
• The test windows wait for each other, because a new window takes the focus and an open combo list closes when its window loses it. The suite passed 5 times in a row.
• End to end (tests/desktop.rs):
  • acting before reading is refused;
  • seeing is asked for and allowed;
  • typing in the password field is refused;
  • a click without a why asks for one, then asks the owner to use the app;
  • once allowed, the click shows "Saved" in the title, the grant is at the act level, and typing goes through without asking again.

Manual check on the real app

1. Ask a bot to fill a field in Notepad's Find dialog or a classic form app.
2. Allow it to use the app.
3. Check the actions in the chat and the reading that comes back.