Skip to content

feat(desktop): bot cursor on the desktop panel, and the panel opens by itself - #182

Merged
httpsphl merged 1 commit into
mainfrom
feat/desktop-cursor
Oct 5, 2026
Merged

httpsphl merged 1 commit into
mainfrom
feat/desktop-cursor

Conversation

@httpsphl

@httpsphl httpsphl commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Part of the owner's feedback on Desktop Use: the desktop should behave like the browser (spec 24.9).

What changes

  • DesktopAction gains x and y: where the action happened, as fractions of the window's frame.
    • Through accessibility, it is the middle of the control (its BoundingRectangle), relative to DWMWA_EXTENDED_FRAME_BOUNDS.
    • For desktop_click_at, it is the point of the picture.
    • Keys and controls outside the window have no point.
  • The desktop panel puts the window on the same stage as the browser (LiveView, now with plain frame and action types). The bot's cursor glides to each point, and a ring marks clicks.
  • The panel opens by itself on the first desktop.changed with a window after 2 minutes without one. It does not open while the owner holds the browser, and it respects "follow the bot". Closed mid-run, it stays closed for the rest of that run.

Tests

  • Unit test for the control's point inside the window frame.
  • The real test window: clicking Save reports a point near the top left.
  • App: the cursor and the ring land at the action's point; the panel opens once per run and again after a pause.

Manual

  • With a real app (Notepad or Chrome), watch the cursor land on the clicked control in the panel.

…itself

Each desktop action now says where it happened in the window: the middle
of the control through accessibility, or the point of a desktop_click_at,
as fractions of the window's frame (DesktopAction x and y). The desktop
panel shows the window on the same stage as the browser, with the bot's
cursor gliding there and a ring on each click.

The panel also opens by itself when the bot starts using the desktop
after two minutes without it, like the browser does, unless the owner
turned off following the bot.
@httpsphl
httpsphl enabled auto-merge October 5, 2026 00:19
@httpsphl
httpsphl merged commit 917820a into main Oct 5, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant