Skip to content

feat: the owner takes a bot's browser into their own hands (N3) - #35

Merged
httpsphl merged 4 commits into
mainfrom
feat/browser-hands
Sep 29, 2026
Merged

httpsphl merged 4 commits into
mainfrom
feat/browser-hands

Conversation

@httpsphl

@httpsphl httpsphl commented Sep 29, 2026 •

Copy link
Copy Markdown
Owner

What

N3 from spec 21.11: the owner clicks, drags, scrolls and types in a bot's live browser to sign in or get past a captcha, with the bot waiting.

  • Daemon: browser.take, browser.input and browser.release, all tied to the app connection that watches the browser. Events go through one queue per hold, in order, and mouse moves in a row collapse to the newest. The bot's browser_* tools wait while the owner holds the browser (up to 10 min). Stopping watching, closing the browser or dropping the connection gives it back.
  • browser_ask_owner {task}: the bot asks for a hand through the approvals path. The owner is done when they answer Done in the chat or give the browser back in the panel; the tool then answers with the page as it is. The bot's rules and the tool description say never to ask for passwords, codes or card numbers in the chat.
  • App:
    • "Take control" under the live screen.
    • A hidden text box captures the keys, so accents, input methods and Ctrl+V paste work.
    • A bar shows the owner is in control, with "Done, give it back to ".
    • A chat card for the request; its Take the browser button opens the panel already in the owner's hands.
    • The request also shows as a notice in the panel, and the header button gets a warning dot.
    • Texts in the three languages.
  • Privacy: nothing the owner types or pastes is logged, sent as browser.action, or given to the bot. The Debug output of BrowserInput prints only the event kind.

Spec: new section 21.10; the milestones are now 21.11. Section 19 records the real test.

Tests

  • cargo test --workspace: 253 pass.
    • New real-Edge tests in tests/browser_hands.rs:
      • the owner signs in on a local page while browser_ask_owner waits;
      • the bot's browser_look waits until the browser is given back;
      • Ctrl+A replaces a field's text;
      • a second connection cannot take a browser someone already has;
      • "I won't do it" reaches the bot with the note.
    • Unit tests for the key mapping and for event checks.
  • pnpm check: 132 pass. New hands.test.tsx covers clicks mapped to page pixels, keys, paste, give back, the chat card opening the panel in the owner's hands, and a denial.
  • Real Claude Code (2.1.284, Haiku, Manual mode), dev daemon and app preview:
    • Asked to open a local sign-in page and call me, the bot loaded browser_ask_owner via ToolSearch by itself.
    • I took the browser from the card, typed e-mail and password on the live screen, pressed Enter and gave the browser back.
    • The bot listed the orders it saw after signing in.
    • The password showed up nowhere: not in the daemon log, the database or the bot's transcript.

Checked by hand

  • Ctrl+V and an accent (´ + a) reach the page (checked by the owner)
  • The wheel over the live screen scrolls the bot's page, not the panel: this branch's daemon, the real Edge, the app in the Claude desktop Browser pane
  • The same in the app window (pnpm tauri dev with BOTLOFT_HOME pointing at a dev folder), since the installed 0.5.0 has no browser

The app connection that watches a bot's browser can take it
(browser.take), send the owner's mouse, wheel, keys and pasted text to
the page (browser.input) and give it back (browser.release). Events go
through one queue per hold, in order; moves in a row collapse to the
newest. Shortcuts run from the Windows virtual key, AltGr types. While
the owner holds it, the bot's browser tools wait, up to 10 minutes.
Watching stops, the browser closes or the connection drops: the browser
goes back to the bot.

browser_ask_owner lets a bot ask for a hand (sign-ins, captchas, codes)
through the approvals path, and answers with the page once the owner is
done; giving the browser back answers it. The bot's rules say to ask
this way instead of asking for passwords in the chat. What the owner
types is never logged nor sent to the bot.

Spec 21.10 describes it; tested with the real Edge.
"Take control" under the live screen puts the owner's hands on the
page: clicks, drags, the wheel and keys go to the bot's browser through
a hidden text box, so accents, input methods and pasting work. A bar
says the owner is in control and gives the browser back. A bot's
request for a hand shows as a card in the chat, whose button opens the
panel already in the owner's hands, and as a notice in the panel; the
browser button marks it. Texts in the three languages; the preview has
a Clerk that asks to sign in.
@httpsphl
httpsphl enabled auto-merge September 29, 2026 17:37
@httpsphl
httpsphl merged commit 8e2a2e0 into main Sep 29, 2026
3 checks passed
@httpsphl httpsphl mentioned this pull request Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant