Repository navigation
feat: the owner takes a bot's browser into their own hands (N3) - #35
Merged
Merged
Conversation
The app connection that watches a bot's browser can take it (browser.take), send the owner's mouse, wheel, keys and pasted text to the page (browser.input) and give it back (browser.release). Events go through one queue per hold, in order; moves in a row collapse to the newest. Shortcuts run from the Windows virtual key, AltGr types. While the owner holds it, the bot's browser tools wait, up to 10 minutes. Watching stops, the browser closes or the connection drops: the browser goes back to the bot. browser_ask_owner lets a bot ask for a hand (sign-ins, captchas, codes) through the approvals path, and answers with the page once the owner is done; giving the browser back answers it. The bot's rules say to ask this way instead of asking for passwords in the chat. What the owner types is never logged nor sent to the bot. Spec 21.10 describes it; tested with the real Edge.
"Take control" under the live screen puts the owner's hands on the page: clicks, drags, the wheel and keys go to the bot's browser through a hidden text box, so accents, input methods and pasting work. A bar says the owner is in control and gives the browser back. A bot's request for a hand shows as a card in the chat, whose button opens the panel already in the owner's hands, and as a notice in the panel; the browser button marks it. Texts in the three languages; the preview has a Clerk that asks to sign in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
N3 from spec 21.11: the owner clicks, drags, scrolls and types in a bot's live browser to sign in or get past a captcha, with the bot waiting.
browser.take,browser.inputandbrowser.release, all tied to the app connection that watches the browser. Events go through one queue per hold, in order, and mouse moves in a row collapse to the newest. The bot'sbrowser_*tools wait while the owner holds the browser (up to 10 min). Stopping watching, closing the browser or dropping the connection gives it back.browser_ask_owner {task}: the bot asks for a hand through the approvals path. The owner is done when they answer Done in the chat or give the browser back in the panel; the tool then answers with the page as it is. The bot's rules and the tool description say never to ask for passwords, codes or card numbers in the chat.browser.action, or given to the bot. TheDebugoutput ofBrowserInputprints only the event kind.Spec: new section 21.10; the milestones are now 21.11. Section 19 records the real test.
Tests
cargo test --workspace: 253 pass.tests/browser_hands.rs:browser_ask_ownerwaits;browser_lookwaits until the browser is given back;pnpm check: 132 pass. Newhands.test.tsxcovers clicks mapped to page pixels, keys, paste, give back, the chat card opening the panel in the owner's hands, and a denial.browser_ask_ownerviaToolSearchby itself.Checked by hand
pnpm tauri devwithBOTLOFT_HOMEpointing at a dev folder), since the installed 0.5.0 has no browser