Skip to content

feat(browser): teach a bot a task by doing it once - #159

Merged
httpsphl merged 1 commit into
mainfrom
feat/teach-task
Oct 4, 2026
Merged

httpsphl merged 1 commit into
mainfrom
feat/teach-task

Conversation

@httpsphl

@httpsphl httpsphl commented Oct 4, 2026

Copy link
Copy Markdown
Owner

Part 8 of the ideas from 2026-10-04, the last one. New spec section 21.13 (milestone N3.5).

Daemon

  • browser.teach {botId, on} (only from the connection holding the browser): starts a lesson with an open step for the active tab, or ends it and returns the steps. BrowserState.lesson carries the steps live; giving the browser back, closing the connection or the browser drops it.
  • lesson.rs: with a lesson on, each owner move is read before it reaches the page, through the isolated-world script (actions.js gains at(x, y) and focused(), which return only role, name and whether it is a password field):
    • left click → click with the name of the nearest element with a role;
    • text keys, Backspace, pasted/composed text → one type step per field, secret for password fields;
    • Enter/Escape/Tab → press; an address the owner types → open.
  • Never kept: what is typed, pasted or chosen; an address keeps scheme, site and path only (no user:pass, query or fragment). Labels ≤ 80 chars, ≤ 100 steps, memory only, never logged.
  • Input helpers of the Edge tests moved to tests/common/browsing.rs.

App

  • Under the owner's pill: Teach a task. Recording, a "Recording the lesson" strip lists the steps live with Cancel, and the pill's button becomes Finish the lesson.
  • Then a dialog: the task's name and its steps (each removable), and Make it a routine… (routine dialog prefilled with the lesson in words + "Do it now.") or Send to to remember (owner message asking to keep it in memory).
  • The lesson in words is built in the owner's language and says where things were typed, never what; passwords: ask the owner.

Test plan

  • cargo fmt, clippy -D warnings, cargo test --workspace (new tests/browser_lesson.rs with the real Edge: steps, secret, nothing typed kept; unit tests for addresses and steps), Edge tests run 3 more times
  • pnpm check (new lesson.test.tsx: words, live strip, send, routine prefill)
  • Preview: lesson strip, pill, dialog
  • Real app with a bot: teach a short task on a real site, make it a routine, run it now and see the bot follow the steps

With the browser in their hands, the owner starts a lesson: each click,
field typed in, key and address becomes a step by what it means, read
from the page by the isolated-world script before the step reaches it.
What is typed is never read, and addresses lose their query. At the end
the owner names the task and makes it a routine, or sends it for the
bot to remember.
@httpsphl
httpsphl enabled auto-merge October 4, 2026 18:10
@httpsphl
httpsphl merged commit 22b7217 into main Oct 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant