Repository navigation
feat(desktop): let the agent read another app's window as named elements, approved per app (#21096, PR 1) - #21120
Open
sauranshbhardwaj wants to merge 19 commits into
Open
sauranshbhardwaj wants to merge 19 commits into
sauranshbhardwaj wants to merge 19 commits into
Conversation
…ool policy A dispatch_required decision for a sensitive on-device tool now has a policy-owned shape for what the user is asked: the capability, operation and resource the grant would be matched against, a bounded display preview of the input (attachment paths reduced to a file name, text and scripts capped at the 4 KiB card bound with a truncation flag the broker fails closed on), and the answers. Deny is the default; allow_once is always offered; allow_session is offered only when there is an exact resource for a grant to cover and carries plain words for what it would cover, so a script grant is shown as covering only that exact script. The 180s expiry matches the mobile device-tool transport and stays under the pi-mono long tool wait minus the Swift executor timeout, so a late answer cannot run an effect the model already reported as failed. Only this module decides the question and the options; adapters and the model never choose allow or deny.
2 tasks done
Collaborator
|
Reviewed this statically against the diff. This is careful work — the single-source refusal floor ( Leaving this for human maintainer review rather than approving — it's stacked on the approval chain (#20973 → #20975 → #20997 → #20998), it's a first PR from this account, and it's a privacy-sensitive design decision that has real open questions, several called out in the PR body:
One more thing maintainers should look at directly: this PR changes agent-facing guidance and model-facing tool copy — Thanks for the exceptionally thorough PR description and evidence artifacts — it made review much easier. by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
Git-on-my-level
added
needs-maintainer-review
Contributor
Author
|
Thanks for the careful read. Taking your points in order:
On the agent-facing guidance and docs: agreed that it needs a human read together with the security surface. The new copy ("Call it directly: Omi asks the person to approve it in the app") is only true once the parking flow in #20973 → #20997 is live. If you'd prefer, I can keep |
sauranshbhardwaj
force-pushed
the
feat/mac-ui-snapshot
branch
2 times, most recently
from
October 10, 2026 21:03
cc5d706 to
169d868
Compare
… approval.requested send_message, run_applescript, read_message_history, list_message_chats and list_mail_messages used to fail with approval_required whenever no scoped grant existed, and nothing in the app could create one. capture_screen did not fail at all: it took and shared a full-screen screenshot with no approval, although its own description promised one, so it now parks the same way under the stable resource "screen". A screenshot holds whatever is on screen, including windows the person keeps out of capture elsewhere, so each one is allowed once: its card offers no session grant, no grant row covers a capture, and the kernel refuses to mint a grant the card did not offer. The realtime voice screenshot tool is no longer advertised to pi-mono, where it sat ungated in every typed chat, pill and workstream run; realtime voice runs still get it through their surface projection. The capability broker now prepares the ledger row and inserts one approval dispatch bound to that exact invocation (id, input hash, daemon boot epoch) in the same transaction, the kernel moves the run and attempt to waiting_approval and appends run.waiting_approval plus approval.requested, and the broker refuses to dispatch the invocation until the dispatch is resolved. resolve_desktop_dispatch gains the two user answers: decision allow without a grant admits exactly this invocation once and mints nothing; allow with a grant whose runId is null mints a session-scoped grant through the existing insertGrant path, capped at 24 hours. Deny, cancellation, expiry, run termination and owner change fail the prepared invocation closed as failed, never outcome_unknown, each with its own approval.resolved, and the run resumes so the model receives approval_denied. A new kernel operation, endChatApprovalScope, ends what a chat granted when the person clears it: the main chat keeps one kernel session across clears, so it closes every approval still parked in that session and revokes every grant the person minted for it, with one grants.revoked audit event. Startup reconciliation expires any pending bound approval and records approval.resolved, so a restart never replays a parked call. Every relay-callable tool is now classified on purpose (named in a policy set or declared unbundled, never by the read-only default), and a test holds the approval set to every relay-callable tool in a sensitive bundle, with each deliberate exception named and justified, so a new tool cannot ship unclassified or ungated by omission. Grants for desktop.context.screenshot_image must name their exact resource. The gated tools are held to their manifest schema at the kernel boundary and the resource a grant covers is derived only from the fields the tool acts on, so a model-supplied resource_ref can no longer become the grant pattern. A dispatch created through create_desktop_dispatch has its invocation payload key stripped and can never pose as a parked call. Authority is revalidated before a resolution is admitted and that check runs before the resolving transaction, so a revocation it triggers commits on its own; a rolled-back decision restores the broker's memory. The model still cannot resolve its own dispatch: the relay path keeps returning policy_denied without trustedUserControl. Parking sits behind a kernel switch, desktopToolApprovalsEnabled, that defaults to off and is not reachable from the daemon in this change, so runtime behavior is unchanged until a client that renders approval cards turns it on. The Swift card PR flips the default and deletes the switch.
…ol calls The stdio relay no longer answers a parked device tool call immediately. A new relay-tool-approvals module, taking the kernel and the frame and result writers, sends Swift an approval_requested frame carrying the dispatch id, the bound invocation, the policy preview, the options and the expiry, keeps the relay call pending, and settles it from the kernel's approval.resolved event: an allow decision hands the invocation to Swift through the ordinary authorized_tool_execution path exactly once, every other outcome returns approval_denied to the model. Every resolution is mirrored to Swift as approval_resolved so a card can close whatever ended it, including owner changes. Expiry fires from the runtime, a disconnected relay client cancels its own dispatch, and terminal run events fail parked calls like any other pending tool call. The Swift hand-off itself moves into the same module so the whole relay side is covered by hermetic tests with injected timers. Clearing a chat through journal_clear_turns now ends what that chat granted before the journal is cleared: the kernel closes the session's parked approvals and revokes the grants the person minted for it, because the main chat keeps one kernel session across clears and an "Allow for This Chat" grant would otherwise cover the new conversation. No tool reaches this; only the app's own clear message does. Parking is on only after a client declares desktop_tool_approval_cards through the new client_capabilities message; until the Swift card ships no client does, and the relay keeps today's immediate approval_required. The runtime advertises desktop_tool_approval_requests so that client can require it first. Realtime voice keeps today's behavior because its Swift tool wait is 180s and would race the approval TTL.
The pi-mono extension waits 30s for a normal Swift tool result and gives up with a timeout error. A human answering an approval card needs longer than that, and a late approval would then run a send the model had already reported as failed. The six device tools that park behind an approval, capture_screen among them, now declare the long timeout class, which the extension maps to its 10 minute wait, and a test holds every gated tool to it; the approval TTL of 180s plus the 120s Swift executor timeout stays well inside it. Swift does not consume timeoutClass outside the generated surfaces, so the regenerated files only carry the new manifest digests.
The approval UX design doc keeps its design status and points at the device tool surface doc for the runtime half that now exists; the Swift card, the Claude Code lane and the voice surface stay deferred and section 24 still applies. The device tool surface doc gains the ask-and-answer table, the schema and resource rules that stop the model naming what a grant covers, the session-grant options and their 24 hour cap, the restart and expiry behavior, and the interim client-capability gate. The changelog fragment is kind none because nothing is user-visible until the Swift card lands.
Render the kernel's approval_requested frame as a card in the main chat, workstream and agent pill threads, matched to the asking session, with the exact target, the content preview, the expiry and the answers the runtime offered; answer through signed direct control (allow once, allow for this chat with a 1 h session grant, deny) and close the card on approval_resolved whoever ended it. Declare desktop_tool_approval_cards after the handshake so the relay parks a sensitive call instead of failing it, require desktop_tool_approval_requests from the runtime, and add tool_approval_snapshot and answer_tool_approval bridge actions so omi-ctl can drive the card without Accessibility permission.
… stall In the live run the chat's 90 s tool no-progress guard revoked the turn and interrupted the bridge while a send_message card was waiting for the person, so the kernel cancelled the run and the card read Cancelled, not run. Teach StallDetector a waitingOnUser state the turn's tick loop sets from the approval store: while a card for the session is pending nothing promotes to slow or stalled, no tool is ever overdue, the tool row reads Waiting for your approval, and when the card closes every clock restarts from that moment; the stop button still cancels. Give the hermes and openclaw ACP adapters the same rule, treating a run the kernel holds in waiting_approval as progress for their 150 s no-progress cancel.
Six in-process captures from the packaged named bundle: a parked send_message card, the same card after Deny, parked list_message_chats and list_mail_messages cards, and the Allowed once state after an answer, with evidence.json recording agent capture through the automation bridge.
The five sensitive device tools described themselves as always requiring an approval dispatch, so the in-app model answered a send request by calling create_desktop_dispatch instead of send_message; that dispatch has no card, nothing renders it, and it never expires. Now that the kernel parks a direct call behind the approval card, say so in the descriptions and capability guidance and tell the model to call the tool directly, including capture_screen, which is now gated the same way, and regenerate the Swift tool surfaces for the changed manifest digest.
Add the Swift core of ui_snapshot, not yet wired to any tool. WindowSnapshotWalker reads one window through an AccessibilityElementSource that has no press, focus, raise or type method, so it can only read; its one write is the app-level switch that makes an Electron (AXManualAccessibility) or Chromium (AXEnhancedUserInterface) app expose its elements; the switch is turned back off on every exit, only if Omi turned it on and it is still on, so a read leaves no lasting change in the other app. The walk keeps every labelled, valued or actionable element and walks through unlabelled containers; anonymous wrapper groups (Electron and Chromium nest dozens around web content) cost no depth, so the depth limit of 12 counts informative elements, with an absolute raw depth of 64; it stops at 400 emitted elements, 2,000 visited, a 3 s deadline that also covers the window list and the pane check, the calling task's cancellation, or the first timeout from a hung app, keeping what it read, and it reads at most 200 children of any one container (its first and last 100) and carries on with the rest of the window. It reads role and subrole first and never requests a secure field's value or length or walks into it, leaves out any app text over 300 characters (value, label, window title) with only its length, strips the query string and fragment of every URL-like token in app text (any scheme, including file: and mailto:, or none) and blanks long opaque path segments, prefers a list's visible rows, sanitizes and quotes app text so it cannot forge another line or field, and gives each element an a:/n:/p: reference plus a fingerprint from its role, label and window-relative frame. A result with fewer than five elements, none actionable, or a depth-cut walk that found almost nothing beyond window chrome is marked sparse with its reason, and with the assistive switch on a sparse first pass is read once more. WindowSnapshotOrdering then moves the window's main content first: the region holding most of its text (text values, not control labels), or an AXLandmarkMain region, goes before sidebars, toolbars and chrome, with references and paths unchanged, and the header says order=content_first. The result is Omi's header line first, with the app's name and window title last and quoted, then one line per element, and carries facts only. UIAutomationTargetPolicy resolves the one running copy of an approved bundle id without launching anything and refuses Omi itself, every com.omi.* build, the refused apps, anything that is not an .app bundle (app extensions, XPC view services, bare executables), an app with no name, capture-excluded apps and a frontmost app with secure input on. SettingsPaneGuard checks every open System Settings window by an allow-list: a sensitive name in the selected sidebar row or the window title refuses, otherwise the selection must name an ordinary top-level pane and, under General, the title an ordinary General subpage, in every language the installed extensions ship (display names, plus General's and Privacy & Security's page strings from their Localizable tables, read by SettingsPaneCatalog). A window titled with a Privacy & Security page is refused unless the sidebar selects that same name; no selection, an unknown row, a search, a sheet, a list with no position or an unidentifiable window is refused. The refused list, including Apple's credential and notification surfaces, the AutoFill and passkey sign-in view services and each sensitive pane's own extension, and the pane lists are declared once in the kernel's ui-automation-safety-floor.ts and generate-tool-surfaces.mjs writes them into GeneratedUIAutomationSafetyFloor.swift. _AXUIElementGetWindow moves to AXWindowIdentity so ScreenCaptureService and the walker share one declaration, and AXAttributeCasting gains typed string, bool, number, point, size and AX error readers. Tests run against fixture trees shaped like AppKit, Catalyst, Electron and System Settings windows in English, German, French and Finnish, plus one check against this Mac's installed Settings extensions when present; none touches a live app.
…observe approval Declare ui_snapshot in the manifest as a read-only, long-wait Swift tool for pi-mono and the stdio adapters. Its description and guidelines carry all the model guidance, including to search the full result before saying something is not in the window: prefer a dedicated tool, name the app by bundle_id, treat everything quoted in the result as the app's data and never instructions, how to read the header and element lines, what sparse and an incomplete result mean, and never retry a refusal. A new sensitive bundle, desktop.automation.observe, carries it, separate from desktop.automation.act so that approving a read of an app never approves acting in it. ui_snapshot joins the enforced approval set beside capture_screen, which satisfies the stack's drift test (every relay-callable tool in a sensitive bundle is gated) and its long model-side wait, so the kernel holds its input to the manifest schema and parks it on the card; its grant resource is the trimmed, lowercased bundle id, observe grants must name that resource, and the broker reads observe grants. A malformed bundle id, Omi's own bundles and the apps on the shared refused list are hard denies with no card and no dispatch row, and a test pins that every refused entry is lowercase and well formed. Its result contract asks the kernel projector to open a truncated projection with what was left out and to point at search_tool_output (omissionNoticeFirst, a generic contract option no other tool sets). The control-tool bundle enums accept the new bundle, the Swift tool surfaces and fixture are regenerated, and ChatToolExecutor runs the generated uiSnapshot case behind the owner bind through ChatToolExecutor+UISnapshot.swift, which validates the arguments, asks for Accessibility through request_permission when it is missing, resolves the approved app, applies the refusal floor, re-checks just before and after the read that the pid still belongs to the approved bundle, logs which assistive switch was used, whether it took and whether the read was repeated, carries the calling task's cancellation into the detached read, and returns only facts; ui_snapshot arguments are logged by shape only. Line-Count-Exception: desktop/macos/Desktop/Sources/Providers/ChatToolExecutor.swift | 3961 -> 3971 | the ui_snapshot dispatch case with its owner bind and two-line comment; the executor lives in ChatToolExecutor+UISnapshot.swift
The kernel's ui_snapshot card reads "Read an app window" and asks "Let Omi read the window of <bundle id>?", previews only window_title and window_id (never the pid or the element limit), and offers Allow for This Chat as covering any window of this app for the next hour. The Swift card resolves the bundle id through Omi's own LaunchServices lookup once, when the frame arrives, and asks "Let Omi read the window of TextEdit?" with an App row of "TextEdit · com.apple.TextEdit", so a look-alike name cannot hide which app is read; the app's own name is made display safe and kept to letters, digits, spaces and plain punctuation, so the · separator, its look-alikes and invisible format characters become spaces, and it is bounded to 40 characters so it cannot forge the bundle id beside it. An app it cannot resolve keeps the kernel's words and the bare bundle id, other tools' resources are never looked up as apps, and the window title passes through the card's display-safe rendering. The note under the answers now names the answer it is about, in one pattern for every tool that offers it ("Allow for This Chat also allows reading any window of TextEdit for the next hour.", and the same "Allow for This Chat also allows … for the next hour." for the message, script and read tools; capture_screen offers Allow Once only and shows no note), so the hour can no longer read as part of Allow Once. Window fields get plain labels. The card reuses the existing Ink, OmiType, OmiSpacing and OmiButtonStyle tokens and adds no component.
The device tool surface doc gains the ui_snapshot row and a Reading app windows section: the per-app desktop.automation.observe grant and why it is separate from act, why bundle_id is required, the shared refusal floor and where each half checks it, what the floor does not cover (credential screens inside an approved app), the System Settings allow-list that refuses sensitive and unidentified panes and Privacy & Security pages by title on every read in every installed language, the long-text, secure-field and URL-token rules, the facts-only result with its content-first order and the kernel's omission notice, the walk's bounds, and the Electron and Chromium assistive switch as the tool's only write, set only after the person approved the app. The macOS agent guide's on-device bullet and the approval UX design doc's list of parked tools name the new tool, a manual-tier e2e flow (ui-snapshot-approval) records the packaged-app check, including a non-English System Settings pass and a Privacy deep link, and covers the new Swift files, and a changelog fragment describes the change for people.
…screen capture_screen grabbed the whole display, so a screenshot the person approved could still show Terminal, a password manager, Keychain, a sign-in prompt, Omi itself or an app they had excluded from capture, the same apps ui_snapshot refuses to read. CaptureScreenExclusion now works out those apps from the shared generated safety floor plus the person's capture exclusions and Omi's own process, and capture_screen takes its image through a ScreenCaptureKit filter that removes their windows, so one that is frontmost or fills the screen is simply absent. When the filter cannot be used, the plain display capture is taken only if none of those apps has a window on screen, and otherwise nothing is captured. The selection, the filter's application list and the fallback rule are tested without a display; the device tool surface doc, the manual flow and a changelog fragment describe it. The ChatToolExecutor call site swaps the synchronous capture for the async one in the same four lines, so the file does not grow.
Nine in-process captures of the main chat from the named bundle built from f6951e094a, signed in to a fresh account: the TextEdit card pending, denied, and allowed for the chat with the read quoting the scratch document, a third read running with no card, a Notes card asking again in the same chat, the Terminal refusal with no card, the System Settings card pending and allowed for the chat, and the capture_screen card offering only Allow Once and Deny, with evidence.json recording agent capture through the automation bridge.
sauranshbhardwaj
force-pushed
the
feat/mac-ui-snapshot
branch
from
October 11, 2026 00:40
169d868 to
cd8227a
Compare
Collaborator
|
Reviewed the ui_snapshot work on head cd8227a. This is a careful, defensively-designed implementation; notes from the pass, then the items a maintainer should weigh. What checked out:
For a maintainer to weigh:
Agent-guidance docs: this PR updates .github/agent-docs/agent-grants-approval-ux.mdx (its status flips from "design-only, do not treat as implementation scope" to describing the now-live approval dispatch), desktop/macos/AGENTS.md (observe joins the dispatch-or-grant rule), desktop/macos/docs/device-tool-surface.md, and the generated tool bullets now tell the model to call sensitive tools directly because the app asks the person to approve. That guidance matches the implemented card behavior as verified in the kernel and relay, but since these files steer AI coding/review agents in this repo, maintainer confirmation of the copy is warranted. The committed UI-evidence PNGs under .agent-artifacts/ui-evidence/ follow the existing convention; I could not verify image contents. Leaving for human maintainer review: this opens a privacy-sensitive surface (Accessibility reads of other apps' windows, with content sent to the model) that needs maintainer sign-off, and it is PR 1 of a stacked sequence (#20973 -> #20975 -> #20997 -> #20998 -> this), so merge-order decisions belong to a human. by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
Git-on-my-level
added
the
positive-signal
… own words WhatsApp starts its display name and nearly every label with U+200E LEFT-TO-RIGHT MARK, so the approval card showed the app as ⟨U+200E⟩WhatsApp. One InvisibleFormatMarks type now names the harmless marks (U+200E, U+200F, U+061C, U+200B, U+2060-U+2064, U+FEFF, the soft hyphen, U+180E) and the bidirectional controls apart; the card's app name drops the harmless ones and still shows an override or isolate as its code point, and snapshot lines drop both as before.
… controls and show zero-width marks in the card's app name
U+200F and U+061C are strong right-to-left characters, so in a left-to-right line they move the digits and spaces beside them ("Acct 1234 <RLM>5678" can read as "5678 1234"); InvisibleFormatMarks now counts them with the embeddings, overrides and isolates instead of dropping them as harmless. The card's App row drops only U+200E, the mark WhatsApp starts its own name with, and again shows U+200B, U+2060-U+2064, U+FEFF and the rest as code points, so a look-alike app name cannot hide one. Snapshot lines drop the same set of characters as before.
…d app with ui_snapshot In a live test "Send the draft in WhatsApp" made the model list Messages conversations and read Messages history first. The ui_snapshot guideline now says to act in the app the person names, that send_message, read_message_history and list_message_chats are for Apple Messages only, and that WhatsApp and other Catalyst apps have no dedicated tool; each Messages tool's own guidelines say the same. The docs and the ui_snapshot flow gain a step for it.
…ep one leading direction mark as a label's own Safari answered ui_snapshot with app_not_responding while in the background because WebKit builds a background window's Accessibility tree lazily, and WhatsApp's first read failed the same way right after it was reopened. When the walk stops with fewer than five elements because the app did not answer, it is walked once more after 0.5 s; when the whole read fails with app_not_responding, it is read again after 0.5 s and then 1 s; all within the same 3 s deadline. A right-to-left locale starts its labels with U+200F the way an English one starts them with U+200E, so InvisibleFormatMarks treats one U+200E or U+200F at the very start of a label as the label's own (strippingLeadingDirectionMark, labelContainsBidiControl) and the card's app name drops it; any other right-to-left or Arabic letter mark, embedding, override or isolate is still a bidi control and shows as its code point.
Contributor
Author
|
Pushed four small fixes from a packaged-app test with WhatsApp: hide the invisible direction marks apps put on their own labels (WhatsApp prefixes almost every label with U+200E), refuse the bidi marks that could disguise a label, keep the Messages tools to Apple Messages, and retry a read once when an app is slow to answer. Details are in the updated description. |
Collaborator
|
Follow-up review of the four commits since my earlier pass (a1640d6, 4c348e9, 265095a, 768c15e) — the feedback from that pass is addressed, and the additions hold the same standard as the rest of the work. What the new commits get right:
Still open for a maintainer (unchanged from the stack, now concrete in this PR):
Note for maintainers: this PR also changes agent-facing guidance, not just code — the tool descriptions the shipping model reads now tell it to call capture_screen/send_message directly (the app asks for approval itself) and to use ui_snapshot when the person names another app, and desktop/macos/AGENTS.md marks desktop.automation.observe as dispatch-gated. The guidance is accurate against the implementation and test-asserted, but it redirects live model behavior on a privacy-sensitive surface, which is another reason the product sign-off above should be explicit. No action requested from the contributor — leaving this for human maintainer review on the product defaults and the refusal-floor composition. by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Invariants carried from the stack (their changes show in this diff until they merge; see each PR's body): INV-AGENT-* and INV-CHAT-1 for the kernel parking, relay and tool descriptions in
desktop/macos/agent/src/runtime/**(#20973, #20975, #20998); INV-AUTH-1 (product/invariants/auth-session.md), INV-CHAT-2 (product/invariants/chat-scroll-placement.md), INV-CHAT-1, INV-UI-1 and INV-UI-2 for the approval card, its post-handshake clearing, the card list inChatMessagesView.swiftand thewaitingApprovalstatus inChatProvider.swift(#20997).Line-count exceptions carried from #20997 (this branch is stacked on it, so its file growth shows in this diff until it merges):
Line-Count-Exception: desktop/macos/Desktop/Sources/Chat/AgentRuntimeProcess.swift | 4373 -> 4384 | two kinds in the closed message switch, the post-handshake clear and conditional capability send, a two-line routing case, the owner-revoke case that clears cards, and one clear on daemon exit; the statics, wire message, routing body and clearing live in AgentRuntimeProcess+ToolApprovals.swift, and messageOwnerIsCurrentlyAuthorized drops private so that extension can call it
Line-Count-Exception: desktop/macos/Desktop/Sources/DesktopAutomationBridge.swift | 5172 -> 5173 | one registerToolApprovalActions() call in the builtin registration list; the actions live in Automation/DesktopAutomationToolApprovalActions.swift
Line-Count-Exception: desktop/macos/Desktop/Sources/FloatingControlBar/FloatingControlBarView.swift | 3018 -> 3020 | the approvalSurface argument at the one AIResponseView call site
Line-Count-Exception: desktop/macos/Desktop/Sources/MainWindow/Components/ChatBubble.swift | 2162 -> 2172 | the "Waiting for your approval" caption under the tool row and its accessibility value; the status itself and the icon live in ChatProvider's enum and ToolActivityTimelineLayout
Line-Count-Exception: desktop/macos/Desktop/Sources/Providers/ChatProvider.swift | 6824 -> 6834 | the waitingApproval case of ToolCallStatus with its doc line, its detector mapping, the turn's session id captured for the stall clocks, and the tick-loop lines that ask the approval store about that session; the lookup lives in ChatProvider+ToolApprovalWait.swift
What changed and why
Phase 1 of #21096 adds
ui_snapshot, a read-only tool that lets the Mac agent read one window of another app as named elements: "a button called Save", "a text area called Document, value 'hello'". It is approved per app on the existing approval card. It never clicks, types, raises a window or moves the cursor.This PR is stacked on
feat/device-tool-descriptions(#20998, now at 703fd95), the top of the approval stack #20973 → #20975 → #20997 → #20998. The stack beneath also gatescapture_screenbehind the same card, Allow Once only (no chat grant can cover a screenshot), and holds every relay-callable tool in a sensitive bundle to the approval set with a drift test;ui_snapshotsits in that set beside it. Theui_snapshotchange is the last ten commits: four for the tool, a fifth that makescapture_screenleave out the same refused apps, the UI evidence for the card, a seventh that drops the harmless invisible marks apps put around their own words (found in phase 3's packaged-app test, below), and two from phase 3's second review and live retest: the right-to-left and Arabic letter marks now count as bidirectional controls and the card's app name shows zero-width marks again, and the Messages tools are kept to Apple Messages so a named app like WhatsApp is read withui_snapshot; and one from the third review and the live retest: a read the app did not answer is tried again (below), and one direction mark at the very start of a label is the label's own.This is deliberately not "let Omi control your Mac" (#12465 was closed over consent breadth). It is one typed read tool behind a kernel capability bundle, granted per app and per chat. Apps that hold credentials, show them or run code are refused before any card appears.
Kernel
desktop.automation.observe, carries the tool. It is separate fromdesktop.automation.act, so a grant to read an app never counts as a grant to act in it.ui_snapshotjoins the enforced approval set. Its input is held to the manifest schema and it parks on the card.bundle_id. Allow for This Chat covers that one app for 1 h; another app asks again. Observe grants must name an app.com.omi.*build;UserNotificationCenteralerts and notification banners (they carry one-time codes);com.apple.SafariPlatformSupport.Helper,com.apple.AuthenticationServices.Helper);One refused list
agent/src/runtime/ui-automation-safety-floor.tsdeclares the refused bundle ids and the System Settings pane lists once.generate-tool-surfaces.mjswrites them intoGeneratedUIAutomationSafetyFloor.swift, so the kernel and Swift cannot drift. A vitest test checks that every entry is lowercase and well formed.Swift
ChatToolExecutor+UISnapshot.swiftruns the generateduiSnapshotcase behind the owner bind. It:bundle_idas untrusted text;request_permissionwhen it is missing;pidonly picks between copies and must belong to the bundle);.app): app extensions, XPC view services and bare executables, an app that reports no name (capture exclusion is keyed by name, so it cannot be checked), capture-excluded apps, and a frontmost app with secure input on;WindowSnapshotWalkerreads throughAccessibilityElementSource, which has no press, focus, raise or type method.System Settings: an allow-list
System Settings is not refused as a whole. On every read, and for every open Settings window, Swift reads the selected sidebar row and the window title:
Localizabletables.refused_settings_pane_unknown: no selection, an unknown row, a search result or an active sidebar search, a sheet over the window, a list in the window with no position (the sidebar cannot be told from a content table), or a second Settings window that cannot be identified.What is returned
a:<AXIdentifier>, elsen:<role>:"<label>", else ap:child path);stale_ref.https:,file:,mailto:,tel:, an app's own) or none (an address bar showsexample.com/reset?token=…), loses its query string and fragment, and long opaque path segments become….app_not_respondingwhile in the background (WebKit builds a background window's tree lazily), and WhatsApp's first read failed the same way right after it was reopened. When the walk stops with fewer than five elements because the app did not answer, it is walked once more after 0.5 s; when the whole read fails withapp_not_responding, it is read again after 0.5 s and then 1 s. All of it stays within the read's 3 s.AXLandmarkMainregion, is moved before sidebars, toolbars and chrome (order=content_firstin the header), so a chat's messages are in view before its channel list. References and paths are unchanged.search_tool_output. This is a generic result-contract option (omissionNoticeFirst) that onlyui_snapshotsets, and the guidelines tell the model to search before saying something is not in the window.sparse=trueand a reason when there are fewer than five elements, none actionable, or a depth-cut walk that found almost nothing beyond window chrome. With the Electron or Chromium switch on, a sparse first pass is read once more. The log records the switch, whether it took, and the retry (assistive_tree=electron enabled=… retried=…), shape only.search_tool_output.Electron and Chromium
These apps expose their elements only to an assistive client. After the person has approved the app, and only then,
ui_snapshotsets:AXManualAccessibilityon an Electron app;AXEnhancedUserInterfaceon a Chromium app.The switch is the only write the tool makes, and it does not outlast the read. On every exit (success, failure, timeout, cancellation) it is turned back off, but only if Omi turned it on and it is still on, so another assistive tool's switch is left alone and a read leaves no lasting change in the other app. A sparse first pass after turning it on is read once more. The app is recognised from its own bundle layout.
Card
The card reads "Read an app window" and asks "Let Omi read the window of TextEdit?", with an App row "TextEdit · com.apple.TextEdit".
·, look-alike separators (•,∙,⋅,‧,・) and invisible format characters become spaces, and it is bounded to 40 characters. U+200E LEFT-TO-RIGHT MARK, which an app may start its own name with, and one U+200F at the very start of the name (a right-to-left locale's labels start with it, as an English one's start with U+200E), are dropped first (WhatsApp's display name does, and the card showed it as⟨U+200E⟩WhatsApp); every other invisible or bidirectional character (a zero-width space, a word joiner, a byte-order mark, an override, an isolate, a right-to-left mark) is still shown as its code point, so a look-alike name cannot hide one.InvisibleFormatMarks.labelContainsBidiControlapplies the same rule to labels for phase 3's card: one leading direction mark is the label's own, any other right-to-left mark is a bidirectional control. Snapshot lines drop the harmless marks (U+200E, U+200B, U+2060–U+2064, U+FEFF, the soft hyphen, U+180E) and the bidirectional controls, as before, now through one sharedInvisibleFormatMarkstype that phases 2 and 3 use too. That type counts the right-to-left mark (U+200F) and the Arabic letter mark (U+061C) as bidirectional controls, not harmless marks: each is a strong right-to-left character that moves the digits and spaces beside it ("Acct 1234 \u200F5678" can read "5678 1234"), so phase 3's commit card offers only Deny for text holding one.capture_screenoffers Allow Once only and shows no note.capture_screen leaves out the same apps
The fifth commit (
fix(desktop): …) makescapture_screenleave out Omi itself, the apps on the same generated safety floor and the person's capture-excluded apps. It takes the image through a ScreenCaptureKit filter that removes those apps' windows, so a terminal that is frontmost or fills the screen is simply not in the picture. If the filter cannot be used, the plain display capture is taken only when none of those apps has a window on screen; otherwise nothing is captured. TheChatToolExecutor.swiftcall site changes in place, so the file does not grow.What the floor does not cover
The floor refuses apps, not screens. Credential UI hosted inside an app the person approved is read like any other content, apart from secure fields. Examples are a sign-in sheet, or a browser's saved-password page once it is revealed. The per-app card is the control there.
Open questions for maintainers
The issue's Q3 and Q4 bear directly on this PR.
bundle_idacceptable, given the kernel cannot scope a pid?Known interactions (not changed by this PR):
x-apple.systempreferences:…) can open with no sidebar row selected. The allow-list then refuses it asrefused_settings_pane_unknownuntil a sidebar row is selected. This is deliberate: a title alone can't tell the top-level Accessibility pane from Privacy & Security → Accessibility.ScreenContextFallbackPolicy). That path predates this stack and is separate fromcapture_screen, which now always asks first.Product invariants affected
INV-AGENT-* (
product/invariants/agent-control-plane.md) is named becausedesktop/macos/agent/src/runtime/**changes.ui_snapshotcall through the existing approval set. Swift runs it only asauthorized_tool_execution, behind the owner bind.read_only/safe_retry, so a restart has no write to replay.device-tool-policy.test.ts:ui-automation-safety-floor.test.ts:run-tool-capability.test.ts:desktop.automation.observeand the lowercased id;invalid_tool_input;timeoutClass: "long".control-tools.test.ts: allow for this chat mints an observe grant for exactly that app.desktop-tool-policy.test.ts: the card copy.INV-CHAT-1 (
product/invariants/chat-continuity.md) is named becausedesktop/macos/agent/src/runtime/**anddesktop/macos/Desktop/Sources/Chat/**match its globs. It is unaffected: nothing writes to the transcript, and the approval store change only resolves an app name when a frame arrives.INV-INT-1 (
product/invariants/integrations.md) is named deliberately, though no glob matches.CloudConnectorFormAutomationis untouched.send_message,read_message_historyandlist_message_chatsare for Apple Messages only (their own guidelines say so too), and WhatsApp and other Catalyst apps have no dedicated tool, so they are read withui_snapshot. In phase 3's retest, "Send the draft in WhatsApp" read Messages history first;omi-tool-manifest.test.tsnow holds this wording.INV-UI-1 and INV-UI-2 (
product/invariants/brand-ui.md,product/invariants/desktop-ux-contract.md): the change is card copy and one target row only. It uses the existingInk,OmiType,OmiSpacingandOmiButtonStyletokens and no new component.check_desktop_ux_contract.pyreports no hand-rolled UX increase.INV-BETA-1, in passing: every
com.omi.*bundle is refused, so Beta never reads stable or a dev build.How it was verified
All commands were run from
desktop/macosat the branch head, on macOS 27 with Xcode 27.xcrun swift build -c debug --package-path Desktop: clean.xcrun swift test --package-path Desktop --filter '<suites>': 278 tests, 0 failures (at cd8227a). After the invisible-marks commit, the whole suite at this head,xcrun swift test --package-path Desktop --skip SubscriptionEntitlementServiceTests(that suite posts a global sign-out notification and traps the test process in a whole-suite run, here and at cd8227a; alone it passes): 7,961 tests, 10 skipped, 3 failures, none from this change:RecentScreensMenuContentTests.testDayOldFrameShowsACalendarDateNotAClockTime(this Mac's date locale),DesktopAutomationWindowPresentationTests.testQuietPresentationAlsoParksNonactivatingOverlayPanels(display-dependent) andMeetingScreenEvidencePassTests.testRetryAfterFinalizeJoinsTheRunTheDeadlineLeftInFlight(timing-dependent) fail the same way at cd8227a, built and run in the same session. After the second review's two commits, the whole suite at this head (265095a), with the same skip: 7,963 tests, 10 skipped, 2 failures, the first two above (the meeting test passed this time). After the third review's commit (this head, 768c15e), the focused suites pass, 113 tests with 0 failures (WindowSnapshotWalkerTests24, two new for the read again after the app did not answer, a walk that stopped and a read that failed, with a hung app read at most twice and an app reopened a moment late caught by the second wait;ApprovalCardTests34, one new for a Hebrew and an Arabic label starting with U+200F;WindowSnapshotSafetyTests,ChatToolExecutorUISnapshotTests,SettingsPaneGuardTests,UIAutomationTargetPolicyTests,CaptureScreen*), and the whole suite ran at the top of the stack, phase 3, which holds this branch's Swift unchanged.WindowSnapshotWalkerTests22 (one for the harmless marks)WindowSnapshotSafetyTests18CaptureScreenExclusionTests3UIAutomationTargetPolicyTests10SettingsPaneGuardTests9 (including one check against this Mac's installed Settings extensions)ChatToolExecutorUISnapshotTests11ApprovalCardTests33 (9 new for ui_snapshot, three of them for the marks in an app's name and the right-to-left and Arabic letter marks, plus the stack's Allow-Once-only capture_screen card test and its Clear Chat seam test).CaptureScreenToolTests4,ScreenCaptureRetentionTests2,AXAttributeCastingTests4,AccessibilityProcessBoundaryTests4,AuthorizedToolExecutionTests15,ToolSurfaceLiteralsTests1,ChatDiscoverabilityTests25,ToolCallStatusTests31,ChatStallWatchdogTests10,AgentRuntimeProcessTests74,ChatToolExecutorPolicyTests5,ScreenCaptureTargetPolicyTests2./System/Library/ExtensionKit/Extensions:./scripts/test-tool-surfaces.shat this head: agent build,generate-tool-surfaces.mjs --check("all outputs match"),npx vitest run74 files / 981 tests passed (one new for the Messages-only routing wording),pi-mono-extension143/143; the same at 768c15e.check_desktop_ux_contract.py: no hand-rolled UX increase.scripts/swiftlint-wrapper.sh lint: 0 violations.check_desktop_test_quality.py,check_agent_doc_references.py,check_agents_md_lean.py,desktop-flow-lint.py,desktop-changelog.py validate.e2e/flows/ui-snapshot-approval.yaml.scripts/pr-preflight --base feat/device-tool-descriptions-> "PR preflight passed: 35 checks" (64 files, base 703fd95; 65 files at this head). Against main with this body:scripts/pr-preflight --base upstream/main(d3208a0) -> "PR preflight passed: 40 checks" (120 files, the stack included). At a1640d6, and again at 265095a and at this head (768c15e), the pre-push gate against upstream with this body:OMI_PR_BODY_FILE=<this body> PRE_PUSH_BASE_REMOTE=upstream scripts/pre-push-singleflight upstream-> "PR preflight passed: 40 checks" (121 files, merge base d3208a0) and "Bounded pre-push gate passed"; against the stack parent with the stack body at this head, 35 checks (65 files).Packaged-app manual pass (the person's live run on the packaged test build, 2026-10-09/10, production servers, macOS 27). No app came to the front and the cursor never moved on any read.
depth_limit); after wrapper groups stopped costing depth, 234 nodes with the channels; after content-first ordering the model quoted recent messagesrefused_settings_pane); Full Disk Access via search (refused_settings_pane_unknown)capture_screen: leaving out 12 app(s))The Electron switch restore (turned back off after every read) came after this pass and is covered by unit tests.
Not yet verified:
Tests
Core path.
WindowSnapshotWalkerTestsandWindowSnapshotSafetyTestsrun against fixture trees shaped like real apps:AXDescription;They check:
stop_reason, and a huge group no longer ends the walk;order=content_first, and a short document keeps document order;mailto:,file:,tel:) or none;Refusals.
UIAutomationTargetPolicyTestscovers each refusal: own pid,com.omi.*, each refused category, an app extension, an XPC service, a bare executable, an unnamed app, a capture-excluded name, and secure input.SettingsPaneGuardTestscovers:capture_screen.
CaptureScreenExclusionTestscovers which apps are left out (Omi,com.omi.*, refused ids, capture-excluded names), the filter's application list, and the rule that the plain capture stands in only when no left-out app is on screen.Executor.
ChatToolExecutorUISnapshotTestscovers:No test touches a real app.
UI evidence
Captured in the packaged named bundle built from f6951e094a, signed in to a fresh account with an empty chat, through the bridge's in-process window capture (
capture_main_window_png) while driving the cards withomi-ctl;captured_by: agent,source: visual-auditin.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/evidence.json. The TextEdit window was a scratch document containing onlyhello snapshot test; Notes was never opened and its card was denied.These captures predate a fix in #20997 that moved the card's field labels, covers note and pending status line, and the tool row's "Waiting for your approval" caption, from
Ink.tertiary(under WCAG AA on the glass panel) toInk.secondary; those runs now draw one step darker, and the images have not been recaptured..agent-artifacts/ui-evidence/desktop-ui-snapshot-card/001-desktop-macos-ui-snapshot-textedit-pending.png.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/002-desktop-macos-ui-snapshot-textedit-denied.pnghello snapshot test:.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/003-desktop-macos-ui-snapshot-textedit-allowed-for-chat.png.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/004-desktop-macos-ui-snapshot-textedit-no-card-in-chat.png.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/005-desktop-macos-ui-snapshot-notes-pending.png.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/006-desktop-macos-ui-snapshot-terminal-refused.png.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/007-desktop-macos-ui-snapshot-settings-pending.png.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/008-desktop-macos-ui-snapshot-settings-allowed-for-chat.pngcapture_screencard with only Allow Once and Deny:.agent-artifacts/ui-evidence/desktop-ui-snapshot-card/009-desktop-macos-capture-screen-pending.pngNot captured: the Privacy & Security, Sharing and Login Items refusals. Resolved cards stay at the live edge of the thread, so by then the card list filled the window and pushed each reply out of view.
Failure class (fixes)
Failure-Class: none
Line-Count-Exception: desktop/macos/Desktop/Sources/Providers/ChatToolExecutor.swift | 3961 -> 3971 | the ui_snapshot dispatch case with its owner bind and two-line comment; the executor lives in ChatToolExecutor+UISnapshot.swift