Repository navigation
feat(desktop): add propose_message and propose_email draft-and-you-send tools - #21166
sauranshbhardwaj wants to merge 4 commits into
Conversation
…nd tools The new tools open a compose window through NSSharingService, filled in with recipients, subject and body, and stop there: the person reviews the draft and presses Send, and Omi has no path that sends it. On current macOS a message draft is placed in the Messages app's own composer and the handoff is reported at once, so Omi cannot see whether it is sent or discarded (a send or cancel report arrives only if the system shows a sheet); email drafts go to whatever app handles mailto links, so results name that app (resolved through NSWorkspace, with app and app_is_browser fields) and, when it is a browser, say the draft opened on its web mail site and that Omi cannot see whether it was sent; nothing assumes Mail. They share the iOS propose_message contract (name, to plus text, does-not-send, cancelled) but never report sent, because the email handoff is reported before Send. propose_message is the default for any request to text or message someone, send_message is only for when the person asks Omi to send for them, and the guidance of send_message, propose_* and create_desktop_dispatch says a message or email is never pre-filed as a dispatch. A new desktop.compose.propose bundle is a platform-confirmed write, so the policy allows it without an Omi approval card; any other write requested beside it keeps its own approval, and the tools are coordinator-only so a background worker cannot raise a window. Because the compose window has no Omi framing, send_message's recipient rule is enforced in code: every recipient must appear in words the person typed, pasted or dictated in this turn or their one previous turn in the same chat (a model-written question-card answer never counts), in a search_contacts result from the same run and owner, or be the account's own address, otherwise recipient_needs_confirmation opens nothing; phone numbers match only as whole numbers and never from dates or reference numbers, a bare name returns recipient_needs_lookup, and addresses use a narrow ASCII form. Return sends from the Messages composer, so a draft is handed off only after three seconds without a key press, with the last idle check immediately before the handoff; the same draft (channel, recipients, text digest) is not handed off again within two minutes for the same owner, which returns draft_already_open, and the guidance tells the model that after a draft the person presses Send themselves, so "send it" never re-opens the draft or calls send_message or create_desktop_dispatch, and send_message itself refuses, before any send, a recipient whose propose_message draft the same owner handed to Messages in the last ten minutes, returning draft_in_messages; each request opens at most one email draft and one message draft and an owner at most three a minute, an open Messages sheet blocks the next draft for up to five minutes, sign-out releases every limit, and a stopped call opens nothing. Results carry sent_by_omi=false and a status of handed_to_email_app, handed_to_messages, cancelled, open or not_opened. Draft content is redacted from the argument log.
…2e probe and document the contract The hermetic device_tools_probe now checks that a compose proposal refuses a bare name, an empty draft and an address the person never gave before any window could open, and the device-tool-surface flow asserts all three. The device tool surface doc gains the propose_message and propose_email rows and explains the no-card policy, that a message draft is placed in the Messages app's composer where Return sends and the send cannot be observed, which app an email draft opens in (the mailto handler, possibly a browser's web mail) and how results name it, the duplicate-draft guard, the send-it guidance and the send_message check that refuses a send to someone whose draft is waiting in Messages, the routing rule that keeps messages off create_desktop_dispatch, the recipient grounding rule and its limits (question-card answers excluded, pasted text and the previous turn counted, whole-number phone matching, no spoken addresses), the three-second typing guard and why it is load-bearing, the per-channel per-request and per-minute limits, the never-sent statuses, what is shared with and different from iOS, and that an email app may autosave an open draft to the server's Drafts mailbox. It also corrects the stale tool count. Adds the user-facing changelog fragment.
…ools Adds three screenshots captured by a person on the packaged build of this branch (production servers, macOS 27, placeholder recipients only): the email draft filled in and opened by the default mailto handler (Google Chrome, Outlook on the web), the message draft placed in the Messages app's own composer, and Omi's reply when the typing guard declined to hand off a draft. The sibling evidence.json records each image under the shared UI evidence contract.
|
Thanks @sauranshbhardwaj - this is a carefully built change and I went through the new code in detail. The guard stacking stands out. Verified highlights
Two decisions I'd like a maintainer to make
CI on the relevant lanes (Desktop Swift Build & Tests, Agent Runtime, Hygiene) is green at this head. Leaving for human maintainer review: product-direction and approval-policy sign-off needed for the new compose tool surface. by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
|
Thanks for the close read. On the two decisions:
On the trade-off you named: yes, a |
|
Thanks @sauranshbhardwaj — this is a careful, unusually well-guarded feature, and the e2e probe extension plus changelog fragment are appreciated. Detailed review below; the product/approval-model decision itself is left for a human maintainer. What this does well, verified against the diff:
Judgment calls that need a maintainer, and why this is not a rubber stamp:
Smaller notes (non-blocking):
Checks: Desktop Swift Build & Tests, Agent Runtime, PR Metadata Preflight, and Mobile Release Eligibility pass at this head; the rest are skipped by path filter as expected. Because this changes the approval model for a security-sensitive surface (messaging/email compose on macOS), it needs maintainer sign-off on the allow-without-dispatch decision and the documented residual risks before merge; keeping security-review and needs-maintainer-review. This feedback is AI-generated on behalf of the Omi maintainers. by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
…fictional number in the propose probe The send_message capability doc repeated the explicitly-asked rule that its prompt guidelines already carry, so the generated capability bullets said it twice in different words; the capability-doc copy is dropped and the regenerated surfaces carry the guideline alone. The propose body-validation probe now uses the fictional +15555550100, which still has to be a valid handle so the empty draft is what gets refused.
|
Thanks, both fixed in one small commit. |
|
Reviewed the full diff at bc1f15c. This is a well-built feature: the draft-and-you-send contract (nothing leaves the Mac without a keystroke in Apple's own compose window) is enforced in code, not just in prompts, and the details hold up on close reading. Thanks also for the docs and test coverage. How the no-send property holds, file by file:
Two non-blocking observations:
The model-guidance changes (omi-tool-manifest.ts, control-tool-manifest.ts, GeneratedToolCapabilities.swift, and the regenerated fixtures) read as safe and accurate: propose_* becomes the default for "text/message/email someone", and the send_message/create_desktop_dispatch guidance steers away from the dead-end dispatch routing described in the PR body. The guidance matches the enforced Swift behavior, which is the right direction. Not approving formally. This adds a no-approval-card write surface ( by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
|
Thank you for this exceptionally careful PR — I did a full static review of the new Swift services and the agent-side tool policy, and the quality bar here is genuinely high. Summarizing what I verified and the small number of product decisions left for a human maintainer. What I verified in the code:
What needs a human maintainer's call before merge:
No security issues found — no exfiltration, no hidden send path, no prompt-injection-reachable bypass of the grounding check that I could construct. Keeping by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with |
What changed and why
Today "email Sam the deck" on the Mac has no proper path.
list_mail_messagesreads headers only,send_messagecovers iMessage only and needs an approval card, and the fallback isrun_applescript. iOS already haspropose_message: it opens the system compose sheet, filled in, and the person taps Send.This PR adds two Mac tools in that spirit:
propose_messageplaces a draft in the Messages app with the recipients and text filled in.propose_emailopens a new email draft in the Mac's default email app with the recipients, subject and body filled in.The person reviews the draft and presses Send in that window. Omi never sends: the tools use
NSSharingService(.composeMessage,.composeEmail) and stop there. There is no AppleScript, no Mail or Messages automation, and no TCC prompt. Nothing is sent until the person presses Send, though Mail may autosave an open draft to the account's Drafts mailbox on the mail server.Which app.
.composeEmailhands the draft to whatever app handlesmailto:links. That may be Mail, another mail app such as Outlook, or a browser, which opens its web mail site. The packaged-app test confirmed this: the tester's default was Google Chrome, and the draft opened in Outlook on the web.NSWorkspace.urlForApplication(toOpen: mailto:), inappandapp_is_browser.Routing. In the same live test, "text me at '…'" made the model file a
create_desktop_dispatchapproval forsend_message. Onmainthat dispatch has no card to render it, so the request dead-ended. The guidance is now explicit:propose_messageis the default for any request to text, message or iMessage someone;send_messageis only for when the person explicitly asks Omi to send it for them;create_desktop_dispatch,send_messageandpropose_*all say a message or email is never pre-filed as a dispatch.send_message's behaviour and the dispatch tool itself are unchanged.This is piece 1 of the follow-through proposal in #21162. That issue is titled "…behind the approval card" and has no maintainer reply yet. This piece deliberately has no Omi approval card (see Policy below), and that is a decision for maintainers to confirm.
The PR goes straight to
main. It does not depend on the approval-card stack (#20973 → #20975 → #20997 → #20998).What is shared with iOS, and what differs
propose_message;to(an array) andtext;sentorcancelledin practice. iOS getssentorcancelledfrom its compose sheet. On the Mac (verified on macOS 27),.composeMessageshows no sheet: it places the draft in the Messages app's own composer and reports the handoff within about half a second. Email apps do the same. So Omi cannot see whether the person sends or discards the draft;propose_messagehas nosubject;propose_email.Results
Every result carries
sent_by_omi: falseand astatus:handed_to_email_app: the email app named inapptook the draft. macOS reports this before the person presses Send, so it never means sent.handed_to_messages: the draft is in Messages for the person to review and send. Whether they send or discard it cannot be observed.cancelledandopen(still on screen after the wait) are produced only if the system shows a sheet or an email app reports a cancel. The code still handles a sheet, but current macOS does not show one.openis also the answer when the turn is stopped after the handoff.not_opened: no window was opened. The reasons are:stopped: the call was stopped before the window opened;person_typing;draft_already_open: the same draft was handed off in the last two minutes;send_message, not a propose tool)draft_in_messages: a send to someone whose draft is waiting in Messages; nothing is sent;compose_already_open(only with a sheet);draft_limit_reached;compose_unavailable:canPerformfalse or "not configured";compose_failedis used for any other error from the share service and reports only the error code.Recipients come from the person, not from what the model read
The compose window has no "Omi wants to…" framing. So the recipient rule from
send_message("never send to a handle the user did not name or confirm") is enforced in code, not only in the prompt.Every recipient must come from one of:
originatingUserTextwith the clock preamble stripped, asisMemoryContentGroundedInUserRequestdoes. For push-to-talk, the run text is the provider's transcript of what the person said, so dictation needs no separate path;search_contactsresult from the same run and owner;If a recipient fails that check, the tool returns
recipient_needs_confirmation, lists the address, and opens nothing. This covers an address the model read on a page, in an email or on screen.Limits of the check:
preparedAnswer. Those turns carry aqri_turn key in the chat transcript and are left out, both as the current turn and as the previous one. The run text is matched to its chat turn by content; when nothing matches (task chat, voice), the run text stands alone.Validation:
recipient_needs_lookupwithnext_tool: search_contacts;?,&,%,/,#,..and lookalike Unicode domains, while punycode is allowed;No handoff while the person is typing (load-bearing)
Return sends from the Messages composer. The live test confirmed it. A keystroke that lands in the draft can send it before the person has reviewed it, so this guard is a safety control, not a nicety:
UserInputPresence;not_opened/person_typing.Residual risk: someone who starts typing again just after the handoff can still type into the composer.
"Send it" after a draft
In the live test, the person replied "send" after a draft. The model called
propose_messageagain, which put three copies in Messages. Two fixes:propose_message,propose_emailandsend_messagenow say that after a draft the person presses Send in Messages or their email app. On "send" or "send it", the model tells them to do that. It never opens the same draft again and never callssend_messageorcreate_desktop_dispatchfor it.send_message. In the next live run, the person typed "Send" after a draft, and the model calledsend_messagewith the same recipient and text. Onmainthe kernel's approval check refused it, so nothing was sent, but the guidance alone did not hold.send_messagenow checks before any send. If the same owner handed apropose_messagedraft to Messages in the last 10 minutes, and the send's recipient matches any recipient of that draft, it returnsnot_sent/draft_in_messagesand sends nothing.send_message(the separate approval stack), the card would still appear first. A kernel-side check would be needed to suppress the card.draft_already_openand is not handed off again. Identical means the same channel, the same recipients in any order or spelling, and the same text. Only a SHA-256 digest is kept. A cancel or failure report clears it, so the person can ask for the same draft again.Limits
All limits are per signed-in owner, and a sign-out releases them:
draft_already_open).compose_already_open). A sheet that never reports stops blocking after 5 minutes.Policy: no Omi approval card, and why
The tools get a new bundle,
desktop.compose.propose, separate fromdesktop.messaging.send. Selecting one never implies the other.It is a write bundle (risk tier medium, privacy
local_private), but it is platform-confirmed. Apple's window shows the person the exact recipients and text, and nothing is sent until they press Send. An Omi card in front of it would ask twice for the same effect. iOS takes the same position (requires_user_confirmation: "the compose sheet is the approval").evaluateDesktopToolPolicyreturnsallow, but narrowly:externalSendorpersistentGrantstill forces a dispatch;Remaining risk without a card: the window looks like the person's own draft. Three things keep that risk bounded:
The tools are also restricted in three ways:
never_auto_retry.Model guidance:
propose_messageunless the person explicitly asked Omi to send;propose_emailfor email;send_message,run_applescriptandsearch_contactsnow point at the propose tools.Privacy
The arguments are logged by shape only. The executor's own log line records counts and the outcome. Results echo the recipients, which the model supplied, and never the body.
Not in this PR, and questions
NSSharingServicehas no CC field. Amailto:path would lose the outcome report and open whichever mail app is the default.send_message(with its card) but not the saferpropose_*. Should pills get the propose tools? That would need a decision on raising windows from background work.ProactiveTaskExecute.systemPromptSuffixstill tells the agent to drive Messages and Mail through osascript.Product invariants
INV-AGENT-*(Authority). The changes are indesktop/macos/agent/src/runtime/**:desktop.compose.proposebundle and its platform-confirmed classification indesktop-tool-policy.ts;evaluate_desktop_tool_policyenums;Run/attempt capability checks are unchanged.
propose_*go through the same broker authorization and ledger preparation as every Swift tool, and the executor checks the owner before and after the window.run-tool-capability.test.tscovers the new authorization path.INV-CHAT-1. Nothing changes in the transcript. Proposal results are ordinary tool results in the kernel's invocation ledger. The only Swift state is in memory: the per-run contacts handles, the draft digests, the open-sheet slot and the limits. All of it is owner-scoped and released on sign-out, so it is not a second store.Testing
xcrun swift build -c debug --package-path Desktop: passes.ComposeProposalCoordinatorTests(20),ComposeProposalTests(19),ComposeRecipientGroundingTests(12),ComposeEmailAppTests(9),ComposeDuplicateDraftTests(4) andComposeSendGuardTests(6): 70 tests, all passing. They use a fakeComposeSurface, a scripted keyboard, an injected sleeper and an injected clock, so no window opens and no test waits on the wall clock. They cover:sent; every result having a status.open.executeSendMessagereturnsdraft_in_messagesbefore reaching Messages; any text to the same person in another spelling is refused; the normal path applies again after 10 minutes and for a different recipient or owner; an email draft or a cancelled draft does not block; every recipient of a group draft is covered.?cc=,%40,&bcc=,/,#, Cyrillic and fullwidth lookalike domains,.., a leading dot, a hyphen-led label and a one-letter TLD are all rejected; ordinary and punycode addresses are accepted; plus the earlier parsing cases.ComposeEmailAppTests(9): the Messages handoff result says the person presses Send in Messages and Omi cannot see the send, with no sheet claimed; a duplicate result points at the app and forbidssend_messageandcreate_desktop_dispatch. Also, with a fake handler for Mail, Microsoft Outlook and Google Chrome, plus an unknown handler:scripts/test-tool-surfaces.sh:generate-tool-surfaces --checkmatches, the full agent vitest suite passes (72 files, 909 tests), andpi-mono-extensionpasses (143). The vitest cases cover:externalSendforces a dispatch;propose_messageis the default for "text me at ''";send_messageno longer says "requires an approval dispatch" and says never to file one;create_desktop_dispatchsays never to use it for a text, message or email;propose_emailnever assumes Mail and reportshanded_to_email_app. The repo's only routing eval (eval-realtime-routing) covers the realtime voice lane, which does not offerpropose_*, so it has no case for this;send_messageorcreate_desktop_dispatchfor it;send_messagemust not send an open draft and namesdraft_in_messages;propose_messagemakes no claim about a sheet;recipient_needs_confirmation, the same-run precondition,coordinatorOnly;never_auto_retry, absent from a leaf capability.device-tool-surfaceflow now asserts that a compose proposal refuses a bare name, an empty draft and an address the person never gave, before any window could open.desktop-flow-lintpasses.xcrun swift test --skip SubscriptionEntitlementServiceTestsexecuted 7905 tests with 10 skipped. Four tests failed (5 assertions), all known and unrelated to this change:DesktopAutomationWindowPresentationTests(2 tests, display-dependent window geometry),RecentScreensMenuContentTests.testDayOldFrameShowsACalendarDateNotAClockTime(locale date format) andMeetingScreenEvidencePassTests.testRetryAfterFinalizeJoinsTheRunTheDeadlineLeftInFlight(timing).SubscriptionEntitlementServiceTestswas skipped because it crashes whole runs on this machine.Packaged-app pass (a person, on the packaged build of this branch at e1894d91ed, production servers, macOS 27, placeholder recipients only):
not_opened/person_typing).draft_already_open, twice).send_messageattempt was stopped by the kernel's approval check, so nothing was sent. That case is now also refused in Swift (draft_in_messages).UI evidence
Captured by a person (
captured_by: human,source: manual). The metadata is in.agent-artifacts/ui-evidence/desktop-propose-compose/evidence.json.Failure class (fixes)
No
fix:commits; no declaration is required.Line-Count-Exception: desktop/macos/Desktop/Sources/Providers/ChatToolExecutor.swift | 3961 -> 3965 | the propose_message/propose_email dispatch case with its turn text and run id, and the run and owner passed to search_contacts so its handles can address a draft; parsing, grounding and result mapping live in ChatToolExecutor+ComposeProposals.swift and Services/Compose*.swift