Skip to content

FE-1834: Teach Brunch voice agent custom words - #9899

Open
kostandinang wants to merge 8 commits into
kd/fe-1833-speak-progressfrom
kostandin/fe-1834-teach-brunch-your-words
Open

kostandinang wants to merge 8 commits into
kd/fe-1833-speak-progressfrom
kostandin/fe-1834-teach-brunch-your-words

Conversation

@kostandinang

@kostandinang kostandinang commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

🌟 What is the purpose of this PR?

Let people teach Brunch names and terms through a default-off Custom words Labs setting. Saved spellings and optional pronunciation hints help the assistant recognize and say those words in Voice conversations.

🔗 Related links

🚫 Blocked by

🔍 What does this change?

  • Add a Voice-only Words header button with a book icon and a Custom words dialog for adding, editing, and removing up to 50 entries.
  • Store words in this browser, scoped to the principal and conversation; clear them with the conversation. Preserve unsaved edits when another tab deletes the edited word.
  • Send spellings to the assistant and both existing voice transcription prompts, and pronunciation hints to the speaking model. Keep the current transcription models and semantic VAD.
  • Keep active Voice sessions on their starting hints until restarted, including retries, and retain the echo guard.
  • Keep the word list, pronunciation and limits in the website's Voice code, and the spelling bounds Brunch receives in the transport. Brunch core, Petrinaut Core and the stock assistant prompt are unchanged.
  • Add a website user guide at apps/petrinaut-website/docs/custom-words.md and a patch changeset for Petrinaut's assistant header controls.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

  • Modifies an npm-publishable library and changeset files are included.

📜 Does this require a change to the docs?

  • Requires changes to docs which are made as part of this PR.

🕸️ Does this require a change to the Turbo Graph?

  • Does not affect the execution graph.

⚠️ Known issues

  • One Live Voice turn was verified with captured microphone and speaker audio; legacy Realtime and physical speaker echo were not tested.
  • The 50-entry limit is also subject to spelling, pronunciation, aggregate-text, and encoded-header size limits.
  • Local frontend checks cannot validate Brunch history without the backend.

🐾 Next steps

Sync the parent branch before marking ready.

🛡 What tests cover this?

  • Tests cover validation, browser persistence, cross-tab edits, transport, prompt wiring, session snapshots, and echo filtering.
  • Latest targeted run: transport, Brunch core, Petrinaut Core, the Petrinaut assistant panel, apps/brunch-agent and the website's Voice and Brunch demo tests pass.
  • Formatting and Markdown lint passed for the latest UI and documentation edits.
  • Browser checks covered the Labs label, dialog copy, accessible Words button, and Voice-only visibility.

❓ How to test this?

  1. Select Use Brunch and enable Custom words in User settings → Labs.
  2. Enter Voice mode, open Words, and add a name with an optional pronunciation.
  3. Edit and remove entries; reload to check persistence. Verify the button is hidden in text mode.
  4. Start Voice and test the name. Change its hint during the session, then restart Voice to apply it.
  5. Clear the conversation and confirm its word list is cleared.

📹 Demo

custom-words-live-demo.mp4

@vercel

vercel Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
hash Ready Ready Preview Oct 2, 2026 10:07pm UTC
petrinaut Ready Ready Preview Oct 2, 2026 10:07pm UTC
petrinaut-docs Ready Ready Preview Oct 2, 2026 10:07pm UTC
1 Skipped Deployment
Project Deployment Actions Updated
hashdotdesign-tokens Ignored Ignored Preview Oct 2, 2026 10:07pm UTC

Request Review

@github-actions github-actions Bot added area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) type/eng > frontend Owned by the @frontend team area/tests New or updated tests area/apps labels Oct 2, 2026
Comment thread libs/@hashintel/brunch-agent/packages/core/src/words.ts Fixed
@kostandinang kostandinang changed the title FE-1834: Add Custom words for Voice FE-1834: Teach Brunch your words Oct 2, 2026
@kostandinang
kostandinang marked this pull request as ready for review October 2, 2026 13:59
@cursor

cursor Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches user-message framing, model instructions, and third-party voice APIs; mistakes could leak host metadata into prompts or admit malformed hints, though validation and projection stripping mitigate injection and history pollution.

Overview
Adds an optional, default-off Custom words Labs setting and a Voice-only Words header control (via new headerActions / actionsInputMode on the Petrinaut assistant panel) so users can maintain per-conversation spellings and pronunciation notes in browser storage.

Spellings ride along on Brunch submissions through a v2 contextual user message envelope (words snapshot captured at send time for retries), while Brunch turns them into a cautious system instruction (useWords) and strips the envelope from model context and human-facing transcripts. Voice pins one word list per session: encoded x-petrinaut-voice-words on Live/Realtime/transcription requests, spellings in transcription prompts, pronunciation hints only on speaking instructions, plus shadow prompt-regurgitation filtering tied to the pinned prompt.

Includes validation limits, server-side header decoding with fail-closed 400s, user docs, and broad unit/integration coverage.

Reviewed by Cursor Bugbot for commit dc79ebc. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread libs/@hashintel/brunch-agent/packages/transport-aisdk/src/words.ts Dismissed
@kostandinang kostandinang changed the title FE-1834: Teach Brunch your words FE-1834: Teach Brunch voice agent custom words Oct 2, 2026
Comment thread libs/@hashintel/brunch-agent/packages/transport-aisdk/src/words.ts Fixed

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9c8d16b. Configure here.

Comment thread apps/petrinaut-website/src/main/app/local-storage-demo/conversation-words.ts Outdated
kube
kube previously approved these changes Oct 2, 2026
@kostandinang
kostandinang added this pull request to stack #9887 October 2, 2026 21:31
kostandinang and others added 8 commits October 2, 2026 21:56
Let a person keep a short, browser-local list of names and terms per Brunch conversation, default off in Labs. Spellings reach Brunch's system prompt through the chat transport and extend the gpt-4o-transcribe prompt on both voice paths; pronunciation notes reach only the speaking model. Voice sessions pin their start snapshot until an explicit restart, and the echo guard compares against the prompt that was actually sent. Adds a header-actions slot and a Teach action on finalized voice lines in Petrinaut, with user docs.

FE-1834

Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: Amp <amp@ampcode.com>
Voice owns the word entry, pronunciation and UI limits in the website; the transport owns the spelling bounds it carries to Brunch. Restore the transport's core-independence lint rule, the stock assistant prompt and the Petrinaut doc catalogue, and move the user guide to the website.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

This branch was successfully deployed

3 active deployments
Preview – hash — dc79ebc8 Deployed Oct 2, 2026 by vercel[bot]
Preview – petrinaut-docs — dc79ebc8 Deployed Oct 2, 2026 by vercel[bot]
Preview – petrinaut — dc79ebc8 Deployed Oct 2, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/apps area/infra Relates to version control, CI, CD or IaC (area) area/libs Relates to first-party libraries/crates/packages (area) area/tests New or updated tests type/eng > frontend Owned by the @frontend team

Development

Successfully merging this pull request may close these issues.

3 participants