Repository navigation
feat(widget): voice input and text-to-speech playback - #153
Merged
Merged
Conversation
Add an opt-in `voice` option (React prop, `window.ClaudiusConfig` key, and `<claudius-chat voice ...>` attributes). Off by default, like attachments. Dictation: a mic button in the composer, shown only where the browser has SpeechRecognition. Toggle (default) or hold-to-talk, with interim results streaming into the message field and appending to typed text. Typing takes over from an open session, and sending ends it. Optional auto-submit fires only after a session that ended normally and heard something. A voice-activity indicator sits inside the field while listening. Read aloud: a speaker control under settled assistant replies, with pause, resume, and stop. One reply at a time, silenced when dictation starts or the chat closes. Markdown is stripped the same way as the screen-reader announcement. Platform quirks handled, each pinned by a test: - stop() before the engine has started aborts instead, so answering the permission prompt after release cannot open the mic - replies are queued as sentence-sized utterances to dodge Chrome's ~15 s cutoff - on Android pause() ends the utterance; the rest of the queue is dropped rather than read on after a "pause" - late error events from cancelled utterances are ignored - an idle engine is not sent cancel() right before speak() - hold mode still toggles on a click with no click count, which is how screen readers and switch devices activate a button Speech language follows the widget locale, refined by the region in <html lang> or the browser language (en-GB page -> British recognition), defaulting to en-US. Eleven new strings in en, es, fr, and de. The test setup gains a PointerEvent polyfill: jsdom has none, so Testing Library was dispatching bare Events with no `button` or `pointerId`. Refs #54 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…snippets
`widget.voice` accepts a boolean or { input, output, mode, autoSubmit, lang }
in clients/_schema.json and in `claudius validate`, with a per-field error
for each malformed option.
The script snippet passes the value straight through to ClaudiusConfig. The
web component snippet emits one attribute per option (voice-mode,
voice-auto-submit, voice-input, voice-output, voice-lang): unlike attachment
limits these are plain enums and booleans, and the element cannot read
ClaudiusConfig, so without them the modes would not be configurable there.
Refs #54
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New configuration/voice.md: enabling voice across the three embed styles, options and attributes, toggle versus hold, auto-submit semantics, language resolution, a browser support matrix, where each browser's voices come from and how to list them, and troubleshooting for blocked microphones. The privacy section corrects the premise in #54. Web Speech recognition is not on-device by default: MDN states that audio "is sent to a web service for recognition processing", and Safari uses the Siri engine without saying where a request is handled. The page says what is true and verifiable instead: the widget never handles audio (no getUserMedia) and none reaches the worker, while the browser vendor does process it, which site owners should reflect in their privacy policy. It also notes that a network voice sends reply text to the vendor. Also: widget options table and attribute list, client config fields, localization keys, two FAQ entries, CLAUDE.md, and the design doc brought in line with what was built. Refs #54 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Voice input and read-aloud (two hooks, the mic button and its hold / toggle gestures, the listening indicator, the read-aloud controls, and eleven strings in four locales) add, measured against main with size-limit: claudius.iife.js +11.9 KB raw +3.86 KB gzip +3.43 KB brotli claudius.js +15.9 KB raw +4.67 KB gzip +3.71 KB brotli claudius.css +1.5 KB raw +262 B gzip +257 B brotli That is in line with attachments (+3.6 KB gzip) for a comparable feature. The IIFE build cannot code-split, so the code cannot be deferred until voice is enabled. New limits are the measured sizes plus the usual 5% headroom, matching the earlier budget bumps. Refs #54 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Deploying chat-widget with
|
| Latest commit: |
766ddd4
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://8c0d8314.chat-widget-ejc.pages.dev |
| Branch Preview URL: | https://54-voice-input-tts.chat-widget-ejc.pages.dev |
Contributor
size-limit report 📦
|
8 of 12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds opt-in voice to the widget, built directly on the browser's Web Speech API with no new dependencies and no worker changes.
SpeechRecognition. Click-to-toggle (default) or hold-to-talk. Words stream into the message field as they are spoken and append to anything already typed. Typing takes over from an open session, and sending ends it. Optional auto-submit.voiceoption on all three embed styles: React prop,window.ClaudiusConfig.voice, and<claudius-chat voice voice-mode voice-auto-submit voice-input voice-output voice-lang>. Off by default, likeattachments, so a minor release does not put a microphone in every existing embed.widget.voiceinclients/_schema.json,claudius validate, and both snippet generators.configuration/voice.md, plus the design doc atdocs/plans/2026-09-18-voice-input-tts-design.md, which records every decision below in more depth.Please read: two deliberate departures from the issue
1. The privacy wording in #54 is not accurate, so the docs say something different. The issue asks for a documented posture that "speech is processed on-device by the browser; no audio leaves the client." That is false for the default Web Speech API. MDN: "By default, using speech recognition on a web page involves a server-based recognition engine. Your audio is sent to a web service for recognition processing." Chrome sends audio to Google; Safari uses the Siri engine and Apple does not say where a request is handled. Publishing the original claim could mislead clients writing their own privacy policies.
The docs state what is true and verifiable instead: the widget never handles audio (it does not call
getUserMedia), and no audio reaches the worker or Anthropic, but the visitor's browser vendor does process it, and site owners should say so. On-device recognition (processLocally, desktop Chrome 139+) exists but is experimental, unavailable on Android, Safari, and Firefox, and needs a language-pack install flow. I left it as a possible follow-up rather than build v1 on it.2. The "waveform / level indicator" shows voice activity, not loudness. The Web Speech API exposes no audio signal. A true meter needs a second microphone capture through
getUserMedia, which conflicts with the recognizer on Android and iOS (the platforms the issue cares most about), double-prompts in Safari, and would give the widget raw audio, making the privacy statement above untrue. The bars instead react to the engine'sspeechstart/speechendand result events, so they still show whether the visitor is being heard.Smaller interpretation calls: "auto-submit toggle" is a configuration option rather than a visitor-facing switch; and the voice is left to the browser (only the language is set), because picking one explicitly lands on the novelty voice "Albert" on some Macs and would replace Edge's Natural voices.
Platform quirks handled, each pinned by a test
pause()ends the utterance; the rest of the queue is dropped instead of reading on after a "pause".errorevents from cancelled utterances are ignored.prefers-reduced-motionstills the indicator.Related Issue
Closes #54
Type of Change
Checklist
pnpm testin bothwidget/andworker/): widget 507, worker 153, scripts 82pnpm lintinwidget/): 0 errors, the same 2 pre-existing warnings asmainpnpm format:checkinwidget/)pnpm typecheckinwidget/), anddocs:api:checkandcheck:exportsare cleancd widget && pnpm build)feat/,fix/,docs/,chore/): uses the repo's existing<issue>-<slug>patternScreenshots / Notes
Bundle size. Budgets are raised in their own commit, to measured size plus the usual 5%. True cost against
main, measured with size-limit:claudius.iife.js+3.86 KB gzip,claudius.js+4.67 KB gzip,claudius.css+262 B gzip. That is in line with attachments (+3.6 KB gzip). The IIFE build cannot code-split, so the code cannot be deferred until voice is enabled.What was and was not verified.
e2e/voice.spec.ts, injects fake engines into a real browser. The whole desktop e2e suite (14 tests) passes locally against system Chrome 153.mobile.spec.tsruns on WebKit, which I cannot run locally, so that one relies on CI.SpeechRecognitionis preferred, and thatspeak()with no voices installed fireserror: synthesis-failed, which returns the control to idle rather than leaving it stuck.Unrelated, noticed in passing: the "Available keys" table in
configuration/localization.mdwas already missing the attachment, streaming, and tool-use string keys. I added the voice row and left the rest alone.Storybook:
ChatInput › WithVoiceInput/WithVoiceHoldToTalk, andChatMessage › WithReadAloud/ReadingAloud.🤖 Generated with Claude Code