Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,9 @@ pnpm test # Run tests
| `ChatMessage` | Renders individual messages with URL linking |
| `ChatSources` | Slide-out sidebar displaying grouped source links |
| `SourceIcon` | Icon button with badge count to trigger source sidebar |
| `VoiceInputButton` | Mic button for dictation; toggle and hold-to-talk gestures |
| `VoiceLevel` | Voice-activity indicator shown inside the input while listening |
| `MessageSpeechControls` | Read aloud / pause / resume / stop buttons under assistant replies |
| `AttachmentPreview` | Image thumbnail / file chip for pending and sent attachments |

### useChat Hook
Expand Down Expand Up @@ -173,6 +176,25 @@ renders. Retrieval failures degrade to ungrounded replies. Ingestion:
`pnpm rag:ingest ./content --index <name>` (chunking helpers in
`scripts/lib/rag-ingest.ts`). Docs: rag/index.md.

### Voice

Voice is widget-only and opt-in (`voice` prop / `ClaudiusConfig.voice` /
`<claudius-chat voice voice-mode voice-auto-submit voice-input voice-output
voice-lang>`), built on the browser's Web Speech API with no dependencies.
`useSpeechRecognition` (owned by `ChatInput`) runs one single-utterance session
at a time and streams interim results into the field; `useSpeechSynthesis`
(owned by `ChatWindow`) reads one settled assistant reply at a time as
sentence-sized utterances. Pure helpers live in `widget/src/utils/voice.ts`
(`resolveVoiceConfig`, `resolveSpeechLang`, `joinDictation`,
`chunkSpeechText`, `voiceOptionsFromAttributes`). Each half is feature
detected separately and renders nothing when unsupported. jsdom has no speech
engines, so tests drive `widget/src/test-utils/fakeSpeech*.ts`.

Privacy: the widget never touches audio (no `getUserMedia`) and none reaches
the worker, but browsers recognize speech on their vendor's servers by
default. Do not describe it as on-device. Docs: configuration/voice.md;
design: docs/plans/2026-09-18-voice-input-tts-design.md.

### Chat Request/Response

```typescript
Expand Down
17 changes: 17 additions & 0 deletions clients/_schema.json
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,23 @@
}
}
]
},
"voice": {
"description": "Voice input (mic button) and read-aloud via the browser's Web Speech API. true enables both with the defaults (click-to-toggle mic, no auto-submit).",
"oneOf": [
{ "type": "boolean" },
{
"type": "object",
"additionalProperties": false,
"properties": {
"input": { "type": "boolean" },
"output": { "type": "boolean" },
"mode": { "type": "string", "enum": ["toggle", "hold"] },
"autoSubmit": { "type": "boolean" },
"lang": { "type": "string", "minLength": 1 }
}
}
]
}
}
},
Expand Down
324 changes: 324 additions & 0 deletions docs/plans/2026-09-18-voice-input-tts-design.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/src/content/docs/configuration/clients.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ pnpm claudius snippet acme # generate the embed snippet(s)
| `slug` | Yes | URL-safe identifier; must match the filename |
| `apiUrl` | Yes | The client's worker chat endpoint |
| `allowedDomains` | Yes | Domains where the widget may be embedded |
| `widget` | No | Appearance: `title`, `subtitle`, `welcomeMessage`, `placeholder`, `theme`, `position`, `accentColor`; `attachments` (`true` or `{ maxSizeBytes, maxCount, allowedTypes }`) |
| `widget` | No | Appearance: `title`, `subtitle`, `welcomeMessage`, `placeholder`, `theme`, `position`, `accentColor`; `attachments` (`true` or `{ maxSizeBytes, maxCount, allowedTypes }`); `voice` (`true` or `{ input, output, mode, autoSubmit, lang }`, see [Voice](/configuration/voice/)) |
| `worker` | No | `model`, `maxTokens` (1–8192), `rateLimitMinute`, `rateLimitHour`, `systemPrompt` (path to a markdown file), `attachments` (`enabled`, `maxBytes`, `maxCount`, `maxRequestBytes`, `allowedTypes`, `storage`, `retentionHours`, `quotaIpBytesPerDay`, `quotaTenantBytesPerDay`) |

`widget.attachments` flows into the generated embed snippet. `worker.attachments`
Expand Down
1 change: 1 addition & 0 deletions docs/src/content/docs/configuration/localization.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ window.ClaudiusConfig = {
| Input | `placeholder`, `sendMessage`, `typeYourMessage` |
| Toggle / bubble | `openChat`, `dismissGreeting` |
| Errors | `errorGeneric`, `errorConnection`, `errorTimeout`, `errorRateLimitMinute`, `errorRateLimitHour`, `errorRetry` |
| [Voice](/configuration/voice/) | `voiceInput`, `voiceInputHold`, `voiceListening`, `voicePermissionDenied`, `voiceNoSpeech`, `voiceNoMicrophone`, `voiceUnavailable`, `readAloud`, `pauseReading`, `resumeReading`, `stopReading` |

Note these localize the widget UI only. The AI's reply language follows the
conversation and your [system prompt](/configuration/worker/#system-prompt) —
Expand Down
247 changes: 247 additions & 0 deletions docs/src/content/docs/configuration/voice.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,247 @@
---
title: Voice
description: Let visitors dictate messages and have replies read aloud, and understand where the audio goes.
sidebar:
order: 8
---

Voice adds two controls, both built on the browser's
[Web Speech API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API):

- A **mic button** in the composer. Visitors dictate instead of typing, and
the words appear in the message field as they speak.
- A **read-aloud button** under each completed assistant reply, with pause and
stop.

Voice is **off by default**. Each half appears only in browsers that support
it, so enabling voice is safe everywhere: an unsupported browser simply shows
the normal text chat. Nothing changes on the worker.

Read [Privacy posture](#privacy-posture) before you switch it on. The widget
never touches audio, but most browsers send it to their vendor for
recognition, and your privacy policy may need to say so.

## Enable voice

```tsx
<ChatWidget apiUrl="https://your-worker.workers.dev" voice />
```

```html
<script>
window.ClaudiusConfig = {
apiUrl: "https://your-worker.workers.dev",
voice: true,
};
</script>
```

```html
<claudius-chat api-url="https://your-worker.workers.dev" voice></claudius-chat>
```

`true` turns on both the mic and read-aloud. Pass an object to tune them:

| Option | Default | Description |
|--------|---------|-------------|
| `input` | `true` | Show the mic button |
| `output` | `true` | Show the read-aloud button on assistant replies |
| `mode` | `"toggle"` | `"toggle"`: click to start, click again to stop. `"hold"`: record only while the button is held |
| `autoSubmit` | `false` | Send the message as soon as dictation ends |
| `lang` | from the widget locale | BCP-47 tag for recognition and playback, e.g. `"en-GB"`. See [Language](#language) |

```tsx
<ChatWidget
apiUrl="…"
voice={{ mode: "hold", autoSubmit: true, output: false }}
/>
```

The web component takes the same options as attributes:

| Attribute | Example |
|-----------|---------|
| `voice` | `voice` or `voice="true"` to enable, `voice="false"` to disable |
| `voice-mode` | `voice-mode="hold"` |
| `voice-auto-submit` | `voice-auto-submit` or `voice-auto-submit="false"` |
| `voice-input` | `voice-input="false"` for read-aloud only |
| `voice-output` | `voice-output="false"` for dictation only |
| `voice-lang` | `voice-lang="en-GB"` |

In a [client config](/configuration/clients/), set `widget.voice` to `true` or
to the same object; `pnpm claudius snippet` carries it into both embed styles.

## Dictation

Dictated text is **appended** to whatever is already in the field, so a
visitor can type part of a message and speak the rest. Recognition stops by
itself at the end of a sentence; pressing the mic again adds more. If the
visitor types while the mic is listening, typing wins: their edit is kept and
the session ends, so a late result can never overwrite it.

Without `autoSubmit`, the text stays in the field to be checked and sent by
hand. Speech recognition makes mistakes, which is why this is the default.
With `autoSubmit`, the message goes out when dictation ends, but only if the
session finished normally and actually heard something. It never fires after
an error or a silent session. It uses the same path as the send button, so
pending [attachments](/configuration/attachments/), the length limit, and
[plugins](/plugins/) all apply.

Starting dictation stops any reply that is being read aloud, so the
microphone does not transcribe the widget's own voice.

### Toggle or hold

`"toggle"` is the default and the more accessible choice, because it needs no
sustained press. While listening, the button shows a stop icon.

`"hold"` is push-to-talk: recording runs from press to release, with a mouse,
a finger, or Space / Enter on the keyboard. If the visitor lets go before the
browser's permission prompt has been answered, nothing starts, so granting
permission afterwards can never open the microphone with nobody at the
button. Screen readers and switch devices activate buttons with a click they
cannot hold, so in hold mode a click of that kind toggles instead.

### Listening indicator

While the mic is open, animated bars appear inside the message field and the
placeholder reads "Listening...". The bars idle while the mic waits and liven
up while speech is being heard.

They show **voice activity, not loudness**. The Web Speech API gives a page no
access to the audio signal. A true level meter would mean opening the
microphone a second time through `getUserMedia`, which conflicts with the
recognizer on Android and iOS, prompts for permission twice in Safari, and
would hand raw audio to the widget. Claudius deliberately does not do that:
see the first point under [Privacy posture](#privacy-posture).

With `prefers-reduced-motion`, the bars hold still and the two states differ
in opacity only.

## Read aloud

The button appears once a reply has finished streaming. Formatting marks are
not spoken and links are shortened to their hostname, the same cleanup used
for screen-reader announcements. One reply plays at a time; starting another
stops the first, and closing the chat stops playback.

Long replies are queued sentence by sentence. Chrome cuts a single long
utterance off after about 15 seconds when it is using a network voice, and
short utterances avoid that.

On **Android**, browsers implement "pause" as "stop". Claudius detects this
and returns the control to its idle state, rather than carrying on with the
next sentence after the visitor pressed pause.

## Language

Recognition and playback use one BCP-47 language tag, chosen in this order:

1. `voice.lang`, when you set it.
2. The page's `<html lang>`, if it names a region of the widget's language.
An English widget on an `en-GB` page listens for, and speaks, British
English.
3. The browser's language, under the same rule.
4. A default for the widget [locale](/configuration/localization/):

| Locale | Default speech language |
|--------|-------------------------|
| `en` | `en-US` |
| `es` | `es-ES` |
| `fr` | `fr-FR` |
| `de` | `de-DE` |

Tags in a different language from the widget are ignored, so a French browser
visiting a Spanish widget still gets `es-ES`. Set `voice.lang` for anything
more specific, such as `es-MX` or `pt-BR`.

## Browser support

| Browser | Dictation | Read aloud | Notes |
|---------|-----------|------------|-------|
| Chrome, desktop and Android | Yes | Yes | Needs a network connection |
| Edge | Yes | Yes | |
| Safari 14.1+ on macOS, 14.5+ on iOS and iPadOS | Yes | Yes | The visitor must have Siri enabled in system settings |
| Firefox | No | Yes | Recognition exists only behind a flag. The mic is hidden; read-aloud works |
| Brave, and many in-app browsers | No | Varies | They expose the API but cannot reach a speech service. The mic appears, and using it shows "Voice input is not available right now." |
| Android WebView | Varies | No | |

Dictation also requires a secure page (HTTPS, or `localhost` in development).

### Voices

Claudius does not pick a voice. It sets the language and lets the browser
choose, because the available voices depend on the visitor's browser *and*
operating system, and the browser's own choice is the most reliable one:

| Browser | Where voices come from |
|---------|------------------------|
| Chrome (desktop) | The operating system's voices, plus Google network voices for about 20 languages |
| Edge | The operating system's voices, plus Microsoft's online "Natural" voices |
| Safari, and every browser on iOS | Apple's system voices |
| Firefox | The operating system's voices |
| Chrome (Android) | The device's text-to-speech engine, usually Google's |

To see exactly what a given browser offers, run this in its console:

```js
speechSynthesis.getVoices().map((v) => `${v.name} (${v.lang})${v.localService ? "" : " - network"}`);
```

Chrome loads the list asynchronously, so run it twice if the first result is
empty. A desktop with no voices installed at all, which is common on Linux,
produces no sound: the button returns to idle straight away.

## Privacy posture

What Claudius does, and does not do:

- **The widget never handles audio.** It does not call `getUserMedia`, never
sees a sample of sound, and cannot record. The browser listens, and gives
the widget text, exactly as if it had been typed.
- **No audio reaches your worker or Anthropic.** Only the resulting text is
sent, and only when the message is sent.
- **Nothing is on unless you enable it**, and the microphone opens only while
the visitor is pressing or has toggled the mic button. Closing the chat or
leaving the page ends the session.

What the browser does is outside Claudius's control, and it is **not
on-device in most browsers**:

| Browser | Where speech is recognized |
|---------|----------------------------|
| Chrome | Google's servers. [MDN](https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API): "Your audio is sent to a web service for recognition processing" |
| Edge | The browser vendor's speech service |
| Safari | The same engine as Siri. Apple does not say whether a given request is handled on the device or on its servers, so treat it as leaving the device |

Read-aloud is similar in one case: when Chrome or Edge choose a **network
voice**, the text of the reply is sent to Google or Microsoft to be
synthesized. System voices are generated on the device. The console snippet
under [Voices](#voices) marks which is which.

If you enable voice, say so in your privacy policy: that dictation is
processed by the visitor's browser vendor, and name the vendors above. The
browser asks the visitor for microphone permission itself, the first time
they use the mic.

On-device recognition exists (`SpeechRecognition.processLocally`, in desktop
Chrome 139 and later) but is experimental, is unavailable on Android, Safari,
and Firefox, and requires downloading a language pack first. Claudius does not
use it yet.

### Site policies that block the microphone

- A `Permissions-Policy: microphone=()` header disables dictation. Allow your
own origin: `Permissions-Policy: microphone=(self)`.
- If the page hosting the widget is itself inside an `<iframe>`, that frame
needs `allow="microphone"`.

In both cases the visitor sees "Microphone access is blocked."

## Customizing the text

Every voice string can be [translated or overridden](/configuration/localization/):
`voiceInput`, `voiceInputHold`, `voiceListening`, `voicePermissionDenied`,
`voiceNoSpeech`, `voiceNoMicrophone`, `voiceUnavailable`, `readAloud`,
`pauseReading`, `resumeReading`, and `stopReading`. English, Spanish, French,
and German ship built in.
5 changes: 4 additions & 1 deletion docs/src/content/docs/configuration/widget.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,14 +27,17 @@ attributes on the `<claudius-chat>` web component.
| `triggers` | `Trigger[]` | `undefined` | Proactive triggers; see [Proactive triggers](/configuration/triggers/) |
| `plugins` | `ClaudiusPlugin[]` | `undefined` | Message middleware run around each send (`onBeforeSend` / `onAfterReceive` / `onError`); see [Plugins](/plugins/) |
| `attachments` | `boolean \| AttachmentsOptions` | `false` | Let visitors attach images and PDFs; `true` for the defaults (5 MB, 5 files) or an object with `maxSizeBytes`, `maxCount`, `allowedTypes`. See [Attachments](/configuration/attachments/) |
| `voice` | `boolean \| VoiceOptions` | `false` | Mic button for dictation and a read-aloud button on replies, using the browser's Web Speech API; `true` for the defaults or an object with `input`, `output`, `mode`, `autoSubmit`, `lang`. See [Voice](/configuration/voice/) |

## Web component attributes

`<claudius-chat>` supports the scalar options as kebab-case attributes:
`api-url`, `title`, `subtitle`, `welcome-message`, `placeholder`,
`persist-messages`, `storage-key-prefix`, `request-timeout-ms`, `theme`,
`accent-color`, `position`, `attachments` (`attachments` or
`attachments="true"` enables the defaults).
`attachments="true"` enables the defaults), and `voice` with its companions
`voice-mode`, `voice-auto-submit`, `voice-input`, `voice-output`, and
`voice-lang` (see [Voice](/configuration/voice/#enable-voice)).

```html
<claudius-chat
Expand Down
8 changes: 7 additions & 1 deletion docs/src/content/docs/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ API key can access, and `MAX_TOKENS` to control response length (default
- **Attachments** (optional): forwarded to Anthropic and discarded by default;
with the R2 backend they are kept for a configurable retention window behind
signed URLs. Details in [Attachments](/configuration/attachments/#privacy-posture).
- **Voice** (optional, off by default): the widget never handles audio and
none reaches your worker. The visitor's browser does the listening, and most
browsers send that audio to their own vendor (Google, Apple) to be
recognized. Details in [Voice](/configuration/voice/#privacy-posture).

## How big is the embed?

Expand All @@ -58,7 +62,9 @@ existing React.
## Is it accessible?

WCAG 2.1 AA: focus trap, full keyboard navigation, ARIA labels, screen-reader
announcements, reduced-motion-aware animations. Lighthouse accessibility
announcements, reduced-motion-aware animations. Optional
[voice input and read-aloud](/configuration/voice/) help visitors who find
typing or reading difficult. Lighthouse accessibility
scores 98/100.

## How do I report a security issue?
Expand Down
Loading
Loading