Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .changeset/tidy-voice-dock.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@
"@hashintel/petrinaut": patch
---

Allow host-owned Voice sessions to report a neutral connected state and omit unsupported microphone, resume, reconnect, and playback controls from the dock. Keep the viewport controls above a collapsed Voice dock instead of reserving full-height chat panel space.
Allow host-owned Voice sessions to report a neutral connected state and omit unsupported controls, keeping viewport controls above the collapsed dock. Contain Voice failures and explicit recovery warnings behind a warning icon with hover previews and a scrollable details popover instead of global notifications, while temporary session notices remain in the dock status.

Keep voice permission content above the compact dock without moving the controls, and show Voice setup until a connection actually starts. Keep the error icon in the same dock position when expanding or collapsing a Voice session.
19 changes: 15 additions & 4 deletions libs/@hashintel/petrinaut/docs/ai-assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,9 @@ action appears in the first-run prompt and the assistant panel; if voice is unav
composer retains a disabled **Send** action. Starting Voice mode keeps the transcript in place and
opens the existing one-time disclosure. Voice selected from the first-run prompt starts compact: the
disclosure and microphone check appear in a card immediately above a **Voice setup** dock, while the
AI header, transcript, and composer stay hidden. Select **Expand voice setup** to restore the full
AI header, transcript, and composer stay hidden. The card opens without shifting the dock or viewport
controls, and scrolls within the available screen height. Setup does not show **Connecting** before
you start. Select **Expand voice setup** to restore the full
panel. Voice started from the composer keeps that full panel visible. Review that OpenAI processes
live audio and speaks the interviewer's words while Petrinaut keeps finalized answers in the
conversation rather than the audio. You can check your microphone before confirming that you
Expand Down Expand Up @@ -87,6 +89,15 @@ flicker above the line. While the assistant speaks the ribbon takes on a restrai
motion instead, colour crossfading as the turn changes hands, so which side holds it is readable at a
glance. It flattens to near a line whenever nobody holds the turn.

Voice failures and recovery warnings, including unconfirmed submissions and input that was not
retained, join the Voice warning indicator rather than appearing as global notifications. Hover to
preview or click to read the complete details, including while the panel is collapsed. Distinct
issues share one icon with a count; long details wrap and scroll in the popover, where they can be
copied or dismissed with **Dismiss Voice issues**. Issues stay in the list until dismissed or the
chat is cleared; dismissing them does not trigger a retry or mean that unsent input was retained.
Temporary status notices, such as a pending handoff, replace the short dock state only while they
apply and do not become Voice issues.

Spoken turns appear in the conversation as soon as their finalized text arrives, so the transcript
stays current while the session runs and tools that edit the net remain visible. Select **Collapse
voice session** to reduce the panel to the Voice dock alone; this hides the AI header, transcript, and
Expand Down Expand Up @@ -141,9 +152,9 @@ reopen the panel. The same mounted session stays paused; choose **Resume voice m
ready. **Clear AI chat** is unavailable while a Voice
session is active.

If voice cannot continue, the status reads **Voice interrupted** and the actionable error arrives as
a persistent toast that names the microphone, connection, or Voice failure in one sentence, followed
by any diagnostic reference in parentheses. **Reconnect voice mode** replaces the microphone action until
If voice cannot continue, the status reads **Voice interrupted** and the warning icon contains
the microphone, connection, or Voice failure, followed by any diagnostic reference in parentheses.
**Reconnect voice mode** replaces the microphone action until
the session recovers. For microphone permission or device errors, allow access or connect/select a
microphone before reconnecting. For an interrupted request, network error, or timeout, check the
connection and reconnect. If the preview is unavailable, continue with the text composer. An invalid
Expand Down
4 changes: 3 additions & 1 deletion libs/@hashintel/petrinaut/src/react/voice-session/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,9 @@ export type PetrinautAiVoiceSessionState = {
microphoneMuted: boolean;
/** Normalized 0–1 input level driving the listening indicator. */
microphoneLevel: number;
/** Recoverable feedback about an utterance which was not submitted. */
/** Temporary operational status shown in place of the current phase. */
notice?: string | null;
phase: PetrinautAiVoiceSessionPhase;
/** Recoverable issue retained behind the Voice warning indicator. */
warningMessage?: string | null;
};
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,16 @@ export const useVoiceSessionNotice = (): string | null => {
);
};

export const useVoiceSessionWarningMessage = (): string | null => {
const store = use(VoiceSessionContext);

return useSyncExternalStore(
store.subscribe,
() => store.getSnapshot().state?.warningMessage ?? null,
() => null,
);
};

export const useVoiceSessionInterruptionBySpeaking = (): boolean => {
const store = use(VoiceSessionContext);
return useSyncExternalStore(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1837,7 +1837,9 @@ describe("AiAssistantPanel composer submissions", () => {
expect(textarea.value).toBe("Keep this draft");
expect(textarea.disabled).toBe(false);
expect(latestVoiceContext?.inputMode).toBe("voice");
expect(screen.getByText("Voice invalidation failed.")).not.toBeNull();
expect(
await screen.findByText("Voice invalidation failed."),
).not.toBeNull();
});

test("restores a typed draft when its post-Voice message submission rejects", async () => {
Expand Down Expand Up @@ -3329,7 +3331,7 @@ describe("AiAssistantPanel composer submissions", () => {
await act(async () => latest?.stop());
expect(requestStop).toHaveBeenCalledOnce();
expect(
screen.getAllByText(/Durable stop failed/u).length,
(await screen.findAllByText(/Durable stop failed/u)).length,
).toBeGreaterThan(0);

await act(async () => {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { type ReactNode, useState } from "react";
import { userEvent, within } from "storybook/test";
import { expect, userEvent, within } from "storybook/test";

import { Button } from "@hashintel/ds-components";
import { css } from "@hashintel/ds-helpers/css";
Expand Down Expand Up @@ -348,6 +348,24 @@ export const VoiceModeAwaitingConsentCompact: Story = {
voiceModeAvailable
/>
),
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
const consent = canvas.getByTestId("ai-voice-mode");
const dock = canvas.getByRole("region", { name: "Voice setup" });
const shell = dock.closest("aside")!;
const initialShellHeight = shell.getBoundingClientRect().height;
const initialDockTop = dock.getBoundingClientRect().top;

// Host content can grow or disappear; neither should move the controls
// whose position is derived from the compact shell's reported height.
consent.style.minHeight = "320px";
await expect(shell.getBoundingClientRect().height).toBe(initialShellHeight);
await expect(dock.getBoundingClientRect().top).toBe(initialDockTop);
consent.style.display = "none";
await expect(shell.getBoundingClientRect().height).toBe(initialShellHeight);
consent.style.removeProperty("min-height");
consent.style.removeProperty("display");
},
};

export const VoiceSessionListening: Story = {
Expand All @@ -363,6 +381,38 @@ export const VoiceSessionListening: Story = {
),
};

export const VoiceSessionLongWarning: Story = {
render: () => (
<Frame
initialVoiceDockCollapsed
inputMode="voice"
messages={[userMessage, assistantMarkdownMessage]}
voiceModeAvailable
voiceSession={liveSession({
phase: "connected",
warningMessage:
"Voice admission could not be confirmed. Check canonical history before sending again; no automatic retry was made.",
})}
/>
),
};

export const VoiceSessionInputNotRetained: Story = {
render: () => (
<Frame
initialVoiceDockCollapsed
inputMode="voice"
messages={[userMessage, assistantMarkdownMessage]}
voiceModeAvailable
voiceSession={liveSession({
phase: "connected",
warningMessage:
Comment thread
cursor[bot] marked this conversation as resolved.
"That utterance was not retained. Wait for the pending input, then use the composer to send it.",
})}
/>
),
};

export const VoiceSessionCollapsed: Story = {
render: () => (
<Frame
Expand Down Expand Up @@ -531,6 +581,65 @@ export const NetworkError: Story = {
render: () => <Frame error={errorMessage} messages={[userMessage]} />,
};

export const MultipleVoiceIssues: Story = {
render: () => (
<Frame
messages={[userMessage]}
voiceSession={liveSession({
phase: "error",
errorMessage:
"Voice connection interrupted. Check your connection before reconnecting.",
warningMessage:
"Voice admission could not be confirmed. Check canonical history before sending again; no automatic retry was made.",
})}
/>
),
};

export const CollapsedVoiceIssues: Story = {
render: () => (
<Frame
initialVoiceDockCollapsed
messages={[userMessage]}
voiceSession={liveSession({
phase: "error",
errorMessage:
"Voice connection interrupted. Check your connection before reconnecting.",
warningMessage:
"Voice admission could not be confirmed. Check canonical history before sending again; no automatic retry was made.",
})}
/>
),
play: async ({ canvasElement }) => {
const canvas = within(canvasElement);
const indicator = canvas.getByRole("button", {
name: "Show 2 Voice issues",
});
const { x, y, width, height } = indicator.getBoundingClientRect();
await userEvent.click(
canvas.getByRole("button", { name: "Expand voice session" }),
);
await expect(
canvas.getByRole("button", { name: "Show 2 Voice issues" }),
).toBe(indicator);
await expect(indicator.getBoundingClientRect()).toMatchObject({
x,
y,
width,
height,
});
await userEvent.click(
canvas.getByRole("button", { name: "Collapse voice session" }),
);
await expect(indicator.getBoundingClientRect()).toMatchObject({
x,
y,
width,
height,
});
},
};

export const StoppedResponse: Story = {
render: () => (
<Frame
Expand Down
Loading
Loading