FE-1722: Improve Brunch Voice controls

.changeset/clear-voice-controls.md

@@ -0,0 +1,5 @@
+---
+"@hashintel/petrinaut": patch
+---
+
+Improve Voice controls with provider-specific audio options, independent microphone and speaker state, distinct Stop and End actions, and a responsive compact dock.

apps/petrinaut-website/MISSION.md

@@ -1,29 +1,38 @@
-# GPT-Live capture mitigation
+# Improve Brunch Voice controls
 
 The child branch's sole execution authority is the
-[Brunch mission](../../libs/@hashintel/brunch-agent/MISSION.md).
-This file is a pointer, not a second mission.
+[Brunch mission](../../libs/@hashintel/brunch-agent/MISSION.md). This file is a
+pointer, not a second mission.
 
-FE-1712 permits explicit browser capture preferences, semantic VAD with medium
-eagerness on the separate Live transcription session, and provider-free checks.
-It also permits a 500 ms Speaking-indicator hold and a patient-listening Live
-instruction. Submission timing, separate finalized transcription and Realtime stay
-unchanged. The indicator does not control playback or establish turn completion.
-Acoustic benefit remains Kostandin's matched speaker/headphone witness; no
-deterministic feedback prevention or migration-readiness claim is established.
-The sole mission specifies bounded headless probe allocations and their results;
-natural human turn boundaries still require the owner witness. No new publication authority.
+FE-1722 selects the reduced Voice-control cut: one compact dock, direct
+microphone mute, one secondary audio popover and the existing conversation
+panel for output. Both Live and Realtime expose canonical Stop only while
+Brunch is submitted or streaming; End remains separate Voice teardown.
+Show/Hide conversation changes visibility only.
 
-The publication base is restacked FE-1664 at
-[6d188da42f](https://github.com/hashintel/hash/commit/6d188da42f86b2d6ef3d211ec55058685185e8c7).
-Its integration contract and earlier standalone comparisons are retained in the
-[future spine](../../libs/@hashintel/brunch-agent/MISSION.next.md#voice-feedback-follow-up).
-The parent has removed its superseded `PR_DESCRIPTION.md` draft; its PR body on
-GitHub is its authority. This child's local Git branch description mirrors its mission.
+Live microphone mute gates the existing shared capture track without silencing
+playback. Realtime preserves its current microphone gating. Both providers gain
+session-local speaker mute and normalized volume, reset for every new session.
+Read-full-response, repeat-question and interruption-by-speaking remain
+Realtime-only. Speaker settings do not redefine Speaking, and a
+provider-finalized partial transcript after mid-utterance mute is allowed.
 
-Kostandin authorizes pushing this child and opening its draft PR against FE-1664.
-The authorized conflict fix preserves the parent's consent and Thinking dock behavior
-and refreshes this draft's proof record. No other issue/PR changes, agent
-microphone access, merge or deployment are authorized. The sole provider exception
-is the bounded synthetic transcription probe specified in the mission.
-Delegation-driven invocation and transcript filtering remain deferred.
+The branch is stacked on FE-1664 at
+[`023a26b96b`](https://github.com/hashintel/hash/commit/023a26b96b51169da0acdb188697e159d001bcc0),
+the squash base incorporating merged
+[FE-1712 PR #9704](https://github.com/hashintel/hash/pull/9704). FE-1712 remains
+the protected behavior and evidence source inherited through that base. Its
+capture preferences, semantic VAD, patient listening, 500 ms output hold and
+unfinished owner-held obligations remain unchanged. Device switching, voice
+and speed selection, helmet animation and persistence remain deferred in the
+[future spine](../../libs/@hashintel/brunch-agent/MISSION.next.md#voice-control-follow-up).
+
+FE-1722 implementation and deterministic verification are established on this
+branch through `49ffd6836a..bdc699c923`. Post-rebase evidence passed 160
+focused Petrinaut tests, 349 focused website unit tests and 5 Voice preview
+tests under network denial; the Brunch mission owns the exact proof and
+remaining limitations. Real microphone, speaker and headphone behavior remains
+unproven and owner-held. No agent microphone or provider session is authorized.
+Current owner direction authorizes the subsequent branch push and stacked draft
+PR after the committed final gate; this correction task performs neither.
+Merge, deployment, Linear writes and other tracker changes remain unauthorized.

apps/petrinaut-website/src/main/app/voice-interview/live-conversation-control.test.tsx

@@ -23,6 +23,9 @@ import type { FlueClient, FlueConversationState } from "@flue/sdk";
 import type { PetrinautAiVoiceModeContext } from "@hashintel/petrinaut/ui";
 
 const liveConversationMocks = vi.hoisted(() => ({
+  setMicrophoneMuted: vi.fn(),
+  setSpeakerMuted: vi.fn(),
+  setSpeakerVolume: vi.fn(),
   stop: vi.fn(async () => {}),
 }));
 
@@ -33,6 +36,9 @@ vi.mock("./live-conversation", () => ({
     stop: liveConversationMocks.stop,
     appendCommentary: vi.fn(() => true),
     appendInstructions: vi.fn(() => true),
+    setMicrophoneMuted: liveConversationMocks.setMicrophoneMuted,
+    setSpeakerMuted: liveConversationMocks.setSpeakerMuted,
+    setSpeakerVolume: liveConversationMocks.setSpeakerVolume,
   })),
 }));
 afterEach(() => {
@@ -204,6 +210,8 @@ test("reuses setup and reports failure to the host dock and notification surface
     phase: "listening",
     microphoneLevel: 0,
     microphoneMuted: false,
+    speakerMuted: false,
+    speakerVolume: 1,
     errorMessage: null,
     notice: null,
     warningMessage: null,
@@ -265,6 +273,8 @@ test("reuses setup and reports failure to the host dock and notification surface
     microphoneMuted: false,
     errorMessage: null,
     notice: playbackNotice,
+    speakerMuted: false,
+    speakerVolume: 1,
     warningMessage: null,
   });
   if (!props.registerVoiceModeSessionControls)
@@ -276,6 +286,9 @@ test("reuses setup and reports failure to the host dock and notification surface
     "end",
     "pause",
     "retryPlayback",
+    "setMicrophoneMuted",
+    "setSpeakerMuted",
+    "setSpeakerVolume",
   ]);
   controls.retryPlayback?.();
   const liveSession = vi.mocked(createLiveConversation).mock.results[0]!
@@ -295,6 +308,8 @@ test("reuses setup and reports failure to the host dock and notification surface
     microphoneLevel: 0,
     microphoneMuted: true,
     notice: null,
+    speakerMuted: false,
+    speakerVolume: 1,
     warningMessage: null,
   });
   expect(screen.getByText(connectionError)).toBeTruthy();
@@ -306,6 +321,246 @@ test("reuses setup and reports failure to the host dock and notification surface
   expect(createLiveConversation).toHaveBeenCalledOnce();
 });
 
+test("registers truthful microphone and speaker controls for the same Live session", async () => {
+  const props = context();
+  const { rerender } = render(
+    <VoiceInterviewControl {...props} config={config} />,
+  );
+  await start();
+  const onState = vi.mocked(createLiveConversation).mock.lastCall![0];
+  act(() =>
+    onState({
+      phase: "connected",
+      message: null,
+      activity: { microphoneLevel: 0.42, outputActive: false },
+    }),
+  );
+  if (!props.registerVoiceModeSessionControls)
+    throw new Error("Session control registration was not provided");
+  const controls = vi.mocked(props.registerVoiceModeSessionControls).mock
+    .lastCall![0];
+
+  act(() => controls.setMicrophoneMuted?.(true));
+  expect(liveConversationMocks.setMicrophoneMuted).toHaveBeenCalledWith(true);
+  expect(props.reportVoiceSessionState).toHaveBeenLastCalledWith(
+    expect.objectContaining({
+      microphoneLevel: 0,
+      microphoneMuted: true,
+      phase: "muted",
+    }),
+  );
+
+  act(() =>
+    onState({
+      phase: "connected",
+      message: null,
+      activity: { microphoneLevel: 0.8, outputActive: true },
+    }),
+  );
+  expect(props.reportVoiceSessionState).toHaveBeenLastCalledWith(
+    expect.objectContaining({
+      microphoneLevel: 0,
+      microphoneMuted: true,
+      phase: "speaking",
+    }),
+  );
+
+  act(() => controls.setSpeakerMuted?.(true));
+  act(() => controls.setSpeakerVolume?.(0));
+  expect(liveConversationMocks.setSpeakerMuted).toHaveBeenCalledWith(true);
+  expect(liveConversationMocks.setSpeakerVolume).toHaveBeenCalledWith(0);
+  expect(props.reportVoiceSessionState).toHaveBeenLastCalledWith(
+    expect.objectContaining({
+      phase: "speaking",
+      speakerMuted: true,
+      speakerVolume: 0,
+    }),
+  );
+
+  act(() =>
+    onState({
+      phase: "connected",
+      message: null,
+      activity: { microphoneLevel: 0.8, outputActive: false },
+    }),
+  );
+  rerender(
+    <VoiceInterviewControl {...props} config={config} status="streaming" />,
+  );
+  expect(props.reportVoiceSessionState).toHaveBeenLastCalledWith(
+    expect.objectContaining({
+      microphoneMuted: true,
+      phase: "thinking",
+      speakerMuted: true,
+      speakerVolume: 0,
+    }),
+  );
+
+  act(() => controls.setMicrophoneMuted?.(false));
+  expect(liveConversationMocks.setMicrophoneMuted).toHaveBeenLastCalledWith(
+    false,
+  );
+});
+
+test("ignores media controls without a usable Live session", async () => {
+  const props = context();
+  render(<VoiceInterviewControl {...props} config={config} />);
+  if (!props.registerVoiceModeSessionControls)
+    throw new Error("Session control registration was not provided");
+  const controls = vi.mocked(props.registerVoiceModeSessionControls).mock
+    .lastCall![0];
+
+  act(() => {
+    controls.setMicrophoneMuted?.(true);
+    controls.setSpeakerMuted?.(true);
+    controls.setSpeakerVolume?.(0.4);
+  });
+
+  expect(liveConversationMocks.setMicrophoneMuted).not.toHaveBeenCalled();
+  expect(liveConversationMocks.setSpeakerMuted).not.toHaveBeenCalled();
+  expect(liveConversationMocks.setSpeakerVolume).not.toHaveBeenCalled();
+
+  await start();
+  const onState = vi.mocked(createLiveConversation).mock.lastCall![0];
+  act(() => onState({ phase: "connected", message: null }));
+  act(() =>
+    onState({
+      phase: "error",
+      message: "Live media connection ended.",
+    }),
+  );
+  liveConversationMocks.setMicrophoneMuted.mockClear();
+  liveConversationMocks.setSpeakerMuted.mockClear();
+  liveConversationMocks.setSpeakerVolume.mockClear();
+  vi.mocked(props.reportVoiceSessionState).mockClear();
+
+  act(() => {
+    controls.setMicrophoneMuted?.(true);
+    controls.setSpeakerMuted?.(true);
+    controls.setSpeakerVolume?.(0.4);
+  });
+
+  expect(liveConversationMocks.setMicrophoneMuted).not.toHaveBeenCalled();
+  expect(liveConversationMocks.setSpeakerMuted).not.toHaveBeenCalled();
+  expect(liveConversationMocks.setSpeakerVolume).not.toHaveBeenCalled();
+  expect(props.reportVoiceSessionState).not.toHaveBeenCalled();
+});
+
+test("caches speaker controls while a Live session is connecting", async () => {
+  const props = context();
+  render(<VoiceInterviewControl {...props} config={config} />);
+  await start();
+  if (!props.registerVoiceModeSessionControls)
+    throw new Error("Session control registration was not provided");
+  const controls = vi.mocked(props.registerVoiceModeSessionControls).mock
+    .lastCall![0];
+  liveConversationMocks.setSpeakerMuted.mockClear();
+  liveConversationMocks.setSpeakerVolume.mockClear();
+
+  act(() => {
+    controls.setSpeakerMuted?.(true);
+    controls.setSpeakerVolume?.(0.4);
+  });
+
+  expect(liveConversationMocks.setSpeakerMuted).toHaveBeenCalledExactlyOnceWith(
+    true,
+  );
+  expect(
+    liveConversationMocks.setSpeakerVolume,
+  ).toHaveBeenCalledExactlyOnceWith(0.4);
+  expect(props.reportVoiceSessionState).toHaveBeenLastCalledWith(
+    expect.objectContaining({
+      phase: "connecting",
+      speakerMuted: true,
+      speakerVolume: 0.4,
+    }),
+  );
+});
+
+test("ignores media controls as soon as a Live session ends", async () => {
+  const props = context();
+  render(<VoiceInterviewControl {...props} config={config} />);
+  await start();
+  if (!props.registerVoiceModeSessionControls)
+    throw new Error("Session control registration was not provided");
+  const controls = vi.mocked(props.registerVoiceModeSessionControls).mock
+    .lastCall![0];
+  const onState = vi.mocked(createLiveConversation).mock.lastCall![0];
+  act(() => onState({ phase: "connected", message: null }));
+
+  await act(() => controls.end());
+  liveConversationMocks.setMicrophoneMuted.mockClear();
+  liveConversationMocks.setSpeakerMuted.mockClear();
+  liveConversationMocks.setSpeakerVolume.mockClear();
+  vi.mocked(props.reportVoiceSessionState).mockClear();
+  act(() => {
+    controls.setMicrophoneMuted?.(true);
+    controls.setSpeakerMuted?.(true);
+    controls.setSpeakerVolume?.(0.4);
+  });
+
+  expect(liveConversationMocks.setMicrophoneMuted).not.toHaveBeenCalled();
+  expect(liveConversationMocks.setSpeakerMuted).not.toHaveBeenCalled();
+  expect(liveConversationMocks.setSpeakerVolume).not.toHaveBeenCalled();
+  expect(props.reportVoiceSessionState).not.toHaveBeenCalled();
+});
+
+test("resets and applies audio defaults when a Live session restarts", async () => {
+  const props = context();
+  render(<VoiceInterviewControl {...props} config={config} />);
+  await start();
+  const firstOnState = vi.mocked(createLiveConversation).mock.calls[0]![0];
+  act(() => firstOnState({ phase: "connected", message: null }));
+  if (!props.registerVoiceModeSessionControls)
+    throw new Error("Session control registration was not provided");
+  const controls = vi.mocked(props.registerVoiceModeSessionControls).mock
+    .lastCall![0];
+
+  act(() => controls.setMicrophoneMuted?.(true));
+  act(() => controls.setSpeakerMuted?.(true));
+  act(() => controls.setSpeakerVolume?.(0.25));
+  expect(props.reportVoiceSessionState).toHaveBeenLastCalledWith(
+    expect.objectContaining({
+      microphoneMuted: true,
+      speakerMuted: true,
+      speakerVolume: 0.25,
+    }),
+  );
+  await act(() => controls.end());
+  act(() =>
+    firstOnState({
+      phase: "ended",
+      message: "Microphone and playback stopped.",
+    }),
+  );
+  liveConversationMocks.setMicrophoneMuted.mockClear();
+  liveConversationMocks.setSpeakerMuted.mockClear();
+  liveConversationMocks.setSpeakerVolume.mockClear();
+
+  await start();
+
+  expect(createLiveConversation).toHaveBeenCalledTimes(2);
+  expect(
+    liveConversationMocks.setMicrophoneMuted,
+  ).toHaveBeenCalledExactlyOnceWith(false);
+  expect(liveConversationMocks.setSpeakerMuted).toHaveBeenCalledExactlyOnceWith(
+    false,
+  );
+  expect(
+    liveConversationMocks.setSpeakerVolume,
+  ).toHaveBeenCalledExactlyOnceWith(1);
+  const secondOnState = vi.mocked(createLiveConversation).mock.calls[1]![0];
+  act(() => secondOnState({ phase: "connected", message: null }));
+  expect(props.reportVoiceSessionState).toHaveBeenLastCalledWith(
+    expect.objectContaining({
+      microphoneMuted: false,
+      phase: "listening",
+      speakerMuted: false,
+      speakerVolume: 1,
+    }),
+  );
+});
+
 test("does not show a successful prior session close on the next consent card", async () => {
   render(<VoiceInterviewControl {...context()} config={config} />);
   await start();