FE-1664: Experiment Live Full Brunch Integration

[kostandinang]

Note

Experiment 2: This PR demonstrates the end-to-end GPT-Live and Brunch integration, including tool-backed turns. Manual testing passed. Stabilization issues identified during the experiment are addressed in follow-up #9704.

🌟 What is the purpose of this PR?

Experiment with GPT-Live as the full-duplex spoken interface for Brunch while
keeping Brunch as the only domain and tool authority. A separate
gpt-4o-transcribe session turns finalized speech into ordinary Brunch input;
settled Brunch answers are returned to Live as commentary to speak.

The path works end to end in manual sessions, including tool-backed Brunch turns.
Manual testing of this experiment passed. Stabilization issues identified during
testing are owned by downstream #9704. Merge dependencies #9671 and #9719 land
first; this PR should not be deployed without the stabilization follow-up.

🔗 Related links

• FE-1664 (internal)
• Parent FE-1663 / #9671
• Petrinaut Voice platform #9719
• Merged Brunch base FE-1573 / #9667
• Capture follow-up FE-1712 (internal) / #9704
• Mission contract

🚫 Blocked by

• Stabilization acceptance is owned by downstream FE-1712: Stabilize GPT-Live full-duplex voice feedback #9704.
• Merge dependencies: FE-1663: Experiment Live Full-Duplex Migration #9671, then FE-1664: Refine host-owned Voice sessions in Petrinaut #9719; merge FE-1664: Experiment Live Full Brunch Integration #9673 before FE-1712: Stabilize GPT-Live full-duplex voice feedback #9704.

🔍 What does this change?

• Opens GPT-Live and gpt-4o-transcribe WebRTC sessions from one consented
  microphone capture. Only finalized, provider-ordered transcription items enter
  the existing composer and Flue admission path.
• Correlates each canonical input with the newest unclaimed Live delegation.
  Complete settled Brunch prose is offered once as commentary; dropped, failed,
  or textless turns instruct Live to ask the person to continue.
• Keeps Brunch responsible for questions, answers, model operations, and history.
  Live has no tools. Its native audio remains best effort and is not treated as
  canonical output or proof of playback.
• Matches commentary acknowledgements and rejections by client_event_id, avoids
  truncation/retry/replay. Explicit Voice Stop, mode exit, panel close, and provider failure tear down both transports; stopping only the Brunch response keeps Voice listening.
• Reports Live phases and explicit recovery warnings through the host-owned Voice APIs from
  FE-1664: Refine host-owned Voice sessions in Petrinaut #9719. Provider-independent dock layout and error presentation live in that PR.
• Adds server, conversation, bridge, controller, UI, and browser-tool regression
  coverage plus the corresponding user and mission documentation.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• does not modify any publishable blocks or libraries, or modifications do not need publishing

This PR updates Petrinaut user documentation for the Live integration; #9719 owns the publishable library changes and changeset.

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Phantom input: a Spanish utterance was admitted while the tester was silent.
  Echo, background audio, and provider hallucination have not been distinguished.
• Lost input: server VAD can split hesitation into more finalized items than
  the existing one-waiting-input policy retains.
• Heuristic correlation: delegation timing does not prove that a delegation
  belongs to a particular transcription item.
• Best-effort speech: Live may paraphrase or speak independently before
  settlement. Commentary acceptance is not playback proof.
• Transcription fidelity, second-tab recovery, and migration readiness remain
  unproved.

🐾 Next steps

Run the structured acceptance steps with headphones. Use FE-1712's
speaker-versus-headphone comparison to classify phantom input, then reassess the
capture constraints and one-waiting-input policy.

🛡 What tests cover this?

• live-conversation.test.ts: dual-session lifecycle, transcription ordering,
  delegation correlation, rejection formats, timeout/recovery, and Stop.
• live-brunch-bridge.test.ts: admission, settlement, textless/failed turns,
  source freezing, and no replay.
• live-conversation-control.test.tsx: consent, Brunch wiring, dock states, and
  stale-message handling.
• openai-transcription-session.test.ts: client-secret/SDP exchange, redaction,
  limits, timeout, and abort behavior.
• Provider-independent Petrinaut Voice UI behavior is covered by FE-1664: Refine host-owned Voice sessions in Petrinaut #9719.
• Browser-tool integration: Voice input through canonical tool-backed Brunch work.

Local verification for this revision:

• seven affected website Live/Voice test files: 185 passed
• Petrinaut website typecheck and ESLint: passed
• the full website run additionally passed 729 tests; 13 unrelated fixture tests could not access localStorage in the local Node environment

These checks do not establish naturalness, echo safety, transcription fidelity, or
heard playback.

❓ How to test this?

Prepare the website dependencies, codegen, and examples, then start one provider:

turbo run build --filter '@apps/brunch-agent^...' --filter '@apps/petrinaut-website^...'
yarn workspace @apps/petrinaut-website codegen
yarn workspace @apps/petrinaut-website examples:generate

# Realtime baseline
PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=realtime yarn dev:brunch

# Brunch-backed Live experiment
PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=live yarn dev:brunch

Open http://localhost:4915/new, wear headphones, and:

1. Give an account and answer one Brunch clarification.
2. Hesitate, elaborate, and correct a consequential detail.
3. Request one available model operation and inspect the canonical result.
4. Speak during playback; separately test acoustic interruption and durable Stop.
5. Compare canonical history, actual tool effects, settled Brunch text, and audible
   output. Confirm ending Voice stops capture and playback.

📹 Demo

No shareable demo is available.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 15, 2026 3:22pm UTC
petrinaut	Ready	Preview	Sep 15, 2026 3:22pm UTC
petrinaut-docs	Ready	Preview	Sep 15, 2026 3:22pm UTC

1 Skipped Deployment

Project	Deployment	Actions	Updated
hashdotdesign-tokens	Ignored	Preview	Sep 15, 2026 3:22pm UTC

[CLAassistant]

All committers have signed the CLA.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[lunelson]

Code review findings:

1. Blocking: canonical Brunch Stop currently invokes the tracker stop subscription, and LiveConversationControl handles that by calling end(). This tears down both WebRTC sessions and clears consent, contradicting this PRs stated behavior that stopping only the Brunch response keeps Voice listening. The test around live-conversation-control.test.tsx:449 currently codifies the teardown. The Stop notification should cancel/settle the Brunch turn without ending Live.

2. Settlement race: LiveBrunchBridge requires turn.sawBusy before offering an otherwise correlated, completed settlement. A fast or batched ready-to-ready transition can avoid rendering submitted/streaming, leaving a completed canonical response permanently unoffered. The submission ID, response-completion event, settlement, and finalized canonical prose should be sufficient authorities.

3. Delegation leak: when two admissions share one answering submission, the first turn marks the segment offered; the second turn is deleted and silently skipped because every segment is already offered. Its distinct Live delegation receives neither commentary nor a continue instruction. The existing coalesced-answer test verifies once-only speech but does not verify resolution of both delegations.

4. Correlation edge: whitespace-only finalized input returns before consuming or resolving the newest unclaimed delegation. The next non-empty transcript can therefore claim a stale delegation. Please explicitly resolve/expire that delegation or prove why it cannot belong to the empty finalized item.

Focused Voice tests pass (5 files, 146 tests), but the first and third behaviors are currently encoded by tests rather than falsified by them.

[kostandinang]

@lunelson Addressed at the restacked head 6d188da. (1) Brunch Stop clears only the interrupted bridge turn and leaves Live/WebRTC plus consent active. (2) Completed correlated responses no longer depend on observing an intermediate busy render. (3) A coalesced turn whose prose was already delivered now receives a targeted continue-without-repeating instruction for its distinct delegation. (4) Empty finalized input consumes and resolves its delegation before later input. The focused Live regression set passes (99 tests across the conversation, bridge, and control suites).

[lunelson]

Follow-up remote review confirms the four earlier findings were addressed, but the Stop fix leaves one related delegation leak.

stopResponse() sets #chat.stopped = true and then calls #interruptTurns(). Because #interruptTurns() calls #unserved() only when !this.#chat.stopped, an in-flight turn with an attached Live delegation is cleared without commentary or continuation instructions while the Live session remains active. The provider can retain that unresolved delegation indefinitely. Unclaimed delegations are also left available for later correlation.

Please resolve attached/unclaimed delegations when stopping only the Brunch response, while reserving silent cleanup for full Voice-session shutdown. Add a regression that attaches a delegation, calls stopResponse(), and verifies it receives a targeted no-completion/continue instruction and cannot be claimed by the next utterance.

[kostandinang]

@lunelson Addressed in f6203c2. Response-only Stop now sends the targeted no-completion/continue instruction to every attached and unclaimed Live delegation before clearing turn correlation. Full Voice-session shutdown remains silent because stop() still aborts and clears directly. The regression covers both delegation states and verifies the next utterance receives its answer without claiming either stopped delegation.

[lunelson]

Re-reviewed the current remote head. The response-only Stop path now resolves attached and unclaimed Live delegations before clearing correlation, and the regression verifies a later utterance cannot inherit them. I found no remaining child-specific code issue. The parent activity-state concern remains owned by #9671.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d9b6d6a. Configure here.}

[cursor]

Stop drops Live tracking mid-submit

High Severity

stopResponse() tells Live the backend could not take the turn and removes it from tracking, but it never aborts the in-flight submit(). That call still uses the session abort signal, so a queued utterance spoken during playback is later admitted to Brunch with no Live correlation. The waiting-input slot is also cleared, so a later utterance can overlap that submit. Full stop() aborts; response-only Stop does not.

Additional Locations (2)

• apps/petrinaut-website/src/main/app/voice-interview/live-brunch-bridge.ts#L170-L198
• apps/petrinaut-website/src/main/app/voice-interview/live-brunch-bridge.ts#L274-L289

 

^{Reviewed by Cursor Bugbot for commit d9b6d6a. Configure here.}