FE-1663: Experiment Live Full-Duplex Migration

[kostandinang]

Note

Experiment 1: This PR evaluates a standalone GPT-Live interview using the existing Voice controls. Local testing passed. It does not execute application changes; domain-specific Brunch integration is in Experiment 2 (#9673).

🌟 What is the purpose of this PR?

Test whether GPT-Live-1 natively improves process interviewing: concise relevant
follow-ups, room for hesitation, and interruption/correction without losing meaning.
Smooth audio alone is not acceptance.

This first experiment is a labelled standalone conversation through the existing
Voice UX, not Brunch output or application execution. It asks whether the person
feels heard and the interview advances; full Brunch integration is deferred.

🔗 Related links

• FE-1663 (internal)
• Brunch base FE-1573 / #9667
• FE-1661 (internal; retained migration assessment)
• Petrinaut Voice platform #9719, stacked on this branch
• Child experiment FE-1664 / #9673, stacked on FE-1664: Refine host-owned Voice sessions in Petrinaut #9719
• Official Live migration guide
• Official Live prompting guide

🚫 Blocked by

• Manual conversation/media acceptance before merge.

🔍 What does this change?

• Adds an isolated Live controller and website session endpoint using existing
  Voice enablement/credentials. Creation starts WebRTC; wait for session.started
  without sending another session.start.
• Gives Live a purpose-led process-interviewer prompt structured around the official
  backchannel, interruption and delegation policies. It explicitly has no backend
  capabilities and cannot claim application execution or saved changes.
• Reuses Voice consent and the dock, with experimental wording only at consent.
  Local WebRTC audio levels drive Listening/Speaking and the microphone ribbon.
  Unsupported lifecycle/playback actions stay hidden; no separate experiment panel.
• Keeps transcript deltas and delegation metadata out of canonical submission.
  Local media stops immediately on End; remote closure is not guaranteed.
• Selects realtime or live through PETRINAUT_VOICE_PROVIDER, pinned for the
  mounted conversation. Unset remains Realtime; its prompts, admission, queue,
  settlement and playback remain unchanged. Live cannot submit transcripts,
  execute tools or insert canonical history. No automatic retry or input replay.

Pre-Merge Checklist 🚀

🚢 Has this modified a publishable library?

This PR:

• modifies an npm-publishable library and Kostandin has added a changeset file

Adds a patch changeset for @hashintel/petrinaut.

📜 Does this require a change to the docs?

The changes in this PR:

• require changes to docs which are made as part of this PR

Updates the website README, mission and assistant user guide. Existing Realtime
screenshots remain applicable; experimental dock captures are local review evidence.

🕸️ Does this require a change to the Turbo Graph?

The changes in this PR:

• do not affect the execution graph

⚠️ Known issues

• Kostandin manually tried the standalone experiment and reported that it looked
  fine; the exact tested revision was not supplied. This supports proceeding, not
  comprehensive acoustic, naturalness or Brunch compatibility acceptance.
  Model instructions are guidance, not an output guarantee.
• Lost connections can leave remote closure unconfirmed. No transcripts or recordings
  are retained. Dock activity is not a turn boundary or proof of heard playback.
  Browsers without audio-level telemetry retain Listening without an input meter.
• Existing unauthenticated Voice endpoint risk applies (FE-1622); keep the experiment local.
• The forced Turbo gate hit a sandbox restriction on tsx's Unix socket. Equivalent
  package checks ran directly against prepared dependencies with external network denied.
• A pre-commit website run failed six tests while generated Petrinaut build chunks
  changed underneath it. Rerunning against the settled build passed all 518 tests.

🐾 Next steps

Review and merge stack: main (including merged FE-1573 / #9667)
→ FE-1663 / #9671
→ Petrinaut Voice platform / #9719
→ FE-1664 / #9673
→ FE-1712 / #9704.
Domain-specific elicitation details, modelling rules and application execution are
deferred to experiment 2, Experiment Live Full Brunch Integration (see the
preserved integration scope).
This experiment tests general interviewing and native conversational behavior only;
it does not port Brunch's domain prompt or grant Live domain authority in experiment 2.
The second experiment's authority recut and implementation live in the child PR,
not this standalone comparison.

PR 2 must retain finalized input → composer → Flue → Brunch → authorized Petrinaut
execution → complete-turn settlement → frozen canonical reply → Live delivery.
Brunch alone authors domain answers/questions. Approval, settlement and playback
eligibility remain distinct; unknown outcomes are not replayed or spoken as success.

The child records the accepted experimental policy in a separate authority commit:
authoritative transcription plus native, unbuffered Live delivery with best-effort
speech control. Settlement gates supplied Brunch context, not all audible speech;
this is neither strict output gating nor exact relay. The child's reported connection
failure remains unresolved, and no live integrated exchange is claimed. Standalone
success does not reopen FE-1624. This PR adds neither an audio evaluation harness
nor PR 2 integration.

🛡 What tests cover this?

2026-09-14 restack: all eight parent commits are patch-equivalent to their
pre-restack versions. The focused Live/config suites pass (4 files, 47 tests), and
website typecheck/lint pass with no warnings. The full Turbo gate is not green:
website example generation fails in the generated Petrinaut Core HIR bundle with
TypeError: r.platform is not a function, including after rebuilding core. Core
source is unchanged from #9667. No provider session or microphone was started.

Previous, pre-restack evidence:

• Provider-free checks passed: 883 Petrinaut tests, 520 website tests, both package
  builds and typechecks. Lint has zero errors and one pre-existing Realtime effect
  warning. Formatting and diff checks passed.
• New mocked tests cover default/pinned provider selection, client-delegated
  handshake, consent, canonical isolation, optional dock actions, failure and
  cleanup. Activity tests cover simultaneous input/output, silence, unavailable
  telemetry and samples arriving after Stop. Realtime retains its regression coverage.
• Browser checks covered consent, mocked connection/error, collapse/expand, End,
  fresh consent, Exit and unchanged Realtime setup. Screenshots were inspected.
  The real Live controller was also exercised with mocked media and SDP to verify
  Listening/Speaking transitions and End cleanup in the existing dock.
  Verification made no real microphone or provider requests.

Package commands: test:unit (run for Petrinaut), lint:tsc, lint:eslint, build.
Kostandin's manual conversational/media acceptance remains outstanding.

❓ How to test this?

From this branch's repository root, with OPENAI_VOICE_API_KEY exported or in the
website's .env.local (full preparation commands are in the website README):

PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=live yarn dev:brunch:panel

Open http://localhost:4915/new → AI panel → Start voice mode → consent → Start voice.
Only the final action activates microphone/provider. Allow 10–15 minutes:

1. Explain a familiar process and allow follow-ups.
2. Hesitate, pause, answer briefly, then elaborate.
3. Interrupt mid-sentence and correct an earlier detail.
4. Speak during output; note lost meaning, verbosity and acknowledgements.
5. End voice mode; verify microphone indicator and speaker silence. Stop the dev
   command, then start Realtime with the existing Brunch backend environment:

PETRINAUT_OPENAI_VOICE_ENABLED=true PETRINAUT_VOICE_PROVIDER=realtime yarn dev:brunch

Reload for a new session. Judge feeling heard/interview progress separately from
transport correctness. Prior experiments do not constitute a passing baseline.

📹 Demo

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hash	Ready	Preview	Sep 15, 2026 2:20pm UTC
petrinaut	Ready	Preview	Sep 15, 2026 2:20pm UTC
petrinaut-docs	Ready	Preview	Sep 15, 2026 2:20pm UTC

1 Skipped Deployment

Project	Deployment	Actions	Updated
hashdotdesign-tokens	Ignored	Preview	Sep 15, 2026 2:20pm UTC

[cursor]

PR Summary

Medium Risk
New billable Live WebRTC path reuses the same unauthenticated voice endpoint boundary as Realtime, plus substantial client media lifecycle code; Realtime is isolated by provider pinning but Petrinaut dock/session APIs changed for all hosts.

Overview
Adds an experimental GPT-Live-1 path selected by PETRINAUT_VOICE_PROVIDER=live, alongside unchanged Realtime. A new /api/voice/live-session handler validates same-origin SDP, creates a client-delegated WebRTC session with a standalone process-interviewer prompt (no tools or Brunch), and returns the answer SDP. Voice config now exposes provider and treats invalid values as unavailable.

On the client, createLiveConversation runs the full-duplex WebRTC loop: wait for session.started (never send session.start), ignore transcript/delegation events, drive the existing dock from local getStats activity, recover brief disconnects on the same session without replay, and stop with bounded session.close / cleanup. Blocked autoplay surfaces a notice and retryPlayback without restarting the session. LiveConversationControl wires consent, host session state, and minimal dock actions (end / pause-as-end / retry playback) with no canonical chat, tools, or admission.

Petrinaut gains registerVoiceModeSessionControls so hosts can omit Realtime-only actions (mute, resume, reconnect); the dock shows Play voice audio when needed and a neutral connected phase. Collapsed Voice reports aiAssistantDockHeight so viewport controls sit above the compact dock instead of reserving full panel width. Realtime behavior and the full registerVoiceModeControls contract stay as-is; provider choice is pinned per mounted conversation.

^{Reviewed by Cursor Bugbot for commit c5f816a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[lunelson]

Code review concern (non-blocking for the experiment, but relevant to the published package): this changes PetrinautAiVoiceModeControls.reconnect, .resume, and .setMicrophoneMuted from required to optional. That lets Live omit unsupported controls, but it is source-breaking for existing typed consumers that call those methods without narrowing, while the changeset is a patch. Could we preserve the existing control contract and introduce a capability/session-specific variant, or otherwise document and test the compatibility boundary?

[kostandinang]

@lunelson Addressed in 06b4f30. PetrinautAiVoiceModeControls again keeps reconnect, resume, and setMicrophoneMuted required for source-compatible typed consumers. Session registration now accepts a separately exported PetrinautAiVoiceModeSessionControls capability subset, with a type-level regression test, updated changeset, and a corrected website registration harness.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[lunelson]

Follow-up remote review found two remaining issues:

1. Blocked playback remains silent and connected. live-conversation.ts now enables autoplay but swallows both synchronous and asynchronous audio.play() failure. Browser autoplay-policy rejection is not retried merely because autoplay remains true, so the microphone can keep transmitting while the user hears nothing and sees a connected session. Please surface an actionable warning and provide a user-gesture playback retry, or stop safely. The current regression proves connection survival, not restored playback.

2. The public context remains source-incompatible. Restoring required members on PetrinautAiVoiceModeControls fixes direct users of that type, but PetrinautAiVoiceModeContext.registerVoiceModeControls changed from accepting complete controls to accepting PetrinautAiVoiceModeSessionControls. Under strict function variance, an existing context implementation/mock whose callback accepts required controls is no longer assignable because the new contract may pass partial controls. Please preserve the old context signature and expose provider-capability registration through a separate compatible seam, or add a type regression covering assignability of an existing complete context implementation.

[kostandinang]

@lunelson Addressed in 87659d4. Blocked Live playback now keeps an actionable dock warning, exposes a Play voice audio user-gesture retry, preserves the warning through activity updates, and clears it only after playback succeeds. The public context restores registerVoiceModeControls(PetrinautAiVoiceModeControls) and exposes partial provider capabilities through a separate optional registerVoiceModeSessionControls seam, with a strict-variance assignability regression.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 87659d4. Configure here.}

[lunelson]

Re-reviewed the current remote head. Playback failure and recovery now preserve the latest activity state, the user-gesture retry remains actionable, and the public Voice registration contract is source-compatible. I found no remaining code issue; all current checks pass.