Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/activity-without-nested-scroll.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@hashintel/petrinaut": patch
---

Show every reasoning step and tool in the Brunch Activity fold instead of clipping them in a nested scroll area, so the visible list always matches the tool count.
5 changes: 5 additions & 0 deletions .changeset/brunch-activity-labels.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@hashintel/petrinaut": patch
---

Show activity wording instead of naming Brunch during a turn. The waiting row and the default working status read "Working…", and the Voice brief reads "Preparing request", "Sending request", then "Request sent".
5 changes: 5 additions & 0 deletions .changeset/brunch-activity-stays-put.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@hashintel/petrinaut": patch
---

Keep the Brunch Activity row in place through a turn. A spoken Live reply no longer takes over the Working… row, and the row's label no longer shifts when the spinner gives way to the Activity icon.
5 changes: 5 additions & 0 deletions .changeset/brunch-tab-fade.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@hashintel/petrinaut": patch
---

Stop Brunch assistant tabs inheriting the stock tab fade, so the Brunch selected and hover treatment applies on its own.
5 changes: 5 additions & 0 deletions .changeset/brunch-waiting-in-activity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@hashintel/petrinaut": patch
---

Show Brunch's waiting state in the transcript, where the turn's Activity will appear, and keep the spinner in the Activity header while Brunch works. The status row above the composer now appears only while the host tab is shown.
5 changes: 5 additions & 0 deletions .changeset/stock-assistant-parity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@hashintel/petrinaut": patch
---

Keep the stock AI assistant presentation unchanged by the Brunch conversation UI: chronological messages, stock tool rows, tabs, scrolling and composer. A tool stopped mid-run now shows Cancelled instead of a spinner.
Original file line number Diff line number Diff line change
Expand Up @@ -427,7 +427,7 @@ describe("local storage demo Brunch voice integration", () => {
]),
);
expect(aiAssistant.resolveToolPresentation).toBeTypeOf("function");
expect(aiAssistant.workingLabel).toBe("Brunch is working");
expect(aiAssistant.workingLabel).toBe("Working…");
expect(
aiAssistant.resolveToolPresentation?.({
toolName: "layout_petrinaut_net",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -969,7 +969,7 @@ export const LocalStorageDemoApp = ({
presentation: "brunch" as const,
mapMessagesForDisplay: mapVoiceMessages,
resolveToolPresentation: resolveBrunchToolPresentation,
workingLabel: "Brunch is working",
workingLabel: "Working…",
renderComposerControl: (
context: PetrinautAiComposerControlContext,
) => <BrunchExperimentFollowUp context={context} />,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -472,7 +472,7 @@ test("drops a superseded transcript quietly when another turn holds the composer
await current;
});

test("a turn keeps its annotations in the history it began in when the conversation switches", async () => {
test("does not admit a turn prepared in a conversation that was switched away from", async () => {
const brief = Promise.withResolvers<Record<string, string>>();
const original = new VoiceMediationHistory("original");
const switched = new VoiceMediationHistory("switched");
Expand All @@ -485,25 +485,66 @@ test("a turn keeps its annotations in the history it began in when the conversat
summarize: vi.fn(),
offered: vi.fn(),
});
fixture.bridge.acceptDelegation("delegation");
const turn = fixture.bridge.accept(
speech("one", "Compare two to eight agents"),
);
current = switched;
const fields = { decide: "two to eight agents" };
brief.resolve(fields);
brief.resolve({ decide: "two to eight agents" });
await turn;
const admitted = [

expect(fixture.submit).not.toHaveBeenCalled();
expect(original.project([])).toEqual([]);
expect(switched.project([])).toEqual([
{
id: "one",
role: "user" as const,
parts: [{ type: "text" as const, text: "Brief" }],
role: "user",
metadata: { source: "voice" },
parts: [{ type: "text", text: "Compare two to eight agents" }],
},
];
expect(original.project(admitted)[0]?.parts[1]).toEqual({
type: "data-brief",
data: { fields, state: "done" },
]);
expect(fixture.notice).toHaveBeenLastCalledWith(
expect.stringContaining("the conversation changed"),
);
expect(fixture.appendInstructions).toHaveBeenCalledWith(
expect.stringContaining("not submitted because the conversation changed"),
"delegation",
);

await fixture.bridge.accept(speech("two", "Then check the queue"));
expect(fixture.submit).toHaveBeenCalledOnce();
});

test("stops waiting on an admitted turn once its conversation is switched away from", async () => {
const original = new VoiceMediationHistory("original");
const switched = new VoiceMediationHistory("switched");
let current = original;
const fixture = setup({
get history() {
return current;
},
prepare: async () => ({}),
summarize: vi.fn(async (text: string) => text),
offered: vi.fn(),
});
fixture.bridge.acceptDelegation("delegation");
await fixture.bridge.accept(speech("one", "Compare two to eight"));
current = switched;
fixture.update({ messages: [] });

expect(fixture.appendInstructions).toHaveBeenCalledWith(
expect.stringContaining("answer stays in the original conversation"),
"delegation",
);
fixture.bridge.responseStarted(started);
fixture.bridge.responseCompleted({
...started,
position: { batch: 2, index: 0 },
});
expect(switched.project(admitted)[0]?.parts).toHaveLength(1);
fixture.update({ segments: [segment()], settlements: completed });
await Promise.resolve();
expect(fixture.mediation.summarize).not.toHaveBeenCalled();
expect(fixture.appendCommentary).not.toHaveBeenCalled();
});

test("speech cancels preparation and stale asynchronous wrap-ups without cancelling admitted Brunch work", async () => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,56 @@ export class LiveBrunchBridge {
this.#waitingForComposer = undefined;
}

/**
* Submission follows the conversation shown now, so words prepared in
* another one are not admitted here. Like words cancelled by speech, they
* stay sendable in the conversation shown now.
*/
#withdrawSwitched(turn: Turn): void {
this.#turns.delete(turn);
this.#preparations.delete(turn.preparation);
turn.history.failed(turn.inputId);
this.#dependencies.mediation.history.unsent(
turn.inputId,
turn.inputText,
this.#chat.messages?.at(-1)?.id,
);
logLiveDiagnostic("brunch.conversation-switched", {
inputId: turn.inputId,
delegationId: turn.delegationId,
submitted: false,
});
this.#dependencies.notice(
"Your earlier utterance was not sent because the conversation changed. Use the composer to send it.",
);
this.#unserved(
turn.delegationId,
"The request was not submitted because the conversation changed.",
);
}

/** Settlement follows the conversation shown now, where a turn submitted elsewhere never settles. */
#releaseSwitchedTurns(): void {
const history = this.#dependencies.mediation.history;
for (const turn of this.#turns) {
if (!turn.submitted || turn.history === history) continue;
this.#turns.delete(turn);
this.#preparations.delete(turn.preparation);
if (this.#waitingForComposer === turn)
this.#waitingForComposer = undefined;
logLiveDiagnostic("brunch.conversation-switched", {
inputId: turn.inputId,
submissionId: turn.submissionId,
delegationId: turn.delegationId,
submitted: true,
});
this.#unserved(
turn.delegationId,
"The conversation was switched before Brunch finished; its answer stays in the original conversation.",
);
}
}

/** Frees the composer from one stale turn; other turns and delegations stay. */
#evict(turn: Turn): void {
turn.preparation.abort();
Expand Down Expand Up @@ -394,6 +444,10 @@ export class LiveBrunchBridge {
logLiveDiagnostic("brief.unavailable", { inputId: input.id });
turn.history.preparationFailed(input.id);
}
if (turn.history !== this.#dependencies.mediation.history) {
this.#withdrawSwitched(turn);
return;
}
logLiveDiagnostic("brunch.submit", { inputId: input.id, delegationId });
turn.submitted = true;
const result = await this.#dependencies.submit({
Expand Down Expand Up @@ -499,6 +553,7 @@ export class LiveBrunchBridge {
const enteredError =
chat.status === "error" && this.#chat.status !== "error";
this.#chat = chat;
this.#releaseSwitchedTurns();
if (stopped || enteredError) {
this.#interruptTurns(stopped ? "stopped" : "error");
return;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -482,13 +482,13 @@ test("starts acknowledged Live after the previous session finishes stopping", ()

rerender(<VoiceInterviewControl {...props} config={config} />);

expect(screen.getByText("Stopping voice…")).toBeTruthy();
expect(screen.queryByText("Voice disconnected")).toBeNull();
expect(screen.queryByRole("region", { name: "Voice mode retry" })).toBeNull();
expect(
screen
.getByRole("button", { name: "Retry voice" })
.hasAttribute("disabled"),
).toBe(true);
screen.queryByRole("region", { name: "Voice mode consent" }),
).toBeNull();
expect(props.reportVoiceSessionState).toHaveBeenLastCalledWith(
expect.objectContaining({ phase: "connecting" }),
);
expect(createLiveConversation).toHaveBeenCalledOnce();
act(() =>
onState({
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,7 @@ export const LiveConversationControl = ({
);
const sessionActive = useRef(false);
const handledVoiceSelection = useRef(false);
const [startAwaitingStop, setStartAwaitingStop] = useState(false);
const bridge = useRef<LiveBrunchBridge | null>(null);
const latest = useRef({
submit,
Expand Down Expand Up @@ -367,6 +368,7 @@ export const LiveConversationControl = ({
useLayoutEffect(() => {
if (inputMode !== "voice" || !isAiAssistantOpen) {
handledVoiceSelection.current = false;
setStartAwaitingStop(false);
return;
}
if (handledVoiceSelection.current) return;
Expand All @@ -375,12 +377,17 @@ export const LiveConversationControl = ({
return;
}
handledVoiceSelection.current = tryStartLiveConversation();
setStartAwaitingStop(!handledVoiceSelection.current);
}, [
disclosureAcknowledged,
inputMode,
isAiAssistantOpen,
tryStartLiveConversation,
]);
// Reopening Voice while the previous session is still closing starts the
// next one as soon as it ends, so that wait is part of connecting.
const sessionPhase =
startAwaitingStop && phase === "stopping" ? "connecting" : phase;
const setMicrophoneMuted = useCallback((muted: boolean) => {
if (!sessionActive.current || !session.current) return;
session.current.setMicrophoneMuted(muted);
Expand Down Expand Up @@ -443,13 +450,15 @@ export const LiveConversationControl = ({
reportVoiceSessionState(
inputMode === "voice" &&
isAiAssistantOpen &&
(phase === "connecting" || phase === "connected" || phase === "error")
(sessionPhase === "connecting" ||
sessionPhase === "connected" ||
sessionPhase === "error")
? {
audioSettings,
phase:
phase === "error"
sessionPhase === "error"
? "error"
: phase === "connecting"
: sessionPhase === "connecting"
? "connecting"
: activity?.outputActive
? "speaking"
Expand Down Expand Up @@ -478,6 +487,7 @@ export const LiveConversationControl = ({
inputMode,
isAiAssistantOpen,
phase,
sessionPhase,
message,
playbackBlocked,
activity,
Expand Down Expand Up @@ -508,7 +518,11 @@ export const LiveConversationControl = ({
};
}, [reportVoiceSessionState, setVoiceActive]);

if (inputMode !== "voice" || phase === "connecting" || phase === "connected")
if (
inputMode !== "voice" ||
sessionPhase === "connecting" ||
sessionPhase === "connected"
)
return null;
const exitVoiceMode = () => {
void end();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ export const toVoiceSessionState = ({
microphoneLevel: snapshot.microphoneLevel,
notice:
snapshot.inputNotice === "answer-pending"
? "Answer captured. Waiting for Brunch."
? "Answer captured. Working on it."
: snapshot.inputNotice === "answer-already-pending"
? "Previous answer waiting. Please try again after it is sent."
: snapshot.inputNotice === "not-heard"
Expand Down
33 changes: 17 additions & 16 deletions libs/@hashintel/petrinaut/docs/ai-assistant.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,24 +37,25 @@ In the stock assistant, an empty conversation asks you to **Describe the process

The primary tab reads **AI** unless the host names it. In Brunch it reads **Chat**, or **Voice** while Voice mode is selected. Your messages appear in right-aligned grey bubbles. Each Brunch turn groups its work in a neutral disclosure, followed by a light-blue answer and any produced cards. In Voice, the written Brunch answer is inside the work disclosure instead. The stock assistant keeps plain answers and a horizontally scrolling row of suggestion chips; Brunch's chips wrap to fit the panel.

Before Brunch starts a response, **Waiting for Brunch** and a small blue spinner
appear above the composer or Voice dock. This status uses reserved space, so
appearing or clearing it does not move the transcript or input controls.
Suggestion chips keep their space while hidden and cannot be activated during
work. Once the response arrives, its **Working…** header takes over; there is no
duplicate status or information button below it. The spinner stays still with
reduced motion enabled. While viewing Ledger, the reserved row reports
**Brunch is working** instead. The stock assistant shows its working label without this reserved Brunch status row.
As soon as you send, **Working…** and a small blue spinner appear in the
transcript, where that turn's activity will appear, below any spoken Voice
reply. Once the response arrives, its **Working…** activity header takes over
in the same place and keeps the spinner until the work ends; there is no
duplicate status below it. Suggestion chips keep their space while hidden and
cannot be activated during work. The spinner stays still with reduced motion
enabled. While viewing Ledger, a status row above the composer reports
**Working…** instead.
The stock assistant shows its working label without these Brunch statuses.

Completed Chat answers offer **Copy** and **Retry**. The latest answer keeps these controls visible; older answers reveal them on hover or keyboard focus. Touch screens keep them visible. Copy keeps the answer's Markdown and briefly shows a check mark after copying succeeds. Retry sends that answer's original prompt as a new turn, keeping the previous answer and any unsent draft. It can lead to new tool calls, just like sending the prompt yourself. Retry is unavailable while another response or voice handoff is active.

Brunch opens activity and streaming thoughts when work starts. Activity, thoughts,
and tools keep your open/closed choices as more text arrives, tools finish, or the
response completes. Long activity scrolls inside a bounded area instead of
continually pushing the answer down. **Working…** stays until the response ends,
then becomes **Activity · Ns**; history without timing says **Activity**. Previously
completed turns start collapsed. A new **Approval required** request reveals its
controls and removes the height limit so the decision is not hidden. Expand
response completes. Every thought and tool stays in the transcript, so long
activity scrolls with the conversation rather than inside its own area. **Working…**
stays until the response ends, then becomes **Activity · Ns**; history without
timing says **Activity**. Previously completed turns start collapsed. A new
**Approval required** request reveals its controls. Expand
activity to inspect **Thought for Ns** and **Used N tools**. Other hosts retain
their collapsed reasoning default and automatic work/tool collapsing. Stopped
work says **Stopped after N tools** and retains a **Response stopped** note.
Expand All @@ -65,7 +66,7 @@ pending, completed, and failed tools.

Before Brunch removes model elements, an approval lists the requested removals. Associated arcs or references may also be removed. **Allow** applies that removal; **Deny** withholds that call and tells Brunch nothing was changed. Brunch's later calls in the same response wait until you answer, then continue to run. **Always allow** permits later removals only in the current mounted conversation, until you leave or reload. It does not grant permission for another conversation or browser session. Stop cancels a pending approval. Auto-layout asks separately; see `applyAutoLayout` below.

When the host supplies them, Voice also shows a collapsed brief directly under your message, an immediate spoken-agent reply before the work, and a wrap-up after the produced cards. The brief says **Preparing for Brunch** while its fields are being prepared, **Sending to Brunch** once the fields are ready but not yet accepted, and **Sent to Brunch** after acceptance. Expand a prepared brief to see **Prepared from what you said** and its right-aligned fields. Missing excerpts display **Still open**; that placeholder is not sent as an extracted fact. If preparation fails, your original words are sent without prepared fields: the brief says **Sending without preparation**, then **Sent without preparation** after acceptance. These optional parts are absent in hosts that do not provide them. In Chat, a small neutral voice-bars icon marks user messages sent using Voice; typed messages have no icon. In Voice, those per-message icons are hidden.
When the host supplies them, Voice also shows a collapsed brief directly under your message, an immediate spoken-agent reply before the work, and a wrap-up after the produced cards. The brief says **Preparing request** while its fields are being prepared, **Sending request** once the fields are ready but not yet accepted, and **Request sent** after acceptance. Expand a prepared brief to see **Prepared from what you said** and its right-aligned fields. Missing excerpts display **Still open**; that placeholder is not sent as an extracted fact. If preparation fails, your original words are sent without prepared fields: the brief says **Sending without preparation**, then **Sent without preparation** after acceptance. These optional parts are absent in hosts that do not provide them. In Chat, a small neutral voice-bars icon marks user messages sent using Voice; typed messages have no icon. In Voice, those per-message icons are hidden.

Hosts that provide live input captions can show your words while you speak. This partial text is display-only: it does not submit work or start preparing a brief. The finalized transcript replaces it in the same bubble before preparation starts. New spoken words and status labels fade in; reduced-motion preferences disable these effects.

Expand Down Expand Up @@ -130,7 +131,7 @@ history, and changes to the net. In the website's Live mode, your complete
finalized words go to Brunch alongside excerpts from a prepared brief, so short
replies, corrections and requests to use defaults are not lost during preparation.
If the brief cannot be prepared, your words are sent without excerpts.
Brunch uses the prior conversation to interpret them. Expand **Sent to Brunch**
Brunch uses the prior conversation to interpret them. Expand **Request sent**
to inspect the brief. **Still open** means a detail was not extracted from this
turn, not that a previous answer has been forgotten. Those placeholders are not
sent to Brunch. The brief is not editable before sending in this version.
Expand Down Expand Up @@ -299,7 +300,7 @@ pause, or an error makes playback unsafe.
With **Allow interruptions** enabled, start speaking while Brunch is
talking to stop its audio and give your answer. Your interrupting words are
captured; you do not need to repeat them. If Brunch is still finishing its
previous turn, the dock shows **Answer captured. Waiting for Brunch.** and
previous turn, the dock shows **Answer captured. Working on it.** and
sends that answer when it is ready. Wait for it to be sent before giving
another one. Disable **Allow interruptions** to use manual handover. In
manual mode, select **Your turn**, wait for cancellation to finish, then speak;
Expand Down
Loading
Loading