From 315ae97d3361627a98162043a440dccc116c8222 Mon Sep 17 00:00:00 2001 From: Steven Roussey Date: Mon, 12 Oct 2026 05:34:45 +0000 Subject: [PATCH 1/7] ai: AgentTask deadlines, time-left notices, review, reasoning and native-turn storage - Each tool's ToolExecuteContext carries the turn's deadline. announceTimeLeft appends the time left to each round's last tool result. A retry wait that would pass the deadline ends the turn "budget". - reviewPrompt asks the model for one check of its answer before the turn ends. - Assistant turns keep their reasoning, and a provider's own turn ({ provider, payload }) rides on the reasoning block so the provider can replay it; history trimming keeps the current turn whole and measures earlier ones without their payloads. - Tool results may carry images. Empty tool results and reasoning-only turns are sent as strings, and Ollama and llama.cpp server histories are converted instead of forwarded as content blocks. - Each round's reasoning streams on the task's own reasoning port, held with the text until the attempt settles; AgentRoundContext gains onReasoningDelta. --- .claude/CLAUDE.md | 89 +++- packages/ai/src/task/AgentRoundRunner.ts | 6 + packages/ai/src/task/AgentTask.ts | 256 +++++++++++- packages/ai/src/task/AgentToolExecution.ts | 135 ++++-- packages/ai/src/task/AiChatTask.ts | 1 + packages/ai/src/task/ChatHistory.ts | 53 ++- packages/ai/src/task/ChatMessage.ts | 44 +- packages/ai/src/task/MessageConversion.ts | 192 ++++++++- packages/ai/src/task/ToolCallingTask.ts | 77 +++- packages/ai/src/task/ToolCallingUtils.ts | 18 + .../__tests__/AgentDeadlineSignal.test.ts | 39 ++ .../task/__tests__/CacheCheckpoint.test.ts | 52 +++ .../ai/src/task/__tests__/ChatHistory.test.ts | 79 ++++ packages/test/src/test/ai/AgentTask.test.ts | 136 ++++++- .../ai/AgentTaskDeadlineAndReview.test.ts | 352 ++++++++++++++++ .../src/test/ai/AgentTaskNativeTurn.test.ts | 133 ++++++ .../ai/AgentTaskReasoningAndImages.test.ts | 379 +++++++++++++++++ .../src/test/ai/AgentTaskTurnControls.test.ts | 36 ++ .../src/test/task/MessageConversion.test.ts | 383 +++++++++++++++++- 19 files changed, 2384 insertions(+), 76 deletions(-) create mode 100644 packages/ai/src/task/__tests__/AgentDeadlineSignal.test.ts create mode 100644 packages/test/src/test/ai/AgentTaskDeadlineAndReview.test.ts create mode 100644 packages/test/src/test/ai/AgentTaskNativeTurn.test.ts create mode 100644 packages/test/src/test/ai/AgentTaskReasoningAndImages.test.ts diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 7c83846fee..3b85dab4ea 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -243,15 +243,52 @@ The rest are controls a batch host needs: `toolConcurrency` (opt-in; a round hol put to a person still runs one at a time, and results return in the order asked), `maxRoundRetries` (default 2, for a `RetryableJobError`, waiting as the provider's retry-after says within 1–60 s), `roundTimeoutMs` (a provider can accept a request and never answer; -past it the round is abandoned as a retryable failure, so the retries cover it; a round's text is -forwarded only once the attempt settles, since a failed attempt's partial cannot be taken back -from the accumulated text port), and budgets — `maxInputTokens`, `maxCostUsd` (refused -without a price card, since a budget it cannot measure never stops anything, and failing closed -for a round that reports no usage: it cannot be priced, so the turn ends `"budget"` after it), +past it the round is abandoned as a retryable failure, so the retries cover it; a round's text and +reasoning are forwarded only once the attempt settles, since a failed attempt's partial cannot be +taken back from the accumulated port; reasoning goes out on the turn's own `reasoning` port, a +blank line between rounds, and is absent from the output when no round returned any), and +budgets — `maxInputTokens`, `maxCostUsd` (refused without a price card, +since a budget it cannot measure never stops anything, and failing closed for a round that reports no usage: it cannot be priced, so the turn ends `"budget"` after it), `maxDurationMs` — each ending the turn `"budget"`, never between a `tool_use` and its result. Every round leaves an `AgentStep` on `steps` (timings, attempts, each tool's outcome and size, usage, cost) and rides on the `snapshot` beside `messages`; `costUsd` totals them when every round could be priced. +A tool with an `execute` function is handed the turn's `deadline` on its `ToolExecuteContext` +(undefined without `maxDurationMs`; a task-backed tool never receives it): the loop stops only +between rounds, so a tool that can bound its own work caps it there. A retry wait that would pass +the deadline ends the turn `"budget"` instead of sleeping. `announceTimeLeft` appends the time +left to the text of each round's last tool result, and the notice stays in the returned +`messages` — the model cannot budget time it is never told about, and a long command started in +the last minutes is how a turn runs past its budget. +`reviewPrompt` is sent once per turn, the first time the model replies without a tool call, so it +checks its answer against what it was asked before the turn ends; the model may go back to its +tools. A chat host that passes it on every turn gets a review each turn. It is skipped with +`outputSchema` (that is `checkSubmission`'s job) and when no round, budget or time is left. The +turn's `text` then carries both replies. + +**A provider can be handed its own turn back.** A run-fn that can replay its own output emits it +once per round on `ToolCallingTask`'s `nativeTurn` port (`{ provider, payload }`), and `AgentTask` +keeps it on the assistant message's `reasoning` block (`provider`, `payload`). That provider's +converter then replays the payload verbatim instead of rebuilding the turn — the OpenAI Responses +items with their encrypted reasoning, Anthropic's content blocks with their signed thinking — and +only while the turn's call ids still match it (by count and order: each call's id, name and arguments, and for Anthropic the turn's text too), so +a host that changed or dropped a call id gets the rebuilt turn. The `provider` is `openai:` +or `anthropic:` (the provider name and its API model name), because encrypted reasoning and +signed thinking are readable only by the model that wrote them, so a turn is replayed only to that +model and rebuilt after a model switch. A reasoning model handed nothing back starts its thinking +over every round, spending output tokens re-deriving it. The payload stays on the message, so a +turn that is not truly replayable must not be emitted at all: OpenAI emits one only when encrypted +reasoning was requested (`include: ["reasoning.encrypted_content"]`, sent only for a reasoning +model, that is when reasoning effort is not `none`) and every reasoning item came back with it, +since a reasoning item replayed without it is looked up by id and fails where nothing is stored. +Anthropic emits none when it cannot capture the turn verbatim — a block or delta type it does not +reproduce, a thinking block without its signature, or an empty text block dropped ahead of a +thinking block. Anthropic binds a thinking block's signature to the conversation before it, so +the replay must be append-only; a request it rejects as edited is retried once with every thinking +block stripped. History trimming measures a message without its payload. +`provider_config.replay_reasoning: false` turns the OpenAI replay off; Anthropic has no such +switch. DeepSeek's `replay_reasoning` is a different mechanism: it controls replaying the +`reasoning_content` text, not a native turn. **The loop and the model call can live in different places.** A host whose tools are closures (they draw on a screen, ask a person, read state only that process holds) but whose model is @@ -422,6 +459,22 @@ Rules for writing a run-fn: discriminate on a field one caller always sends and the other never does (e.g. `Array.isArray(input.messages) && input.messages.length > 0`). +**Reasoning travels between rounds, and each provider carries it its own way.** DeepSeek and +the llama.cpp server replay `reasoning_content`; Ollama replays `thinking`; HF Transformers +hands `reasoning_content` to the chat template; node-llama-cpp replays thought segments. +OpenAI and Anthropic replay a native turn, and OpenRouter carries `reasoning_details` as one +keyed `openrouter:`; all three replay it only while the turn's calls still match (each call's +id, name and arguments, in order), so a host that changed or dropped one gets the rebuilt turn, and OpenRouter emits none for Anthropic +thinking cut off before its signature, or for a turn one of whose calls it filtered out. **A tool-calling run-fn converts `messages` with a +converter that keeps tool calls and prefers `messages` over `prompt`:** `AgentTask` sends both +every round, and a converter that reads `prompt` or flattens to text shows the model its first +turn again and never a tool result. A history holding tool calls or tool results must never go +through `toTextFlatMessages`; HF Transformers still uses it, but only on the path where the +history has no tool messages and no reasoning yet, where it loses nothing. Tool-result images go +in a user message after the tool message (OpenAI, xAI, OpenRouter, HF Inference, Ollama); Gemini 3+ takes them in +`functionResponse.parts`, and older models, which reject that with HTTP 400, get a user turn of +their own — beside the response they leak their thinking into the answer. + ### `examples/cli` — the CLI and its web console `workglow web` (`src/web/`, client in `src/web/client/`) serves the same commands the @@ -539,6 +592,32 @@ The client bundles via `bun run build-web` into `dist/web` and is part of `build deliberately non-render-blocking; a blocking font link blanks the console on any machine that cannot reach the CDN. +### `examples/agent-eval` — the harness comparison + +This example runs `AgentTask`, opencode and pi on public benchmarks (Terminal-Bench 2.0, +Aider Polyglot, SWE-bench Verified) through **Harbor**, with the same model for all three, and +compares them task by task. Harbor is a Python tool and needs Docker; it ships the opencode and +pi adapters. + +What the Workglow arm is: + +- **The agent.** `src/bin/workglow-agent.ts` is `AgentTask` plus four coding tools (`read`, + `bash`, `edit`, `write`). The tools match pi's defaults and limits on purpose, so the + Workglow–pi gap measures the loop and not the tools. +- **How it gets into the container.** It is bundled into one Node file. `harbor/workglow_agent.py` + uploads that bundle into each task container, so a run measures the checkout it was built + from: rebuild with `bun run build-agent` after touching `AgentTask`. + +Two rules hold for every run: + +- **Parity.** `run` sets the turn cap and the reasoning level explicitly for every arm, because + the three harnesses' defaults differ enough to decide the result. +- **Blocked trials.** A trial whose container or install failed is _blocked_, and is excluded + from every rate rather than counted as a failure. + +`mock-model` is a scripted Anthropic API. With it, the whole pipeline runs offline with no key +and no cost. + ### `@workglow/util` `EventEmitter`, `ServiceRegistry` (DI), `DirectedAcyclicGraph`, `DataPortSchema`/`JsonSchema`, diff --git a/packages/ai/src/task/AgentRoundRunner.ts b/packages/ai/src/task/AgentRoundRunner.ts index 8707ce924a..5d64e482d7 100644 --- a/packages/ai/src/task/AgentRoundRunner.ts +++ b/packages/ai/src/task/AgentRoundRunner.ts @@ -13,6 +13,12 @@ export interface AgentRoundContext { readonly signal: AbortSignal; /** The round's text as it streams, re-emitted as the turn's own. */ readonly onTextDelta: (delta: string) => void; + /** + * The round's reasoning as it streams, re-emitted as the turn's own on its + * `reasoning` port. Held with the round's text until the attempt settles, so + * a round that fails and is retried contributes neither. + */ + readonly onReasoningDelta: (delta: string) => void; /** * Progress while the round has nothing to say yet — a model being downloaded * or loaded. `progress` is 0..100, or undefined when there is no denominator. diff --git a/packages/ai/src/task/AgentTask.ts b/packages/ai/src/task/AgentTask.ts index c025327b2f..d0f60bd7e0 100644 --- a/packages/ai/src/task/AgentTask.ts +++ b/packages/ai/src/task/AgentTask.ts @@ -48,7 +48,9 @@ import { import type { ChatMessage, ContentBlock, + ContentBlockImage, ContentBlockInToolResultBody, + ContentBlockText, ContentBlockToolResult, } from "./ChatMessage"; import { ChatMessageSchema } from "./ChatMessage"; @@ -85,6 +87,9 @@ const DEFAULT_MAX_TOOL_RESULT_CHARS = 20_000; */ const DEFAULT_MAX_ROUND_RETRIES = 2; +/** Between two rounds' reasoning, which would otherwise run together into one thought. */ +const REASONING_ROUND_SEPARATOR = "\n\n"; + /** * Times a turn with an `outputSchema` reminds a model that replied in text to * submit instead. Some models narrate a final answer out of habit and submit @@ -158,6 +163,8 @@ export const AgentInputSchema = { description: "Time one model call may take before it is abandoned and retried as a retryable failure", minimum: 1, + // The longest timer that can be set; past it the round would be abandoned at once. + maximum: 2_147_483_647, "x-ui-group": "Configuration", }, toolConcurrency: { @@ -200,6 +207,20 @@ export const AgentInputSchema = { minimum: 1, "x-ui-group": "Configuration", }, + announceTimeLeft: { + type: "boolean", + title: "Announce Time Left", + description: + "Append the time left before maxDurationMs to the last tool result of every round, so the model can plan the work it has time for. Needs maxDurationMs", + "x-ui-group": "Configuration", + }, + reviewPrompt: { + type: "string", + title: "Review Prompt", + description: + "Sent once, as a user message, the first time the model replies without calling a tool, so it checks its work before the turn ends. Not sent on a turn with outputSchema, or when no round, budget or time is left to act on it", + "x-ui-group": "Configuration", + }, approval: { type: "string", title: "Approval", @@ -223,6 +244,12 @@ export const AgentOutputSchema = { "Everything the assistant said this turn, including what it narrated between tool calls", "x-stream": "append", }, + reasoning: { + type: "string", + title: "Reasoning", + description: "The model's reasoning this turn, as the provider streamed it", + "x-stream": "append", + }, messages: { type: "array", items: ChatMessageSchema, @@ -296,6 +323,8 @@ export type AgentTaskInput = { readonly maxInputTokens?: number | undefined; readonly maxCostUsd?: number | undefined; readonly maxDurationMs?: number | undefined; + readonly announceTimeLeft?: boolean | undefined; + readonly reviewPrompt?: string | undefined; readonly approval?: AgentApprovalMode | undefined; }; @@ -303,6 +332,12 @@ export type AgentStopReason = "answered" | "submitted" | "budget" | "max-rounds" export type AgentTaskOutput = { text: string; + /** + * Each round's reasoning, a blank line between rounds that reasoned. Absent + * when no round returned any, so a turn against a model that does not think + * has the output it always had. + */ + reasoning?: string | undefined; messages: ChatMessage[]; rounds: number; stopReason: AgentStopReason; @@ -329,8 +364,27 @@ function isAnswerable(call: ToolCall): boolean { return typeof call.id === "string" && call.id.length > 0 && typeof call.name === "string"; } -function assistantMessage(text: string, calls: readonly ToolCall[]): ChatMessage { +function assistantMessage( + text: string, + calls: readonly ToolCall[], + reasoning: string | undefined, + nativeTurn: { readonly provider: string; readonly payload: string } | undefined +): ChatMessage { const content: ContentBlock[] = []; + // Reasoning rides only on a reply that has something else: on its own it is + // not a reply, and an assistant turn with no text and no call is rejected. + if (text.length > 0 || calls.length > 0) { + if (nativeTurn) { + content.push({ + type: "reasoning", + text: reasoning ?? "", + provider: nativeTurn.provider, + payload: nativeTurn.payload, + }); + } else if (reasoning) { + content.push({ type: "reasoning", text: reasoning }); + } + } if (text.length > 0) content.push({ type: "text", text }); for (const call of calls) { content.push({ @@ -362,24 +416,83 @@ function toolResult( call: ToolCall, text: string, isError: boolean, - maxChars: number + maxChars: number, + content?: ReadonlyArray ): ContentBlockToolResult { - const body: ContentBlockInToolResultBody[] = [ - { type: "text", text: clampToolText(text, maxChars) }, - ]; return { type: "tool_result", tool_use_id: call.id, - content: body, + content: content ? orderedToolBody(content, maxChars) : [textBlock(text, maxChars)], is_error: isError ? true : undefined, }; } +function textBlock(text: string, maxChars: number): ContentBlockText { + return { type: "text", text: clampToolText(text, maxChars) }; +} + +/** + * A tool's own blocks in the order it returned them, so text describing an image + * stays beside it. Adjacent text joins into one block, which keeps a result of + * text alone a single block, as providers send it as a plain string. The text + * shares one budget: past it a block is cut and later text dropped, while images + * stay. A result of images alone carries no text block: an empty one is not a + * block Anthropic accepts, and there is nothing in it for the model to read. + */ +function orderedToolBody( + blocks: ReadonlyArray, + maxChars: number +): ContentBlockInToolResultBody[] { + const runs: Array = []; + for (const block of blocks) { + const previous = runs[runs.length - 1]; + if (block.type === "text" && previous?.type === "text") { + runs[runs.length - 1] = { type: "text", text: `${previous.text}\n${block.text}` }; + } else { + runs.push(block); + } + } + const total = runs.reduce((n, block) => n + (block.type === "text" ? block.text.length : 0), 0); + const note = `[truncated, ${total} chars total]`; + let left = maxChars; + let noted = false; + const body: ContentBlockInToolResultBody[] = []; + for (const block of runs) { + if (block.type === "image") { + body.push(block); + } else if (block.text.length === 0) { + continue; + } else if (block.text.length <= left) { + body.push(block); + left -= block.text.length; + } else if (left > 0) { + body.push({ type: "text", text: `${block.text.slice(0, left)}… ${note}` }); + left = 0; + noted = true; + } else if (!noted) { + // The budget ran out exactly at an earlier block: the model is still told + // that text was left out, where it was. + body.push({ type: "text", text: `… ${note}` }); + noted = true; + } + } + return body.length > 0 ? body : [textBlock("", maxChars)]; +} + +/** The time a turn has left, as the model is shown it: whole seconds, rounded down. */ +function timeLeftNotice(ms: number): string { + const total = Math.max(0, Math.floor(ms / 1000)); + const minutes = Math.floor(total / 60); + return `\n\n[Time left: ${minutes > 0 ? `${minutes}m ` : ""}${total % 60}s]`; +} + interface RunCallOptions { readonly byName: ReadonlyMap; readonly validators: ReturnType; readonly approval: AgentApprovalMode; readonly maxResultChars: number; + /** When the turn's time budget runs out; see {@link ToolExecuteContext.deadline}. */ + readonly deadline: number | undefined; /** The tool that submits the answer, when the turn has an `outputSchema`. */ readonly submitToolName: string | undefined; } @@ -540,8 +653,15 @@ export class AgentTask extends Task ({ messages, steps }); /** * The transcript so far, for a host drawing one while the turn is still @@ -571,11 +694,13 @@ export class AgentTask extends Task; let text = ""; + let reasoning = ""; let rounds = 0; const finish = (stopReason: AgentStopReason): StreamEvent => ({ type: "finish", data: { text, + ...(reasoning.length > 0 ? { reasoning } : {}), messages, rounds, stopReason, @@ -596,8 +721,14 @@ export class AgentTask extends Task= deadline) { + yield finish("budget"); + return; + } + rounds = round + 1; const roundStart = Date.now(); const captured: { output: ToolCallingTaskOutput | undefined } = { output: undefined }; @@ -610,9 +741,11 @@ export class AgentTask extends Task 0 )) { yield event; } @@ -621,8 +754,23 @@ export class AgentTask extends Task maxRoundRetries || !isRetryableRoundError(err)) throw err; context.signal.throwIfAborted(); const wait = retryDelayMs(err, attempts); + // Nothing from the failed attempt has been recorded, so the turn can end + // here as cleanly as between rounds. + if (deadline !== undefined && Date.now() + wait >= deadline) { + yield finish("budget"); + return; + } await context.updateProgress(undefined, `Retrying in ${Math.ceil(wait / 1000)}s`); await sleepUnlessAborted(wait, context.signal); + // The wait can overrun (a slow progress write, a late timer); an attempt + // started past the deadline is one the budget no longer covers. + if (deadline !== undefined && Date.now() >= deadline) { + yield finish("budget"); + return; + } + // Said as the attempt starts, not before the wait: a wait cut short + // (the run stopped during it) is no attempt made. + await context.updateProgress(undefined, `Thinking (attempt ${attempts + 1})`); } } const modelMs = Date.now() - roundStart; @@ -634,6 +782,10 @@ export class AgentTask extends Task 0) { + reasoning += (reasoning.length > 0 ? REASONING_ROUND_SEPARATOR : "") + roundReasoning; + } const record = (toolRecords: readonly AgentToolRecord[]): void => { steps.push({ @@ -661,13 +813,33 @@ export class AgentTask extends Task 0) { messages.push(reply); yield transcript(); } if (calls.length === 0) { record([]); + // A turn with an outputSchema has checkSubmission for this, and a review + // sent with no round, budget or time left could never be acted on. + if ( + input.reviewPrompt && + !reviewed && + submitTool === undefined && + round + 1 < maxRounds && + !overBudget() && + (deadline === undefined || Date.now() < deadline) + ) { + reviewed = true; + messages.push({ role: "user", content: [{ type: "text", text: input.reviewPrompt }] }); + yield transcript(); + continue; + } if (submitTool !== undefined && reminders < MAX_SUBMIT_REMINDERS && !overBudget()) { reminders++; messages.push({ role: "user", content: [{ type: "text", text: SUBMIT_REMINDER }] }); @@ -707,10 +879,34 @@ export class AgentTask extends Task 0) { + const last = results[results.length - 1]!; + const notice = timeLeftNotice(deadline - Date.now()); + // Joined onto the last text block, not added as a second one: a result of + // one text block stays one, which providers send as a plain string (or + // parse as a whole) and a second block would turn into a list of parts. + // A result of images alone has no text to join, so the notice leads them. + const content = [...last.content]; + const at = content.findLastIndex((block) => block.type === "text"); + const block = content[at]; + if (block?.type === "text") content[at] = { ...block, text: block.text + notice }; + else content.unshift({ type: "text", text: notice }); + results[results.length - 1] = { ...last, content }; + // The record counts what the model is shown, which now includes the notice. + const recordAt = toolRecords.findIndex((r) => r.id === last.tool_use_id); + if (recordAt !== -1) { + const counted = toolRecords[recordAt]!; + toolRecords[recordAt] = { ...counted, chars: counted.chars + notice.length }; + } + } messages.push({ role: "tool", content: results }); record(toolRecords); yield transcript(); @@ -793,7 +989,7 @@ export class AgentTask extends Task> { const roundInput: ToolCallingTaskInput = { model: input.model, prompt: input.prompt, systemPrompt: input.systemPrompt, - messages: normalizeHistoryForModel(trimHistoryForModel(messages, maxHistoryChars)), + // The current turn is held whole: a reminder or review prompt is a user + // message, and a cut there would drop the prompt the model is asked to + // check its answer against and every tool result it gathered. Earlier + // turns are still cut to fit. + messages: normalizeHistoryForModel(trimHistoryForModel(messages, maxHistoryChars, turnStart)), tools: input.tools, temperature: input.temperature, maxTokens: input.maxTokens, @@ -844,6 +1046,8 @@ export class AgentTask extends Task queue.push({ type: "text-delta", port: "text", textDelta: delta }), + onReasoningDelta: (delta) => + queue.push({ type: "text-delta", port: "reasoning", textDelta: delta }), // Progress is advisory: a failed update must not surface as an // unhandled rejection, nor fail a round that is otherwise fine. onProgress: (progress, message) => { @@ -856,8 +1060,9 @@ export class AgentTask extends Task turn.abort(); off = turn.subscribe("stream_chunk", (event: StreamEvent) => { if (event.type !== "text-delta") return; - if ((event.port ?? "text") !== "text") return; - queue.push({ type: "text-delta", port: "text", textDelta: event.textDelta }); + const port = event.port ?? "text"; + if (port !== "text" && port !== "reasoning") return; + queue.push({ type: "text-delta", port, textDelta: event.textDelta }); }); runRound = () => turn.run(roundInput); } @@ -887,13 +1092,25 @@ export class AgentTask extends Task {}); // Held until the attempt settles: the stream accumulates every delta it is - // handed into the task's text port and offers no way to take one back, so - // text forwarded from an attempt that then fails and is retried would be - // joined to the retry's. A failed attempt throws here and its text is dropped. + // handed into the task's text and reasoning ports and offers no way to take + // one back, so either forwarded from an attempt that then fails and is + // retried would be joined to the retry's. A failed attempt throws here and + // its text and reasoning are dropped together. const held: StreamEvent[] = []; for await (const event of queue.iterable) held.push(event); await run; - for (const event of held) yield event; + let separated = !reasonedBefore; + for (const event of held) { + // Rounds are told apart on the wire exactly as in the settled output, and + // only once this round has something to say: a separator ahead of nothing + // would leave the streamed reasoning ending differently than the final one. + if (!separated && event.type === "text-delta" && event.port === "reasoning") { + if (event.textDelta.length === 0) continue; + yield { type: "text-delta", port: "reasoning", textDelta: REASONING_ROUND_SEPARATOR }; + separated = true; + } + yield event; + } } /** @@ -936,8 +1153,9 @@ export class AgentTask extends Task | undefined; } /** @@ -160,13 +166,15 @@ export function toolCallNeedsApproval( async function askToApprove( context: IExecuteContext, toolName: string, - request: IHumanRequest + request: IHumanRequest, + signal: AbortSignal ): Promise { try { - return await context.registry.get(HUMAN_CONNECTOR).send(request, context.signal); + return await context.registry.get(HUMAN_CONNECTOR).send(request, signal); } catch (error) { if (context.signal.aborted) throw error; - getLogger().warn(`Could not ask for approval of "${toolName}"`, { error }); + // Withdrawn at the turn's deadline: the caller reports that, not a failure to ask. + if (!signal.aborted) getLogger().warn(`Could not ask for approval of "${toolName}"`, { error }); return undefined; } } @@ -198,7 +206,8 @@ function describeToolReach( async function approveToolCall( tool: ToolDefinition, call: ToolCall, - context: IExecuteContext + context: IExecuteContext, + signal: AbortSignal ): Promise { if (!context.registry.has(HUMAN_CONNECTOR)) { return { @@ -209,21 +218,26 @@ async function approveToolCall( }; } const ctor = getTaskConstructors(context.registry).get(backingTaskType(tool)); - const response = await askToApprove(context, tool.name, { - requestId: uuid4(), - targetHumanId: "default", - kind: "confirm", - message: `Run "${tool.name}"?`, - contentSchema: APPROVAL_SCHEMA as DataPortSchema, - contentData: { - tool: tool.name, - reach: describeToolReach(tool, ctor), - arguments: clampToolText(stringifyForModel(call.input), MAX_APPROVAL_ARGUMENT_CHARS), + const response = await askToApprove( + context, + tool.name, + { + requestId: uuid4(), + targetHumanId: "default", + kind: "confirm", + message: `Run "${tool.name}"?`, + contentSchema: APPROVAL_SCHEMA as DataPortSchema, + contentData: { + tool: tool.name, + reach: describeToolReach(tool, ctor), + arguments: clampToolText(stringifyForModel(call.input), MAX_APPROVAL_ARGUMENT_CHARS), + }, + expectsResponse: true, + mode: "single", + metadata: { toolUseId: call.id, toolName: tool.name }, }, - expectsResponse: true, - mode: "single", - metadata: { toolUseId: call.id, toolName: tool.name }, - }); + signal + ); if (response === undefined) { return { text: @@ -243,6 +257,51 @@ async function approveToolCall( }; } +const MAX_TIMER_MS = 2_147_483_647; + +/** + * A signal that aborts at `deadline`, however far off. A timer past 2^31-1 ms + * fires at once, so a deadline further away than that is reached in steps, each + * re-armed for what is left. + */ +export function deadlineSignal(deadline: number): { + readonly signal: AbortSignal; + readonly cancel: () => void; +} { + const expiry = new AbortController(); + let timer: ReturnType | undefined; + const arm = (): void => { + const left = deadline - Date.now(); + timer = + left > MAX_TIMER_MS + ? setTimeout(arm, MAX_TIMER_MS) + : setTimeout(() => expiry.abort(), Math.max(0, left)); + }; + arm(); + return { signal: expiry.signal, cancel: () => clearTimeout(timer) }; +} + +/** Asks for approval, withdrawing the request when the turn's deadline passes. */ +async function approveWithinDeadline( + tool: ToolDefinition, + call: ToolCall, + context: IExecuteContext, + deadline: number | undefined +): Promise { + if (deadline === undefined) return approveToolCall(tool, call, context, context.signal); + const expiry = deadlineSignal(deadline); + try { + return await approveToolCall( + tool, + call, + context, + AbortSignal.any([context.signal, expiry.signal]) + ); + } finally { + expiry.cancel(); + } +} + /** * Runs one tool call and returns what the model reads back. * @@ -255,15 +314,40 @@ export async function runAgentTool( tool: ToolDefinition, call: ToolCall, context: IExecuteContext, - options: { readonly approval: AgentApprovalMode; readonly maxResultChars: number } + options: { + readonly approval: AgentApprovalMode; + readonly maxResultChars: number; + readonly deadline: number | undefined; + } ): Promise { - const refusal = toolCallNeedsApproval(tool, options.approval, context.registry) - ? await approveToolCall(tool, call, context) + const needsApproval = toolCallNeedsApproval(tool, options.approval, context.registry); + const refusal = needsApproval + ? await approveWithinDeadline(tool, call, context, options.deadline) : undefined; + // A person can take longer to answer than the turn has left: the request is + // withdrawn at the deadline, and an approval landing after it does not start the tool. + if (needsApproval && options.deadline !== undefined && Date.now() >= options.deadline) { + return { + text: + `The turn's time ran out while "${tool.name}" waited for approval, so it was not run. ` + + `Do not call it again this turn.`, + isError: true, + }; + } if (refusal) return refusal; try { - const output = await invokeTool(tool, call, context); + const output = await invokeTool(tool, call, context, options.deadline); + if (output instanceof ToolResultContent) { + const text = output.content + .flatMap((block) => (block.type === "text" ? [block.text] : [])) + .join("\n"); + return { + text: clampToolText(text, options.maxResultChars), + isError: false, + content: output.content, + }; + } return { text: clampToolText(stringifyForModel(output), options.maxResultChars), isError: false, @@ -287,13 +371,14 @@ export async function runAgentTool( async function invokeTool( tool: ToolDefinition, call: ToolCall, - context: IExecuteContext + context: IExecuteContext, + deadline: number | undefined ): Promise { if (tool.type === "function" || (tool.type === undefined && tool.execute)) { if (!tool.execute) { throw new Error(`Tool "${tool.name}" is declared type "function" but supplies no execute()`); } - return await tool.execute(call.input, { toolUseId: call.id, signal: context.signal }); + return await tool.execute(call.input, { toolUseId: call.id, signal: context.signal, deadline }); } const ctor = getTaskConstructors(context.registry).get(backingTaskType(tool)); if (!ctor) { diff --git a/packages/ai/src/task/AiChatTask.ts b/packages/ai/src/task/AiChatTask.ts index 97dc2603af..2e482b1845 100644 --- a/packages/ai/src/task/AiChatTask.ts +++ b/packages/ai/src/task/AiChatTask.ts @@ -208,6 +208,7 @@ export type AiChatTaskInput = Omit< )[]; tool_use_id: string; } + | { type: "reasoning"; text: string; provider?: string; payload?: string } )[]; }, "messages" diff --git a/packages/ai/src/task/ChatHistory.ts b/packages/ai/src/task/ChatHistory.ts index de73eb4311..ef60e4aae6 100644 --- a/packages/ai/src/task/ChatHistory.ts +++ b/packages/ai/src/task/ChatHistory.ts @@ -62,14 +62,29 @@ export function normalizeHistoryForModel(history: readonly ChatMessage[]): ChatM */ export const DEFAULT_MAX_HISTORY_CHARS = 120_000; +/** + * A message's size as the model reads it. A reasoning block's `payload` is a + * provider's own turn kept for replay, not text this budget is protecting the + * context window from, so it is measured without it. + */ function messageChars(message: ChatMessage): number { try { - return JSON.stringify(message).length; + const content = message.content.map((block) => { + if (block.type !== "reasoning" || block.payload === undefined) return block; + const { payload: _payload, ...rest } = block; + return rest; + }); + return JSON.stringify({ ...message, content }).length; } catch { return 0; } } +/** `history`'s size as {@link trimHistoryForModel} measures it against its budget. */ +export function measureHistoryForModel(history: readonly ChatMessage[]): number { + return history.reduce((sum, message) => sum + messageChars(message), 0); +} + /** * Caps `history` at `max` characters by dropping whole turns from the front. * @@ -82,14 +97,24 @@ function messageChars(message: ChatMessage): number { * otherwise be nothing for the model to answer, and silently returning an empty * list turns "this turn is too long" into "the conversation is gone". * - * A leading run of `system` messages belongs to no turn and is never cut: it - * carries the instructions every later turn is answered under, and dropping it - * for being old changes how the model behaves with nothing to show for it. Its - * size still counts against the budget, so the turns cut to fit around it. + * A `system` message is never cut, leading or not: it carries instructions + * every later turn is answered under, and dropping one for being old changes + * how the model behaves with nothing to show for it. A later one stays in its + * place ahead of the turns that survive. Their size still counts against the + * budget, so the turns cut to fit around them. + * + * `keepFrom` holds everything from that index on, for a caller whose newest + * turn holds user messages of its own (a reminder, a review prompt): cutting at + * one of those would keep the prompt and drop the question it is about. + * + * A reasoning block's `payload` is not counted, so a provider replaying its own + * turns (encrypted reasoning, signed thinking) sends more than `max`. The budget + * caps the conversation's text, not those opaque payloads. */ export function trimHistoryForModel( history: readonly ChatMessage[], - max: number = DEFAULT_MAX_HISTORY_CHARS + max: number = DEFAULT_MAX_HISTORY_CHARS, + keepFrom: number = history.length ): ChatMessage[] { const sizes = history.map(messageChars); let total = sizes.reduce((sum, n) => sum + n, 0); @@ -99,15 +124,25 @@ export function trimHistoryForModel( while (history[prefixEnd]?.role === "system") prefixEnd++; const turnStarts: number[] = []; + // A run of user messages is one pending turn (as normalizeHistoryForModel + // merges it), so only its first message is a place to cut: cutting inside it + // would keep the newest prompt and drop the question it follows. for (let i = prefixEnd; i < history.length; i++) { - if (history[i]?.role === "user") turnStarts.push(i); + if (history[i]?.role === "user" && history[i - 1]?.role !== "user") turnStarts.push(i); } + // A system message past the leading run is an instruction too, and every + // provider lifts it into the system prompt wherever it sits, so a cut keeps it + // and it stays counted. let cut = prefixEnd; for (const start of turnStarts.slice(1)) { - for (let i = cut; i < start; i++) total -= sizes[i] ?? 0; + if (start > keepFrom) break; + for (let i = cut; i < start; i++) { + if (history[i]?.role !== "system") total -= sizes[i] ?? 0; + } cut = start; if (total <= max) break; } - return [...history.slice(0, prefixEnd), ...history.slice(cut)]; + const keptSystem = history.slice(prefixEnd, cut).filter((message) => message.role === "system"); + return [...history.slice(0, prefixEnd), ...keptSystem, ...history.slice(cut)]; } diff --git a/packages/ai/src/task/ChatMessage.ts b/packages/ai/src/task/ChatMessage.ts index a9d379494a..4202535f55 100644 --- a/packages/ai/src/task/ChatMessage.ts +++ b/packages/ai/src/task/ChatMessage.ts @@ -32,6 +32,28 @@ export type ContentBlockToolUse = { readonly providerSignature?: string; }; +/** + * What a thinking model reasoned before the rest of its reply, kept so it can + * be handed back on later rounds of the same turn. Some providers ask for it + * back (DeepSeek's `reasoning_content` on an assistant turn that called tools), + * and a model given its own reasoning continues from it rather than working the + * plan out again each round. A provider that has no use for it skips the block. + * `payload` is a provider's whole turn, kept so that provider can resume its own + * reasoning, which it returns only in its own encoded form. + */ +export type ContentBlockReasoning = { + readonly type: "reasoning"; + readonly text: string; + /** + * The provider whose assistant turn `payload` is, when the round kept it. That + * provider replays the payload verbatim on later rounds; every other provider + * reads `text` or skips the block. + */ + readonly provider?: string; + /** The provider's own assistant output for this turn, serialized in its wire format and order. */ + readonly payload?: string; +}; + /** * Blocks that may appear in a `tool_result`'s `content` array. Provider payloads * typically use text, image, and tool_use; nested `tool_result` is not modeled @@ -55,7 +77,8 @@ export type ContentBlock = | ContentBlockText | ContentBlockImage | ContentBlockToolUse - | ContentBlockToolResult; + | ContentBlockToolResult + | ContentBlockReasoning; export type ChatRole = "user" | "assistant" | "tool" | "system"; @@ -98,6 +121,18 @@ const ContentBlockToolUseSchema = { additionalProperties: false, } as const; +const ContentBlockReasoningSchema = { + type: "object", + properties: { + type: { type: "string", enum: ["reasoning"] }, + text: { type: "string" }, + provider: { type: "string" }, + payload: { type: "string" }, + }, + required: ["type", "text"], + additionalProperties: false, +} as const; + /** `tool_result.content` — text, image, and tool_use only (no nested `tool_result`). */ const ContentBlockInToolResultBodySchema = { oneOf: [ContentBlockTextSchema, ContentBlockImageSchema, ContentBlockToolUseSchema], @@ -126,6 +161,7 @@ export const ContentBlockSchema = { ContentBlockImageSchema, ContentBlockToolUseSchema, ContentBlockToolResultSchema, + ContentBlockReasoningSchema, ], title: "ContentBlock", description: "A single content block within a chat message", @@ -189,6 +225,12 @@ export function isContentBlock(value: unknown): value is ContentBlock { Array.isArray(v.content) && v.content.every(isContentBlockInToolResultBody) ); + case "reasoning": + return ( + typeof v.text === "string" && + (v.provider === undefined || typeof v.provider === "string") && + (v.payload === undefined || typeof v.payload === "string") + ); default: return false; } diff --git a/packages/ai/src/task/MessageConversion.ts b/packages/ai/src/task/MessageConversion.ts index 5e22b19d1c..a81b363101 100644 --- a/packages/ai/src/task/MessageConversion.ts +++ b/packages/ai/src/task/MessageConversion.ts @@ -53,6 +53,32 @@ export interface OpenAICompatMessage { function: { name: string; arguments: string }; }>; tool_call_id?: string; + /** An assistant turn's reasoning, under the name the provider reads it back by. */ + reasoning_content?: string; + /** The provider's own output items for this turn, for the Responses converter to replay. */ + native_items?: unknown[]; +} + +export interface OpenAIMessageOptions { + /** + * Send each assistant turn's reasoning back as `reasoning_content`. DeepSeek's + * thinking models read it on the turns of a tool-calling loop; other + * OpenAI-compatible APIs do not know the field, so it is opt-in and reasoning + * blocks are otherwise dropped. + */ + readonly replayReasoning?: boolean | undefined; + /** + * Chat-completions accepts only text in a `tool` message. With this set, the + * images a turn's tools returned are sent in one user message after that + * turn's tool results, and each tool message says an image follows. Without + * it they stay inline in the tool message, for APIs that accept that. + */ + readonly toolImagesInUserMessage?: boolean | undefined; + /** + * Attach an assistant turn's native items (`native_items`) when its reasoning + * block was produced by this provider, for the Responses converter to replay. + */ + readonly nativeTurnProvider?: string | undefined; } /** @@ -61,14 +87,24 @@ export interface OpenAICompatMessage { * * Multi-turn capable: preserves full tool call metadata across turns. */ -export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessage[] { +export function toOpenAIMessages( + input: ToolCallingTaskInput, + options: OpenAIMessageOptions = {} +): OpenAICompatMessage[] { const messages: OpenAICompatMessage[] = []; - if (input.systemPrompt) { - messages.push({ role: "system", content: input.systemPrompt }); + // System turns in the history join the system prompt; the loop below has no + // place for them, and a caller's instructions must not be dropped. + const history = getInputMessages(input); + const lifted = history ? liftSystemMessages(history, input.systemPrompt) : undefined; + const systemPrompt = lifted ? lifted.systemPrompt : input.systemPrompt; + if (systemPrompt) { + messages.push({ role: "system", content: systemPrompt }); } - const inputMessages = getInputMessages(input); + // A history that held only system turns is still the history: the prompt is + // what a caller with no history sends, never a stand-in for one. + const inputMessages = lifted?.messages; if (!inputMessages) { if (!Array.isArray(input.prompt)) { messages.push({ role: "user", content: input.prompt }); @@ -102,7 +138,26 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa return messages; } + // Images a tool returned wait until every result of the run of tool messages + // is out: a user turn between two results of one assistant turn would leave a + // call unanswered before it. + let deferredImages: Array<{ type: string; [key: string]: unknown }> = []; + let deferredImageCount = 0; + const flushDeferredImages = (): void => { + if (deferredImages.length === 0) return; + messages.push({ + role: "user", + content: [ + { type: "text", text: "Images returned by the tool calls above:" }, + ...deferredImages, + ], + }); + deferredImages = []; + deferredImageCount = 0; + }; + for (const msg of inputMessages) { + if (msg.role !== "tool") flushDeferredImages(); if (msg.role === "user") { const parts: Array<{ type: string; [key: string]: unknown }> = []; for (const block of msg.content) { @@ -113,6 +168,9 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa type: "image_url", image_url: { url: `data:${block.mimeType};base64,${block.data}` }, }); + } else if ((block as { type: string }).type === "image_url") { + // Already in this format: a caller written against chat-completions. + parts.push(block as unknown as { type: string; [key: string]: unknown }); } // tool_use / tool_result not valid in a user message — skip } @@ -134,18 +192,86 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa })); const entry: OpenAICompatMessage = { role: "assistant", - content: textParts.length > 0 ? textParts : null, + // A turn with neither text nor calls (the model only reasoned) still needs a + // content string: chat-completions APIs reject `null` without `tool_calls`. + content: textParts.length > 0 ? textParts : toolCalls.length > 0 ? null : "", }; if (toolCalls.length > 0) { entry.tool_calls = toolCalls; } + if (options.replayReasoning) { + // Sent even when empty: DeepSeek returns an empty reasoning on some + // turns and still expects the field on every assistant turn it reads. + entry.reasoning_content = msg.content + .filter((b): b is Extract => b.type === "reasoning") + .map((b) => b.text) + .join(""); + } + if (options.nativeTurnProvider !== undefined) { + const own = msg.content.find( + (b): b is Extract => + b.type === "reasoning" && + b.provider === options.nativeTurnProvider && + b.payload !== undefined + ); + if (own) { + try { + const items = JSON.parse(own.payload!) as unknown; + // Each item must at least be a typed object; `[null]` or a bare value + // would be sent as an input item and refused, where a rebuilt turn + // is not. + if ( + Array.isArray(items) && + items.length > 0 && + items.every( + (item) => + item !== null && + typeof item === "object" && + !Array.isArray(item) && + typeof (item as { type?: unknown }).type === "string" + ) + ) { + entry.native_items = items; + } + } catch { + // A payload this provider cannot read is not replayed; the turn is rebuilt. + } + } + } messages.push(entry); } else if (msg.role === "tool") { for (const block of msg.content) { if (block.type !== "tool_result") continue; let content: string | Array<{ type: string; [key: string]: unknown }>; - if (block.content.length === 1 && block.content[0].type === "text") { - content = block.content[0].text; + const onlyText = block.content.every((inner) => inner.type === "text"); + if (onlyText) { + // Several text blocks, or none, still become one string: some chat-completions + // APIs reject a parts array in a tool message. + content = block.content.map((inner) => (inner as { text: string }).text).join("\n"); + } else if (options.toolImagesInUserMessage) { + // Each image leaves a numbered marker where it stood, and the user message + // labels it with the same number, so text around an image still points at + // the right one. A block that is neither text nor an image (a nested + // tool_use) is data the tool returned, and goes as JSON rather than dropped. + const lines: string[] = []; + for (const inner of block.content) { + if (inner.type === "text") { + lines.push(inner.text); + } else if (inner.type === "image") { + const label = `[image ${++deferredImageCount}]`; + lines.push(`${label} follows in the next message`); + deferredImages.push( + { type: "text", text: label }, + { + type: "image_url", + image_url: { url: `data:${inner.mimeType};base64,${inner.data}` }, + } + ); + } else { + lines.push(JSON.stringify(inner)); + } + } + content = lines.join("\n"); } else { const parts: Array<{ type: string; [key: string]: unknown }> = []; for (const inner of block.content) { @@ -156,6 +282,8 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa type: "image_url", image_url: { url: `data:${inner.mimeType};base64,${inner.data}` }, }); + } else { + parts.push({ type: "text", text: JSON.stringify(inner) }); } } content = parts; @@ -164,10 +292,56 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa } } } + flushDeferredImages(); return messages; } +/** A message whose content is either workglow content blocks or a bare string. */ +export interface LooseChatMessage { + readonly role: string; + readonly content: string | ReadonlyArray; +} + +/** + * Chat history as content blocks, accepting the bare-string `content` that + * text-generation callers have always been allowed to pass. A provider runs + * this before its converter, so a string message and a block message reach the + * wire the same way instead of the array being forwarded unconverted. + */ +export function normalizeChatMessages(messages: ReadonlyArray): ChatMessage[] { + return messages.map((message) => ({ + role: message.role as ChatMessage["role"], + content: + typeof message.content === "string" + ? [{ type: "text", text: message.content }] + : message.content, + })); +} + +/** + * Moves system-role messages out of a history and into the system prompt. Most + * converters take the system prompt as its own input and skip system turns in + * `messages`, so a caller that put its instructions in the history would + * otherwise lose them; `toOpenAIMessages` lifts them itself. + */ +export function liftSystemMessages( + history: ReadonlyArray, + systemPrompt: string | undefined +): { readonly messages: ChatMessage[]; readonly systemPrompt: string | undefined } { + const lifted = history + .filter((message) => message.role === "system") + .map((message) => + message.content.flatMap((block) => (block.type === "text" ? [block.text] : [])).join("") + ) + .filter((text) => text.length > 0); + const parts = [systemPrompt, ...lifted].filter((text): text is string => !!text); + return { + messages: history.filter((message) => message.role !== "system"), + systemPrompt: parts.length > 0 ? parts.join("\n\n") : undefined, + }; +} + export interface TextFlatMessage { role: string; content: string; @@ -175,8 +349,8 @@ export interface TextFlatMessage { /** * Converts ToolCallingTaskInput to a simplified text-only message format. - * Used by providers that don't natively support structured multi-turn - * tool calling (Ollama, HuggingFace Transformers). + * Used for single-turn prompts by providers whose template takes only role and + * text. A tool-calling loop must not use it: * * NOTE: This format discards tool_use blocks from assistant messages. * The LLM will not see what tools it previously called. Multi-turn tool diff --git a/packages/ai/src/task/ToolCallingTask.ts b/packages/ai/src/task/ToolCallingTask.ts index 3d0a5706bc..c10180f70b 100644 --- a/packages/ai/src/task/ToolCallingTask.ts +++ b/packages/ai/src/task/ToolCallingTask.ts @@ -258,6 +258,26 @@ export const ToolCallingOutputSchema = { description: "Tool calls requested by the model", "x-stream": "object", }, + reasoning: { + type: "string", + title: "Reasoning", + description: + "What a thinking model reasoned before replying, when its provider returns it as text", + "x-stream": "append", + }, + nativeTurn: { + type: "object", + title: "Native Turn", + description: + "The provider's own assistant output for this round, kept so the provider can be handed it back verbatim on the next round", + properties: { + provider: { type: "string" }, + payload: { type: "string" }, + }, + required: ["provider", "payload"], + additionalProperties: false, + "x-stream": "object", + }, ...CheckpointOutputProperty, }, required: ["text", "toolCalls"], @@ -320,6 +340,10 @@ export type ToolCallingTaskOutput = { input: { [x: string]: unknown }; providerSignature?: string; }[]; + /** Present when the provider returned the model's reasoning as text. */ + reasoning?: string | undefined; + /** Present when the provider handed back its own turn for replay. */ + nativeTurn?: { readonly provider: string; readonly payload: string } | undefined; checkpoint?: string | undefined; }; export type ToolCallingTaskConfig = TaskConfig; @@ -450,7 +474,12 @@ export class ToolCallingTask extends StreamingAiTask< private async finalizeCheckpoint( input: ToolCallingTaskInput, - out: { text: string; toolCalls: ToolCallingTaskOutput["toolCalls"] } + out: { + text: string; + toolCalls: ToolCallingTaskOutput["toolCalls"]; + reasoning?: string | undefined; + nativeTurn?: ToolCallingTaskOutput["nativeTurn"]; + } ): Promise { const resolved = this._resolvedCheckpoint; if (!resolved?.emitCheckpointId) return; @@ -459,12 +488,41 @@ export class ToolCallingTask extends StreamingAiTask< input.messages && input.messages.length > 0 ? [...input.messages] : [promptToUserMessage(input.prompt)]; - const assistantContent = [ + const replyContent = [ ...(out.text ? ([{ type: "text", text: out.text }] as const) : []), + // A call's provider signature (Gemini's thought signature) rides with it, + // as the agent loop records it: Gemini refuses a replayed call without it. ...out.toolCalls.map( - (tc) => ({ type: "tool_use", id: tc.id, name: tc.name, input: tc.input }) as const + (tc) => + ({ + type: "tool_use", + id: tc.id, + name: tc.name, + input: tc.input, + ...(tc.providerSignature === undefined + ? {} + : { providerSignature: tc.providerSignature }), + }) as const ), ]; + // The turn is recorded with its reasoning and the provider's own output, as + // the agent loop records it, so a provider that replays its turn (signed + // thinking, encrypted reasoning) is handed it back from the checkpoint too. + // Reasoning alone is not a reply, so it rides only beside one. + const reasoningBlock = + replyContent.length === 0 + ? undefined + : out.nativeTurn + ? ({ + type: "reasoning", + text: out.reasoning ?? "", + provider: out.nativeTurn.provider, + payload: out.nativeTurn.payload, + } as const) + : out.reasoning + ? ({ type: "reasoning", text: out.reasoning } as const) + : undefined; + const assistantContent = reasoningBlock ? [reasoningBlock, ...replyContent] : replyContent; await finalizeEmittedCheckpoint({ model, resolved, @@ -547,12 +605,18 @@ export class ToolCallingTask extends StreamingAiTask< } let text = ""; + let reasoning = ""; + let nativeTurn: ToolCallingTaskOutput["nativeTurn"]; let toolCalls: ToolCallingTaskOutput["toolCalls"] = []; let finalized = false; try { for await (const event of super.executeStream(input, context)) { if (event.type === "text-delta" && (event.port ?? "text") === "text") { text += event.textDelta; + } else if (event.type === "text-delta" && event.port === "reasoning") { + reasoning += event.textDelta; + } else if (event.type === "object-delta" && event.port === "nativeTurn") { + nativeTurn = event.objectDelta as unknown as ToolCallingTaskOutput["nativeTurn"]; } else if (event.type === "object-delta" && event.port === "toolCalls") { // Mirror StreamProcessor's canonical accumulation: array deltas are // upserts by id (OpenAI-shaped providers emit one single-element @@ -571,7 +635,12 @@ export class ToolCallingTask extends StreamingAiTask< } } if (event.type === "finish") { - await this.finalizeCheckpoint(input, { text, toolCalls }); + await this.finalizeCheckpoint(input, { + text, + toolCalls, + reasoning: reasoning || undefined, + nativeTurn, + }); finalized = true; yield { type: "text-delta", diff --git a/packages/ai/src/task/ToolCallingUtils.ts b/packages/ai/src/task/ToolCallingUtils.ts index d31c492bc9..0d1d597924 100644 --- a/packages/ai/src/task/ToolCallingUtils.ts +++ b/packages/ai/src/task/ToolCallingUtils.ts @@ -8,6 +8,7 @@ import type { SchemaNode } from "@workglow/util/schema"; import { compileSchema } from "@workglow/util/schema"; import type { JsonSchema } from "@workglow/util/worker"; import { getLogger } from "@workglow/util/worker"; +import type { ContentBlockImage, ContentBlockText } from "./ChatMessage"; /** * A tool definition that can be passed to an LLM for tool calling. @@ -77,6 +78,23 @@ export interface ToolDefinition { export interface ToolExecuteContext { readonly toolUseId: string; readonly signal: AbortSignal; + /** + * When the turn's `maxDurationMs` runs out, in epoch milliseconds; undefined + * when the turn has no time budget. The loop stops only between rounds, so a + * tool that can bound its own work (a command timeout, a poll) caps it here, + * or one call can run the turn past its budget. + */ + readonly deadline: number | undefined; +} + +/** + * A tool result made of content blocks rather than one string — what a tool's + * `execute` returns to hand the model an image. Anything else a tool returns is + * shown to the model as text, so without this an image would arrive as a + * base64 string, clamped. + */ +export class ToolResultContent { + constructor(readonly content: ReadonlyArray) {} } /** diff --git a/packages/ai/src/task/__tests__/AgentDeadlineSignal.test.ts b/packages/ai/src/task/__tests__/AgentDeadlineSignal.test.ts new file mode 100644 index 0000000000..61cf61c5f8 --- /dev/null +++ b/packages/ai/src/task/__tests__/AgentDeadlineSignal.test.ts @@ -0,0 +1,39 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { afterEach, describe, expect, it, vi } from "vitest"; +import { deadlineSignal } from "../AgentToolExecution"; + +const MAX_TIMER_MS = 2_147_483_647; + +describe("deadlineSignal", () => { + afterEach(() => { + vi.useRealTimers(); + }); + + it("aborts at a deadline too far off for one timer, and not before", () => { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] }); + const days = 30 * 24 * 60 * 60 * 1000; + const { signal } = deadlineSignal(Date.now() + days); + // An overflowed timer would have fired at once. + vi.advanceTimersByTime(1); + expect(signal.aborted).toBe(false); + vi.advanceTimersByTime(MAX_TIMER_MS); + expect(signal.aborted).toBe(false); + vi.advanceTimersByTime(days - MAX_TIMER_MS - 2); + expect(signal.aborted).toBe(false); + vi.advanceTimersByTime(1); + expect(signal.aborted).toBe(true); + }); + + it("stops when cancelled", () => { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] }); + const { signal, cancel } = deadlineSignal(Date.now() + 100); + cancel(); + vi.advanceTimersByTime(1_000); + expect(signal.aborted).toBe(false); + }); +}); diff --git a/packages/ai/src/task/__tests__/CacheCheckpoint.test.ts b/packages/ai/src/task/__tests__/CacheCheckpoint.test.ts index 31aa5db9a0..27eaa953a9 100644 --- a/packages/ai/src/task/__tests__/CacheCheckpoint.test.ts +++ b/packages/ai/src/task/__tests__/CacheCheckpoint.test.ts @@ -519,9 +519,18 @@ describe("CacheCheckpointTask storage cost inside a graph", () => { describe("ToolCallingTask checkpoint ports", () => { let toolCalls: { session: AiSessionContext | undefined }[]; + let withNativeTurn = false; const toolUseFn: AiProviderRunFn = async (_input, _model, _signal, emit, _schema, session) => { toolCalls.push({ session }); + if (withNativeTurn) { + emit({ type: "text-delta", port: "reasoning", textDelta: "thought" } as any); + emit({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: "p:m", payload: "SIGNED" }, + } as any); + } emit({ type: "text-delta", port: "text", textDelta: "done" } as any); emit({ type: "finish", data: { text: "", toolCalls: [] } } as any); }; @@ -545,6 +554,7 @@ describe("ToolCallingTask checkpoint ports", () => { setAiProviderRegistry(new AiProviderRegistry()); clearCheckpointsForTesting(); toolCalls = []; + withNativeTurn = false; const provider = new CheckpointTestProvider([ { serves: ["text.generation", "tool-use"] as Capability[], runFn: toolUseFn }, { serves: ["cache.checkpoint"] as Capability[], runFn: ckptWarmFn }, @@ -598,6 +608,48 @@ describe("ToolCallingTask checkpoint ports", () => { expect(entry?.prefix.messages?.at(-2)?.role).toBe("user"); }); + it("records the turn's reasoning and native turn in the emitted checkpoint", async () => { + withNativeTurn = true; + const scope = new ResourceScope(); + const out = await new ToolCallingTask().run( + { model: toolModel(), prompt: "hi", tools: [aTool], emitCheckpoint: true }, + { resourceScope: scope } + ); + const entry = getCheckpoint((out as { checkpoint?: string }).checkpoint!); + expect(entry?.prefix.messages?.at(-1)).toEqual({ + role: "assistant", + content: [ + { type: "reasoning", text: "thought", provider: "p:m", payload: "SIGNED" }, + { type: "text", text: "done" }, + ], + }); + }); + + it("records a tool call's provider signature in the emitted checkpoint", async () => { + const signedCall = async (_i: any, _m: any, _s: any, emit: any) => { + emit({ + type: "object-delta", + port: "toolCalls", + objectDelta: [{ id: "c1", name: "a", input: {}, providerSignature: "SIG" }], + }); + emit({ type: "finish", data: { text: "", toolCalls: [] } }); + }; + setAiProviderRegistry(new AiProviderRegistry()); + clearCheckpointsForTesting(); + await new CheckpointTestProvider([ + { serves: ["text.generation", "tool-use"] as Capability[], runFn: signedCall as any }, + { serves: ["cache.checkpoint"] as Capability[], runFn: ckptWarmFn }, + ]).register({ queue: { autoCreate: false } }); + const out = await new ToolCallingTask().run( + { model: toolModel(), prompt: "hi", tools: [aTool], emitCheckpoint: true }, + { resourceScope: new ResourceScope() } + ); + const entry = getCheckpoint((out as { checkpoint?: string }).checkpoint!); + expect(entry?.prefix.messages?.at(-1)?.content).toEqual([ + { type: "tool_use", id: "c1", name: "a", input: {}, providerSignature: "SIG" }, + ]); + }); + it("keepParentCheckpoint preserves the parent", async () => { registerCheckpoint("ckpt-parent", { provider: CKPT_PROVIDER, diff --git a/packages/ai/src/task/__tests__/ChatHistory.test.ts b/packages/ai/src/task/__tests__/ChatHistory.test.ts index 69ee679c43..e2a5eb23e4 100644 --- a/packages/ai/src/task/__tests__/ChatHistory.test.ts +++ b/packages/ai/src/task/__tests__/ChatHistory.test.ts @@ -7,6 +7,7 @@ import type { ChatMessage } from "@workglow/ai"; import { DEFAULT_MAX_HISTORY_CHARS, + measureHistoryForModel, normalizeHistoryForModel, trimHistoryForModel, } from "@workglow/ai"; @@ -67,6 +68,28 @@ describe("trimHistoryForModel", () => { expect(trimHistoryForModel(history)).toEqual(history); }); + it("keeps a system message past the leading run when it cuts the turn around it", () => { + const filler = "z".repeat(400); + const system: ChatMessage = { role: "system", content: [{ type: "text", text: "be terse" }] }; + const history = [user(`old ${filler}`), system, assistant(`reply ${filler}`), user("now")]; + const trimmed = trimHistoryForModel(history, 100, 3); + expect(trimmed).toEqual([system, user("now")]); + }); + + it("keeps a run of user messages whole, never cutting between them", () => { + const filler = "z".repeat(400); + const history = [ + user(`old ${filler}`), + assistant(`reply ${filler}`), + user("the question"), + user("the new prompt"), + ]; + // Too small for everything: the old turn goes, the pending user turn stays whole, + // even with keepFrom at the newest prompt. + const trimmed = trimHistoryForModel(history, 100, 3); + expect(trimmed).toEqual([user("the question"), user("the new prompt")]); + }); + it("cuts only at a user message, so no tool_result outlives its tool_use", () => { // The invariant is not "the first message is a user message" — it is that // every surviving `tool_result` still has the `tool_use` it answers, which @@ -99,6 +122,32 @@ describe("trimHistoryForModel", () => { for (const id of results) expect(uses.has(id)).toBe(true); }); + it("does not count a reasoning block's payload against the budget", () => { + // The payload is a provider's own turn, replayed verbatim and never shown + // to the model as text, so a long one must not push earlier turns out. + const build = (payload: string | undefined): ChatMessage[] => { + const history: ChatMessage[] = []; + for (let i = 0; i < 6; i++) { + history.push(user(`ask ${i} ${"q".repeat(200)}`)); + history.push({ + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: "openai:m", ...(payload ? { payload } : {}) }, + { type: "text", text: `done ${i} ${"a".repeat(200)}` }, + ], + }); + } + return history; + }; + const budget = 1500; + const plain = trimHistoryForModel(build(undefined), budget); + expect(plain.length).toBeLessThan(12); + expect(plain.length).toBeGreaterThan(0); + const withPayload = trimHistoryForModel(build("p".repeat(5000)), budget); + expect(withPayload).toHaveLength(plain.length); + expect(withPayload[0]).toEqual(plain[0]); + }); + it("keeps the newest turn even when it alone exceeds the budget", () => { // Discarding here is how a conversation gets erased for being too long. const history = [user("old"), assistant("old reply"), user("x".repeat(5000))]; @@ -153,6 +202,36 @@ describe("trimHistoryForModel", () => { expect(trimmed.map((m) => m.role)).toEqual(["system", "system", "user"]); }); + it("never cuts past keepFrom", () => { + // A turn's own reminder is a user message; cutting there would keep the + // reminder and drop the question it is about. + const history = [ + user("old".repeat(200)), + assistant("old reply"), + user("the question"), + assistant("x".repeat(1000)), + user("check your work"), + ]; + expect(trimHistoryForModel(history, 500)).toEqual([history[4]]); + expect(trimHistoryForModel(history, 500, 2)).toEqual(history.slice(2)); + }); + + it("measures history as the trim does, payload left out", () => { + const plain: ChatMessage[] = [user("a"), assistant("b")]; + const withPayload: ChatMessage[] = [ + user("a"), + { + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: "openai:m", payload: "p".repeat(5000) }, + { type: "text", text: "b" }, + ], + }, + ]; + expect(measureHistoryForModel(plain)).toBeGreaterThan(0); + expect(measureHistoryForModel(withPayload)).toBeLessThan(5000); + }); + it("exposes a default budget callers can reason about", () => { expect(DEFAULT_MAX_HISTORY_CHARS).toBeGreaterThan(0); }); diff --git a/packages/test/src/test/ai/AgentTask.test.ts b/packages/test/src/test/ai/AgentTask.test.ts index 0fe42ccf82..1bc6e195c4 100644 --- a/packages/test/src/test/ai/AgentTask.test.ts +++ b/packages/test/src/test/ai/AgentTask.test.ts @@ -43,6 +43,8 @@ const MODEL: ModelConfig = { interface ScriptedRound { readonly text?: string; + /** Used by the host-runner script only: handed to `onReasoningDelta`. */ + readonly reasoning?: string; /** Report the text on the finish event only, as a non-streaming provider does. */ readonly withoutDeltas?: boolean; readonly calls?: ReadonlyArray<{ @@ -648,6 +650,85 @@ describe("AgentTask", () => { expect(JSON.stringify(toolResults(output.messages)[0])).toContain("asking for one failed"); }); + it("withdraws an unanswered approval at the turn's deadline instead of waiting on", async () => { + scriptModel([ + { + calls: [ + { id: "c1", name: "AgentTest_FetchTask", input: { url: "https://example.test" } }, + ], + }, + { text: "ok" }, + ]); + let withdrawn = false; + // A person who never answers: the request settles only when it is withdrawn. + registry.registerInstance(HUMAN_CONNECTOR, { + send: (_request: unknown, signal: AbortSignal) => + new Promise((_resolve, reject) => { + signal.addEventListener("abort", () => { + withdrawn = true; + reject(signal.reason); + }); + }), + }); + + const started = Date.now(); + const output = await new AgentTask().run( + { model: MODEL, prompt: "fetch", tools: [FETCH_TOOL], maxDurationMs: 300 }, + { registry } + ); + + expect(Date.now() - started).toBeLessThan(5_000); + expect(withdrawn).toBe(true); + expect(fetchRuns).toBe(0); + expect(output.stopReason).toBe("budget"); + expect(JSON.stringify(toolResults(output.messages)[0])).toContain("time ran out"); + }); + + it("waits on an approval normally under a deadline too far off for a timer", async () => { + scriptModel([ + { + calls: [ + { id: "c1", name: "AgentTest_FetchTask", input: { url: "https://example.test" } }, + ], + }, + { text: "ok" }, + ]); + // Answers after a moment, and withdraws on abort as a real connector does: a + // timer that overflowed to 1 ms would withdraw it before the answer lands. + registry.registerInstance(HUMAN_CONNECTOR, { + send: (request: IHumanRequest, signal: AbortSignal): Promise => + new Promise((resolve, reject) => { + const timer = setTimeout( + () => + resolve({ + requestId: request.requestId, + action: "accept", + content: undefined, + done: true, + }), + 50 + ); + signal.addEventListener("abort", () => { + clearTimeout(timer); + reject(signal.reason); + }); + }), + }); + + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "fetch", + tools: [FETCH_TOOL], + maxDurationMs: 30 * 24 * 60 * 60 * 1000, + }, + { registry } + ); + + expect(fetchRuns).toBe(1); + expect(output.stopReason).toBe("answered"); + }); + it("refuses rather than runs when there is nobody to ask", async () => { scriptModel([ { @@ -1133,8 +1214,13 @@ describe("AgentTask", () => { inputs.push(input); signals.push(context.signal); context.onProgress(40, "Loading model"); + if (round.reasoning) context.onReasoningDelta(round.reasoning); if (round.text) context.onTextDelta(round.text); - return { text: round.text ?? "", toolCalls: [...(round.calls ?? [])] }; + return { + text: round.text ?? "", + toolCalls: [...(round.calls ?? [])], + ...(round.reasoning ? { reasoning: round.reasoning } : {}), + }; }; return { runner, inputs, signals }; } @@ -1196,6 +1282,54 @@ describe("AgentTask", () => { expect(signals.every((signal) => signal instanceof AbortSignal)).toBe(true); }); + it("forwards the reasoning a runner reports on the turn's own port, apart from the text", async () => { + const { runner } = scriptRunner([ + { + reasoning: "Need the tool.", + text: "Looking. ", + calls: [{ id: "c1", name: "AgentTest_EchoTask", input: { text: "hi" } }], + }, + { reasoning: "Got it.", text: "It said HI." }, + ]); + registry.registerInstance(AGENT_ROUND_RUNNER, runner); + const task = new AgentTask(); + const events: Array<{ port: string; delta: string }> = []; + task.subscribe("stream_chunk", (event) => { + if (event.type === "text-delta") { + events.push({ port: event.port ?? "text", delta: event.textDelta }); + } + }); + + const output = await task.run( + { model: MODEL, prompt: "echo hi", tools: [ECHO_TOOL], approval: "never" }, + { registry } + ); + + expect(events.filter((e) => e.port === "reasoning").map((e) => e.delta)).toEqual([ + "Need the tool.", + "\n\n", + "Got it.", + ]); + expect(events.filter((e) => e.port === "text").map((e) => e.delta)).toEqual([ + "Looking. ", + "It said HI.", + ]); + expect(output.reasoning).toBe("Need the tool.\n\nGot it."); + expect(output.text).toBe("Looking. It said HI."); + }); + + it("leaves reasoning off the output when the runner reported none", async () => { + const { runner } = scriptRunner([{ text: "Plain." }]); + registry.registerInstance(AGENT_ROUND_RUNNER, runner); + + const output = await new AgentTask().run( + { model: MODEL, prompt: "hi", tools: [ECHO_TOOL], approval: "never" }, + { registry } + ); + + expect("reasoning" in output).toBe(false); + }); + it("still answers every call the model made through the runner", async () => { const { runner } = scriptRunner([ { calls: [{ id: "c1", name: "nope", input: {} }] }, diff --git a/packages/test/src/test/ai/AgentTaskDeadlineAndReview.test.ts b/packages/test/src/test/ai/AgentTaskDeadlineAndReview.test.ts new file mode 100644 index 0000000000..9ab4d3a0c1 --- /dev/null +++ b/packages/test/src/test/ai/AgentTaskDeadlineAndReview.test.ts @@ -0,0 +1,352 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { + AiProviderRunFn, + ChatMessage, + ModelConfig, + ToolCallingTaskInput, + ToolDefinition, +} from "@workglow/ai"; +import { + AgentTask, + AiProviderRegistry, + DirectExecutionStrategy, + getAiProviderRegistry, + setAiProviderRegistry, +} from "@workglow/ai"; +import { RetryableJobError } from "@workglow/job-queue"; +import { TaskConfigurationError } from "@workglow/task-graph"; +import { Container, ServiceRegistry } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const PROVIDER = "mock-deadline-provider"; + +const MODEL: ModelConfig = { + model_id: "mock/deadline-1", + provider: PROVIDER, + title: "", + description: "", + capabilities: ["tool-use"], + provider_config: {}, + metadata: {}, +}; + +interface Round { + readonly text?: string; + readonly calls?: ReadonlyArray<{ id: string; name: string; input: Record }>; +} + +/** A scripted model, one entry per call with the last repeating; returns what each call was sent. */ +function script(rounds: readonly Round[]): ToolCallingTaskInput[] { + const seen: ToolCallingTaskInput[] = []; + const runFn: AiProviderRunFn = async (input, _model, _signal, emit) => { + const round = rounds[Math.min(seen.length, rounds.length - 1)]!; + seen.push(input as ToolCallingTaskInput); + if (round.text) emit({ type: "text-delta", port: "text", textDelta: round.text }); + if (round.calls?.length) { + emit({ type: "object-delta", port: "toolCalls", objectDelta: [...round.calls] }); + } + emit({ type: "finish", data: {} }); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + return seen; +} + +const EMPTY = { type: "object", properties: {}, additionalProperties: false } as const; + +function tool(name: string, execute: ToolDefinition["execute"]): ToolDefinition { + return { name, description: name, inputSchema: EMPTY, execute }; +} + +describe("AgentTask deadline", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + it("hands a tool the moment the turn's time budget runs out", async () => { + script([{ calls: [{ id: "c1", name: "clock", input: {} }] }, { text: "done" }]); + const seen: Array = []; + const before = Date.now(); + await new AgentTask().run( + { + model: MODEL, + prompt: "?", + maxDurationMs: 60_000, + tools: [ + tool("clock", async (_input, context) => { + seen.push(context.deadline); + return "ok"; + }), + ], + }, + { registry } + ); + expect(seen).toHaveLength(1); + expect(seen[0]).toBeGreaterThanOrEqual(before + 60_000); + expect(seen[0]).toBeLessThanOrEqual(Date.now() + 60_000); + }); + + it("hands no deadline on a turn with no time budget", async () => { + script([{ calls: [{ id: "c1", name: "clock", input: {} }] }, { text: "done" }]); + const seen: Array = []; + await new AgentTask().run( + { + model: MODEL, + prompt: "?", + tools: [ + tool("clock", async (_input, context) => { + seen.push(context.deadline); + return "ok"; + }), + ], + }, + { registry } + ); + expect(seen).toEqual([undefined]); + }); +}); + +describe("AgentTask announceTimeLeft", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + const look = tool("look", async () => "looked"); + + it("appends the time left to the round's last tool result", async () => { + const seen = script([ + { + calls: [ + { id: "c1", name: "look", input: {} }, + { id: "c2", name: "look", input: {} }, + ], + }, + { text: "done" }, + ]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look], maxDurationMs: 600_000, announceTimeLeft: true }, + { registry } + ); + const sent = (seen[1]!.messages as ChatMessage[]).find((m) => m.role === "tool")!; + const [first, last] = sent.content as unknown as Array<{ + content: Array<{ type: string; text?: string }>; + }>; + expect(first!.content.map((block) => block.text)).toEqual(["looked"]); + expect(last!.content).toHaveLength(1); + expect(last!.content[0]!.type).toBe("text"); + expect(last!.content[0]!.text).toMatch(/^looked\n\n\[Time left: (9m [0-5]?\d|10m 0)s\]$/); + // The step records what the model was shown, notice included. + expect(output.steps![0]!.tools.map((t) => t.chars)).toEqual([ + "looked".length, + last!.content[0]!.text!.length, + ]); + }); + + it("says nothing about time unless asked", async () => { + const seen = script([{ calls: [{ id: "c1", name: "look", input: {} }] }, { text: "done" }]); + await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look], maxDurationMs: 600_000 }, + { registry } + ); + expect(JSON.stringify(seen[1]!.messages)).not.toContain("Time left"); + }); + + it("refuses announceTimeLeft without a time budget to announce", async () => { + script([{ text: "done" }]); + await expect( + new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look], announceTimeLeft: true }, + { registry } + ) + ).rejects.toBeInstanceOf(TaskConfigurationError); + }); +}); + +describe("AgentTask deadline and round retries", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + it("ends the turn instead of sleeping through a retry wait that passes the deadline", async () => { + let calls = 0; + const runFn: AiProviderRunFn = async () => { + calls++; + throw new RetryableJobError("429", new Date(Date.now() + 30_000)); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + const started = Date.now(); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [], maxDurationMs: 5_000 }, + { registry } + ); + expect(output.stopReason).toBe("budget"); + expect(Date.now() - started).toBeLessThan(5_000); + expect(calls).toBe(1); + }); + + it("does not start a round a slow progress write carried past the deadline", async () => { + let calls = 0; + const runFn: AiProviderRunFn = async () => { + calls++; + throw new Error("the model must not be called"); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + const task = new AgentTask(); + task.events.on("progress", (_progress, message) => { + if (message !== "Thinking") return; + const until = Date.now() + 300; + while (Date.now() < until) { + // A slow progress listener holds the turn past its deadline. + } + }); + const output = await task.run( + { model: MODEL, prompt: "?", tools: [], maxDurationMs: 200 }, + { registry } + ); + expect(output.stopReason).toBe("budget"); + expect(output.rounds).toBe(0); + expect(calls).toBe(0); + }); + + it("does not start a retry the wait carried past the deadline", async () => { + let calls = 0; + const runFn: AiProviderRunFn = async () => { + calls++; + // Asks for a 1 s wait with 1.5 s left; the progress write then runs long. + throw new RetryableJobError("429", new Date(Date.now() + 1_000)); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + const task = new AgentTask(); + task.events.on("progress", (_progress, message) => { + if (!message?.startsWith("Retrying")) return; + const until = Date.now() + 700; + while (Date.now() < until) { + // A slow progress listener holds the turn past its deadline. + } + }); + const output = await task.run( + { model: MODEL, prompt: "?", tools: [], maxDurationMs: 1_500, maxRoundRetries: 5 }, + { registry } + ); + expect(output.stopReason).toBe("budget"); + expect(calls).toBe(1); + }); +}); + +describe("AgentTask reviewPrompt", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + const reviewText = (messages: readonly ChatMessage[]): number => + messages.filter( + (m) => m.role === "user" && JSON.stringify(m.content).includes("Check your work.") + ).length; + + it("asks once for a review when the model first tries to finish", async () => { + const seen = script([{ text: "done" }, { text: "checked" }]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [], reviewPrompt: "Check your work." }, + { registry } + ); + expect(seen).toHaveLength(2); + expect(reviewText(output.messages)).toBe(1); + expect(output.stopReason).toBe("answered"); + }); + + it("lets the model go back to its tools after the review", async () => { + let looked = 0; + const seen = script([ + { text: "done" }, + { calls: [{ id: "c1", name: "look", input: {} }] }, + { text: "fixed" }, + ]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "?", + tools: [ + tool("look", async () => { + looked++; + return "looked"; + }), + ], + reviewPrompt: "Check your work.", + }, + { registry } + ); + expect(seen).toHaveLength(3); + expect(looked).toBe(1); + expect(reviewText(output.messages)).toBe(1); + }); + + it("sends no review when no round is left to act on it", async () => { + const seen = script([{ text: "done" }]); + await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [], reviewPrompt: "Check your work.", maxRounds: 1 }, + { registry } + ); + expect(seen).toHaveLength(1); + }); + + it("keeps the turn's prompt and work in the review round when history is over budget", async () => { + const seen = script([{ calls: [{ id: "c1", name: "look", input: {} }] }, { text: "done" }]); + await new AgentTask().run( + { + model: MODEL, + prompt: "Write the report.", + messages: [ + { role: "user", content: [{ type: "text", text: "an earlier question" }] }, + { role: "assistant", content: [{ type: "text", text: "an earlier answer" }] }, + ], + tools: [tool("look", async () => "x".repeat(2_000))], + reviewPrompt: "Check your work.", + maxHistoryChars: 500, + }, + { registry } + ); + expect(seen).toHaveLength(3); + const review = seen[2]!.messages!; + // The earlier turn is cut to fit; the review is not a turn of its own. + expect(JSON.stringify(review)).not.toContain("an earlier question"); + expect(JSON.stringify(review)).toContain("Write the report."); + expect(review.some((m) => m.role === "tool")).toBe(true); + expect(reviewText(review)).toBe(1); + }); +}); diff --git a/packages/test/src/test/ai/AgentTaskNativeTurn.test.ts b/packages/test/src/test/ai/AgentTaskNativeTurn.test.ts new file mode 100644 index 0000000000..415fd11f11 --- /dev/null +++ b/packages/test/src/test/ai/AgentTaskNativeTurn.test.ts @@ -0,0 +1,133 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn, ChatMessage, ModelConfig, ToolCallingTaskInput } from "@workglow/ai"; +import { + AgentTask, + AiProviderRegistry, + DirectExecutionStrategy, + getAiProviderRegistry, + setAiProviderRegistry, +} from "@workglow/ai"; +import { Container, ServiceRegistry } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const PROVIDER = "mock-native-turn-provider"; + +const MODEL: ModelConfig = { + model_id: "mock/native-1", + provider: PROVIDER, + title: "", + description: "", + capabilities: ["tool-use"], + provider_config: {}, + metadata: {}, +}; + +interface Round { + readonly reasoning?: string; + readonly text?: string; + readonly calls?: ReadonlyArray<{ id: string; name: string; input: Record }>; + readonly native?: { readonly provider: string; readonly payload: string }; +} + +/** A scripted model that may hand back its own turn, and records what each round was sent. */ +function script(rounds: readonly Round[]): ToolCallingTaskInput[] { + const seen: ToolCallingTaskInput[] = []; + const runFn: AiProviderRunFn = async (input, _model, _signal, emit) => { + const round = rounds[Math.min(seen.length, rounds.length - 1)]!; + seen.push(input as ToolCallingTaskInput); + if (round.reasoning) + emit({ type: "text-delta", port: "reasoning", textDelta: round.reasoning }); + if (round.text) emit({ type: "text-delta", port: "text", textDelta: round.text }); + if (round.calls?.length) { + emit({ type: "object-delta", port: "toolCalls", objectDelta: [...round.calls] }); + } + if (round.native) { + emit({ type: "object-delta", port: "nativeTurn", objectDelta: round.native }); + } + emit({ type: "finish", data: {} }); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + return seen; +} + +const look = { + name: "look", + description: "Look", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => "looked", +} as const; + +describe("AgentTask native turn", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + it("keeps a round's native turn on its reply and sends it back next round", async () => { + const seen = script([ + { + calls: [{ id: "c1", name: "look", input: {} }], + native: { provider: "mock", payload: '[{"kind":"reasoning","enc":"xyz"}]' }, + }, + { text: "done" }, + ]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look] }, + { registry } + ); + const reply = output.messages.find((m) => m.role === "assistant")!; + expect(reply.content[0]).toEqual({ + type: "reasoning", + text: "", + provider: "mock", + payload: '[{"kind":"reasoning","enc":"xyz"}]', + }); + const sent = (seen[1]!.messages as ChatMessage[]).find((m) => m.role === "assistant")!; + expect(sent.content[0]).toEqual({ + type: "reasoning", + text: "", + provider: "mock", + payload: '[{"kind":"reasoning","enc":"xyz"}]', + }); + }); + + it("carries text reasoning and the native turn on one block", async () => { + script([ + { + reasoning: "Plan.", + calls: [{ id: "c1", name: "look", input: {} }], + native: { provider: "mock", payload: "[]" }, + }, + { text: "done" }, + ]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look] }, + { registry } + ); + const blocks = output.messages + .find((m) => m.role === "assistant")! + .content.filter((b) => b.type === "reasoning"); + expect(blocks).toEqual([{ type: "reasoning", text: "Plan.", provider: "mock", payload: "[]" }]); + }); + + it("records no native turn on a reply that has neither text nor calls", async () => { + script([{ native: { provider: "mock", payload: "[]" } }]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [] }, + { registry } + ); + expect(output.messages.filter((m) => m.role === "assistant")).toEqual([]); + }); +}); diff --git a/packages/test/src/test/ai/AgentTaskReasoningAndImages.test.ts b/packages/test/src/test/ai/AgentTaskReasoningAndImages.test.ts new file mode 100644 index 0000000000..d1b7c93755 --- /dev/null +++ b/packages/test/src/test/ai/AgentTaskReasoningAndImages.test.ts @@ -0,0 +1,379 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn, ChatMessage, ModelConfig, ToolCallingTaskInput } from "@workglow/ai"; +import { + AgentTask, + AiProviderRegistry, + DirectExecutionStrategy, + getAiProviderRegistry, + setAiProviderRegistry, + ToolResultContent, +} from "@workglow/ai"; +import { RetryableJobError } from "@workglow/job-queue"; +import { Container, ServiceRegistry } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const PROVIDER = "mock-reasoning-provider"; + +const MODEL: ModelConfig = { + model_id: "mock/reasoning-1", + provider: PROVIDER, + title: "", + description: "", + capabilities: ["tool-use"], + provider_config: {}, + metadata: {}, +}; + +interface Round { + readonly reasoning?: string; + readonly text?: string; + readonly calls?: ReadonlyArray<{ id: string; name: string; input: Record }>; + /** Streamed, then thrown instead of answering, as a provider failing mid-stream does. */ + readonly failWith?: { + readonly error: Error; + readonly reasoning?: string; + readonly text?: string; + }; +} + +/** A scripted model that streams reasoning beside its reply, and records what each round was sent. */ +function script(rounds: readonly Round[]): ToolCallingTaskInput[] { + const seen: ToolCallingTaskInput[] = []; + const runFn: AiProviderRunFn = async (input, _model, _signal, emit) => { + const round = rounds[Math.min(seen.length, rounds.length - 1)]!; + seen.push(input as ToolCallingTaskInput); + if (round.failWith) { + const { error, reasoning, text } = round.failWith; + if (reasoning) emit({ type: "text-delta", port: "reasoning", textDelta: reasoning }); + if (text) emit({ type: "text-delta", port: "text", textDelta: text }); + throw error; + } + if (round.reasoning) + emit({ type: "text-delta", port: "reasoning", textDelta: round.reasoning }); + if (round.text) emit({ type: "text-delta", port: "text", textDelta: round.text }); + if (round.calls?.length) { + emit({ type: "object-delta", port: "toolCalls", objectDelta: [...round.calls] }); + } + emit({ type: "finish", data: {} }); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + return seen; +} + +describe("AgentTask reasoning and tool images", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + it("keeps a round's reasoning on its reply, so the next round is sent it", async () => { + const seen = script([ + { + reasoning: "The file is the place to start.", + calls: [{ id: "c1", name: "look", input: {} }], + }, + { reasoning: "Done thinking.", text: "It is fine." }, + ]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "Check it", + tools: [ + { + name: "look", + description: "Look", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => "looked", + }, + ], + }, + { registry } + ); + + const replies = output.messages.filter((message) => message.role === "assistant"); + expect(replies[0]!.content[0]).toEqual({ + type: "reasoning", + text: "The file is the place to start.", + }); + expect(replies[1]!.content).toEqual([ + { type: "reasoning", text: "Done thinking." }, + { type: "text", text: "It is fine." }, + ]); + // The second round was sent the first round's reasoning with its call. + const sentReply = (seen[1]!.messages as ChatMessage[]).find((m) => m.role === "assistant"); + expect(sentReply!.content.map((block) => block.type)).toEqual(["reasoning", "tool_use"]); + // Reasoning is not part of what the turn said. + expect(output.text).toBe("It is fine."); + }); + + it("records no reply for a round that only reasoned", async () => { + script([{ reasoning: "Hmm." }]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "Say nothing", tools: [] }, + { registry } + ); + expect(output.messages.filter((message) => message.role === "assistant")).toEqual([]); + }); + + it("hands the model an image a tool returned through ToolResultContent", async () => { + const seen = script([ + { calls: [{ id: "c1", name: "snap", input: {} }] }, + { text: "I see a board." }, + ]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "What is in the picture?", + tools: [ + { + name: "snap", + description: "Take a picture", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => + new ToolResultContent([ + { type: "text", text: "Image: board.png" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ]), + }, + ], + }, + { registry } + ); + + const result = output.messages.find((message) => message.role === "tool")!.content[0]!; + expect(result).toEqual({ + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "Image: board.png" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + is_error: undefined, + }); + const sentTool = (seen[1]!.messages as ChatMessage[]).find((m) => m.role === "tool")!; + expect(JSON.stringify(sentTool)).toContain('"type":"image"'); + // The step record counts the text the model read, not the image bytes. + expect(output.steps[0]!.tools[0]!.chars).toBe("Image: board.png".length); + }); + + it("keeps a tool's text beside the image it describes, and the text within budget", async () => { + script([{ calls: [{ id: "c1", name: "pair", input: {} }] }, { text: "done" }]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "Compare them", + maxToolResultChars: 12, + tools: [ + { + name: "pair", + description: "Two pictures", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => + new ToolResultContent([ + { type: "text", text: "before:" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "after:" }, + { type: "text", text: "second" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + ]), + }, + ], + }, + { registry } + ); + + const result = output.messages.find((message) => message.role === "tool")!.content[0]!; + expect(result.type === "tool_result" && result.content).toEqual([ + { type: "text", text: "before:" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "after… [truncated, 20 chars total]" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + ]); + }); + + it("counts the separator joining a tool's adjacent text blocks, as the model reads them", async () => { + script([{ calls: [{ id: "c1", name: "two", input: {} }] }, { text: "done" }]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "go", + tools: [ + { + name: "two", + description: "Two blocks", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => + new ToolResultContent([ + { type: "text", text: "a" }, + { type: "text", text: "b" }, + ]), + }, + ], + }, + { registry } + ); + const result = output.messages.find((message) => message.role === "tool")!.content[0]!; + expect(result.type === "tool_result" && result.content).toEqual([ + { type: "text", text: "a\nb" }, + ]); + expect(output.steps[0]!.tools[0]!.chars).toBe(3); + }); + + it("says text was left out when the budget ran out exactly at an earlier block", async () => { + script([{ calls: [{ id: "c1", name: "pair", input: {} }] }, { text: "done" }]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "Compare them", + maxToolResultChars: 7, + tools: [ + { + name: "pair", + description: "Two pictures", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => + new ToolResultContent([ + { type: "text", text: "before:" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "after" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + { type: "text", text: "end" }, + ]), + }, + ], + }, + { registry } + ); + + const result = output.messages.find((message) => message.role === "tool")!.content[0]!; + expect(result.type === "tool_result" && result.content).toEqual([ + { type: "text", text: "before:" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "… [truncated, 15 chars total]" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + ]); + }); + + it("refuses a round timeout longer than a timer can be set", async () => { + script([{ text: "done" }]); + await expect( + new AgentTask().run( + { model: MODEL, prompt: "hi", tools: [], roundTimeoutMs: 2_147_483_648 }, + { registry } + ) + ).rejects.toThrow(/roundTimeoutMs|2147483647|maximum/); + }); + + describe("reasoning on the turn's own port", () => { + const LOOK = { + name: "look", + description: "Look", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => "looked", + } as const; + + function listen(task: AgentTask): Array<{ port: string; delta: string }> { + const events: Array<{ port: string; delta: string }> = []; + task.subscribe("stream_chunk", (event) => { + if (event.type === "text-delta") { + events.push({ port: event.port ?? "text", delta: event.textDelta }); + } + }); + return events; + } + + it("streams a round's reasoning on its own port, apart from the text", async () => { + script([ + { reasoning: "Check the file first.", calls: [{ id: "c1", name: "look", input: {} }] }, + { reasoning: "It looks fine.", text: "All good." }, + ]); + const task = new AgentTask(); + const events = listen(task); + + const output = await task.run( + { model: MODEL, prompt: "Check it", tools: [LOOK], approval: "never" }, + { registry } + ); + + expect(events.filter((event) => event.port === "reasoning").map((e) => e.delta)).toEqual([ + "Check the file first.", + // A round's reasoning is told from the one before it. + "\n\n", + "It looks fine.", + ]); + expect(events.filter((event) => event.port === "text").map((e) => e.delta)).toEqual([ + "All good.", + ]); + expect(output.reasoning).toBe("Check the file first.\n\nIt looks fine."); + expect(output.text).toBe("All good."); + }); + + it("leaves reasoning off the output when the model streamed none", async () => { + script([{ text: "Plain answer." }]); + const task = new AgentTask(); + const events = listen(task); + + const output = await task.run( + { model: MODEL, prompt: "Hi", tools: [], approval: "never" }, + { registry } + ); + + expect(events.some((event) => event.port === "reasoning")).toBe(false); + expect(output.reasoning).toBeUndefined(); + expect("reasoning" in output).toBe(false); + expect(output.text).toBe("Plain answer."); + }); + + it("does not separate a round that reasoned from one before it that did not", async () => { + script([ + { calls: [{ id: "c1", name: "look", input: {} }] }, + { reasoning: "Only now.", text: "Done." }, + ]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "Check it", tools: [LOOK], approval: "never" }, + { registry } + ); + expect(output.reasoning).toBe("Only now."); + }); + + it("drops a failed attempt's reasoning and text, so the retry's are not joined to them", async () => { + script([ + { + failWith: { + error: new RetryableJobError("overloaded", new Date(Date.now())), + reasoning: "Half a thought", + text: "Half an ans", + }, + }, + { reasoning: "A whole thought.", text: "A whole answer." }, + ]); + const task = new AgentTask(); + const events = listen(task); + + const output = await task.run( + { model: MODEL, prompt: "?", tools: [], approval: "never" }, + { registry } + ); + + expect(output.steps[0]!.attempts).toBe(2); + expect(output.reasoning).toBe("A whole thought."); + expect(output.text).toBe("A whole answer."); + expect(events).toEqual([ + { port: "reasoning", delta: "A whole thought." }, + { port: "text", delta: "A whole answer." }, + ]); + }); + }); +}); diff --git a/packages/test/src/test/ai/AgentTaskTurnControls.test.ts b/packages/test/src/test/ai/AgentTaskTurnControls.test.ts index e0e0861e60..77e3e13ab8 100644 --- a/packages/test/src/test/ai/AgentTaskTurnControls.test.ts +++ b/packages/test/src/test/ai/AgentTaskTurnControls.test.ts @@ -333,6 +333,42 @@ describe("AgentTask turn controls", () => { ]); expect(overlap.peak).toBe(1); }); + + it("does not start a tool whose approval arrives after the deadline", async () => { + script([{ calls: [{ id: "a", name: "gated", input: {} }] }, { text: "done" }]); + const slowApproval: IHumanConnector = { + send: async (request) => { + await new Promise((resolve) => setTimeout(resolve, 120)); + return { requestId: request.requestId, action: "accept", content: undefined, done: true }; + }, + }; + registry.registerInstance(HUMAN_CONNECTOR, slowApproval); + let ran = false; + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "?", + tools: [ + { + name: "gated", + description: "gated", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + requiresApproval: true, + execute: async () => { + ran = true; + return "ran"; + }, + }, + ], + maxDurationMs: 50, + }, + { registry } + ); + expect(ran).toBe(false); + const [result] = toolResults(output.messages); + expect(result!.is_error).toBe(true); + expect(JSON.stringify(result!.content)).toContain("time ran out"); + }); }); describe("steps", () => { diff --git a/packages/test/src/test/task/MessageConversion.test.ts b/packages/test/src/test/task/MessageConversion.test.ts index 8c4def1bc8..89ba5318af 100644 --- a/packages/test/src/test/task/MessageConversion.test.ts +++ b/packages/test/src/test/task/MessageConversion.test.ts @@ -5,7 +5,12 @@ */ import type { ToolCallingTaskInput, ToolDefinition } from "@workglow/ai"; -import { toOpenAIMessages, toTextFlatMessages } from "@workglow/ai"; +import { + liftSystemMessages, + normalizeChatMessages, + toOpenAIMessages, + toTextFlatMessages, +} from "@workglow/ai"; import { describe, expect, test } from "vitest"; const dummyTools: ToolDefinition[] = [ @@ -45,6 +50,72 @@ describe("toOpenAIMessages", () => { expect(msgs[1].role).toBe("user"); }); + test("keeps a nested tool_use a tool returned, as JSON, whichever way images go", () => { + const nested = { type: "tool_use", id: "x", name: "inner", input: { a: 1 } }; + const input = makeInput({ + messages: [ + { role: "assistant", content: [{ type: "tool_use", id: "c1", name: "t", input: {} }] }, + { + role: "tool", + content: [{ type: "tool_result", tool_use_id: "c1", content: [nested] }], + }, + ], + } as any); + expect(toOpenAIMessages(input, { toolImagesInUserMessage: true })[1]).toEqual({ + role: "tool", + content: JSON.stringify(nested), + tool_call_id: "c1", + }); + expect(toOpenAIMessages(input)[1]!.content).toEqual([ + { type: "text", text: JSON.stringify(nested) }, + ]); + }); + + test("passes an image_url part a caller wrote for chat-completions through", () => { + const part = { type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } }; + const msgs = toOpenAIMessages( + makeInput({ + messages: [{ role: "user", content: [{ type: "text", text: "what is this" }, part] }], + } as any) + ); + expect(msgs).toEqual([ + { role: "user", content: [{ type: "text", text: "what is this" }, part] }, + ]); + }); + + test("joins system turns in the history to the system prompt", () => { + const msgs = toOpenAIMessages( + makeInput({ + systemPrompt: "base", + messages: [ + { role: "system", content: [{ type: "text", text: "rule one" }] }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ], + } as any) + ); + expect(msgs).toEqual([ + { role: "system", content: "base\n\nrule one" }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ]); + const only = toOpenAIMessages( + makeInput({ + messages: [ + { role: "system", content: [{ type: "text", text: "rule" }] }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ], + } as any) + ); + expect(only[0]).toEqual({ role: "system", content: "rule" }); + // A history of system turns alone is still the history: the prompt is not sent. + const systemOnly = toOpenAIMessages( + makeInput({ + prompt: "stale", + messages: [{ role: "system", content: [{ type: "text", text: "rule" }] }], + } as any) + ); + expect(systemOnly).toEqual([{ role: "system", content: "rule" }]); + }); + test("should convert multi-turn messages with user and assistant", () => { const input = makeInput({ messages: [ @@ -125,7 +196,7 @@ describe("toOpenAIMessages", () => { }); }); - test("should set content to null for empty assistant text", () => { + test("should send empty assistant text as an empty string when there are no tool calls", () => { const input = makeInput({ messages: [ { role: "user", content: [{ type: "text", text: "Go" }] }, @@ -134,7 +205,7 @@ describe("toOpenAIMessages", () => { }); const msgs = toOpenAIMessages(input); - expect(msgs[1].content).toBeNull(); + expect(msgs[1].content).toBe(""); }); test("should convert user message with image content blocks to OpenAI format", () => { @@ -217,6 +288,279 @@ describe("toOpenAIMessages", () => { expect(parts[0]).toEqual({ type: "text", text: "Plain text" }); expect(parts[1].type).toBe("image_url"); }); + + test("sends an empty tool result as an empty string, not an empty parts array", () => { + const out = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "t", input: {} }], + }, + { role: "tool", content: [{ type: "tool_result", tool_use_id: "c1", content: [] }] }, + ] as any, + }) + ); + expect(out.at(-1)).toEqual({ role: "tool", content: "", tool_call_id: "c1" }); + }); + + test("sends a reasoning-only assistant turn with empty-string content", () => { + const out = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { role: "assistant", content: [{ type: "reasoning", text: "thinking" }] }, + { role: "user", content: [{ type: "text", text: "again" }] }, + ] as any, + }) + ); + expect(out[1]).toEqual({ role: "assistant", content: "" }); + }); + + test("keeps null content for an assistant turn that only calls tools", () => { + const out = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "t", input: {} }], + }, + ] as any, + }) + ); + expect(out[1]!.content).toBeNull(); + expect(out[1]!.tool_calls).toHaveLength(1); + }); + + describe("reasoning and tool images", () => { + const history = makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "Look" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "I should read the file." }, + { type: "tool_use", id: "c1", name: "read", input: { path: "board.png" } }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "Image: board.png" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + is_error: undefined, + }, + ], + }, + ], + }); + + test("drops reasoning blocks unless asked to replay them", () => { + const msgs = toOpenAIMessages(history); + expect(msgs[1]).not.toHaveProperty("reasoning_content"); + expect(msgs[1]!.content).toBeNull(); + }); + + test("replays reasoning as reasoning_content when asked", () => { + const msgs = toOpenAIMessages(history, { replayReasoning: true }); + expect(msgs[1]).toMatchObject({ + role: "assistant", + reasoning_content: "I should read the file.", + tool_calls: [{ id: "c1" }], + }); + }); + + test("keeps tool images inline by default", () => { + const msgs = toOpenAIMessages(history); + expect(msgs[2]).toMatchObject({ role: "tool", tool_call_id: "c1" }); + expect(msgs[2]!.content).toEqual([ + { type: "text", text: "Image: board.png" }, + { type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } }, + ]); + }); + + test("moves tool images into a user message after the turn's results when asked", () => { + const msgs = toOpenAIMessages(history, { toolImagesInUserMessage: true }); + expect(msgs[2]).toEqual({ + role: "tool", + tool_call_id: "c1", + content: "Image: board.png\n[image 1] follows in the next message", + }); + expect(msgs[3]).toEqual({ + role: "user", + content: [ + { type: "text", text: "Images returned by the tool calls above:" }, + { type: "text", text: "[image 1]" }, + { type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } }, + ], + }); + }); + + test("marks each moved image where it stood, numbered across the turn's results", () => { + const image = (data: string) => ({ type: "image", mimeType: "image/png", data }); + const msgs = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "Compare" }] }, + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "before:" }, + image("AAAA"), + { type: "text", text: "after:" }, + image("BBBB"), + ], + }, + { type: "tool_result", tool_use_id: "c2", content: [image("CCCC")] }, + ], + }, + ], + } as never), + { toolImagesInUserMessage: true } + ); + expect(msgs[2]!.content).toBe( + "before:\n[image 1] follows in the next message\nafter:\n[image 2] follows in the next message" + ); + expect(msgs[3]!.content).toBe("[image 3] follows in the next message"); + const labels = ( + msgs[4]!.content as Array<{ type: string; text?: string; image_url?: { url: string } }> + ) + .slice(1) + .map((part) => part.text ?? part.image_url!.url.slice(-4)); + expect(labels).toEqual(["[image 1]", "AAAA", "[image 2]", "BBBB", "[image 3]", "CCCC"]); + }); + }); + + test("holds tool images until every separate tool message of the turn is out", () => { + const msgs = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "Look" }] }, + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "c2", content: [{ type: "text", text: "ok" }] }, + ], + }, + { role: "user", content: [{ type: "text", text: "Next" }] }, + ], + } as never), + { toolImagesInUserMessage: true } + ); + expect(msgs.map((m) => (m.role === "tool" ? `tool:${m.tool_call_id}` : m.role))).toEqual([ + "user", + "assistant", + "tool:c1", + "tool:c2", + "user", + "user", + ]); + expect(JSON.stringify(msgs[4]!.content)).toContain("Images returned by the tool calls above"); + }); + + describe("native turn replay", () => { + const withBlock = (block: Record) => + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "Look" }] }, + { + role: "assistant", + content: [block, { type: "tool_use", id: "c1", name: "read", input: {} }], + }, + ], + } as never); + const items = [{ type: "reasoning", id: "rs_1", encrypted_content: "ENC" }]; + + test("attaches the items of a reasoning block this provider produced", () => { + const msgs = toOpenAIMessages( + withBlock({ + type: "reasoning", + text: "", + provider: "openai", + payload: JSON.stringify(items), + }), + { nativeTurnProvider: "openai" } + ); + expect(msgs[1]!.native_items).toEqual(items); + }); + + test("ignores a block another provider produced", () => { + const msgs = toOpenAIMessages( + withBlock({ + type: "reasoning", + text: "", + provider: "anthropic", + payload: JSON.stringify(items), + }), + { nativeTurnProvider: "openai" } + ); + expect(msgs[1]).not.toHaveProperty("native_items"); + }); + + test("ignores a payload that is not JSON, not an array, or not typed items", () => { + for (const payload of [ + "{not json", + JSON.stringify({ a: 1 }), + "[]", + "[null]", + JSON.stringify([{ type: "reasoning" }, 7]), + JSON.stringify([{ id: "rs_1" }]), + ]) { + const msgs = toOpenAIMessages( + withBlock({ type: "reasoning", text: "", provider: "openai", payload }), + { nativeTurnProvider: "openai" } + ); + expect(msgs[1]).not.toHaveProperty("native_items"); + } + }); + + test("attaches nothing unless asked", () => { + const msgs = toOpenAIMessages( + withBlock({ + type: "reasoning", + text: "", + provider: "openai", + payload: JSON.stringify(items), + }) + ); + expect(msgs[1]).not.toHaveProperty("native_items"); + }); + }); }); // ======================================================================== @@ -367,3 +711,36 @@ describe("toTextFlatMessages", () => { expect(msgs[0]).toEqual({ role: "user", content: "Look at this" }); }); }); + +describe("normalizeChatMessages", () => { + test("turns bare-string content into a text block and leaves blocks alone", () => { + const blocks = [{ type: "image" as const, mimeType: "image/png", data: "AAAA" }]; + expect( + normalizeChatMessages([ + { role: "user", content: "hi" }, + { role: "user", content: blocks }, + ]) + ).toEqual([ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { role: "user", content: blocks }, + ]); + }); +}); + +describe("liftSystemMessages", () => { + test("moves system turns into the system prompt, after the one given", () => { + const lifted = liftSystemMessages( + [ + { role: "system", content: [{ type: "text", text: "rule one" }] }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ], + "base" + ); + expect(lifted.systemPrompt).toBe("base\n\nrule one"); + expect(lifted.messages).toEqual([{ role: "user", content: [{ type: "text", text: "hi" }] }]); + }); + + test("leaves the system prompt undefined when there is none anywhere", () => { + expect(liftSystemMessages([], undefined).systemPrompt).toBeUndefined(); + }); +}); From d534a043026fd200476698b4532e1d9ac61444ea Mon Sep 17 00:00:00 2001 From: Steven Roussey Date: Mon, 12 Oct 2026 05:34:45 +0000 Subject: [PATCH 2/7] openai, anthropic: hand a model its own turn back each round OpenAI replays the Responses items with their encrypted reasoning, and Anthropic its content blocks with their signed thinking, keyed by provider and model so another model gets the rebuilt turn. A turn is replayed only while its call ids still match, and emitted only when it is truly replayable: OpenAI requests encrypted reasoning only for a reasoning model and rebuilds earlier turns when it does not ask for it; Anthropic retries once with thinking stripped when a request is rejected as edited. Both send tool-result images on a user turn after the results (OpenAI) or beside them (Anthropic). --- .../provider-utils/OpenAIShapedResponses.ts | 112 +++- .../AnthropicCheckpointParams.test.ts | 38 ++ .../ai-provider/Anthropic_NativeTurn.test.ts | 508 ++++++++++++++++++ .../Anthropic_ReasoningAndToolImages.test.ts | 152 ++++++ .../OpenAIGeminiCheckpointParams.test.ts | 115 ++++ .../ai-provider/OpenAIShapedResponses.test.ts | 141 +++++ .../ai-provider/OpenAI_NativeTurn.test.ts | 250 +++++++++ .../ai/common/Anthropic_CacheCheckpoint.ts | 7 +- .../src/ai/common/Anthropic_TextGeneration.ts | 3 +- .../src/ai/common/Anthropic_ToolCalling.ts | 423 ++++++++++++--- .../src/ai/common/OpenAI_CacheCheckpoint.ts | 41 +- .../src/ai/common/OpenAI_ModelSchema.ts | 6 + .../src/ai/common/OpenAI_TextGeneration.ts | 17 +- .../src/ai/common/OpenAI_ToolCalling.ts | 40 +- 14 files changed, 1745 insertions(+), 108 deletions(-) create mode 100644 packages/test/src/test/ai-provider/Anthropic_NativeTurn.test.ts create mode 100644 packages/test/src/test/ai-provider/Anthropic_ReasoningAndToolImages.test.ts create mode 100644 packages/test/src/test/ai-provider/OpenAI_NativeTurn.test.ts diff --git a/packages/ai/src/provider-utils/OpenAIShapedResponses.ts b/packages/ai/src/provider-utils/OpenAIShapedResponses.ts index c4e83e86bb..3e1187a68d 100644 --- a/packages/ai/src/provider-utils/OpenAIShapedResponses.ts +++ b/packages/ai/src/provider-utils/OpenAIShapedResponses.ts @@ -43,6 +43,8 @@ export interface OpenAIResponsesMessageSource { readonly function?: { readonly name?: string; readonly arguments?: string }; }[]; readonly tool_call_id?: string; + /** The provider's own output items for this assistant turn, replayed verbatim. */ + readonly native_items?: readonly unknown[] | undefined; } /** A Responses request `input` item — a role message or a tool item. */ @@ -143,6 +145,18 @@ export function buildResponsesInput(args: { } if (message.role === "assistant") { + // The provider's own turn, replayed as it produced it: its reasoning only + // continues from items it can decrypt, and only while the calls still + // match — a host that renamed or dropped a call gets the rebuilt turn. + const native = message.native_items; + if ( + native && + sameCalls(native, message.tool_calls ?? []) && + nativeText(native) === contentToString(message.content) + ) { + items.push(...(native as OpenAIResponsesInputItem[])); + continue; + } // Preserve any assistant text, then each tool call as a `function_call` // item (Responses represents tool calls as items, not message fields). const text = contentToString(message.content); @@ -171,6 +185,66 @@ export function buildResponsesInput(args: { }; } +/** + * The text the native items said, run together as the turn records it: an + * edited or removed reply means the items are no longer the turn, so it is + * rebuilt rather than replayed. + */ +function nativeText(native: readonly unknown[]): string { + return native + .filter((item) => (item as { type?: unknown } | null)?.type === "message") + .flatMap((item) => { + const content = (item as { content?: unknown }).content; + return Array.isArray(content) ? content : []; + }) + .map((part) => { + const p = part as { type?: unknown; text?: unknown }; + return p?.type === "output_text" && typeof p.text === "string" ? p.text : ""; + }) + .join(""); +} + +/** A call's arguments as one canonical string: the API's own spacing is not part of it. */ +function canonicalArguments(args: unknown): string | undefined { + if (typeof args !== "string") return undefined; + try { + return JSON.stringify(JSON.parse(args === "" ? "{}" : args)); + } catch { + return undefined; + } +} + +/** + * Whether the native items' calls are the message's calls exactly: the same + * ids, names and arguments, in order. A host that changed any of them gets the + * rebuilt turn, since replaying the items would send the call it replaced. + */ +function sameCalls( + native: readonly unknown[], + calls: readonly { + readonly id?: string; + readonly function?: { readonly name?: string; readonly arguments?: string }; + }[] +): boolean { + const nativeCalls = native.filter( + (item): item is { type: string; call_id: string; name?: unknown; arguments?: unknown } => + (item as { type?: string } | null)?.type === "function_call" + ); + return ( + nativeCalls.length === calls.length && + nativeCalls.every((item, i) => { + const call = calls[i]!; + const args = canonicalArguments(item.arguments); + return ( + item.call_id === (call.id ?? "") && + item.name === call.function?.name && + args !== undefined && + args === canonicalArguments(call.function?.arguments ?? "") + ); + }) + ); +} + export function buildResponsesTools(tools: readonly ToolDefinition[]): OpenAIResponsesTool[] { return tools.map((t) => ({ type: "function" as const, @@ -246,11 +320,21 @@ interface ResponsesToolCallEntry { * * Pass `promptText` to emit a provisional ↑ estimate before the first delta — * Responses only attaches billed usage to the terminal lifecycle event. + * + * Pass `nativeTurnProvider` to also emit the turn's finished output items, + * verbatim and in order, as one `nativeTurn` `object-delta` keyed by that + * provider, so the next round can replay them. Nothing is emitted when the turn + * has no items, or when any reasoning item came back without + * `encrypted_content`: replaying that item makes the API look it up by id, + * which fails under `store: false`, zero data retention, or once it expired. */ export async function accumulateOpenAIResponsesStream>( stream: AsyncIterable, emit: (event: StreamEvent) => void, - options: { readonly promptText?: string | undefined } = {} + options: { + readonly promptText?: string | undefined; + readonly nativeTurnProvider?: string | undefined; + } = {} ): Promise { // Keyed by the output item's stable id (`item.id` on `output_item.{added,done}`, // `item_id` on `function_call_arguments.delta`). Falls back to `output_index` @@ -258,6 +342,9 @@ export async function accumulateOpenAIResponsesStream(); + // Every finished output item by position, for replay on the next round; kept + // only when the caller asked for a native turn. + const outputItems = new Map(); // Responses only reports billed usage on the terminal lifecycle event; when the // caller supplies promptText, estimate ↑ before the first delta and ↓ from // content / tool-arg deltas so the CLI counter moves during the call. The @@ -343,6 +430,12 @@ export async function accumulateOpenAIResponsesStream 0) { + const items = [...outputItems.entries()].sort((a, b) => a[0] - b[0]).map(([, item]) => item); + const replayable = items.every((item) => { + const { type, encrypted_content } = item as { type?: unknown; encrypted_content?: unknown }; + return ( + type !== "reasoning" || (typeof encrypted_content === "string" && encrypted_content !== "") + ); + }); + if (replayable) { + emit({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: options.nativeTurnProvider, payload: JSON.stringify(items) }, + } as StreamEvent); + } + } + provisionalUsage?.flush(); return usage; } diff --git a/packages/test/src/test/ai-provider/AnthropicCheckpointParams.test.ts b/packages/test/src/test/ai-provider/AnthropicCheckpointParams.test.ts index a143081088..c45993049d 100644 --- a/packages/test/src/test/ai-provider/AnthropicCheckpointParams.test.ts +++ b/packages/test/src/test/ai-provider/AnthropicCheckpointParams.test.ts @@ -102,6 +102,44 @@ describe("applyAnthropicPrefixReplay", () => { expect((params.system as any[])[0].text).toBe("sys"); }); + it("replays a prefix turn's signed thinking for the model that wrote it", () => { + const thinking = { type: "thinking", thinking: "plan", signature: "SIG" }; + const turn = (provider: string) => ({ + ...prefix, + messages: [ + prefix.messages[0]!, + { + role: "assistant" as const, + content: [ + { + type: "reasoning" as const, + text: "plan", + provider, + payload: JSON.stringify([thinking, { type: "text", text: "hi" }]), + }, + { type: "text" as const, text: "hi" }, + ], + }, + ], + }); + const replay = (provider: string): any[] => { + const params: Record = { + model: "claude-x", + messages: [{ role: "user", content: [{ type: "text", text: "tail" }] }], + }; + applyAnthropicPrefixReplay(params, { sessionId: "ckpt", prefix: turn(provider) }); + return (params.messages as any[])[1].content; + }; + expect(replay("anthropic:claude-x")[0]).toEqual(thinking); + expect(replay("anthropic:other")[0]).toEqual({ + type: "text", + text: "hi", + cache_control: { type: "ephemeral" }, + }); + const warm = buildAnthropicCheckpointParams(turn("anthropic:claude-x") as any, "claude-x"); + expect((warm.messages as any[])[1].content[0]).toEqual(thinking); + }); + it("annotates the final turn when emitting a chained checkpoint", () => { const session: AiSessionContext = { sessionId: "ckpt", emitCheckpointId: "next", prefix }; const params: Record = { diff --git a/packages/test/src/test/ai-provider/Anthropic_NativeTurn.test.ts b/packages/test/src/test/ai-provider/Anthropic_NativeTurn.test.ts new file mode 100644 index 0000000000..26a539095b --- /dev/null +++ b/packages/test/src/test/ai-provider/Anthropic_NativeTurn.test.ts @@ -0,0 +1,508 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn } from "@workglow/ai"; +import { ANTHROPIC, _testOnly } from "@workglow/anthropic/ai"; +import { _testOnly as runtimeTestOnly } from "@workglow/anthropic/ai-runtime"; +import { getLogger } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const { ANTHROPIC_RUN_FNS, setAnthropicClientForTests } = _testOnly; + +const MODEL_NAME = "claude-sonnet-5-5"; +const NATIVE_KEY = `anthropic:${MODEL_NAME}`; + +function toolCallingRunFn(): AiProviderRunFn { + const registration = ANTHROPIC_RUN_FNS.find(({ serves }) => + (serves as readonly string[]).includes("tool-use") + ); + expect(registration).toBeDefined(); + return registration!.runFn as AiProviderRunFn; +} + +const model = { + model_id: MODEL_NAME, + title: "", + description: "", + provider: ANTHROPIC, + provider_config: { model_name: MODEL_NAME, api_key: "test-key" }, + capabilities: ["text.generation", "tool-use"], + metadata: {}, +} as never; + +type Scripted = readonly unknown[] | Error; + +const thinkingThenCall = [ + { type: "content_block_start", index: 0, content_block: { type: "thinking", thinking: "" } }, + { + type: "content_block_delta", + index: 0, + delta: { type: "thinking_delta", thinking: "Plan" }, + }, + { + type: "content_block_delta", + index: 0, + delta: { type: "signature_delta", signature: "SIG" }, + }, + { type: "content_block_stop", index: 0 }, + { + type: "content_block_start", + index: 1, + content_block: { type: "tool_use", id: "t1", name: "look", input: {} }, + }, + { + type: "content_block_delta", + index: 1, + delta: { type: "input_json_delta", partial_json: '{"a":' }, + }, + { + type: "content_block_delta", + index: 1, + delta: { type: "input_json_delta", partial_json: "1}" }, + }, + { type: "content_block_stop", index: 1 }, +]; + +const signedBlocks = [ + { type: "thinking", thinking: "Plan", signature: "SIG" }, + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, +]; + +const turnMessages = (provider: string, toolUseId: string) => [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "", provider, payload: JSON.stringify(signedBlocks) }, + { type: "tool_use", id: toolUseId, name: "look", input: { a: 1 } }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: toolUseId, content: [{ type: "text", text: "ok" }] }, + ], + }, +]; + +const bindingError = () => + Object.assign( + new Error( + "messages.1.content.0: Invalid `signature` in `thinking` block. The block is bound to a different conversation." + ), + { status: 400 } + ); + +describe("Anthropic native turn", () => { + let calls: Array>; + let script: Scripted[]; + + const install = () => { + const fakeClient = { + messages: { + stream: (params: Record) => { + calls.push(structuredClone(params)); + const next = script.shift() ?? []; + if (next instanceof Error) throw next; + return { + async *[Symbol.asyncIterator]() { + for (const event of next) { + if (event instanceof Error) throw event; + yield event; + } + }, + }; + }, + }, + }; + setAnthropicClientForTests(fakeClient); + runtimeTestOnly.setAnthropicClientForTests?.(fakeClient); + }; + + beforeEach(() => { + calls = []; + script = []; + vi.spyOn(getLogger(), "warn").mockImplementation(() => {}); + install(); + }); + + afterEach(() => { + setAnthropicClientForTests(undefined); + runtimeTestOnly.setAnthropicClientForTests?.(undefined); + vi.restoreAllMocks(); + }); + + async function run(messages: unknown[]): Promise>> { + const events: Array> = []; + await toolCallingRunFn()( + { model, prompt: "", messages, tools: [] } as never, + model, + undefined as never, + ((event: Record) => events.push(event)) as never + ); + return events; + } + + const nativeTurns = (events: Array>) => + events.filter((e) => e.type === "object-delta" && e.port === "nativeTurn"); + + const sentAssistant = (call: Record) => + (call.messages as Array<{ role: string; content: any[] }>).find((m) => m.role === "assistant")!; + + it("captures the reply's blocks in order, signature included", async () => { + script = [thinkingThenCall]; + const events = await run([{ role: "user", content: [{ type: "text", text: "go" }] }]); + const native = nativeTurns(events); + expect(native).toHaveLength(1); + expect(native[0]!.objectDelta).toEqual({ + provider: NATIVE_KEY, + payload: JSON.stringify(signedBlocks), + }); + expect(events.indexOf(native[0]!)).toBeLessThan(events.findIndex((e) => e.type === "finish")); + }); + + it("streams the thinking on the reasoning port as well as keeping it for replay", async () => { + script = [thinkingThenCall]; + const events = await run([{ role: "user", content: [{ type: "text", text: "go" }] }]); + expect(events).toContainEqual({ type: "text-delta", port: "reasoning", textDelta: "Plan" }); + }); + + it("rebuilds rather than replays a stored tool_use block with no input", async () => { + script = [[]]; + const blocks = [{ type: "tool_use", id: "t1", name: "look" }]; + await run([ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: NATIVE_KEY, payload: JSON.stringify(blocks) }, + { type: "tool_use", id: "t1", name: "look", input: {} }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "t1", content: [{ type: "text", text: "ok" }] }, + ], + }, + ]); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: {} }, + ]); + }); + + it("rebuilds rather than replays a stored turn holding an empty text block", async () => { + script = [[]]; + const blocks = [{ type: "text", text: "" }, ...signedBlocks]; + await run([ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: NATIVE_KEY, payload: JSON.stringify(blocks) }, + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "t1", content: [{ type: "text", text: "ok" }] }, + ], + }, + ]); + expect(sentAssistant(calls[0]!).content).not.toContainEqual({ type: "text", text: "" }); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + }); + + it.each([ + ["a null entry", [null]], + ["a block of a type it never writes", [{ type: "image", source: {} }]], + ["thinking without its signature", [{ type: "thinking", thinking: "hm" }]], + ["redacted thinking without its data", [{ type: "redacted_thinking" }]], + ["text that is not a string", [{ type: "text", text: 7 }]], + ])("rebuilds rather than replays a stored turn holding %s", async (_label, extra) => { + script = [[]]; + await run([ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { + type: "reasoning", + text: "", + provider: NATIVE_KEY, + payload: JSON.stringify([...(extra as unknown[]), ...signedBlocks]), + }, + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "t1", content: [{ type: "text", text: "ok" }] }, + ], + }, + ]); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + }); + + it("captures redacted thinking as it arrived", async () => { + script = [ + [ + { + type: "content_block_start", + index: 0, + content_block: { type: "redacted_thinking", data: "R" }, + }, + { type: "content_block_stop", index: 0 }, + ], + ]; + const events = await run([{ role: "user", content: [{ type: "text", text: "go" }] }]); + expect(nativeTurns(events)[0]!.objectDelta.payload).toBe( + JSON.stringify([{ type: "redacted_thinking", data: "R" }]) + ); + }); + + const textBlock = (index: number, text: string) => [ + { type: "content_block_start", index, content_block: { type: "text", text: "" } }, + { type: "content_block_delta", index, delta: { type: "text_delta", text } }, + { type: "content_block_stop", index }, + ]; + + const thinkingBlock = (index: number, signature: string | undefined) => [ + { type: "content_block_start", index, content_block: { type: "thinking", thinking: "" } }, + { type: "content_block_delta", index, delta: { type: "thinking_delta", thinking: "Plan" } }, + ...(signature === undefined + ? [] + : [{ type: "content_block_delta", index, delta: { type: "signature_delta", signature } }]), + { type: "content_block_stop", index }, + ]; + + const toolBlock = (index: number) => [ + { + type: "content_block_start", + index, + content_block: { type: "tool_use", id: "t1", name: "look", input: {} }, + }, + { + type: "content_block_delta", + index, + delta: { type: "input_json_delta", partial_json: '{"a":1}' }, + }, + { type: "content_block_stop", index }, + ]; + + const lookTool = [ + { name: "look", description: "", inputSchema: { type: "object", properties: {} } }, + ]; + + /** Runs one reply and asserts it carries no native turn but its text and call came through. */ + async function expectNoNativeTurn( + reply: readonly unknown[], + expectedText: string | undefined + ): Promise { + script = [reply]; + const events: Array> = []; + await toolCallingRunFn()( + { model, prompt: "", messages: [], tools: lookTool } as never, + model, + undefined as never, + ((event: Record) => events.push(event)) as never + ); + expect(nativeTurns(events)).toHaveLength(0); + expect(events.some((e) => e.type === "finish")).toBe(true); + const text = events + .filter((e) => e.type === "text-delta" && e.port === "text") + .map((e) => e.textDelta) + .join(""); + expect(text).toBe(expectedText ?? ""); + const lastCalls = events.filter((e) => e.type === "object-delta" && e.port === "toolCalls"); + expect(lastCalls.at(-1)!.objectDelta).toEqual([{ id: "t1", name: "look", input: { a: 1 } }]); + } + + it("emits no native turn when a block of an unknown type arrived", async () => { + await expectNoNativeTurn( + [ + ...thinkingBlock(0, "SIG"), + { + type: "content_block_start", + index: 1, + content_block: { type: "server_tool_use", id: "s1", name: "web_search", input: {} }, + }, + { type: "content_block_stop", index: 1 }, + ...textBlock(2, "hello"), + ...toolBlock(3), + ], + "hello" + ); + }); + + it("emits no native turn when a delta of an unknown type arrived", async () => { + await expectNoNativeTurn( + [ + ...thinkingBlock(0, "SIG"), + { type: "content_block_start", index: 1, content_block: { type: "text", text: "" } }, + { + type: "content_block_delta", + index: 1, + delta: { type: "citations_delta", citation: { type: "char_location" } }, + }, + { type: "content_block_delta", index: 1, delta: { type: "text_delta", text: "hello" } }, + { type: "content_block_stop", index: 1 }, + ...toolBlock(2), + ], + "hello" + ); + }); + + it("emits no native turn when a thinking block never received its signature", async () => { + await expectNoNativeTurn([...thinkingBlock(0, undefined), ...toolBlock(1)], undefined); + }); + + it("emits no native turn when an empty text block precedes thinking", async () => { + await expectNoNativeTurn( + [ + { type: "content_block_start", index: 0, content_block: { type: "text", text: "" } }, + { type: "content_block_stop", index: 0 }, + ...thinkingBlock(1, "SIG"), + ...toolBlock(2), + ], + undefined + ); + }); + + it("still emits the native turn when the empty text block follows the last thinking", async () => { + script = [ + [ + ...thinkingBlock(0, "SIG"), + { type: "content_block_start", index: 1, content_block: { type: "text", text: "" } }, + { type: "content_block_stop", index: 1 }, + ...toolBlock(2), + ], + ]; + const events = await run([{ role: "user", content: [{ type: "text", text: "go" }] }]); + expect(nativeTurns(events)[0]!.objectDelta.payload).toBe(JSON.stringify(signedBlocks)); + }); + + it("replays the captured turn verbatim while the tool calls still match", async () => { + await run(turnMessages(NATIVE_KEY, "t1")); + expect(sentAssistant(calls[0]!).content).toEqual(signedBlocks); + }); + + it("rebuilds the turn without thinking when a tool call id changed", async () => { + await run(turnMessages(NATIVE_KEY, "t1-2")); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1-2", name: "look", input: { a: 1 } }, + ]); + }); + + it("rebuilds the turn without thinking when a call kept its id but changed its input", async () => { + const edited = turnMessages(NATIVE_KEY, "t1").map((message) => + message.role === "assistant" + ? { + ...message, + content: message.content.map((block) => + block.type === "tool_use" ? { ...block, input: { a: 2 } } : block + ), + } + : message + ); + await run(edited as never); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 2 } }, + ]); + }); + + it("rebuilds the turn without thinking when the host added text the model never wrote", async () => { + const edited = turnMessages(NATIVE_KEY, "t1").map((message) => + message.role === "assistant" + ? { ...message, content: [...message.content, { type: "text", text: "edited" }] } + : message + ); + await run(edited as never); + expect(sentAssistant(calls[0]!).content).not.toContainEqual(signedBlocks[0]); + }); + + it("rebuilds the turn without thinking when another model produced it", async () => { + await run(turnMessages("anthropic:some-other-model", "t1")); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + }); + + it("strips thinking and retries once when Anthropic rejects the replayed signature", async () => { + script = [bindingError(), thinkingThenCall]; + const events = await run(turnMessages(NATIVE_KEY, "t1")); + expect(calls).toHaveLength(2); + expect(sentAssistant(calls[0]!).content).toEqual(signedBlocks); + expect(sentAssistant(calls[1]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + expect(events.some((e) => e.type === "finish")).toBe(true); + expect(getLogger().warn).toHaveBeenCalledWith( + "Anthropic rejected replayed thinking; retrying without it.", + { model: MODEL_NAME } + ); + }); + + it("strips redacted thinking on the retry too", async () => { + script = [bindingError(), []]; + const messages = turnMessages(NATIVE_KEY, "t1"); + (messages[1]!.content as unknown[])[0] = { + type: "reasoning", + text: "", + provider: NATIVE_KEY, + payload: JSON.stringify([ + { type: "redacted_thinking", data: "R" }, + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]), + }; + await run(messages); + expect(sentAssistant(calls[1]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + }); + + it("does not loop: a second rejection is rethrown", async () => { + script = [bindingError(), bindingError()]; + await expect(run(turnMessages(NATIVE_KEY, "t1"))).rejects.toThrow(/Invalid `signature`/); + expect(calls).toHaveLength(2); + }); + + it("does not retry an unrelated 400", async () => { + script = [Object.assign(new Error("max_tokens too large"), { status: 400 })]; + await expect(run(turnMessages(NATIVE_KEY, "t1"))).rejects.toThrow(/max_tokens/); + expect(calls).toHaveLength(1); + }); + + it("rethrows a binding 400 when the request carried no thinking, after one call", async () => { + script = [bindingError()]; + await expect(run([{ role: "user", content: [{ type: "text", text: "go" }] }])).rejects.toThrow( + /Invalid `signature`/ + ); + expect(calls).toHaveLength(1); + }); + + it("does not retry after output reached the consumer", async () => { + script = [[...textBlock(0, "partial"), bindingError()]]; + const events: Array> = []; + await expect( + toolCallingRunFn()( + { model, prompt: "", messages: turnMessages(NATIVE_KEY, "t1"), tools: [] } as never, + model, + undefined as never, + ((event: Record) => events.push(event)) as never + ) + ).rejects.toThrow(/Invalid `signature`/); + expect(calls).toHaveLength(1); + expect(events.filter((e) => e.type === "text-delta")).toHaveLength(1); + expect(events.find((e) => e.type === "text-delta")!.textDelta).toBe("partial"); + }); +}); diff --git a/packages/test/src/test/ai-provider/Anthropic_ReasoningAndToolImages.test.ts b/packages/test/src/test/ai-provider/Anthropic_ReasoningAndToolImages.test.ts new file mode 100644 index 0000000000..ead4394a25 --- /dev/null +++ b/packages/test/src/test/ai-provider/Anthropic_ReasoningAndToolImages.test.ts @@ -0,0 +1,152 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn } from "@workglow/ai"; +import { ANTHROPIC, _testOnly } from "@workglow/anthropic/ai"; +import { _testOnly as runtimeTestOnly } from "@workglow/anthropic/ai-runtime"; +import { getLogger } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const { ANTHROPIC_RUN_FNS, setAnthropicClientForTests } = _testOnly; + +function toolCallingRunFn(): AiProviderRunFn { + const registration = ANTHROPIC_RUN_FNS.find(({ serves }) => + (serves as readonly string[]).includes("tool-use") + ); + expect(registration).toBeDefined(); + return registration!.runFn as AiProviderRunFn; +} + +const model = { + model_id: "claude-sonnet-5-5", + title: "", + description: "", + provider: ANTHROPIC, + provider_config: { model_name: "claude-sonnet-5-5", api_key: "test-key" }, + capabilities: ["text.generation", "tool-use"], + metadata: {}, +} as never; + +/** + * A conversation another provider produced: its reasoning was streamed as text + * and kept on the assistant turn, and a tool returned an image. + */ +const messages = [ + { role: "user", content: [{ type: "text", text: "What is on the board?" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "I should look at the image." }, + { type: "tool_use", id: "c1", name: "read", input: { path: "board.png" } }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "Image: board.png" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + is_error: undefined, + }, + ], + }, +]; + +describe("Anthropic tool-calling request with reasoning and tool images", () => { + let captured: Record[]; + + beforeEach(() => { + captured = []; + vi.spyOn(getLogger(), "warn").mockImplementation(() => {}); + const fakeClient = { + messages: { + stream: (params: Record) => { + captured.push(params); + return { + async *[Symbol.asyncIterator]() { + // No events: the run-fn finishes on an empty stream. + }, + }; + }, + }, + }; + setAnthropicClientForTests(fakeClient); + runtimeTestOnly.setAnthropicClientForTests?.(fakeClient); + }); + + afterEach(() => { + setAnthropicClientForTests(undefined); + runtimeTestOnly.setAnthropicClientForTests?.(undefined); + vi.restoreAllMocks(); + }); + + it("drops reasoning blocks, which have no unsigned Anthropic form, and keeps tool images", async () => { + await toolCallingRunFn()( + { model, prompt: "", messages, tools: [] } as never, + model, + undefined as never, + (() => {}) as never + ); + expect(captured).toHaveLength(1); + const sent = captured[0]!.messages as Array<{ role: string; content: unknown[] }>; + const assistant = sent.find((message) => message.role === "assistant")!; + expect(assistant.content).toEqual([ + { type: "tool_use", id: "c1", name: "read", input: { path: "board.png" } }, + ]); + expect(JSON.stringify(sent)).not.toContain('"reasoning"'); + const toolResult = sent.at(-1)!.content[0] as { content: unknown[] }; + expect(toolResult.content[1]).toEqual({ + type: "image", + source: { type: "base64", media_type: "image/png", data: "AAAA" }, + }); + }); + + it("sends no empty text block inside a tool result", async () => { + const results = [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + { type: "tool_result", tool_use_id: "c2", content: [{ type: "text", text: "" }] }, + ]; + const history = [ + messages[0], + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "read", input: { path: "board.png" } }, + { type: "tool_use", id: "c2", name: "touch", input: { path: "x" } }, + ], + }, + { role: "tool", content: results }, + ]; + await toolCallingRunFn()( + { model, prompt: "", messages: history, tools: [] } as never, + model, + undefined as never, + (() => {}) as never + ); + const sent = captured[0]!.messages as Array<{ role: string; content: unknown[] }>; + expect(sent.at(-1)!.content).toEqual([ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "image", source: { type: "base64", media_type: "image/png", data: "AAAA" } }, + ], + }, + { type: "tool_result", tool_use_id: "c2" }, + ]); + }); +}); diff --git a/packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts b/packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts index 50c780fa57..8847706f12 100644 --- a/packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts +++ b/packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts @@ -217,6 +217,121 @@ describe("OpenAI checkpoint prompt_cache_key parity", () => { expect(consumerRequest.tool_choice).toBeUndefined(); }); + it.each([ + ["a reasoning model replays", "gpt-5", true], + ["a model without reasoning rebuilds", "gpt-4.1", false], + ])( + "writes a native turn and a tool image the same way in warm-up and tool calling: %s it", + async (_label, modelName, replays) => { + const model = { provider_config: { model_name: modelName } } as never; + const nativeItems = [ + { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "ENC" }, + { type: "function_call", call_id: "c1", name: "a", arguments: "{}" }, + ]; + const replayPrefix = { + systemPrompt: "sys", + tools: prefix.tools, + messages: [ + { role: "user" as const, content: [{ type: "text" as const, text: "look" }] }, + { + role: "assistant" as const, + content: [ + { + type: "reasoning" as const, + text: "", + provider: `openai:${modelName}`, + payload: JSON.stringify(nativeItems), + }, + { type: "tool_use" as const, id: "c1", name: "a", input: {} }, + ], + }, + { + role: "tool" as const, + content: [ + { + type: "tool_result" as const, + tool_use_id: "c1", + content: [{ type: "image" as const, mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + ], + }; + const session = { sessionId: `ckpt-${modelName}`, prefix: replayPrefix } as AiSessionContext; + await findOpenAiRunFn("cache.checkpoint")( + { model: modelName } as CacheCheckpointTaskInput, + model, + new AbortController().signal, + () => {}, + undefined, + session + ); + await findOpenAiRunFn("tool-use")( + { model: modelName, prompt: "tail", tools: [] } as never, + model, + new AbortController().signal, + () => {}, + undefined, + session + ); + + expect(openAiRequests).toHaveLength(2); + const [warmUpRequest, consumerRequest] = openAiRequests; + const warmed = warmUpRequest.input as unknown[]; + const sent = consumerRequest.input as unknown[]; + expect(sent.slice(0, warmed.length)).toEqual(warmed); + expect(sent.length).toBeGreaterThan(warmed.length); + if (replays) expect(warmed).toContainEqual(nativeItems[0]); + else expect(warmed).not.toContainEqual(nativeItems[0]); + } + ); + + it("writes a tool image the same way in warm-up and text generation", async () => { + const model = { provider_config: { model_name: "gpt-4.1" } } as never; + const imagePrefix = { + systemPrompt: "sys", + messages: [ + { role: "user" as const, content: [{ type: "text" as const, text: "look" }] }, + { + role: "assistant" as const, + content: [{ type: "tool_use" as const, id: "c1", name: "a", input: {} }], + }, + { + role: "tool" as const, + content: [ + { + type: "tool_result" as const, + tool_use_id: "c1", + content: [{ type: "image" as const, mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + ], + }; + const session = { sessionId: "ckpt-text-image", prefix: imagePrefix } as AiSessionContext; + await findOpenAiRunFn("cache.checkpoint")( + { model: "gpt-4.1" } as CacheCheckpointTaskInput, + model, + new AbortController().signal, + () => {}, + undefined, + session + ); + await findOpenAiRunFn("text.generation")( + { model: "gpt-4.1", prompt: "tail" } as TextGenerationTaskInput, + model, + new AbortController().signal, + () => {}, + undefined, + session + ); + const [warmUpRequest, consumerRequest] = openAiRequests; + const warmed = warmUpRequest!.input as unknown[]; + const sent = consumerRequest!.input as unknown[]; + expect(sent.slice(0, warmed.length)).toEqual(warmed); + expect(JSON.stringify(warmed)).toContain("data:image/png;base64,AAAA"); + }); + it("an empty-string system prompt keeps the warmed instructions and cache key", async () => { const session: AiSessionContext = { sessionId: "ckpt-oai-2", prefix }; await runWarmUp(session); diff --git a/packages/test/src/test/ai-provider/OpenAIShapedResponses.test.ts b/packages/test/src/test/ai-provider/OpenAIShapedResponses.test.ts index da8339e45f..4e85d5b66b 100644 --- a/packages/test/src/test/ai-provider/OpenAIShapedResponses.test.ts +++ b/packages/test/src/test/ai-provider/OpenAIShapedResponses.test.ts @@ -29,6 +29,69 @@ async function collect(list: readonly unknown[]): Promise { } describe("accumulateOpenAIResponsesStream", () => { + it("hands back every output item in order when asked for the native turn", async () => { + const out: StreamEvent>[] = []; + const reasoning = { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "ENC" }; + const call = { + type: "function_call", + id: "fc_1", + call_id: "c1", + name: "look", + arguments: "{}", + }; + await accumulateOpenAIResponsesStream( + events([ + { type: "response.output_item.done", output_index: 1, item: call }, + { type: "response.output_item.done", output_index: 0, item: reasoning }, + { type: "response.completed", response: { usage: undefined } }, + ]), + (e) => out.push(e), + { nativeTurnProvider: "openai" } + ); + const native = out.find((e) => e.type === "object-delta" && e.port === "nativeTurn"); + expect(native).toEqual({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: "openai", payload: JSON.stringify([reasoning, call]) }, + }); + }); + + it("emits no native turn when a reasoning item came back without encrypted content", async () => { + for (const reasoning of [ + { type: "reasoning", id: "rs_1", summary: [] }, + { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "" }, + { type: "reasoning", id: "rs_1", summary: [], encrypted_content: null }, + ]) { + const out: StreamEvent>[] = []; + await accumulateOpenAIResponsesStream( + events([ + { type: "response.output_item.done", output_index: 0, item: reasoning }, + { + type: "response.output_item.done", + output_index: 1, + item: { + type: "function_call", + id: "fc_1", + call_id: "c1", + name: "look", + arguments: "{}", + }, + }, + ]), + (e) => out.push(e), + { nativeTurnProvider: "openai" } + ); + expect(out.some((e) => e.type === "object-delta" && e.port === "nativeTurn")).toBe(false); + } + }); + + it("emits no native turn unless asked", async () => { + const out = await collect([ + { type: "response.output_item.done", output_index: 0, item: { type: "reasoning", id: "r" } }, + ]); + expect(out.some((e) => e.type === "object-delta" && e.port === "nativeTurn")).toBe(false); + }); + it("maps output_text deltas to text-delta events on the text port", async () => { const out = await collect([ { type: "response.created" }, @@ -340,6 +403,84 @@ describe("mapResponsesToolChoice", () => { }); describe("buildResponsesInput", () => { + it("replays an assistant turn's native items verbatim instead of rebuilding it", () => { + const items = [ + { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "ENC" }, + { type: "function_call", id: "fc_1", call_id: "c1", name: "look", arguments: "{}" }, + ]; + const { input } = buildResponsesInput({ + messages: [ + { role: "user", content: "go" }, + { + role: "assistant", + content: null, + tool_calls: [{ id: "c1", type: "function", function: { name: "look", arguments: "{}" } }], + native_items: items, + }, + { role: "tool", tool_call_id: "c1", content: "looked" }, + ] as never, + }); + expect(input).toEqual([ + { role: "user", content: "go" }, + ...items, + { type: "function_call_output", call_id: "c1", output: "looked" }, + ]); + }); + + it("rebuilds the turn when its calls no longer match the native items", () => { + const { input } = buildResponsesInput({ + messages: [ + { + role: "assistant", + content: null, + tool_calls: [ + { id: "c1-2", type: "function", function: { name: "look", arguments: "{}" } }, + ], + native_items: [{ type: "function_call", call_id: "c1", name: "look", arguments: "{}" }], + }, + ] as never, + }); + expect(input).toEqual([ + { type: "function_call", call_id: "c1-2", name: "look", arguments: "{}" }, + ]); + }); + + it("replays despite the API's spacing, and rebuilds when a call's arguments changed", () => { + const turn = (args: string) => + buildResponsesInput({ + messages: [ + { + role: "assistant", + content: null, + tool_calls: [ + { id: "c1", type: "function", function: { name: "look", arguments: args } }, + ], + native_items: [ + { type: "function_call", call_id: "c1", name: "look", arguments: '{"a": 1}' }, + ], + }, + ] as never, + }).input; + expect(turn('{"a":1}')).toEqual([ + { type: "function_call", call_id: "c1", name: "look", arguments: '{"a": 1}' }, + ]); + expect(turn('{"a":2}')).toEqual([ + { type: "function_call", call_id: "c1", name: "look", arguments: '{"a":2}' }, + ]); + }); + + it("rebuilds the turn when its text no longer matches the native items", () => { + const items = [ + { type: "message", role: "assistant", content: [{ type: "output_text", text: "Hi" }] }, + ]; + const turn = (content: string) => + buildResponsesInput({ + messages: [{ role: "assistant", content, native_items: items }] as never, + }).input; + expect(turn("Hi")).toEqual(items); + expect(turn("Edited")).toEqual([{ role: "assistant", content: "Edited" }]); + }); + it("uses the prompt string as input when there are no messages", () => { expect(buildResponsesInput({ prompt: "hello" })).toEqual({ input: "hello", diff --git a/packages/test/src/test/ai-provider/OpenAI_NativeTurn.test.ts b/packages/test/src/test/ai-provider/OpenAI_NativeTurn.test.ts new file mode 100644 index 0000000000..87d9cd7496 --- /dev/null +++ b/packages/test/src/test/ai-provider/OpenAI_NativeTurn.test.ts @@ -0,0 +1,250 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn } from "@workglow/ai"; +import { OPENAI, _testOnly } from "@workglow/openai/ai"; +import { _testOnly as runtimeTestOnly } from "@workglow/openai/ai-runtime"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const { OPENAI_RUN_FNS, setOpenAIClientForTests } = _testOnly; + +function toolCallingRunFn(): AiProviderRunFn { + const registration = OPENAI_RUN_FNS.find(({ serves }) => + (serves as readonly string[]).includes("tool-use") + ); + expect(registration).toBeDefined(); + return registration!.runFn as AiProviderRunFn; +} + +const modelWith = (providerConfig: Record) => + ({ + model_id: "gpt-5.5", + title: "gpt-5.5", + description: "", + provider: OPENAI, + provider_config: { + model_name: MODEL_NAME, + api_key: "test-key", + reasoning: { effort: "high" }, + ...providerConfig, + }, + capabilities: ["text.generation", "tool-use"], + metadata: {}, + }) as never; + +const reasoningItem = { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "ENC" }; +const callItem = { + type: "function_call", + id: "fc_1", + call_id: "c1", + name: "look", + arguments: "{}", +}; + +const MODEL_NAME = "gpt-5.5"; + +const messagesFrom = (provider: string) => [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { + type: "reasoning", + text: "", + provider, + payload: JSON.stringify([reasoningItem, callItem]), + }, + { type: "tool_use", id: "c1", name: "look", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "text", text: "looked" }], + is_error: undefined, + }, + ], + }, +]; + +const messages = messagesFrom(`openai:${MODEL_NAME}`); + +describe("OpenAI tool-calling native turn", () => { + let captured: Record[]; + let streamEvents: unknown[]; + + beforeEach(() => { + captured = []; + streamEvents = []; + const fakeClient = { + responses: { + create: async (params: Record) => { + captured.push(params); + return { + async *[Symbol.asyncIterator]() { + for (const event of streamEvents) yield event; + }, + }; + }, + }, + }; + setOpenAIClientForTests(fakeClient); + runtimeTestOnly.setOpenAIClientForTests(fakeClient); + }); + + afterEach(() => { + setOpenAIClientForTests(undefined); + runtimeTestOnly.setOpenAIClientForTests(undefined); + }); + + const run = async ( + model: never, + emitted: unknown[] = [], + history: unknown[] = messages + ): Promise> => { + await toolCallingRunFn()( + { + model, + prompt: "", + messages: history, + tools: [{ name: "look", description: "", inputSchema: {} }], + } as never, + model, + undefined as never, + ((event: unknown) => emitted.push(event)) as never + ); + expect(captured).toHaveLength(1); + return captured[0]!; + }; + + it("asks for the encrypted reasoning when the model reasons", async () => { + const params = await run(modelWith({})); + expect(params.include).toEqual(["reasoning.encrypted_content"]); + }); + + it("asks for nothing extra when replay is switched off", async () => { + const params = await run(modelWith({ replay_reasoning: false })); + expect(params).not.toHaveProperty("include"); + expect(JSON.stringify(params.input)).not.toContain("ENC"); + }); + + it("asks for nothing extra when reasoning is off", async () => { + const params = await run(modelWith({ reasoning: { effort: "none" } })); + expect(params).not.toHaveProperty("include"); + }); + + it("rebuilds an earlier native turn when reasoning is off", async () => { + const params = await run(modelWith({ reasoning: { effort: "none" } })); + expect(params.input).toEqual([ + { role: "user", content: [{ type: "input_text", text: "go" }] }, + { type: "function_call", call_id: "c1", name: "look", arguments: "{}" }, + { type: "function_call_output", call_id: "c1", output: "looked" }, + ]); + expect(JSON.stringify(params.input)).not.toContain("ENC"); + }); + + it("replays the native items it produced earlier", async () => { + const params = await run(modelWith({})); + expect(params.input).toEqual([ + { role: "user", content: [{ type: "input_text", text: "go" }] }, + reasoningItem, + callItem, + { type: "function_call_output", call_id: "c1", output: "looked" }, + ]); + }); + + it("sends a tool result's images on a user turn after the results", async () => { + const withImage = messages.map((message) => + message.role === "tool" + ? { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "looked" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + is_error: undefined, + }, + ], + } + : message + ); + const params = await run(modelWith({}), [], withImage); + const input = params.input as Record[]; + expect(input.at(-2)).toMatchObject({ type: "function_call_output", call_id: "c1" }); + expect(input.at(-1)).toEqual({ + role: "user", + content: [ + { type: "input_text", text: "Images returned by the tool calls above:" }, + { type: "input_text", text: "[image 1]" }, + { type: "input_image", image_url: "data:image/png;base64,AAAA" }, + ], + }); + }); + + it("rebuilds a turn another model produced instead of replaying it", async () => { + const params = await run(modelWith({}), [], messagesFrom("openai:some-other-model")); + expect(params.input).toEqual([ + { role: "user", content: [{ type: "input_text", text: "go" }] }, + { type: "function_call", call_id: "c1", name: "look", arguments: "{}" }, + { type: "function_call_output", call_id: "c1", output: "looked" }, + ]); + expect(JSON.stringify(params.input)).not.toContain("ENC"); + }); + + it("emits the turn's items as the native turn", async () => { + streamEvents = [ + { type: "response.output_item.done", output_index: 0, item: reasoningItem }, + { type: "response.output_item.done", output_index: 1, item: callItem }, + { type: "response.completed", response: {} }, + ]; + const emitted: Array<{ type: string; port?: string }> = []; + await run(modelWith({}), emitted); + expect(emitted.find((e) => e.port === "nativeTurn")).toEqual({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { + provider: `openai:${MODEL_NAME}`, + payload: JSON.stringify([reasoningItem, callItem]), + }, + }); + }); + + it("emits no native turn when replay is switched off", async () => { + streamEvents = [{ type: "response.output_item.done", output_index: 0, item: reasoningItem }]; + const emitted: Array<{ type: string; port?: string }> = []; + await run(modelWith({ replay_reasoning: false }), emitted); + expect(emitted.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("emits no native turn when the model takes no reasoning", async () => { + streamEvents = [ + { type: "response.output_item.done", output_index: 0, item: reasoningItem }, + { type: "response.output_item.done", output_index: 1, item: callItem }, + ]; + const emitted: Array<{ type: string; port?: string }> = []; + const params = await run(modelWith({ model_name: "gpt-4o", reasoning: undefined }), emitted); + expect(params).not.toHaveProperty("include"); + expect(emitted.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("emits no native turn when reasoning effort is none", async () => { + streamEvents = [ + { type: "response.output_item.done", output_index: 0, item: reasoningItem }, + { type: "response.output_item.done", output_index: 1, item: callItem }, + ]; + const emitted: Array<{ type: string; port?: string }> = []; + const params = await run(modelWith({ reasoning: { effort: "none" } }), emitted); + expect(params).not.toHaveProperty("include"); + expect(emitted.some((e) => e.port === "nativeTurn")).toBe(false); + }); +}); diff --git a/providers/anthropic/src/ai/common/Anthropic_CacheCheckpoint.ts b/providers/anthropic/src/ai/common/Anthropic_CacheCheckpoint.ts index 28fe1d36ea..3f3c51ff5a 100644 --- a/providers/anthropic/src/ai/common/Anthropic_CacheCheckpoint.ts +++ b/providers/anthropic/src/ai/common/Anthropic_CacheCheckpoint.ts @@ -90,7 +90,7 @@ export function buildAnthropicCheckpointParams( // entries, so a non-empty prefix.messages can still convert to []. const messages = prefix.messages && prefix.messages.length > 0 - ? buildAnthropicMessages(prefix.messages, "") + ? buildAnthropicMessages(prefix.messages, "", `anthropic:${modelName}`) : []; if (messages.length > 0) { annotateLastBlock(messages[messages.length - 1] as { content: unknown }); @@ -138,9 +138,12 @@ export function applyAnthropicPrefixReplay( // Guard on the CONVERTED array: buildAnthropicMessages skips system-role // entries, so a non-empty prefix.messages can still convert to []. + // The prefix's own turns replay with their signed thinking, keyed as tool + // calling keys them, so a checkpointed turn reaches the model as it wrote it. + const nativeKey = typeof params.model === "string" ? `anthropic:${params.model}` : undefined; const prefixMessages = prefix.messages && prefix.messages.length > 0 - ? buildAnthropicMessages(prefix.messages, "") + ? buildAnthropicMessages(prefix.messages, "", nativeKey) : []; if (prefixMessages.length > 0) { annotateLastBlock(prefixMessages[prefixMessages.length - 1] as { content: unknown }); diff --git a/providers/anthropic/src/ai/common/Anthropic_TextGeneration.ts b/providers/anthropic/src/ai/common/Anthropic_TextGeneration.ts index 2613b8e5a0..5b71abec19 100644 --- a/providers/anthropic/src/ai/common/Anthropic_TextGeneration.ts +++ b/providers/anthropic/src/ai/common/Anthropic_TextGeneration.ts @@ -65,7 +65,8 @@ export const Anthropic_TextGeneration_Stream: AiProviderRunFn< const messages = hasMessages ? buildAnthropicMessages( unified.messages as Parameters[0], - unified.prompt ?? "" + unified.prompt ?? "", + undefined ) : [{ role: "user" as const, content: input.prompt }]; diff --git a/providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts b/providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts index 0c6f67c23e..45fa431936 100644 --- a/providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts +++ b/providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts @@ -43,9 +43,91 @@ function isEmptyTextBlock(block: { type?: unknown; text?: unknown }): boolean { return block.type === "text" && (block.text === undefined || block.text === ""); } +/** + * Whether a stored block is one of the kinds this provider captures, with the + * fields Anthropic requires of it. An empty text block is refused outright, and + * one cannot be dropped from a signed turn, so a payload holding one (never + * emitted here) is rebuilt instead; so is a tool_use without an input object. + */ +function isReplayableBlock(block: unknown): boolean { + if (block === null || typeof block !== "object" || Array.isArray(block)) return false; + const b = block as Record; + switch (b.type) { + case "text": + return typeof b.text === "string" && !isEmptyTextBlock(b); + case "thinking": + return ( + typeof b.thinking === "string" && typeof b.signature === "string" && b.signature !== "" + ); + case "redacted_thinking": + return typeof b.data === "string" && b.data !== ""; + case "tool_use": + return ( + typeof b.id === "string" && + typeof b.name === "string" && + b.input !== null && + typeof b.input === "object" && + !Array.isArray(b.input) + ); + default: + return false; + } +} + +/** + * The turn a model produced, as the blocks the API returned in the order it + * returned them — or `undefined` when the payload is unusable or the host + * changed the turn's tool calls (an id, a name or an input) or its text, in + * which case it is rebuilt from the message. + */ +function parseOwnTurn(payload: string, content: ChatMessage["content"]): any[] | undefined { + let blocks: unknown; + try { + blocks = JSON.parse(payload); + } catch { + return undefined; + } + // Every block must be one this provider writes, carrying what Anthropic + // requires of it; anything else would be a 400 rather than a rebuilt turn. + if (!Array.isArray(blocks) || !blocks.every(isReplayableBlock)) return undefined; + // The payload's calls must be the message's calls exactly — id, name and + // input — or the replay would send a call the host has since changed. + const sent = blocks.filter((b) => b?.type === "tool_use") as Array<{ + id?: unknown; + name?: unknown; + input?: unknown; + }>; + const calls = content.filter((b) => b.type === "tool_use"); + if (sent.length !== calls.length) return undefined; + const same = sent.every( + (block, i) => + block.id === calls[i]!.id && + block.name === calls[i]!.name && + JSON.stringify(block.input) === JSON.stringify(calls[i]!.input ?? {}) + ); + if (!same) return undefined; + // And its text must be the message's text: the turn's text arrives as its + // text blocks run together, so an edited reply rebuilds rather than replays. + const sentText = blocks + .filter((b) => b?.type === "text" && typeof b.text === "string") + .map((b) => (b as { text: string }).text) + .join(""); + const messageText = content + .filter((b) => b.type === "text") + .map((b) => (b as { text: string }).text) + .join(""); + if (sentText !== messageText) return undefined; + return blocks.length > 0 ? blocks : undefined; +} + +/** + * `nativeTurnProvider` is the key a reasoning block must carry to be replayed; + * without it every assistant turn is rebuilt from its text and tool calls. + */ export function buildAnthropicMessages( messages: ReadonlyArray | undefined, - prompt: unknown + prompt: unknown, + nativeTurnProvider: string | undefined ): any[] { if (!messages || messages.length === 0) { return [{ role: "user", content: prompt }]; @@ -54,6 +136,7 @@ export function buildAnthropicMessages( for (const msg of messages) { if (msg.role === "user") { const blocks = msg.content + .filter((b) => b.type !== "reasoning") .map((b) => { if (b.type === "text") return { type: "text", text: b.text }; if (b.type === "image") { @@ -68,7 +151,28 @@ export function buildAnthropicMessages( if (blocks.length === 0) continue; out.push({ role: "user", content: blocks }); } else if (msg.role === "assistant") { + // The model's own turn, replayed exactly as it produced it: a thinking + // block's signature binds it to the conversation before it, so its text, + // the tool inputs and the block order are sent unchanged — and only while + // the calls still match, since a host that renamed one changed the turn. + const own = + nativeTurnProvider === undefined + ? undefined + : msg.content.find( + (b): b is Extract => + b.type === "reasoning" && + b.provider === nativeTurnProvider && + b.payload !== undefined + ); + const replayed = own ? parseOwnTurn(own.payload!, msg.content) : undefined; + if (replayed) { + out.push({ role: "assistant", content: replayed }); + continue; + } + // Reasoning another provider streamed as text has no Anthropic form (its + // thinking blocks are signed), and an unknown block is a 400. const blocks = msg.content + .filter((b) => b.type !== "reasoning") .map((b) => { if (b.type === "text") return { type: "text", text: b.text }; if (b.type === "tool_use") { @@ -86,20 +190,24 @@ export function buildAnthropicMessages( b.type === "tool_result" ) .map((b) => { - const content = b.content.map((inner) => { - if (inner.type === "text") return { type: "text", text: inner.text }; - if (inner.type === "image") { - return { - type: "image", - source: { type: "base64", media_type: inner.mimeType, data: inner.data }, - }; - } - return inner; - }); + const content = b.content + .map((inner) => { + if (inner.type === "text") return { type: "text", text: inner.text }; + if (inner.type === "image") { + return { + type: "image", + source: { type: "base64", media_type: inner.mimeType, data: inner.data }, + }; + } + return inner; + }) + .filter((inner) => !isEmptyTextBlock(inner)); + // A tool that returned nothing still answers its call: `content` is + // optional on a tool_result, where an empty text block is a 400. return { type: "tool_result", tool_use_id: b.tool_use_id, - content, + ...(content.length > 0 ? { content } : {}), ...(b.is_error ? { is_error: true } : {}), }; }); @@ -127,6 +235,84 @@ type AnthropicBlockMeta = } | { readonly type: "text" }; +/** A content block as the API produced it, accumulated for replay. */ +type NativeBlock = + | { type: "thinking"; thinking: string; signature: string } + | { type: "redacted_thinking"; data: string } + | { type: "text"; text: string } + | { type: "tool_use"; id: string; name: string; raw: string }; + +const THINKING_BLOCK_TYPES: ReadonlySet = new Set(["thinking", "redacted_thinking"]); + +/** Anthropic's 400 for a replayed thinking block whose signature no longer matches. */ +function isThinkingSignatureError(err: unknown): boolean { + const e = err as { status?: unknown; message?: unknown } | undefined; + return ( + e?.status === 400 && + typeof e.message === "string" && + e.message.includes("Invalid `signature` in `thinking` block") + ); +} + +/** + * The messages with every thinking block removed (a message left empty is + * dropped), or `undefined` when there was none to remove — a request without + * thinking cannot have been rejected for it, so there is nothing to retry. + */ +function withoutThinking(messages: readonly any[]): any[] | undefined { + const out: any[] = []; + let removed = false; + for (const msg of messages) { + if (msg?.role !== "assistant" || !Array.isArray(msg.content)) { + out.push(msg); + continue; + } + const content = msg.content.filter((b: any) => !THINKING_BLOCK_TYPES.has(b?.type)); + if (content.length !== msg.content.length) removed = true; + if (content.length > 0) out.push({ ...msg, content }); + } + return removed ? out : undefined; +} + +/** + * The reply's blocks in index order as the API produced them, or `undefined` + * when there is nothing to replay or the turn cannot be replayed verbatim: a + * tool input that cannot be parsed from its raw JSON, a thinking block that + * never received its signature, or an empty text block that must be dropped + * ahead of a thinking block (the signature binds the blocks before it). The + * replay must carry exactly what the model wrote, so a guess is worse than + * rebuilding the turn. + */ +function serializeNativeTurn(native: ReadonlyMap): string | undefined { + const ordered = [...native.entries()].sort((a, b) => a[0] - b[0]).map(([, block]) => block); + let lastThinking = -1; + ordered.forEach((block, i) => { + if (THINKING_BLOCK_TYPES.has(block.type)) lastThinking = i; + }); + const blocks: unknown[] = []; + for (const [i, block] of ordered.entries()) { + if (block.type === "tool_use") { + let input: unknown; + try { + input = block.raw ? JSON.parse(block.raw) : {}; + } catch { + return undefined; + } + blocks.push({ type: "tool_use", id: block.id, name: block.name, input }); + } else if (block.type === "text") { + // The API rejects an empty text block, so one is dropped — which is only + // faithful when no thinking block follows it. + if (block.text !== "") blocks.push(block); + else if (i < lastThinking) return undefined; + } else if (block.type === "thinking" && block.signature === "") { + return undefined; + } else { + blocks.push(block); + } + } + return blocks.length > 0 ? JSON.stringify(blocks) : undefined; +} + function mapAnthropicToolChoice( toolChoice: string | undefined ): { type: "auto" } | { type: "any" } | { type: "tool"; name: string } | undefined { @@ -165,7 +351,10 @@ export const Anthropic_ToolCalling_Stream: AiProviderRunFn< toolChoice = { type: "auto" }; } - const messages = buildAnthropicMessages(input.messages, input.prompt); + // Thinking blocks are bound to the model that produced them, so the native + // turn is keyed by model and another model gets the rebuilt turn. + const nativeKey = `anthropic:${modelName}`; + const messages = buildAnthropicMessages(input.messages, input.prompt, nativeKey); const params: any = { model: modelName, @@ -216,90 +405,160 @@ export const Anthropic_ToolCalling_Stream: AiProviderRunFn< } } - const stream = client.messages.stream(params, { signal }); - - const blockMeta = new Map(); - /** Keyed by Anthropic content block index — avoids collisions when `id` is missing during early deltas. */ - const toolCallsByBlockIndex = new Map(); - - const toolCallsInStreamOrder = (): ToolCall[] => - [...toolCallsByBlockIndex.entries()].sort((a, b) => a[0] - b[0]).map(([, tc]) => tc); + /** Whether anything has reached the consumer, since a retry after output would repeat it. */ + let emitted = false; + const emitTracked: typeof emit = (event) => { + emitted = true; + emit(event); + }; /** - * Defence-in-depth: drop tool calls whose name isn't in the effective - * declared tool set (the caller's tools, or the prefix's on fallback) - * before yielding to the consumer. Anthropic's tool API normally respects - * the declarations, but a stray response from a model that hallucinates a - * function name would otherwise propagate to dispatch. + * One request and the consumption of its stream. All streaming state lives + * inside, so a retry starts clean. */ - const validatedToolCallsInStreamOrder = (): ToolCall[] => - filterValidToolCalls(toolCallsInStreamOrder(), toolDefinitions); - - const usageCollector = createAnthropicUsageCollector(); - const snapshotUsage = createUsageSnapshotEmitter(emit); - for await (const event of stream) { - usageCollector.observe(event); - snapshotUsage(usageCollector.result()); - maybeEmitAnthropicRefusal(event, emit); - if (event.type === "content_block_start") { - const block = event.content_block; - const index = event.index as number; - if (block.type === "tool_use") { - blockMeta.set(index, { - type: "tool_use", - id: block.id ?? "", - name: block.name ?? "", - json: createPartialJsonStream(), - }); - } else if (block.type === "text") { - blockMeta.set(index, { type: "text" }); - } - } else if (event.type === "content_block_delta") { - const index = event.index as number; - const delta = event.delta as any; - if (delta.type === "text_delta") { - emit({ type: "text-delta", port: "text", textDelta: delta.text }); - } else if (delta.type === "input_json_delta") { + const consume = async (requestParams: any): Promise => { + const stream = client.messages.stream(requestParams, { signal }); + + const blockMeta = new Map(); + const native = new Map(); + /** Keyed by Anthropic content block index — avoids collisions when `id` is missing during early deltas. */ + const toolCallsByBlockIndex = new Map(); + + const toolCallsInStreamOrder = (): ToolCall[] => + [...toolCallsByBlockIndex.entries()].sort((a, b) => a[0] - b[0]).map(([, tc]) => tc); + + /** + * Defence-in-depth: drop tool calls whose name isn't in the effective + * declared tool set (the caller's tools, or the prefix's on fallback) + * before yielding to the consumer. Anthropic's tool API normally respects + * the declarations, but a stray response from a model that hallucinates a + * function name would otherwise propagate to dispatch. + */ + const validatedToolCallsInStreamOrder = (): ToolCall[] => + filterValidToolCalls(toolCallsInStreamOrder(), toolDefinitions); + + /** Cleared by any block or delta the replay cannot reproduce exactly. */ + let verbatim = true; + + const usageCollector = createAnthropicUsageCollector(); + const snapshotUsage = createUsageSnapshotEmitter(emitTracked); + for await (const event of stream) { + usageCollector.observe(event); + snapshotUsage(usageCollector.result()); + maybeEmitAnthropicRefusal(event, emitTracked); + if (event.type === "content_block_start") { + const block = event.content_block; + const index = event.index as number; + if (block.type === "tool_use") { + blockMeta.set(index, { + type: "tool_use", + id: block.id ?? "", + name: block.name ?? "", + json: createPartialJsonStream(), + }); + native.set(index, { + type: "tool_use", + id: block.id ?? "", + name: block.name ?? "", + raw: "", + }); + } else if (block.type === "text") { + blockMeta.set(index, { type: "text" }); + native.set(index, { type: "text", text: "" }); + } else if (block.type === "thinking") { + native.set(index, { + type: "thinking", + thinking: block.thinking ?? "", + signature: block.signature ?? "", + }); + } else if (block.type === "redacted_thinking") { + native.set(index, { type: "redacted_thinking", data: block.data ?? "" }); + } else { + verbatim = false; + } + } else if (event.type === "content_block_delta") { + const index = event.index as number; + const delta = event.delta as any; + const nativeBlock = native.get(index); + if (delta.type === "text_delta") { + if (nativeBlock?.type === "text") nativeBlock.text += delta.text ?? ""; + emitTracked({ type: "text-delta", port: "text", textDelta: delta.text }); + } else if (delta.type === "thinking_delta") { + if (nativeBlock?.type === "thinking") nativeBlock.thinking += delta.thinking ?? ""; + if (delta.thinking) { + emitTracked({ type: "text-delta", port: "reasoning", textDelta: delta.thinking }); + } + } else if (delta.type === "signature_delta") { + if (nativeBlock?.type === "thinking") nativeBlock.signature += delta.signature ?? ""; + } else if (delta.type === "input_json_delta") { + if (nativeBlock?.type === "tool_use") nativeBlock.raw += delta.partial_json ?? ""; + const meta = blockMeta.get(index); + if (meta?.type === "tool_use") { + // `push` returns the parser's live root; `sanitizeToolArgs` copies it, + // so the emitted call is detached from later mutations. + const parsedInput = meta.json.push(delta.partial_json ?? "") ?? {}; + toolCallsByBlockIndex.set(index, { + id: meta.id, + name: meta.name, + input: sanitizeToolArgs(parsedInput) as Record, + }); + emitTracked({ + type: "object-delta", + port: "toolCalls", + objectDelta: validatedToolCallsInStreamOrder(), + }); + } + } else { + verbatim = false; + } + } else if (event.type === "content_block_stop") { + const index = event.index as number; const meta = blockMeta.get(index); if (meta?.type === "tool_use") { - // `push` returns the parser's live root; `sanitizeToolArgs` copies it, - // so the emitted call is detached from later mutations. - const parsedInput = meta.json.push(delta.partial_json ?? "") ?? {}; + const finalInput = meta.json.finishObject(); toolCallsByBlockIndex.set(index, { id: meta.id, name: meta.name, - input: sanitizeToolArgs(parsedInput) as Record, + input: sanitizeToolArgs(finalInput) as Record, }); - emit({ + emitTracked({ type: "object-delta", port: "toolCalls", objectDelta: validatedToolCallsInStreamOrder(), }); } + blockMeta.delete(index); } - } else if (event.type === "content_block_stop") { - const index = event.index as number; - const meta = blockMeta.get(index); - if (meta?.type === "tool_use") { - const finalInput = meta.json.finishObject(); - toolCallsByBlockIndex.set(index, { - id: meta.id, - name: meta.name, - input: sanitizeToolArgs(finalInput) as Record, - }); - emit({ - type: "object-delta", - port: "toolCalls", - objectDelta: validatedToolCallsInStreamOrder(), - }); - } - blockMeta.delete(index); } - } - emit({ - type: "finish", - data: { text: "", toolCalls: [] } as ToolCallingTaskOutput, - usage: usageCollector.result(), - }); + const nativeTurn = verbatim ? serializeNativeTurn(native) : undefined; + if (nativeTurn !== undefined) { + emitTracked({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: nativeKey, payload: nativeTurn }, + }); + } + + emitTracked({ + type: "finish", + data: { text: "", toolCalls: [] } as ToolCallingTaskOutput, + usage: usageCollector.result(), + }); + }; + + try { + await consume(params); + } catch (err) { + // A replayed thinking block whose signature no longer matches is a 400 + // decided before any output; the documented recovery is to send the turn + // without its thinking, once. + if (emitted || !isThinkingSignatureError(err) || !Array.isArray(params.messages)) throw err; + const stripped = withoutThinking(params.messages); + if (stripped === undefined) throw err; + getLogger().warn("Anthropic rejected replayed thinking; retrying without it.", { + model: modelName, + }); + await consume({ ...params, messages: stripped }); + } }; diff --git a/providers/openai/src/ai/common/OpenAI_CacheCheckpoint.ts b/providers/openai/src/ai/common/OpenAI_CacheCheckpoint.ts index ae65639721..d3967ecdc0 100644 --- a/providers/openai/src/ai/common/OpenAI_CacheCheckpoint.ts +++ b/providers/openai/src/ai/common/OpenAI_CacheCheckpoint.ts @@ -21,6 +21,34 @@ import { promptToTailMessages, toOpenAIMessages } from "@workglow/ai/worker"; import { finalizeResponsesRequest, getClient, getModelName } from "./OpenAI_Client"; import type { OpenAiModelConfig } from "./OpenAI_ModelSchema"; +/** + * How a conversation is turned into chat messages for a finalized Responses + * request. Warming a checkpoint and sending a turn over it must agree, or the + * request diverges from the warmed prefix at the first turn they write + * differently and the prompt cache stops matching there. + * + * A native turn is replayed only into a request that asks for encrypted + * reasoning back: with reasoning off its items would be looked up by id and + * fail where nothing is stored, so the turn is rebuilt instead. It is keyed by + * model, because encrypted reasoning is readable only by the model that wrote it. + */ +export function openAIConversationOptions( + model: OpenAiModelConfig | undefined, + params: Record +): { readonly toolImagesInUserMessage: true; readonly nativeTurnProvider?: string | undefined } { + const replay = + (model?.provider_config as { replay_reasoning?: boolean } | undefined)?.replay_reasoning !== + false; + const reasoning = params.reasoning as { effort?: string } | undefined; + const wantsEncryptedReasoning = replay && reasoning !== undefined && reasoning.effort !== "none"; + return { + // A function_call_output carries text only, so images a tool returned go on + // a user turn after the results. + toolImagesInUserMessage: true, + ...(wantsEncryptedReasoning ? { nativeTurnProvider: `openai:${getModelName(model)}` } : {}), + }; +} + /** * Merges a checkpoint prefix into the unified generation input: the prefix's * messages come first, the caller's tail follows (its `messages`, or its @@ -89,13 +117,15 @@ export const OpenAI_CacheCheckpoint_Stream: AiProviderRunFn< // The "." prompt is a throwaway user turn used only when the prefix has no // messages of its own (toOpenAIMessages ignores it otherwise); the cached // value is the system/tools region ahead of it. - const messages = toOpenAIMessages({ + const conversation = { messages: prefix.messages ?? [], systemPrompt: prefix.systemPrompt, prompt: ".", tools: [], - } as never); - const { input: responsesInput, instructions } = buildResponsesInput({ messages }); + } as never; + const { input: responsesInput, instructions } = buildResponsesInput({ + messages: toOpenAIMessages(conversation, { toolImagesInUserMessage: true }), + }); const params: Record = { model: getModelName(model), @@ -107,6 +137,11 @@ export const OpenAI_CacheCheckpoint_Stream: AiProviderRunFn< params.tools = buildResponsesTools(prefix.tools); } finalizeResponsesRequest(model, params); + // Written the way a turn over this checkpoint writes it, now that the + // request's reasoning is settled. + params.input = buildResponsesInput({ + messages: toOpenAIMessages(conversation, openAIConversationOptions(model, params)), + }).input; const response = await (client.responses.create as (p: unknown, o: unknown) => Promise)( params, diff --git a/providers/openai/src/ai/common/OpenAI_ModelSchema.ts b/providers/openai/src/ai/common/OpenAI_ModelSchema.ts index 3bde7e83cf..38a2ab020e 100644 --- a/providers/openai/src/ai/common/OpenAI_ModelSchema.ts +++ b/providers/openai/src/ai/common/OpenAI_ModelSchema.ts @@ -77,6 +77,12 @@ export const OpenAiModelSchema = { "Override whether the model accepts a pinned temperature with reasoning effort 'none'. Absent means 'decide from the model id'. A wrong `true` is an HTTP 400.", "x-ui-hidden": true, }, + replay_reasoning: { + type: "boolean", + description: + "Hand the model its own reasoning back on later rounds (default true); false is the ablation switch.", + "x-ui-hidden": true, + }, }, required: ["model_name"], additionalProperties: false, diff --git a/providers/openai/src/ai/common/OpenAI_TextGeneration.ts b/providers/openai/src/ai/common/OpenAI_TextGeneration.ts index 641334fb1d..df99be3e43 100644 --- a/providers/openai/src/ai/common/OpenAI_TextGeneration.ts +++ b/providers/openai/src/ai/common/OpenAI_TextGeneration.ts @@ -49,12 +49,17 @@ function buildResponsesParams( ): Record { const hasMessages = Array.isArray(input.messages) && input.messages.length > 0; const messages = hasMessages - ? toOpenAIMessages({ - messages: input.messages, - systemPrompt: input.systemPrompt, - prompt: "", - tools: [], - } as never) + ? toOpenAIMessages( + { + messages: input.messages, + systemPrompt: input.systemPrompt, + prompt: "", + tools: [], + } as never, + // A function_call_output carries text only: images a tool returned go on a + // user turn after the results, as the checkpoint warm-up writes them. + { toolImagesInUserMessage: true } + ) : undefined; const { input: responsesInput, instructions } = buildResponsesInput({ diff --git a/providers/openai/src/ai/common/OpenAI_ToolCalling.ts b/providers/openai/src/ai/common/OpenAI_ToolCalling.ts index 37cc044e64..c05682448a 100644 --- a/providers/openai/src/ai/common/OpenAI_ToolCalling.ts +++ b/providers/openai/src/ai/common/OpenAI_ToolCalling.ts @@ -19,7 +19,7 @@ import { promptTextForResponsesUsageEstimate, } from "@workglow/ai/provider-utils"; import { filterValidToolCalls, toOpenAIMessages } from "@workglow/ai/worker"; -import { mergeOpenAICheckpointPrefix } from "./OpenAI_CacheCheckpoint"; +import { mergeOpenAICheckpointPrefix, openAIConversationOptions } from "./OpenAI_CacheCheckpoint"; import { finalizeResponsesRequest, getClient, getModelName } from "./OpenAI_Client"; import type { OpenAiModelConfig } from "./OpenAI_ModelSchema"; @@ -27,7 +27,10 @@ import type { OpenAiModelConfig } from "./OpenAI_ModelSchema"; * Streaming run-fn for `["text.generation", "tool-use"]`. Calls the OpenAI * Responses endpoint with `stream: true` and forwards delta events via * {@link accumulateOpenAIResponsesStream}, which emits `text-delta` and tool-call - * `object-delta` events plus a final empty `finish`. + * `object-delta` events plus a final empty `finish`. The turn's own output + * items go back on the `nativeTurn` port only when the request asked for + * encrypted reasoning, since a replayed reasoning item without it is looked up + * by id and fails where the response is not stored. * * Defence-in-depth: each tool-call `object-delta` is filtered against the * effective tool declarations (the caller's tools, or the checkpoint prefix's @@ -49,17 +52,16 @@ export const OpenAI_ToolCalling_Stream: AiProviderRunFn< const merged = mergeOpenAICheckpointPrefix(sessionContext, input); const toolDefinitions = merged?.tools ?? input.tools; const tools = buildResponsesTools(toolDefinitions); + const conversation = merged + ? ({ + ...input, + messages: merged.messages, + systemPrompt: merged.systemPrompt, + prompt: "", + } as ToolCallingTaskInput) + : input; const { input: responsesInput, instructions } = buildResponsesInput({ - messages: toOpenAIMessages( - merged - ? ({ - ...input, - messages: merged.messages, - systemPrompt: merged.systemPrompt, - prompt: "", - } as ToolCallingTaskInput) - : input - ), + messages: toOpenAIMessages(conversation, { toolImagesInUserMessage: true }), }); const toolChoice = mapResponsesToolChoice(input.toolChoice); @@ -73,6 +75,18 @@ export const OpenAI_ToolCalling_Stream: AiProviderRunFn< if (input.maxTokens !== undefined) params.max_output_tokens = input.maxTokens; if (input.temperature !== undefined) params.temperature = input.temperature; finalizeResponsesRequest(model, params); + // A reasoning model starts its thinking over every round unless it is handed + // its encrypted reasoning back; the items come back on `nativeTurn`. The + // input is written the way the checkpoint warm-up writes it, so a prefix + // still matches. + const conversationOptions = openAIConversationOptions(model, params); + const nativeKey = conversationOptions.nativeTurnProvider; + if (nativeKey !== undefined) { + params.include = ["reasoning.encrypted_content"]; + params.input = buildResponsesInput({ + messages: toOpenAIMessages(conversation, conversationOptions), + }).input; + } const promptText = promptTextForResponsesUsageEstimate(params); createEstimatedOutputUsageReporter(emit).onPrompt(promptText); @@ -94,7 +108,7 @@ export const OpenAI_ToolCalling_Stream: AiProviderRunFn< } emit(event); }, - { promptText } + { promptText, nativeTurnProvider: nativeKey } ); emit({ type: "finish", data: { text: "", toolCalls: [] } as ToolCallingTaskOutput, usage }); }; From 692115ce158db8c905d2728ac365ce58909d3896 Mon Sep 17 00:00:00 2001 From: Steven Roussey Date: Mon, 12 Oct 2026 05:34:45 +0000 Subject: [PATCH 3/7] providers: carry reasoning and tool-result images between rounds on cloud APIs - DeepSeek sends its reasoning effort and replays reasoning_content. - OpenRouter replays reasoning_details beside the same tool-call ids, never thinking truncated before its signature, and never a turn with a call it filtered out. - xAI, OpenRouter and HF Inference send tool-result images in a user message; HF Inference gains an effort setting. - Gemini sends tool results as objects with errors under error, emits thought summaries as reasoning on Gemini 2.5 and later, and puts tool-result images in functionResponse.parts on Gemini 3+, or in a labelled turn of their own on older models, which reject that field. --- .../ai/src/provider-utils/OpenAIShapedChat.ts | 11 + .../ai-provider-api/DeepSeekProvider.test.ts | 32 ++ .../GoogleGeminiProvider.test.ts | 226 ++++++++- .../test/ai-provider-api/HFI_Effort.test.ts | 81 +++ .../OpenAICompatToolImages.test.ts | 151 ++++++ .../OpenRouter_ReasoningDetails.test.ts | 466 ++++++++++++++++++ .../ai-provider/Gemini_ThinkingBudget.test.ts | 67 +++ .../OpenAIShapedChatToolArgs.test.ts | 17 + .../deepseek/src/ai/common/DeepSeek_Client.ts | 28 ++ .../src/ai/common/DeepSeek_ModelSchema.ts | 6 + .../common/DeepSeek_StructuredGeneration.ts | 2 + .../src/ai/common/DeepSeek_TextGeneration.ts | 2 + .../src/ai/common/DeepSeek_TextRewriter.ts | 3 +- .../src/ai/common/DeepSeek_TextSummary.ts | 3 +- .../src/ai/common/DeepSeek_ToolCalling.ts | 10 +- providers/deepseek/src/ai/index.ts | 1 + .../src/ai/common/Gemini_CacheCheckpoint.ts | 6 +- .../src/ai/common/Gemini_Client.ts | 47 +- .../src/ai/common/Gemini_ToolCalling.ts | 85 +++- providers/google-gemini/src/ai/index.ts | 7 +- .../src/ai/HfInferenceProvider.ts | 7 +- .../src/ai/HfInferenceQueuedProvider.ts | 7 +- .../src/ai/common/HFI_EffortPolicy.ts | 51 ++ .../src/ai/common/HFI_TextGeneration.ts | 2 + .../src/ai/common/HFI_ToolCalling.ts | 4 +- .../huggingface-inference/src/ai/index.ts | 1 + .../huggingface-inference/src/ai/runtime.ts | 1 + .../ai/common/OpenRouter_ReasoningDetails.ts | 188 +++++++ .../src/ai/common/OpenRouter_ToolCalling.ts | 92 +++- .../xai/src/ai/common/Xai_ToolCalling.ts | 2 +- 30 files changed, 1574 insertions(+), 32 deletions(-) create mode 100644 packages/test/src/test/ai-provider-api/HFI_Effort.test.ts create mode 100644 packages/test/src/test/ai-provider-api/OpenAICompatToolImages.test.ts create mode 100644 packages/test/src/test/ai-provider-api/OpenRouter_ReasoningDetails.test.ts create mode 100644 providers/huggingface-inference/src/ai/common/HFI_EffortPolicy.ts create mode 100644 providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails.ts diff --git a/packages/ai/src/provider-utils/OpenAIShapedChat.ts b/packages/ai/src/provider-utils/OpenAIShapedChat.ts index f938968c3c..f5745bb919 100644 --- a/packages/ai/src/provider-utils/OpenAIShapedChat.ts +++ b/packages/ai/src/provider-utils/OpenAIShapedChat.ts @@ -182,6 +182,17 @@ export async function accumulateOpenAIChatStream( const choice = chunk?.choices?.[0]; if (!choice) continue; + // Thinking models on OpenAI-compatible APIs stream their reasoning beside + // the content: DeepSeek as `reasoning_content`, OpenRouter as `reasoning`. + // A chunk can carry both with one empty, so the first non-empty one wins. + const reasoningDelta = [choice.delta?.reasoning_content, choice.delta?.reasoning].find( + (value: unknown): value is string => typeof value === "string" && value.length > 0 + ); + if (reasoningDelta !== undefined) { + provisionalUsage?.onText(reasoningDelta); + emit({ type: "text-delta", port: "reasoning", textDelta: reasoningDelta }); + } + const contentDelta: string = choice.delta?.content ?? ""; if (contentDelta) { provisionalUsage?.onText(contentDelta); diff --git a/packages/test/src/test/ai-provider-api/DeepSeekProvider.test.ts b/packages/test/src/test/ai-provider-api/DeepSeekProvider.test.ts index ab68797b1a..b3c7ba55d8 100644 --- a/packages/test/src/test/ai-provider-api/DeepSeekProvider.test.ts +++ b/packages/test/src/test/ai-provider-api/DeepSeekProvider.test.ts @@ -11,6 +11,7 @@ import { DEEPSEEK_ALLOWED_HOSTS, DEEPSEEK_DEFAULT_BASE_URL, DEEPSEEK_DEFAULT_REASONING_ALLOWANCE, + reasoningParams, resolveMaxTokens, } from "@workglow/deepseek/ai"; import { describe, expect, it } from "vitest"; @@ -208,6 +209,37 @@ describe("resolveMaxTokens", () => { }); }); +describe("reasoningParams", () => { + const flash = (effort?: string) => + ({ + provider: "DEEPSEEK", + provider_config: { model_name: "deepseek-flash" }, + ...(effort ? { effort } : {}), + }) as never; + + it("sends the effort a caller set, in DeepSeek's names", () => { + expect(reasoningParams(flash("low"))).toEqual({ reasoning_effort: "low" }); + expect(reasoningParams(flash("high"))).toEqual({ reasoning_effort: "high" }); + expect(reasoningParams(flash("extra"))).toEqual({ reasoning_effort: "xhigh" }); + expect(reasoningParams(flash("ultra"))).toEqual({ reasoning_effort: "max" }); + }); + + it("turns thinking off for none, which DeepSeek spells as its own switch", () => { + expect(reasoningParams(flash("none"))).toEqual({ thinking: { type: "disabled" } }); + }); + + it("sends nothing when no effort is set, or the model does not reason", () => { + expect(reasoningParams(flash())).toEqual({}); + expect( + reasoningParams({ + provider: "DEEPSEEK", + provider_config: { model_name: "deepseek-chat" }, + effort: "high", + } as never) + ).toEqual({}); + }); +}); + describe("assertNotTruncatedByReasoning", () => { it("throws when the budget was exhausted before any content was emitted", () => { expect(() => assertNotTruncatedByReasoning("length", "", 4096)).toThrow( diff --git a/packages/test/src/test/ai-provider-api/GoogleGeminiProvider.test.ts b/packages/test/src/test/ai-provider-api/GoogleGeminiProvider.test.ts index a77112e242..c42a880323 100644 --- a/packages/test/src/test/ai-provider-api/GoogleGeminiProvider.test.ts +++ b/packages/test/src/test/ai-provider-api/GoogleGeminiProvider.test.ts @@ -6,7 +6,7 @@ import type { ChatMessage, ModelRecord } from "@workglow/ai"; import { GEMINI_FALLBACK_MODELS, _testOnly } from "@workglow/google-gemini/ai"; -import { describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it } from "vitest"; const { GoogleGeminiQueuedProvider, GEMINI_RUN_FN_SPECS, GEMINI_RUN_FNS } = _testOnly; @@ -296,3 +296,227 @@ describe("buildGeminiContents multi-turn", () => { expect("thoughtSignature" in fnPart).toBe(false); }); }); + +describe("buildGeminiContents tool results", () => { + const { buildGeminiContents } = _testOnly; + + const call = (id: string, name: string) => ({ + role: "assistant", + content: [{ type: "tool_use", id, name, input: {} }], + }); + const result = (id: string, content: any[], is_error?: boolean) => ({ + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: id, + content, + ...(is_error ? { is_error } : {}), + }, + ], + }); + + it.each([ + ["42", { result: 42 }], + ["null", { result: null }], + ["[1,2]", { result: [1, 2] }], + ['"s"', { result: "s" }], + ['{"a":1}', { a: 1 }], + ["plain words", { result: "plain words" }], + ])("wraps a %s tool result as an object", (text, response) => { + const contents = buildGeminiContents( + [call("c1", "t"), result("c1", [{ type: "text", text }])] as any, + "x" + ); + expect(contents.at(-1).parts[0].functionResponse.response).toEqual(response); + }); + + it("holds tool images until every separate tool message of the turn is out", () => { + const contents = buildGeminiContents( + [ + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + result("c1", [{ type: "image", mimeType: "image/png", data: "AAAA" }]), + result("c2", [{ type: "text", text: "ok" }]), + ] as any, + "x" + ); + const responses = contents.filter((c: any) => c.parts[0]?.functionResponse); + expect(responses).toHaveLength(2); + expect(contents.indexOf(responses[1])).toBeLessThan(contents.length - 1); + expect(contents.at(-1)).toEqual({ + role: "user", + parts: [ + { text: "Images returned by the tool calls above:" }, + { inlineData: { mimeType: "image/png", data: "AAAA" } }, + ], + }); + }); + + it("reports an error result under `error`", () => { + const contents = buildGeminiContents( + [call("c1", "t"), result("c1", [{ type: "text", text: "boom" }], true)] as any, + "x" + ); + expect(contents.at(-1).parts[0].functionResponse.response).toEqual({ error: "boom" }); + }); + + it("reports an is_error result whose text is JSON under `error`, unparsed", () => { + const contents = buildGeminiContents( + [call("c1", "t"), result("c1", [{ type: "text", text: '{"code":1}' }], true)] as any, + "x" + ); + expect(contents.at(-1).parts[0].functionResponse.response).toEqual({ + error: '{"code":1}', + }); + }); + + it("returns images in functionResponse.parts when the model takes them there", () => { + const contents = buildGeminiContents( + [ + call("c1", "t"), + result("c1", [ + { type: "text", text: "here" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ]), + ] as any, + "x", + { multimodalFunctionResponses: true } + ); + expect(contents.at(-1).parts[0].functionResponse).toEqual({ + name: "t", + response: { result: "here" }, + parts: [{ inlineData: { mimeType: "image/png", data: "AAAA" } }], + }); + }); + + it("sends images in a separate user turn after the function responses by default", () => { + const contents = buildGeminiContents( + [ + call("c1", "t"), + result("c1", [ + { type: "text", text: "here" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ]), + ] as any, + "x" + ); + expect(contents).toHaveLength(3); + expect(contents[1].parts).toEqual([ + { functionResponse: { name: "t", response: { result: "here" } } }, + ]); + expect(contents[2]).toEqual({ + role: "user", + parts: [ + { text: "Images returned by the tool calls above:" }, + { inlineData: { mimeType: "image/png", data: "AAAA" } }, + ], + }); + }); + + it("puts every image of a tool message in one labelled turn after all of its function responses", () => { + const toolMessage = { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + { + type: "tool_result", + tool_use_id: "c2", + content: [{ type: "text", text: "two" }], + }, + ], + }; + const contents = buildGeminiContents( + [call("c1", "a"), call("c2", "b"), toolMessage] as any, + "x" + ); + const [responses, images] = contents.slice(-2); + expect(responses.parts).toEqual([ + { functionResponse: { name: "a", response: { result: "" } } }, + { functionResponse: { name: "b", response: { result: "two" } } }, + ]); + expect(images.parts).toEqual([ + { text: "Images returned by the tool calls above:" }, + { inlineData: { mimeType: "image/png", data: "AAAA" } }, + ]); + }); +}); + +describe("geminiSupportsMultimodalFunctionResponses", () => { + const { geminiSupportsMultimodalFunctionResponses } = _testOnly; + + it.each(["gemini-3-flash-preview", "gemini-3.5-flash", "gemini-10-pro"])( + "accepts multimodal function responses for %s", + (name) => { + expect(geminiSupportsMultimodalFunctionResponses(name)).toBe(true); + } + ); + + it.each(["gemini-2.5-flash", "gemini-flash-latest", "gemini-2.0-flash"])( + "rejects multimodal function responses for %s", + (name) => { + expect(geminiSupportsMultimodalFunctionResponses(name)).toBe(false); + } + ); +}); + +describe("Gemini tool calling thought summaries", () => { + afterEach(() => { + _testOnly.setGeminiClientForTests(undefined); + }); + + const run = async (model_name: string, parts: readonly Record[]) => { + const requests: Array> = []; + _testOnly.setGeminiClientForTests({ + models: { + generateContentStream: async (request: Record) => { + requests.push(request); + return { + async *[Symbol.asyncIterator]() { + yield { candidates: [{ content: { parts } }] }; + }, + }; + }, + }, + } as never); + const registration = GEMINI_RUN_FNS.find(({ serves }) => + (serves as readonly string[]).includes("tool-use") + ); + const events: any[] = []; + await (registration!.runFn as any)( + { prompt: "hi", tools: [] }, + { provider: "GOOGLE_GEMINI", provider_config: { api_key: "k", model_name } }, + undefined, + (event: unknown) => events.push(event), + undefined + ); + return { requests, events }; + }; + + it("emits thought parts on reasoning and the rest on text", async () => { + const { requests, events } = await run("gemini-2.5-flash", [ + { text: "pondering", thought: true }, + { text: "answer" }, + ]); + const deltas = events.filter((e) => e.type === "text-delta"); + expect(deltas).toEqual([ + { type: "text-delta", port: "reasoning", textDelta: "pondering" }, + { type: "text-delta", port: "text", textDelta: "answer" }, + ]); + expect(requests[0].config.thinkingConfig.includeThoughts).toBe(true); + }); + + it("sends no thinkingConfig to a model that does not think", async () => { + const { requests } = await run("gemini-embedding-001", [{ text: "answer" }]); + expect(requests[0].config.thinkingConfig).toBeUndefined(); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/HFI_Effort.test.ts b/packages/test/src/test/ai-provider-api/HFI_Effort.test.ts new file mode 100644 index 0000000000..d11a26af5b --- /dev/null +++ b/packages/test/src/test/ai-provider-api/HFI_Effort.test.ts @@ -0,0 +1,81 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { _testOnly, hfiEffortPolicy, hfiReasoningParams } from "@workglow/huggingface-inference/ai"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const model = (effort?: string, model_name = "openai/gpt-oss-120b") => + ({ provider_config: { model_name }, ...(effort ? { effort } : {}) }) as any; + +describe("Hugging Face Inference effort", () => { + it("sends reasoning_effort for a set effort", () => { + expect(hfiReasoningParams(model("high"))).toEqual({ reasoning_effort: "high" }); + expect(hfiReasoningParams(model("low"))).toEqual({ reasoning_effort: "low" }); + }); + + it("maps extra and ultra to high and sends nothing for none or unset", () => { + expect(hfiReasoningParams(model("ultra"))).toEqual({ reasoning_effort: "high" }); + expect(hfiReasoningParams(model("extra"))).toEqual({ reasoning_effort: "high" }); + expect(hfiReasoningParams(model("none"))).toEqual({}); + expect(hfiReasoningParams(model())).toEqual({}); + }); + + it("gives embedding models no effort", () => { + expect(hfiEffortPolicy(model(undefined, "BAAI/bge-small-en-v1.5")).supported).toEqual([]); + expect(hfiReasoningParams(model("high", "BAAI/bge-small-en-v1.5"))).toEqual({}); + }); + + it("excludes whisper and tts ids by name, not by substring", () => { + expect(hfiEffortPolicy(model(undefined, "openai/whisper-large-v3")).supported).toEqual([]); + expect(hfiEffortPolicy(model(undefined, "hexgrad/tts-1")).supported).toEqual([]); + expect(hfiEffortPolicy(model(undefined, "org/matts-chat")).supported).toContain("high"); + expect(hfiReasoningParams(model("high", "org/matts-chat"))).toEqual({ + reasoning_effort: "high", + }); + }); + + it("is what the provider reports for a model", () => { + const provider = new _testOnly.HfInferenceQueuedProvider() as any; + expect(provider.effortPolicy(model("high"))?.supported).toContain("high"); + expect(provider.effortPolicy(model(undefined, "BAAI/bge-small-en-v1.5"))?.supported).toEqual( + [] + ); + }); +}); + +describe("Hugging Face Inference text generation effort", () => { + let sent: Record[] = []; + class CapturingClient { + chatCompletionStream(params: Record) { + sent.push(params); + return (async function* () { + yield { choices: [{ delta: { content: "ok" } }] }; + })(); + } + } + + beforeEach(() => { + sent = []; + _testOnly._setHfInferenceSDKForTesting({ InferenceClient: CapturingClient } as any); + }); + + afterEach(() => { + _testOnly._setHfInferenceSDKForTesting(undefined); + }); + + it("sends the model's effort on a plain text-generation request", async () => { + const runFn = _testOnly.HFI_RUN_FNS.find( + (r) => [...r.serves].sort().join(",") === "text.generation" + )!.runFn; + const withKey = (effort?: string) => ({ + ...model(effort), + provider_config: { model_name: "openai/gpt-oss-120b", api_key: "k" }, + }); + await runFn({ prompt: "hi" } as any, withKey("high"), new AbortController().signal, () => {}); + await runFn({ prompt: "hi" } as any, withKey(), new AbortController().signal, () => {}); + expect(sent.map((params) => params.reasoning_effort)).toEqual(["high", undefined]); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/OpenAICompatToolImages.test.ts b/packages/test/src/test/ai-provider-api/OpenAICompatToolImages.test.ts new file mode 100644 index 0000000000..2588cc6c2f --- /dev/null +++ b/packages/test/src/test/ai-provider-api/OpenAICompatToolImages.test.ts @@ -0,0 +1,151 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { _testOnly as hfiTestOnly } from "@workglow/huggingface-inference/ai"; +import { _testOnly as openRouterTestOnly } from "@workglow/openrouter/ai"; +import { _testOnly as xaiTestOnly } from "@workglow/xai/ai"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +function findRunFn( + fns: readonly { readonly serves: readonly string[]; readonly runFn: any }[], + label: string +) { + const reg = fns.find((r) => [...r.serves].sort().join(",") === "text.generation,tool-use"); + if (!reg) throw new Error(`no ${label} tool-calling run-fn registered`); + return reg.runFn; +} + +const input = { + prompt: "look", + tools: [{ name: "shot", description: "", inputSchema: { type: "object", properties: {} } }], + messages: [ + { role: "user", content: [{ type: "text", text: "look" }] }, + { role: "assistant", content: [{ type: "tool_use", id: "c1", name: "shot", input: {} }] }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "here" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + ], + }, + ], +} as any; + +function expectImageInFollowingUserMessage(messages: any[]): void { + const toolIndex = messages.findIndex((m) => m.role === "tool"); + expect(toolIndex).toBeGreaterThan(-1); + const toolMessage = messages[toolIndex]; + expect(typeof toolMessage.content).toBe("string"); + expect(toolMessage.content).toContain("[image 1] follows in the next message"); + + const next = messages[toolIndex + 1]; + expect(next.role).toBe("user"); + expect(next.content).toContainEqual({ + type: "image_url", + image_url: { url: "data:image/png;base64,AAAA" }, + }); +} + +function sseEmptyResponse(): Response { + const encoder = new TextEncoder(); + const stream = new ReadableStream({ + start(controller) { + controller.enqueue(encoder.encode(`data: [DONE]\n\n`)); + controller.close(); + }, + }); + return new Response(stream, { + status: 200, + headers: { "content-type": "text/event-stream" }, + }); +} + +describe("tool-result images reach the model in a user message", () => { + describe("over the openai SDK (xAI, OpenRouter)", () => { + let fetchSpy: ReturnType; + + beforeEach(() => { + fetchSpy = vi.spyOn(globalThis, "fetch"); + fetchSpy.mockImplementation(async () => sseEmptyResponse()); + }); + + afterEach(() => { + fetchSpy.mockRestore(); + }); + + function sentMessages(): any[] { + const init = fetchSpy.mock.calls[0]![1] as RequestInit; + return JSON.parse(init.body as string).messages; + } + + it("xAI", async () => { + const runFn = findRunFn(xaiTestOnly.XAI_RUN_FNS, "xAI"); + await runFn( + input, + { + model_id: "grok-x", + provider_config: { model_name: "grok-4", api_key: "test-key" }, + } as any, + new AbortController().signal, + () => {} + ); + expectImageInFollowingUserMessage(sentMessages()); + }); + + it("OpenRouter", async () => { + const runFn = findRunFn(openRouterTestOnly.OPENROUTER_RUN_FNS, "OpenRouter"); + await runFn( + input, + { + model_id: "openrouter-x", + provider_config: { model_name: "anthropic/claude-sonnet-4", api_key: "test-key" }, + } as any, + new AbortController().signal, + () => {} + ); + expectImageInFollowingUserMessage(sentMessages()); + }); + }); + + describe("Hugging Face Inference", () => { + let captured: any; + + beforeEach(() => { + captured = undefined; + class FakeInferenceClient { + chatCompletionStream(params: unknown) { + captured = params; + return (async function* () {})(); + } + } + hfiTestOnly._setHfInferenceSDKForTesting({ InferenceClient: FakeInferenceClient } as any); + }); + + afterEach(() => { + hfiTestOnly._setHfInferenceSDKForTesting(undefined); + }); + + it("sends the image in a user message", async () => { + const runFn = findRunFn(hfiTestOnly.HFI_RUN_FNS, "Hugging Face Inference"); + await runFn( + input, + { + model_id: "hfi-x", + provider_config: { model_name: "meta-llama/Llama-3.3-70B-Instruct", api_key: "test-key" }, + } as any, + new AbortController().signal, + () => {} + ); + expectImageInFollowingUserMessage(captured.messages); + }); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/OpenRouter_ReasoningDetails.test.ts b/packages/test/src/test/ai-provider-api/OpenRouter_ReasoningDetails.test.ts new file mode 100644 index 0000000000..99f8612c3e --- /dev/null +++ b/packages/test/src/test/ai-provider-api/OpenRouter_ReasoningDetails.test.ts @@ -0,0 +1,466 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { _testOnly as openRouterTestOnly } from "@workglow/openrouter/ai"; +import { + createReasoningDetailAccumulator, + encodeReasoningTurn, + mergeReasoningDetailDelta, + withReasoningDetails, +} from "../../../../../providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const MODEL = "anthropic/claude-sonnet-5-5"; +const KEY = `openrouter:${MODEL}`; + +describe("reasoning detail accumulator", () => { + it("joins pieces by index and sorts the result", () => { + const acc = createReasoningDetailAccumulator(); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.text", index: 0, text: "Let me " }]); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.text", index: 0, text: "think." }]); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.text", index: 0, signature: "sig" }]); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.encrypted", index: 1, data: "xyz" }]); + expect(acc.items()).toEqual([ + { type: "reasoning.text", index: 0, text: "Let me think.", signature: "sig" }, + { type: "reasoning.encrypted", index: 1, data: "xyz" }, + ]); + }); + + it("is undefined when nothing arrived", () => { + expect(createReasoningDetailAccumulator().items()).toBeUndefined(); + }); + + it("appends a piece with no index after the last one", () => { + const acc = createReasoningDetailAccumulator(); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.summary", index: 0, summary: "a" }]); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.summary", summary: "b" }]); + expect(acc.items()).toEqual([ + { type: "reasoning.summary", index: 0, summary: "a" }, + { type: "reasoning.summary", summary: "b" }, + ]); + }); + + it("ignores a delta that is not an array of objects", () => { + const acc = createReasoningDetailAccumulator(); + mergeReasoningDetailDelta(acc, undefined); + mergeReasoningDetailDelta(acc, "x"); + mergeReasoningDetailDelta(acc, [null, 3]); + expect(acc.items()).toBeUndefined(); + }); +}); + +describe("withReasoningDetails", () => { + const detail = { type: "reasoning.text", text: "x", signature: "sig" }; + const written = (ids: string[]) => ids.map((id) => ({ id, name: "shot", arguments: "{}" })); + const turn = ( + callIds: string[], + payloadOf: unknown = { calls: written(callIds), details: [detail] } + ): any => ({ + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: KEY, payload: JSON.stringify(payloadOf) }, + ...callIds.map((id) => ({ type: "tool_use", id, name: "shot", input: {} })), + ], + }); + const converted = (ids: string[]): any => ({ + role: "assistant", + content: null, + ...(ids.length > 0 + ? { + tool_calls: ids.map((id) => ({ + id, + type: "function", + function: { name: "shot", arguments: "{}" }, + })), + } + : {}), + }); + + it("attaches the details to the turn whose call ids match, without mutating", () => { + const input = [{ role: "user", content: "hi" }, converted(["a", "b"])]; + const out = withReasoningDetails( + input, + [{ role: "user", content: [{ type: "text", text: "hi" }] }, turn(["a", "b"])], + KEY + ); + expect(out[0]).toEqual({ role: "user", content: "hi" }); + expect(out[1]).toMatchObject({ reasoning_details: [detail] }); + expect(input[1]).not.toHaveProperty("reasoning_details"); + }); + + it("matches a turn with no tool calls against an empty id list", () => { + const out = withReasoningDetails([converted([])], [turn([])], KEY); + expect(out[0]).toMatchObject({ reasoning_details: [detail] }); + }); + + it("does not replay when an id changed, was dropped or was added", () => { + const history = [turn(["a", "b"])]; + expect(withReasoningDetails([converted(["a", "z"])], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + expect(withReasoningDetails([converted(["a"])], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + expect(withReasoningDetails([converted(["a", "b", "c"])], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + expect(withReasoningDetails([converted(["b", "a"])], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + }); + + it("does not replay when a kept id now carries another name or arguments", () => { + const history = [turn(["a"])]; + const renamed = converted(["a"]); + renamed.tool_calls[0].function.name = "other"; + expect(withReasoningDetails([renamed], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + const edited = converted(["a"]); + edited.tool_calls[0].function.arguments = '{"x":1}'; + expect(withReasoningDetails([edited], history, KEY)[0]).not.toHaveProperty("reasoning_details"); + }); + + it("does not replay a payload that is not a turn, or another model's", () => { + expect(withReasoningDetails([converted([])], [turn([], [detail])], KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + expect( + withReasoningDetails([converted([])], [turn([], "not json{")], KEY)[0] + ).not.toHaveProperty("reasoning_details"); + expect( + withReasoningDetails([converted([])], [turn([])], "openrouter:other/model")[0] + ).not.toHaveProperty("reasoning_details"); + }); + + it("does not replay details that are not typed objects", () => { + for (const details of [[null], [7], [{ text: "untyped" }], [detail, null]]) { + expect( + withReasoningDetails([converted([])], [turn([], { calls: [], details })], KEY)[0] + ).not.toHaveProperty("reasoning_details"); + } + }); +}); + +describe("encodeReasoningTurn", () => { + it("carries the calls beside the details, arguments as the converter writes them", () => { + expect( + JSON.parse(encodeReasoningTurn([{ id: "c1", name: "shot", input: { a: 1 } }], details)!) + ).toEqual({ calls: [{ id: "c1", name: "shot", arguments: '{"a":1}' }], details }); + }); + + it("refuses an anthropic thinking block cut off before its signature", () => { + const cut = [{ type: "reasoning.text", format: "anthropic-claude-v1", text: "hm" }]; + expect(encodeReasoningTurn([], cut)).toBeUndefined(); + expect(encodeReasoningTurn([], [{ ...cut[0], signature: "" }])).toBeUndefined(); + expect(encodeReasoningTurn([], [{ ...cut[0], signature: "sig" }])).toBeDefined(); + }); + + it("keeps a signature-less text detail from another format", () => { + const plain = [{ type: "reasoning.text", format: "unknown", text: "hm" }]; + expect(encodeReasoningTurn([], plain)).toBeDefined(); + }); +}); + +function sseResponse(chunks: readonly unknown[]): Response { + const encoder = new TextEncoder(); + const body = new ReadableStream({ + start(controller) { + for (const chunk of chunks) { + controller.enqueue(encoder.encode(`data: ${JSON.stringify(chunk)}\n\n`)); + } + controller.enqueue(encoder.encode(`data: [DONE]\n\n`)); + controller.close(); + }, + }); + return new Response(body, { status: 200, headers: { "content-type": "text/event-stream" } }); +} + +function chunk(delta: Record, finish: string | null = null): unknown { + return { + id: "c", + object: "chat.completion.chunk", + choices: [{ index: 0, delta, finish_reason: finish }], + }; +} + +const details = [ + { type: "reasoning.text", index: 0, text: "Let me think.", signature: "sig" }, + { type: "reasoning.encrypted", index: 1, data: "xyz" }, +]; + +const tools = [{ name: "shot", description: "", inputSchema: { type: "object", properties: {} } }]; + +const modelConfig = { + model_id: "openrouter-x", + provider_config: { model_name: MODEL, api_key: "test-key" }, +} as any; + +function findRunFn() { + const reg = openRouterTestOnly.OPENROUTER_RUN_FNS.find( + (r: any) => + [...r.serves].sort((a: string, b: string) => a.localeCompare(b)).join(",") === + "text.generation,tool-use" + ); + if (!reg) throw new Error("no OpenRouter tool-calling run-fn registered"); + return reg.runFn as any; +} + +describe("OpenRouter tool calling replays reasoning_details", () => { + let fetchSpy: ReturnType; + + beforeEach(() => { + fetchSpy = vi.spyOn(globalThis, "fetch"); + }); + + afterEach(() => { + fetchSpy.mockRestore(); + }); + + function sentMessages(): any[] { + const init = fetchSpy.mock.calls[0]![1] as RequestInit; + return JSON.parse(init.body as string).messages; + } + + function replayInput(provider: string, callId = "c1", payloadCallId = callId): any { + return { + prompt: "look", + tools, + messages: [ + { role: "user", content: [{ type: "text", text: "look" }] }, + { + role: "assistant", + content: [ + { + type: "reasoning", + text: "", + provider, + payload: JSON.stringify({ + calls: [{ id: payloadCallId, name: "shot", arguments: "{}" }], + details, + }), + }, + { type: "tool_use", id: callId, name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: callId, content: [{ type: "text", text: "ok" }] }, + ], + }, + ], + }; + } + + it("forwards thinking carried only in reasoning_details to the reasoning port, once", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + // A null entry is malformed and skipped, not a reason to fail the run. + chunk({ + reasoning_details: [null, { type: "reasoning.text", index: 0, text: "only here" }], + }), + // An empty legacy field must not hide the details' text. + chunk({ + reasoning_content: "", + reasoning_details: [{ type: "reasoning.text", index: 0, text: ", and here" }], + }), + chunk({ + reasoning: "legacy", + reasoning_details: [{ type: "reasoning.text", index: 0, text: "legacy" }], + }), + // Nor an empty alias hide the other one's text, with or without details. + chunk({ + reasoning: " too", + reasoning_content: "", + reasoning_details: [{ type: "reasoning.text", index: 0, text: " too" }], + }), + chunk({ reasoning: "!", reasoning_content: "" }), + chunk({ content: "done" }), + chunk({}, "stop"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + const reasoning = events + .filter((e) => e.type === "text-delta" && e.port === "reasoning") + .map((e) => e.textDelta) + .join(""); + expect(reasoning).toBe("only here, and herelegacy too!"); + }); + + it("emits the streamed details as the native turn before finish", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + chunk({ reasoning_details: [{ type: "reasoning.text", index: 0, text: "Let me " }] }), + chunk({ + reasoning_details: [ + { type: "reasoning.text", index: 0, text: "think.", signature: "sig" }, + { type: "reasoning.encrypted", index: 1, data: "xyz" }, + ], + }), + chunk({ + tool_calls: [ + { index: 0, id: "c1", type: "function", function: { name: "shot", arguments: "{}" } }, + ], + }), + chunk({}, "tool_calls"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + const nativeIndex = events.findIndex( + (e) => e.type === "object-delta" && e.port === "nativeTurn" + ); + expect(nativeIndex).toBeGreaterThan(-1); + expect(events[nativeIndex]).toEqual({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { + provider: KEY, + payload: JSON.stringify({ calls: [{ id: "c1", name: "shot", arguments: "{}" }], details }), + }, + }); + expect(nativeIndex).toBeLessThan(events.findIndex((e) => e.type === "finish")); + }); + + it("emits an empty call list for a round that made no tool calls", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + chunk({ reasoning_details: [{ type: "reasoning.encrypted", index: 0, data: "xyz" }] }), + chunk({ content: "done" }, "stop"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + const native = events.find((e) => e.port === "nativeTurn"); + expect(JSON.parse(native.objectDelta.payload)).toEqual({ + calls: [], + details: [{ type: "reasoning.encrypted", index: 0, data: "xyz" }], + }); + }); + + it("emits no native turn when a call it made was filtered out", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + chunk({ reasoning_details: [{ type: "reasoning.encrypted", index: 0, data: "xyz" }] }), + chunk({ + tool_calls: [ + { index: 0, id: "bad", type: "function", function: { name: "nope", arguments: "{}" } }, + { index: 1, id: "c2", type: "function", function: { name: "shot", arguments: "{}" } }, + ], + }), + chunk({}, "tool_calls"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + expect( + events.find((e) => e.type === "object-delta" && e.port === "toolCalls")?.objectDelta + ).toEqual([expect.objectContaining({ id: "c2", name: "shot" })]); + expect(events.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("emits no native turn when thinking was cut off before its signature", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + chunk({ + reasoning_details: [ + { type: "reasoning.text", index: 0, format: "anthropic-claude-v1", text: "hm" }, + ], + }), + chunk({ content: "partial" }, "length"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + expect(events.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("emits no native turn when the stream carried no details", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([chunk({ content: "hi" }), chunk({}, "stop")]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + expect(events.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("sends the turn's own details back as reasoning_details", async () => { + fetchSpy.mockImplementation(async () => sseResponse([])); + await findRunFn()(replayInput(KEY), modelConfig, new AbortController().signal, () => {}); + const assistant = sentMessages().find((m) => m.role === "assistant"); + expect(assistant.reasoning_details).toEqual(details); + expect(assistant).not.toHaveProperty("native_items"); + expect(assistant.tool_calls).toHaveLength(1); + }); + + it("does not replay when the host changed the call id", async () => { + fetchSpy.mockImplementation(async () => sseResponse([])); + await findRunFn()( + replayInput(KEY, "c1-renamed", "c1"), + modelConfig, + new AbortController().signal, + () => {} + ); + const assistant = sentMessages().find((m) => m.role === "assistant"); + expect(assistant).not.toHaveProperty("reasoning_details"); + expect(assistant.tool_calls).toHaveLength(1); + }); + + it("does not replay when the host dropped the call", async () => { + fetchSpy.mockImplementation(async () => sseResponse([])); + const input = replayInput(KEY); + input.messages[1].content = input.messages[1].content.filter((b: any) => b.type !== "tool_use"); + input.messages.pop(); + await findRunFn()(input, modelConfig, new AbortController().signal, () => {}); + const assistant = sentMessages().find((m) => m.role === "assistant"); + expect(assistant).not.toHaveProperty("reasoning_details"); + }); + + it("does not replay details another model produced", async () => { + fetchSpy.mockImplementation(async () => sseResponse([])); + await findRunFn()( + replayInput("openrouter:openai/gpt-6-luna"), + modelConfig, + new AbortController().signal, + () => {} + ); + const assistant = sentMessages().find((m) => m.role === "assistant"); + expect(assistant).not.toHaveProperty("reasoning_details"); + expect(assistant).not.toHaveProperty("native_items"); + }); +}); diff --git a/packages/test/src/test/ai-provider/Gemini_ThinkingBudget.test.ts b/packages/test/src/test/ai-provider/Gemini_ThinkingBudget.test.ts index 824622b8eb..0e3f004e74 100644 --- a/packages/test/src/test/ai-provider/Gemini_ThinkingBudget.test.ts +++ b/packages/test/src/test/ai-provider/Gemini_ThinkingBudget.test.ts @@ -90,4 +90,71 @@ describe("resolveThinkingConfig", () => { maxOutputTokens: 1000, }); }); + describe("includeThoughts option", () => { + const flash = (extra: Record = {}) => + model({ provider_config: { model_name: "gemini-2.5-flash" }, ...extra }); + + it("asks for thoughts with no budget when no effort is set", () => { + expect(resolveThinkingConfig(flash(), 100, undefined, { includeThoughts: true })).toEqual({ + thinkingConfig: { includeThoughts: true }, + maxOutputTokens: 100, + }); + }); + + it("adds includeThoughts beside the effort budget and pads the output cap", () => { + expect( + resolveThinkingConfig(flash({ effort: "high" }), 100, undefined, { includeThoughts: true }) + ).toEqual({ + thinkingConfig: { thinkingBudget: 2048, includeThoughts: true }, + maxOutputTokens: 2148, + }); + }); + + it("sends nothing for a model that does not think", () => { + for (const model_name of ["gemini-embedding-001", "imagen-4.0-generate-001"]) { + expect( + resolveThinkingConfig(model({ provider_config: { model_name } }), 100, undefined, { + includeThoughts: true, + }) + ).toEqual({ thinkingConfig: undefined, maxOutputTokens: 100 }); + } + }); + + it("does not ask a model from before Gemini 2.5 for thoughts", () => { + for (const model_name of ["gemini-1.5-pro", "gemini-2.0-flash", "gemini-pro"]) { + expect( + resolveThinkingConfig(model({ provider_config: { model_name } }), 100, undefined, { + includeThoughts: true, + }) + ).toEqual({ thinkingConfig: undefined, maxOutputTokens: 100 }); + } + }); + + it("asks a -latest alias and Gemini 3 for thoughts", () => { + for (const model_name of ["gemini-flash-latest", "gemini-3.5-pro"]) { + expect( + resolveThinkingConfig(model({ provider_config: { model_name } }), 100, undefined, { + includeThoughts: true, + }) + ).toEqual({ thinkingConfig: { includeThoughts: true }, maxOutputTokens: 100 }); + } + }); + + it("does not ask for thoughts when the budget is zero", () => { + expect( + resolveThinkingConfig(flash({ effort: "none" }), 100, undefined, { includeThoughts: true }) + ).toEqual({ thinkingConfig: { thinkingBudget: 0 }, maxOutputTokens: 100 }); + }); + + it("leaves calls without the option unchanged", () => { + expect(resolveThinkingConfig(flash(), 100)).toEqual({ + thinkingConfig: undefined, + maxOutputTokens: 100, + }); + expect(resolveThinkingConfig(flash({ effort: "high" }), 100, undefined, {})).toEqual({ + thinkingConfig: { thinkingBudget: 2048 }, + maxOutputTokens: 2148, + }); + }); + }); }); diff --git a/packages/test/src/test/ai-provider/OpenAIShapedChatToolArgs.test.ts b/packages/test/src/test/ai-provider/OpenAIShapedChatToolArgs.test.ts index 5f16c4700b..ed16fb6cfc 100644 --- a/packages/test/src/test/ai-provider/OpenAIShapedChatToolArgs.test.ts +++ b/packages/test/src/test/ai-provider/OpenAIShapedChatToolArgs.test.ts @@ -98,3 +98,20 @@ describe("accumulateOpenAIChatStream tool-call arguments", () => { expect(deltas.at(-1)).toEqual([{ id: "c0", name: "now", input: {} }]); }); }); + +describe("accumulateOpenAIChatStream reasoning", () => { + it("streams reasoning_content and reasoning deltas to the reasoning port", async () => { + const out: string[] = []; + await accumulateOpenAIChatStream( + chunks([ + { choices: [{ delta: { reasoning_content: "think " } }] }, + { choices: [{ delta: { reasoning: "more" } }] }, + { choices: [{ delta: { content: "answer" } }] }, + ]), + (e) => { + if (e.type === "text-delta") out.push(`${e.port}:${e.textDelta}`); + } + ); + expect(out).toEqual(["reasoning:think ", "reasoning:more", "text:answer"]); + }); +}); diff --git a/providers/deepseek/src/ai/common/DeepSeek_Client.ts b/providers/deepseek/src/ai/common/DeepSeek_Client.ts index 0616792054..b3f7ac855d 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_Client.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_Client.ts @@ -122,6 +122,34 @@ export function getModelName(model: DeepSeekModelConfig | undefined): string { return name; } +/** + * DeepSeek's names for the shared effort levels. `none` is not a level here: + * it turns thinking off, which DeepSeek spells as a separate switch. + */ +const EFFORT_TO_REASONING_EFFORT: Record, string> = { + low: "low", + medium: "medium", + high: "high", + extra: "xhigh", + ultra: "max", +}; + +/** + * The request fields that carry `model.effort` to DeepSeek. Without them the + * effort a caller set only sized `max_tokens`, and the model reasoned at its own + * default whatever was asked. Nothing is sent for a model whose policy has no + * reasoning, or when no effort is set. + */ +export function reasoningParams(model: DeepSeekModelConfig | undefined): { + reasoning_effort?: string; + thinking?: { type: "disabled" }; +} { + const effort = resolveEnabledEffort(model, deepseekEffortPolicy(model)); + if (effort === undefined) return {}; + if (effort === "none") return { thinking: { type: "disabled" } }; + return { reasoning_effort: EFFORT_TO_REASONING_EFFORT[effort] }; +} + /** * Reasoning headroom added when the model sets no `reasoning_allowance`. Sized * above the ~4k reasoning tokens the thinking models typically spend. diff --git a/providers/deepseek/src/ai/common/DeepSeek_ModelSchema.ts b/providers/deepseek/src/ai/common/DeepSeek_ModelSchema.ts index d2e57d2161..476bbfe7be 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_ModelSchema.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_ModelSchema.ts @@ -48,6 +48,12 @@ export const DeepSeekModelSchema = { description: "Tokens added to the answer budget to leave room for reasoning. Set 0 for a non-thinking model; omit for the default.", }, + replay_reasoning: { + type: "boolean", + description: + "Send each assistant turn's reasoning back on later rounds of a tool-calling loop (default true). Off saves the prompt tokens it costs, at the price of the model working its plan out again each round.", + "x-ui-hidden": true, + }, }, required: ["model_name"], additionalProperties: false, diff --git a/providers/deepseek/src/ai/common/DeepSeek_StructuredGeneration.ts b/providers/deepseek/src/ai/common/DeepSeek_StructuredGeneration.ts index 4054aaae50..4d9c9defc0 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_StructuredGeneration.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_StructuredGeneration.ts @@ -21,6 +21,7 @@ import { getClient, getModelName, resolveMaxTokens, + reasoningParams, } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -62,6 +63,7 @@ export const DeepSeek_StructuredGeneration_Stream: AiProviderRunFn< messages: [{ role: "user", content: userContent }], response_format: responseFormat as never, max_tokens: resolveMaxTokens(model, input.maxTokens), + ...reasoningParams(model), temperature: input.temperature, stream: true, ...OPENAI_STREAM_USAGE_OPTIONS, diff --git a/providers/deepseek/src/ai/common/DeepSeek_TextGeneration.ts b/providers/deepseek/src/ai/common/DeepSeek_TextGeneration.ts index adc2282381..d06c951b20 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_TextGeneration.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_TextGeneration.ts @@ -21,6 +21,7 @@ import { getClient, getModelName, resolveMaxTokens, + reasoningParams, } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -62,6 +63,7 @@ function buildChatParams( }; const resolvedMaxTokens = resolveMaxTokens(model, input.maxTokens); if (resolvedMaxTokens !== undefined) params.max_tokens = resolvedMaxTokens; + Object.assign(params, reasoningParams(model)); if (input.temperature !== undefined) params.temperature = input.temperature; if ((input as { topP?: number }).topP !== undefined) params.top_p = (input as { topP?: number }).topP; diff --git a/providers/deepseek/src/ai/common/DeepSeek_TextRewriter.ts b/providers/deepseek/src/ai/common/DeepSeek_TextRewriter.ts index 8ab702687a..3fbc2bd757 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_TextRewriter.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_TextRewriter.ts @@ -14,7 +14,7 @@ import { createEstimatedOutputUsageReporter, OPENAI_STREAM_USAGE_OPTIONS, } from "@workglow/ai/provider-utils"; -import { getClient, getModelName } from "./DeepSeek_Client"; +import { getClient, getModelName, reasoningParams } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -42,6 +42,7 @@ export const DeepSeek_TextRewriter_Stream: AiProviderRunFn< { role: "system", content: input.prompt }, { role: "user", content: input.text }, ], + ...reasoningParams(model), stream: true, ...OPENAI_STREAM_USAGE_OPTIONS, }, diff --git a/providers/deepseek/src/ai/common/DeepSeek_TextSummary.ts b/providers/deepseek/src/ai/common/DeepSeek_TextSummary.ts index 923c48ad2b..08834f2ccb 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_TextSummary.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_TextSummary.ts @@ -14,7 +14,7 @@ import { createEstimatedOutputUsageReporter, OPENAI_STREAM_USAGE_OPTIONS, } from "@workglow/ai/provider-utils"; -import { getClient, getModelName } from "./DeepSeek_Client"; +import { getClient, getModelName, reasoningParams } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -42,6 +42,7 @@ export const DeepSeek_TextSummary_Stream: AiProviderRunFn< { role: "system", content: "Summarize the following text concisely." }, { role: "user", content: input.text }, ], + ...reasoningParams(model), stream: true, ...OPENAI_STREAM_USAGE_OPTIONS, }, diff --git a/providers/deepseek/src/ai/common/DeepSeek_ToolCalling.ts b/providers/deepseek/src/ai/common/DeepSeek_ToolCalling.ts index ba84fcf971..946a0db542 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_ToolCalling.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_ToolCalling.ts @@ -17,7 +17,7 @@ import { } from "@workglow/ai/provider-utils"; import { filterValidToolCalls, toOpenAIMessages } from "@workglow/ai/worker"; import { RetryableJobError } from "@workglow/job-queue"; -import { getClient, getModelName, resolveMaxTokens } from "./DeepSeek_Client"; +import { getClient, getModelName, reasoningParams, resolveMaxTokens } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -125,7 +125,12 @@ export const DeepSeek_ToolCalling_Stream: AiProviderRunFn< const modelName = getModelName(model); const tools = buildOpenAITools(input.tools); - const messages = toOpenAIMessages(input); + // DeepSeek's thinking models read their reasoning back on each turn of a + // tool-calling loop, and an image a tool returned has to ride a user message. + const messages = toOpenAIMessages(input, { + replayReasoning: model?.provider_config?.replay_reasoning !== false, + toolImagesInUserMessage: true, + }); const toolChoice = mapDeepSeekToolChoice(input.toolChoice); const stream = await client.chat.completions.create( @@ -133,6 +138,7 @@ export const DeepSeek_ToolCalling_Stream: AiProviderRunFn< model: modelName, messages, max_tokens: resolveMaxTokens(model, input.maxTokens), + ...reasoningParams(model), temperature: input.temperature, stream: true, tools, diff --git a/providers/deepseek/src/ai/index.ts b/providers/deepseek/src/ai/index.ts index 2134f94893..37ebca6f6b 100644 --- a/providers/deepseek/src/ai/index.ts +++ b/providers/deepseek/src/ai/index.ts @@ -11,6 +11,7 @@ export { DEEPSEEK_ALLOWED_HOSTS, DEEPSEEK_DEFAULT_BASE_URL, DEEPSEEK_DEFAULT_REASONING_ALLOWANCE, + reasoningParams, resolveMaxTokens, } from "./common/DeepSeek_Client"; export * from "./common/DeepSeek_Constants"; diff --git a/providers/google-gemini/src/ai/common/Gemini_CacheCheckpoint.ts b/providers/google-gemini/src/ai/common/Gemini_CacheCheckpoint.ts index 88d3d1d5b0..8c9bd8b492 100644 --- a/providers/google-gemini/src/ai/common/Gemini_CacheCheckpoint.ts +++ b/providers/google-gemini/src/ai/common/Gemini_CacheCheckpoint.ts @@ -21,6 +21,7 @@ import { createGeminiClient, getModelName } from "./Gemini_Client"; import type { GeminiModelConfig } from "./Gemini_ModelSchema"; import { sanitizeSchemaForGemini } from "./Gemini_Schema"; import { buildGeminiContents } from "./Gemini_ToolCalling"; +import type { IGeminiContentsOptions } from "./Gemini_ToolCalling"; /** Default TTL for explicit cached content — Gemini bills storage per token-hour. */ const GEMINI_CACHE_TTL = "3600s"; @@ -229,11 +230,12 @@ function canonicalSortKey(value: unknown): string { export function buildGeminiPrefixedContents( prefix: CheckpointPrefix, messages: ReadonlyArray | undefined, - prompt: unknown + prompt: unknown, + options: IGeminiContentsOptions = {} ): any[] { const tail: ChatMessage[] = messages && messages.length > 0 ? [...messages] : promptToTailMessages(prompt); - return buildGeminiContents([...(prefix.messages ?? []), ...tail], ""); + return buildGeminiContents([...(prefix.messages ?? []), ...tail], "", options); } /** diff --git a/providers/google-gemini/src/ai/common/Gemini_Client.ts b/providers/google-gemini/src/ai/common/Gemini_Client.ts index 6fddf42433..6329d5c544 100644 --- a/providers/google-gemini/src/ai/common/Gemini_Client.ts +++ b/providers/google-gemini/src/ai/common/Gemini_Client.ts @@ -117,6 +117,28 @@ export function getModelName(model: GeminiModelConfig | undefined): string { return name; } +/** + * Whether the model accepts images inside `functionResponse.parts`. Only the + * Gemini 3 series and later do; older ids answer such a request with a 400. + * Aliases such as `gemini-flash-latest` do not match and take the portable shape. + */ +export function geminiSupportsMultimodalFunctionResponses(modelName: string): boolean { + return /^gemini-(?:[3-9]|\d{2,})(?:[.-]|$)/i.test(modelName); +} + +/** + * Whether the model returns thought summaries: Gemini 2.5 and later, and the + * `-latest` aliases, which point at them. Earlier ids do not think and reject + * `includeThoughts`. + */ +export function geminiReturnsThoughts(modelName: string | undefined): boolean { + if (!modelName) return false; + return ( + /^gemini-(?:2\.5|[3-9]|\d{2,})(?:[.-]|$)/i.test(modelName) || + /^gemini-[a-z-]*-latest$/i.test(modelName) + ); +} + /** * Reasoning-token budget for thinking models. Native * `provider_config.thinking_budget` wins; otherwise map `model.effort`. @@ -158,15 +180,30 @@ export function getGeminiSeed(model: GeminiModelConfig | undefined): number | un export function resolveThinkingConfig( model: GeminiModelConfig | undefined, maxTokens: number | undefined, - defaultBudget?: number + defaultBudget?: number, + options?: { readonly includeThoughts?: boolean | undefined } ): { - thinkingConfig: { thinkingBudget: number } | undefined; + thinkingConfig: { thinkingBudget?: number; includeThoughts?: boolean } | undefined; maxOutputTokens: number | undefined; } { const budget = getThinkingBudget(model) ?? defaultBudget; - if (budget === undefined) { + // Thought summaries exist only when the model thinks: a zero budget turns + // thinking off, and models before Gemini 2.5 reject the config. + const thinks = + budget !== 0 && + geminiEffortPolicy(model).supported.length > 0 && + geminiReturnsThoughts(model?.provider_config?.model_name); + const includeThoughts = options?.includeThoughts === true && thinks; + if (budget === undefined && !includeThoughts) { return { thinkingConfig: undefined, maxOutputTokens: maxTokens }; } - const maxOutputTokens = maxTokens !== undefined && budget > 0 ? maxTokens + budget : maxTokens; - return { thinkingConfig: { thinkingBudget: budget }, maxOutputTokens }; + const maxOutputTokens = + maxTokens !== undefined && budget !== undefined && budget > 0 ? maxTokens + budget : maxTokens; + return { + thinkingConfig: { + ...(budget !== undefined ? { thinkingBudget: budget } : {}), + ...(includeThoughts ? { includeThoughts: true } : {}), + }, + maxOutputTokens, + }; } diff --git a/providers/google-gemini/src/ai/common/Gemini_ToolCalling.ts b/providers/google-gemini/src/ai/common/Gemini_ToolCalling.ts index 3664aecb3a..feef096799 100644 --- a/providers/google-gemini/src/ai/common/Gemini_ToolCalling.ts +++ b/providers/google-gemini/src/ai/common/Gemini_ToolCalling.ts @@ -24,6 +24,7 @@ import { generateGeminiStreamWithCacheFallback } from "./Gemini_CachedContentFal import { evictIfStaleGeminiCachedContent, getGeminiCachedContent } from "./Gemini_CacheStore"; import { createGeminiClient, + geminiSupportsMultimodalFunctionResponses, getGeminiSeed, getModelName, resolveThinkingConfig, @@ -32,9 +33,20 @@ import type { GeminiModelConfig } from "./Gemini_ModelSchema"; import { emitGeminiRefusal, geminiRefusalCategory } from "./Gemini_Refusal"; import { mapGeminiUsage } from "./Gemini_Usage"; +export interface IGeminiContentsOptions { + /** + * Whether the model takes images inside `functionResponse.parts`. When it does + * not, a tool message's images go in their own user turn after its function + * responses, behind a label. Images beside the response in the same turn made + * gemini-2.5-flash write its thinking into the answer text. + */ + readonly multimodalFunctionResponses?: boolean | undefined; +} + export function buildGeminiContents( messages: ReadonlyArray | undefined, - prompt: unknown + prompt: unknown, + options: IGeminiContentsOptions = {} ): any[] { if (!messages || messages.length === 0) { return [{ role: "user", parts: [{ text: prompt }] }]; @@ -50,7 +62,20 @@ export function buildGeminiContents( const toolUseNames = new Map(); const contents: any[] = []; + // Images a tool returned wait until every result of the run of tool messages + // is out: a user turn between two results of one model turn would leave a + // call unanswered before it. + let deferredImages: any[] = []; + const flushDeferredImages = (): void => { + if (deferredImages.length === 0) return; + contents.push({ + role: "user", + parts: [{ text: "Images returned by the tool calls above:" }, ...deferredImages], + }); + deferredImages = []; + }; for (const msg of messages) { + if (msg.role !== "tool") flushDeferredImages(); if (msg.role === "user") { const parts: any[] = []; for (const block of msg.content) { @@ -87,21 +112,46 @@ export function buildGeminiContents( for (const block of msg.content) { if (block.type !== "tool_result") continue; const name = toolUseNames.get(block.tool_use_id) ?? "unknown"; - const textContent = block.content + const text = block.content .filter((b) => b.type === "text") .map((b) => (b as { type: "text"; text: string }).text) .join(""); + const images = block.content.filter( + (b): b is Extract => b.type === "image" + ); let response: Record; - try { - response = JSON.parse(textContent); - } catch { - response = { result: textContent }; + if (block.is_error) { + response = { error: text }; + } else { + let parsed: unknown = text; + try { + parsed = JSON.parse(text); + } catch { + // Not JSON: sent as the text it is. + } + // `response` must be an object; a scalar, array or null result is wrapped. + response = + parsed !== null && typeof parsed === "object" && !Array.isArray(parsed) + ? (parsed as Record) + : { result: parsed }; } - parts.push({ functionResponse: { name, response } }); + const functionResponse: Record = { name, response }; + const imageParts = images.map((image) => ({ + inlineData: { mimeType: image.mimeType, data: image.data }, + })); + if (imageParts.length > 0) { + if (options.multimodalFunctionResponses === true) { + functionResponse.parts = imageParts; + } else { + deferredImages.push(...imageParts); + } + } + parts.push({ functionResponse }); } if (parts.length > 0) contents.push({ role: "user", parts }); } } + flushDeferredImages(); return contents; } @@ -134,6 +184,10 @@ export const Gemini_ToolCalling_Stream: AiProviderRunFn< > = async (input, model, signal, emit, _outputSchema, sessionContext) => { const ai = await createGeminiClient(model); + const contentOptions: IGeminiContentsOptions = { + multimodalFunctionResponses: geminiSupportsMultimodalFunctionResponses(getModelName(model)), + }; + const functionDeclarations = buildGeminiFunctionDeclarations(input.tools); const toolConfig = mapGeminiToolConfig(input.toolChoice); @@ -186,12 +240,17 @@ export const Gemini_ToolCalling_Stream: AiProviderRunFn< // Thinking is opt-in here (no default budget): the model uses its own default // reasoning unless `provider_config.thinking_budget` is set, in which case the // output cap is padded so reasoning can't starve the tool call / answer. - const { thinkingConfig, maxOutputTokens } = resolveThinkingConfig(model, input.maxTokens); + const { thinkingConfig, maxOutputTokens } = resolveThinkingConfig( + model, + input.maxTokens, + undefined, + { includeThoughts: true } + ); /** Build the tail-only request that references the CachedContent handle. */ const buildCachedRequest = (): Record => ({ model: getModelName(model), - contents: buildGeminiContents(input.messages, input.prompt), + contents: buildGeminiContents(input.messages, input.prompt, contentOptions), config: { abortSignal: signal ?? undefined, systemInstruction: undefined, @@ -206,8 +265,8 @@ export const Gemini_ToolCalling_Stream: AiProviderRunFn< /** Build the full inline-replay request (prefix messages + tail + tools). */ const buildInlineReplayRequest = (): Record => { const contents = prefix - ? buildGeminiPrefixedContents(prefix, input.messages, input.prompt) - : buildGeminiContents(input.messages, input.prompt); + ? buildGeminiPrefixedContents(prefix, input.messages, input.prompt, contentOptions) + : buildGeminiContents(input.messages, input.prompt, contentOptions); const systemInstruction = input.systemPrompt || (prefix ? prefix.systemPrompt : undefined); return { model: getModelName(model), @@ -246,7 +305,9 @@ export const Gemini_ToolCalling_Stream: AiProviderRunFn< refusalCategory = refusalCategory ?? geminiRefusalCategory(chunk); const parts = chunk.candidates?.[0]?.content?.parts ?? []; for (const part of parts) { - if (part.text && !part.thought) { + if (part.text && part.thought) { + emit({ type: "text-delta", port: "reasoning", textDelta: part.text }); + } else if (part.text) { emit({ type: "text-delta", port: "text", textDelta: part.text }); } if (part.functionCall) { diff --git a/providers/google-gemini/src/ai/index.ts b/providers/google-gemini/src/ai/index.ts index cc0f12be70..21650bd9e7 100644 --- a/providers/google-gemini/src/ai/index.ts +++ b/providers/google-gemini/src/ai/index.ts @@ -19,7 +19,11 @@ import { generateGeminiStreamWithCacheFallback, isGeminiCachedContentNotFoundError, } from "./common/Gemini_CachedContentFallback"; -import { _testOnly as clientTestOnly, resolveThinkingConfig } from "./common/Gemini_Client"; +import { + _testOnly as clientTestOnly, + geminiSupportsMultimodalFunctionResponses, + resolveThinkingConfig, +} from "./common/Gemini_Client"; import { _cacheStoreTestOnly, getGeminiCachedContent, @@ -42,6 +46,7 @@ export const _testOnly = { GEMINI_RUN_FN_SPECS, GEMINI_RUN_FNS, buildGeminiContents, + geminiSupportsMultimodalFunctionResponses, geminiRefusalCategory, emitGeminiRefusal, setGeminiClientForTests: clientTestOnly.setGeminiClientForTests, diff --git a/providers/huggingface-inference/src/ai/HfInferenceProvider.ts b/providers/huggingface-inference/src/ai/HfInferenceProvider.ts index 7302329b21..b9ad353a71 100644 --- a/providers/huggingface-inference/src/ai/HfInferenceProvider.ts +++ b/providers/huggingface-inference/src/ai/HfInferenceProvider.ts @@ -5,13 +5,14 @@ */ import { createCloudProviderClass } from "@workglow/ai/provider-utils"; -import type { Capability, ModelRecord } from "@workglow/ai/worker"; +import type { Capability, ModelEffortPolicy, ModelRecord } from "@workglow/ai/worker"; import { AiProvider } from "@workglow/ai/worker"; import { hfInferenceWorkerRunFnSpecs, inferHfInferenceCapabilities, } from "./common/HFI_Capabilities"; import { HF_INFERENCE } from "./common/HFI_Constants"; +import { hfiEffortPolicy } from "./common/HFI_EffortPolicy"; import type { HfInferenceModelConfig } from "./common/HFI_ModelSchema"; /** @@ -29,6 +30,10 @@ export class HfInferenceProvider extends createCloudProviderClass + * SPDX-License-Identifier: Apache-2.0 + */ + +import { + EFFORT_POLICY_ALL, + EFFORT_POLICY_NONE, + makeEffortPolicy, + resolveEnabledEffort, + type ModelEffort, + type ModelEffortPolicyFn, +} from "@workglow/ai/worker"; +import type { HfInferenceModelConfig } from "./HFI_ModelSchema"; + +/** Ids on the router that are not chat models, so take no reasoning dial. */ +const NON_TEXT = [ + /embed/i, + /(?:^|\/)(?:bge|gte|e5)-/i, + /rerank/i, + /stable-diffusion|flux|sdxl/i, + /(?:^|[/-])(?:whisper|tts)/i, +]; + +/** + * The router forwards to many backends and drops nothing on its own, so an id + * no rule recognizes keeps the dial; the field is sent only when a caller set + * an effort. + */ +export const hfiEffortPolicy: ModelEffortPolicyFn = makeEffortPolicy({ + rules: [{ when: NON_TEXT, policy: EFFORT_POLICY_NONE }], + fallback: EFFORT_POLICY_ALL, +}); + +const EFFORT_TO_REASONING_EFFORT: Record, string> = { + low: "low", + medium: "medium", + high: "high", + extra: "high", + ultra: "high", +}; + +/** The chat-completions `reasoning_effort` field for a model's effort, or nothing. */ +export function hfiReasoningParams(model: HfInferenceModelConfig | undefined): { + reasoning_effort?: string; +} { + const effort = resolveEnabledEffort(model, hfiEffortPolicy(model)); + if (effort === undefined || effort === "none") return {}; + return { reasoning_effort: EFFORT_TO_REASONING_EFFORT[effort] }; +} diff --git a/providers/huggingface-inference/src/ai/common/HFI_TextGeneration.ts b/providers/huggingface-inference/src/ai/common/HFI_TextGeneration.ts index fd61b539aa..12fad774f7 100644 --- a/providers/huggingface-inference/src/ai/common/HFI_TextGeneration.ts +++ b/providers/huggingface-inference/src/ai/common/HFI_TextGeneration.ts @@ -16,6 +16,7 @@ import { } from "@workglow/ai/provider-utils"; import { toOpenAIMessages } from "@workglow/ai/worker"; import { getClient, getModelName, getProvider } from "./HFI_Client"; +import { hfiReasoningParams } from "./HFI_EffortPolicy"; import type { HfInferenceModelConfig } from "./HFI_ModelSchema"; /** @@ -76,6 +77,7 @@ export const HFI_TextGeneration_Stream: AiProviderRunFn< top_p: input.topP, frequency_penalty: input.frequencyPenalty, provider, + ...hfiReasoningParams(model), }, { signal } ); diff --git a/providers/huggingface-inference/src/ai/common/HFI_ToolCalling.ts b/providers/huggingface-inference/src/ai/common/HFI_ToolCalling.ts index becef88d9a..26563eda1f 100644 --- a/providers/huggingface-inference/src/ai/common/HFI_ToolCalling.ts +++ b/providers/huggingface-inference/src/ai/common/HFI_ToolCalling.ts @@ -14,6 +14,7 @@ import { } from "@workglow/ai/provider-utils"; import { toOpenAIMessages } from "@workglow/ai/worker"; import { getClient, getModelName, getProvider } from "./HFI_Client"; +import { hfiReasoningParams } from "./HFI_EffortPolicy"; import type { HfInferenceModelConfig } from "./HFI_ModelSchema"; export const HFI_ToolCalling_Stream: AiProviderRunFn< @@ -26,7 +27,7 @@ export const HFI_ToolCalling_Stream: AiProviderRunFn< const provider = getProvider(model); const tools = buildOpenAITools(input.tools); - const messages = toOpenAIMessages(input); + const messages = toOpenAIMessages(input, { toolImagesInUserMessage: true }); const toolChoice = mapOpenAIToolChoice(input.toolChoice, false); const promptText = messages .map((m) => (typeof m.content === "string" ? m.content : "")) @@ -40,6 +41,7 @@ export const HFI_ToolCalling_Stream: AiProviderRunFn< temperature: input.temperature, provider, stream: true, + ...hfiReasoningParams(model), }; if (toolChoice !== "none") { params.tools = tools; diff --git a/providers/huggingface-inference/src/ai/index.ts b/providers/huggingface-inference/src/ai/index.ts index 0793d28295..8b7e812ec6 100644 --- a/providers/huggingface-inference/src/ai/index.ts +++ b/providers/huggingface-inference/src/ai/index.ts @@ -8,6 +8,7 @@ export * from "./common/HFI_AspectRatio"; export * from "./common/HFI_Constants"; +export * from "./common/HFI_EffortPolicy"; export * from "./common/HFI_ImageValidation"; export * from "./common/HFI_ModelSchema"; export * from "./common/HFI_ModelSearch"; diff --git a/providers/huggingface-inference/src/ai/runtime.ts b/providers/huggingface-inference/src/ai/runtime.ts index 1d96fa057b..6bf8d23573 100644 --- a/providers/huggingface-inference/src/ai/runtime.ts +++ b/providers/huggingface-inference/src/ai/runtime.ts @@ -14,5 +14,6 @@ // organize-imports-ignore export * from "./common/HFI_Client"; +export * from "./common/HFI_EffortPolicy"; export * from "./registerHfInferenceInline"; export * from "./registerHfInferenceWorker"; diff --git a/providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails.ts b/providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails.ts new file mode 100644 index 0000000000..3f617f32b0 --- /dev/null +++ b/providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails.ts @@ -0,0 +1,188 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ChatMessage, ContentBlockReasoning } from "@workglow/ai"; +import type { OpenAICompatMessage } from "@workglow/ai/worker"; + +/** Fields of a reasoning detail whose streamed pieces are text to be joined. */ +const CONCATENATED_KEYS: ReadonlySet = new Set(["text", "summary", "data"]); + +/** + * Joins the pieces of `reasoning_details` OpenRouter streams on + * `choices[0].delta`. Each piece carries the `index` of the item it belongs to. + */ +export interface ReasoningDetailAccumulator { + readonly entries: Map>; + /** The joined items by index, or `undefined` when no piece arrived. */ + items(): unknown[] | undefined; +} + +export function createReasoningDetailAccumulator(): ReasoningDetailAccumulator { + const entries = new Map>(); + return { + entries, + items() { + if (entries.size === 0) return undefined; + return [...entries.entries()].sort(([a], [b]) => a - b).map(([, item]) => item); + }, + }; +} + +/** Folds one streamed `reasoning_details` array into the accumulator. */ +export function mergeReasoningDetailDelta(acc: ReasoningDetailAccumulator, delta: unknown): void { + if (!Array.isArray(delta)) return; + for (const piece of delta) { + if (piece === null || typeof piece !== "object" || Array.isArray(piece)) continue; + const incoming = piece as Record; + const index = + typeof incoming.index === "number" && Number.isFinite(incoming.index) + ? incoming.index + : nextIndex(acc); + const existing = acc.entries.get(index); + if (!existing) { + acc.entries.set(index, { ...incoming }); + continue; + } + for (const [key, value] of Object.entries(incoming)) { + const current = existing[key]; + existing[key] = + CONCATENATED_KEYS.has(key) && typeof current === "string" && typeof value === "string" + ? current + value + : value; + } + } +} + +function nextIndex(acc: ReasoningDetailAccumulator): number { + let max = -1; + for (const index of acc.entries.keys()) max = Math.max(max, index); + return max + 1; +} + +/** A call as the replay check compares it: its arguments as the converter writes them. */ +export interface ReasoningTurnCall { + readonly id: string; + readonly name: string; + readonly arguments: string; +} + +/** What one round's native turn carries: its reasoning and the calls it was written beside. */ +export interface ReasoningTurn { + readonly calls: readonly ReasoningTurnCall[]; + readonly details: readonly unknown[]; +} + +/** + * Whether the details stand as the model wrote them. An Anthropic thinking + * block streamed through OpenRouter is only replayable with its signature, and + * the stream can end before that arrives. + */ +function isReplayable(details: readonly unknown[]): boolean { + return !details.some((detail) => { + if (detail === null || typeof detail !== "object") return false; + const { type, format, signature } = detail as Record; + return ( + type === "reasoning.text" && + typeof format === "string" && + format.startsWith("anthropic-") && + !(typeof signature === "string" && signature.length > 0) + ); + }); +} + +/** + * The native-turn payload for a round, or `undefined` when the details cannot + * be replayed and the turn is better rebuilt. + */ +export function encodeReasoningTurn( + calls: ReadonlyArray<{ readonly id: string; readonly name: string; readonly input: unknown }>, + details: readonly unknown[] +): string | undefined { + if (!isReplayable(details)) return undefined; + return JSON.stringify({ + calls: calls.map((call) => ({ + id: call.id, + name: call.name, + arguments: JSON.stringify(call.input ?? {}), + })), + details, + }); +} + +function isTurnCall(value: unknown): value is ReasoningTurnCall { + if (value === null || typeof value !== "object") return false; + const { id, name, arguments: args } = value as Record; + return typeof id === "string" && typeof name === "string" && typeof args === "string"; +} + +function decodeReasoningTurn(payload: string): ReasoningTurn | undefined { + let parsed: unknown; + try { + parsed = JSON.parse(payload); + } catch { + return undefined; + } + if (parsed === null || typeof parsed !== "object") return undefined; + const { calls, details } = parsed as Record; + // Each detail must at least be a typed object: `[null]` or a bare value would + // be sent back as `reasoning_details` and refused, where a rebuilt turn is not. + if ( + !Array.isArray(details) || + !details.every( + (detail) => + detail !== null && + typeof detail === "object" && + !Array.isArray(detail) && + typeof (detail as { type?: unknown }).type === "string" + ) + ) { + return undefined; + } + if (!Array.isArray(calls) || !calls.every(isTurnCall)) return undefined; + return { calls, details }; +} + +/** + * OpenRouter reads an assistant turn's reasoning back as `reasoning_details`. + * `converted` is the shared converter's output for `history`, which keeps one + * assistant message per assistant turn, in order. Each turn's own reasoning + * block, written by `nativeKey`, is attached to its message only while the + * turn's tool calls still equal the ones the details were written beside, by + * id, name and arguments in order: a host that changed, dropped or added a call + * gets the rebuilt turn. The inputs are not mutated. + */ +export function withReasoningDetails( + converted: readonly OpenAICompatMessage[], + history: ReadonlyArray | undefined, + nativeKey: string +): Array { + const turns = (history ?? []).filter((message) => message.role === "assistant"); + const aligned = + turns.length === converted.filter((message) => message.role === "assistant").length; + let next = 0; + return converted.map((message) => { + if (message.role !== "assistant" || !aligned) return message; + const turn = turns[next++]!; + const block = turn.content.find( + (b): b is ContentBlockReasoning => + b.type === "reasoning" && b.provider === nativeKey && b.payload !== undefined + ); + const decoded = block === undefined ? undefined : decodeReasoningTurn(block.payload!); + if (decoded === undefined) return message; + const sent = message.tool_calls ?? []; + const matches = + sent.length === decoded.calls.length && + sent.every((call, i) => { + const written = decoded.calls[i]!; + return ( + call.id === written.id && + call.function.name === written.name && + call.function.arguments === written.arguments + ); + }); + return matches ? { ...message, reasoning_details: decoded.details } : message; + }); +} diff --git a/providers/openrouter/src/ai/common/OpenRouter_ToolCalling.ts b/providers/openrouter/src/ai/common/OpenRouter_ToolCalling.ts index bf8186180c..8a0884f87b 100644 --- a/providers/openrouter/src/ai/common/OpenRouter_ToolCalling.ts +++ b/providers/openrouter/src/ai/common/OpenRouter_ToolCalling.ts @@ -20,8 +20,62 @@ import { filterValidToolCalls, toOpenAIMessages } from "@workglow/ai/worker"; import { getClient, getModelName } from "./OpenRouter_Client"; import type { OpenRouterModelConfig } from "./OpenRouter_ModelSchema"; import { buildOpenRouterExtras } from "./OpenRouter_RequestParams"; +import { + createReasoningDetailAccumulator, + encodeReasoningTurn, + mergeReasoningDetailDelta, + withReasoningDetails, +} from "./OpenRouter_ReasoningDetails"; +import type { ReasoningDetailAccumulator } from "./OpenRouter_ReasoningDetails"; import { mapOpenRouterUsage } from "./OpenRouter_Usage"; +/** The readable text of a chunk's reasoning details: `reasoning.text` and summaries. */ +function detailText(details: readonly unknown[]): string { + return details + .map((detail) => { + // A malformed entry is skipped, as the accumulator skips it. + if (detail === null || typeof detail !== "object") return ""; + const d = detail as { type?: unknown; text?: unknown; summary?: unknown }; + if (d.type === "reasoning.text" && typeof d.text === "string") return d.text; + if (d.type === "reasoning.summary" && typeof d.summary === "string") return d.summary; + return ""; + }) + .join(""); +} + +/** + * Passes the stream through while collecting each chunk's `reasoning_details`. + * A chunk whose thinking arrives only there, with no `reasoning` of its own, + * gets that text as `reasoning`, so it reaches the reasoning port as well as + * the replay; one that carries both is left alone, so nothing is said twice. + */ +async function* tapReasoningDetails( + stream: AsyncIterable, + acc: ReasoningDetailAccumulator +): AsyncGenerator { + for await (const chunk of stream) { + const delta = chunk?.choices?.[0]?.delta; + const details = delta?.reasoning_details; + if (!Array.isArray(details)) { + yield chunk; + continue; + } + mergeReasoningDetailDelta(acc, details); + const text = detailText(details); + if (text && !delta.reasoning && !delta.reasoning_content) { + yield { + ...chunk, + choices: [ + { ...chunk.choices[0], delta: { ...delta, reasoning: text } }, + ...chunk.choices.slice(1), + ], + }; + continue; + } + yield chunk; + } +} + /** * Streaming run-fn for `["text.generation", "tool-use"]`. Forwards deltas via * {@link accumulateOpenAIChatStream}; each tool-call delta is filtered against @@ -36,7 +90,14 @@ export const OpenRouter_ToolCalling_Stream: AiProviderRunFn< const modelName = getModelName(model); const tools = buildOpenAITools(input.tools); - const messages = toOpenAIMessages(input); + // The details are readable only by the model that wrote them, so the native + // turn is keyed by model and another model gets the rebuilt turn. + const nativeKey = `openrouter:${modelName}`; + const messages = withReasoningDetails( + toOpenAIMessages(input, { toolImagesInUserMessage: true }), + input.messages, + nativeKey + ); const toolChoice = mapOpenAIToolChoice(input.toolChoice, true); const stream = await client.chat.completions.create( @@ -54,12 +115,24 @@ export const OpenRouter_ToolCalling_Stream: AiProviderRunFn< { signal } ); + const details = createReasoningDetailAccumulator(); + // The calls this round actually emitted, in order, each as last streamed: the + // replay is valid only beside the same calls. + const emittedCalls = new Map(); + // Whether each call the model made survived validation, as last streamed. The + // details were produced beside every call, so a turn missing one of them is + // not the turn they belong to and is not replayed. + const callKept = new Map(); const usage = await accumulateOpenAIChatStream( - stream, + tapReasoningDetails(stream, details), (event) => { if (event.type === "object-delta" && event.port === "toolCalls") { - const validated = filterValidToolCalls(event.objectDelta as ToolCalls, input.tools); + const calls = event.objectDelta as ToolCalls; + const validated = filterValidToolCalls(calls, input.tools); + const kept = new Set(validated.map((call) => call.id)); + for (const call of calls) callKept.set(call.id, kept.has(call.id)); if (validated.length > 0) { + for (const call of validated) emittedCalls.set(call.id, call); emit({ type: "object-delta", port: "toolCalls", objectDelta: validated }); } return; @@ -74,5 +147,18 @@ export const OpenRouter_ToolCalling_Stream: AiProviderRunFn< .join("\n"), } ); + const items = details.items(); + const dropped = [...callKept.values()].some((isKept) => !isKept); + const payload = + items === undefined || dropped + ? undefined + : encodeReasoningTurn([...emittedCalls.values()], items); + if (payload !== undefined) { + emit({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: nativeKey, payload }, + }); + } emit({ type: "finish", data: { text: "", toolCalls: [] } as ToolCallingTaskOutput, usage }); }; diff --git a/providers/xai/src/ai/common/Xai_ToolCalling.ts b/providers/xai/src/ai/common/Xai_ToolCalling.ts index f7bfe3f9a5..dd65ab16ed 100644 --- a/providers/xai/src/ai/common/Xai_ToolCalling.ts +++ b/providers/xai/src/ai/common/Xai_ToolCalling.ts @@ -40,7 +40,7 @@ export const Xai_ToolCalling_Stream: AiProviderRunFn< const modelName = getModelName(model); const tools = buildOpenAITools(input.tools); - const messages = toOpenAIMessages(input); + const messages = toOpenAIMessages(input, { toolImagesInUserMessage: true }); const toolChoice = mapOpenAIToolChoice(input.toolChoice, true); const reasoningEffort = getXaiReasoningEffort(model); From 3603213e49c17e6872133b999074e2a34a7a411c Mon Sep 17 00:00:00 2001 From: Steven Roussey Date: Mon, 12 Oct 2026 05:34:45 +0000 Subject: [PATCH 4/7] providers: carry reasoning, tool calls and images between rounds on local runtimes - Ollama and llama.cpp server send the whole conversation with their own tool calls and replay thinking / reasoning_content; Ollama gains a think effort setting and sends tool-result images on a user turn after the results. - node-llama-cpp emits and replays thought segments, maps effort to a thought budget and never inlines image data. - HF Transformers routes to reasoning (ThinkTagFilter), hands it back to the chat template whenever the history carries it, and names tool results. - Cactus pairs the latest user turn with the turn's prompt and emits its reasoning; Chrome AI's finish has the same shape as the other providers. --- packages/ai/src/provider-utils.ts | 1 + .../ai/src/provider-utils/ThinkTagFilter.ts | 84 ++++ ...lamaCppServer_TextGenerationStream.test.ts | 57 +++ .../LlamaCppServer_ToolCalling.test.ts | 145 ++++++ .../Ollama_TextGenerationStream.test.ts | 102 ++++ .../Ollama_ToolCalling.test.ts | 441 ++++++++++++++++++ .../Cactus_ToolCalling.test.ts | 149 +++++- .../ai-provider-hft/HFT_ToolCalling.test.ts | 223 ++++++++- .../LlamaCpp_ToolCalling.test.ts | 233 +++++++++ .../LlamaCpp_ToolCallingCheckpoint.test.ts | 8 +- .../test/ai-provider/ThinkTagFilter.test.ts | 119 +++++ .../ai-provider/WebBrowserProvider.test.ts | 22 + .../src/ai/common/Cactus_ParseToolCalls.ts | 52 ++- .../src/ai/common/Cactus_ToolCallingCore.ts | 40 +- .../src/ai/common/WebBrowser_ToolCalling.ts | 2 +- .../src/ai/common/HFT_ToolCalling.ts | 102 +++- .../src/ai/common/LlamaCppServer_Client.ts | 7 +- .../common/LlamaCppServer_TextGeneration.ts | 42 +- .../ai/common/LlamaCppServer_ToolCalling.ts | 25 +- .../node-llama-cpp/src/ai/LlamaCppProvider.ts | 6 + .../src/ai/LlamaCppQueuedProvider.ts | 6 + .../src/ai/common/LlamaCpp_Chat.ts | 2 + .../src/ai/common/LlamaCpp_EffortPolicy.ts | 45 ++ .../src/ai/common/LlamaCpp_TextGeneration.ts | 2 + .../src/ai/common/LlamaCpp_ToolCalling.ts | 86 +++- providers/node-llama-cpp/src/ai/index.ts | 1 + providers/node-llama-cpp/src/ai/runtime.ts | 1 + providers/ollama/src/ai/OllamaProvider.ts | 7 +- .../ollama/src/ai/OllamaQueuedProvider.ts | 13 +- .../src/ai/common/Ollama_EffortPolicy.ts | 88 ++++ .../src/ai/common/Ollama_JobRunFns.browser.ts | 20 +- .../ollama/src/ai/common/Ollama_JobRunFns.ts | 7 +- .../ollama/src/ai/common/Ollama_Messages.ts | 173 +++++++ .../src/ai/common/Ollama_TextGeneration.ts | 34 +- .../src/ai/common/Ollama_ToolCalling.ts | 12 +- providers/ollama/src/ai/index.browser.ts | 1 + providers/ollama/src/ai/index.ts | 1 + providers/ollama/src/ai/runtime.browser.ts | 1 + providers/ollama/src/ai/runtime.ts | 1 + 39 files changed, 2249 insertions(+), 112 deletions(-) create mode 100644 packages/ai/src/provider-utils/ThinkTagFilter.ts create mode 100644 packages/test/src/test/ai-provider-api/Ollama_ToolCalling.test.ts create mode 100644 packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCalling.test.ts create mode 100644 packages/test/src/test/ai-provider/ThinkTagFilter.test.ts create mode 100644 providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy.ts create mode 100644 providers/ollama/src/ai/common/Ollama_EffortPolicy.ts create mode 100644 providers/ollama/src/ai/common/Ollama_Messages.ts diff --git a/packages/ai/src/provider-utils.ts b/packages/ai/src/provider-utils.ts index 11b2387e5d..64382898e5 100644 --- a/packages/ai/src/provider-utils.ts +++ b/packages/ai/src/provider-utils.ts @@ -18,6 +18,7 @@ export * from "./provider-utils/registerProvider"; export * from "./provider-utils/modelSearchQuery"; export * from "./provider-utils/ToolCallParsers"; export * from "./provider-utils/ToolCallMarkupFilter"; +export * from "./provider-utils/ThinkTagFilter"; export * from "./provider-utils/HfModelSearch"; export * from "./provider-utils/PipelineTaskMapping"; export * from "./provider-utils/imageOutputHelpers"; diff --git a/packages/ai/src/provider-utils/ThinkTagFilter.ts b/packages/ai/src/provider-utils/ThinkTagFilter.ts new file mode 100644 index 0000000000..d834ad868c --- /dev/null +++ b/packages/ai/src/provider-utils/ThinkTagFilter.ts @@ -0,0 +1,84 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** One routed piece of a stream: which output port it belongs to, and its text. */ +export interface ThinkTagPart { + readonly port: "text" | "reasoning"; + readonly text: string; +} + +export interface ThinkTagFilter { + /** Routes one streamed delta; returns the parts to emit, in order. */ + push(delta: string): ReadonlyArray; + /** Releases anything held back and returns to the starting state. */ + flush(): ReadonlyArray; +} + +const OPEN_TAG = ""; +const CLOSE_TAG = ""; + +/** Length of the longest suffix of `text` that is a proper prefix of `tag`. */ +function danglingPrefixLength(text: string, tag: string): number { + for (let len = Math.min(text.length, tag.length - 1); len >= 1; len--) { + if (text.endsWith(tag.slice(0, len))) return len; + } + return 0; +} + +/** + * Splits a stream of text deltas into reasoning and answer text, using the + * `…` convention of Qwen3-style chat templates. + * + * A tag can arrive split across deltas, so a suffix that might still become + * the tag being looked for is held back until it resolves. A block the stream + * never closes stays on `reasoning` to the end: a generation cut off at the + * token limit loses its closing tag, and what follows the opening one is still + * thinking. + * + * `startInside` is for a template that opens the tag in the prompt, so the + * generation begins mid-block and only the closing tag appears in the stream. + */ +export function createThinkTagFilter(options?: { + readonly startInside?: boolean | undefined; +}): ThinkTagFilter { + const startInside = options?.startInside === true; + let inside = startInside; + let held = ""; + + function part(port: "text" | "reasoning", text: string, into: ThinkTagPart[]): void { + if (text.length > 0) into.push({ port, text }); + } + + function push(delta: string): ReadonlyArray { + const out: ThinkTagPart[] = []; + let buffer = held + delta; + held = ""; + for (;;) { + const tag = inside ? CLOSE_TAG : OPEN_TAG; + const port = inside ? "reasoning" : "text"; + const at = buffer.indexOf(tag); + if (at === -1) { + const keep = danglingPrefixLength(buffer, tag); + part(port, buffer.slice(0, buffer.length - keep), out); + held = buffer.slice(buffer.length - keep); + return out; + } + part(port, buffer.slice(0, at), out); + buffer = buffer.slice(at + tag.length); + inside = !inside; + } + } + + function flush(): ReadonlyArray { + const out: ThinkTagPart[] = []; + part(inside ? "reasoning" : "text", held, out); + held = ""; + inside = startInside; + return out; + } + + return { push, flush }; +} diff --git a/packages/test/src/test/ai-provider-api/LlamaCppServer_TextGenerationStream.test.ts b/packages/test/src/test/ai-provider-api/LlamaCppServer_TextGenerationStream.test.ts index e1ab72b9ad..0f7862b801 100644 --- a/packages/test/src/test/ai-provider-api/LlamaCppServer_TextGenerationStream.test.ts +++ b/packages/test/src/test/ai-provider-api/LlamaCppServer_TextGenerationStream.test.ts @@ -73,6 +73,63 @@ describe("createLlamaCppServerTextGenerationStream", () => { ]); }); + it("sends a history of system turns alone with no invented user turn", async () => { + const fetchSpy = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(sseResponse([dataLine("ok"), "data: [DONE]\n"])); + const fn = createLlamaCppServerTextGenerationStream({}); + await fn( + { prompt: "", messages: [{ role: "system", content: "be brief" }] } as any, + model, + undefined as any, + (_e: any) => undefined + ); + const body = JSON.parse(asText((fetchSpy.mock.calls[0]![1] as RequestInit).body)); + expect(body.messages).toEqual([{ role: "system", content: "be brief" }]); + }); + + it("converts block history to chat-completions parts and drops blocks it has no form for", async () => { + const fetchSpy = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(sseResponse([dataLine("ok"), "data: [DONE]\n"])); + const fn = createLlamaCppServerTextGenerationStream({}); + await fn( + { + prompt: "", + messages: [ + { + role: "user", + content: [ + { type: "text", text: "What is this?" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "It looks red." }, + { type: "text", text: "A red square." }, + ], + }, + ], + } as any, + model, + undefined as any, + () => undefined + ); + const body = JSON.parse(asText((fetchSpy.mock.calls[0]![1] as RequestInit).body)); + expect(body.messages).toEqual([ + { + role: "user", + content: [ + { type: "text", text: "What is this?" }, + { type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } }, + ], + }, + { role: "assistant", content: "A red square." }, + ]); + }); + it("throws on non-2xx with informative message", async () => { vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response("nope", { status: 500 })); const fn = createLlamaCppServerTextGenerationStream({}); diff --git a/packages/test/src/test/ai-provider-api/LlamaCppServer_ToolCalling.test.ts b/packages/test/src/test/ai-provider-api/LlamaCppServer_ToolCalling.test.ts index a4bc78031b..7de871624b 100644 --- a/packages/test/src/test/ai-provider-api/LlamaCppServer_ToolCalling.test.ts +++ b/packages/test/src/test/ai-provider-api/LlamaCppServer_ToolCalling.test.ts @@ -120,4 +120,149 @@ describe("createLlamaCppServerToolCallingStream", () => { const body = JSON.parse(asText((fetchSpy.mock.calls[0]![1] as RequestInit).body)); expect(body.tools).toBeUndefined(); }); + describe("conversation and reasoning", () => { + async function requestBodyFor(input: object): Promise { + const fetchSpy = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(sseChunks([{ choices: [{ delta: { content: "ok" } }] }])); + const fn = createLlamaCppServerToolCallingStream({}); + await fn( + { tools: TOOLS, toolChoice: "auto", ...input } as any, + model, + undefined as any, + () => undefined + ); + return JSON.parse(asText((fetchSpy.mock.calls[0]![1] as RequestInit).body)); + } + + it("sends the whole conversation, including its own tool calls and their results", async () => { + const body = await requestBodyFor({ + prompt: "p", + messages: [ + { role: "user", content: [{ type: "text", text: "p" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "lookup", input: {} }], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "text", text: "42" }], + }, + ], + }, + ], + }); + expect(body.messages).toEqual([ + { role: "user", content: [{ type: "text", text: "p" }] }, + { + role: "assistant", + content: null, + tool_calls: [ + { id: "c1", type: "function", function: { name: "lookup", arguments: "{}" } }, + ], + reasoning_content: "", + }, + { role: "tool", content: "42", tool_call_id: "c1" }, + ]); + }); + + it("counts a parts-array user message in the provisional prompt estimate", async () => { + vi.spyOn(globalThis, "fetch").mockResolvedValue( + sseChunks([{ choices: [{ delta: { content: "ok" } }] }]) + ); + const events: any[] = []; + await createLlamaCppServerToolCallingStream({})( + { + tools: TOOLS, + toolChoice: "auto", + prompt: "", + messages: [{ role: "user", content: [{ type: "text", text: "word ".repeat(400) }] }], + } as any, + model, + undefined as any, + (event: any) => events.push(event) + ); + const first = events.find((event) => event.type === "usage"); + expect(first?.usage?.input).toBeGreaterThan(0); + }); + + it("replays an assistant turn's reasoning as reasoning_content", async () => { + const body = await requestBodyFor({ + prompt: "p", + messages: [ + { role: "user", content: [{ type: "text", text: "p" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "I should look it up" }, + { type: "tool_use", id: "c1", name: "lookup", input: {} }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "c1", content: [{ type: "text", text: "42" }] }, + ], + }, + ], + }); + const assistant = body.messages.find((m: any) => m.role === "assistant"); + expect(assistant.reasoning_content).toBe("I should look it up"); + }); + + it("sends a tool-result image as a user message after the string tool message", async () => { + const body = await requestBodyFor({ + prompt: "p", + messages: [ + { role: "user", content: [{ type: "text", text: "p" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "shot", input: {} }], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + ], + }); + const toolIdx = body.messages.findIndex((m: any) => m.role === "tool"); + expect(typeof body.messages[toolIdx].content).toBe("string"); + const after = body.messages[toolIdx + 1]; + expect(after.role).toBe("user"); + expect(after.content).toContainEqual({ + type: "image_url", + image_url: { url: "data:image/png;base64,AAAA" }, + }); + }); + + it("emits delta.reasoning_content on the reasoning port, not on text", async () => { + vi.spyOn(globalThis, "fetch").mockResolvedValue( + sseChunks([ + { choices: [{ delta: { reasoning_content: "hmm" } }] }, + { choices: [{ delta: { content: "answer" } }] }, + ]) + ); + const fn = createLlamaCppServerToolCallingStream({}); + const events: any[] = []; + await fn( + { prompt: "p", tools: TOOLS, toolChoice: "auto" } as any, + model, + undefined as any, + (e: any) => events.push(e) + ); + expect(events).toContainEqual({ type: "text-delta", port: "reasoning", textDelta: "hmm" }); + const textDeltas = events.filter((e) => e.type === "text-delta" && e.port === "text"); + expect(textDeltas.map((e) => e.textDelta)).toEqual(["answer"]); + }); + }); }); diff --git a/packages/test/src/test/ai-provider-api/Ollama_TextGenerationStream.test.ts b/packages/test/src/test/ai-provider-api/Ollama_TextGenerationStream.test.ts index 0fdb41b6b8..1203f8f7e7 100644 --- a/packages/test/src/test/ai-provider-api/Ollama_TextGenerationStream.test.ts +++ b/packages/test/src/test/ai-provider-api/Ollama_TextGenerationStream.test.ts @@ -207,3 +207,105 @@ describe("createOllamaTextGenerationStream usage", () => { expect(finish.usage.output).toBe(3); }); }); + +describe("createOllamaTextGenerationStream chat history", () => { + const model = { + model_id: "ollama:test", + provider_config: { model_name: "llava" }, + } as any; + + it("sends block history as Ollama messages: string content, images beside it, thinking replayed", async () => { + const chat = vi.fn().mockResolvedValue(makeFakeStream(["ok"])); + const streamFn = createOllamaTextGenerationStream(vi.fn().mockResolvedValue({ chat })); + await streamFn( + { + prompt: "", + systemPrompt: "be brief", + messages: [ + { role: "system", content: [{ type: "text", text: "answer in English" }] }, + { + role: "user", + content: [ + { type: "text", text: "What is this?" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "It looks red." }, + { type: "text", text: "A red square." }, + ], + }, + { role: "user", content: "Thanks" }, + ], + } as any, + model, + new AbortController().signal, + () => {} + ); + + expect(chat).toHaveBeenCalledTimes(1); + expect(chat.mock.calls[0]![0].messages).toEqual([ + { role: "system", content: "be brief\n\nanswer in English" }, + { role: "user", content: "What is this?", images: ["AAAA"] }, + { role: "assistant", content: "A red square.", thinking: "It looks red." }, + { role: "user", content: "Thanks" }, + ]); + }); + + it("keeps a tool exchange in the history whole: the call and the result it answers", async () => { + const chat = vi.fn().mockResolvedValue(makeFakeStream(["ok"])); + const streamFn = createOllamaTextGenerationStream(vi.fn().mockResolvedValue({ chat })); + await streamFn( + { + prompt: "", + messages: [ + { role: "user", content: [{ type: "text", text: "weather?" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "weather", input: { city: "Paris" } }], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "text", text: "sunny" }], + }, + ], + }, + { role: "user", content: "and tomorrow?" }, + ], + } as any, + model, + new AbortController().signal, + () => {} + ); + expect(chat.mock.calls[0]![0].messages).toEqual([ + { role: "user", content: "weather?" }, + { + role: "assistant", + content: "", + tool_calls: [{ function: { name: "weather", arguments: { city: "Paris" } } }], + }, + { role: "tool", content: "sunny", tool_name: "weather" }, + { role: "user", content: "and tomorrow?" }, + ]); + }); + it("sends the model's effort as think, and nothing for a model with no effort set", async () => { + const chat = vi.fn().mockResolvedValue(makeFakeStream(["ok"])); + const streamFn = createOllamaTextGenerationStream(vi.fn().mockResolvedValue({ chat })); + const qwen = (effort?: string): any => ({ + model_id: "ollama:qwen3", + provider: "OLLAMA", + provider_config: { model_name: "qwen3:8b" }, + ...(effort === undefined ? {} : { effort }), + }); + await streamFn({ prompt: "hi" } as any, qwen("high"), new AbortController().signal, () => {}); + await streamFn({ prompt: "hi" } as any, qwen("none"), new AbortController().signal, () => {}); + await streamFn({ prompt: "hi" } as any, qwen(), new AbortController().signal, () => {}); + expect(chat.mock.calls.map((call) => call[0].think)).toEqual([true, false, undefined]); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/Ollama_ToolCalling.test.ts b/packages/test/src/test/ai-provider-api/Ollama_ToolCalling.test.ts new file mode 100644 index 0000000000..1c1b9a727f --- /dev/null +++ b/packages/test/src/test/ai-provider-api/Ollama_ToolCalling.test.ts @@ -0,0 +1,441 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { ollamaEffortPolicy } from "@workglow/ollama/ai"; +import { readFileSync } from "node:fs"; +import { describe, expect, it, vi } from "vitest"; +import { ollamaThinkParam } from "../../../../../providers/ollama/src/ai/common/Ollama_EffortPolicy"; +import { toOllamaMessages } from "../../../../../providers/ollama/src/ai/common/Ollama_Messages"; +import { createOllamaToolCallingStream } from "../../../../../providers/ollama/src/ai/common/Ollama_ToolCalling"; + +const model = (name: string, effort?: string): any => ({ + provider: "OLLAMA", + provider_config: { model_name: name }, + ...(effort === undefined ? {} : { effort }), +}); + +describe("toOllamaMessages", () => { + it("refuses audio rather than sending the turn without it", () => { + const audio = { type: "audio", mimeType: "audio/wav", data: "AAAA" }; + expect(() => toOllamaMessages({ prompt: [audio], tools: [] } as never)).toThrow(/audio/); + expect(() => + toOllamaMessages({ + prompt: "", + tools: [], + messages: [{ role: "user", content: [{ type: "text", text: "hear this" }, audio] }], + } as never) + ).toThrow(/audio/); + }); + + it("keeps tool calls, tool names, thinking and images across a turn", () => { + const input = { + systemPrompt: "be brief", + prompt: "weather?", + tools: [], + messages: [ + { + role: "user", + content: [ + { type: "text", text: "weather?" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "need the tool" }, + { type: "text", text: "Checking." }, + { type: "tool_use", id: "c1", name: "weather", input: { city: "Paris" } }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "sunny" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + ], + }, + ], + }, + ], + } as any; + expect(toOllamaMessages(input)).toEqual([ + { role: "system", content: "be brief" }, + { role: "user", content: "weather?", images: ["AAAA"] }, + { + role: "assistant", + content: "Checking.", + thinking: "need the tool", + tool_calls: [{ function: { name: "weather", arguments: { city: "Paris" } } }], + }, + { + role: "tool", + content: "sunny\n[image 1] follows in the next message", + tool_name: "weather", + }, + { + role: "user", + content: "Images returned by the tool calls above, in order: [image 1]", + images: ["BBBB"], + }, + ]); + }); + + it("marks each moved image where it stood, numbered across the turn's results", () => { + const image = (data: string) => ({ type: "image", mimeType: "image/png", data }); + const out = toOllamaMessages({ + prompt: "", + tools: [], + messages: [ + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "before:" }, + image("AAAA"), + { type: "text", text: "after:" }, + image("BBBB"), + ], + }, + { type: "tool_result", tool_use_id: "c2", content: [image("CCCC")] }, + ], + }, + ], + } as never); + expect(out.map((message) => message.content)).toEqual([ + "", + "before:\n[image 1] follows in the next message\nafter:\n[image 2] follows in the next message", + "[image 3] follows in the next message", + "Images returned by the tool calls above, in order: [image 1], [image 2], [image 3]", + ]); + expect(out.at(-1)!.images).toEqual(["AAAA", "BBBB", "CCCC"]); + }); + + it("holds tool images until every separate tool message of the turn is out", () => { + const out = toOllamaMessages({ + prompt: "", + tools: [], + messages: [ + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "c2", content: [{ type: "text", text: "ok" }] }, + ], + }, + ], + } as any); + expect(out.map((m) => m.role)).toEqual(["assistant", "tool", "tool", "user"]); + expect(out[3]!.images).toEqual(["AAAA"]); + }); + + it("keeps a non-text block a tool returned, as JSON", () => { + const nested = { type: "tool_use", id: "x", name: "inner", input: { a: 1 } }; + const out = toOllamaMessages({ + prompt: "", + tools: [], + messages: [ + { role: "assistant", content: [{ type: "tool_use", id: "c1", name: "t", input: {} }] }, + { role: "tool", content: [{ type: "tool_result", tool_use_id: "c1", content: [nested] }] }, + ], + } as any); + expect(out[1]).toEqual({ role: "tool", content: JSON.stringify(nested), tool_name: "t" }); + }); + + it("lifts system turns in the history into the system prompt", () => { + const out = toOllamaMessages({ + systemPrompt: "be brief", + prompt: "", + tools: [], + messages: [ + { role: "system", content: [{ type: "text", text: "answer in French" }] }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ], + } as any); + expect(out).toEqual([ + { role: "system", content: "be brief\n\nanswer in French" }, + { role: "user", content: "hi" }, + ]); + }); + + it("sends no prompt for a history of system turns alone", () => { + const out = toOllamaMessages({ + prompt: "stale", + tools: [], + messages: [{ role: "system", content: [{ type: "text", text: "rule" }] }], + } as any); + expect(out).toEqual([{ role: "system", content: "rule" }]); + }); + + it("prefers messages over prompt when both are present", () => { + const out = toOllamaMessages({ + prompt: "first", + tools: [], + messages: [ + { role: "user", content: [{ type: "text", text: "first" }] }, + { role: "assistant", content: [{ type: "text", text: "second" }] }, + ], + } as any); + expect(out).toEqual([ + { role: "user", content: "first" }, + { role: "assistant", content: "second" }, + ]); + }); + + it("uses a string prompt when there are no messages", () => { + expect(toOllamaMessages({ systemPrompt: "s", prompt: "hi", tools: [] } as any)).toEqual([ + { role: "system", content: "s" }, + { role: "user", content: "hi" }, + ]); + expect(toOllamaMessages({ prompt: "hi", tools: [], messages: [] } as any)).toEqual([ + { role: "user", content: "hi" }, + ]); + }); + + it("joins an array prompt's text with newlines and keeps its images", () => { + const out = toOllamaMessages({ + prompt: [ + "one", + { type: "text", text: "two" }, + { type: "image", mimeType: "image/png", data: "IMG" }, + ], + tools: [], + } as any); + expect(out).toEqual([{ role: "user", content: "one\ntwo", images: ["IMG"] }]); + }); + + it("gives an assistant turn holding only a tool call empty content", () => { + const out = toOllamaMessages({ + prompt: "x", + tools: [], + messages: [ + { role: "user", content: [{ type: "text", text: "x" }] }, + { role: "assistant", content: [{ type: "tool_use", id: "a", name: "t", input: {} }] }, + ], + } as any); + expect(out[1]).toEqual({ + role: "assistant", + content: "", + tool_calls: [{ function: { name: "t", arguments: {} } }], + }); + }); + + it("omits tool_name for a result whose call id is unknown", () => { + const out = toOllamaMessages({ + prompt: "x", + tools: [], + messages: [ + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "missing", + content: [{ type: "text", text: "r" }], + }, + ], + }, + ], + } as any); + expect(out).toEqual([{ role: "tool", content: "r" }]); + }); + + it("prefixes an error result's content with Error:", () => { + const out = toOllamaMessages({ + prompt: "x", + tools: [], + messages: [ + { role: "assistant", content: [{ type: "tool_use", id: "a", name: "t", input: {} }] }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "a", + is_error: true, + content: [{ type: "text", text: "boom" }], + }, + ], + }, + ], + } as any); + expect(out[1]).toEqual({ role: "tool", content: "Error: boom", tool_name: "t" }); + }); + + it("resolves a repeated call id against the most recent preceding call", () => { + const result = (text: string): any => ({ + role: "tool", + content: [{ type: "tool_result", tool_use_id: "call_0", content: [{ type: "text", text }] }], + }); + const out = toOllamaMessages({ + prompt: "x", + tools: [], + messages: [ + { role: "assistant", content: [{ type: "tool_use", id: "call_0", name: "a", input: {} }] }, + result("ra"), + { role: "assistant", content: [{ type: "tool_use", id: "call_0", name: "b", input: {} }] }, + result("rb"), + ], + } as any); + expect(out.filter((m) => m.role === "tool").map((m) => m.tool_name)).toEqual(["a", "b"]); + }); +}); + +describe("ollamaThinkParam", () => { + it("maps gpt-oss effort onto low/medium/high", () => { + expect(ollamaThinkParam(model("gpt-oss:20b", "high"))).toEqual({ think: "high" }); + expect(ollamaThinkParam(model("gpt-oss:20b", "low"))).toEqual({ think: "low" }); + expect(ollamaThinkParam(model("gpt-oss:20b", "medium"))).toEqual({ think: "medium" }); + expect(ollamaThinkParam(model("gpt-oss:20b", "extra"))).toEqual({ think: "high" }); + expect(ollamaThinkParam(model("gpt-oss:20b", "ultra"))).toEqual({ think: "high" }); + }); + + it("sends nothing for gpt-oss at effort none, which its policy does not list", () => { + expect(ollamaEffortPolicy(model("gpt-oss:20b")).supported).not.toContain("none"); + expect(ollamaThinkParam(model("gpt-oss:20b", "none"))).toEqual({}); + }); + + it("maps boolean-thinking families onto true/false", () => { + expect(ollamaThinkParam(model("qwen3:8b", "none"))).toEqual({ think: false }); + expect(ollamaThinkParam(model("qwen3:8b", "medium"))).toEqual({ think: true }); + expect(ollamaThinkParam(model("deepseek-r1:7b", "high"))).toEqual({ think: true }); + }); + + it("sends nothing for qwen3 variants that cannot think", () => { + expect(ollamaThinkParam(model("qwen3-coder:30b", "high"))).toEqual({}); + expect(ollamaThinkParam(model("qwen3-embedding:8b", "high"))).toEqual({}); + expect(ollamaThinkParam(model("qwen3:30b-a3b-instruct-2507", "high"))).toEqual({}); + expect(ollamaEffortPolicy(model("qwen3-coder:30b")).supported).toEqual([]); + expect(ollamaEffortPolicy(model("qwen3:30b-a3b-thinking-2507")).supported).toContain("high"); + expect(ollamaThinkParam(model("qwen3:8b", "high"))).toEqual({ think: true }); + }); + + it("reads the model name the way the policy does, trimmed", () => { + expect(ollamaThinkParam(model(" gpt-oss:20b ", "high"))).toEqual({ think: "high" }); + }); + + it("sends nothing for a model that cannot think, or when no effort is set", () => { + expect(ollamaThinkParam(model("llama3.2", "high"))).toEqual({}); + expect(ollamaThinkParam(model("qwen3:8b"))).toEqual({}); + expect(ollamaThinkParam(undefined)).toEqual({}); + }); +}); + +function chunks(items: unknown[]): any { + async function* gen(): AsyncGenerator { + for (const item of items) yield item; + } + return Object.assign(gen(), { abort() {} }); +} + +async function runStream(m: any): Promise<{ events: any[]; request: any }> { + let request: any; + const chat = vi.fn(async (req: any) => { + request = req; + return chunks([ + { message: { thinking: "hm", content: "" } }, + { message: { content: "ok" } }, + { + message: { + content: "", + tool_calls: [{ function: { name: "weather", arguments: { city: "Paris" } } }], + }, + done: true, + }, + ]); + }); + const run = createOllamaToolCallingStream(async () => ({ chat }), toOllamaMessages); + const events: any[] = []; + await run( + { + prompt: "weather?", + tools: [ + { + name: "weather", + description: "d", + inputSchema: { type: "object", properties: { city: { type: "string" } } }, + }, + ], + } as any, + m, + new AbortController().signal, + (e: any) => events.push(e) + ); + return { events, request }; +} + +describe("createOllamaToolCallingStream", () => { + it("streams thinking on the reasoning port, text on text, and sends think for a thinking model", async () => { + const { events, request } = await runStream(model("qwen3:8b", "medium")); + expect(events).toContainEqual({ type: "text-delta", port: "reasoning", textDelta: "hm" }); + expect(events).toContainEqual({ type: "text-delta", port: "text", textDelta: "ok" }); + expect(events).toContainEqual({ + type: "object-delta", + port: "toolCalls", + objectDelta: [{ id: "call_0", name: "weather", input: { city: "Paris" } }], + }); + expect(request.think).toBe(true); + expect(request.messages).toEqual([{ role: "user", content: "weather?" }]); + }); + + it("omits think for a model that cannot think", async () => { + const { request } = await runStream(model("llama3.2", "medium")); + expect("think" in request).toBe(false); + }); +}); + +describe("browser build", () => { + it("passes toOllamaMessages and no longer carries its own prompt-only converter", () => { + const source = readFileSync( + new URL( + "../../../../../providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts", + import.meta.url + ), + "utf8" + ); + expect(source).not.toContain("buildBrowserToolCallingMessages"); + expect(source).toContain("toOllamaMessages"); + }); + + it("node build no longer uses the text-flat converter", () => { + const source = readFileSync( + new URL("../../../../../providers/ollama/src/ai/common/Ollama_JobRunFns.ts", import.meta.url), + "utf8" + ); + expect(source).not.toContain("toTextFlatMessages"); + expect(source).toContain("toOllamaMessages"); + }); +}); diff --git a/packages/test/src/test/ai-provider-cactus/Cactus_ToolCalling.test.ts b/packages/test/src/test/ai-provider-cactus/Cactus_ToolCalling.test.ts index 5b18207816..a8f616527f 100644 --- a/packages/test/src/test/ai-provider-cactus/Cactus_ToolCalling.test.ts +++ b/packages/test/src/test/ai-provider-cactus/Cactus_ToolCalling.test.ts @@ -273,9 +273,9 @@ describe("Cactus_ToolCalling v3 reasoning on the text port", () => { expect(text).toContain("Paris is sunny."); }); - it("releases an unterminated rather than swallowing the generation", async () => { - // Cut off at the token limit: there is no closing tag, and the parser - // deliberately leaves such a block alone, so the stream must too. + it("keeps an unterminated off the text port", async () => { + // Cut off at the token limit: there is no closing tag, and what follows the + // opening one is still reasoning, so it goes to the reasoning port. const { text } = await runToolCalling({ run: () => "unused", run_stream: async (_q, _t, cb) => { @@ -283,7 +283,7 @@ describe("Cactus_ToolCalling v3 reasoning on the text port", () => { return "reasoning that never closes"; }, }); - expect(text).toBe("reasoning that never closes"); + expect(text).toBe(""); }); it("does not hold back text that merely starts like a tag", async () => { @@ -580,3 +580,144 @@ describe("Cactus_ToolCalling tool-call markup on the text port", () => { expect(text).toBe("ends mid-tag { + const toolUse = { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "lookup_weather", input: { city: "Paris" } }], + }; + const toolResult = { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "text", text: "sunny" }], + }, + ], + }; + const user = (text: string) => ({ role: "user", content: [{ type: "text", text }] }); + + async function queryFor(extra: Record): Promise { + let seen = ""; + cactusEngines.set("needle-26m", { + run: (q: string) => { + seen = q; + return "[]"; + }, + } as never); + await Cactus_ToolCalling( + { prompt: "first", tools, toolChoice: "auto", ...extra } as never, + model as never, + new AbortController().signal, + () => {} + ); + return seen; + } + + it("sends the most recent user message after the prompt, not the prompt twice", async () => { + const query = await queryFor({ + messages: [user("first"), toolUse, toolResult, user("second")], + }); + expect(query).toBe("first\n\nsecond"); + }); + + it("sends the prompt alone while it is the latest user message", async () => { + expect(await queryFor({ messages: [user("first"), toolUse, toolResult] })).toBe("first"); + }); + + it("leaves an earlier request in the chat history out of the query", async () => { + const query = await queryFor({ + prompt: "and in Rome?", + messages: [ + user("weather in Paris?"), + { role: "assistant", content: [{ type: "text", text: "sunny" }] }, + user("and in Rome?"), + ], + }); + expect(query).toBe("and in Rome?"); + }); + + it("keeps the question a review prompt asks the model to check against", async () => { + const query = await queryFor({ + prompt: "weather in Paris?", + messages: [ + user("weather in Paris?"), + toolUse, + toolResult, + { role: "assistant", content: [{ type: "text", text: "sunny" }] }, + user("Check your answer against what you were asked."), + ], + }); + expect(query).toBe("weather in Paris?\n\nCheck your answer against what you were asked."); + }); + + it("uses the prompt when there are no messages", async () => { + expect(await queryFor({})).toBe("first"); + expect(await queryFor({ messages: [] })).toBe("first"); + }); + + it("prepends the system prompt and a blank line", async () => { + const query = await queryFor({ + systemPrompt: "rules", + messages: [user("first"), toolUse, toolResult, user("second")], + }); + expect(query).toBe("rules\n\nfirst\n\nsecond"); + expect(await queryFor({ systemPrompt: "rules" })).toBe("rules\n\nfirst"); + }); +}); + +describe("Cactus_ToolCalling emits text on the reasoning port", () => { + const reasoningOf = (events: StreamEvent[]): string => + events + .filter((e) => e.type === "text-delta" && e.port === "reasoning") + .map((e) => (e as { textDelta: string }).textDelta) + .join(""); + + it("sends the think contents of a one-shot generation to reasoning", async () => { + const { names, events, text } = await runToolCalling({ + run: () => + `pick weather` + + `[{"name":"lookup_weather","arguments":{"city":"Paris"}}]`, + }); + expect(names).toEqual(["lookup_weather"]); + expect(events).toContainEqual({ + type: "text-delta", + port: "reasoning", + textDelta: "pick weather", + }); + expect(text).toBe(""); + }); + + it("streams think contents as they arrive, whatever the tag boundaries", async () => { + const { events, text } = await runToolCalling({ + run: () => "unused", + run_stream: async (_q, _t, cb) => { + for (const piece of ["pick ", "wea", "therDone."]) cb(piece); + return "pick weatherDone."; + }, + }); + expect(reasoningOf(events)).toBe("pick weather"); + expect(text).toBe("Done."); + }); + + it("sends an unterminated think block to reasoning, not text", async () => { + const { events, text } = await runToolCalling({ + run: () => "unused", + run_stream: async (_q, _t, cb) => { + cb("reasoning that never "); + cb("closesreasoning that never closes { + const { events } = await runToolCalling({ + run: () => `[{"name":"lookup_weather","arguments":{"city":"Paris"}}]`, + }); + expect(events.some((e) => e.type === "text-delta" && e.port === "reasoning")).toBe(false); + }); +}); diff --git a/packages/test/src/test/ai-provider-hft/HFT_ToolCalling.test.ts b/packages/test/src/test/ai-provider-hft/HFT_ToolCalling.test.ts index 84a29bc6f8..3d85182597 100644 --- a/packages/test/src/test/ai-provider-hft/HFT_ToolCalling.test.ts +++ b/packages/test/src/test/ai-provider-hft/HFT_ToolCalling.test.ts @@ -15,7 +15,11 @@ import { loadTransformersSDK, pipelines, } from "../../../../../providers/huggingface-transformers/src/ai/common/HFT_Pipeline"; -import { HFT_ToolCalling } from "../../../../../providers/huggingface-transformers/src/ai/common/HFT_ToolCalling"; +import { + buildHFTMessages, + createHftDeltaRouter, + HFT_ToolCalling, +} from "../../../../../providers/huggingface-transformers/src/ai/common/HFT_ToolCalling"; // "qwen" in the path selects the Hermes-style `` parser family. const model = { @@ -31,22 +35,27 @@ const PREAMBLE = "Let me check the weather."; * The open tag is split across pieces so the markup filter has to hold a * partial tag back rather than leak it onto the text port. */ -const PIECES: readonly string[] = [ +const TOOL_PIECES: readonly string[] = [ "Let me check ", "the weather.{"name": "get_weather", ', '"arguments": {"city": "Paris"}}', ]; -function makeFakePipeline(): any { +const RENDERED_PROMPT = "What is the weather?\n"; + +function makeFakePipeline( + pieces: readonly string[] = TOOL_PIECES, + rendered: string = RENDERED_PROMPT +): any { const tokenizer = Object.assign((text: string) => ({ input_ids: { dims: [1, text.length] } }), { all_special_ids: [] as number[], - apply_chat_template: () => "What is the weather?\n", + apply_chat_template: () => rendered, decode: () => "", }); const hfModel = { generate: async (args: Record) => { - for (const piece of PIECES) args.streamer?.callback_function?.(piece); + for (const piece of pieces) args.streamer?.callback_function?.(piece); return { dims: [1, args.input_ids.dims[1]] }; }, }; @@ -133,3 +142,207 @@ describe("HFT_ToolCalling stream output", () => { ]); }); }); + +function portText(events: StreamEvent[], port: string): string { + return events + .filter((e) => e.type === "text-delta" && e.port === port) + .map((e) => (e as { textDelta: string }).textDelta) + .join(""); +} + +describe("HFT_ToolCalling thinking", () => { + it("routes a block to the reasoning port and keeps it out of the answer", async () => { + pipelines.set( + cacheKey, + makeFakePipeline(["I should check", " the map.It is sunny."]) + ); + const { events, output } = await run(undefined); + + expect(portText(events, "reasoning")).toBe("I should check the map."); + expect(portText(events, "text")).toBe("It is sunny."); + expect(output.text).toBe("It is sunny."); + expect(output.toolCalls).toEqual([]); + expectNoFinishAccumulation(events); + }); + + it("treats generation as already inside when the prompt opens the tag", async () => { + pipelines.set( + cacheKey, + makeFakePipeline(["planning\n\nDone."], "\n\n") + ); + const { events } = await run(undefined); + + expect(portText(events, "reasoning")).toBe("planning"); + expect(portText(events, "text")).toBe("\n\nDone."); + }); + + it("parses the tool call after the thinking, and ignores one quoted inside it", async () => { + pipelines.set( + cacheKey, + makeFakePipeline([ + 'Maybe {"name": "get_weather", "arguments": {"city": "Rome"}}?', + "", + '{"name": "get_weather", "arguments": {"city": "Paris"}}', + ]) + ); + const { events, output } = await run(undefined); + + expect(portText(events, "text")).toBe(""); + expect(output.toolCalls).toEqual([ + { id: expect.any(String), name: "get_weather", input: { city: "Paris" } }, + ]); + }); +}); + +describe("HFT_ToolCalling chat history", () => { + it("hands earlier reasoning to the template when no tool has run yet", async () => { + const pipeline = makeFakePipeline(["It is still sunny."]); + const rendered: unknown[] = []; + pipeline.tokenizer.apply_chat_template = (messages: unknown) => { + rendered.push(messages); + return RENDERED_PROMPT; + }; + pipelines.set(cacheKey, pipeline); + + await HFT_ToolCalling( + { + ...input, + messages: [ + { role: "user", content: [{ type: "text", text: "Weather?" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "No tool needed." }, + { type: "text", text: "Sunny." }, + ], + }, + { role: "user", content: [{ type: "text", text: "And now?" }] }, + ], + } as unknown as ToolCallingTaskInput, + model, + new AbortController().signal, + () => {} + ); + + expect(rendered[0]).toEqual([ + { role: "user", content: "Weather?" }, + { role: "assistant", content: "Sunny.", reasoning_content: "No tool needed." }, + { role: "user", content: "And now?" }, + ]); + }); +}); + +describe("createHftDeltaRouter", () => { + it("sends content to reasoning and the answer to text", () => { + const seen: Array<{ port: string; textDelta: string }> = []; + const router = createHftDeltaRouter((e) => seen.push(e), false); + router.feed("ab"); + router.flush(); + expect(seen).toEqual([ + { type: "text-delta", port: "reasoning", textDelta: "a" }, + { type: "text-delta", port: "text", textDelta: "b" }, + ]); + expect(router.answerText()).toBe("b"); + }); + + it("starts on reasoning when told the prompt opened the tag", () => { + const seen: Array<{ port: string; textDelta: string }> = []; + const router = createHftDeltaRouter((e) => seen.push(e), true); + router.feed("ab"); + router.flush(); + expect(seen.map((e) => [e.port, e.textDelta])).toEqual([ + ["reasoning", "a"], + ["text", "b"], + ]); + }); +}); + +describe("buildHFTMessages", () => { + const history = [ + { role: "user", content: [{ type: "text", text: "Weather in Paris?" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "Need the weather tool." }, + { type: "text", text: "Checking." }, + { type: "tool_use", id: "call_1", name: "get_weather", input: { city: "Paris" } }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "call_1", + content: [{ type: "text", text: "sunny" }], + }, + ], + }, + ] as never; + + it("replays reasoning as reasoning_content on the assistant turn", () => { + const out = buildHFTMessages(history, undefined, undefined, undefined); + expect(out[1]).toEqual({ + role: "assistant", + content: "Checking.", + reasoning_content: "Need the weather tool.", + tool_calls: [{ id: "call_1", name: "get_weather", arguments: { city: "Paris" } }], + }); + }); + + it("omits reasoning_content when the turn has no reasoning", () => { + const out = buildHFTMessages( + [{ role: "assistant", content: [{ type: "text", text: "Hi" }] }] as never, + undefined, + undefined, + undefined + ); + expect(out[0]).toEqual({ role: "assistant", content: "Hi" }); + }); + + it("names a tool message after the tool_use it answers", () => { + const out = buildHFTMessages(history, undefined, undefined, undefined); + expect(out[2]).toEqual({ + role: "tool", + content: "sunny", + tool_call_id: "call_1", + name: "get_weather", + }); + }); + + it("resolves a repeated id to the most recent tool_use", () => { + const out = buildHFTMessages( + [ + { role: "assistant", content: [{ type: "tool_use", id: "x", name: "first", input: {} }] }, + { role: "assistant", content: [{ type: "tool_use", id: "x", name: "second", input: {} }] }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "x", content: [{ type: "text", text: "r" }] }, + ], + }, + ] as never, + undefined, + undefined, + undefined + ); + expect(out[2]).toMatchObject({ role: "tool", name: "second" }); + }); + + it("leaves the name off a result with no matching tool_use", () => { + const out = buildHFTMessages( + [ + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "z", content: [{ type: "text", text: "r" }] }, + ], + }, + ] as never, + undefined, + undefined, + undefined + ); + expect(out[0]).toEqual({ role: "tool", content: "r", tool_call_id: "z" }); + }); +}); diff --git a/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCalling.test.ts b/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCalling.test.ts new file mode 100644 index 0000000000..ae651ea818 --- /dev/null +++ b/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCalling.test.ts @@ -0,0 +1,233 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from "vitest"; +import { + llamaCppEffortPolicy, + llamaCppThoughtBudget, + llamaCppThoughtBudgetOption, +} from "../../../../../providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy"; +import { + convertMessagesToChatHistory, + messagesToPureChatHistoryForPrefix, + routeLlamaResponseChunk, +} from "../../../../../providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling"; + +const IMAGE_MARKER = "[image/png image returned; this model reads text only]"; + +describe("convertMessagesToChatHistory", () => { + it("replays a reasoning block as an ended thought segment ahead of the calls", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "let me call it" }, + { type: "tool_use", id: "c1", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0]).toEqual({ + type: "segment", + segmentType: "thought", + text: "let me call it", + ended: true, + }); + expect(history[1].response[1].result).toBe(IMAGE_MARKER); + expect(history[1].response[1].result).not.toContain("AAAA"); + }); + + it("skips a reasoning block with no text", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "" }, + { type: "text", text: "hello" }, + ], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response).toEqual(["hello"]); + }); + + it("keeps the text of a mixed result and puts the image marker on its own line", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "shot", input: {} }], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "captured" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + ], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0].result).toBe(`captured\n${IMAGE_MARKER}`); + }); + + it("keeps each block of a result in the place it held", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { role: "assistant", content: [{ type: "tool_use", id: "c1", name: "shot", input: {} }] }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "before" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "after" }, + ], + }, + ], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0].result).toBe(`before\n${IMAGE_MARKER}\nafter`); + }); + + it("leaves an empty result empty instead of serializing the blocks", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "noop", input: {} }], + }, + { + role: "tool", + content: [{ type: "tool_result", tool_use_id: "c1", content: [] }], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0].result).toBe(""); + }); + + it("keeps a non-text block a tool returned, as JSON", () => { + const nested = { type: "tool_use", id: "x", name: "inner", input: { a: 1 } }; + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "noop", input: {} }], + }, + { + role: "tool", + content: [{ type: "tool_result", tool_use_id: "c1", content: [nested] }], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0].result).toBe(JSON.stringify(nested)); + }); + + it("renders the same thought segment for a checkpoint prefix", () => { + const history = messagesToPureChatHistoryForPrefix( + [{ role: "assistant", content: [{ type: "reasoning", text: "thinking" }] }] as any, + undefined + ); + expect(history[0].response[0]).toMatchObject({ type: "segment", segmentType: "thought" }); + }); +}); + +describe("routeLlamaResponseChunk", () => { + it("routes a thought segment to the reasoning port", () => { + expect( + routeLlamaResponseChunk({ type: "segment", segmentType: "thought", text: "hm" }) + ).toEqual({ port: "reasoning", text: "hm" }); + }); + + it("drops any other segment type", () => { + expect( + routeLlamaResponseChunk({ type: "segment", segmentType: "comment", text: "note" }) + ).toBeUndefined(); + }); + + it("routes a main-response chunk to the text port, untyped or typed text", () => { + for (const type of [undefined, "text"]) { + expect(routeLlamaResponseChunk({ type, text: "answer" })).toEqual({ + port: "text", + text: "answer", + }); + } + }); +}); + +describe("llamaCppThoughtBudget", () => { + const budget = (effort?: string): number | undefined => + llamaCppThoughtBudget(effort === undefined ? {} : ({ effort } as object)); + + it("is undefined when no effort is set", () => { + expect(budget()).toBeUndefined(); + }); + + it.each([ + ["none", 0], + ["low", 512], + ["medium", 1024], + ["high", 2048], + ["extra", 4096], + ["ultra", 8192], + ])("maps %s to %i thought tokens", (effort, tokens) => { + expect(budget(effort)).toBe(tokens); + }); + + it("is the prompt option text generation and chat send, empty when no effort is set", () => { + expect(llamaCppThoughtBudgetOption({ effort: "high" } as object)).toEqual({ + budgets: { thoughtTokens: 2048 }, + }); + expect(llamaCppThoughtBudgetOption({})).toEqual({}); + }); +}); + +describe("llamaCppEffortPolicy", () => { + it("offers every level on any local model", () => { + expect( + llamaCppEffortPolicy({ provider_config: { model_path: "models/x.gguf" } }).supported + ).toEqual(["none", "low", "medium", "high", "extra", "ultra"]); + }); +}); diff --git a/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCallingCheckpoint.test.ts b/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCallingCheckpoint.test.ts index a0c8569d74..a39f8af438 100644 --- a/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCallingCheckpoint.test.ts +++ b/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCallingCheckpoint.test.ts @@ -65,7 +65,11 @@ vi.mock("node-llama-cpp", () => ({ generateResponse( history: any[], options: { - readonly onTextChunk: (chunk: string) => void; + readonly onResponseChunk: (chunk: { + type: undefined; + segmentType: undefined; + text: string; + }) => void; readonly signal: AbortSignal; } ): Promise<{ @@ -84,7 +88,7 @@ vi.mock("node-llama-cpp", () => ({ return Promise.reject(options.signal.reason); } advanceFakeSequence(this.sequence, TURN_TOKENS); - options.onTextChunk("calling"); + options.onResponseChunk({ type: undefined, segmentType: undefined, text: "calling" }); const cleanHistory = [ ...history, { diff --git a/packages/test/src/test/ai-provider/ThinkTagFilter.test.ts b/packages/test/src/test/ai-provider/ThinkTagFilter.test.ts new file mode 100644 index 0000000000..80736b01b2 --- /dev/null +++ b/packages/test/src/test/ai-provider/ThinkTagFilter.test.ts @@ -0,0 +1,119 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { createThinkTagFilter } from "@workglow/ai/provider-utils"; +import { describe, expect, it } from "vitest"; + +type Part = { readonly port: "text" | "reasoning"; readonly text: string }; + +/** Merges adjacent same-port parts so assertions do not depend on chunking. */ +function merged(parts: ReadonlyArray): Part[] { + const out: Part[] = []; + for (const part of parts) { + if (part.text === "") continue; + const last = out[out.length - 1]; + if (last && last.port === part.port) { + out[out.length - 1] = { port: last.port, text: last.text + part.text }; + } else { + out.push(part); + } + } + return out; +} + +describe("createThinkTagFilter", () => { + it("routes a think block to reasoning and the rest to text, across split tags", () => { + const f = createThinkTagFilter(); + const parts = [ + ...f.push("plan"), + ...f.push(" more\n\nAnswer"), + ...f.flush(), + ]; + expect(merged(parts)).toEqual([ + { port: "reasoning", text: "plan more" }, + { port: "text", text: "\n\nAnswer" }, + ]); + expect(parts.filter((p) => p.port === "reasoning").map((p) => p.text)).toEqual([ + "plan", + " more", + ]); + }); + + it("starts inside a think block when asked", () => { + const g = createThinkTagFilter({ startInside: true }); + expect(merged([...g.push("whyok"), ...g.flush()])).toEqual([ + { port: "reasoning", text: "why" }, + { port: "text", text: "ok" }, + ]); + }); + + it("passes text containing other tags through untouched", () => { + const h = createThinkTagFilter(); + expect(merged([...h.push("plain text"), ...h.flush()])).toEqual([ + { port: "text", text: "plain text" }, + ]); + }); + + it("releases a held partial open tag as text once it cannot be a tag", () => { + const f = createThinkTagFilter(); + const parts = [...f.push("a { + const outside = createThinkTagFilter(); + expect(merged([...outside.push("done { + const f = createThinkTagFilter(); + expect(merged([...f.push("cut off"), ...f.push(" here"), ...f.flush()])).toEqual([ + { port: "reasoning", text: "cut off here" }, + ]); + }); + + it("handles several blocks and a whole stream in one push", () => { + const f = createThinkTagFilter(); + expect(merged([...f.push("abcd"), ...f.flush()])).toEqual([ + { port: "reasoning", text: "a" }, + { port: "text", text: "b" }, + { port: "reasoning", text: "c" }, + { port: "text", text: "d" }, + ]); + }); + + it("gives the same result however the stream is chunked", () => { + const whole = "xwhy\nbecause\nanswer again end"; + const expected = merged( + (() => { + const f = createThinkTagFilter(); + return [...f.push(whole), ...f.flush()]; + })() + ); + for (let size = 1; size <= 8; size++) { + const f = createThinkTagFilter(); + const parts: Part[] = []; + for (let i = 0; i < whole.length; i += size) parts.push(...f.push(whole.slice(i, i + size))); + parts.push(...f.flush()); + expect(merged(parts)).toEqual(expected); + } + }); + + it("is usable again after a flush", () => { + const f = createThinkTagFilter(); + f.push("a"); + f.flush(); + expect(merged([...f.push("b"), ...f.flush()])).toEqual([{ port: "text", text: "b" }]); + }); +}); diff --git a/packages/test/src/test/ai-provider/WebBrowserProvider.test.ts b/packages/test/src/test/ai-provider/WebBrowserProvider.test.ts index 549bb60630..69b7d5faf7 100644 --- a/packages/test/src/test/ai-provider/WebBrowserProvider.test.ts +++ b/packages/test/src/test/ai-provider/WebBrowserProvider.test.ts @@ -1106,6 +1106,28 @@ describe("WebBrowser_ToolCalling session lifecycle", () => { restore(); } }); + + it("finishes with the full empty scaffold, leaving accumulation to the deltas", async () => { + const { factory } = makeFakeToolCallingModel(); + const restore = installLanguageModelGlobal(factory); + try { + const events: Array<{ type?: string; data?: unknown }> = []; + await WebBrowser_ToolCalling( + asTCI({ prompt: "do it", tools: [toolA, toolB] }), + undefined, + new AbortController().signal, + (e: unknown) => { + events.push(e as { type?: string; data?: unknown }); + }, + undefined, + { sessionId: sid } + ); + const finish = events.find((e) => e.type === "finish"); + expect(finish?.data).toEqual({ text: "", toolCalls: [] }); + } finally { + restore(); + } + }); }); // -------------------------------------------------------------------------- diff --git a/providers/cactus/src/ai/common/Cactus_ParseToolCalls.ts b/providers/cactus/src/ai/common/Cactus_ParseToolCalls.ts index 6b8bbb0ca2..72d1318fef 100644 --- a/providers/cactus/src/ai/common/Cactus_ParseToolCalls.ts +++ b/providers/cactus/src/ai/common/Cactus_ParseToolCalls.ts @@ -55,22 +55,33 @@ function danglingPrefixLength(text: string, tag: string): number { * would leave a host rendering the chain of thought beside the tool cards, and * a host logging `text` persisting it. * + * What the filter drops is handed to `onReasoning` as it arrives, so the + * caller can put it on the `reasoning` port instead of losing it. The tags + * themselves are not part of it. + * * Streaming makes this more than a `replace`: a tag can straddle a delta * boundary, so the filter holds back any trailing run that could still turn - * into `` and releases it once it cannot. + * into `` (or ``, inside a block) and releases it once it + * cannot. * * `flush` exists for the case the fence never closes. A generation cut short - * at the token limit has no ``, and `stripNeedleReasoning` deliberately - * leaves such a block alone — so the filter releases what it held rather than - * swallowing the whole generation. + * at the token limit has no ``: everything after the opening tag is + * still reasoning, so it goes to `onReasoning` — including a partial closing + * tag held back at the end — and none of it reaches the visible text. */ -export function createNeedleReasoningFilter(): NeedleTextFilter { +export function createNeedleReasoningFilter( + onReasoning?: (text: string) => void +): NeedleTextFilter { const OPEN = ""; const CLOSE = ""; - /** Text not yet classifiable: a partial tag, or a fence still open. */ + /** Text not yet classifiable: a partial tag. */ let held = ""; let inside = false; + const think = (text: string): void => { + if (text.length > 0) onReasoning?.(text); + }; + return { push(delta: string): string { held += delta; @@ -78,8 +89,13 @@ export function createNeedleReasoningFilter(): NeedleTextFilter { for (;;) { if (inside) { const close = held.indexOf(CLOSE); - // No closing tag yet: keep holding, in case the stream ends here. - if (close === -1) return visible; + if (close === -1) { + const dangling = danglingPrefixLength(held, CLOSE); + think(held.slice(0, held.length - dangling)); + held = held.slice(held.length - dangling); + return visible; + } + think(held.slice(0, close)); held = held.slice(close + CLOSE.length); inside = false; continue; @@ -92,15 +108,18 @@ export function createNeedleReasoningFilter(): NeedleTextFilter { return visible; } visible += held.slice(0, open); - // Keep the opening tag, so an unterminated block flushes intact. - held = held.slice(open); + held = held.slice(open + OPEN.length); inside = true; } }, flush(): string { const rest = held; held = ""; - inside = false; + if (inside) { + inside = false; + think(rest); + return ""; + } return rest; }, }; @@ -136,8 +155,9 @@ function isBareToolCallPayload(text: string): boolean { * * Three stages, in the order the parser reads the generation: * - * 1. {@link createNeedleReasoningFilter} drops `` blocks, so a fence the - * model only talked about counts for nothing below. + * 1. {@link createNeedleReasoningFilter} drops `` blocks, handing them to + * `onReasoning`, so a fence the model only talked about counts for nothing + * below. * 2. {@link createToolCallMarkupFilter} drops each `` block, holding * back a partial opening tag until it is disambiguated and suppressing an * unclosed block to the end — the parser recovers that tail as a payload. @@ -151,8 +171,10 @@ function isBareToolCallPayload(text: string): boolean { * Output that is nothing but whitespace is dropped at the end too: it is the * newline between `` and ``, not an answer. */ -export function createNeedleVisibleTextFilter(): NeedleTextFilter { - const reasoning = createNeedleReasoningFilter(); +export function createNeedleVisibleTextFilter( + onReasoning?: (text: string) => void +): NeedleTextFilter { + const reasoning = createNeedleReasoningFilter(onReasoning); /** A `` opened outside reasoning, so no bare payload is read. */ let sawFence = false; /** The end of the last routed delta, for an opening tag split across two. */ diff --git a/providers/cactus/src/ai/common/Cactus_ToolCallingCore.ts b/providers/cactus/src/ai/common/Cactus_ToolCallingCore.ts index 5af31e85b0..4d40720218 100644 --- a/providers/cactus/src/ai/common/Cactus_ToolCallingCore.ts +++ b/providers/cactus/src/ai/common/Cactus_ToolCallingCore.ts @@ -31,12 +31,38 @@ function buildToolsJson(tools: ReadonlyArray): string { ); } +/** + * What the engine is asked this round. Needle answers one query with no + * conversation of its own, and each round's input carries the whole history + * beside the turn's prompt, so the question is the latest user turn: sending + * the prompt every round would repeat it after a tool result. A later user turn + * within the same request can lean on the prompt (a review asking the model to + * check its answer against what it was asked), so the prompt goes ahead of it + * whenever the two differ. Earlier requests in the history are left out. + */ function promptText(input: ToolCallingTaskInput): string { - if (typeof input.prompt === "string") return input.prompt; - if (input.prompt) return extractMessageText(input.prompt); - if (input.messages && input.messages.length > 0) { - const last = input.messages[input.messages.length - 1]; - return extractMessageText(last.content); + const query = queryText(input); + return input.systemPrompt ? `${input.systemPrompt}\n\n${query}` : query; +} + +function queryText(input: ToolCallingTaskInput): string { + const prompt = + typeof input.prompt === "string" + ? input.prompt + : input.prompt + ? extractMessageText(input.prompt) + : ""; + const messages = input.messages; + if (messages && messages.length > 0) { + for (let i = messages.length - 1; i >= 0; i--) { + if (messages[i].role !== "user") continue; + const latest = extractMessageText(messages[i].content); + return prompt !== "" && latest !== prompt ? `${prompt}\n\n${latest}` : latest; + } + } + if (prompt !== "") return prompt; + if (messages && messages.length > 0) { + return extractMessageText(messages[messages.length - 1].content); } return ""; } @@ -76,7 +102,9 @@ export function createCactusToolCalling( // One definition of "what the caller sees as text", so the streamed and // one-shot paths cannot disagree about it. - const visible = createNeedleVisibleTextFilter(); + const visible = createNeedleVisibleTextFilter((chunk) => { + emit({ type: "text-delta", port: "reasoning", textDelta: chunk }); + }); const emitVisible = (chunk: string): void => { if (chunk.length > 0) emit({ type: "text-delta", port: "text", textDelta: chunk }); }; diff --git a/providers/chrome-ai/src/ai/common/WebBrowser_ToolCalling.ts b/providers/chrome-ai/src/ai/common/WebBrowser_ToolCalling.ts index de4ab0041d..aac1e4f55e 100644 --- a/providers/chrome-ai/src/ai/common/WebBrowser_ToolCalling.ts +++ b/providers/chrome-ai/src/ai/common/WebBrowser_ToolCalling.ts @@ -197,7 +197,7 @@ export const WebBrowser_ToolCalling: AiProviderRunFn< if (validated.length > 0) { emit({ type: "object-delta", port: "toolCalls", objectDelta: validated }); } - emit({ type: "finish", data: {} as ToolCallingTaskOutput }); + emit({ type: "finish", data: { text: "", toolCalls: [] } as ToolCallingTaskOutput }); } finally { try { session.destroy(); diff --git a/providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts b/providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts index a6de439372..2b36942ee1 100644 --- a/providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts +++ b/providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts @@ -16,6 +16,7 @@ import type { } from "@workglow/ai"; import { adaptParserResult, + createThinkTagFilter, createToolCallMarkupFilter, forcedToolSelection, getAvailableParsers, @@ -156,7 +157,9 @@ function resolveHFTToolsAndMessages( * Unlike `toTextFlatMessages` (which flattens everything to `{role, content}` * strings), this preserves tool_calls on assistant messages and the tool name * on tool-result messages — both required by HFT chat templates that support - * tool calling. + * tool calling. An assistant turn's reasoning rides as `reasoning_content`, + * which Qwen3-style templates read to keep the thinking of the current tool + * loop in the rendered history; templates that do not know it ignore it. */ export function buildHFTMessages( messages: ReadonlyArray | undefined, @@ -178,6 +181,9 @@ export function buildHFTMessages( out.push({ role: "user", content: extractPromptText(prompt) }); return out; } + // A tool_result names only the call it answers, so the tool's name is + // resolved from the tool_use that raised it (most recent id wins). + const toolNames = new Map(); for (const msg of messages) { if (msg.role === "user") { const text = msg.content @@ -201,7 +207,13 @@ export function buildHFTMessages( }; return { id: tu.id, name: tu.name, arguments: tu.input }; }); + for (const call of toolCalls) toolNames.set(call.id, call.name); + const reasoning = msg.content + .filter((b) => b.type === "reasoning") + .map((b) => (b as { type: "reasoning"; text: string }).text) + .join(""); const entry: Record = { role: "assistant", content: text }; + if (reasoning.length > 0) entry.reasoning_content = reasoning; if (toolCalls.length > 0) entry.tool_calls = toolCalls; out.push(entry); } else if (msg.role === "tool") { @@ -211,10 +223,12 @@ export function buildHFTMessages( .filter((inner) => inner.type === "text") .map((inner) => (inner as { type: "text"; text: string }).text) .join(""); + const name = toolNames.get(b.tool_use_id); out.push({ role: "tool", content: text, tool_call_id: b.tool_use_id, + ...(name !== undefined ? { name } : {}), }); } } @@ -267,6 +281,21 @@ function hasToolMessages(input: ToolCallingTaskInput): boolean { return input.messages?.some((m) => m.role === "tool") ?? false; } +/** + * Whether the history needs {@link buildHFTMessages} rather than flat text: + * tool messages, or an assistant turn carrying reasoning, which only the + * structured messages hand to the template as `reasoning_content`. + */ +function needsStructuredMessages(input: ToolCallingTaskInput): boolean { + return ( + hasToolMessages(input) || + (input.messages?.some( + (m) => m.role === "assistant" && m.content.some((b) => b.type === "reasoning" && b.text) + ) ?? + false) + ); +} + function buildPromptAndPrefix( tokenizer: TextGenerationPipeline["tokenizer"], input: ToolCallingTaskInput, @@ -274,9 +303,9 @@ function buildPromptAndPrefix( ): { prompt: string; responsePrefix: string | undefined } { let basePrompt: string; - if (hasToolMessages(input)) { - // Multi-turn with tool results: use structured messages so the tokenizer - // can format tool_calls and tool responses correctly. + if (needsStructuredMessages(input)) { + // Multi-turn with tool results or reasoning: use structured messages so the + // tokenizer can format tool_calls, tool responses and reasoning correctly. const messages = buildHFTMessages( input.messages, input.systemPrompt, @@ -290,7 +319,7 @@ function buildPromptAndPrefix( add_generation_prompt: true, }) as string; } else { - // Single-turn or no tool results: flat messages work fine. + // No tool results and no reasoning: flat messages lose nothing. const messages = toTextFlatMessages(input); const tools = resolveHFTToolsAndMessages(input, messages); basePrompt = tokenizer.apply_chat_template(messages, { @@ -374,6 +403,52 @@ function buildCheckpointPromptAndPrefix( }; } +// ============================================================================ +// Streamed delta routing +// ============================================================================ + +/** + * Routes decoded deltas to the ports: `` content goes to `reasoning`, + * and what remains goes through the `` markup filter to `text`. + * Thinking is separated first, so a tool-call tag quoted inside it is not + * mistaken for markup and neither is it recorded as the answer. + * + * `answerText` is the text-port content before the markup filter — what the + * tool-call parser must read, without the model's thinking. + */ +export function createHftDeltaRouter( + emit: (event: { type: "text-delta"; port: "text" | "reasoning"; textDelta: string }) => void, + startInside: boolean +): { + readonly feed: (delta: string) => void; + readonly flush: () => void; + readonly answerText: () => string; +} { + let answer = ""; + const markup = createToolCallMarkupFilter((text) => { + emit({ type: "text-delta", port: "text", textDelta: text }); + }); + const think = createThinkTagFilter({ startInside }); + const route = (parts: ReturnType): void => { + for (const part of parts) { + if (part.port === "reasoning") { + emit({ type: "text-delta", port: "reasoning", textDelta: part.text }); + } else { + answer += part.text; + markup.feed(part.text); + } + } + }; + return { + feed: (delta) => route(think.push(delta)), + flush: () => { + route(think.flush()); + markup.flush(); + }, + answerText: () => answer, + }; +} + // ============================================================================ // Provider run functions // ============================================================================ @@ -425,12 +500,15 @@ export const HFT_ToolCalling: AiProviderRunFn< } const { prompt, responsePrefix } = promptParts; - // Accumulate raw tokens for post-hoc tool-call parsing, and feed each - // delta through a markup filter that emits cleaned text-delta events. + // Accumulate raw tokens (the KV cache encodes them verbatim), and route + // each delta: thinking to `reasoning`, the rest through the tool-call + // markup filter to `text`. A template that opens `` in the prompt + // leaves the generation starting inside the block. let fullText = ""; - const filter = createToolCallMarkupFilter((text) => { - emit({ type: "text-delta", port: "text", textDelta: text }); - }); + const filter = createHftDeltaRouter( + (event) => emit(event), + prompt.trimEnd().endsWith("") + ); const streamer = createStreamingTextStreamer( generateText.tokenizer, @@ -517,7 +595,9 @@ export const HFT_ToolCalling: AiProviderRunFn< // Parse the accumulated text for tool calls using the model-family-aware parser. // For models that use a generation prefix, prepend it so the parser sees the // full markup pattern. - const parseableFullText = responsePrefix ? `${responsePrefix}${fullText}` : fullText; + // Thinking is left out: a call quoted inside it was never made. + const answerText = filter.answerText(); + const parseableFullText = responsePrefix ? `${responsePrefix}${answerText}` : answerText; // Only the calls are taken: the parser's cleaned text is the same text the // markup filter already streamed, and the stream is the one copy. const { toolCalls } = adaptParserResult( diff --git a/providers/llamacpp-server/src/ai/common/LlamaCppServer_Client.ts b/providers/llamacpp-server/src/ai/common/LlamaCppServer_Client.ts index 1e4af70e1e..97370e3a83 100644 --- a/providers/llamacpp-server/src/ai/common/LlamaCppServer_Client.ts +++ b/providers/llamacpp-server/src/ai/common/LlamaCppServer_Client.ts @@ -102,6 +102,8 @@ const noopRelease = async (): Promise => {}; /** One parsed delta from an OpenAI-compatible `/v1/chat/completions` stream. */ export interface IChatCompletionDelta { readonly contentDelta?: string; + /** The model's thinking, streamed as `delta.reasoning_content` by a reasoning model. */ + readonly reasoningDelta?: string | undefined; readonly toolCallDeltas?: ReadonlyArray<{ readonly index?: number; readonly id?: string; @@ -159,6 +161,7 @@ export async function* readChatCompletionDeltas( choices?: Array<{ delta?: { content?: string; + reasoning_content?: string; tool_calls?: IChatCompletionDelta["toolCallDeltas"]; }; finish_reason?: string; @@ -172,6 +175,7 @@ export async function* readChatCompletionDeltas( } const choice = chunk.choices?.[0]; const contentDelta = choice?.delta?.content; + const reasoningDelta = choice?.delta?.reasoning_content; const toolCallDeltas = choice?.delta?.tool_calls; const finishReason = choice?.finish_reason; // The usage-bearing chunk carries an empty `choices` array, so it must @@ -179,11 +183,12 @@ export async function* readChatCompletionDeltas( const usage = chunk.usage ?? undefined; if ( contentDelta !== undefined || + reasoningDelta !== undefined || toolCallDeltas !== undefined || finishReason || usage !== undefined ) { - yield { contentDelta, toolCallDeltas, finishReason, usage }; + yield { contentDelta, reasoningDelta, toolCallDeltas, finishReason, usage }; } } } diff --git a/providers/llamacpp-server/src/ai/common/LlamaCppServer_TextGeneration.ts b/providers/llamacpp-server/src/ai/common/LlamaCppServer_TextGeneration.ts index b0654e4539..97209468df 100644 --- a/providers/llamacpp-server/src/ai/common/LlamaCppServer_TextGeneration.ts +++ b/providers/llamacpp-server/src/ai/common/LlamaCppServer_TextGeneration.ts @@ -6,10 +6,13 @@ import type { AiProviderRunFn, + LooseChatMessage, + OpenAICompatMessage, TextGenerationTaskInput, TextGenerationTaskOutput, Usage, } from "@workglow/ai"; +import { normalizeChatMessages, toOpenAIMessages } from "@workglow/ai/worker"; import { createEstimatedOutputUsageReporter, localOnlyFetch, @@ -28,15 +31,7 @@ import { getLlamaCppServerModelName } from "./LlamaCppServer_ModelUtil"; type AcquireFn = typeof acquireBaseUrl; interface UnifiedTextGenerationInput extends TextGenerationTaskInput { - readonly messages?: ReadonlyArray<{ - readonly role: string; - readonly content: - | string - | ReadonlyArray< - | { readonly type: "text"; readonly text: string } - | { readonly type: "image_url"; readonly image_url: { readonly url: string } } - >; - }>; + readonly messages?: ReadonlyArray; readonly systemPrompt?: string; } @@ -62,11 +57,18 @@ export function createLlamaCppServerTextGenerationStream( const unified = input as UnifiedTextGenerationInput; const hasMessages = Array.isArray(unified.messages) && unified.messages.length > 0; + // llama-server speaks chat-completions, so history goes through the shared + // OpenAI converter: workglow image blocks become `image_url` parts (llava), + // and blocks the endpoint has no form for are dropped rather than sent. const messages = hasMessages - ? [ - ...(unified.systemPrompt ? [{ role: "system", content: unified.systemPrompt }] : []), - ...unified.messages!.map((m) => ({ role: m.role, content: m.content })), - ] + ? toOpenAIMessages({ + // The converter lifts system turns into the system prompt itself, and + // keeps a history of system turns alone from becoming an empty user turn. + messages: normalizeChatMessages(unified.messages!), + systemPrompt: unified.systemPrompt, + prompt: "", + tools: [], + } as never).map(collapseTextOnlyContent) : [{ role: "user", content: input.prompt }]; const body = JSON.stringify({ @@ -124,6 +126,20 @@ export function createLlamaCppServerTextGenerationStream( }; } +/** + * A user turn that is text alone goes out as a plain string, as it always has: + * only a turn carrying an image needs the parts array, and a server built + * without multimodal support is the one that may not accept one. + */ +function collapseTextOnlyContent(message: OpenAICompatMessage): OpenAICompatMessage { + const content = message.content; + if (!Array.isArray(content) || !content.every((part) => part.type === "text")) return message; + return { + ...message, + content: content.map((part) => (typeof part.text === "string" ? part.text : "")).join(""), + }; +} + function promptTextFromMessages(messages: ReadonlyArray<{ readonly content: unknown }>): string { const parts: string[] = []; for (const message of messages) { diff --git a/providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts b/providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts index a0707f5fc0..89cbe65367 100644 --- a/providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts +++ b/providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts @@ -23,7 +23,7 @@ import { filterValidToolCalls, isAllowedToolName, sanitizeToolArgs, - toTextFlatMessages, + toOpenAIMessages, } from "@workglow/ai/worker"; import type { PartialJsonStream } from "@workglow/util/worker"; import { createPartialJsonStream } from "@workglow/util/worker"; @@ -62,13 +62,28 @@ function mapTools(tools: readonly ToolDefinition[]) { })); } +/** The text of a message's content, string or parts, for the prompt-size estimate. */ +function promptText(content: unknown): string { + if (typeof content === "string") return content; + if (!Array.isArray(content)) return ""; + return content + .map((part: { type?: unknown; text?: unknown } | null) => + part?.type === "text" && typeof part.text === "string" ? part.text : "" + ) + .join("\n"); +} + export function createLlamaCppServerToolCallingStream( opts: ILlamaCppServerProviderOptions, acquire: AcquireFn = acquireBaseUrl ): AiProviderRunFn { return async (input, model, signal, emit) => { signal?.throwIfAborted?.(); - const messages = toTextFlatMessages(input); + // llama.cpp's server reads `reasoning_content` back on assistant messages. + const messages = toOpenAIMessages(input, { + replayReasoning: true, + toolImagesInUserMessage: true, + }); const tools = input.toolChoice === "none" ? undefined : mapTools(input.tools); const body = JSON.stringify({ model: getLlamaCppServerModelName(model), @@ -84,7 +99,7 @@ export function createLlamaCppServerToolCallingStream( const provisionalUsage = createEstimatedOutputUsageReporter(emit); provisionalUsage.onPrompt( messages - .map((m) => (typeof m.content === "string" ? m.content : "")) + .map((m) => promptText(m.content)) .filter(Boolean) .join("\n") ); @@ -113,6 +128,10 @@ export function createLlamaCppServerToolCallingStream( for await (const delta of readChatCompletionDeltas(response, signal)) { if (delta.done) break; usage = mapOpenAIChatUsage(delta.usage) ?? usage; + if (delta.reasoningDelta) { + provisionalUsage.onText(delta.reasoningDelta); + emit({ type: "text-delta", port: "reasoning", textDelta: delta.reasoningDelta }); + } if (delta.contentDelta) { provisionalUsage.onText(delta.contentDelta); emit({ type: "text-delta", port: "text", textDelta: delta.contentDelta }); diff --git a/providers/node-llama-cpp/src/ai/LlamaCppProvider.ts b/providers/node-llama-cpp/src/ai/LlamaCppProvider.ts index 5b10bbdb9d..39f742411c 100644 --- a/providers/node-llama-cpp/src/ai/LlamaCppProvider.ts +++ b/providers/node-llama-cpp/src/ai/LlamaCppProvider.ts @@ -9,6 +9,7 @@ import type { AiProviderRunFnRegistration, Capability, ModelConfig, + ModelEffortPolicy, ModelRecord, } from "@workglow/ai/worker"; import { AiProvider } from "@workglow/ai/worker"; @@ -17,6 +18,7 @@ import { llamaCppWorkerRunFnSpecs, } from "./common/LlamaCpp_Capabilities"; import { LOCAL_LLAMACPP } from "./common/LlamaCpp_Constants"; +import { llamaCppEffortPolicy } from "./common/LlamaCpp_EffortPolicy"; import type { LlamaCppModelConfig } from "./common/LlamaCpp_ModelSchema"; import { deleteLlamaCppSession } from "./common/LlamaCpp_Runtime"; @@ -39,6 +41,10 @@ export class LlamaCppProvider extends AiProvider { return inferLlamaCppCapabilities(model); } + override effortPolicy(model: LlamaCppModelConfig): ModelEffortPolicy | undefined { + return llamaCppEffortPolicy(model); + } + protected override workerRunFnSpecs(): readonly { serves: readonly Capability[] }[] { return llamaCppWorkerRunFnSpecs(); } diff --git a/providers/node-llama-cpp/src/ai/LlamaCppQueuedProvider.ts b/providers/node-llama-cpp/src/ai/LlamaCppQueuedProvider.ts index 03bdad6819..3cc8ee9015 100644 --- a/providers/node-llama-cpp/src/ai/LlamaCppQueuedProvider.ts +++ b/providers/node-llama-cpp/src/ai/LlamaCppQueuedProvider.ts @@ -9,6 +9,7 @@ import type { AiProviderRunFnRegistration, Capability, ModelConfig, + ModelEffortPolicy, ModelPricing, ModelRecord, } from "@workglow/ai"; @@ -24,6 +25,7 @@ import { llamaCppWorkerRunFnSpecs, } from "./common/LlamaCpp_Capabilities"; import { LOCAL_LLAMACPP } from "./common/LlamaCpp_Constants"; +import { llamaCppEffortPolicy } from "./common/LlamaCpp_EffortPolicy"; import type { LlamaCppModelConfig } from "./common/LlamaCpp_ModelSchema"; import { deleteLlamaCppSession } from "./common/LlamaCpp_Runtime"; @@ -58,6 +60,10 @@ export class LlamaCppQueuedProvider extends QueuedAiProvider { queue.push(chunk); provisionalUsage.onText(chunk); diff --git a/providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy.ts b/providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy.ts new file mode 100644 index 0000000000..98c12901d1 --- /dev/null +++ b/providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy.ts @@ -0,0 +1,45 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ModelEffort, ModelEffortPolicyFn } from "@workglow/ai/worker"; +import { EFFORT_POLICY_ALL, makeEffortPolicy, resolveEnabledEffort } from "@workglow/ai/worker"; + +/** + * A local GGUF's ability to think is not knowable from its path, so every + * level is offered. A thought budget on a model that does not think is ignored + * by the chat wrapper, which makes offering the dial harmless where it does + * not apply. + */ +export const llamaCppEffortPolicy: ModelEffortPolicyFn = makeEffortPolicy({ + rules: [], + fallback: EFFORT_POLICY_ALL, +}); + +const THOUGHT_TOKENS: Readonly> = { + none: 0, + low: 512, + medium: 1024, + high: 2048, + extra: 4096, + ultra: 8192, +}; + +/** + * The `budgets.thoughtTokens` a caller's effort asks for, or `undefined` when + * no effort is set so the SDK keeps its own default (a share of the context). + */ +export function llamaCppThoughtBudget(model: object | undefined): number | undefined { + const effort = resolveEnabledEffort(model, llamaCppEffortPolicy(model)); + return effort === undefined ? undefined : THOUGHT_TOKENS[effort]; +} + +/** `session.prompt`'s thought budget for the effort a caller set; nothing when unset. */ +export function llamaCppThoughtBudgetOption(model: object | undefined): { + budgets?: { thoughtTokens: number }; +} { + const thoughtTokens = llamaCppThoughtBudget(model); + return thoughtTokens === undefined ? {} : { budgets: { thoughtTokens } }; +} diff --git a/providers/node-llama-cpp/src/ai/common/LlamaCpp_TextGeneration.ts b/providers/node-llama-cpp/src/ai/common/LlamaCpp_TextGeneration.ts index bbef17f1ff..43de24f1a7 100644 --- a/providers/node-llama-cpp/src/ai/common/LlamaCpp_TextGeneration.ts +++ b/providers/node-llama-cpp/src/ai/common/LlamaCpp_TextGeneration.ts @@ -32,6 +32,7 @@ import { withModelInUse, withSessionLock, } from "./LlamaCpp_Runtime"; +import { llamaCppThoughtBudgetOption } from "./LlamaCpp_EffortPolicy"; export const LlamaCpp_TextGeneration_Stream: AiProviderRunFn< TextGenerationTaskInput, @@ -175,6 +176,7 @@ export const LlamaCpp_TextGeneration_Stream: AiProviderRunFn< ...(input.temperature !== undefined && { temperature: input.temperature }), ...(input.maxTokens !== undefined && { maxTokens: input.maxTokens }), ...(input.topP !== undefined && { topP: input.topP }), + ...llamaCppThoughtBudgetOption(model), }); }, signal, diff --git a/providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling.ts b/providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling.ts index d994477531..efa787c081 100644 --- a/providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling.ts +++ b/providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling.ts @@ -24,6 +24,7 @@ import { renderLlamaCppPrefixChatHistory, renderLlamaCppPrefixFunctions, } from "./LlamaCpp_CacheCheckpoint"; +import { llamaCppThoughtBudget } from "./LlamaCpp_EffortPolicy"; import type { LlamaCppModelConfig } from "./LlamaCpp_ModelSchema"; import type { LlamaCppSessionState } from "./LlamaCpp_Runtime"; import { @@ -111,7 +112,16 @@ function messagesToPureChatHistory( const response: any[] = []; for (const block of msg.content) { - if (block.type === "text" && block.text) { + if (block.type === "reasoning") { + if (block.text) { + response.push({ + type: "segment", + segmentType: "thought", + text: block.text, + ended: true, + }); + } + } else if (block.type === "text" && block.text) { response.push(block.text); } else if (block.type === "tool_use") { // Create functionCall entry — result will be filled by subsequent tool message @@ -156,12 +166,21 @@ function messagesToPureChatHistory( item.result === undefined ); if (fnCall) { - // Flatten nested text blocks from tool_result content into a string - let resultText = ""; + // The result is a string, so an image cannot ride along. Say it was + // dropped rather than inlining its base64 into the prompt, in the place + // it held among the result's blocks. + const lines: string[] = []; for (const inner of block.content) { - if (inner.type === "text") resultText += inner.text; + if (inner.type === "text") { + if (inner.text) lines.push(inner.text); + } else if (inner.type === "image") { + lines.push(`[${inner.mimeType} image returned; this model reads text only]`); + } else { + // Any other block is data the tool returned; it goes as JSON. + lines.push(JSON.stringify(inner)); + } } - fnCall.result = resultText || JSON.stringify(block.content); + fnCall.result = lines.join("\n"); } } continue; @@ -246,6 +265,10 @@ function llamaCppChatGenerateOptions( } else if (toolChoiceForcesToolCall(input.toolChoice)) { opts.temperature = 0.2; } + const thoughtTokens = llamaCppThoughtBudget(model); + if (thoughtTokens !== undefined) { + opts.budgets = { thoughtTokens }; + } return opts; } @@ -263,18 +286,43 @@ function extractNativeFunctionCalls( })); } +interface LlamaResponseChunk { + readonly type?: string | undefined; + readonly segmentType?: string | undefined; + readonly text: string; +} + +/** + * Decides which output port a streamed chunk belongs to. `onTextChunk` leaves + * thought segments out entirely, so a run-fn that wants the model's thinking + * has to read `onResponseChunk` and sort the chunks itself: a thought goes to + * `reasoning`, the main response to `text`, and any other segment (a comment + * the model wrote about its own turn) is not part of the answer at all. + */ +export function routeLlamaResponseChunk( + chunk: LlamaResponseChunk +): { readonly port: "text" | "reasoning"; readonly text: string } | undefined { + // The SDK marks a main-response chunk with no type; "text" is taken too, so + // the answer still arrives should a release name it. + if (chunk.type === undefined || chunk.type === "text") return { port: "text", text: chunk.text }; + if (chunk.type === "segment" && chunk.segmentType === "thought") { + return { port: "reasoning", text: chunk.text }; + } + return undefined; +} + /** - * Drives an async generation call that pushes text chunks via `onTextChunk`, - * yielding `text-delta` events (and provisional `usage` snapshots) as they - * arrive. Returns accumulated text and the generation result (if any) once - * complete. + * Drives an async generation call that pushes response chunks via + * `onResponseChunk`, yielding `text-delta` events (and provisional `usage` + * snapshots) as they arrive. Returns the answer text (thoughts excluded) and + * the generation result (if any) once complete. */ async function* streamTextChunks( - startGeneration: (onTextChunk: (chunk: string) => void) => Promise, + startGeneration: (onResponseChunk: (chunk: LlamaResponseChunk) => void) => Promise, signal: AbortSignal, options: { readonly promptText?: string | undefined } = {} ): AsyncGenerator, { text: string; result: T | undefined }> { - const queue: string[] = []; + const queue: { readonly port: "text" | "reasoning"; readonly text: string }[] = []; const pendingUsage: StreamEvent[] = []; let isComplete = false; let completionError: unknown; @@ -295,9 +343,11 @@ async function* streamTextChunks( provisionalUsage.onPrompt(options.promptText); } - const generationPromise = startGeneration((chunk: string) => { - queue.push(chunk); - provisionalUsage.onText(chunk); + const generationPromise = startGeneration((chunk) => { + const routed = routeLlamaResponseChunk(chunk); + if (routed === undefined || routed.text === "") return; + queue.push(routed); + provisionalUsage.onText(routed.text); notifyWaiter(); }) .then((res) => { @@ -317,8 +367,8 @@ async function* streamTextChunks( } while (queue.length > 0) { const chunk = queue.shift()!; - accumulatedText += chunk; - yield { type: "text-delta", port: "text", textDelta: chunk }; + if (chunk.port === "text") accumulatedText += chunk.text; + yield { type: "text-delta", port: chunk.port, textDelta: chunk.text }; } }; @@ -377,13 +427,13 @@ async function generateToolResponse( .join("\n"); gen = streamTextChunks( - (onTextChunk) => + (onResponseChunk) => llamaChat.generateResponse(chatHistory, { signal, ...llamaCppChatGenerateOptions(input, model), functions, ...(toolChoiceForcesToolCall(input.toolChoice) && { documentFunctionParams: true }), - onTextChunk, + onResponseChunk, }), signal, { promptText } diff --git a/providers/node-llama-cpp/src/ai/index.ts b/providers/node-llama-cpp/src/ai/index.ts index fa357bef89..c47be21443 100644 --- a/providers/node-llama-cpp/src/ai/index.ts +++ b/providers/node-llama-cpp/src/ai/index.ts @@ -8,6 +8,7 @@ export * from "./common/LlamaCpp_Constants"; export * from "./common/LlamaCpp_ModelSchema"; +export { llamaCppEffortPolicy } from "./common/LlamaCpp_EffortPolicy"; // Mutable runtime state (e.g. llamaCppSessions) is intentionally NOT re-exported // here. The `ai-provider` and `ai-runtime` entry points are bundled // separately, so re-exporting from both creates two distinct module instances diff --git a/providers/node-llama-cpp/src/ai/runtime.ts b/providers/node-llama-cpp/src/ai/runtime.ts index b546871ee7..5fb86a1292 100644 --- a/providers/node-llama-cpp/src/ai/runtime.ts +++ b/providers/node-llama-cpp/src/ai/runtime.ts @@ -18,6 +18,7 @@ export { renderLlamaCppPrefixChatHistory, renderLlamaCppPrefixFunctions, } from "./common/LlamaCpp_CacheCheckpoint"; +export { llamaCppEffortPolicy, llamaCppThoughtBudget } from "./common/LlamaCpp_EffortPolicy"; export { LlamaCpp_SessionDispose } from "./common/LlamaCpp_SessionDispose"; export * from "./registerLlamaCppInline"; export * from "./registerLlamaCppWorker"; diff --git a/providers/ollama/src/ai/OllamaProvider.ts b/providers/ollama/src/ai/OllamaProvider.ts index c93e435251..beb9a2de67 100644 --- a/providers/ollama/src/ai/OllamaProvider.ts +++ b/providers/ollama/src/ai/OllamaProvider.ts @@ -5,10 +5,11 @@ */ import { createCloudProviderClass } from "@workglow/ai/provider-utils"; -import type { Capability, ModelRecord } from "@workglow/ai/worker"; +import type { Capability, ModelEffortPolicy, ModelRecord } from "@workglow/ai/worker"; import { AiProvider } from "@workglow/ai/worker"; import { inferOllamaCapabilities, ollamaWorkerRunFnSpecs } from "./common/Ollama_Capabilities"; import { OLLAMA } from "./common/Ollama_Constants"; +import { ollamaEffortPolicy } from "./common/Ollama_EffortPolicy"; import type { OllamaModelConfig } from "./common/Ollama_ModelSchema"; /** @@ -27,6 +28,10 @@ export class OllamaProvider extends createCloudProviderClass( return inferOllamaCapabilities(model); } + override effortPolicy(model: OllamaModelConfig): ModelEffortPolicy | undefined { + return ollamaEffortPolicy(model); + } + protected override workerRunFnSpecs(): readonly { serves: readonly Capability[] }[] { return ollamaWorkerRunFnSpecs(); } diff --git a/providers/ollama/src/ai/OllamaQueuedProvider.ts b/providers/ollama/src/ai/OllamaQueuedProvider.ts index c487458009..79e2a513e0 100644 --- a/providers/ollama/src/ai/OllamaQueuedProvider.ts +++ b/providers/ollama/src/ai/OllamaQueuedProvider.ts @@ -4,11 +4,18 @@ * SPDX-License-Identifier: Apache-2.0 */ -import type { Capability, ModelConfig, ModelPricing, ModelRecord } from "@workglow/ai"; +import type { + Capability, + ModelConfig, + ModelEffortPolicy, + ModelPricing, + ModelRecord, +} from "@workglow/ai"; import { AiProvider, FREE_LOCAL_PRICING } from "@workglow/ai"; import { createCloudProviderClass } from "@workglow/ai/provider-utils"; import { inferOllamaCapabilities, ollamaWorkerRunFnSpecs } from "./common/Ollama_Capabilities"; import { OLLAMA } from "./common/Ollama_Constants"; +import { ollamaEffortPolicy } from "./common/Ollama_EffortPolicy"; import type { OllamaModelConfig } from "./common/Ollama_ModelSchema"; /** Main-thread registration (inline or worker-backed). No queue — uses direct execution. */ @@ -28,6 +35,10 @@ export class OllamaQueuedProvider extends createCloudProviderClass + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ModelEffort, ModelEffortPolicyFn } from "@workglow/ai/worker"; +import { + EFFORT_POLICY_ALL, + EFFORT_POLICY_NONE, + makeEffortPolicy, + readModelName, + resolveEnabledEffort, +} from "@workglow/ai/worker"; +import type { OllamaModelConfig } from "./Ollama_ModelSchema"; + +const GPT_OSS = /^gpt-oss/i; + +/** + * The qwen3 tags that cannot think: the coder and embedding models, and the + * `-instruct` releases. Ollama answers `think: true` on them with HTTP 400. + */ +const QWEN3_NON_THINKING = /^qwen3(?:-(?:coder|embedding)|.*instruct)/i; + +function isThinkingQwen3(id: string): boolean { + return /^qwen3/i.test(id) && !QWEN3_NON_THINKING.test(id); +} + +/** Families Ollama marks as thinking-capable; they take `think: true | false`. */ +const THINKING = [ + isThinkingQwen3, + /^deepseek-r1/i, + /^deepseek-v3\.[1-9]/i, + /^magistral/i, + /^qwq/i, + /^phi4-reasoning/i, + /^cogito/i, + /^granite3\.[2-9]/i, +]; + +/** + * gpt-oss takes a level and cannot turn thinking off, so `none` is not listed: + * a level the policy omits resolves to no effort and nothing is sent. + */ +export const ollamaEffortPolicy: ModelEffortPolicyFn = makeEffortPolicy({ + rules: [ + { + when: GPT_OSS, + policy: { supported: ["low", "medium", "high", "extra", "ultra"], default: undefined }, + }, + { when: THINKING, policy: EFFORT_POLICY_ALL }, + ], + fallback: EFFORT_POLICY_NONE, +}); + +type OllamaThink = boolean | "low" | "medium" | "high"; + +function gptOssLevel(effort: ModelEffort): "low" | "medium" | "high" | undefined { + switch (effort) { + case "low": + case "medium": + case "high": + return effort; + case "extra": + case "ultra": + return "high"; + default: + return undefined; + } +} + +/** + * The request's `think` field for the effort a caller set. A model that cannot + * think rejects `think: true`, so nothing is sent unless the policy says the + * model thinks and an effort is set. + */ +export function ollamaThinkParam(model: OllamaModelConfig | undefined): { + think?: OllamaThink; +} { + const effort = resolveEnabledEffort(model, ollamaEffortPolicy(model)); + if (effort === undefined) return {}; + const name = readModelName(model); + if (GPT_OSS.test(name)) { + const level = gptOssLevel(effort); + return level === undefined ? {} : { think: level }; + } + return { think: effort !== "none" }; +} diff --git a/providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts b/providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts index cb630b5596..0f7b8468d9 100644 --- a/providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts +++ b/providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts @@ -4,7 +4,7 @@ * SPDX-License-Identifier: Apache-2.0 */ -import type { AiProviderRunFnRegistration, ToolCallingTaskInput } from "@workglow/ai"; +import type { AiProviderRunFnRegistration } from "@workglow/ai"; import { OLLAMA_JSON_MODE, OLLAMA_MODEL_INFO, @@ -16,6 +16,7 @@ import { OLLAMA_TOOL_USE, } from "./Ollama_CapabilitySets"; import { getClient } from "./Ollama_Client.browser"; +import { toOllamaMessages } from "./Ollama_Messages"; import { createOllamaModelInfoStream } from "./Ollama_ModelInfo"; import type { OllamaModelConfig } from "./Ollama_ModelSchema"; import { createOllamaModelSearchStream } from "./Ollama_ModelSearch"; @@ -28,27 +29,12 @@ import { createOllamaToolCallingStream } from "./Ollama_ToolCalling"; export { getClient, getModelName, loadOllamaSDK } from "./Ollama_Client.browser"; -function buildBrowserToolCallingMessages(input: ToolCallingTaskInput): Array<{ - role: string; - content: string; -}> { - const messages: Array<{ role: string; content: string }> = []; - if (input.systemPrompt) { - messages.push({ role: "system", content: input.systemPrompt as string }); - } - messages.push({ role: "user", content: input.prompt as string }); - return messages; -} - export const Ollama_TextGeneration_Stream = createOllamaTextGenerationStream(getClient); export const Ollama_StructuredGeneration_Stream = createOllamaStructuredGenerationStream(getClient); export const Ollama_TextRewriter_Stream = createOllamaTextRewriterStream(getClient); export const Ollama_TextSummary_Stream = createOllamaTextSummaryStream(getClient); export const Ollama_TextEmbedding_Stream = createOllamaTextEmbeddingStream(getClient); -export const Ollama_ToolCalling_Stream = createOllamaToolCallingStream( - getClient, - buildBrowserToolCallingMessages -); +export const Ollama_ToolCalling_Stream = createOllamaToolCallingStream(getClient, toOllamaMessages); export const Ollama_ModelInfo_Stream = createOllamaModelInfoStream(getClient); export const Ollama_ModelSearch_Stream = createOllamaModelSearchStream(getClient); diff --git a/providers/ollama/src/ai/common/Ollama_JobRunFns.ts b/providers/ollama/src/ai/common/Ollama_JobRunFns.ts index 54c6cff1fa..e0d40e8e7e 100644 --- a/providers/ollama/src/ai/common/Ollama_JobRunFns.ts +++ b/providers/ollama/src/ai/common/Ollama_JobRunFns.ts @@ -5,7 +5,6 @@ */ import type { AiProviderRunFnRegistration } from "@workglow/ai"; -import { toTextFlatMessages } from "@workglow/ai/worker"; import { OLLAMA_JSON_MODE, OLLAMA_MODEL_INFO, @@ -17,6 +16,7 @@ import { OLLAMA_TOOL_USE, } from "./Ollama_CapabilitySets"; import { getClient } from "./Ollama_Client"; +import { toOllamaMessages } from "./Ollama_Messages"; import { createOllamaModelInfoStream } from "./Ollama_ModelInfo"; import type { OllamaModelConfig } from "./Ollama_ModelSchema"; import { createOllamaModelSearchStream } from "./Ollama_ModelSearch"; @@ -32,10 +32,7 @@ export const Ollama_StructuredGeneration_Stream = createOllamaStructuredGenerati export const Ollama_TextRewriter_Stream = createOllamaTextRewriterStream(getClient); export const Ollama_TextSummary_Stream = createOllamaTextSummaryStream(getClient); export const Ollama_TextEmbedding_Stream = createOllamaTextEmbeddingStream(getClient); -export const Ollama_ToolCalling_Stream = createOllamaToolCallingStream( - getClient, - toTextFlatMessages -); +export const Ollama_ToolCalling_Stream = createOllamaToolCallingStream(getClient, toOllamaMessages); export const Ollama_ModelInfo_Stream = createOllamaModelInfoStream(getClient); export const Ollama_ModelSearch_Stream = createOllamaModelSearchStream(getClient); diff --git a/providers/ollama/src/ai/common/Ollama_Messages.ts b/providers/ollama/src/ai/common/Ollama_Messages.ts new file mode 100644 index 0000000000..093be4f9d6 --- /dev/null +++ b/providers/ollama/src/ai/common/Ollama_Messages.ts @@ -0,0 +1,173 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolCallingTaskInput } from "@workglow/ai"; +import { liftSystemMessages } from "@workglow/ai/worker"; +import { PermanentJobError } from "@workglow/job-queue"; + +/** + * One message on Ollama's `/api/chat`. `tool_calls` carry no ids — Ollama pairs + * a result to its call by `tool_name` on the `tool` message instead. + */ +export interface OllamaChatMessage { + readonly role: string; + readonly content: string; + readonly thinking?: string; + readonly images?: readonly string[]; + readonly tool_calls?: ReadonlyArray<{ + readonly function: { + readonly name: string; + readonly arguments: Record; + }; + }>; + readonly tool_name?: string; +} + +interface TextOrImage { + readonly type: string; + readonly text?: unknown; + readonly data?: unknown; +} + +/** + * Splits text/image blocks into the joined text and the base64 image list. With + * `keepOther`, any other block (data a tool returned) is kept as JSON text. + * With `markImagesAfter`, each image leaves a numbered marker where it stood, + * counting on from that many images already set aside, so text around an image + * still points at it once the image itself has moved to a later message. + * Audio is refused wherever it appears. + */ +function flattenBlocks( + blocks: ReadonlyArray, + separator: string, + keepOther = false, + markImagesAfter?: number +): { readonly text: string; readonly images: string[] } { + const texts: string[] = []; + const images: string[] = []; + for (const block of blocks) { + if (block.type === "audio") { + // Ollama's chat API has no audio field, and sending the turn without it + // would have the model answer as if it had heard nothing. + throw new PermanentJobError("Ollama's chat API does not accept audio input."); + } + if (block.type === "text" && typeof block.text === "string") { + texts.push(block.text); + } else if (block.type === "image" && typeof block.data === "string") { + images.push(block.data); + if (markImagesAfter !== undefined) { + texts.push(`[image ${markImagesAfter + images.length}] follows in the next message`); + } + } else if (keepOther && block.type !== "text" && block.type !== "image") { + texts.push(JSON.stringify(block)); + } + } + return { text: texts.join(separator), images }; +} + +function withImages(message: OllamaChatMessage, images: readonly string[]): OllamaChatMessage { + return images.length > 0 ? { ...message, images } : message; +} + +/** + * Converts a tool-calling request into Ollama chat messages, keeping the model's + * own tool calls, the tool name each result answers, thinking, and images. + * + * `messages` wins over `prompt` whenever it is non-empty: the agent loop sends + * both, and `messages` already starts with that prompt. + */ +export function toOllamaMessages(input: ToolCallingTaskInput): OllamaChatMessage[] { + const out: OllamaChatMessage[] = []; + // System turns in the history join the system prompt, as on the + // text-generation path, rather than being dropped. + const lifted = liftSystemMessages(input.messages ?? [], input.systemPrompt); + + if (lifted.systemPrompt) { + out.push({ role: "system", content: lifted.systemPrompt }); + } + + // A history of system turns alone is still the history: the prompt is what a + // caller with no history sends, never a stand-in for one. + const history = lifted.messages; + if (!input.messages || input.messages.length === 0) { + const prompt = input.prompt as unknown; + if (typeof prompt === "string") { + out.push({ role: "user", content: prompt }); + } else if (Array.isArray(prompt)) { + const blocks = prompt.map((item): TextOrImage => + typeof item === "string" ? { type: "text", text: item } : (item as TextOrImage) + ); + const { text, images } = flattenBlocks(blocks, "\n"); + out.push(withImages({ role: "user", content: text }, images)); + } + return out; + } + + // Provider-assigned ids restart at `call_0` each run, so they repeat across + // turns. The map is updated as each assistant turn is read, so a result + // resolves against the most recent preceding call with its id. + const toolNames = new Map(); + // Ollama's chat templates render a tool message's text only, so images a tool + // returned ride on a user turn — after the whole run of tool messages, so it + // never sits between two results of one assistant turn. + let toolImages: string[] = []; + const flushToolImages = (): void => { + if (toolImages.length === 0) return; + const labels = toolImages.map((_, i) => `[image ${i + 1}]`).join(", "); + out.push({ + role: "user", + content: `Images returned by the tool calls above, in order: ${labels}`, + images: toolImages, + }); + toolImages = []; + }; + + for (const msg of history) { + if (msg.role !== "tool") flushToolImages(); + if (msg.role === "user") { + const { text, images } = flattenBlocks(msg.content, "\n"); + out.push(withImages({ role: "user", content: text }, images)); + } else if (msg.role === "assistant") { + const texts: string[] = []; + const thoughts: string[] = []; + const calls: Array<{ + function: { name: string; arguments: Record }; + }> = []; + for (const block of msg.content) { + if (block.type === "text") { + texts.push(block.text); + } else if (block.type === "reasoning") { + if (block.text) thoughts.push(block.text); + } else if (block.type === "tool_use") { + toolNames.set(block.id, block.name); + calls.push({ function: { name: block.name, arguments: block.input } }); + } + } + const content = texts.join(""); + if (content === "" && thoughts.length === 0 && calls.length === 0) continue; + let message: OllamaChatMessage = { role: "assistant", content }; + if (thoughts.length > 0) message = { ...message, thinking: thoughts.join("") }; + if (calls.length > 0) message = { ...message, tool_calls: calls }; + out.push(message); + } else if (msg.role === "tool") { + for (const block of msg.content) { + if (block.type !== "tool_result") continue; + const { text, images } = flattenBlocks(block.content, "\n", true, toolImages.length); + let message: OllamaChatMessage = { + role: "tool", + content: block.is_error ? `Error: ${text}` : text, + }; + const name = toolNames.get(block.tool_use_id); + if (name !== undefined) message = { ...message, tool_name: name }; + out.push(message); + toolImages.push(...images); + } + } + } + flushToolImages(); + + return out; +} diff --git a/providers/ollama/src/ai/common/Ollama_TextGeneration.ts b/providers/ollama/src/ai/common/Ollama_TextGeneration.ts index baf77452cd..33234218ff 100644 --- a/providers/ollama/src/ai/common/Ollama_TextGeneration.ts +++ b/providers/ollama/src/ai/common/Ollama_TextGeneration.ts @@ -6,11 +6,16 @@ import type { AiProviderRunFn, + LooseChatMessage, TextGenerationTaskInput, TextGenerationTaskOutput, Usage, } from "@workglow/ai"; +import { normalizeChatMessages } from "@workglow/ai/worker"; import { createEstimatedOutputUsageReporter } from "@workglow/ai/provider-utils"; +import { ollamaThinkParam } from "./Ollama_EffortPolicy"; +import type { OllamaChatMessage } from "./Ollama_Messages"; +import { toOllamaMessages } from "./Ollama_Messages"; import type { OllamaModelConfig } from "./Ollama_ModelSchema"; import { getOllamaModelName } from "./Ollama_ModelUtil"; import { mapOllamaUsage } from "./Ollama_Usage"; @@ -18,10 +23,29 @@ import { mapOllamaUsage } from "./Ollama_Usage"; type GetClient = (model: OllamaModelConfig | undefined) => Promise; interface UnifiedTextGenerationInput extends TextGenerationTaskInput { - readonly messages?: readonly { readonly role: string; readonly content: string }[]; + readonly messages?: ReadonlyArray; readonly systemPrompt?: string; } +/** + * Chat history in Ollama's shape, through the same converter tool calling uses: + * an assistant turn keeps its tool calls and each tool result keeps the call it + * answers, so a history seeded with a tool exchange is not sent half of it. + * System turns join the system prompt, and a user turn's images ride in its own + * `images` field. + */ +export function toOllamaChatMessages( + messages: ReadonlyArray, + systemPrompt: string | undefined +): OllamaChatMessage[] { + return toOllamaMessages({ + messages: normalizeChatMessages(messages), + systemPrompt, + prompt: "", + tools: [], + } as never); +} + /** * Streaming run-fn factory for the `["text.generation"]` capability. Returns * an async generator that yields `text-delta` events and a final empty @@ -41,11 +65,8 @@ export function createOllamaTextGenerationStream( const unified = input as UnifiedTextGenerationInput; const hasMessages = Array.isArray(unified.messages) && unified.messages.length > 0; - const messages = hasMessages - ? [ - ...(unified.systemPrompt ? [{ role: "system", content: unified.systemPrompt }] : []), - ...unified.messages!.map((m) => ({ role: m.role, content: m.content })), - ] + const messages: OllamaChatMessage[] = hasMessages + ? toOllamaChatMessages(unified.messages!, unified.systemPrompt) : [{ role: "user", content: input.prompt }]; // Ollama only reports counts on the terminal `done: true` chunk; estimate ↑ @@ -61,6 +82,7 @@ export function createOllamaTextGenerationStream( const stream = await client.chat({ model: modelName, messages, + ...ollamaThinkParam(model), options: { temperature: input.temperature, top_p: input.topP, diff --git a/providers/ollama/src/ai/common/Ollama_ToolCalling.ts b/providers/ollama/src/ai/common/Ollama_ToolCalling.ts index effbbfde65..cee852ae92 100644 --- a/providers/ollama/src/ai/common/Ollama_ToolCalling.ts +++ b/providers/ollama/src/ai/common/Ollama_ToolCalling.ts @@ -17,13 +17,13 @@ import { buildToolDescription, filterValidToolCalls, sanitizeToolArgs } from "@w import { parsePartialJson } from "@workglow/util/worker"; import type { OllamaModelConfig } from "./Ollama_ModelSchema"; import { getOllamaModelName } from "./Ollama_ModelUtil"; +import { ollamaThinkParam } from "./Ollama_EffortPolicy"; +import type { OllamaChatMessage } from "./Ollama_Messages"; import { mapOllamaUsage } from "./Ollama_Usage"; type GetClient = (model: OllamaModelConfig | undefined) => Promise; -export type OllamaToolCallingMessagesFn = ( - input: ToolCallingTaskInput -) => Array<{ role: string; content: string }>; +export type OllamaToolCallingMessagesFn = (input: ToolCallingTaskInput) => OllamaChatMessage[]; function mapOllamaTools(tools: ReadonlyArray) { return tools.map((t) => ({ @@ -60,6 +60,7 @@ export function createOllamaToolCallingStream( model: modelName, messages, tools, + ...ollamaThinkParam(model), options: { temperature: input.temperature, num_predict: input.maxTokens, @@ -76,6 +77,11 @@ export function createOllamaToolCallingStream( try { for await (const chunk of stream) { usage = mapOllamaUsage(chunk) ?? usage; + const thinking = chunk.message.thinking; + if (thinking) { + provisionalUsage.onText(thinking); + emit({ type: "text-delta", port: "reasoning", textDelta: thinking }); + } const delta = chunk.message.content; if (delta) { provisionalUsage.onText(delta); diff --git a/providers/ollama/src/ai/index.browser.ts b/providers/ollama/src/ai/index.browser.ts index cd6a8e6d38..82dfa4fb86 100644 --- a/providers/ollama/src/ai/index.browser.ts +++ b/providers/ollama/src/ai/index.browser.ts @@ -7,5 +7,6 @@ // organize-imports-ignore export * from "./common/Ollama_Constants"; +export * from "./common/Ollama_EffortPolicy"; export * from "./common/Ollama_ModelSchema"; export * from "./registerOllama"; diff --git a/providers/ollama/src/ai/index.ts b/providers/ollama/src/ai/index.ts index 42ba9df194..ac73c45b45 100644 --- a/providers/ollama/src/ai/index.ts +++ b/providers/ollama/src/ai/index.ts @@ -7,6 +7,7 @@ // organize-imports-ignore export * from "./common/Ollama_Constants"; +export * from "./common/Ollama_EffortPolicy"; export * from "./common/Ollama_ModelSchema"; export * from "./registerOllama"; diff --git a/providers/ollama/src/ai/runtime.browser.ts b/providers/ollama/src/ai/runtime.browser.ts index 59877518ad..452971a482 100644 --- a/providers/ollama/src/ai/runtime.browser.ts +++ b/providers/ollama/src/ai/runtime.browser.ts @@ -14,5 +14,6 @@ // organize-imports-ignore export * from "./common/Ollama_Client.browser"; +export * from "./common/Ollama_EffortPolicy"; export * from "./registerOllamaInline.browser"; export * from "./registerOllamaWorker.browser"; diff --git a/providers/ollama/src/ai/runtime.ts b/providers/ollama/src/ai/runtime.ts index f394dee74b..dc699e8d15 100644 --- a/providers/ollama/src/ai/runtime.ts +++ b/providers/ollama/src/ai/runtime.ts @@ -14,6 +14,7 @@ // organize-imports-ignore export * from "./common/Ollama_Client"; +export * from "./common/Ollama_EffortPolicy"; export * from "./common/Ollama_TextGeneration"; export * from "./common/Ollama_StructuredGeneration"; export * from "./registerOllamaInline"; From 2c8e06ef66a8ff21946fcc070c141d780014bcaa Mon Sep 17 00:00:00 2001 From: Steven Roussey Date: Mon, 12 Oct 2026 05:34:45 +0000 Subject: [PATCH 5/7] cli, web: draw a task's reasoning apart from its answer Ink rows, the web console and examples/web's TaskNode show the reasoning port separately and drop it once the answer starts; every other port is still the task's text. workglow agent chat and the run-event projection no longer print reasoning as answer text, and StreamingTextRow also draws while a task is streaming. --- examples/cli/src/agent/runAgentChat.ts | 6 +- examples/cli/src/run-events/RunEventTypes.ts | 2 + .../src/run-events/projectRunEvents.test.ts | 44 +++++++ .../cli/src/run-events/projectRunEvents.ts | 13 ++- examples/cli/src/test/agentChat.test.ts | 15 +++ .../src/test/streaming-reasoning-row.test.ts | 109 ++++++++++++++++++ examples/cli/src/ui/TaskRunApp.tsx | 34 +++--- .../src/ui/components/StreamedReasoning.tsx | 34 ++++++ .../cli/src/ui/model/streamedText.test.ts | 63 ++++++++++ examples/cli/src/ui/model/streamedText.ts | 77 +++++++++++++ examples/cli/src/ui/rows/ChatTaskRow.tsx | 40 +++++-- examples/cli/src/ui/rows/StreamingTextRow.tsx | 25 ++-- examples/cli/src/ui/rows/useTaskStreamText.ts | 18 ++- examples/cli/src/web/client/app.css | 14 +++ examples/cli/src/web/client/state.test.ts | 11 ++ examples/cli/src/web/client/state.ts | 7 ++ .../cli/src/web/client/views/RunConsole.tsx | 17 ++- examples/web/src/graph/TaskNode.test.ts | 75 +++++++++++- examples/web/src/graph/TaskNode.tsx | 32 ++++- 19 files changed, 579 insertions(+), 57 deletions(-) create mode 100644 examples/cli/src/test/streaming-reasoning-row.test.ts create mode 100644 examples/cli/src/ui/components/StreamedReasoning.tsx create mode 100644 examples/cli/src/ui/model/streamedText.test.ts create mode 100644 examples/cli/src/ui/model/streamedText.ts diff --git a/examples/cli/src/agent/runAgentChat.ts b/examples/cli/src/agent/runAgentChat.ts index 25ba06907f..498c57c87b 100644 --- a/examples/cli/src/agent/runAgentChat.ts +++ b/examples/cli/src/agent/runAgentChat.ts @@ -198,7 +198,11 @@ async function runTurn( let phase: string | undefined; const offStream = task.subscribe("stream_chunk", (event: StreamEvent) => { - if (event.type === "text-delta") transcript.delta(event.textDelta); + // The turn streams its reasoning beside its reply on its own port; printed + // here it would read as part of the answer. + if (event.type === "text-delta" && (event.port ?? "text") === "text") { + transcript.delta(event.textDelta); + } }); const offProgress = task.subscribe("progress", (_progress, message) => { // Only the tool lines are worth a row: "Thinking" is what the blank space diff --git a/examples/cli/src/run-events/RunEventTypes.ts b/examples/cli/src/run-events/RunEventTypes.ts index 1484e18859..2daba2d195 100644 --- a/examples/cli/src/run-events/RunEventTypes.ts +++ b/examples/cli/src/run-events/RunEventTypes.ts @@ -59,6 +59,8 @@ export type RunEvent = readonly message?: string; } | { readonly k: "text"; readonly id: string; readonly delta: string } + /** A model's thinking, streamed beside its answer; never folded into `text`. */ + | { readonly k: "reasoning"; readonly id: string; readonly delta: string } | { readonly k: "messages"; readonly id: string; readonly messages: unknown } | { readonly k: "human_request"; diff --git a/examples/cli/src/run-events/projectRunEvents.test.ts b/examples/cli/src/run-events/projectRunEvents.test.ts index 2d39e5596a..71d07142f0 100644 --- a/examples/cli/src/run-events/projectRunEvents.test.ts +++ b/examples/cli/src/run-events/projectRunEvents.test.ts @@ -66,6 +66,50 @@ describe("projectRunEvents", () => { }); }); +const STREAMED_PORTS = { + type: "object", + properties: { + text: { type: "string", "x-stream": "append" }, + reasoning: { type: "string", "x-stream": "append" }, + }, + additionalProperties: false, +} as const satisfies DataPortSchema; + +/** Streams a reply beside its reasoning, as `AgentTask` does. */ +class ThinkingTask extends Task, { text: string; reasoning: string }> { + static override readonly type = "ThinkingTask"; + static override readonly title = "Thinking task"; + static override readonly category = "Test"; + static override readonly cacheable = false; + static override inputSchema(): DataPortSchema { + return EMPTY; + } + static override outputSchema(): DataPortSchema { + return STREAMED_PORTS; + } + async *executeStream() { + yield { type: "text-delta" as const, port: "reasoning", textDelta: "Let me think." }; + yield { type: "text-delta" as const, port: "text", textDelta: "Hello." }; + yield { type: "finish" as const, data: {} as { text: string; reasoning: string } }; + } +} + +describe("projectRunEvents streamed text", () => { + it("reports a task's text and reasoning as separate kinds", async () => { + const { events, sink } = collector(); + const workflow = new Workflow(); + workflow.pipe(new ThinkingTask() as never); + const stop = projectRunEvents(workflow.graph, sink); + await workflow.run({}); + stop(); + + const deltas = events.flatMap((e) => (e.k === "text" ? [e.delta] : [])); + expect(deltas).toEqual(["Hello."]); + const thinking = events.flatMap((e) => (e.k === "reasoning" ? [e.delta] : [])); + expect(thinking).toEqual(["Let me think."]); + }); +}); + const ITEM_SCHEMA = { type: "object", properties: { item: { type: "number" } }, diff --git a/examples/cli/src/run-events/projectRunEvents.ts b/examples/cli/src/run-events/projectRunEvents.ts index 50dd9a88a4..0ec1111a6d 100644 --- a/examples/cli/src/run-events/projectRunEvents.ts +++ b/examples/cli/src/run-events/projectRunEvents.ts @@ -5,6 +5,7 @@ */ import type { ITask, StreamEvent, TaskGraph, TaskIdType, Usage } from "@workglow/task-graph"; +import { classifyTextDelta } from "../ui/model/streamedText"; import { cliTaskLabel } from "../ui/taskGraphCliSubscriptions"; import { projectTaskSubtree } from "./projectSubtrees"; import type { RunEventSink } from "./runEventChannel"; @@ -119,8 +120,16 @@ export function projectRunEvents( }); const onChunk = (taskId: TaskIdType, event: StreamEvent): void => { if (event.type === "text-delta") { - const delta = (event as { textDelta?: string; text?: string }).textDelta ?? ""; - if (delta) sink.emit({ k: "text", id: String(taskId), delta }); + // A task's reasoning streams beside its reply on its own port; folded into + // the row's text it would read as part of the answer, so it is its own kind. + const classified = classifyTextDelta(event); + if (classified) { + sink.emit({ + k: classified.port === "text" ? "text" : "reasoning", + id: String(taskId), + delta: classified.delta, + }); + } } else if (event.type === "object-delta" && (event as { port?: string }).port === "messages") { const task = graph.getTask(taskId); sink.emit({ diff --git a/examples/cli/src/test/agentChat.test.ts b/examples/cli/src/test/agentChat.test.ts index 52c8ffa9fe..c729f2ad7e 100644 --- a/examples/cli/src/test/agentChat.test.ts +++ b/examples/cli/src/test/agentChat.test.ts @@ -99,6 +99,21 @@ describe("agent chat loop", () => { expect(out()).toContain("Hello back."); }); + it("prints the reply and not the reasoning streamed beside it", async () => { + const runFn: AiProviderRunFn = async (_input, _model, _signal, emit) => { + emit({ type: "text-delta", port: "reasoning", textDelta: "Pondering quietly." }); + emit({ type: "text-delta", port: "text", textDelta: "Hello back." }); + emit({ type: "finish", data: {} }); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + const { io, out } = scriptedIo(["hello", "/exit"]); + + await runAgentChat(baseOptions, io); + + expect(out()).toContain("Hello back."); + expect(out()).not.toContain("Pondering"); + }); + it("ends at end of input as well as on /exit", async () => { scriptModel(["…"]); const { io } = scriptedIo(["hello", undefined as unknown as string]); diff --git a/examples/cli/src/test/streaming-reasoning-row.test.ts b/examples/cli/src/test/streaming-reasoning-row.test.ts new file mode 100644 index 0000000000..140cfa294a --- /dev/null +++ b/examples/cli/src/test/streaming-reasoning-row.test.ts @@ -0,0 +1,109 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { StreamEvent } from "@workglow/task-graph"; +import { Task, Workflow } from "@workglow/task-graph"; +import type { DataPortSchema } from "@workglow/util/schema"; +import { render } from "ink"; +import { EventEmitter } from "node:events"; +import React from "react"; +import { describe, expect, it } from "vitest"; +import { WorkflowRunApp } from "../ui/WorkflowRunApp"; + +const OUTPUT = { + type: "object", + properties: { + text: { type: "string", "x-stream": "append" }, + reasoning: { type: "string", "x-stream": "append" }, + }, +} as const satisfies DataPortSchema; + +const EMPTY = { type: "object", properties: {} } as const satisfies DataPortSchema; + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** Streams its thinking, pauses, then streams its answer, pausing after each. */ +class ThinkingThenAnswerTask extends Task, { text: string }> { + static override readonly type = "ThinkingThenAnswerTask"; + static override readonly title = "Thinking then answer"; + static override readonly category = "Test"; + static override readonly cacheable = false; + static override inputSchema(): DataPortSchema { + return EMPTY; + } + static override outputSchema(): DataPortSchema { + return OUTPUT; + } + async *executeStream(): AsyncIterable> { + // Give the mounted row time to subscribe before the first delta. + await sleep(300); + yield { type: "text-delta", port: "reasoning", textDelta: "Weighing the options." }; + await sleep(400); + yield { type: "text-delta", port: "text", textDelta: "The answer is 42." }; + await sleep(400); + yield { type: "finish", data: {} as { text: string } }; + } +} + +class CapturingStdout extends EventEmitter { + readonly columns = 120; + readonly rows = 40; + readonly isTTY = true; + readonly frames: string[] = []; + write(data: string): boolean { + this.frames.push(data); + return true; + } +} + +// eslint-disable-next-line no-control-regex +const stripAnsi = (s: string): string => s.replace(/\x1b\[[0-9;]*[A-Z]/gi, ""); + +async function renderFrames(): Promise { + const workflow = new Workflow(); + workflow.pipe(new ThinkingThenAnswerTask() as never); + const stdout = new CapturingStdout(); + let finished = false; + const instance = render( + React.createElement(WorkflowRunApp, { + graph: workflow.graph, + input: {}, + runExecutor: () => workflow.run({}), + onComplete: () => { + finished = true; + }, + onError: () => { + finished = true; + }, + }), + // Under a CI runner Ink would otherwise write only the final frame, and the + // thinking this reads is drawn mid-run. + { stdout: stdout as never, patchConsole: false, exitOnCtrlC: false, interactive: true } + ); + const deadline = Date.now() + 6000; + while (!finished && Date.now() < deadline) await sleep(20); + await sleep(100); + instance.unmount(); + return stdout.frames.map(stripAnsi); +} + +describe("streaming row with a reasoning port", () => { + it("draws thinking apart from the answer, and drops it once the answer starts", async () => { + const frames = await renderFrames(); + + const thinkingFrame = frames.find((f) => f.includes("Weighing the options.")); + expect(thinkingFrame, "reasoning was never drawn").toBeDefined(); + expect(thinkingFrame).toContain("thinking"); + expect(thinkingFrame).not.toContain("The answer is 42."); + + const answerFrame = frames.find((f) => f.includes("The answer is 42.")); + expect(answerFrame, "the answer was never drawn").toBeDefined(); + expect(answerFrame).not.toContain("Weighing the options."); + expect(answerFrame).not.toContain("thinking"); + }); +}); diff --git a/examples/cli/src/ui/TaskRunApp.tsx b/examples/cli/src/ui/TaskRunApp.tsx index d93e7711db..e397addb3d 100644 --- a/examples/cli/src/ui/TaskRunApp.tsx +++ b/examples/cli/src/ui/TaskRunApp.tsx @@ -4,7 +4,7 @@ * SPDX-License-Identifier: Apache-2.0 */ -import type { IRunConfig, ITask } from "@workglow/task-graph"; +import type { IRunConfig, ITask, StreamEvent } from "@workglow/task-graph"; import { Box, Static, Text, useWindowSize } from "ink"; import path from "node:path"; import React, { useEffect, useMemo, useRef, useState } from "react"; @@ -15,11 +15,18 @@ import { ProgressBar } from "./components/ProgressBar"; import { deriveRunState, RunStatusBar } from "./components/RunStatusBar"; import { RunViewportProvider } from "./components/RunViewport"; import { ScrollRegion } from "./components/ScrollRegion"; +import { StreamedReasoning } from "./components/StreamedReasoning"; import { StreamOutput } from "./components/StreamOutput"; import { TaskErrorDetail } from "./components/TaskErrorDetail"; import { TaskStatusProgressRow } from "./components/TaskStatusProgressRow"; import { HumanInteractionHost } from "./HumanInteractionHost"; import { planRunViewport } from "./model/runViewport"; +import { + EMPTY_STREAMED_TEXT, + foldTextDelta, + visibleReasoning, + type StreamedText, +} from "./model/streamedText"; import { isRedundantSubgraph, SubtaskRows } from "./rows/SubtaskRows"; import { settledTaskDurationMs } from "./rows/taskDuration"; import { useSubtaskRows } from "./rows/useSubtaskRows"; @@ -99,7 +106,7 @@ export function TaskRunApp({ message: undefined, }); const [downloadFiles, setDownloadFiles] = useState([]); - const [streamText, setStreamText] = useState(""); + const [streamed, setStreamed] = useState(EMPTY_STREAMED_TEXT); const [logs, setLogs] = useState([]); const runUsageLine = useGraphUsageLine(task.subGraph); const subtasks = useSubtaskRows(task); @@ -152,21 +159,18 @@ export function TaskRunApp({ if (msg) progressRef.current.msg = msg; }); - // `textDelta`, not `text`: that is the field name on StreamTextDelta, and - // the loose local type here is why reading the wrong one showed as an - // empty panel rather than as a compile error. - task.events.on("stream_chunk", (event: { type: string; textDelta?: string }) => { - if (event.type === "text-delta" && event.textDelta) { - setStreamText((prev) => prev + event.textDelta); - } + task.events.on("stream_chunk", (event: StreamEvent) => { + setStreamed((prev) => foldTextDelta(prev, event)); }); + // Only the answer is kept in the scrollback; the thinking behind it is + // shown while it streams and then dropped. task.events.on("stream_end", () => { - setStreamText((prev) => { - if (prev) { - setLogs((logs) => [...logs, { id: logCounter++, text: prev }]); + setStreamed((prev) => { + if (prev.text) { + setLogs((logs) => [...logs, { id: logCounter++, text: prev.text }]); } - return ""; + return EMPTY_STREAMED_TEXT; }); }); @@ -188,6 +192,7 @@ export function TaskRunApp({ }, [onComplete, onError, task, taskType, overrides, runConfig]); /** Job queue may register multiple subgraph tasks of the same type (mirrors); hide when all match the parent. */ + const reasoning = visibleReasoning(streamed); const subgraphIsAllSameTypeMirror = isModelDownloadTask && subtasks.rows.length > 1 && @@ -242,7 +247,8 @@ export function TaskRunApp({ ))} )} - {streamText && } + {streamed.text && } + {reasoning && } {showSubtasksSection && ( + * SPDX-License-Identifier: Apache-2.0 + */ + +import { Box, Text } from "ink"; +import React from "react"; +import { STREAM_PANEL_MAX_LINES, tailLines } from "../model/streamedText"; + +interface StreamedReasoningProps { + readonly reasoning: string; + /** Total height including the label; the thinking itself gets one line fewer. */ + readonly maxLines?: number; +} + +/** + * A model's thinking, drawn dim under a "thinking" label so it cannot be + * mistaken for the answer. Bounded to `maxLines` rows like the answer panel. + */ +export function StreamedReasoning({ + reasoning, + maxLines = STREAM_PANEL_MAX_LINES, +}: StreamedReasoningProps): React.ReactElement | null { + if (!reasoning) return null; + return ( + + + thinking + + {tailLines(reasoning, Math.max(1, maxLines - 1))} + + ); +} diff --git a/examples/cli/src/ui/model/streamedText.test.ts b/examples/cli/src/ui/model/streamedText.test.ts new file mode 100644 index 0000000000..1c7bcddced --- /dev/null +++ b/examples/cli/src/ui/model/streamedText.test.ts @@ -0,0 +1,63 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from "vitest"; +import { + EMPTY_STREAMED_TEXT, + foldTextDelta, + tailLines, + visibleReasoning, + type StreamedText, +} from "./streamedText"; + +const fold = (events: ReadonlyArray[1]>): StreamedText => + events.reduce(foldTextDelta, EMPTY_STREAMED_TEXT); + +describe("foldTextDelta", () => { + it("keeps text and reasoning apart", () => { + const out = fold([ + { type: "text-delta", port: "reasoning", textDelta: "Let me " }, + { type: "text-delta", port: "reasoning", textDelta: "think." }, + { type: "text-delta", port: "text", textDelta: "Hel" }, + { type: "text-delta", port: "text", textDelta: "lo" }, + ]); + expect(out).toEqual({ text: "Hello", reasoning: "Let me think." }); + }); + + it("treats a missing port as text", () => { + expect(fold([{ type: "text-delta", textDelta: "hi" }])).toEqual({ + text: "hi", + reasoning: "", + }); + }); + + it("reads a task's own output port as its text", () => { + expect( + foldTextDelta(EMPTY_STREAMED_TEXT, { type: "text-delta", port: "code", textDelta: "x" }) + ).toEqual({ text: "x", reasoning: "" }); + }); + + it("ignores other event types and empty deltas, returning the same object", () => { + const prev: StreamedText = { text: "a", reasoning: "b" }; + expect(foldTextDelta(prev, { type: "finish" })).toBe(prev); + expect(foldTextDelta(prev, { type: "text-delta", port: "text", textDelta: "" })).toBe(prev); + expect(foldTextDelta(prev, { type: "text-delta", port: "text" })).toBe(prev); + }); +}); + +describe("visibleReasoning", () => { + it("shows thinking until the answer starts", () => { + expect(visibleReasoning({ text: "", reasoning: "hmm" })).toBe("hmm"); + expect(visibleReasoning({ text: "A", reasoning: "hmm" })).toBe(""); + }); +}); + +describe("tailLines", () => { + it("keeps the last n lines and leaves a short text alone", () => { + expect(tailLines("a\nb\nc\nd", 2)).toBe("c\nd"); + expect(tailLines("a\nb", 5)).toBe("a\nb"); + }); +}); diff --git a/examples/cli/src/ui/model/streamedText.ts b/examples/cli/src/ui/model/streamedText.ts new file mode 100644 index 0000000000..755e568671 --- /dev/null +++ b/examples/cli/src/ui/model/streamedText.ts @@ -0,0 +1,77 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * What a streaming task has said so far, split by the port it said it on. + * + * A task streams its answer on port `text` and a reasoning model's thinking on + * port `reasoning`. Folded into one string the thinking reads as part of the + * answer, so every surface that draws a streaming row folds through here and + * decides for itself how to show the two apart. + */ +export interface StreamedText { + readonly text: string; + readonly reasoning: string; +} + +export const EMPTY_STREAMED_TEXT: StreamedText = { text: "", reasoning: "" }; + +/** The part of a stream event the fold reads; any `StreamEvent` is assignable. */ +export interface TextDeltaLike { + readonly type: string; + readonly port?: string; + readonly textDelta?: string; +} + +/** Lines of a streaming panel shown at once; the oldest scroll off the top. */ +export const STREAM_PANEL_MAX_LINES = 8; + +/** A text delta on a port worth showing: which one, and what it said. */ +export interface ClassifiedDelta { + readonly port: "text" | "reasoning"; + readonly delta: string; +} + +/** + * Reads a stream event as a text delta, or `undefined` when it is not one worth + * showing. Port `reasoning` is the model's thinking; every other port, a missing + * one included, is what the task is saying, since a task may stream its output + * on a port of its own name (`code`, `summary`). An empty delta is ignored. + */ +export function classifyTextDelta(event: TextDeltaLike): ClassifiedDelta | undefined { + if (event.type !== "text-delta") return undefined; + const delta = event.textDelta ?? ""; + if (!delta) return undefined; + return { port: event.port === "reasoning" ? "reasoning" : "text", delta }; +} + +/** + * Folds one stream event into the accumulated text. Returns `prev` itself when + * nothing changed so a state setter can bail out of a re-render. + */ +export function foldTextDelta(prev: StreamedText, event: TextDeltaLike): StreamedText { + const classified = classifyTextDelta(event); + if (!classified) return prev; + return classified.port === "text" + ? { text: prev.text + classified.delta, reasoning: prev.reasoning } + : { text: prev.text, reasoning: prev.reasoning + classified.delta }; +} + +/** The last `n` lines of `text`, so a long stream never grows its panel. */ +export function tailLines(text: string, n: number): string { + const lines = text.split("\n"); + if (lines.length <= n) return text; + return lines.slice(lines.length - n).join("\n"); +} + +/** + * The thinking worth drawing, or "" when there is none. Reasoning is shown only + * until the answer starts: once text is flowing it is the thing being read, and + * the thinking behind it is no longer news. + */ +export function visibleReasoning(streamed: StreamedText): string { + return streamed.text.length > 0 ? "" : streamed.reasoning; +} diff --git a/examples/cli/src/ui/rows/ChatTaskRow.tsx b/examples/cli/src/ui/rows/ChatTaskRow.tsx index eb05416779..6e8ce603b6 100644 --- a/examples/cli/src/ui/rows/ChatTaskRow.tsx +++ b/examples/cli/src/ui/rows/ChatTaskRow.tsx @@ -9,7 +9,14 @@ import type { ITask, StreamEvent } from "@workglow/task-graph"; import { Box, Text } from "ink"; import React, { useEffect, useState, useSyncExternalStore } from "react"; import { TaskErrorDetail } from "../components/TaskErrorDetail"; +import { StreamedReasoning } from "../components/StreamedReasoning"; import { TaskStatusProgressRow } from "../components/TaskStatusProgressRow"; +import { + EMPTY_STREAMED_TEXT, + foldTextDelta, + visibleReasoning, + type StreamedText, +} from "../model/streamedText"; import type { TaskRowProps } from "./pickRenderer"; import { settledTaskDurationMs } from "./taskDuration"; import { useTaskUsageLine } from "./useTaskUsageLine"; @@ -39,6 +46,9 @@ function blockToText(block: ContentBlock): string { return `[tool_use: ${block.name}]`; case "tool_result": return "[tool_result]"; + case "reasoning": + // Kept for the model, not shown as part of the reply. + return ""; } } @@ -72,25 +82,24 @@ export function ChatTaskRow({ task, line }: TaskRowProps): React.ReactElement { () => messagesFromTask(task), () => messagesFromTask(task) ); - // `streamText` is the in-progress assistant text for the *current* turn. - // Appends on every text-delta; resets whenever history grows (meaning the - // previous turn's assistant message just landed, or a new user turn has - // started), and on stream_end. - const [streamText, setStreamText] = useState(""); + // `streamed` is the in-progress assistant text (and thinking, kept apart by + // port) for the *current* turn. Appends on every text-delta; resets whenever + // history grows (meaning the previous turn's assistant message just landed, + // or a new user turn has started), and on stream_end. + const [streamed, setStreamed] = useState(EMPTY_STREAMED_TEXT); const usageLine = useTaskUsageLine(task); useEffect(() => { const onChunk = (event: StreamEvent): void => { if (event.type === "text-delta") { - const delta = (event as { textDelta?: string; text?: string }).textDelta ?? ""; - if (delta) setStreamText((prev) => prev + delta); + setStreamed((prev) => foldTextDelta(prev, event)); } else if (event.type === "object-delta" && event.port === "messages") { // Each object-delta marks a turn boundary — clear the per-turn // streaming buffer so the next turn's text-deltas start fresh. - setStreamText(""); + setStreamed(EMPTY_STREAMED_TEXT); } }; - const onEnd = (): void => setStreamText(""); + const onEnd = (): void => setStreamed(EMPTY_STREAMED_TEXT); task.events.on("stream_chunk", onChunk); task.events.on("stream_end", onEnd); return () => { @@ -148,7 +157,8 @@ export function ChatTaskRow({ task, line }: TaskRowProps): React.ReactElement { // PROCESSING const tail = messages.slice(Math.max(0, messages.length - HISTORY_TAIL)); - const showStreamingBubble = streamText.length > 0; + const showStreamingBubble = streamed.text.length > 0; + const reasoning = visibleReasoning(streamed); return ( @@ -179,7 +189,15 @@ export function ChatTaskRow({ task, line }: TaskRowProps): React.ReactElement { assistant:{" "} - {streamText} + {streamed.text} + + )} + {reasoning.length > 0 && ( + + + assistant:{" "} + + )} diff --git a/examples/cli/src/ui/rows/StreamingTextRow.tsx b/examples/cli/src/ui/rows/StreamingTextRow.tsx index f8c5df4ffb..cf2fae78fc 100644 --- a/examples/cli/src/ui/rows/StreamingTextRow.tsx +++ b/examples/cli/src/ui/rows/StreamingTextRow.tsx @@ -7,25 +7,22 @@ import { Box, Text } from "ink"; import React from "react"; import { TaskErrorDetail } from "../components/TaskErrorDetail"; +import { StreamedReasoning } from "../components/StreamedReasoning"; import { TaskStatusProgressRow } from "../components/TaskStatusProgressRow"; +import { STREAM_PANEL_MAX_LINES, tailLines, visibleReasoning } from "../model/streamedText"; import type { TaskRowProps } from "./pickRenderer"; import { settledTaskDurationMs } from "./taskDuration"; import { useTaskStreamText } from "./useTaskStreamText"; import { useTaskUsageLine } from "./useTaskUsageLine"; -const MAX_LINES = 8; - -function tailLines(text: string, n: number): string { - const lines = text.split("\n"); - if (lines.length <= n) return text; - return lines.slice(lines.length - n).join("\n"); -} - export function StreamingTextRow({ task, line }: TaskRowProps): React.ReactElement { - const streamText = useTaskStreamText(task); + const streamed = useTaskStreamText(task); + const reasoning = visibleReasoning(streamed); const usageLine = useTaskUsageLine(task); - const isActive = line.status === "PROCESSING"; - const showPanel = isActive && streamText.length > 0; + // The first text-delta flips a task to STREAMING, so PROCESSING alone would + // hide the panel for exactly the tasks that have something to show in it. + const isActive = line.status === "PROCESSING" || line.status === "STREAMING"; + const showPanel = isActive && (streamed.text.length > 0 || reasoning.length > 0); return ( @@ -47,7 +44,11 @@ export function StreamingTextRow({ task, line }: TaskRowProps): React.ReactEleme borderColor="gray" paddingX={1} > - {tailLines(streamText, MAX_LINES)} + {streamed.text.length > 0 ? ( + {tailLines(streamed.text, STREAM_PANEL_MAX_LINES)} + ) : ( + + )} )} diff --git a/examples/cli/src/ui/rows/useTaskStreamText.ts b/examples/cli/src/ui/rows/useTaskStreamText.ts index 1f7a8b5c8f..c4292e2d7e 100644 --- a/examples/cli/src/ui/rows/useTaskStreamText.ts +++ b/examples/cli/src/ui/rows/useTaskStreamText.ts @@ -6,27 +6,25 @@ import type { ITask, StreamEvent } from "@workglow/task-graph"; import { useEffect, useState } from "react"; +import { EMPTY_STREAMED_TEXT, foldTextDelta, type StreamedText } from "../model/streamedText"; /** * Subscribes to `stream_chunk` text-delta events on a task and returns the - * accumulated text. Clears on `stream_end` or when the task status transitions - * to COMPLETED. + * accumulated answer and reasoning, kept apart by port. Clears on `stream_end` + * or when the task status transitions to COMPLETED. */ -export function useTaskStreamText(task: ITask): string { - const [text, setText] = useState(""); +export function useTaskStreamText(task: ITask): StreamedText { + const [text, setText] = useState(EMPTY_STREAMED_TEXT); useEffect(() => { const onChunk = (event: StreamEvent): void => { - if (event.type !== "text-delta") return; - const delta = event.textDelta ?? ""; - if (!delta) return; - setText((prev) => prev + delta); + setText((prev) => foldTextDelta(prev, event)); }; const onEnd = (): void => { - setText(""); + setText(EMPTY_STREAMED_TEXT); }; const onStatus = (status: string): void => { - if (status === "COMPLETED") setText(""); + if (status === "COMPLETED") setText(EMPTY_STREAMED_TEXT); }; task.events.on("stream_chunk", onChunk); diff --git a/examples/cli/src/web/client/app.css b/examples/cli/src/web/client/app.css index f9e0ac8c9b..5fe8193f88 100644 --- a/examples/cli/src/web/client/app.css +++ b/examples/cli/src/web/client/app.css @@ -1303,6 +1303,20 @@ details.adv > summary:hover { .bubble .txt { color: var(--scr-fg); } +.bubble.thinking { + border-left-color: var(--scr-faint); + background: none; +} +.bubble.thinking .r { + margin-right: 8px; + font-style: italic; + font-weight: 400; + color: var(--scr-faint); +} +.bubble.thinking .txt { + color: var(--scr-dim); + font-style: italic; +} .bubble .caret { display: inline-block; width: 7px; diff --git a/examples/cli/src/web/client/state.test.ts b/examples/cli/src/web/client/state.test.ts index 25b1b61672..ff485bf397 100644 --- a/examples/cli/src/web/client/state.test.ts +++ b/examples/cli/src/web/client/state.test.ts @@ -61,6 +61,17 @@ describe("reduceRunEvent", () => { expect(state.rows.get("t1")!.streamText).toBe("Hello"); }); + it("keeps a row's reasoning apart from its streamed text", () => { + const state = apply([ + added("t1"), + { k: "reasoning", id: "t1", delta: "Let me " }, + { k: "reasoning", id: "t1", delta: "think." }, + { k: "text", id: "t1", delta: "Hello" }, + ]); + expect(state.rows.get("t1")!.reasoningText).toBe("Let me think."); + expect(state.rows.get("t1")!.streamText).toBe("Hello"); + }); + it("drops a row the run released", () => { const state = apply([added("t1"), { k: "task_removed", id: "t1" }]); expect(state.rows.has("t1")).toBe(false); diff --git a/examples/cli/src/web/client/state.ts b/examples/cli/src/web/client/state.ts index 3921e23e3f..abac520685 100644 --- a/examples/cli/src/web/client/state.ts +++ b/examples/cli/src/web/client/state.ts @@ -26,6 +26,7 @@ export interface RunRow { readonly progress: number | undefined; readonly message: string | undefined; readonly streamText: string; + readonly reasoningText: string; readonly messages: unknown; readonly usage: { input: number; output: number; cached: number } | undefined; readonly startedAt: number | undefined; @@ -192,6 +193,7 @@ export function reduceRunEvent( progress: undefined, message: undefined, streamText: "", + reasoningText: "", messages: undefined, usage: undefined, startedAt: undefined, @@ -222,6 +224,11 @@ export function reduceRunEvent( if (!row) return state; return withRow(state, event.id, { streamText: row.streamText + event.delta }); } + case "reasoning": { + const row = state.rows.get(event.id); + if (!row) return state; + return withRow(state, event.id, { reasoningText: row.reasoningText + event.delta }); + } case "messages": return withRow(state, event.id, { messages: event.messages }); case "usage": diff --git a/examples/cli/src/web/client/views/RunConsole.tsx b/examples/cli/src/web/client/views/RunConsole.tsx index 2664e41a61..f2892dc3b9 100644 --- a/examples/cli/src/web/client/views/RunConsole.tsx +++ b/examples/cli/src/web/client/views/RunConsole.tsx @@ -21,6 +21,9 @@ import { MapView } from "./MapView"; /** Braille frames, the same family the terminal spins. */ const SPINNER = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"] as const; +/** The most of a model's thinking a row keeps on screen; older text scrolls off. */ +const REASONING_TAIL_CHARS = 600; + const RUNNING = new Set(["PROCESSING", "STREAMING", "ABORTING"]); function durationOf(row: RunRow): number | undefined { @@ -51,7 +54,10 @@ function TaskRow({ const iteration = state.iterations.get(row.id); const detail = taskDetailText(row.progress, durationOf(row), running); const showBar = running && row.progress !== undefined; - const indeterminate = running && row.progress === undefined && row.streamText.length > 0; + const indeterminate = + running && + row.progress === undefined && + (row.streamText.length > 0 || row.reasoningText.length > 0); return (
@@ -84,6 +90,15 @@ function TaskRow({
) : null} {iteration ? : null} + {row.reasoningText && !row.streamText && running ? ( +
+
+ thinking + {row.reasoningText.slice(-REASONING_TAIL_CHARS)} + +
+
+ ) : null} {row.streamText && running ? (
diff --git a/examples/web/src/graph/TaskNode.test.ts b/examples/web/src/graph/TaskNode.test.ts index ef84517535..9a36112d10 100644 --- a/examples/web/src/graph/TaskNode.test.ts +++ b/examples/web/src/graph/TaskNode.test.ts @@ -57,8 +57,15 @@ afterEach(() => { container.remove(); }); -function renderTaskNode(usageValue: Usage | undefined): void { - const task = new NoopTask(); +function renderTaskNode(usageValue: Usage | undefined, task: Task = new NoopTask()): void { + act(() => { + root = createRoot(container); + }); + rerenderTaskNode(usageValue, task); +} + +/** Renders into the existing root, as ReactFlow does when it reuses a node for another task. */ +function rerenderTaskNode(usageValue: Usage | undefined, task: Task): void { const nodeProps = { id: String(task.id), type: "task", @@ -79,7 +86,6 @@ function renderTaskNode(usageValue: Usage | undefined): void { }; act(() => { - root = createRoot(container); root.render( createElement(ReactFlowProvider, null, createElement(TaskNode, nodeProps as never)) ); @@ -98,3 +104,66 @@ describe("TaskNode per-node usage badge (the node actually rendered on the graph expect(container.querySelector(".task-node-usage")).toBeNull(); }); }); + +describe("TaskNode streaming panel", () => { + const delta = (port: string | undefined, textDelta: string) => + port === undefined + ? { type: "text-delta" as const, textDelta } + : { type: "text-delta" as const, port, textDelta }; + + it("shows reasoning in its own subdued block, above the answer", () => { + const task = new NoopTask(); + renderTaskNode(undefined, task); + act(() => { + task.emit("stream_start"); + task.emit("stream_chunk", delta("reasoning", "Weighing it.") as never); + task.emit("stream_chunk", delta("text", "The answer.") as never); + }); + + const reasoning = container.querySelector("details.task-node-reasoning"); + expect(reasoning?.textContent).toContain("Weighing it."); + expect(reasoning?.textContent).not.toContain("The answer."); + + const answer = [...container.querySelectorAll("pre")].find((pre) => !reasoning?.contains(pre)); + expect(answer?.textContent).toBe("The answer."); + expect( + reasoning!.compareDocumentPosition(answer!) & Node.DOCUMENT_POSITION_FOLLOWING + ).toBeTruthy(); + }); + + it("drops the previous task's stream when the node is reused for another task", () => { + const first = new NoopTask(); + renderTaskNode(undefined, first); + act(() => { + first.emit("stream_start"); + first.emit("stream_chunk", delta("reasoning", "First task's thinking.") as never); + first.emit("stream_chunk", delta("text", "First answer.") as never); + }); + expect(container.textContent).toContain("First task's thinking."); + + rerenderTaskNode(undefined, new NoopTask()); + expect(container.textContent).not.toContain("First task's thinking."); + expect(container.textContent).not.toContain("First answer."); + }); + + it("treats a delta with no port as the answer", () => { + const task = new NoopTask(); + renderTaskNode(undefined, task); + act(() => { + task.emit("stream_start"); + task.emit("stream_chunk", delta(undefined, "plain") as never); + }); + expect(container.querySelector("details.task-node-reasoning")).toBeNull(); + expect(container.querySelector("pre")?.textContent).toBe("plain"); + }); + + it("draws no reasoning block for a task that sends none", () => { + const task = new NoopTask(); + renderTaskNode(undefined, task); + act(() => { + task.emit("stream_start"); + task.emit("stream_chunk", delta("text", "only answer") as never); + }); + expect(container.querySelector("details.task-node-reasoning")).toBeNull(); + }); +}); diff --git a/examples/web/src/graph/TaskNode.tsx b/examples/web/src/graph/TaskNode.tsx index 7d6bafe746..228be6a064 100644 --- a/examples/web/src/graph/TaskNode.tsx +++ b/examples/web/src/graph/TaskNode.tsx @@ -49,6 +49,8 @@ export function TaskNode(props: NodeProps>) { const [streamingText, setStreamingText] = useState(""); const [isStreaming, setIsStreaming] = useState(false); const streamingTextRef = useRef(""); + const [reasoningText, setReasoningText] = useState(""); + const reasoningTextRef = useRef(""); const task = data.task; const [trackedTask, setTrackedTask] = useState(task); if (trackedTask !== task) { @@ -58,10 +60,17 @@ export function TaskNode(props: NodeProps>) { setProgressMessage(undefined); setSubTasks(task.hasChildren() ? task.subGraph.getTasks() : []); setIsExpandable(task.hasChildren()); + // What the previous task streamed is not this one's, and this one may have + // started streaming before the effect below subscribes to it. + setIsStreaming(false); + setStreamingText(""); + setReasoningText(""); } useEffect(() => { const unsubscribes: (() => void)[] = []; + streamingTextRef.current = ""; + reasoningTextRef.current = ""; // Helper to update consolidated progress const updateConsolidatedProgress = () => { @@ -93,14 +102,23 @@ export function TaskNode(props: NodeProps>) { setIsStreaming(true); setStreamingText(""); streamingTextRef.current = ""; + setReasoningText(""); + reasoningTextRef.current = ""; }) ); unsubscribes.push( task.subscribe("stream_chunk", (event: StreamEvent) => { if (event.type === "text-delta") { - streamingTextRef.current += event.textDelta; - setStreamingText(streamingTextRef.current); + // A model's thinking streams beside its answer on its own port; every + // other port, a missing one included, is what the task is saying. + if (event.port === "reasoning") { + reasoningTextRef.current += event.textDelta; + setReasoningText(reasoningTextRef.current); + } else { + streamingTextRef.current += event.textDelta; + setStreamingText(streamingTextRef.current); + } } else if (event.type === "snapshot") { const text = typeof event.data === "string" ? event.data : JSON.stringify(event.data); streamingTextRef.current = text; @@ -203,12 +221,20 @@ export function TaskNode(props: NodeProps>) { message={progressMessage} /> - {(isStreaming || (status === TaskStatus.STREAMING && streamingText)) && ( + {(isStreaming || (status === TaskStatus.STREAMING && (streamingText || reasoningText))) && (
Streaming
+ {reasoningText && ( +
+ Thinking + 
+                  {reasoningText}
+
+
+ )} 
{streamingText}
)} From e5ee4b3992731d917dfa1cec8fa592b02e7b87cf Mon Sep 17 00:00:00 2001 From: Steven Roussey Date: Mon, 12 Oct 2026 05:34:45 +0000 Subject: [PATCH 6/7] agent-eval: compare AgentTask with opencode and pi on public benchmarks Runs AgentTask, opencode and pi on Terminal-Bench 2.0, Aider Polyglot and SWE-bench Verified through Harbor, with the same model, turn cap and reasoning level for every arm, and compares them task by task. The workglow arm is AgentTask plus read, bash, edit and write tools matching pi's: bash defaults to 120 s and is capped at the trial deadline, edit applies the edits that match and names the rest, and the system prompt lists the commands installed and asks for one verification once the change is complete. Includes ATIF trajectories, retries for trials the infrastructure lost, --tasks-dir for local task copies and a scripted mock model for offline runs. --- bun.lock | 21 + examples/agent-eval/.gitignore | 2 + examples/agent-eval/README.md | 206 +++++ examples/agent-eval/harbor/workglow_agent.py | 376 +++++++++ examples/agent-eval/package.json | 39 + examples/agent-eval/src/agent/atif.ts | 150 ++++ .../agent-eval/src/agent/credentialsFile.ts | 50 ++ examples/agent-eval/src/agent/environment.ts | 61 ++ examples/agent-eval/src/agent/models.ts | 106 +++ examples/agent-eval/src/agent/providers.ts | 33 + .../agent-eval/src/agent/runCodingAgent.ts | 271 ++++++ examples/agent-eval/src/agent/runSummary.ts | 148 ++++ examples/agent-eval/src/agent/systemPrompt.ts | 79 ++ examples/agent-eval/src/agent/tools/bash.ts | 286 +++++++ .../agent-eval/src/agent/tools/codingTools.ts | 28 + .../agent-eval/src/agent/tools/context.ts | 46 + examples/agent-eval/src/agent/tools/edit.ts | 323 +++++++ .../agent-eval/src/agent/tools/fileLock.ts | 45 + examples/agent-eval/src/agent/tools/read.ts | 197 +++++ .../agent-eval/src/agent/tools/truncate.ts | 150 ++++ examples/agent-eval/src/agent/tools/write.ts | 50 ++ .../agent-eval/src/bin/workglow-agent-eval.ts | 275 ++++++ examples/agent-eval/src/bin/workglow-agent.ts | 243 ++++++ examples/agent-eval/src/compare/harborJobs.ts | 377 +++++++++ examples/agent-eval/src/compare/report.ts | 400 +++++++++ examples/agent-eval/src/compare/stats.ts | 70 ++ examples/agent-eval/src/mock/anthropicMock.ts | 224 +++++ examples/agent-eval/src/run/jobConfig.ts | 189 +++++ examples/agent-eval/src/run/runHarbor.ts | 97 +++ examples/agent-eval/src/run/suites.ts | 200 +++++ examples/agent-eval/src/test/atif.test.ts | 136 +++ examples/agent-eval/src/test/compare.test.ts | 444 ++++++++++ .../agent-eval/src/test/jobConfig.test.ts | 199 +++++ .../src/test/runCodingAgent.test.ts | 386 +++++++++ .../agent-eval/src/test/runHarbor.test.ts | 50 ++ examples/agent-eval/src/test/tools.test.ts | 796 ++++++++++++++++++ examples/agent-eval/tsconfig.json | 13 + examples/agent-eval/vitest.config.ts | 13 + 38 files changed, 6779 insertions(+) create mode 100644 examples/agent-eval/.gitignore create mode 100644 examples/agent-eval/README.md create mode 100644 examples/agent-eval/harbor/workglow_agent.py create mode 100644 examples/agent-eval/package.json create mode 100644 examples/agent-eval/src/agent/atif.ts create mode 100644 examples/agent-eval/src/agent/credentialsFile.ts create mode 100644 examples/agent-eval/src/agent/environment.ts create mode 100644 examples/agent-eval/src/agent/models.ts create mode 100644 examples/agent-eval/src/agent/providers.ts create mode 100644 examples/agent-eval/src/agent/runCodingAgent.ts create mode 100644 examples/agent-eval/src/agent/runSummary.ts create mode 100644 examples/agent-eval/src/agent/systemPrompt.ts create mode 100644 examples/agent-eval/src/agent/tools/bash.ts create mode 100644 examples/agent-eval/src/agent/tools/codingTools.ts create mode 100644 examples/agent-eval/src/agent/tools/context.ts create mode 100644 examples/agent-eval/src/agent/tools/edit.ts create mode 100644 examples/agent-eval/src/agent/tools/fileLock.ts create mode 100644 examples/agent-eval/src/agent/tools/read.ts create mode 100644 examples/agent-eval/src/agent/tools/truncate.ts create mode 100644 examples/agent-eval/src/agent/tools/write.ts create mode 100644 examples/agent-eval/src/bin/workglow-agent-eval.ts create mode 100644 examples/agent-eval/src/bin/workglow-agent.ts create mode 100644 examples/agent-eval/src/compare/harborJobs.ts create mode 100644 examples/agent-eval/src/compare/report.ts create mode 100644 examples/agent-eval/src/compare/stats.ts create mode 100644 examples/agent-eval/src/mock/anthropicMock.ts create mode 100644 examples/agent-eval/src/run/jobConfig.ts create mode 100644 examples/agent-eval/src/run/runHarbor.ts create mode 100644 examples/agent-eval/src/run/suites.ts create mode 100644 examples/agent-eval/src/test/atif.test.ts create mode 100644 examples/agent-eval/src/test/compare.test.ts create mode 100644 examples/agent-eval/src/test/jobConfig.test.ts create mode 100644 examples/agent-eval/src/test/runCodingAgent.test.ts create mode 100644 examples/agent-eval/src/test/runHarbor.test.ts create mode 100644 examples/agent-eval/src/test/tools.test.ts create mode 100644 examples/agent-eval/tsconfig.json create mode 100644 examples/agent-eval/vitest.config.ts diff --git a/bun.lock b/bun.lock index c87f621e16..100e4224fa 100644 --- a/bun.lock +++ b/bun.lock @@ -34,6 +34,25 @@ "@sqliteai/sqlite-vector-win32-x86_64": "^1.1.2", }, }, + "examples/agent-eval": { + "name": "@workglow/agent-eval", + "version": "0.6.12", + "bin": { + "workglow-agent-eval": "./dist/bin/workglow-agent-eval.js", + }, + "dependencies": { + "@workglow/ai": "workspace:*", + "@workglow/anthropic": "workspace:*", + "@workglow/deepseek": "workspace:*", + "@workglow/google-gemini": "workspace:*", + "@workglow/openai": "workspace:*", + "@workglow/openrouter": "workspace:*", + "@workglow/task-graph": "workspace:*", + "@workglow/util": "workspace:*", + "@workglow/xai": "workspace:*", + "commander": "catalog:", + }, + }, "examples/cli": { "name": "@workglow/cli", "version": "0.6.12", @@ -1763,6 +1782,8 @@ "@workglow/a2ui": ["@workglow/a2ui@workspace:packages/a2ui"], + "@workglow/agent-eval": ["@workglow/agent-eval@workspace:examples/agent-eval"], + "@workglow/ai": ["@workglow/ai@workspace:packages/ai"], "@workglow/anthropic": ["@workglow/anthropic@workspace:providers/anthropic"], diff --git a/examples/agent-eval/.gitignore b/examples/agent-eval/.gitignore new file mode 100644 index 0000000000..cdb6529f54 --- /dev/null +++ b/examples/agent-eval/.gitignore @@ -0,0 +1,2 @@ +jobs/ +__pycache__/ diff --git a/examples/agent-eval/README.md b/examples/agent-eval/README.md new file mode 100644 index 0000000000..13264da8a4 --- /dev/null +++ b/examples/agent-eval/README.md @@ -0,0 +1,206 @@ +# Workglow Agent Harness Eval + +Runs Workglow's `AgentTask` loop, [opencode](https://github.com/sst/opencode) and +[pi](https://github.com/badlogic/pi-mono) on the same public agentic benchmarks, with the +same model, and compares them task by task. + +The point is to steer work on `AgentTask`. A pass rate alone doesn't do that. What does is +seeing which tasks another harness solved and ours didn't, why ours failed, and whether a +change to the loop moves those tasks. + +## How it works + +[Harbor](https://harborframework.com) (the Terminal-Bench harness) runs everything: + +- It builds each task's container and runs the agent inside it. +- It scores the result with the task's own tests. +- It ships maintained adapters for opencode and pi. +- Its registry carries Terminal-Bench 2.0, Aider Polyglot and SWE-bench Verified, among about + seventy others. + +This package adds three pieces: + +| Piece | What it is | +| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `src/bin/workglow-agent.ts` | A headless coding agent: `AgentTask` plus four tools (`read`, `bash`, `edit`, `write`). It runs one instruction and writes a summary, an event log and an ATIF trajectory. It is bundled into one Node file (`dist/workglow-agent.mjs`) so it runs in any task container. | +| `harbor/workglow_agent.py` | The Harbor adapter. It installs Node, uploads the bundle and runs it. Because it uploads the bundle you just built, the eval measures your `libs` checkout. | +| `src/bin/workglow-agent-eval.ts` | The driver. `run` puts every arm in one Harbor job, with parity settings, then writes the comparison. `compare` reads any Harbor job directories. `mock-model` serves a scripted model for offline checks. | + +### What is held equal + +Each harness's defaults differ in ways that would decide the result, so `run` sets them +explicitly: + +| | workglow | opencode | pi | +| -------------------------------------- | -------------------------------------------------- | ------------------------------------------ | ------------------------------------ | +| Turn cap (`--max-turns`, default 100) | `max_rounds` (AgentTask's own default is 8) | `agent.build.steps` (unlimited by default) | `max_turns` (unlimited by default) | +| Reasoning (`--effort`, default `high`) | `effort` | `variant` (`--opencode-variant`) | `thinking` (pi defaults to `medium`) | +| Model | `-m provider/model`, the same string for all three | | | +| Time limit | the task's own agent timeout, enforced by Harbor | | | + +Reasoning defaults to `high` rather than `off` because "off" does not mean the same thing +everywhere. On Claude 5.x, a request without `thinking` runs adaptive thinking. pi's `off` +and opencode's default omit the field, while Workglow sends Sonnet 5.5's real off switch. + +**The tools are not held equal, deliberately.** The Workglow arm gets pi's default four, +with pi's limits: 2000 lines or 50KB per output, the tail for commands, and the head plus an +offset hint for reads. So the gap between the Workglow and pi columns measures the loop +around the tools. opencode's richer toolset (grep, glob, todo, subagents, a fuzzier edit) is +part of what its column measures. + +## Getting started + +```bash +uv tool install harbor # Harbor 0.24+; needs Docker +bun install && bun run build-example # in examples/agent-eval: bundles dist/workglow-agent.mjs + +export ANTHROPIC_API_KEY=... # whichever provider the model needs + +# Does every arm install and run? hello-world + the 10-task Terminal-Bench sample +./dist/bin/workglow-agent-eval.js run -m anthropic/claude-sonnet-5-5 -s smoke + +# The decision set: Terminal-Bench 2.0 + Polyglot (Python/JS) + SWE-bench Verified 50 +./dist/bin/workglow-agent-eval.js run -m anthropic/claude-sonnet-5-5 -s core -n 8 -k 2 +``` + +`run` writes `//comparison.md` and `comparison.json` and prints the Markdown. +To re-compare any Harbor jobs later, including ones run by hand or merged from several +machines: + +```bash +./dist/bin/workglow-agent-eval.js compare jobs/core-2026-10-06T18-00-00 [more job dirs…] +./dist/bin/workglow-agent-eval.js compare jobs/ --format json --out comparison.json +``` + +`harbor view jobs/` shows each trial's trajectory. The Workglow arm writes ATIF, so it is +drawn the same way as the other two. + +### Suites + +`./dist/bin/workglow-agent-eval.js suites` lists them: + +| Suite | Contents | Exercises | +| ---------------------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------- | +| `smoke` | `hello-world` + `terminal-bench-sample@2.0` (10) | install and plumbing | +| `terminal-bench` | Terminal-Bench 2.0 (89) | long-horizon terminal work; context growth, command timeouts, recovery | +| `polyglot` / `polyglot-lite` | Aider Polyglot, all six languages (225) / Python and JS (83) | edit-run-fix cycles; edit reliability, reading test output | +| `swebench-50` | SWE-bench Verified, first 50 | navigating a large repository, minimal patches | +| `core` | the last three together | the set to decide on | + +For any other registry dataset, use `-d name@version` (repeatable). To narrow a run, use +`--tasks 'glob,glob'` and `-l N`. Both only narrow a suite: `--tasks` keeps the tasks that match +both it and the suite's own list (`--tasks '*_001'` on `polyglot-lite` runs each language's +`_001`), and fails if none do or the overlap is not one it can express, and `-l` lowers the suite's cap but never +raises it. opencode's `--opencode-variant` defaults to the `--effort` level. + +To run task folders on disk instead — a downloaded dataset you have patched, or tasks of your +own — use `--tasks-dir ` (repeatable). A host behind a TLS-intercepting proxy needs this: +the registry tasks fetch over HTTPS while their images build, before any compose overlay can +put the proxy's CA in the container, so the CA has to go into each task's Dockerfile. + +### Ablations: testing a change to the loop + +An arm is a harness plus agent kwargs. Extra arms run in the same job, on the same tasks: + +```bash +./dist/bin/workglow-agent-eval.js run -m anthropic/claude-sonnet-5-5 -s polyglot-lite \ + -a workglow --arm workglow:tool_concurrency=8 --arm workglow:max_history_chars=400000 +``` + +The report labels arms by the kwargs that differ (`workglow[tool_concurrency=8]`), and pairs +them task by task like any other arms. For a change to `AgentTask` itself, edit `libs`, run +`bun run build-agent`, and re-run the same suite. The job config is saved beside the results, +so the before and after runs are the same experiment. + +Values are JSON where they parse as JSON. A value that holds a comma followed by `key=` goes in +double quotes: `--arm 'workglow:append_system_prompt="When x=a, set y=b"'`. + +The Workglow kwargs (`--arm workglow:k=v`) are: `effort`, `max_rounds`, +`max_tool_result_chars`, `max_history_chars`, `tool_concurrency`, `max_tokens`, +`temperature`, `round_timeout_sec`, `max_round_retries`, `command_timeout_sec`, +`append_system_prompt`, `concise`, `images`, `replay_reasoning`, `time_budget`, `time_left`, +`review`, `max_duration_sec`. opencode takes `opencode_config` and `variant`. pi takes +`thinking` and `max_turns`. + +- `time_budget` (default on): the adapter reads the trial's agent time limit and passes it, + less a reserve of 10% (30 s to 120 s), as `--max-duration-sec`. The loop then ends cleanly with + `stopReason: "budget"` and writes its summary before Harbor would cancel it. Harbor gives the + opencode and pi arms no such signal; this is a harness capability, not a benchmark setting. +- `time_left` (default on): the model sees `[Time left: 7m 12s]` after each round's tool results. +- `review` (default on): the first time the model tries to finish, it is asked once to check its + work against the instruction's exact requirements. +- `max_duration_sec`: an explicit budget, overriding `time_budget`. + +The budget stops the loop only between rounds, so the adapter also passes `--stop-after-sec` +a little under Harbor's limit: the agent then stops outright, a model call in flight included, +records the run as `budget`, and writes its summary and trajectory before Harbor cancels it. + +`bash` now defaults to a 120 s timeout (was 600 s), and never runs past the deadline. +Its commands run without the agent's credential variables (`*_API_KEY`, `*_TOKEN`, `*_SECRET` +and the like), so the model cannot print its own key into the transcript or the logs. + +Filtering the shell's environment is not enough on its own: a process's starting environment +stays readable at `/proc//environ`, and every command the model runs descends from the +agent. So under Harbor the keys never enter that environment. The adapter writes them to a +`0600` file in an agent-only directory, passes only its path (`--credentials-file`), and the +agent loads them after it starts and deletes the file before any tool can run. + +## Reading the report + +- **Pass rate (95% CI)** uses a Wilson interval. Arms run on tens of tasks, so expect wide + intervals. Use the head-to-head table to tell two arms apart, not the overlap of their + intervals. +- **Head to head** pairs the arms on the tasks both ran. A task solved by both or by neither + says nothing about which arm is better. The p-value is an exact sign test (McNemar) over the + tasks where the arms differ, and those tasks are listed by name. Read their transcripts + first. +- **Blocked** trials are excluded from every rate. These are trials where the container or + the agent's install failed before the agent saw the task. Counting them as failures would + charge a harness for a flaky registry. +- **Why runs failed** sorts failures into modes: `wrong-answer`, `agent-timeout`, + `context-overflow`, `max-rounds`, `budget`, `rate-limit`, `provider-error`, `agent-crash`, + `verifier-error`. The Workglow arm also reports outcomes, rounds, retries and tool error + counts from the loop itself. +- **Cost.** Each harness prices its own runs. For the Workglow arm, that is `AgentTask`'s + per-round step records. When a custom endpoint is configured, pi runs it as a provider with + no price card, so its cost is blank. + +## Offline check (no key, no bill) + +`mock-model` serves a scripted Anthropic Messages API. Each reply calls the harness's `bash` +tool with the next scripted command, and replies in text once the script is used up. All +three harnesses can target an Anthropic base URL, so one mock serves the whole pipeline: + +```bash +./dist/bin/workglow-agent-eval.js mock-model -p 18089 \ + -c "echo 'Hello, world!' > hello.txt" --request-log requests.jsonl & +ANTHROPIC_BASE_URL=http://127.0.0.1:18089 ANTHROPIC_API_KEY=mock \ + ./dist/bin/workglow-agent-eval.js run -m anthropic/claude-sonnet-5-5 -d hello-world@1.0 \ + --harbor-config hostnet.yaml +``` + +The task container has to reach the mock. On Linux, point a Harbor config at a compose +overlay that sets `network_mode: host` for the `main` service: + +```yaml +environment: + extra_docker_compose: [./hostnet.compose.yaml] +``` + +`requests.jsonl` captures what each harness actually sends the model: the system prompt, the +tool schemas, the thinking settings and the cache markers. That is the quickest way to see +how the three differ before spending anything. + +The same mock drives `src/test/runCodingAgent.test.ts`, which runs the real provider, the +real `AgentTask` and the real tools in process. + +## Running the Workglow agent by itself + +```bash +node dist/workglow-agent.mjs -m anthropic/claude-sonnet-5-5 --cwd ./repo \ + --logs-dir ./logs --effort high "Fix the failing test in src/parse.ts" +``` + +Providers: `anthropic`, `openai`, `google`/`gemini`, `deepseek`, `xai`, `openrouter`. Keys +are read from the usual environment variables, and a custom endpoint from `_BASE_URL`; +`--credentials-file ` loads them from a JSON object instead and deletes the file. diff --git a/examples/agent-eval/harbor/workglow_agent.py b/examples/agent-eval/harbor/workglow_agent.py new file mode 100644 index 0000000000..eac9afc643 --- /dev/null +++ b/examples/agent-eval/harbor/workglow_agent.py @@ -0,0 +1,376 @@ +# @license +# Copyright 2026 Steven Roussey +# SPDX-License-Identifier: Apache-2.0 +"""Harbor adapter for the headless Workglow coding agent (AgentTask). + +Run it beside Harbor's built-in ``opencode`` and ``pi`` agents: + + harbor run -d terminal-bench@2.0 -m anthropic/claude-sonnet-5-5 \\ + -a workglow_agent:WorkglowAgent + +with this directory on ``PYTHONPATH``. It installs Node in the task container, +uploads the single-file agent bundle (``dist/workglow-agent.mjs``, built by +``bun run build-agent``), and runs it on the task instruction. The bundle is +uploaded rather than installed from a registry so a run always measures the +``libs`` checkout it was built from: change AgentTask, rebuild, re-run. +""" + +import json +import os +import shlex +import tempfile +import time +from pathlib import Path, PurePosixPath +from typing import Annotated, Any, override + +from pydantic import Field + +from harbor.agents.capabilities import AgentCapabilities +from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template +from harbor.agents.installed.node_install import nvm_node_install_snippet +from harbor.agents.model_connection import ModelConnectionSpec +from harbor.agents.options import Cli, InstalledAgentOptions +from harbor.environments.base import BaseEnvironment +from harbor.models.agent.context import AgentContext +from harbor.models.task.task import Task +from harbor.models.trial.config import TrialConfig + +_REMOTE_DIR = PurePosixPath("/installed-agent") +_REMOTE_BUNDLE = _REMOTE_DIR / "workglow-agent.mjs" +_REMOTE_INSTRUCTION = _REMOTE_DIR / "instruction.md" +# Owned by the agent user and closed to everyone else, so the agent can delete +# the credentials file it reads from here. +_REMOTE_SECRETS_DIR = _REMOTE_DIR / "secrets" +_REMOTE_CREDENTIALS = _REMOTE_SECRETS_DIR / "credentials.json" +_DEFAULT_BUNDLE = Path(__file__).resolve().parent.parent / "dist" / "workglow-agent.mjs" +_NODE_PREFIX = "[ -f ~/.nvm/nvm.sh ] && . ~/.nvm/nvm.sh; " +# The longest timer Node can set, in whole seconds; the agent refuses anything longer. +_MAX_TIMER_SEC = 2_147_483 + + +class WorkglowOptions(InstalledAgentOptions): + """Agent kwargs (``--ak name=value``), passed through as CLI flags. + + Every knob that changes AgentTask's behaviour is here so a job can run an + ablation — the same model and tasks with one setting changed — without a + rebuild. + """ + + effort: Annotated[str | None, Cli("--effort")] = Field( + default=None, + description="Reasoning effort: none|low|medium|high|extra|ultra (or off|minimal|xhigh|max).", + ) + max_rounds: Annotated[int | None, Cli("--max-rounds")] = Field( + default=None, ge=1, description="Model calls before the run stops (agent default 100)." + ) + max_tool_result_chars: Annotated[int | None, Cli("--max-tool-result-chars")] = Field( + default=None, ge=1, description="Characters of one tool result the model is shown." + ) + max_history_chars: Annotated[int | None, Cli("--max-history-chars")] = Field( + default=None, ge=1, description="History budget per round (AgentTask default if unset)." + ) + tool_concurrency: Annotated[int | None, Cli("--tool-concurrency")] = Field( + default=None, ge=1, description="Tool calls of one round run at once." + ) + max_tokens: Annotated[int | None, Cli("--max-tokens")] = Field( + default=None, ge=1, description="Output tokens per model call." + ) + temperature: Annotated[float | None, Cli("--temperature")] = Field( + default=None, description="Sampling temperature." + ) + round_timeout_sec: Annotated[int | None, Cli("--round-timeout-sec")] = Field( + default=None, + ge=1, + le=_MAX_TIMER_SEC, + description="Abandon and retry a model call after this long.", + ) + max_round_retries: Annotated[int | None, Cli("--max-round-retries")] = Field( + default=None, ge=0, description="Retries of a failed model call." + ) + command_timeout_sec: Annotated[int | None, Cli("--command-timeout-sec")] = Field( + default=None, ge=1, le=_MAX_TIMER_SEC, description="Default bash timeout." + ) + append_system_prompt: Annotated[str | None, Cli("--append-system-prompt")] = Field( + default=None, description="Text appended to the system prompt." + ) + concise: Annotated[str | None, Cli("--concise")] = Field( + default=None, description="on|off: ask for terse replies between tool calls (default on)." + ) + images: Annotated[str | None, Cli("--images")] = Field( + default=None, description="on|off: let read show images to the model (default on)." + ) + replay_reasoning: Annotated[str | None, Cli("--replay-reasoning")] = Field( + default=None, + description="on|off: send the model's reasoning back on later rounds (default on).", + ) + max_duration_sec: Annotated[int | None, Cli("--max-duration-sec")] = Field( + default=None, + ge=1, + description="Wall-clock budget; set from the trial's time limit unless given.", + ) + time_budget: str | None = Field( + default=None, + pattern="^(on|off)$", + description="on|off: give the agent the trial's time limit, less a reserve (default on).", + ) + time_left: Annotated[str | None, Cli("--time-left")] = Field( + default=None, + pattern="^(on|off)$", + description="on|off: show the model its remaining time (default on)." + ) + review: Annotated[str | None, Cli("--review")] = Field( + default=None, + pattern="^(on|off)$", + description="on|off: one check against the instructions before finishing (default on).", + ) + bundle_path: str | None = Field( + default=None, + description="Local path of workglow-agent.mjs (default: ../dist, or $WORKGLOW_AGENT_BUNDLE).", + ) + + +class WorkglowAgent(BaseInstalledAgent): + capabilities = AgentCapabilities(atif=True) + MODEL_CONNECTION = ModelConnectionSpec(passthrough=True) + + options_model = WorkglowOptions + options: WorkglowOptions + + _OUTPUT_FILENAME = "workglow.txt" + _SUMMARY_FILENAME = "workglow-summary.json" + + @staticmethod + @override + def name() -> str: + return "workglow" + + @override + def get_version_command(self) -> str | None: + return f"{_NODE_PREFIX}node {shlex.quote(_REMOTE_BUNDLE.as_posix())} --version" + + def _agent_timeout_sec(self) -> float | None: + """The time Harbor gives run(), resolved as the trial resolves it. + + Harbor hands an agent no deadline: it cancels run() when the time is + up. The trial's config and task sit next to the logs directory, so the + agent can learn its limit and stop cleanly, with its summary written, + before that. + """ + try: + config = TrialConfig.model_validate_json( + (self.logs_dir.parent / "config.json").read_text() + ) + base = config.agent.override_timeout_sec or Task( + config.task.get_local_path() + ).config.agent.timeout_sec + except (OSError, ValueError) as error: + self.logger.warning( + "Could not resolve the trial's agent time limit (%s); the workglow agent " + "runs without a time budget.", + error, + ) + return None + if base is None: + return None + multiplier = ( + config.agent_timeout_multiplier + if config.agent_timeout_multiplier is not None + else config.timeout_multiplier + ) + return min(base, config.agent.max_timeout_sec or float("inf")) * multiplier + + def _max_duration_sec(self) -> int | None: + if self.options.max_duration_sec is not None: + return self.options.max_duration_sec + if self.options.time_budget == "off": + return None + limit = self._agent_timeout_sec() + if limit is None: + return None + # The loop stops only between rounds; the reserve covers the round in + # flight and writing the summary, and never takes more than half a + # short limit. + # The budget is whole seconds and must end before the limit, so the shortest + # limit that leaves a one-second budget is two seconds. + if limit < 2: + raise ValueError( + f"the trial's agent time limit is {limit:.3f}s; a timed run needs at least 2s" + ) + reserve = min(120.0, max(30.0, limit * 0.1), limit / 2) + return max(1, int(limit - reserve)) + + def _stop_after_sec(self) -> int | None: + """When the agent stops outright, a little before Harbor cancels run(). + + The time budget stops the loop only between rounds, so a model call in + flight at Harbor's limit would be cancelled with it and the summary and + trajectory never written. Stopping first, with time left to write them, + keeps the trial's record whatever round was running. Set whenever the + limit is known, ``time_budget`` or not: it changes no behaviour before + the limit. + """ + limit = self._agent_timeout_sec() + if limit is None or limit < 2: + return None + margin = min(10.0, limit / 4) + return min(_MAX_TIMER_SEC, max(1, int(limit - margin))) + + def _bundle(self) -> Path: + configured = self.options.bundle_path or os.environ.get("WORKGLOW_AGENT_BUNDLE") + bundle = Path(configured).expanduser() if configured else _DEFAULT_BUNDLE + if not bundle.is_file(): + raise FileNotFoundError( + f"workglow agent bundle not found at {bundle}; run `bun run build-agent` in " + "examples/agent-eval, or pass --ak bundle_path=..." + ) + return bundle + + @override + async def install(self, environment: BaseEnvironment) -> None: + bundle = self._bundle() + # Only what nvm needs. Distro Node is not asked for: on glibc images nvm + # installs its own, and asking apt for `nodejs` fails the whole install + # on images whose package index has gone stale. + await self.ensure_system_dependencies(environment, ("curl", "bash", "ca_certificates")) + # The bundle needs Node 22+. On glibc, install it with nvm the way the + # opencode and pi adapters do; on musl nvm's binaries do not run, so + # the packaged Node is all there is. + await self.exec_as_root( + environment, + command=( + "if ldd --version 2>&1 | grep -qi musl || [ -f /etc/alpine-release ]; then " + "command -v node >/dev/null || apk add --no-cache nodejs; fi" + ), + ) + await self.exec_as_agent( + environment, + command=( + "set -euo pipefail; " + "if ldd --version 2>&1 | grep -qi musl || [ -f /etc/alpine-release ]; then " + "node --version; " + # The image's own Node can be older than the bundle accepts; say so + # here, as a setup failure, rather than as every trial crashing. + "node -e 'process.exit(Number(process.versions.node.split(\".\")[0]) >= 22 ? 0 : 1)' " + "|| { echo \"workglow agent needs Node 22+, found $(node --version)\" >&2; exit 1; }; " + f"else {nvm_node_install_snippet()}; fi" + ), + ) + await self.exec_as_root(environment, command=f"mkdir -p {_REMOTE_DIR.as_posix()}") + await self._upload_agent_owned_file(environment, bundle, _REMOTE_BUNDLE.as_posix()) + await self.exec_as_agent( + environment, + command=f"{_NODE_PREFIX}node {shlex.quote(_REMOTE_BUNDLE.as_posix())} --version", + ) + + @override + @with_prompt_template + async def run( + self, + instruction: str, + environment: BaseEnvironment, + context: AgentContext, + ) -> None: + # Harbor's limit runs from here, not from when the agent process starts. + entered = time.monotonic() + if not self.model_name or "/" not in self.model_name: + raise ValueError("Model name must be in the format provider/model_name") + + # Through a file, not argv: a long instruction is past what a shell + # command line safely carries. + self.logs_dir.mkdir(parents=True, exist_ok=True) + local_instruction = self.logs_dir / "instruction.md" + local_instruction.write_text(instruction) + await self._upload_agent_owned_file( + environment, local_instruction, _REMOTE_INSTRUCTION.as_posix() + ) + + await self._upload_credentials(environment) + try: + logs_dir = self.environment_logs_dir.as_posix() + flags = self.build_cli_flags() + # Both are relative to the agent's own start, so the time setup took + # (the uploads above) comes off them. + spent = int(time.monotonic() - entered) + if self.options.max_duration_sec is None: + budget = self._max_duration_sec() + if budget is not None: + flags = f"{flags} --max-duration-sec {max(1, budget - spent)}".strip() + stop = self._stop_after_sec() + if stop is not None: + flags = f"{flags} --stop-after-sec {max(1, stop - spent)}".strip() + await self.exec_as_agent( + environment, + command=( + # Without pipefail the run's status is `tee`'s, and an agent that + # crashed would be recorded as one that finished. + "set -o pipefail; " + f"{_NODE_PREFIX}" + f"node {shlex.quote(_REMOTE_BUNDLE.as_posix())} " + f"--model {shlex.quote(self.model_name)} " + f"--instruction-file {shlex.quote(_REMOTE_INSTRUCTION.as_posix())} " + f"--credentials-file {shlex.quote(_REMOTE_CREDENTIALS.as_posix())} " + f'--cwd "$PWD" --logs-dir {shlex.quote(logs_dir)} ' + f"{flags + ' ' if flags else ''}" + f"2>&1 None: + """Hands the agent its API keys in a file rather than in its environment. + + A process's starting environment stays readable at ``/proc//environ`` + for its whole life, and every command the model runs is a descendant of + the shell and the Node process this adapter starts. Keys passed in the + ``exec`` environment could be read back by the model however the bash + tool filters its own; the agent loads the file into its environment after + it starts, where that file does not show it, and deletes it. + """ + secrets_dir = shlex.quote(_REMOTE_SECRETS_DIR.as_posix()) + user = environment.default_user + owner = f"chown {shlex.quote(str(user))} {secrets_dir} && " if user is not None else "" + await self.exec_as_root( + environment, command=f"mkdir -p {secrets_dir} && {owner}chmod 700 {secrets_dir}" + ) + # Never under logs_dir: that directory is kept with the job's results. + with tempfile.TemporaryDirectory(prefix="workglow-agent-") as temp_dir: + local = Path(temp_dir) / "credentials.json" + fd = os.open(local, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(fd, "w") as handle: + json.dump(dict(self.model_connection.env), handle) + await self._upload_agent_owned_file(environment, local, _REMOTE_CREDENTIALS.as_posix()) + await self.exec_as_root( + environment, command=f"chmod 600 {shlex.quote(_REMOTE_CREDENTIALS.as_posix())}" + ) + + def _summary(self) -> dict[str, Any] | None: + path = self.logs_dir / self._SUMMARY_FILENAME + try: + return json.loads(path.read_text()) + except (OSError, json.JSONDecodeError): + return None + + @override + def populate_context_post_run(self, context: AgentContext) -> None: + summary = self._summary() + if summary is None: + return + usage = summary.get("usage") or {} + # A round the provider reported no usage for makes every sum an + # undercount; leaving them unset is what the report marks as partial. + if not usage.get("unreportedRounds"): + context.n_input_tokens = usage.get("promptTokens") + context.n_output_tokens = usage.get("outputTokens") + # A provider that never reported a cache count did not report zero. + if not usage.get("cacheUnreportedRounds"): + context.n_cache_tokens = usage.get("cacheReadTokens") + context.cost_usd = summary.get("costUsd") + # The comparison reads outcome, rounds and tool tallies from here. + context.metadata = { + "workglow": {key: value for key, value in summary.items() if key != "finalText"} + } diff --git a/examples/agent-eval/package.json b/examples/agent-eval/package.json new file mode 100644 index 0000000000..65075dbb59 --- /dev/null +++ b/examples/agent-eval/package.json @@ -0,0 +1,39 @@ +{ + "name": "@workglow/agent-eval", + "private": true, + "type": "module", + "version": "0.6.12", + "repository": { + "type": "git", + "url": "https://github.com/workglow-dev/libs.git", + "directory": "examples/agent-eval" + }, + "bugs": { + "url": "https://github.com/workglow-dev/libs/issues" + }, + "description": "Agent harness eval: run Workglow's AgentTask, opencode and pi on the same public agentic benchmarks through Harbor, and compare them task by task.", + "homepage": "https://workglow.dev", + "scripts": { + "build-example": "concurrently -c 'auto' -n 'agent,cli,types' 'bun run build-agent' 'bun run build-cli' 'bun run build-types'", + "build-clean": "rm -fr dist/* tsconfig.tsbuildinfo", + "build-agent": "bun build --target=node --format=esm --outfile ./dist/workglow-agent.mjs ./src/bin/workglow-agent.ts", + "build-cli": "bun build --target=bun --packages=external --outdir ./dist/bin ./src/bin/workglow-agent-eval.ts", + "build-types": "rm -f tsconfig.tsbuildinfo && tsc --noEmit", + "test": "vitest run --config ../../vitest.config.ts --project agent-eval" + }, + "bin": { + "workglow-agent-eval": "./dist/bin/workglow-agent-eval.js" + }, + "dependencies": { + "@workglow/ai": "workspace:*", + "@workglow/anthropic": "workspace:*", + "@workglow/deepseek": "workspace:*", + "@workglow/google-gemini": "workspace:*", + "@workglow/openai": "workspace:*", + "@workglow/openrouter": "workspace:*", + "@workglow/task-graph": "workspace:*", + "@workglow/util": "workspace:*", + "@workglow/xai": "workspace:*", + "commander": "catalog:" + } +} diff --git a/examples/agent-eval/src/agent/atif.ts b/examples/agent-eval/src/agent/atif.ts new file mode 100644 index 0000000000..55b90eff92 --- /dev/null +++ b/examples/agent-eval/src/agent/atif.ts @@ -0,0 +1,150 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AgentStep, ChatMessage, ContentBlock } from "@workglow/ai"; +import type { AgentRunSummary } from "./runSummary"; +import { hasTokenCounts } from "./runSummary"; + +/** + * A run as an ATIF trajectory (Harbor's Agent Trajectory Interchange Format), + * so `harbor view` draws a workglow trial the way it draws opencode's and + * pi's. Only the fields Harbor reads are filled. + */ +export interface AtifTrajectory { + readonly schema_version: "ATIF-v1.8"; + readonly session_id: string; + readonly agent: { readonly name: string; readonly version: string; readonly model_name: string }; + readonly steps: readonly AtifStep[]; + readonly final_metrics: { + readonly total_prompt_tokens: number; + readonly total_completion_tokens: number; + readonly total_cached_tokens: number | null; + readonly total_cost_usd: number | null; + readonly total_steps: number; + }; +} + +interface AtifStep { + readonly step_id: number; + readonly timestamp?: string | undefined; + readonly source: "user" | "agent"; + readonly message: string; + readonly model_name?: string | undefined; + readonly tool_calls?: + | ReadonlyArray<{ tool_call_id: string; function_name: string; arguments: unknown }> + | undefined; + readonly observation?: + | { readonly results: ReadonlyArray<{ source_call_id: string; content: string }> } + | undefined; + readonly metrics?: + | { + readonly prompt_tokens: number | null; + readonly completion_tokens: number | null; + readonly cached_tokens: number | null; + readonly cost_usd: number | null; + } + | undefined; +} + +function textOf(blocks: readonly ContentBlock[]): string { + return blocks + .map((block) => (block.type === "text" ? block.text : "")) + .filter((text) => text.length > 0) + .join("\n"); +} + +function resultText(block: Extract): string { + return block.content + .map((part) => (part.type === "text" ? part.text : `[${part.type}]`)) + .join(""); +} + +export function toAtifTrajectory( + sessionId: string, + version: string, + messages: readonly ChatMessage[], + steps: readonly AgentStep[], + summary: AgentRunSummary +): AtifTrajectory { + const out: AtifStep[] = []; + let round = 0; + for (let i = 0; i < messages.length; i++) { + const message = messages[i]!; + if (message.role === "user") { + out.push({ step_id: out.length + 1, source: "user", message: textOf(message.content) }); + continue; + } + if (message.role !== "assistant") continue; + // A round that produced neither text nor a call left no assistant message. + while (round < steps.length && steps[round]!.tools.length === 0 && steps[round]!.text === "") { + round++; + } + const step = steps[round++]; + const calls = message.content.filter( + (block): block is Extract => block.type === "tool_use" + ); + const next = messages[i + 1]; + const results = + next?.role === "tool" + ? next.content + .filter( + (block): block is Extract => + block.type === "tool_result" + ) + .map((block) => ({ source_call_id: block.tool_use_id, content: resultText(block) })) + : []; + const usage = step?.usage; + out.push({ + step_id: out.length + 1, + timestamp: step?.startedAt, + source: "agent", + message: textOf(message.content), + model_name: summary.model, + ...(calls.length > 0 + ? { + tool_calls: calls.map((call) => ({ + tool_call_id: call.id, + function_name: call.name, + arguments: call.input, + })), + } + : {}), + ...(results.length > 0 ? { observation: { results } } : {}), + ...(hasTokenCounts(usage) + ? { + metrics: { + prompt_tokens: + (usage.input ?? 0) + + (usage.imageInput ?? 0) + + (usage.cached ?? 0) + + (usage.imageCached ?? 0) + + (usage.cacheWrite ?? 0), + completion_tokens: usage.output ?? null, + cached_tokens: + usage.cached === undefined && usage.imageCached === undefined + ? null + : (usage.cached ?? 0) + (usage.imageCached ?? 0), + cost_usd: step?.costUsd ?? null, + }, + } + : {}), + }); + } + return { + schema_version: "ATIF-v1.8", + session_id: sessionId, + agent: { name: "workglow", version, model_name: summary.model }, + steps: out, + final_metrics: { + total_prompt_tokens: summary.usage.promptTokens, + total_completion_tokens: summary.usage.outputTokens, + total_cached_tokens: + summary.usage.cacheUnreportedRounds > 0 ? null : summary.usage.cacheReadTokens, + total_cost_usd: summary.costUsd ?? null, + total_steps: out.length, + }, + }; +} diff --git a/examples/agent-eval/src/agent/credentialsFile.ts b/examples/agent-eval/src/agent/credentialsFile.ts new file mode 100644 index 0000000000..de22f74737 --- /dev/null +++ b/examples/agent-eval/src/agent/credentialsFile.ts @@ -0,0 +1,50 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { readFileSync, unlinkSync } from "node:fs"; + +/** + * Moves the variables in a JSON file into `env` and deletes the file, returning + * the names it set. + * + * This is how the Harbor adapter hands the agent its API keys. A process's + * starting environment stays readable at `/proc//environ` for its whole + * life, and every command the model runs through `bash` is a descendant of the + * agent, so a key passed in the environment of the command that starts it can be + * read back by the model however the shell's own environment is filtered. + * Variables set after start are not in that file. The file is deleted before any + * tool can run, and a file that cannot be deleted stops the run: left in place, + * the keys would be one `cat` away. + */ +export function loadCredentialsFile(path: string, env: NodeJS.ProcessEnv = process.env): string[] { + const text = readFileSync(path, "utf8"); + // Deleted before it is parsed, so a malformed file does not outlive the run either. + try { + unlinkSync(path); + } catch (error) { + throw new Error( + `could not delete the credentials file ${path} after reading it: ${(error as Error).message}` + ); + } + let parsed: unknown; + try { + parsed = JSON.parse(text); + } catch { + throw new Error(`the credentials file ${path} is not valid JSON`); + } + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) { + throw new Error(`the credentials file ${path} is not a JSON object`); + } + const names: string[] = []; + for (const [name, value] of Object.entries(parsed)) { + if (typeof value !== "string") { + throw new Error(`the credentials file ${path} has a non-string value for ${name}`); + } + env[name] = value; + names.push(name); + } + return names; +} diff --git a/examples/agent-eval/src/agent/environment.ts b/examples/agent-eval/src/agent/environment.ts new file mode 100644 index 0000000000..662fc28177 --- /dev/null +++ b/examples/agent-eval/src/agent/environment.ts @@ -0,0 +1,61 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { accessSync, constants, statSync } from "node:fs"; +import { delimiter, join } from "node:path"; + +/** + * Commands a coding task commonly reaches for. A benchmark image ships some and + * not others (`python3` without `python`, no `rg`), and a model that guesses + * spends a round on "command not found" each time. + */ +export const PROBED_COMMANDS: readonly string[] = [ + "python3", + "python", + "pip3", + "pip", + "node", + "npm", + "git", + "rg", + "gcc", + "g++", + "make", + "cmake", + "cargo", + "go", + "java", + "file", + "xxd", + "strings", + "7zz", + "7z", +]; + +export interface CommandInventory { + readonly found: readonly string[]; + readonly missing: readonly string[]; +} + +function isExecutableFile(file: string): boolean { + try { + accessSync(file, constants.X_OK); + return statSync(file).isFile(); + } catch { + return false; + } +} + +/** Which of `commands` resolve to an executable on `path` (a PATH-style list). */ +export function probeCommands(commands: readonly string[], path: string): CommandInventory { + const dirs = path.split(delimiter).filter((dir) => dir.length > 0); + const found: string[] = []; + const missing: string[] = []; + for (const command of commands) { + (dirs.some((dir) => isExecutableFile(join(dir, command))) ? found : missing).push(command); + } + return { found, missing }; +} diff --git a/examples/agent-eval/src/agent/models.ts b/examples/agent-eval/src/agent/models.ts new file mode 100644 index 0000000000..1ccf5d24d1 --- /dev/null +++ b/examples/agent-eval/src/agent/models.ts @@ -0,0 +1,106 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ModelConfig, ModelEffort } from "@workglow/ai"; +import { isModelEffort } from "@workglow/ai"; +import { getAnthropicModelPricing } from "@workglow/anthropic/ai"; +import { getDeepSeekModelPricing } from "@workglow/deepseek/ai"; +import { getGeminiModelPricing } from "@workglow/google-gemini/ai"; +import { getOpenAiModelPricing } from "@workglow/openai/ai"; +import { getXaiModelPricing } from "@workglow/xai/ai"; + +interface ProviderSpec { + readonly provider: string; + /** Environment variables naming a custom endpoint, first set wins. */ + readonly baseUrlEnvs: readonly string[]; + readonly pricing?: ((model: string) => ModelConfig["pricing"]) | undefined; +} + +/** + * Provider prefixes as opencode, pi and Harbor spell them, so one `--model` + * string means the same model to all three harnesses. + */ +const PROVIDERS: Readonly> = { + anthropic: { + provider: "ANTHROPIC", + baseUrlEnvs: ["ANTHROPIC_BASE_URL"], + pricing: getAnthropicModelPricing, + }, + openai: { + provider: "OPENAI", + baseUrlEnvs: ["OPENAI_BASE_URL", "OPENAI_API_BASE"], + pricing: getOpenAiModelPricing, + }, + google: { provider: "GOOGLE_GEMINI", baseUrlEnvs: [], pricing: getGeminiModelPricing }, + gemini: { provider: "GOOGLE_GEMINI", baseUrlEnvs: [], pricing: getGeminiModelPricing }, + deepseek: { + provider: "DEEPSEEK", + baseUrlEnvs: ["DEEPSEEK_BASE_URL"], + pricing: getDeepSeekModelPricing, + }, + xai: { provider: "XAI", baseUrlEnvs: ["XAI_BASE_URL"], pricing: getXaiModelPricing }, + openrouter: { provider: "OPENROUTER", baseUrlEnvs: ["OPENROUTER_BASE_URL"] }, +}; + +export const SUPPORTED_PROVIDERS = Object.keys(PROVIDERS); + +/** + * Reasoning levels as pi and opencode name them, mapped onto the coarse + * {@link ModelEffort} dial, so a run configured for one harness reads the same + * here. + */ +const EFFORT_ALIASES: Readonly> = { + off: "none", + minimal: "low", + xhigh: "extra", + max: "ultra", +}; + +export function parseEffort(value: string | undefined): ModelEffort | undefined { + if (value === undefined || value === "") return undefined; + const effort = EFFORT_ALIASES[value] ?? value; + if (!isModelEffort(effort)) { + throw new Error( + `unknown effort "${value}" — use none|low|medium|high|extra|ultra (or off|minimal|xhigh|max)` + ); + } + return effort; +} + +/** + * `provider/model` (Harbor's `-m`) to an inline {@link ModelConfig}. A custom + * endpoint in the provider's `*_BASE_URL` variable is trusted: the operator set + * it for this run, which is the case the allow-list exists to make explicit. + */ +export function resolveAgentModel( + id: string, + options: { readonly effort?: ModelEffort | undefined; readonly env?: NodeJS.ProcessEnv } = {} +): ModelConfig { + const slash = id.indexOf("/"); + if (slash <= 0 || slash === id.length - 1) { + throw new Error(`model must be provider/model (e.g. anthropic/claude-sonnet-4-5), got "${id}"`); + } + const prefix = id.slice(0, slash).toLowerCase(); + const model = id.slice(slash + 1); + const spec = PROVIDERS[prefix]; + if (spec === undefined) { + throw new Error( + `unsupported provider "${prefix}" — use one of ${SUPPORTED_PROVIDERS.join(", ")}` + ); + } + const env = options.env ?? process.env; + const baseUrl = spec.baseUrlEnvs.map((name) => env[name]).find((value) => !!value); + const pricing = spec.pricing?.(model); + return { + provider: spec.provider, + provider_config: { + model_name: model, + ...(baseUrl ? { base_url: baseUrl, trustedBaseUrl: true } : {}), + }, + ...(pricing ? { pricing } : {}), + ...(options.effort ? { effort: options.effort } : {}), + } as ModelConfig; +} diff --git a/examples/agent-eval/src/agent/providers.ts b/examples/agent-eval/src/agent/providers.ts new file mode 100644 index 0000000000..4e27c63d8e --- /dev/null +++ b/examples/agent-eval/src/agent/providers.ts @@ -0,0 +1,33 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { registerAnthropicInline } from "@workglow/anthropic/ai-runtime"; +import { registerDeepSeekInline } from "@workglow/deepseek/ai-runtime"; +import { registerGeminiInline } from "@workglow/google-gemini/ai-runtime"; +import { registerOpenAiInline } from "@workglow/openai/ai-runtime"; +import { registerOpenRouterInline } from "@workglow/openrouter/ai-runtime"; +import { EnvCredentialStore, setGlobalCredentialStore } from "@workglow/util"; +import { registerXaiInline } from "@workglow/xai/ai-runtime"; + +const REGISTRATIONS: Readonly Promise>> = { + ANTHROPIC: () => registerAnthropicInline(), + OPENAI: () => registerOpenAiInline(), + GOOGLE_GEMINI: () => registerGeminiInline(), + DEEPSEEK: () => registerDeepSeekInline(), + XAI: () => registerXaiInline(), + OPENROUTER: () => registerOpenRouterInline(), +}; + +/** + * Registers the one provider a run needs, in process. Keys come from the + * environment, as every harness in the comparison reads them. + */ +export async function registerAgentProvider(provider: string): Promise { + setGlobalCredentialStore(new EnvCredentialStore()); + const register = REGISTRATIONS[provider]; + if (register === undefined) throw new Error(`no registration for provider ${provider}`); + await register(); +} diff --git a/examples/agent-eval/src/agent/runCodingAgent.ts b/examples/agent-eval/src/agent/runCodingAgent.ts new file mode 100644 index 0000000000..9ac8e67535 --- /dev/null +++ b/examples/agent-eval/src/agent/runCodingAgent.ts @@ -0,0 +1,271 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { + AgentStep, + AgentTaskInput, + AgentTaskOutput, + ChatMessage, + ModelConfig, +} from "@workglow/ai"; +import { AGENT_APPROVAL_OPT_OUT, AgentTask, measureHistoryForModel } from "@workglow/ai"; +import type { StreamEvent } from "@workglow/task-graph"; +import { globalServiceRegistry, ServiceRegistry } from "@workglow/util"; +import { PROBED_COMMANDS, probeCommands } from "./environment"; +import type { AgentRunOutcome, AgentRunSummary } from "./runSummary"; +import { summarizeUsage, tallyTools, totalCost } from "./runSummary"; +import { codingSystemPrompt } from "./systemPrompt"; +import type { CodingToolContext } from "./tools/context"; +import { createCodingTools } from "./tools/codingTools"; + +/** + * Rounds a benchmark run gets unless told otherwise. AgentTask's own default + * (8) is sized for a chat turn; a terminal task routinely takes dozens. + */ +export const DEFAULT_BENCH_MAX_ROUNDS = 100; +/** + * Room for one full tool output (50KB) plus its truncation notice, so the + * loop's own clamp never cuts a result the tool already sized. + */ +export const DEFAULT_BENCH_MAX_TOOL_RESULT_CHARS = 60_000; + +/** One line of the run's event log. */ +export type AgentRunEvent = + | { readonly type: "start"; readonly at: string; readonly model: string; readonly cwd: string } + | { + readonly type: "tool-call"; + readonly at: string; + readonly status: string; + readonly id: string; + readonly name: string | undefined; + readonly input?: unknown; + readonly result?: string | undefined; + } + | { readonly type: "round"; readonly at: string; readonly step: AgentStep } + | { readonly type: "finish"; readonly at: string; readonly summary: AgentRunSummary }; + +/** The knobs a Harbor job passes through as agent kwargs, for ablations. */ +export interface CodingAgentSettings { + readonly maxRounds: number; + readonly maxToolResultChars: number; + readonly maxHistoryChars: number | undefined; + readonly toolConcurrency: number | undefined; + readonly maxTokens: number | undefined; + readonly temperature: number | undefined; + readonly maxDurationMs: number | undefined; + readonly roundTimeoutMs: number | undefined; + readonly maxRoundRetries: number | undefined; + readonly systemPromptAppend: string | undefined; + /** Ask for terse replies between tool calls. */ + readonly concise: boolean; + /** Show the model its remaining time each round; needs maxDurationMs. */ + readonly announceTimeLeft: boolean; + /** Sent once when the model first tries to finish; undefined sends none. */ + readonly reviewPrompt: string | undefined; +} + +export interface CodingAgentRun { + readonly instruction: string; + /** `provider/model`, as reported. */ + readonly modelId: string; + readonly model: ModelConfig; + readonly tools: CodingToolContext; + readonly settings: CodingAgentSettings; + readonly signal: AbortSignal; + /** + * Stops the run outright this long after it starts, ending it as `budget`. The + * loop's own `maxDurationMs` stops only between rounds, so a model call in + * flight at the outer limit would run on until whatever runs the agent kills + * it, and the summary and trajectory would never be written. + */ + readonly stopAfterMs?: number | undefined; + readonly onEvent?: ((event: AgentRunEvent) => void) | undefined; +} + +export interface CodingAgentResult { + readonly summary: AgentRunSummary; + readonly messages: readonly ChatMessage[]; + readonly steps: readonly AgentStep[]; +} + +function errorText(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +function trimForLog(text: string | undefined, max = 4000): string | undefined { + if (text === undefined || text.length <= max) return text; + return `${text.slice(0, max)}… [${text.length} chars]`; +} + +/** + * One benchmark attempt: the instruction as the only user message, AgentTask + * running the four coding tools until the model stops calling them, a limit + * runs out, or it fails. + * + * The transcript is kept from the loop's own snapshots rather than from its + * final output, so a run that dies mid-way — a context-window overflow is + * the usual one — still leaves every round it completed on the record. + */ +export async function runCodingAgent(run: CodingAgentRun): Promise { + const now = (): string => new Date().toISOString(); + const emit = (event: AgentRunEvent): void => run.onEvent?.(event); + const startedAt = Date.now(); + emit({ type: "start", at: now(), model: run.modelId, cwd: run.tools.cwd }); + + const registry = new ServiceRegistry(globalServiceRegistry.container.createChildContainer()); + // Headless: no person to approve a call, and every tool is a host function. + registry.registerInstance(AGENT_APPROVAL_OPT_OUT, true); + + let messages: readonly ChatMessage[] = []; + let steps: readonly AgentStep[] = []; + let reportedRounds = 0; + const reportRounds = (): void => { + for (; reportedRounds < steps.length; reportedRounds++) { + emit({ type: "round", at: now(), step: steps[reportedRounds]! }); + } + }; + const task = new AgentTask(); + // Counted off the loop's own progress messages, since `steps` holds only the + // rounds that settled: a round that exhausted its retries leaves no step, and + // its attempts would otherwise be missing from the summary. + let roundsStarted = 0; + let retriesSeen = 0; + const offProgress = task.subscribe("progress", (_progress, message) => { + if (message === "Thinking") roundsStarted++; + else if (message?.startsWith("Thinking (attempt ")) retriesSeen++; + }); + const off = task.subscribe("stream_chunk", (event: StreamEvent) => { + if (event.type === "snapshot") { + const data = event.data as { messages?: ChatMessage[]; steps?: AgentStep[] }; + if (data.messages) messages = data.messages; + if (data.steps) { + steps = data.steps; + reportRounds(); + } + } else if (event.type === "tool-call") { + const call = event as unknown as { + status: string; + toolCallId: string; + name?: string; + input?: unknown; + result?: string; + }; + emit({ + type: "tool-call", + at: now(), + status: call.status, + id: call.toolCallId, + name: call.name, + ...(call.input === undefined ? {} : { input: call.input }), + ...(call.result === undefined ? {} : { result: trimForLog(call.result) }), + }); + } + }); + + const settings = run.settings; + const input: AgentTaskInput = { + model: run.model, + prompt: run.instruction, + systemPrompt: + codingSystemPrompt({ + cwd: run.tools.cwd, + platform: `${process.platform} ${process.arch}`, + date: new Date().toISOString().slice(0, 10), + concise: settings.concise, + images: run.tools.images, + commands: probeCommands(PROBED_COMMANDS, process.env.PATH ?? ""), + }) + (settings.systemPromptAppend ? `\n\n${settings.systemPromptAppend}` : ""), + tools: createCodingTools(run.tools), + maxRounds: settings.maxRounds, + maxToolResultChars: settings.maxToolResultChars, + approval: "never", + ...(settings.maxHistoryChars === undefined + ? {} + : { maxHistoryChars: settings.maxHistoryChars }), + ...(settings.toolConcurrency === undefined + ? {} + : { toolConcurrency: settings.toolConcurrency }), + ...(settings.maxTokens === undefined ? {} : { maxTokens: settings.maxTokens }), + ...(settings.temperature === undefined ? {} : { temperature: settings.temperature }), + ...(settings.maxDurationMs === undefined ? {} : { maxDurationMs: settings.maxDurationMs }), + ...(settings.announceTimeLeft && settings.maxDurationMs !== undefined + ? { announceTimeLeft: true } + : {}), + ...(settings.reviewPrompt === undefined ? {} : { reviewPrompt: settings.reviewPrompt }), + ...(settings.roundTimeoutMs === undefined ? {} : { roundTimeoutMs: settings.roundTimeoutMs }), + ...(settings.maxRoundRetries === undefined + ? {} + : { maxRoundRetries: settings.maxRoundRetries }), + }; + + const stopper = new AbortController(); + let stoppedAtLimit = false; + const stopAfterMs = run.stopAfterMs; + const stopTimer = + stopAfterMs === undefined + ? undefined + : setTimeout(() => { + stoppedAtLimit = true; + stopper.abort(new Error(`stopped at the hard time limit of ${stopAfterMs} ms`)); + }, stopAfterMs); + const signal = AbortSignal.any([run.signal, stopper.signal]); + + let outcome: AgentRunOutcome; + let error: string | undefined; + let finalText = ""; + let rounds = 0; + try { + const output = (await task.run(input, { registry, signal })) as AgentTaskOutput; + messages = output.messages; + steps = output.steps; + outcome = output.stopReason; + rounds = output.rounds; + // Only a round that called no tool answered; the narration beside a call is + // not an answer when the turn ran out of rounds or budget after it. + const last = steps.at(-1); + finalText = last !== undefined && last.tools.length === 0 ? last.text : ""; + } catch (caught) { + outcome = stoppedAtLimit ? "budget" : run.signal.aborted ? "aborted" : "error"; + // The provider reports its own cancellation; the limit is the reason it ran into. + error = errorText(stoppedAtLimit ? stopper.signal.reason : caught); + rounds = Math.max(steps.length, roundsStarted); + } finally { + clearTimeout(stopTimer); + off(); + offProgress(); + } + // The last round, when it ends the turn without a call, is in the output + // but was never in a snapshot. + reportRounds(); + + // A round that started and never settled (it failed, was stopped, or ended the + // turn while waiting to retry) has no step, though its request may have been + // billed. Its usage and cost are unknown, so the totals are reported as + // incomplete rather than as the settled rounds' subtotal. + const unsettled = rounds > steps.length; + const usage = summarizeUsage(steps); + const summary: AgentRunSummary = { + harness: "workglow", + model: run.modelId, + outcome, + error, + rounds, + retries: Math.max( + retriesSeen, + steps.reduce((sum, step) => sum + Math.max(0, step.attempts - 1), 0) + ), + durationMs: Date.now() - startedAt, + modelMs: steps.reduce((sum, step) => sum + step.modelMs, 0), + usage: unsettled ? { ...usage, unreportedRounds: usage.unreportedRounds + 1 } : usage, + costUsd: unsettled ? undefined : totalCost(steps), + tools: tallyTools(steps), + finalHistoryChars: measureHistoryForModel(messages), + finalText, + settings: { ...settings }, + }; + emit({ type: "finish", at: now(), summary }); + return { summary, messages, steps }; +} diff --git a/examples/agent-eval/src/agent/runSummary.ts b/examples/agent-eval/src/agent/runSummary.ts new file mode 100644 index 0000000000..25bbe148cc --- /dev/null +++ b/examples/agent-eval/src/agent/runSummary.ts @@ -0,0 +1,148 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AgentStep } from "@workglow/ai"; + +/** How a run ended: the loop's own stop reason, or the error that ended it. */ +export type AgentRunOutcome = + | "answered" + | "submitted" + | "budget" + | "max-rounds" + | "error" + | "aborted"; + +export interface AgentRunUsage { + /** Prompt tokens not served from cache. */ + readonly inputTokens: number; + readonly cacheReadTokens: number; + readonly cacheWriteTokens: number; + readonly outputTokens: number; + /** Every prompt token, cached or not — what Harbor reports as input. */ + readonly promptTokens: number; + /** Rounds whose provider reported no usage at all. */ + readonly unreportedRounds: number; + /** + * Rounds that reported token counts but no cache-read count. Such a provider + * did not say it read nothing from cache, so `cacheReadTokens` is a floor. + */ + readonly cacheUnreportedRounds: number; +} + +export interface AgentToolTally { + readonly calls: number; + readonly errors: number; + readonly byName: Readonly>; +} + +/** + * Everything the comparison reads about one workglow run, written to + * `summary.json` beside the transcript. The Harbor adapter copies the token + * and cost totals into its trial result; the rest rides along as metadata. + */ +export interface AgentRunSummary { + readonly harness: "workglow"; + readonly model: string; + readonly outcome: AgentRunOutcome; + readonly error: string | undefined; + readonly rounds: number; + /** Model calls that failed and were retried. */ + readonly retries: number; + readonly durationMs: number; + readonly modelMs: number; + readonly usage: AgentRunUsage; + /** Undefined when any round could not be priced. */ + readonly costUsd: number | undefined; + readonly tools: AgentToolTally; + /** + * Size of the transcript, in characters, as AgentTask's history budget + * measures it: a reasoning block's replay `payload` is left out, so the + * figure compares with `maxHistoryChars` rather than with encrypted reasoning. + */ + readonly finalHistoryChars: number; + readonly finalText: string; + readonly settings: Readonly>; +} + +/** + * Whether a round's usage carries any token count. One with none (a provider + * that reported only a cost) is as unreported as no usage at all: summing it + * would count the round as zero tokens. + */ +export function hasTokenCounts( + usage: AgentStep["usage"] +): usage is NonNullable { + return ( + usage !== undefined && + [ + usage.input, + usage.output, + usage.cached, + usage.cacheWrite, + usage.imageInput, + usage.imageCached, + ].some((n) => n !== undefined) + ); +} + +export function summarizeUsage(steps: readonly AgentStep[]): AgentRunUsage { + let inputTokens = 0; + let cacheReadTokens = 0; + let cacheWriteTokens = 0; + let outputTokens = 0; + let unreportedRounds = 0; + let cacheUnreportedRounds = 0; + for (const step of steps) { + const usage = step.usage; + if (!hasTokenCounts(usage)) { + unreportedRounds++; + continue; + } + if (usage.cached === undefined && usage.imageCached === undefined) cacheUnreportedRounds++; + inputTokens += (usage.input ?? 0) + (usage.imageInput ?? 0); + cacheReadTokens += (usage.cached ?? 0) + (usage.imageCached ?? 0); + cacheWriteTokens += usage.cacheWrite ?? 0; + outputTokens += usage.output ?? 0; + } + return { + inputTokens, + cacheReadTokens, + cacheWriteTokens, + outputTokens, + promptTokens: inputTokens + cacheReadTokens + cacheWriteTokens, + unreportedRounds, + cacheUnreportedRounds, + }; +} + +export function tallyTools(steps: readonly AgentStep[]): AgentToolTally { + // Keyed by names the model chose, so no prototype: `__proto__` is a name too. + const byName: Record = Object.create(null); + let calls = 0; + let errors = 0; + for (const step of steps) { + for (const tool of step.tools) { + const entry = (byName[tool.name] ??= { calls: 0, errors: 0 }); + entry.calls++; + calls++; + if (tool.isError) { + entry.errors++; + errors++; + } + } + } + return { calls, errors, byName }; +} + +/** Sum of round costs, or undefined as soon as one round has none. */ +export function totalCost(steps: readonly AgentStep[]): number | undefined { + let total = 0; + for (const step of steps) { + if (step.costUsd === undefined) return undefined; + total += step.costUsd; + } + return steps.length > 0 ? total : undefined; +} diff --git a/examples/agent-eval/src/agent/systemPrompt.ts b/examples/agent-eval/src/agent/systemPrompt.ts new file mode 100644 index 0000000000..26cced5d8d --- /dev/null +++ b/examples/agent-eval/src/agent/systemPrompt.ts @@ -0,0 +1,79 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { CommandInventory } from "./environment"; + +export interface SystemPromptEnvironment { + readonly cwd: string; + readonly platform: string; + readonly date: string; + /** Ask for terse replies between tool calls; off reproduces the earlier prompt. */ + readonly concise: boolean; + /** Whether `read` shows images, which the tools line then says. */ + readonly images: boolean; + /** Commands probed on PATH; undefined leaves the line out. */ + readonly commands: CommandInventory | undefined; +} + +/** + * A deliberately small prompt in the shape of pi's: who the agent is, one line + * per tool, the rules that keep tool calls from failing, and where it is. It + * says nothing a benchmark could game; it is the prompt a host would ship. + */ +export function codingSystemPrompt(env: SystemPromptEnvironment): string { + return [ + "You are an expert software engineer working autonomously in a terminal. You complete the " + + "task you are given by reading files, running commands, editing code and writing files. " + + "No one will answer questions: make reasonable decisions and finish the task.", + "", + "", + env.images + ? "- read: read file contents (use offset/limit to page through long files); shows you images (png, jpg, gif, webp)" + : "- read: read file contents (use offset/limit to page through long files)", + "- bash: run shell commands (ls, grep, find, tests, builds, package managers)", + "- edit: replace exact text in a file; several disjoint edits per call", + "- write: create a file or overwrite it completely", + "", + "", + "", + "- Read a file before editing it; oldText must match the current file exactly.", + "- Prefer edit for changes to existing files; use write for new files or full rewrites.", + "- Each bash call is a fresh shell; chain dependent commands with &&.", + "- Verify once the change is complete: run the code or its tests then, not after every edit.", + "- When the task is complete, reply with a short summary and no tool calls.", + ...(env.concise + ? [ + // Narration between calls is output the model pays for and nobody reads. + "- Be concise: do not narrate between tool calls; say only what the next step needs.", + ] + : []), + "", + "", + "", + `cwd: ${env.cwd}`, + `platform: ${env.platform}`, + `date: ${env.date}`, + ...(env.commands === undefined + ? [] + : [ + `commands on PATH: ${env.commands.found.join(", ") || "(none of the usual ones)"}`, + ...(env.commands.missing.length > 0 + ? [`not installed: ${env.commands.missing.join(", ")}`] + : []), + ]), + "", + ].join("\n"); +} + +/** + * Sent once when the model first tries to finish. The tests are often hidden, + * and the requirement a correct-looking answer misses is an exact string the + * instruction named (an error message, an output format, a file name). + */ +export const REVIEW_PROMPT = + "Before you finish, reread the task instructions. Check each exact requirement they state " + + "(names, messages, output formats, file paths, values) against what you built, and fix " + + "any mismatch. Then reply with your summary."; diff --git a/examples/agent-eval/src/agent/tools/bash.ts b/examples/agent-eval/src/agent/tools/bash.ts new file mode 100644 index 0000000000..ff4c904b51 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/bash.ts @@ -0,0 +1,286 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { ToolCallError } from "@workglow/ai"; +import { uuid4 } from "@workglow/util"; +import { spawn } from "node:child_process"; +import { mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import type { CodingToolContext } from "./context"; +import { optionalInteger, requireString } from "./context"; +import { truncateTail } from "./truncate"; + +/** Output held in memory per command; the rest is counted, not kept. */ +const MAX_CAPTURE_BYTES = 16 * 1024 * 1024; +/** Grace between SIGTERM and SIGKILL for a command being stopped. */ +const KILL_GRACE_MS = 2000; +/** Wait after the shell exits for output still in the pipes. */ +const DRAIN_MS = 250; +/** The longest delay a Node timer holds; a longer one fires at once. */ +const MAX_TIMER_MS = 2_147_483_647; +/** The longest timeout a command can ask for: the longest timer Node can set. */ +export const MAX_TIMEOUT_SEC = Math.floor(MAX_TIMER_MS / 1000); + +export interface CommandResult { + readonly output: string; + readonly exitCode: number | null; + readonly timedOut: boolean; + readonly droppedBytes: number; +} + +/** + * Names of variables holding a credential: `*_API_KEY`, `*_TOKEN` (`HF_TOKEN`, + * `ANTHROPIC_AUTH_TOKEN`), `*_SECRET`, `*_SECRET_KEY`, `*_ACCESS_KEY`, `*_PASSWORD`. + */ +const CREDENTIAL_ENV = /(?:_API_KEY|_TOKEN|_SECRET|_SECRET_KEY|_ACCESS_KEY|_PASSWORD)$/i; + +/** + * The environment a command runs in: the agent's own, minus the credentials it + * reads its model's key from. The model writes the commands, so a key left here + * is one `printenv` away from the transcript and the logs. + */ +export function shellEnv(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv { + return Object.fromEntries(Object.entries(env).filter(([name]) => !CREDENTIAL_ENV.test(name))); +} + +/** + * Runs `command` under `bash -c` in its own process group, so a timeout or an + * abort stops everything it started — a test runner's workers, a server it + * backgrounded — rather than only the shell. + */ +export function runCommand( + command: string, + cwd: string, + timeoutMs: number, + signal: AbortSignal +): Promise { + return new Promise((resolve, reject) => { + // An abort that landed before the call would never fire the listener below. + if (signal.aborted) { + reject(signal.reason ?? new Error("aborted")); + return; + } + const child = spawn("bash", ["-c", command], { + cwd, + detached: true, + stdio: ["ignore", "pipe", "pipe"], + env: shellEnv(), + }); + const chunks: Buffer[] = []; + let held = 0; + let dropped = 0; + // The end of a command's output is where its failures and summaries are, so + // past the cap the oldest bytes go, never the newest. + const take = (chunk: Buffer): void => { + chunks.push(chunk); + held += chunk.length; + while (held > MAX_CAPTURE_BYTES) { + const excess = held - MAX_CAPTURE_BYTES; + const first = chunks[0]!; + if (first.length <= excess) { + chunks.shift(); + held -= first.length; + dropped += first.length; + } else { + chunks[0] = first.subarray(excess); + held -= excess; + dropped += excess; + } + } + }; + child.stdout.on("data", take); + child.stderr.on("data", take); + + let timedOut = false; + // Set once the command is told to stop. The command is over only when its + // whole group is: a process it started can ignore SIGTERM and outlive the + // shell, and returning while it still runs would let the next tool call + // race it. So the result waits for the group to go, and sends SIGKILL to + // whatever is left once the grace period is up. + let stopped: Promise | undefined; + const stop = (): void => { + if (child.pid === undefined || stopped !== undefined) return; + const group = child.pid; + try { + process.kill(-group, "SIGTERM"); + } catch { + // Already gone. + } + stopped = new Promise((done) => { + const since = Date.now(); + const check = (): void => { + if (!groupAlive(group)) return done(); + if (Date.now() - since >= KILL_GRACE_MS) { + try { + process.kill(-group, "SIGKILL"); + } catch { + // Already gone. + } + return done(); + } + setTimeout(check, 50); + }; + check(); + }); + }; + const timer = setTimeout( + () => { + timedOut = true; + stop(); + }, + Math.min(timeoutMs, MAX_TIMER_MS) + ); + const onAbort = (): void => stop(); + signal.addEventListener("abort", onAbort, { once: true }); + + let settled = false; + const finish = (code: number | null): void => { + if (settled) return; + settled = true; + clearTimeout(timer); + signal.removeEventListener("abort", onAbort); + child.stdout.destroy(); + child.stderr.destroy(); + if (signal.aborted) { + reject(signal.reason ?? new Error("aborted")); + return; + } + resolve({ + output: Buffer.concat(chunks).toString("utf8"), + exitCode: code, + timedOut, + droppedBytes: dropped, + }); + }; + child.on("error", (error) => { + if (settled) return; + settled = true; + clearTimeout(timer); + signal.removeEventListener("abort", onAbort); + reject(error); + }); + // A command that backgrounds a process (`server &`) leaves it holding the + // pipes, and "close" would wait for it. The shell exiting is the command + // finishing; give the pipes a moment to drain, then stop listening. + // The command is over once the shell exits, so the timeout is too: it must + // not fire during the drain and report a finished command as timed out. + const finishOnceStopped = (code: number | null): void => { + if (stopped === undefined) finish(code); + else void stopped.then(() => finish(code)); + }; + child.on("exit", (code) => { + clearTimeout(timer); + setTimeout(() => finishOnceStopped(code), DRAIN_MS); + }); + child.on("close", (code) => finishOnceStopped(code)); + }); +} + +/** Whether any process in a group is still there (a zombie counts until reaped). */ +function groupAlive(group: number): boolean { + try { + process.kill(-group, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code === "EPERM"; + } +} + +/** + * The timeout a command runs under: what was asked for, cut to the whole + * seconds the task has left. The tool runs nothing with under a second left, + * so the floor of one second never reaches past the deadline. + */ +export function commandTimeoutSec( + requested: number, + deadline: number | undefined, + now: number +): { readonly sec: number; readonly capped: boolean } { + if (deadline === undefined) return { sec: requested, capped: false }; + const left = Math.max(1, Math.floor((deadline - now) / 1000)); + return left < requested ? { sec: left, capped: true } : { sec: requested, capped: false }; +} + +export function createBashTool(context: CodingToolContext): ToolDefinition { + return { + name: "bash", + description: + "Run a bash command in the working directory and return its combined stdout and stderr. " + + "Each call is a fresh shell: cd and exported variables do not carry over, so chain " + + "commands with && when they depend on each other. Output is truncated to the last 2000 " + + `lines or 50KB; the full output is saved to a file named in the result. Commands time out ` + + `after ${context.defaultCommandTimeoutSec}s unless you pass a larger timeout (seconds), ` + + "and never run past the time left for the task: run long jobs in the background and poll. " + + "Do not start interactive programs.", + inputSchema: { + type: "object", + properties: { + command: { type: "string", description: "The command to run" }, + timeout: { + type: "integer", + minimum: 1, + maximum: MAX_TIMEOUT_SEC, + description: "Timeout in seconds", + }, + }, + required: ["command"], + additionalProperties: false, + }, + requiresApproval: false, + execute: async (input, { signal, deadline }) => { + const command = requireString(input, "command"); + const requested = optionalInteger(input, "timeout") ?? context.defaultCommandTimeoutSec; + if (requested < 1) { + throw new ToolCallError(`"timeout" must be at least 1 second; the command was not run.`); + } + // A longer timer cannot be set; refused rather than silently shortened. + if (requested > MAX_TIMEOUT_SEC) { + throw new ToolCallError( + `"timeout" can be at most ${MAX_TIMEOUT_SEC} seconds; the command was not run.` + ); + } + if (deadline !== undefined && deadline - Date.now() < 1000) { + throw new ToolCallError( + "No time is left for this task; the command was not run. Finish with what you have." + ); + } + const { sec: timeoutSec, capped } = commandTimeoutSec(requested, deadline, Date.now()); + const result = await runCommand(command, context.cwd, timeoutSec * 1000, signal); + + const tail = truncateTail(result.output); + let text = tail.text; + if (tail.truncated || result.droppedBytes > 0) { + await mkdir(context.spillDir, { recursive: true }); + const spill = join(context.spillDir, `bash-${uuid4()}.log`); + await writeFile(spill, result.output, "utf8"); + const lost = + result.droppedBytes > 0 ? `, ${result.droppedBytes} earlier bytes discarded` : ""; + text = + `[Output truncated: showing the last ${tail.keptLines} of ${tail.totalLines} lines` + + `${tail.partialLine ? ", the first of them only its end" : ""}. ` + + `Full output (${result.droppedBytes > 0 ? "last " : ""}${result.output.length} characters${lost}) saved to ${spill}]\n` + + text; + } + if (text.length === 0) text = "(no output)"; + if (result.timedOut) { + throw new ToolCallError( + capped + ? `${text}\n\nCommand was killed after ${timeoutSec}s, all the time left for this ` + + "task. Finish with what you have." + : `${text}\n\nCommand timed out after ${timeoutSec}s and was killed. ` + + "Pass a larger timeout, or run it in the background and poll." + ); + } + if (result.exitCode !== 0) { + throw new ToolCallError( + `${text}\n\nCommand exited with code ${result.exitCode ?? "unknown"}` + ); + } + return text; + }, + }; +} diff --git a/examples/agent-eval/src/agent/tools/codingTools.ts b/examples/agent-eval/src/agent/tools/codingTools.ts new file mode 100644 index 0000000000..0763c84323 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/codingTools.ts @@ -0,0 +1,28 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { createBashTool } from "./bash"; +import type { CodingToolContext } from "./context"; +import { createEditTool } from "./edit"; +import { createReadTool } from "./read"; +import { createWriteTool } from "./write"; + +/** + * The toolset the eval hands AgentTask: read, bash, edit, write — pi's default + * four, with its limits. Holding the tools to pi's means a gap between the + * Workglow and pi columns is a gap in the loop around them; opencode's richer + * set (grep, glob, todo, subagents, fuzzier edit) is part of what its column + * measures. + */ +export function createCodingTools(context: CodingToolContext): ToolDefinition[] { + return [ + createReadTool(context), + createBashTool(context), + createEditTool(context), + createWriteTool(context), + ]; +} diff --git a/examples/agent-eval/src/agent/tools/context.ts b/examples/agent-eval/src/agent/tools/context.ts new file mode 100644 index 0000000000..d865dcf06b --- /dev/null +++ b/examples/agent-eval/src/agent/tools/context.ts @@ -0,0 +1,46 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { isAbsolute, resolve } from "node:path"; + +/** What every coding tool is bound to. */ +export interface CodingToolContext { + /** Directory relative paths and commands resolve against. */ + readonly cwd: string; + /** Directory a command's full output is spilled to when the model sees only its tail. */ + readonly spillDir: string; + /** Seconds a command may run when the model does not say. */ + readonly defaultCommandTimeoutSec: number; + /** Whether `read` returns images to the model, or refuses them as binary. */ + readonly images: boolean; +} + +/** + * Where a path the model wrote points. Absolute paths are honoured: benchmark + * instructions routinely name `/app/…`, and the sandbox is the container, not + * the working directory. + */ +export function resolvePath(context: CodingToolContext, path: string): string { + const trimmed = path.trim(); + const expanded = trimmed.startsWith("~/") + ? `${process.env.HOME ?? "/root"}${trimmed.slice(1)}` + : trimmed; + return isAbsolute(expanded) ? expanded : resolve(context.cwd, expanded); +} + +export function requireString(input: Record, key: string): string { + const value = input[key]; + if (typeof value !== "string") throw new Error(`"${key}" must be a string`); + return value; +} + +export function optionalInteger(input: Record, key: string): number | undefined { + const value = input[key]; + if (value === undefined || value === null) return undefined; + const n = typeof value === "string" ? Number(value) : value; + if (typeof n !== "number" || !Number.isFinite(n)) throw new Error(`"${key}" must be a number`); + return Math.trunc(n); +} diff --git a/examples/agent-eval/src/agent/tools/edit.ts b/examples/agent-eval/src/agent/tools/edit.ts new file mode 100644 index 0000000000..f641f86a53 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/edit.ts @@ -0,0 +1,323 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { ToolCallError } from "@workglow/ai"; +import { readFile, stat, writeFile } from "node:fs/promises"; +import type { CodingToolContext } from "./context"; +import { requireString, resolvePath } from "./context"; +import { withFileLock } from "./fileLock"; +import { MAX_READ_BYTES } from "./read"; + +export interface TextEdit { + readonly oldText: string; + readonly newText: string; +} + +interface Range { + readonly start: number; + readonly end: number; + readonly fuzzy: boolean; +} + +/** + * One line as the fuzzy pass compares it: compatibility-normalized, with + * typographic quotes, dashes and spaces folded to ASCII and trailing + * whitespace dropped. These are the differences a model introduces when it + * retypes a line it read, and none of them is a difference the author meant. + */ +export function normalizeLine(line: string): string { + return line + .normalize("NFKC") + .replace(/[‘’‚‛]/g, "'") + .replace(/[“”„‟]/g, '"') + .replace(/[‐-―−]/g, "-") + .replace(/[  -​  　]/g, " ") + .trimEnd(); +} + +function exactRanges(content: string, needle: string): number[] { + const found: number[] = []; + let at = content.indexOf(needle); + while (at !== -1) { + found.push(at); + at = content.indexOf(needle, at + 1); + } + return found; +} + +/** + * Whole-line matches of `oldText` once both sides are normalized. Matching + * whole lines keeps the mapping back to the original text exact: the range is + * the original lines, whatever the normalization changed inside them. + */ +function fuzzyRanges(content: string, oldText: string): Range[] { + const trailingNewline = oldText.endsWith("\n"); + const body = trailingNewline ? oldText.slice(0, -1) : oldText; + const wanted = body.split("\n").map(normalizeLine); + // Knuth–Morris–Pratt over normalized lines: one pass, each file line + // normalized once, so a needle of many repeated lines against a file of them + // costs file + needle rather than file × needle. The file is walked in place + // (a file within the size limit can hold millions of lines); only `wanted`, + // and where its last few candidate lines start, are kept. + const fallback = new Array(wanted.length).fill(0); + for (let i = 1, k = 0; i < wanted.length; i++) { + while (k > 0 && wanted[i] !== wanted[k]) k = fallback[k - 1]!; + if (wanted[i] === wanted[k]) k++; + fallback[i] = k; + } + const starts = new Array(wanted.length).fill(0); + const ranges: Range[] = []; + let matched = 0; + // Lines are the fields `content.split("\n")` would give, the empty one after + // a final newline included. + for (let at = 0, index = 0; at <= content.length; index++) { + const newline = content.indexOf("\n", at); + const lineEnd = newline === -1 ? content.length : newline; + const line = normalizeLine(content.slice(at, lineEnd)); + starts[index % wanted.length] = at; + while (matched > 0 && line !== wanted[matched]) matched = fallback[matched - 1]!; + if (line === wanted[matched]) matched++; + if (matched === wanted.length) { + const start = starts[(index - wanted.length + 1) % wanted.length]!; + // A file ending in a newline splits into a last empty field that is no + // line at all; a match reaching it claimed a blank line the file does not have. + const pastLastLine = content.endsWith("\n") && at === content.length; + let end = lineEnd; + if (trailingNewline && end < content.length) end += 1; + // Blank lines matched against the end of a file that has none (an empty + // file asked for "\n") consume nothing: the edit would insert, not replace. + if (!pastLastLine && end !== start) ranges.push({ start, end, fuzzy: true }); + matched = fallback[matched - 1]!; + } + at = newline === -1 ? content.length + 1 : newline + 1; + } + return ranges; +} + +/** + * Where one edit applies in the original text. Exact first, then the fuzzy + * line match; either way the match must be unique, because guessing which of + * two identical blocks the model meant is how an edit lands in the wrong place. + */ +function locate(content: string, edit: TextEdit, path: string, index: number): Range { + const label = `edits[${index}]`; + if (edit.oldText.length === 0) { + throw new ToolCallError(`${label}: oldText is empty. Use write to create a file.`); + } + const exact = exactRanges(content, edit.oldText); + if (exact.length === 1) { + return { start: exact[0]!, end: exact[0]! + edit.oldText.length, fuzzy: false }; + } + if (exact.length > 1) { + throw new ToolCallError( + `${label}: found ${exact.length} occurrences of oldText in ${path}. ` + + "Include more surrounding lines so it matches exactly one place." + ); + } + const fuzzy = fuzzyRanges(content, edit.oldText); + if (fuzzy.length === 1) return fuzzy[0]!; + if (fuzzy.length > 1) { + throw new ToolCallError( + `${label}: found ${fuzzy.length} places matching oldText in ${path} (ignoring trailing ` + + "whitespace and typographic quotes). Include more surrounding lines so it matches one." + ); + } + throw new ToolCallError( + `${label}: could not find oldText in ${path}. It must match the file exactly, including ` + + "indentation. Read the file again to see its current content." + ); +} + +export interface EditOutcome { + readonly content: string; + /** Applied edits that matched only after normalizing whitespace and quotes. */ + readonly fuzzy: number; + /** Indices of the edits written. */ + readonly applied: readonly number[]; + /** Indices of edits whose newText equals their oldText: nothing to do. */ + readonly unchanged: readonly number[]; + /** Why each remaining edit was not applied, as `edits[i]: reason`. */ + readonly failures: readonly string[]; +} + +/** + * Applies every edit that can be applied, each located against the ORIGINAL + * text so one edit cannot move another's match, and reports the rest. Failing + * the whole call for one bad entry costs the model a round resending the good + * ones; this way it resends only what failed. An edit overlapping one earlier + * in the list is refused rather than resolved in some order. + */ +export function applyEdits(content: string, edits: readonly TextEdit[], path: string): EditOutcome { + const unchanged: number[] = []; + const failures: string[] = []; + const located: Array<{ + readonly index: number; + readonly edit: TextEdit; + readonly range: Range; + }> = []; + edits.forEach((edit, index) => { + if (edit.oldText.length > 0 && edit.oldText === edit.newText) { + unchanged.push(index); + return; + } + try { + const range = locate(content, edit, path, index); + const clash = located.find( + (other) => range.start < other.range.end && other.range.start < range.end + ); + if (clash) { + failures.push( + `edits[${index}]: overlaps edits[${clash.index}], which was applied. Read the file and ` + + "resend this change against its current text." + ); + return; + } + located.push({ index, edit, range }); + } catch (error) { + if (!(error instanceof ToolCallError)) throw error; + failures.push(error.message); + } + }); + let next = content; + for (const { edit, range } of [...located].sort((a, b) => b.range.start - a.range.start)) { + next = next.slice(0, range.start) + edit.newText + next.slice(range.end); + } + return { + content: next, + fuzzy: located.filter((entry) => entry.range.fuzzy).length, + applied: located.map((entry) => entry.index).sort((a, b) => a - b), + unchanged, + failures, + }; +} + +/** + * The edits, as the schema already checked them. A JSON-encoded array or the + * single `oldText`/`newText` shape models also send never get here: AgentTask + * validates against the schema before the tool runs, and has no hook to repair + * arguments first. + */ +function readEdits(input: Record): TextEdit[] { + const raw = input.edits; + if (!Array.isArray(raw) || raw.length === 0) { + throw new ToolCallError("edits must be a non-empty array of {oldText, newText}"); + } + return raw.map((entry, index) => { + const edit = entry as Record; + if (typeof edit?.oldText !== "string" || typeof edit?.newText !== "string") { + throw new ToolCallError(`edits[${index}] needs string oldText and newText`); + } + return { oldText: edit.oldText, newText: edit.newText }; + }); +} + +export function createEditTool(context: CodingToolContext): ToolDefinition { + return { + name: "edit", + description: + "Replace text in an existing file. Each edit's oldText must match exactly one place in " + + "the file as it was before this call (a match that differs only in trailing whitespace " + + "or typographic quotes is accepted). Put several disjoint changes to one file in one call. " + + "Edits that match are applied even when others in the call do not; the result names " + + "the ones that were not. Keep oldText as short as uniqueness allows.", + inputSchema: { + type: "object", + properties: { + path: { type: "string", description: "Path to the file, absolute or relative to the cwd" }, + edits: { + type: "array", + minItems: 1, + items: { + type: "object", + properties: { + oldText: { type: "string", description: "Text to replace" }, + newText: { type: "string", description: "Replacement text" }, + }, + required: ["oldText", "newText"], + additionalProperties: false, + }, + }, + }, + required: ["path", "edits"], + additionalProperties: false, + }, + requiresApproval: false, + execute: async (input) => { + const path = resolvePath(context, requireString(input, "path")); + return withFileLock(path, async () => { + const edits = readEdits(input); + // A file `read` would not load is no file to load here either: a device or + // a pipe has no end to read to (/dev/zero would fill memory), and a huge + // file is not held whole. + const info = await stat(path).catch(() => undefined); + if (info !== undefined && !info.isFile()) { + throw new ToolCallError(`${path} is not a regular file`); + } + if (info !== undefined && info.size > MAX_READ_BYTES) { + throw new ToolCallError( + `${path} is ${info.size} bytes, too large to edit here. Use bash (sed -i, a script) instead.` + ); + } + const bytes = await readFile(path).catch((err: NodeJS.ErrnoException) => { + // Only a missing file is one to create: telling the model to `write` a + // file it merely could not read would have it overwrite the file. + if (err.code === "ENOENT") { + throw new ToolCallError(`File not found: ${path}. Use write to create it.`); + } + throw new ToolCallError(`Could not read ${path}: ${err.message}`); + }); + // Decoding replaces an invalid byte with U+FFFD, and writing the text back + // would replace it in the file too, so a file that is not UTF-8 is not edited. + let original: string; + try { + original = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(bytes); + } catch { + throw new ToolCallError( + `${path} is not UTF-8 text, so editing it here would corrupt it. Use bash instead.` + ); + } + // Matching runs on LF text; the file keeps its BOM and its line ending, + // read from its first line break as pi's edit does (a file mixing the two + // comes back in that one). + const bom = original.startsWith("") ? "" : ""; + const firstBreak = original.indexOf("\n"); + const crlf = firstBreak > 0 && original[firstBreak - 1] === "\r"; + const lf = (text: string): string => text.replace(/\r\n/g, "\n"); + const body = lf(bom ? original.slice(1) : original); + const result = applyEdits( + body, + edits.map((edit) => ({ oldText: lf(edit.oldText), newText: lf(edit.newText) })), + path + ); + if (result.applied.length > 0) { + const out = bom + (crlf ? result.content.replace(/\n/g, "\r\n") : result.content); + await writeFile(path, out, "utf8"); + } + const note = + result.fuzzy > 0 ? ` (${result.fuzzy} matched ignoring whitespace/quotes)` : ""; + const same = + result.unchanged.length > 0 + ? `; ${result.unchanged.length} already matched their newText` + : ""; + if (result.failures.length === 0) { + return result.applied.length > 0 + ? `Applied ${result.applied.length} edit(s) to ${path}${note}${same}` + : `No changes to ${path}: every edit's oldText equals its newText.`; + } + const head = + result.applied.length > 0 + ? `Applied ${result.applied.length} of ${edits.length} edits to ${path}${note}${same}; ` + + "the file now contains them. Not applied:" + : `No edits applied to ${path}:`; + throw new ToolCallError( + `${head}\n${result.failures.join("\n")}\n` + + "Retry only these, against the file as it is now." + ); + }); + }, + }; +} diff --git a/examples/agent-eval/src/agent/tools/fileLock.ts b/examples/agent-eval/src/agent/tools/fileLock.ts new file mode 100644 index 0000000000..c1b157e1cd --- /dev/null +++ b/examples/agent-eval/src/agent/tools/fileLock.ts @@ -0,0 +1,45 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { realpath } from "node:fs/promises"; +import { basename, dirname, join } from "node:path"; + +const tails = new Map>(); + +/** The file a path names, so a symlink and its target share one lock. */ +async function lockKey(path: string): Promise { + try { + return await realpath(path); + } catch { + // Not there yet (a write creating it): the file it will be, in its real directory. + try { + return join(await realpath(dirname(path)), basename(path)); + } catch { + return path; + } + } +} + +/** + * Runs `work` once every earlier holder of `path` has finished. With + * `toolConcurrency` above one, two edits of one file would otherwise both read + * it, each apply its own change, and the later write would silently drop the + * other's. + */ +export async function withFileLock(path: string, work: () => Promise): Promise { + const key = await lockKey(path); + const previous = tails.get(key) ?? Promise.resolve(); + let release!: () => void; + const tail = previous.then(() => new Promise((resolve) => (release = resolve))); + tails.set(key, tail); + await previous; + try { + return await work(); + } finally { + release(); + if (tails.get(key) === tail) tails.delete(key); + } +} diff --git a/examples/agent-eval/src/agent/tools/read.ts b/examples/agent-eval/src/agent/tools/read.ts new file mode 100644 index 0000000000..b117950d14 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/read.ts @@ -0,0 +1,197 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { ToolCallError, ToolResultContent } from "@workglow/ai"; +import { opendir, readFile, stat } from "node:fs/promises"; +import type { CodingToolContext } from "./context"; +import { optionalInteger, requireString, resolvePath } from "./context"; +import { + countLines, + lineAt, + lineStart, + MAX_OUTPUT_BYTES, + MAX_OUTPUT_LINES, + truncateHead, +} from "./truncate"; + +/** Bytes sampled to decide a file is binary. */ +const BINARY_SNIFF_BYTES = 8000; + +/** + * Largest image handed to the model. Providers cap an image at about 5 MB and + * bill it on every later round of the turn, so a bigger one is refused with a + * way to shrink it rather than sent to fail. + */ +const MAX_IMAGE_BYTES = 4 * 1024 * 1024; +/** + * Past this a file is not loaded at all: a page is cut to 50KB anyway, but + * finding it means holding the whole file, and a multi-gigabyte log would take + * the agent's memory with it. + */ +export const MAX_READ_BYTES = 64 * 1024 * 1024; + +/** The image formats every vision provider takes, recognised by their leading bytes. */ +export function imageMimeType(buffer: Buffer): string | undefined { + if (buffer.subarray(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) { + return "image/png"; + } + if (buffer[0] === 0xff && buffer[1] === 0xd8 && buffer[2] === 0xff) return "image/jpeg"; + if (buffer.subarray(0, 6).toString("latin1").startsWith("GIF8")) return "image/gif"; + if ( + buffer.subarray(0, 4).toString("latin1") === "RIFF" && + buffer.subarray(8, 12).toString("latin1") === "WEBP" + ) { + return "image/webp"; + } + return undefined; +} + +function looksBinary(buffer: Buffer): boolean { + const sample = buffer.subarray(0, BINARY_SNIFF_BYTES); + return sample.includes(0); +} + +/** The longest start of `text` that fits `maxBytes` of UTF-8, never splitting a character. */ +function headWithinBytes(text: string, maxBytes: number): string { + let bytes = 0; + let end = 0; + for (const char of text) { + const size = Buffer.byteLength(char, "utf8"); + if (bytes + size > maxBytes) break; + bytes += size; + end += char.length; + } + return text.slice(0, end); +} + +export function createReadTool(context: CodingToolContext): ToolDefinition { + return { + name: "read", + description: + "Read a text file. Output is truncated to 2000 lines or 50KB, whichever comes first; " + + "use offset and limit to page through a longer file. Lines are shown as they are, " + + "without line numbers." + + (context.images + ? " An image (png, jpg, gif, webp) is shown to you as the image itself." + : ""), + inputSchema: { + type: "object", + properties: { + path: { type: "string", description: "Path to the file, absolute or relative to the cwd" }, + offset: { type: "integer", minimum: 1, description: "First line to show (1-based)" }, + limit: { type: "integer", minimum: 1, description: "Maximum lines to show" }, + }, + required: ["path"], + additionalProperties: false, + }, + requiresApproval: false, + execute: async (input) => { + const path = resolvePath(context, requireString(input, "path")); + const offset = optionalInteger(input, "offset") ?? 1; + const limit = optionalInteger(input, "limit"); + // The schema says so, but a tool's arguments reach it unchecked, and a zero + // offset would slice from the end of the file. + if (offset < 1) throw new ToolCallError(`offset must be 1 or more, got ${offset}`); + if (limit !== undefined && limit < 1) { + throw new ToolCallError(`limit must be 1 or more, got ${limit}`); + } + + const info = await stat(path).catch((err: NodeJS.ErrnoException) => { + // Only a missing path is "not found"; anything else (no permission, an + // I/O error) is reported as it is, so the model does not take an + // unreadable file for one it may create. + if (err.code === "ENOENT") throw new ToolCallError(`File not found: ${path}`); + throw new ToolCallError(`Could not read ${path}: ${err.message}`); + }); + if (info.isDirectory()) { + // Kept as they are read, never whole: a directory can hold millions of + // entries and only the first MAX_OUTPUT_LINES are ever shown, so the rest + // are only counted. + const names: string[] = []; + let total = 0; + for await (const entry of await opendir(path)) { + total++; + if (names.length < MAX_OUTPUT_LINES) { + names.push(entry.isDirectory() ? `${entry.name}/` : entry.name); + } + } + names.sort(); + const listing = truncateHead(names.join("\n")); + // Past the cap the names kept are the first the directory returned, not + // the first by name, and the model is told so. + const order = total > names.length ? ", in directory order rather than by name" : ""; + const more = + listing.keptLines < total + ? `\n\n[Showing ${listing.keptLines} of ${total} entries${order}. Use bash (ls, find) to see the rest.]` + : ""; + return `${path} is a directory:\n${listing.text}${more}`; + } + // A device or a pipe has no end to read to (/dev/zero would fill memory). + if (!info.isFile()) throw new ToolCallError(`${path} is not a regular file`); + if (info.size > MAX_READ_BYTES) { + throw new ToolCallError( + `${path} is ${info.size} bytes, too large to read whole. ` + + "Use bash to look at part of it (head, tail, sed -n 'X,Yp', grep)." + ); + } + + const buffer = await readFile(path); + const mimeType = context.images ? imageMimeType(buffer) : undefined; + if (mimeType !== undefined) { + if (buffer.length > MAX_IMAGE_BYTES) { + throw new ToolCallError( + `${path} is a ${buffer.length}-byte image, over the ${MAX_IMAGE_BYTES}-byte limit. ` + + "Shrink it first (e.g. with Python's PIL) and read the smaller copy." + ); + } + return new ToolResultContent([ + { type: "text", text: `Image: ${path} (${mimeType}, ${buffer.length} bytes)` }, + { type: "image", mimeType, data: buffer.toString("base64") }, + ]); + } + if (looksBinary(buffer)) { + throw new ToolCallError(`${path} is a binary file (${buffer.length} bytes)`); + } + const text = buffer.toString("utf8"); + // The window is cut out of the text by line position rather than by + // splitting the file: a file of short lines can hold millions of them. + const totalLines = countLines(text); + if (offset > totalLines) { + throw new ToolCallError( + `offset ${offset} is past the end of ${path} (${totalLines} lines)` + ); + } + const after = limit === undefined ? totalLines + 1 : Math.min(offset + limit, totalLines + 1); + const from = lineStart(text, offset); + const to = + after > totalLines + ? text.endsWith("\n") + ? text.length - 1 + : text.length + : lineStart(text, after) - 1; + const shown = truncateHead(text.slice(from, to)); + + if (shown.keptLines === 0) { + // One line larger than the whole budget: show its start and say how to see the rest. + const first = lineAt(text, offset); + const head = headWithinBytes(first, MAX_OUTPUT_BYTES); + return ( + `${head}\n\n[Line ${offset} is ${first.length} characters; showing the first ` + + `${head.length}. Use bash (e.g. cut -c or head -c) to read the rest.]` + ); + } + const last = offset + shown.keptLines - 1; + if (shown.truncated || (limit !== undefined && last < totalLines)) { + return ( + `${shown.text}\n\n[Showing lines ${offset}-${last} of ${totalLines}. ` + + `Use offset=${last + 1} to continue.]` + ); + } + return shown.text; + }, + }; +} diff --git a/examples/agent-eval/src/agent/tools/truncate.ts b/examples/agent-eval/src/agent/tools/truncate.ts new file mode 100644 index 0000000000..b51e6bb4c8 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/truncate.ts @@ -0,0 +1,150 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Output limits shared by every tool. They are pi's and opencode's numbers on + * purpose: the eval compares loops, and a tool that showed the model five + * times more output than the other harnesses' would be measuring the tool. + */ +export const MAX_OUTPUT_LINES = 2000; +export const MAX_OUTPUT_BYTES = 50 * 1024; + +export interface Truncated { + readonly text: string; + readonly truncated: boolean; + /** Lines in the input, before truncation. */ + readonly totalLines: number; + /** Lines kept. */ + readonly keptLines: number; + /** The first kept line is only the end of a line too long for the budget. */ + readonly partialLine: boolean; +} + +// Lines are walked with indexOf rather than split: a capped output or an allowed +// file of short lines can hold millions of them, and an array of every line is +// many times the text's own size, while only the first or last few are kept. + +/** Where the text's lines end: a final newline ends the last line rather than starting another. */ +function bodyEnd(text: string): number { + return text.endsWith("\n") ? text.length - 1 : text.length; +} + +/** The number of lines in `text`, counted without materializing them. */ +export function countLines(text: string): number { + const end = bodyEnd(text); + let lines = 1; + for (let i = text.indexOf("\n"); i !== -1 && i < end; i = text.indexOf("\n", i + 1)) lines++; + return lines; +} + +/** The index where 1-based line `n` starts; `n` must be at most {@link countLines}. */ +export function lineStart(text: string, n: number): number { + let at = 0; + for (let line = 1; line < n; line++) at = text.indexOf("\n", at) + 1; + return at; +} + +/** Line `n` (1-based) of `text`, without its newline. */ +export function lineAt(text: string, n: number): string { + const start = lineStart(text, n); + const next = text.indexOf("\n", start); + return text.slice(start, next === -1 || next > bodyEnd(text) ? bodyEnd(text) : next); +} + +function byteLength(text: string): number { + return Buffer.byteLength(text, "utf8"); +} + +/** + * The first lines of `text` that fit both limits. A read wants the head: the + * model continues from where it stopped with `offset`. + */ +export function truncateHead( + text: string, + maxLines: number = MAX_OUTPUT_LINES, + maxBytes: number = MAX_OUTPUT_BYTES +): Truncated { + const end = bodyEnd(text); + const totalLines = countLines(text); + const kept: string[] = []; + let bytes = 0; + for (let at = 0; kept.length < totalLines;) { + const next = text.indexOf("\n", at); + const lineEnd = next === -1 || next > end ? end : next; + const line = text.slice(at, lineEnd); + // Each line costs its newline, except a last line the text does not end with. + const size = byteLength(line) + (lineEnd === end && !text.endsWith("\n") ? 0 : 1); + if (kept.length >= maxLines || bytes + size > maxBytes) break; + kept.push(line); + bytes += size; + at = lineEnd + 1; + } + const truncated = kept.length < totalLines; + return { + // The final newline stays when the last line is shown. + text: kept.join("\n") + (!truncated && text.endsWith("\n") ? "\n" : ""), + truncated, + totalLines, + keptLines: kept.length, + partialLine: false, + }; +} + +/** The end of `line` within `maxBytes` of UTF-8, cut on a character boundary. */ +function tailWithinBytes(line: string, maxBytes: number): string { + if (maxBytes <= 0) return ""; + const buffer = Buffer.from(line, "utf8"); + if (buffer.length <= maxBytes) return line; + let start = buffer.length - maxBytes; + // A UTF-8 continuation byte is 10xxxxxx; start on the next character instead. + while (start < buffer.length && (buffer[start]! & 0xc0) === 0x80) start++; + return buffer.subarray(start).toString("utf8"); +} + +/** + * The last lines of `text` that fit both limits. A command wants the tail: + * the error, the test summary and the exit status are at the end. A last line + * longer than the whole budget (minified JSON, a progress bar redrawn with + * carriage returns) keeps its end, rather than leaving nothing to show. + */ +export function truncateTail( + text: string, + maxLines: number = MAX_OUTPUT_LINES, + maxBytes: number = MAX_OUTPUT_BYTES +): Truncated { + const totalLines = countLines(text); + const kept: string[] = []; + let bytes = 0; + let partial = false; + // Walked from the end: `end` is where the current line stops. + for (let end = bodyEnd(text); kept.length < totalLines;) { + if (kept.length >= maxLines) break; + const previous = end === 0 ? -1 : text.lastIndexOf("\n", end - 1); + const line = text.slice(previous + 1, end); + // Each line costs its newline, except a last line the text does not end with. + const newline = kept.length === 0 && !text.endsWith("\n") ? 0 : 1; + const size = byteLength(line) + newline; + if (bytes + size > maxBytes) { + if (kept.every((shown) => shown.length === 0)) { + kept.push(tailWithinBytes(line, maxBytes - bytes - newline)); + partial = true; + } + break; + } + kept.push(line); + bytes += size; + end = previous; + } + kept.reverse(); + return { + // The tail always ends on the last line, so its final newline stays. + text: kept.join("\n") + (kept.length > 0 && text.endsWith("\n") ? "\n" : ""), + truncated: partial || kept.length < totalLines, + totalLines, + keptLines: kept.length, + partialLine: partial, + }; +} diff --git a/examples/agent-eval/src/agent/tools/write.ts b/examples/agent-eval/src/agent/tools/write.ts new file mode 100644 index 0000000000..c391ce8a2d --- /dev/null +++ b/examples/agent-eval/src/agent/tools/write.ts @@ -0,0 +1,50 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { ToolCallError } from "@workglow/ai"; +import { mkdir, stat, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; +import type { CodingToolContext } from "./context"; +import { requireString, resolvePath } from "./context"; +import { withFileLock } from "./fileLock"; + +export function createWriteTool(context: CodingToolContext): ToolDefinition { + return { + name: "write", + description: + "Create a file or overwrite it with new content, creating parent directories as needed. " + + "Use edit for changes to an existing file; use write for new files or complete rewrites.", + inputSchema: { + type: "object", + properties: { + path: { type: "string", description: "Path to the file, absolute or relative to the cwd" }, + content: { type: "string", description: "The complete file content" }, + }, + required: ["path", "content"], + additionalProperties: false, + }, + requiresApproval: false, + execute: async (input) => { + const path = resolvePath(context, requireString(input, "path")); + const content = requireString(input, "content"); + await mkdir(dirname(path), { recursive: true }); + await withFileLock(path, async () => { + // Opening a FIFO for writing blocks until something reads it, and a device + // is not a file to overwrite; only a regular file, or none yet, is written. + const existing = await stat(path).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return undefined; + throw error; + }); + if (existing && !existing.isFile()) { + throw new ToolCallError(`${path} is not a regular file; write only writes files.`); + } + await writeFile(path, content, "utf8"); + }); + return `Wrote ${Buffer.byteLength(content, "utf8")} bytes to ${path}`; + }, + }; +} diff --git a/examples/agent-eval/src/bin/workglow-agent-eval.ts b/examples/agent-eval/src/bin/workglow-agent-eval.ts new file mode 100644 index 0000000000..f265b19ee8 --- /dev/null +++ b/examples/agent-eval/src/bin/workglow-agent-eval.ts @@ -0,0 +1,275 @@ +#!/usr/bin/env bun + +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { existsSync, writeFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { Command, Option } from "commander"; +import { loadTrials } from "../compare/harborJobs"; +import { buildReport, renderMarkdown } from "../compare/report"; +import { startAnthropicMock } from "../mock/anthropicMock"; +import type { ArmSpec, Effort } from "../run/jobConfig"; +import { buildHarborJob, EFFORTS, HARNESSES, parseArm } from "../run/jobConfig"; +import { runHarbor } from "../run/runHarbor"; +import type { JobDataset } from "../run/suites"; +import { getSuite, narrowDataset, SUITES } from "../run/suites"; + +function positiveInteger(value: string): number { + const n = Number(value); + if (!Number.isInteger(n) || n < 1) throw new Error(`expected a positive integer, got ${value}`); + return n; +} + +function positiveNumber(value: string): number { + const n = Number(value); + if (!Number.isFinite(n) || n <= 0) throw new Error(`expected a positive number, got ${value}`); + return n; +} + +function collect(value: string, previous: string[] = []): string[] { + return [...previous, value]; +} + +function timestamp(): string { + return new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19); +} + +function writeReport( + paths: readonly string[], + format: string, + out: string | undefined, + allTasks: boolean +): void { + const trials = loadTrials(paths); + if (trials.length === 0) throw new Error(`no Harbor trial results under ${paths.join(", ")}`); + const report = buildReport(trials); + const text = + format === "json" + ? `${JSON.stringify(report, null, 2)}\n` + : renderMarkdown(report, { allTasks }); + if (out) { + writeFileSync(out, text); + process.stderr.write(`wrote ${out}\n`); + } else process.stdout.write(text); +} + +const program = new Command() + .name("workglow-agent-eval") + .version("0.1.0") + .description( + "Compare Workglow's AgentTask with opencode and pi on public agentic benchmarks, through Harbor" + ); + +program + .command("suites") + .description("List the named benchmark suites") + .action(() => { + for (const [name, suite] of Object.entries(SUITES)) { + const datasets = suite.datasets + .map( + (d) => + `${d.name}@${d.version}${d.task_names ? ` ${d.task_names.join(" ")}` : ""}${d.n_tasks ? ` (first ${d.n_tasks})` : ""}` + ) + .join(" + "); + process.stdout.write( + `${name.padEnd(16)} ${suite.description}\n${"".padEnd(16)} ${datasets}\n` + ); + } + }); + +program + .command("run") + .description("Run every arm on a suite in one Harbor job, then write the comparison") + .requiredOption( + "-m, --model ", + "the model every arm uses, e.g. anthropic/claude-sonnet-5-5" + ) + .option("-s, --suite ", "benchmark suite (see `suites`)", "smoke") + .option( + "-d, --dataset ", + "run this registry dataset instead of a suite (repeatable)", + collect + ) + .option( + "--tasks-dir ", + "run the Harbor task folders in this directory instead of a suite (repeatable)", + collect + ) + .option( + "-a, --agents ", + `comma-separated harnesses: ${HARNESSES.join(",")}`, + HARNESSES.join(",") + ) + .option( + "--arm ", + "an extra arm, e.g. workglow:tool_concurrency=8 (repeatable; agent kwargs as key=value)", + collect + ) + .addOption( + new Option("--effort ", "reasoning level for every arm") + .choices([...EFFORTS]) + .default("high") + ) + .option( + "--opencode-variant ", + "opencode's name for that reasoning level (default: the --effort level)" + ) + .option("--max-turns ", "model calls each arm may make", positiveInteger, 100) + .option("-k, --attempts ", "attempts per task per arm", positiveInteger, 1) + .option("-n, --concurrency ", "trials run at once", positiveInteger, 4) + .option("--tasks ", "only tasks matching these comma-separated globs") + .option("-l, --n-tasks ", "at most this many tasks per dataset", positiveInteger) + .option( + "--timeout-multiplier ", + "scale every task's agent and verifier timeouts", + positiveNumber + ) + .option( + "--pi-model-api ", + "pi's wire API when the provider base URL is overridden (e.g. anthropic-messages)" + ) + .option( + "--harbor-config ", + "extra Harbor config layered on the generated one (repeatable)", + collect + ) + .option("--jobs-dir ", "where Harbor writes results", "./jobs") + .option("--job-name ", "Harbor job name (default -)") + .option("--harbor ", "the harbor executable", "harbor") + .option("--dry-run", "print the Harbor job config and exit") + .action(async (opts) => { + const custom = opts.dataset !== undefined || opts.tasksDir !== undefined; + const suite = custom ? undefined : getSuite(opts.suite); + const suiteDatasets: JobDataset[] = suite + ? [...suite.datasets] + : [ + ...((opts.dataset as string[] | undefined) ?? []).map((spec) => { + const at = spec.lastIndexOf("@"); + if (at <= 0) throw new Error(`dataset must be name@version, got "${spec}"`); + return { name: spec.slice(0, at), version: spec.slice(at + 1) }; + }), + ...((opts.tasksDir as string[] | undefined) ?? []).map((dir) => ({ + path: resolve(dir), + })), + ]; + const arms: ArmSpec[] = [ + ...String(opts.agents) + .split(",") + .map((name) => name.trim()) + .filter((name) => name.length > 0) + .map(parseArm), + ...((opts.arm as string[] | undefined) ?? []).map(parseArm), + ]; + if (arms.length === 0) throw new Error("no arms to run"); + const taskFilter = opts.tasks + ? String(opts.tasks) + .split(",") + .map((glob) => glob.trim()) + : undefined; + // A filter may name tasks of only some of a suite's datasets; the rest drop out. + const datasets = suiteDatasets + .map((dataset) => narrowDataset(dataset, taskFilter, opts.nTasks as number | undefined)) + .filter((dataset): dataset is JobDataset => dataset !== undefined); + if (datasets.length === 0) { + throw new Error(`--tasks ${String(opts.tasks)} matches none of the tasks this suite runs`); + } + const piModelApi = + opts.piModelApi ?? + (String(opts.model).startsWith("anthropic/") && process.env.ANTHROPIC_BASE_URL + ? "anthropic-messages" + : undefined); + const jobName = opts.jobName ?? `${suite ? opts.suite : "custom"}-${timestamp()}`; + const config = buildHarborJob({ + jobName, + jobsDir: opts.jobsDir, + model: opts.model, + arms, + datasets, + parity: { + maxTurns: opts.maxTurns, + effort: opts.effort as Effort, + // opencode names its levels after the effort itself, so the arms run at + // the same one unless the caller names a variant of their own. + opencodeVariant: opts.opencodeVariant || (opts.effort === "off" ? undefined : opts.effort), + piModelApi, + }, + attempts: opts.attempts, + concurrency: opts.concurrency, + timeoutMultiplier: opts.timeoutMultiplier, + }); + if (opts.dryRun) { + process.stdout.write(`${JSON.stringify(config, null, 2)}\n`); + return; + } + // Results already there would be read back as this run's if Harbor then + // wrote none of its own. + const existing = resolve(String(opts.jobsDir), jobName); + if (existsSync(existing)) { + throw new Error(`${existing} already exists; pass another --job-name or move it away`); + } + const { jobDir, exitCode } = await runHarbor({ + config, + extraConfigs: (opts.harborConfig as string[] | undefined) ?? [], + harbor: opts.harbor, + }); + if (!existsSync(jobDir)) throw new Error(`harbor exited with ${exitCode} and wrote no results`); + if (exitCode !== 0) + process.stderr.write(`harbor exited with ${exitCode}; comparing what it wrote\n`); + const trials = loadTrials([jobDir]); + if (trials.length === 0) + throw new Error(`harbor exited with ${exitCode} and wrote no trial results to ${jobDir}`); + const report = buildReport(trials); + const markdown = renderMarkdown(report); + writeFileSync(`${jobDir}/comparison.md`, markdown); + writeFileSync(`${jobDir}/comparison.json`, `${JSON.stringify(report, null, 2)}\n`); + process.stdout.write(`\n${markdown}\nwrote ${jobDir}/comparison.md and comparison.json\n`); + // The partial comparison is still worth having, but a run Harbor failed is + // not a run that succeeded. + if (exitCode !== 0) process.exitCode = exitCode; + }); + +program + .command("compare") + .description("Compare the trials in one or more Harbor job directories") + .argument("", "Harbor job directories, or a jobs directory holding several") + .addOption( + new Option("-f, --format ", "output format").choices(["md", "json"]).default("md") + ) + .option("-o, --out ", "write to a file instead of stdout") + .option("--all-tasks", "list every task, not only those where the arms differ") + .action((paths: string[], opts) => writeReport(paths, opts.format, opts.out, !!opts.allTasks)); + +program + .command("mock-model") + .description("Serve a scripted Anthropic Messages API for an offline end-to-end check") + .option("-p, --port ", "port", positiveInteger, 18089) + .option("--host ", "interface to listen on", "127.0.0.1") + .option("-c, --command ", "shell command the model runs, in order (repeatable)", collect) + .option("--request-log ", "append every request body to this JSONL file") + .action(async (opts) => { + const commands = (opts.command as string[] | undefined) ?? ["echo 'Hello, world!' > hello.txt"]; + await startAnthropicMock({ + port: opts.port, + host: opts.host, + commands, + requestLog: opts.requestLog, + }); + process.stderr.write( + `mock Anthropic API on http://${opts.host}:${opts.port} running ${commands.length} command(s)\n` + + `point the arms at it with ANTHROPIC_BASE_URL=http://${opts.host}:${opts.port} ANTHROPIC_API_KEY=mock\n` + ); + await new Promise(() => {}); + }); + +try { + await program.parseAsync(process.argv); +} catch (error) { + process.stderr.write( + `workglow-agent-eval: ${error instanceof Error ? error.message : String(error)}\n` + ); + process.exitCode = 1; +} diff --git a/examples/agent-eval/src/bin/workglow-agent.ts b/examples/agent-eval/src/bin/workglow-agent.ts new file mode 100644 index 0000000000..9d56dd1df6 --- /dev/null +++ b/examples/agent-eval/src/bin/workglow-agent.ts @@ -0,0 +1,243 @@ +#!/usr/bin/env node + +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * The headless Workglow coding agent: AgentTask plus the four coding tools, + * run once on one instruction. It is what the Harbor adapter installs into a + * task container — the counterpart of `opencode run` and `pi --print` — so it + * is bundled into one file with every dependency and needs only Node. + */ + +import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { Command } from "commander"; +import { toAtifTrajectory } from "../agent/atif"; +import { loadCredentialsFile } from "../agent/credentialsFile"; +import { parseEffort, resolveAgentModel } from "../agent/models"; +import { registerAgentProvider } from "../agent/providers"; +import type { AgentRunEvent } from "../agent/runCodingAgent"; +import { + DEFAULT_BENCH_MAX_ROUNDS, + DEFAULT_BENCH_MAX_TOOL_RESULT_CHARS, + runCodingAgent, +} from "../agent/runCodingAgent"; +import { REVIEW_PROMPT } from "../agent/systemPrompt"; +import { MAX_TIMEOUT_SEC } from "../agent/tools/bash"; + +const VERSION = "0.1.0"; + +function integer(value: string): number { + const n = Number(value); + if (!Number.isInteger(n) || n < 1) throw new Error(`expected a positive integer, got ${value}`); + return n; +} + +/** `on`/`off` switches, spelled so Harbor agent kwargs pass them through as strings. */ +function onOff(value: string): boolean { + if (value === "on" || value === "true") return true; + if (value === "off" || value === "false") return false; + throw new Error(`expected on or off, got ${value}`); +} + +/** Seconds for a timer: past MAX_TIMEOUT_SEC Node would fire it at once. */ +function timerSeconds(value: string): number { + const n = integer(value); + if (n > MAX_TIMEOUT_SEC) + throw new Error(`expected at most ${MAX_TIMEOUT_SEC} seconds, got ${value}`); + return n; +} + +function nonNegativeInteger(value: string): number { + const n = Number(value); + if (!Number.isInteger(n) || n < 0) { + throw new Error(`expected a non-negative integer, got ${value}`); + } + return n; +} + +function number(value: string): number { + const n = Number(value); + if (!Number.isFinite(n)) throw new Error(`expected a number, got ${value}`); + return n; +} + +function describe(event: AgentRunEvent): string | undefined { + switch (event.type) { + case "start": + return `workglow-agent ${VERSION} · ${event.model} · ${event.cwd}`; + case "tool-call": + return event.status === "running" ? ` → ${event.name ?? "?"}` : undefined; + case "round": { + const usage = event.step.usage; + const tokens = usage + ? ` · ${usage.input ?? 0}+${usage.cached ?? 0} in / ${usage.output ?? 0} out` + : ""; + return `round ${event.step.round} · ${event.step.tools.length} tool(s)${tokens}`; + } + case "finish": + return `finished: ${event.summary.outcome}${event.summary.error ? ` (${event.summary.error})` : ""}`; + } +} + +const program = new Command() + .name("workglow-agent") + .version(VERSION) + .description("Run Workglow's AgentTask with coding tools on one instruction") + .argument("[instruction...]", "the task; or pass --instruction-file") + .requiredOption("-m, --model ", "model, e.g. anthropic/claude-sonnet-4-5") + .option("--instruction-file ", "read the instruction from a file") + .option( + "--credentials-file ", + "JSON object of environment variables (API keys) to load, then delete" + ) + .option("--cwd ", "working directory", process.cwd()) + .option("--logs-dir ", "where the events, summary and trajectory are written") + .option("--effort ", "reasoning effort: none|low|medium|high|extra|ultra") + .option("--max-rounds ", "model calls before the run stops", integer, DEFAULT_BENCH_MAX_ROUNDS) + .option( + "--max-tool-result-chars ", + "characters of one tool result the model is shown", + integer, + DEFAULT_BENCH_MAX_TOOL_RESULT_CHARS + ) + .option( + "--max-history-chars ", + "history budget sent per round (AgentTask default if unset)", + integer + ) + .option("--tool-concurrency ", "tool calls of one round run at once", integer) + .option("--max-tokens ", "output tokens per model call", integer) + .option("--temperature ", "sampling temperature", number) + .option("--max-duration-sec ", "wall-clock budget; no round starts after it", integer) + .option( + "--stop-after-sec ", + "stop outright after this long, even mid-call, and still write the logs", + timerSeconds + ) + .option("--round-timeout-sec ", "abandon and retry a model call after this long", timerSeconds) + .option("--max-round-retries ", "retries of a failed model call", nonNegativeInteger) + .option("--command-timeout-sec ", "default bash timeout", timerSeconds, 120) + .option("--append-system-prompt ", "text appended to the system prompt") + .option("--concise ", "ask for terse replies between tool calls", onOff, true) + .option("--images ", "let read show images to the model", onOff, true) + .option( + "--replay-reasoning ", + "send the model's reasoning back on later rounds (providers that take it)", + onOff, + true + ) + .option( + "--time-left ", + "show the model its remaining time each round (needs --max-duration-sec)", + onOff, + true + ) + .option("--review ", "ask the model to check its work once before finishing", onOff, true) + .action(async (words: string[], opts) => { + // First, before anything can start a child process. + if (opts.credentialsFile) loadCredentialsFile(opts.credentialsFile); + const instruction = opts.instructionFile + ? readFileSync(opts.instructionFile, "utf8") + : words.join(" "); + if (instruction.trim() === "") throw new Error("no instruction given"); + + const resolved = resolveAgentModel(opts.model, { effort: parseEffort(opts.effort) }); + // An ablation switch for the provider's reasoning replay; DeepSeek and OpenAI read it, + // and Anthropic has no such switch. + const model = opts.replayReasoning + ? resolved + : ({ + ...resolved, + provider_config: { ...resolved.provider_config, replay_reasoning: false }, + } as typeof resolved); + await registerAgentProvider(model.provider); + + const cwd = resolve(opts.cwd); + const logsDir = opts.logsDir ? resolve(opts.logsDir) : undefined; + if (logsDir) mkdirSync(logsDir, { recursive: true }); + const eventsPath = logsDir ? join(logsDir, "workglow-events.jsonl") : undefined; + // One run per log, like the summary and trajectory beside it. + if (eventsPath) writeFileSync(eventsPath, ""); + + const controller = new AbortController(); + for (const signal of ["SIGINT", "SIGTERM"] as const) { + process.once(signal, () => controller.abort(new Error(`received ${signal}`))); + } + + const result = await runCodingAgent({ + instruction, + modelId: opts.model, + model, + tools: { + cwd, + spillDir: join(logsDir ?? tmpdir(), "spill"), + defaultCommandTimeoutSec: opts.commandTimeoutSec, + images: opts.images, + }, + settings: { + maxRounds: opts.maxRounds, + maxToolResultChars: opts.maxToolResultChars, + maxHistoryChars: opts.maxHistoryChars, + toolConcurrency: opts.toolConcurrency, + maxTokens: opts.maxTokens, + temperature: opts.temperature, + maxDurationMs: opts.maxDurationSec === undefined ? undefined : opts.maxDurationSec * 1000, + roundTimeoutMs: + opts.roundTimeoutSec === undefined ? undefined : opts.roundTimeoutSec * 1000, + maxRoundRetries: opts.maxRoundRetries, + systemPromptAppend: opts.appendSystemPrompt, + concise: opts.concise, + announceTimeLeft: opts.timeLeft, + reviewPrompt: opts.review ? REVIEW_PROMPT : undefined, + }, + signal: controller.signal, + stopAfterMs: opts.stopAfterSec === undefined ? undefined : opts.stopAfterSec * 1000, + onEvent: (event) => { + const line = describe(event); + if (line) process.stderr.write(`${line}\n`); + if (eventsPath) appendFileSync(eventsPath, `${JSON.stringify(event)}\n`); + }, + }); + + if (logsDir) { + writeFileSync( + join(logsDir, "workglow-summary.json"), + JSON.stringify(result.summary, null, 2) + ); + const trajectory = toAtifTrajectory( + `workglow-${Date.now()}`, + VERSION, + result.messages, + result.steps, + result.summary + ); + writeFileSync(join(logsDir, "trajectory.json"), JSON.stringify(trajectory, null, 2)); + } + if (result.summary.finalText) process.stdout.write(`${result.summary.finalText}\n`); + process.exitCode = + result.summary.outcome === "error" || result.summary.outcome === "aborted" ? 1 : 0; + }); + +try { + await program.parseAsync(process.argv); +} catch (error) { + process.stderr.write( + `workglow-agent: ${error instanceof Error ? error.message : String(error)}\n` + ); + process.exitCode = 2; +} +// Provider SDKs can hold sockets open, so the process is ended outright — once +// what was written has drained, since a write to a pipe can still be queued +// where pipes are asynchronous (macOS), and exiting would cut it off. +await Promise.all( + [process.stdout, process.stderr].map( + (stream) => new Promise((resolve) => stream.write("", () => resolve())) + ) +); +process.exit(process.exitCode ?? 0); diff --git a/examples/agent-eval/src/compare/harborJobs.ts b/examples/agent-eval/src/compare/harborJobs.ts new file mode 100644 index 0000000000..7a420500d3 --- /dev/null +++ b/examples/agent-eval/src/compare/harborJobs.ts @@ -0,0 +1,377 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; +import { basename, dirname, join } from "node:path"; + +/** + * How a trial ended, from the comparison's point of view. + * + * `blocked` is the one that never reaches a pass rate: the environment or the + * agent's install failed before the agent saw the task, so the trial measured + * the infrastructure. Counting it as a failure would charge a harness for a + * flaky registry; dropping it silently would shrink one arm's denominator. + * It is excluded and reported. + */ +export type TrialStatus = "pass" | "fail" | "blocked"; + +/** Why a trial that ran did not pass, coarse enough to tally across harnesses. */ +export type FailureMode = + | "wrong-answer" + | "agent-timeout" + | "context-overflow" + | "max-rounds" + | "budget" + | "rate-limit" + | "provider-error" + | "agent-crash" + | "verifier-error"; + +export interface TrialRecord { + readonly job: string; + readonly trial: string; + readonly task: string; + /** + * The comparison column: the agent, plus whatever tells this configuration + * from the agent's others in the same comparison — e.g. + * `workglow[tool_concurrency=8]`, or `pi@openai/gpt-5.5` when models differ. + */ + readonly arm: string; + /** The agent kwargs this trial ran with. */ + readonly kwargs: Readonly>; + readonly agent: string; + readonly agentVersion: string | undefined; + readonly model: string | undefined; + readonly reward: number | undefined; + readonly status: TrialStatus; + readonly failure: FailureMode | undefined; + readonly exceptionType: string | undefined; + readonly exceptionMessage: string | undefined; + readonly inputTokens: number | undefined; + readonly cacheTokens: number | undefined; + readonly outputTokens: number | undefined; + readonly costUsd: number | undefined; + readonly agentSec: number | undefined; + readonly setupSec: number | undefined; + /** The workglow summary the adapter stores on the trial, when this arm is workglow. */ + readonly workglow: WorkglowTrialMetadata | undefined; +} + +export interface WorkglowTrialMetadata { + readonly outcome: string; + readonly error?: string | undefined; + readonly rounds: number; + readonly retries: number; + readonly tools: { + readonly calls: number; + readonly errors: number; + readonly byName: Readonly>; + }; + readonly finalHistoryChars: number; +} + +interface HarborTrialJson { + readonly trial_name?: string; + readonly task_name?: string; + readonly config?: { + readonly agent?: { + name?: string; + import_path?: string; + model_name?: string | null; + kwargs?: Record; + }; + }; + readonly agent_info?: { + readonly name?: string; + readonly version?: string; + readonly model_info?: { readonly name?: string; readonly provider?: string } | null; + }; + readonly agent_result?: { + readonly n_input_tokens?: number | null; + readonly n_cache_tokens?: number | null; + readonly n_output_tokens?: number | null; + readonly cost_usd?: number | null; + readonly metadata?: { readonly workglow?: WorkglowTrialMetadata } | null; + } | null; + readonly verifier_result?: { readonly rewards?: Record | null } | null; + readonly exception_info?: { + readonly exception_type?: string; + readonly exception_message?: string; + } | null; + readonly agent_setup?: Span | null; + readonly agent_execution?: Span | null; + readonly environment_setup?: Span | null; +} + +interface Span { + readonly started_at?: string | null; + readonly finished_at?: string | null; +} + +function seconds(span: Span | null | undefined): number | undefined { + if (!span?.started_at || !span.finished_at) return undefined; + const ms = Date.parse(span.finished_at) - Date.parse(span.started_at); + return Number.isFinite(ms) ? ms / 1000 : undefined; +} + +/** + * A kwarg value as a label shows it. A string goes bare only when it can neither + * split the label (a comma, `=`, a bracket) nor read as another value (`8`, + * `true`): two settings must never share a label, or they are counted as one arm. + */ +function kwargValue(value: unknown): string { + if (typeof value !== "string") return JSON.stringify(value); + if (/[,=[\]"]/.test(value)) return JSON.stringify(value); + try { + JSON.parse(value); + return JSON.stringify(value); + } catch { + return value; + } +} + +/** + * Names each trial's arm. A kwarg appears in a label only when the agent ran + * with more than one value for it in this comparison: the parity settings every + * arm carries would otherwise make every label unreadable, and an ablation's + * one changed setting is exactly what should show. That includes `bundle_path` + * and `version`: they choose the build that ran, so two of them are two agents. + */ +export function labelArms(trials: readonly TrialRecord[]): TrialRecord[] { + // A trial whose model is unknown joins its agent's arm when that agent ran one + // model; left unlabelled it would be an arm of its own, and a blocked trial + // there is missing from the arm it belongs to. + const agentModels = new Map>(); + for (const trial of trials) { + if (trial.model === undefined) continue; + const known = agentModels.get(trial.agent) ?? new Set(); + known.add(trial.model); + agentModels.set(trial.agent, known); + } + const modelOf = (trial: TrialRecord): string | undefined => { + if (trial.model !== undefined) return trial.model; + const known = agentModels.get(trial.agent); + return known?.size === 1 ? [...known][0] : undefined; + }; + const models = new Set(trials.map(modelOf)); + const varying = new Map>(); + const byAgent = new Map(); + for (const trial of trials) { + const list = byAgent.get(trial.agent) ?? []; + list.push(trial); + byAgent.set(trial.agent, list); + } + for (const [agent, list] of byAgent) { + const keys = new Set(list.flatMap((trial) => Object.keys(trial.kwargs))); + const differ = new Set(); + for (const key of keys) { + const values = new Set(list.map((trial) => kwargValue(trial.kwargs[key]))); + if (values.size > 1) differ.add(key); + } + varying.set(agent, differ); + } + return trials.map((trial) => { + const keys = [...(varying.get(trial.agent) ?? [])].sort(); + const parts = keys + .filter((key) => trial.kwargs[key] !== undefined) + .map((key) => `${key}=${kwargValue(trial.kwargs[key])}`); + const resolved = modelOf(trial); + const model = models.size > 1 && resolved ? `@${resolved}` : ""; + return { + ...trial, + arm: `${trial.agent}${model}${parts.length > 0 ? `[${parts.join(",")}]` : ""}`, + }; + }); +} + +const OVERFLOW = + /context(?:[ _](?:length|window))|maximum context|prompt is too long|too many tokens|exceeds the (?:context|maximum)|input token count exceeds/i; +const RATE = /rate.?limit|429|too many requests|overloaded|quota/i; +/** A model API that failed rather than the agent: a 5xx, an API error, a dropped connection. */ +const PROVIDER = + /\b5\d\d\b|api ?error|internal server error|service unavailable|bad gateway|gateway timeout|ECONNRESET|ETIMEDOUT|ENOTFOUND|ECONNREFUSED|fetch failed|socket hang up/i; + +/** + * Classifies a trial. A reward is believed whenever the verifier produced + * one, even after an agent exception: an agent that timed out after writing + * the right answer did solve the task, which is how Harbor scores it too. + */ +export function classifyTrial( + reward: number | undefined, + exceptionType: string | undefined, + exceptionMessage: string | undefined, + agentRan: boolean, + workglow: WorkglowTrialMetadata | undefined +): { status: TrialStatus; failure: FailureMode | undefined } { + if (reward !== undefined && reward >= 1 - 1e-9) return { status: "pass", failure: undefined }; + if (!agentRan && exceptionType !== undefined) return { status: "blocked", failure: undefined }; + + const text = `${exceptionType ?? ""} ${exceptionMessage ?? ""} ${workglow?.error ?? ""}`; + let failure: FailureMode; + if (exceptionType === "AgentTimeoutError") failure = "agent-timeout"; + // Ahead of the API-error branch: a provider reports an over-long prompt as an + // API error too, and that is the overflow it describes. A rate-limit message + // ("too many tokens per minute") is not one. + else if ( + exceptionType === "ContextWindowExceededError" || + (OVERFLOW.test(text) && !RATE.test(text)) + ) + failure = "context-overflow"; + else if (workglow?.outcome === "max-rounds") failure = "max-rounds"; + else if (workglow?.outcome === "budget") failure = "budget"; + else if (exceptionType !== undefined && /RateLimit|UsageLimit|Overloaded/.test(exceptionType)) + failure = "rate-limit"; + else if ( + exceptionType !== undefined && + /Api|Network|Authentication|ModelNotFound/.test(exceptionType) + ) + failure = RATE.test(text) ? "rate-limit" : "provider-error"; + else if (exceptionType !== undefined && /Verifier|RewardFile|AddTests/.test(exceptionType)) + failure = "verifier-error"; + // The Workglow agent exits non-zero on a provider failure, which Harbor reports + // as a generic exit-code error; its own recorded error says what failed. + else if (workglow?.error !== undefined && RATE.test(workglow.error)) failure = "rate-limit"; + else if (workglow?.error !== undefined && PROVIDER.test(workglow.error)) + failure = "provider-error"; + else if (exceptionType !== undefined) + failure = OVERFLOW.test(text) ? "context-overflow" : "agent-crash"; + else if (workglow?.outcome === "error") + failure = OVERFLOW.test(text) ? "context-overflow" : "agent-crash"; + else failure = "wrong-answer"; + return { status: "fail", failure }; +} + +export function parseTrial(json: HarborTrialJson, job: string): TrialRecord | undefined { + if (!json.trial_name || !json.task_name) return undefined; + const configured = json.config?.agent; + const agent = json.agent_info?.name ?? configured?.name ?? configured?.import_path ?? "unknown"; + const rewards = json.verifier_result?.rewards ?? undefined; + const reward = rewards ? (rewards.reward ?? Object.values(rewards)[0]) : undefined; + const exceptionType = json.exception_info?.exception_type ?? undefined; + const exceptionMessage = json.exception_info?.exception_message ?? undefined; + const agentRan = !!json.agent_execution?.started_at; + const workglow = json.agent_result?.metadata?.workglow ?? undefined; + const { status, failure } = classifyTrial( + reward, + exceptionType, + exceptionMessage, + agentRan, + workglow + ); + // A trial that failed in setup has no agent_info yet; the model it was + // configured with still says which arm it belongs to. + const model = json.agent_info?.model_info + ? `${json.agent_info.model_info.provider ?? "?"}/${json.agent_info.model_info.name ?? "?"}` + : (json.config?.agent?.model_name ?? undefined); + const result = json.agent_result ?? undefined; + return { + job, + trial: json.trial_name, + task: json.task_name, + arm: agent, + kwargs: configured?.kwargs ?? {}, + agent, + agentVersion: json.agent_info?.version, + model, + reward, + status, + failure, + exceptionType, + exceptionMessage, + inputTokens: result?.n_input_tokens ?? undefined, + cacheTokens: result?.n_cache_tokens ?? undefined, + outputTokens: result?.n_output_tokens ?? undefined, + costUsd: result?.cost_usd ?? undefined, + agentSec: seconds(json.agent_execution), + setupSec: seconds(json.agent_setup), + workglow, + }; +} + +/** + * Whether a `result.json` is a trial's rather than its job's. Told apart by the + * trial's own fields, not by an `agent/` directory beside it: a trial that + * failed in setup can leave its result before it has any agent logs, and is the + * blocked trial every rate has to exclude. + */ +function isTrialResult(file: string): boolean { + try { + const json = JSON.parse(readFileSync(file, "utf8")) as Record | null; + return ( + json !== null && + typeof json === "object" && + (typeof json.trial_name === "string" || typeof json.task_name === "string") + ); + } catch { + return false; + } +} + +/** Every trial `result.json` at or below `path` (a trial dir, a job dir, or a jobs dir). */ +function findTrialResults(path: string, depth = 0): string[] { + if (!existsSync(path)) throw new Error(`no such path: ${path}`); + if (statSync(path).isFile()) return [path]; + const own = join(path, "result.json"); + const found: string[] = []; + if (existsSync(own) && isTrialResult(own)) return [own]; + if (depth >= 3) return found; + for (const entry of readdirSync(path, { withFileTypes: true })) { + if (entry.isDirectory()) found.push(...findTrialResults(join(path, entry.name), depth + 1)); + } + return found; +} + +/** + * Reasoning tokens opencode reported for a trial, from its own step records. + * + * Harbor's opencode adapter counts `tokens.output` alone, while opencode + * reports reasoning beside it as `tokens.reasoning`. Providers bill reasoning + * as output, and the other two arms' counts already include it. Without this, + * opencode's output column reads a third of what it generated. Its cost is + * unaffected: opencode prices both. + */ +export function opencodeReasoningTokens(trialDir: string): number | undefined { + const path = join(trialDir, "agent", "opencode.txt"); + if (!existsSync(path)) return undefined; + let total = 0; + for (const line of readFileSync(path, "utf8").split("\n")) { + if (!line.includes('"step_finish"')) continue; + try { + const event = JSON.parse(line) as { + type?: string; + part?: { tokens?: { reasoning?: number } }; + }; + if (event.type === "step_finish") total += event.part?.tokens?.reasoning ?? 0; + } catch { + // A line that is not an event. + } + } + return total; +} + +export function loadTrials(paths: readonly string[]): TrialRecord[] { + const trials: TrialRecord[] = []; + for (const path of paths) { + for (const file of findTrialResults(path)) { + let json: HarborTrialJson; + try { + json = JSON.parse(readFileSync(file, "utf8")) as HarborTrialJson; + } catch { + continue; + } + const record = parseTrial(json, basename(dirname(dirname(file)))); + if (!record) continue; + const reasoning = + record.agent === "opencode" && record.outputTokens !== undefined + ? opencodeReasoningTokens(dirname(file)) + : undefined; + trials.push( + reasoning ? { ...record, outputTokens: record.outputTokens! + reasoning } : record + ); + } + } + return labelArms(trials); +} diff --git a/examples/agent-eval/src/compare/report.ts b/examples/agent-eval/src/compare/report.ts new file mode 100644 index 0000000000..dae9b17407 --- /dev/null +++ b/examples/agent-eval/src/compare/report.ts @@ -0,0 +1,400 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { FailureMode, TrialRecord } from "./harborJobs"; +import type { Interval } from "./stats"; +import { mcnemarExactP, mean, median, passAtK, wilsonInterval } from "./stats"; + +export interface ArmSummary { + readonly arm: string; + readonly models: readonly string[]; + readonly versions: readonly string[]; + readonly tasks: number; + readonly trials: number; + /** Trials the infrastructure lost before the agent ran; excluded from every rate. */ + readonly blocked: number; + readonly passed: number; + /** Passes over trials that ran; undefined when none ran, which is no measurement, not zero. */ + readonly passRate: number | undefined; + readonly passRateCi: Interval | undefined; + /** Mean over tasks of the task's pass fraction: pass@1, unbiased with several attempts. */ + readonly passAt1: number | undefined; + /** pass@k at the largest k every task in this arm has attempts for, when above 1. */ + readonly passAtK: { readonly k: number; readonly value: number } | undefined; + readonly costUsd: number | undefined; + readonly costPerPass: number | undefined; + /** Trials that reported no cost; a total over the rest would understate. */ + readonly unpricedTrials: number; + /** Trials that reported no token counts; the token totals cover the rest. */ + readonly usageMissingTrials: number; + readonly inputTokens: number; + readonly cacheTokens: number; + readonly outputTokens: number; + readonly medianAgentSec: number | undefined; + readonly failures: Readonly>>; + readonly workglow: WorkglowArmDetail | undefined; +} + +export interface WorkglowArmDetail { + readonly outcomes: Readonly>; + readonly meanRounds: number | undefined; + readonly retries: number; + readonly toolCalls: number; + readonly toolErrors: number; + readonly byTool: Readonly>; + readonly maxHistoryChars: number; +} + +export interface PairComparison { + readonly a: string; + readonly b: string; + readonly commonTasks: number; + /** Tasks every attempt of both arms passed. */ + readonly bothPass: number; + readonly neither: number; + /** Tasks both arms passed on the same share of attempts, short of all of them. */ + readonly partialTie: number; + /** Tasks where `a`'s pass fraction is higher. */ + readonly aBetter: readonly string[]; + readonly bBetter: readonly string[]; + /** Exact sign / McNemar p-value over the tasks where the arms differ. */ + readonly pValue: number; +} + +export interface TaskCell { + readonly passes: number; + readonly runs: number; + readonly blocked: number; + readonly failures: readonly FailureMode[]; +} + +export interface ComparisonReport { + readonly generatedAt: string; + readonly arms: readonly ArmSummary[]; + readonly pairs: readonly PairComparison[]; + readonly tasks: ReadonlyArray<{ + readonly task: string; + readonly cells: Readonly>; + }>; +} + +function groupBy(items: readonly T[], key: (item: T) => string): Map { + const groups = new Map(); + for (const item of items) { + const k = key(item); + const list = groups.get(k); + if (list) list.push(item); + else groups.set(k, [item]); + } + return groups; +} + +function sum(values: ReadonlyArray): number { + return values.reduce((total, value) => total + (value ?? 0), 0); +} + +function workglowDetail(trials: readonly TrialRecord[]): WorkglowArmDetail | undefined { + const runs = trials.flatMap((trial) => (trial.workglow ? [trial.workglow] : [])); + if (runs.length === 0) return undefined; + const outcomes: Record = {}; + // Tool names are the model's to choose, `__proto__` included. + const byTool: Record = Object.create(null); + for (const run of runs) { + outcomes[run.outcome] = (outcomes[run.outcome] ?? 0) + 1; + for (const [name, tally] of Object.entries(run.tools.byName)) { + const entry = (byTool[name] ??= { calls: 0, errors: 0 }); + entry.calls += tally.calls; + entry.errors += tally.errors; + } + } + return { + outcomes, + meanRounds: mean(runs.map((run) => run.rounds)), + retries: sum(runs.map((run) => run.retries)), + toolCalls: sum(runs.map((run) => run.tools.calls)), + toolErrors: sum(runs.map((run) => run.tools.errors)), + byTool, + maxHistoryChars: Math.max(...runs.map((run) => run.finalHistoryChars)), + }; +} + +function summarizeArm(arm: string, trials: readonly TrialRecord[]): ArmSummary { + const ran = trials.filter((trial) => trial.status !== "blocked"); + const passed = ran.filter((trial) => trial.status === "pass").length; + const byTask = groupBy(ran, (trial) => trial.task); + const fractions = [...byTask.values()].map( + (list) => list.filter((trial) => trial.status === "pass").length / list.length + ); + const minAttempts = Math.min(...[...byTask.values()].map((list) => list.length)); + const passAtKValue = + byTask.size > 0 && minAttempts > 1 + ? { + k: minAttempts, + value: + mean( + [...byTask.values()].map((list) => + passAtK( + list.length, + list.filter((trial) => trial.status === "pass").length, + minAttempts + ) + ) + ) ?? 0, + } + : undefined; + const priced = ran.filter((trial) => trial.costUsd !== undefined); + const costUsd = priced.length > 0 ? sum(priced.map((trial) => trial.costUsd)) : undefined; + const failures: Partial> = {}; + for (const trial of ran) { + if (trial.failure) failures[trial.failure] = (failures[trial.failure] ?? 0) + 1; + } + return { + arm, + models: [...new Set(trials.flatMap((trial) => (trial.model ? [trial.model] : [])))], + versions: [ + ...new Set(trials.flatMap((trial) => (trial.agentVersion ? [trial.agentVersion] : []))), + ], + tasks: new Set(trials.map((trial) => trial.task)).size, + trials: trials.length, + blocked: trials.length - ran.length, + passed, + passRate: ran.length > 0 ? passed / ran.length : undefined, + passRateCi: ran.length > 0 ? wilsonInterval(passed, ran.length) : undefined, + passAt1: mean(fractions), + passAtK: passAtKValue, + costUsd, + // Only when every trial was priced: a total missing some trials, divided by + // every pass, would understate what a pass costs. + costPerPass: + costUsd !== undefined && passed > 0 && priced.length === ran.length + ? costUsd / passed + : undefined, + unpricedTrials: ran.length - priced.length, + usageMissingTrials: ran.filter( + (trial) => + trial.inputTokens === undefined || + trial.cacheTokens === undefined || + trial.outputTokens === undefined + ).length, + inputTokens: sum(ran.map((trial) => trial.inputTokens)), + cacheTokens: sum(ran.map((trial) => trial.cacheTokens)), + outputTokens: sum(ran.map((trial) => trial.outputTokens)), + medianAgentSec: median( + ran.flatMap((trial) => (trial.agentSec === undefined ? [] : [trial.agentSec])) + ), + failures, + workglow: workglowDetail(ran), + }; +} + +function cell(trials: readonly TrialRecord[]): TaskCell { + const ran = trials.filter((trial) => trial.status !== "blocked"); + return { + passes: ran.filter((trial) => trial.status === "pass").length, + runs: ran.length, + blocked: trials.length - ran.length, + failures: ran.flatMap((trial) => (trial.failure ? [trial.failure] : [])), + }; +} + +function fraction(c: TaskCell | undefined): number | undefined { + return c === undefined || c.runs === 0 ? undefined : c.passes / c.runs; +} + +/** + * The paired comparison is the one that answers "is A better than B": both + * arms ran the same tasks, so a task solved by both or by neither says + * nothing, and the tasks where they differ are both the evidence and the + * list worth reading transcripts for. + */ +function comparePair(a: string, b: string, tasks: ComparisonReport["tasks"]): PairComparison { + let commonTasks = 0; + let bothPass = 0; + let neither = 0; + let partialTie = 0; + const aBetter: string[] = []; + const bBetter: string[] = []; + for (const { task, cells } of tasks) { + const fa = fraction(cells[a]); + const fb = fraction(cells[b]); + if (fa === undefined || fb === undefined) continue; + commonTasks++; + if (fa > fb) aBetter.push(task); + else if (fb > fa) bBetter.push(task); + else if (fa === 0) neither++; + else if (fa === 1) bothPass++; + else partialTie++; + } + return { + a, + b, + commonTasks, + bothPass, + neither, + partialTie, + aBetter, + bBetter, + pValue: mcnemarExactP(aBetter.length, bBetter.length), + }; +} + +/** Whether any of the arm's trials ran: an arm whose every trial was blocked has no rate. */ +function ranAny(arm: ArmSummary): boolean { + return arm.trials > arm.blocked; +} + +export function buildReport(trials: readonly TrialRecord[]): ComparisonReport { + const byArm = groupBy(trials, (trial) => trial.arm); + const arms = [...byArm.entries()] + .map(([arm, list]) => summarizeArm(arm, list)) + // An arm none of whose trials ran has no rate to rank by, which is not a + // rate of zero; it goes last. + .sort( + (x, y) => + Number(ranAny(y)) - Number(ranAny(x)) || + (y.passAt1 ?? 0) - (x.passAt1 ?? 0) || + x.arm.localeCompare(y.arm) + ); + const armNames = arms.map((arm) => arm.arm); + + const tasks = [...groupBy(trials, (trial) => trial.task).entries()] + .map(([task, list]) => { + // Arm names come from job directories, so no inherited keys to collide with. + const cells: Record = Object.create(null); + for (const [arm, armTrials] of groupBy(list, (trial) => trial.arm)) + cells[arm] = cell(armTrials); + return { task, cells }; + }) + .sort((x, y) => x.task.localeCompare(y.task)); + + const pairs: PairComparison[] = []; + for (let i = 0; i < armNames.length; i++) { + for (let j = i + 1; j < armNames.length; j++) { + pairs.push(comparePair(armNames[i]!, armNames[j]!, tasks)); + } + } + return { generatedAt: new Date().toISOString(), arms, pairs, tasks }; +} + +const pct = (value: number): string => `${(value * 100).toFixed(1)}%`; +const usd = (value: number | undefined): string => + value === undefined ? "—" : `$${value.toFixed(value < 1 ? 4 : 2)}`; +const int = (value: number): string => value.toLocaleString("en-US"); +const secs = (value: number | undefined): string => + value === undefined ? "—" : `${value.toFixed(0)}s`; + +function markSymbol(c: TaskCell | undefined): string { + if (c === undefined) return "·"; + if (c.runs === 0) return c.blocked > 0 ? "B" : "·"; + if (c.runs === 1) return c.passes === 1 ? "✓" : `✗ ${c.failures[0] ?? ""}`.trim(); + return `${c.passes}/${c.runs}`; +} + +export function renderMarkdown( + report: ComparisonReport, + options: { readonly allTasks?: boolean } = {} +): string { + const lines: string[] = []; + lines.push(`# Agent harness comparison`, "", `Generated ${report.generatedAt}.`, ""); + + lines.push( + "| Arm | Model | Tasks | Ran | Blocked | Pass rate (95% CI) | pass@1 | pass@k | Cost | $/pass | In tok | Cache tok | Out tok | Median time |", + "|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|" + ); + for (const arm of report.arms) { + const cost = + arm.unpricedTrials > 0 && arm.costUsd !== undefined + ? `${usd(arm.costUsd)}*` + : usd(arm.costUsd); + const partial = arm.usageMissingTrials > 0 ? "†" : ""; + const rates = + arm.passRate !== undefined && arm.passRateCi !== undefined && arm.passAt1 !== undefined + ? `${pct(arm.passRate)} (${pct(arm.passRateCi.low)}–${pct(arm.passRateCi.high)}) | ${pct(arm.passAt1)}` + : "— | —"; + lines.push( + `| ${arm.arm} | ${arm.models.join(", ")} | ${arm.tasks} | ${arm.trials - arm.blocked} | ${arm.blocked} | ` + + `${rates} | ` + + `${arm.passAtK ? `${pct(arm.passAtK.value)} (k=${arm.passAtK.k})` : "—"} | ${cost} | ${usd(arm.costPerPass)} | ` + + `${int(arm.inputTokens)}${partial} | ${int(arm.cacheTokens)}${partial} | ${int(arm.outputTokens)}${partial} | ${secs(arm.medianAgentSec)} |` + ); + } + if (report.arms.some((arm) => arm.unpricedTrials > 0)) { + lines.push("", "\\* some trials reported no cost; the total covers the rest."); + } + if (report.arms.some((arm) => arm.usageMissingTrials > 0)) { + lines.push("", "† some trials reported no token counts; the totals cover the rest."); + } + + lines.push("", "## Head to head", ""); + lines.push( + "| A | B | Tasks | Both ✓ | Neither | Tied partial | A better | B better | p (exact sign) |", + "|---|---|---:|---:|---:|---:|---:|---:|---:|" + ); + for (const pair of report.pairs) { + lines.push( + `| ${pair.a} | ${pair.b} | ${pair.commonTasks} | ${pair.bothPass} | ${pair.neither} | ${pair.partialTie} | ` + + `${pair.aBetter.length} | ${pair.bBetter.length} | ${pair.pValue.toFixed(3)} |` + ); + } + for (const pair of report.pairs) { + if (pair.aBetter.length + pair.bBetter.length === 0) continue; + lines.push("", `### ${pair.a} vs ${pair.b}`, ""); + if (pair.aBetter.length > 0) + lines.push(`- **${pair.a}** better on: ${pair.aBetter.join(", ")}`); + if (pair.bBetter.length > 0) + lines.push(`- **${pair.b}** better on: ${pair.bBetter.join(", ")}`); + } + + lines.push("", "## Why runs failed", ""); + const modes = [ + ...new Set(report.arms.flatMap((arm) => Object.keys(arm.failures))), + ].sort() as FailureMode[]; + if (modes.length === 0) lines.push("No failures."); + else { + lines.push(`| Arm | ${modes.join(" | ")} |`, `|---|${modes.map(() => "---:").join("|")}|`); + for (const arm of report.arms) { + lines.push(`| ${arm.arm} | ${modes.map((mode) => arm.failures[mode] ?? 0).join(" | ")} |`); + } + } + + const workglowArms = report.arms.filter((arm) => arm.workglow); + if (workglowArms.length > 0) { + lines.push("", "## Workglow loop detail", ""); + for (const arm of workglowArms) { + const detail = arm.workglow!; + const outcomes = Object.entries(detail.outcomes) + .map(([outcome, n]) => `${outcome} ${n}`) + .join(", "); + const tools = Object.entries(detail.byTool) + .sort(([, x], [, y]) => y.calls - x.calls) + .map(([name, t]) => `${name} ${t.calls} (${t.errors} err)`) + .join(", "); + lines.push( + `- **${arm.arm}**: outcomes ${outcomes}; mean rounds ${detail.meanRounds?.toFixed(1) ?? "—"}; ` + + `retries ${detail.retries}; tool calls ${detail.toolCalls} with ${detail.toolErrors} errors ` + + `(${tools}); largest transcript ${int(detail.maxHistoryChars)} chars` + ); + } + } + + const arms = report.arms.map((arm) => arm.arm); + const rows = report.tasks.filter((row) => { + if (options.allTasks) return true; + const values = arms.map((arm) => fraction(row.cells[arm])).filter((v) => v !== undefined); + // An arm with no result for the task (blocked, or never run) is a gap in the + // comparison, and the row is how the reader finds it. + return new Set(values).size > 1 || values.length < arms.length; + }); + lines.push("", options.allTasks ? "## Every task" : "## Tasks where the arms differ", ""); + if (rows.length === 0) lines.push("None."); + else { + lines.push(`| Task | ${arms.join(" | ")} |`, `|---|${arms.map(() => "---").join("|")}|`); + for (const row of rows) + lines.push(`| ${row.task} | ${arms.map((arm) => markSymbol(row.cells[arm])).join(" | ")} |`); + } + lines.push(""); + return lines.join("\n"); +} diff --git a/examples/agent-eval/src/compare/stats.ts b/examples/agent-eval/src/compare/stats.ts new file mode 100644 index 0000000000..29b612fa90 --- /dev/null +++ b/examples/agent-eval/src/compare/stats.ts @@ -0,0 +1,70 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +export interface Interval { + readonly low: number; + readonly high: number; +} + +/** + * Wilson score interval for a pass rate. Benchmark arms are small (tens of + * tasks) and rates near 0 or 1 are common, which is where the normal + * approximation gives intervals outside [0, 1]. + */ +export function wilsonInterval(successes: number, trials: number, z = 1.96): Interval { + if (trials === 0) return { low: 0, high: 0 }; + const p = successes / trials; + const z2 = z * z; + const denominator = 1 + z2 / trials; + const center = (p + z2 / (2 * trials)) / denominator; + const margin = (z * Math.sqrt((p * (1 - p)) / trials + z2 / (4 * trials * trials))) / denominator; + return { low: Math.max(0, center - margin), high: Math.min(1, center + margin) }; +} + +function logChoose(n: number, k: number): number { + let sum = 0; + for (let i = 1; i <= k; i++) sum += Math.log(n - k + i) - Math.log(i); + return sum; +} + +/** + * Unbiased pass@k for one task from n attempts with c passes (Chen et al., + * 2021): the chance that at least one of k attempts drawn without replacement + * passes. + */ +export function passAtK(n: number, c: number, k: number): number { + if (k > n) throw new Error(`pass@${k} needs at least ${k} attempts, got ${n}`); + if (n - c < k) return 1; + return 1 - Math.exp(logChoose(n - c, k) - logChoose(n, k)); +} + +/** + * Exact two-sided McNemar test on the discordant pairs of a paired + * comparison: `onlyA` tasks the first arm solved and the second did not, and + * the reverse. The concordant tasks carry no information about which arm is + * better, which is why a paired test needs far fewer tasks than comparing two + * pass rates as if they were independent samples. + */ +export function mcnemarExactP(onlyA: number, onlyB: number): number { + const n = onlyA + onlyB; + if (n === 0) return 1; + const k = Math.min(onlyA, onlyB); + let tail = 0; + for (let i = 0; i <= k; i++) tail += Math.exp(logChoose(n, i) - n * Math.LN2); + return Math.min(1, 2 * tail); +} + +export function median(values: readonly number[]): number | undefined { + if (values.length === 0) return undefined; + const sorted = [...values].sort((a, b) => a - b); + const mid = Math.floor(sorted.length / 2); + return sorted.length % 2 === 1 ? sorted[mid]! : (sorted[mid - 1]! + sorted[mid]!) / 2; +} + +export function mean(values: readonly number[]): number | undefined { + if (values.length === 0) return undefined; + return values.reduce((sum, value) => sum + value, 0) / values.length; +} diff --git a/examples/agent-eval/src/mock/anthropicMock.ts b/examples/agent-eval/src/mock/anthropicMock.ts new file mode 100644 index 0000000000..6e4364d7d9 --- /dev/null +++ b/examples/agent-eval/src/mock/anthropicMock.ts @@ -0,0 +1,224 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { appendFileSync } from "node:fs"; +import type { IncomingMessage, Server, ServerResponse } from "node:http"; +import { createServer } from "node:http"; + +/** + * A scripted stand-in for the Anthropic Messages API, for checking the whole + * pipeline — install, run, usage capture, verification, comparison — without a + * key or a bill. Every harness in the comparison can be pointed at an + * Anthropic base URL, so one fake serves all three. + * + * The script is a list of shell commands. Each request is answered with a call + * to whatever tool the harness named `bash`, running the next command — the + * position is the number of assistant turns already in the conversation — + * and once the commands run out, with a short text answer. That makes the + * behaviour a function of the conversation alone, so it is the same for every + * harness and survives the retries and side requests (titles, summaries) a + * harness makes. + */ +export interface AnthropicMockOptions { + readonly port: number; + readonly host?: string | undefined; + readonly commands: readonly string[]; + /** Append every request body here as JSON lines, for inspection. */ + readonly requestLog?: string | undefined; + readonly finalText?: string | undefined; +} + +interface MessagesRequest { + readonly model?: string; + readonly stream?: boolean; + readonly system?: unknown; + readonly messages?: ReadonlyArray<{ role: string; content: unknown }>; + readonly tools?: ReadonlyArray<{ + name: string; + input_schema?: { properties?: Record; required?: string[] }; + }>; +} + +type Reply = + | { readonly kind: "text"; readonly text: string } + | { readonly kind: "tool"; readonly name: string; readonly input: Record }; + +let nextId = 0; +const id = (prefix: string): string => `${prefix}_mock${(++nextId).toString().padStart(6, "0")}`; + +/** Fills a tool's required arguments: the command where it wants one, a placeholder elsewhere. */ +function bashInput( + tool: NonNullable[number], + command: string +): Record { + const input: Record = { command }; + const properties = tool.input_schema?.properties ?? {}; + for (const name of tool.input_schema?.required ?? []) { + if (name in input) continue; + const type = properties[name]?.type; + input[name] = + type === "number" || type === "integer" + ? 60 + : type === "boolean" + ? false + : "Run the next step"; + } + return input; +} + +export function decideReply(request: MessagesRequest, options: AnthropicMockOptions): Reply { + const finalText = options.finalText ?? "Done. The task is complete."; + const bash = request.tools?.find((tool) => tool.name.toLowerCase() === "bash"); + if (bash === undefined) return { kind: "text", text: finalText }; + const step = (request.messages ?? []).filter((message) => message.role === "assistant").length; + const command = options.commands[step]; + if (command === undefined) return { kind: "text", text: finalText }; + return { kind: "tool", name: bash.name, input: bashInput(bash, command) }; +} + +function promptSize(request: MessagesRequest): number { + return Math.ceil(JSON.stringify([request.system, request.messages, request.tools]).length / 4); +} + +function sse(res: ServerResponse, event: string, data: unknown): void { + res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`); +} + +function respond(res: ServerResponse, request: MessagesRequest, reply: Reply): void { + const messageId = id("msg"); + const model = request.model ?? "mock"; + const usage = { + input_tokens: promptSize(request), + output_tokens: 24, + cache_creation_input_tokens: 0, + cache_read_input_tokens: 0, + }; + const stopReason = reply.kind === "tool" ? "tool_use" : "end_turn"; + const block = + reply.kind === "tool" + ? { type: "tool_use", id: id("toolu"), name: reply.name, input: reply.input } + : { type: "text", text: reply.text }; + + if (!request.stream) { + res.writeHead(200, { "content-type": "application/json" }); + res.end( + JSON.stringify({ + id: messageId, + type: "message", + role: "assistant", + model, + content: [block], + stop_reason: stopReason, + stop_sequence: null, + usage, + }) + ); + return; + } + + res.writeHead(200, { + "content-type": "text/event-stream", + "cache-control": "no-cache", + connection: "keep-alive", + }); + sse(res, "message_start", { + type: "message_start", + message: { + id: messageId, + type: "message", + role: "assistant", + model, + content: [], + stop_reason: null, + stop_sequence: null, + usage: { ...usage, output_tokens: 1 }, + }, + }); + if (block.type === "tool_use") { + sse(res, "content_block_start", { + type: "content_block_start", + index: 0, + content_block: { type: "tool_use", id: block.id, name: block.name, input: {} }, + }); + sse(res, "content_block_delta", { + type: "content_block_delta", + index: 0, + delta: { type: "input_json_delta", partial_json: JSON.stringify(block.input) }, + }); + } else { + sse(res, "content_block_start", { + type: "content_block_start", + index: 0, + content_block: { type: "text", text: "" }, + }); + sse(res, "content_block_delta", { + type: "content_block_delta", + index: 0, + delta: { type: "text_delta", text: block.text }, + }); + } + sse(res, "content_block_stop", { type: "content_block_stop", index: 0 }); + sse(res, "message_delta", { + type: "message_delta", + delta: { stop_reason: stopReason, stop_sequence: null }, + usage: { output_tokens: usage.output_tokens }, + }); + sse(res, "message_stop", { type: "message_stop" }); + res.end(); +} + +async function readBody(req: IncomingMessage): Promise { + const chunks: Buffer[] = []; + for await (const chunk of req) chunks.push(chunk as Buffer); + return Buffer.concat(chunks).toString("utf8"); +} + +export function startAnthropicMock(options: AnthropicMockOptions): Promise { + const server = createServer(async (req, res) => { + try { + const path = (req.url ?? "").split("?")[0] ?? ""; + if (req.method === "POST" && path.endsWith("/messages/count_tokens")) { + const request = JSON.parse(await readBody(req)) as MessagesRequest; + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ input_tokens: promptSize(request) })); + return; + } + if (req.method === "POST" && path.endsWith("/messages")) { + const body = await readBody(req); + const request = JSON.parse(body) as MessagesRequest; + if (options.requestLog) { + appendFileSync( + options.requestLog, + `${JSON.stringify({ at: new Date().toISOString(), request })}\n` + ); + } + respond(res, request, decideReply(request, options)); + return; + } + res.writeHead(404, { "content-type": "application/json" }); + res.end(JSON.stringify({ type: "error", error: { type: "not_found_error", message: path } })); + } catch (error) { + res.writeHead(400, { "content-type": "application/json" }); + res.end( + JSON.stringify({ + type: "error", + error: { + type: "invalid_request_error", + message: error instanceof SyntaxError ? "request body is not JSON" : "invalid request", + }, + }) + ); + } + }); + return new Promise((resolve, reject) => { + // A port already in use is reported as an error event, not thrown. + server.once("error", reject); + server.listen(options.port, options.host ?? "127.0.0.1", () => { + server.off("error", reject); + resolve(server); + }); + }); +} diff --git a/examples/agent-eval/src/run/jobConfig.ts b/examples/agent-eval/src/run/jobConfig.ts new file mode 100644 index 0000000000..09d80f928f --- /dev/null +++ b/examples/agent-eval/src/run/jobConfig.ts @@ -0,0 +1,189 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { JobDataset } from "./suites"; + +export const HARNESSES = ["workglow", "opencode", "pi"] as const; +export type Harness = (typeof HARNESSES)[number]; + +/** One column of the comparison: a harness and the kwargs that make it this arm. */ +export interface ArmSpec { + readonly harness: Harness; + readonly kwargs: Readonly>; +} + +/** Reasoning levels every harness can be set to, in pi's vocabulary. */ +export const EFFORTS = ["off", "minimal", "low", "medium", "high", "xhigh", "max"] as const; +export type Effort = (typeof EFFORTS)[number]; + +export interface ParityOptions { + /** Model calls each arm may make. Every harness gets the same cap. */ + readonly maxTurns: number; + readonly effort: Effort; + /** opencode names reasoning levels per model ("high", "max", …); unset leaves it at its default. */ + readonly opencodeVariant: string | undefined; + /** + * pi needs the wire API spelled out when the provider endpoint is + * overridden (a proxy or a mock): `anthropic-messages`, `openai-responses`, … + */ + readonly piModelApi: string | undefined; +} + +/** + * The kwargs that hold each harness to the same budget and reasoning level. + * + * Without them the arms are not comparable: opencode and pi run until the + * model stops (no turn cap at all), pi thinks at `medium` by default while the + * other two do not think unless asked, and AgentTask stops after 8 rounds. + * An arm's own kwargs win, so an ablation can still move one of these. + */ +export function parityKwargs(harness: Harness, options: ParityOptions): Record { + switch (harness) { + case "workglow": + return { max_rounds: options.maxTurns, effort: options.effort }; + case "opencode": + return { + opencode_config: { agent: { build: { steps: options.maxTurns } } }, + ...(options.opencodeVariant ? { variant: options.opencodeVariant } : {}), + }; + case "pi": + return { + max_turns: options.maxTurns, + thinking: options.effort, + ...(options.piModelApi ? { model_api: options.piModelApi } : {}), + }; + } +} + +/** + * Where the value starting at `start` ends. A JSON string, array or object runs to + * its closing quote or bracket, so it may hold anything; a bare value runs to the + * next comma that a `key=` follows, so a sentence or a list can carry its own commas. + */ +function kwargValueEnd(text: string, start: number): number { + let i = start; + while (i < text.length && /\s/.test(text[i]!)) i++; + const open = text[i]; + if (open === '"' || open === "{" || open === "[") { + let depth = 0; + let inString = false; + for (; i < text.length; i++) { + const c = text[i]; + if (inString) { + if (c === "\\") i++; + else if (c === '"') { + inString = false; + if (depth === 0) return i + 1; + } + } else if (c === '"') { + inString = true; + } else if (c === "{" || c === "[") { + depth++; + } else if (c === "}" || c === "]") { + depth--; + if (depth === 0) return i + 1; + } + } + // An opened value that never closes would otherwise swallow every pair after it. + throw new Error(`arm kwarg value starting ${text.slice(start).trim()} is never closed`); + } + const next = text.slice(start).search(/,(?=\s*[A-Za-z_][\w-]*=)/); + return next === -1 ? text.length : start + next; +} + +/** + * `workglow`, `pi:thinking=high`, `workglow:tool_concurrency=8,max_history_chars=400000`. + * Values are JSON where they parse as JSON, strings otherwise — Harbor's own + * `--ak` rule. A value holding a comma followed by `key=` is written as a JSON + * string: `workglow:append_system_prompt="When x=a, set y=b"`. + */ +export function parseArm(spec: string): ArmSpec { + const colon = spec.indexOf(":"); + const name = (colon === -1 ? spec : spec.slice(0, colon)).trim(); + if (!(HARNESSES as readonly string[]).includes(name)) { + throw new Error(`unknown harness "${name}" — one of ${HARNESSES.join(", ")}`); + } + const kwargs: Record = {}; + const body = colon === -1 ? "" : spec.slice(colon + 1); + let pos = 0; + while (pos < body.length) { + const eq = body.indexOf("=", pos); + const comma = body.indexOf(",", pos); + if (eq === -1 || (comma !== -1 && comma < eq)) { + const pair = body.slice(pos, comma === -1 ? undefined : comma).trim(); + if (pair !== "") throw new Error(`arm kwarg "${pair}" must be key=value`); + pos = comma === -1 ? body.length : comma + 1; + continue; + } + const key = body.slice(pos, eq).trim(); + const end = kwargValueEnd(body, eq + 1); + const raw = body.slice(eq + 1, end).trim(); + let value: unknown = raw; + try { + value = JSON.parse(raw); + } catch { + // A bare string. + } + kwargs[key] = value; + const after = body.indexOf(",", end); + const trailing = body.slice(end, after === -1 ? undefined : after).trim(); + if (trailing !== "") + throw new Error(`arm kwarg "${key}" has text after its value: ${trailing}`); + pos = after === -1 ? body.length : after + 1; + } + return { harness: name as Harness, kwargs }; +} + +export interface HarborJobOptions { + readonly jobName: string; + readonly jobsDir: string; + readonly model: string; + readonly arms: readonly ArmSpec[]; + readonly datasets: readonly JobDataset[]; + readonly parity: ParityOptions; + readonly attempts: number; + readonly concurrency: number; + readonly timeoutMultiplier: number | undefined; +} + +/** + * Exceptions that mean the infrastructure failed before or around the agent: + * a container that would not build or start (Docker reports those as + * `RuntimeError`), or an install that timed out. Retrying them is free of + * bias. Nothing the agent itself did is listed — retrying an agent failure + * would turn one attempt into best-of-n for whichever arm failed more. + */ +export const INFRASTRUCTURE_RETRY_EXCEPTIONS = [ + "RuntimeError", + "EnvironmentStartTimeoutError", + "AgentSetupTimeoutError", +] as const; + +/** + * One Harbor job holding every arm, so all of them run the same task list + * under the same environment settings; the comparison then pairs trials by + * task. The workglow arm is loaded by import path, which needs this + * package's `harbor/` directory on PYTHONPATH. + */ +export function buildHarborJob(options: HarborJobOptions): Record { + return { + job_name: options.jobName, + jobs_dir: options.jobsDir, + n_attempts: options.attempts, + n_concurrent_trials: options.concurrency, + retry: { max_retries: 2, include_exceptions: [...INFRASTRUCTURE_RETRY_EXCEPTIONS] }, + ...(options.timeoutMultiplier === undefined + ? {} + : { timeout_multiplier: options.timeoutMultiplier }), + agents: options.arms.map((arm) => { + const kwargs = { ...parityKwargs(arm.harness, options.parity), ...arm.kwargs }; + return arm.harness === "workglow" + ? { import_path: "workglow_agent:WorkglowAgent", model_name: options.model, kwargs } + : { name: arm.harness, model_name: options.model, kwargs }; + }), + datasets: options.datasets.map((dataset) => ({ ...dataset })), + }; +} diff --git a/examples/agent-eval/src/run/runHarbor.ts b/examples/agent-eval/src/run/runHarbor.ts new file mode 100644 index 0000000000..781f05bdeb --- /dev/null +++ b/examples/agent-eval/src/run/runHarbor.ts @@ -0,0 +1,97 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { spawn } from "node:child_process"; +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { delimiter, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +/** This package's root, found from this module whether it runs from src or a bundle in dist. */ +export function packageRoot(): string { + let dir = fileURLToPath(new URL(".", import.meta.url)); + for (let i = 0; i < 4; i++) { + if (existsSync(join(dir, "harbor", "workglow_agent.py"))) return dir; + dir = resolve(dir, ".."); + } + throw new Error("cannot find the agent-eval package root (harbor/workglow_agent.py)"); +} + +export interface HarborRun { + readonly config: Record; + /** Extra Harbor config files layered after the generated one (environment overlays and the like). */ + readonly extraConfigs: readonly string[]; + readonly harbor: string; +} + +interface HarborAgentConfig { + readonly import_path?: string; + readonly kwargs?: Record; +} + +/** + * What stops a workglow arm from finding the agent bundle it would upload, if + * anything. An arm's own `bundle_path` wins over the shared one, as it does in + * the adapter, which expands `~` and resolves it from the same working directory. + */ +export function missingBundle( + agents: readonly HarborAgentConfig[], + bundle: string +): string | undefined { + for (const agent of agents) { + if (!agent.import_path?.startsWith("workglow_agent")) continue; + const own = agent.kwargs?.bundle_path; + if (typeof own === "string" && own.length > 0) { + const path = resolve(own.replace(/^~(?=$|\/)/, homedir())); + if (!existsSync(path)) return `the workglow arm's bundle_path ${path} does not exist`; + } else if (!existsSync(bundle)) { + return `the workglow arm needs ${bundle}; run \`bun run build-agent\` first`; + } + } + return undefined; +} + +/** + * Writes the job config beside the job's results and runs `harbor run` on it, + * with the workglow adapter importable and the bundle it uploads located. + * Harbor's own progress display goes straight to the terminal. + */ +export async function runHarbor(run: HarborRun): Promise<{ jobDir: string; exitCode: number }> { + const root = packageRoot(); + const bundle = process.env.WORKGLOW_AGENT_BUNDLE ?? join(root, "dist", "workglow-agent.mjs"); + const jobsDir = resolve(String(run.config.jobs_dir)); + const jobDir = join(jobsDir, String(run.config.job_name)); + mkdirSync(jobsDir, { recursive: true }); + const configPath = join(jobsDir, `${String(run.config.job_name)}.config.json`); + writeFileSync(configPath, JSON.stringify({ ...run.config, jobs_dir: jobsDir }, null, 2)); + + const problem = missingBundle(run.config.agents as readonly HarborAgentConfig[], bundle); + if (problem !== undefined) throw new Error(problem); + + const args = [ + "run", + "-c", + configPath, + ...run.extraConfigs.flatMap((path) => ["-c", resolve(path)]), + ]; + const env = { + ...process.env, + PYTHONPATH: [join(root, "harbor"), process.env.PYTHONPATH].filter(Boolean).join(delimiter), + WORKGLOW_AGENT_BUNDLE: bundle, + }; + const exitCode = await new Promise((resolvePromise, reject) => { + const child = spawn(run.harbor, args, { stdio: "inherit", env }); + child.on("error", (error) => + reject( + new Error( + `could not start ${run.harbor} (${error.message}); install Harbor with \`uv tool install harbor\`` + ) + ) + ); + child.on("exit", (code) => resolvePromise(code ?? 1)); + }); + return { jobDir, exitCode }; +} diff --git a/examples/agent-eval/src/run/suites.ts b/examples/agent-eval/src/run/suites.ts new file mode 100644 index 0000000000..8c8ef83785 --- /dev/null +++ b/examples/agent-eval/src/run/suites.ts @@ -0,0 +1,200 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** One Harbor registry dataset, optionally narrowed to some of its tasks. */ +export interface SuiteDataset { + readonly name: string; + readonly version: string; + /** Glob patterns over task names. */ + readonly task_names?: readonly string[] | undefined; + /** Cap applied after `task_names`, in registry order. */ + readonly n_tasks?: number | undefined; +} + +/** + * A directory of Harbor task folders on disk — a custom benchmark, or a local + * copy of a registry dataset prepared for an environment the registry copy + * cannot build in. + */ +export interface LocalDataset { + readonly path: string; + readonly task_names?: readonly string[] | undefined; + readonly n_tasks?: number | undefined; +} + +export type JobDataset = SuiteDataset | LocalDataset; + +export interface Suite { + readonly description: string; + readonly datasets: readonly SuiteDataset[]; +} + +/** + * Named slices of public benchmarks in Harbor's registry. Each is chosen for + * what it exercises in a loop, not only for its name: + * + * - Terminal-Bench 2.0 is long-horizon terminal work — builds, services, + * debugging — where context growth, command timeouts and recovery from tool + * errors decide the outcome. + * - Aider Polyglot is short edit-run-fix cycles in six languages, where edit + * reliability and reading test output dominate. + * - SWE-bench Verified is repository navigation and a minimal patch in a + * large Python codebase: search, reading at scale, and not breaking + * neighbouring tests. + */ +export const SUITES: Readonly> = { + smoke: { + description: + "hello-world plus the 10-task Terminal-Bench sample: does every arm install and run", + datasets: [ + { name: "hello-world", version: "1.0" }, + { name: "terminal-bench-sample", version: "2.0" }, + ], + }, + "terminal-bench": { + description: "Terminal-Bench 2.0, all 89 tasks", + datasets: [{ name: "terminal-bench", version: "2.0" }], + }, + polyglot: { + description: "Aider Polyglot, all 225 exercises in six languages", + datasets: [{ name: "aider-polyglot", version: "1.0" }], + }, + "polyglot-lite": { + description: "Aider Polyglot, Python and JavaScript only (83 exercises)", + datasets: [ + { + name: "aider-polyglot", + version: "1.0", + task_names: ["polyglot_python_*", "polyglot_javascript_*"], + }, + ], + }, + "swebench-50": { + description: "SWE-bench Verified, the first 50 tasks in registry order", + datasets: [{ name: "swebench-verified", version: "1.0", n_tasks: 50 }], + }, + core: { + description: + "Terminal-Bench 2.0 + Polyglot (Python/JS) + SWE-bench Verified 50: the decision set", + datasets: [ + { name: "terminal-bench", version: "2.0" }, + { + name: "aider-polyglot", + version: "1.0", + task_names: ["polyglot_python_*", "polyglot_javascript_*"], + }, + { name: "swebench-verified", version: "1.0", n_tasks: 50 }, + ], + }, +}; + +export function getSuite(name: string): Suite { + const suite = SUITES[name]; + if (suite === undefined) { + throw new Error(`unknown suite "${name}" — one of ${Object.keys(SUITES).join(", ")}`); + } + return suite; +} + +function globRegExp(glob: string): RegExp { + const body = glob + .replace(/[.+^${}()|[\]\\]/g, "\\$&") + .replace(/\*/g, ".*") + .replace(/\?/g, "."); + return new RegExp(`^${body}$`); +} + +/** A glob with exactly one `*` and no `?`, split at the star. */ +function splitStar(glob: string): { readonly head: string; readonly tail: string } | undefined { + const star = glob.indexOf("*"); + if (star < 0 || glob.indexOf("*", star + 1) >= 0 || glob.includes("?")) return undefined; + return { head: glob.slice(0, star), tail: glob.slice(star + 1) }; +} + +/** + * Globs matching exactly the names that start with `prefix` and end with `suffix`. + * `prefix*suffix` alone misses the names where the two overlap (`ab` and `bc` both + * match `abc`), so each such overlap is listed as a literal name beside it. + */ +function prefixSuffixGlobs(prefix: string, suffix: string): string[] { + if (suffix === "") return [`${prefix}*`]; + const globs = [`${prefix}*${suffix}`]; + for (let k = Math.min(prefix.length, suffix.length); k > 0; k--) { + if (prefix.endsWith(suffix.slice(0, k))) globs.push(prefix + suffix.slice(k)); + } + return globs; +} + +/** + * The task names `a` and `b` both match, as globs Harbor takes as one list: `[]` when + * no name can match both, `undefined` when the overlap is not one this can write down. + * Decided for literals, and for a single-`*` glob against a prefix glob (`P*`), which + * is the shape every suite here restricts itself with; never for a `[...]` class. + */ +function globIntersection(a: string, b: string): string[] | undefined { + if (a === b) return [a]; + // Harbor matches with fnmatch, where `[...]` is a character class; nothing here + // reasons about classes, so an overlap involving one is not written down. + if (a.includes("[") || b.includes("[")) return undefined; + if (!/[*?]/.test(a)) return globRegExp(b).test(a) ? [a] : []; + if (!/[*?]/.test(b)) return globRegExp(a).test(b) ? [b] : []; + const sa = splitStar(a); + const sb = splitStar(b); + if (!sa || !sb) return undefined; + const [glob, prefix] = sb.tail === "" ? [sa, sb.head] : sa.tail === "" ? [sb, sa.head] : []; + if (!glob || prefix === undefined) return undefined; + if (glob.head.startsWith(prefix)) return [`${glob.head}*${glob.tail}`]; + if (prefix.startsWith(glob.head)) return prefixSuffixGlobs(prefix, glob.tail); + return []; +} + +/** + * A dataset narrowed by `--tasks` and `--n-tasks`. Both only ever narrow: the + * filter is intersected with the tasks the suite already restricts itself to + * (Harbor takes one list, so the intersection is computed here), and the cap + * replaces the suite's only when it is lower. Undefined when the filter shares + * no task with the dataset, which then has nothing to run. + */ +export function narrowDataset( + dataset: T, + taskFilter: readonly string[] | undefined, + nTasks: number | undefined +): T | undefined { + let taskNames = dataset.task_names; + if (taskFilter && taskFilter.length > 0) { + if (!taskNames) { + taskNames = [...taskFilter]; + } else { + const kept = new Set(); + for (const wanted of taskFilter) { + for (const allowed of taskNames) { + const both = globIntersection(wanted, allowed); + if (both === undefined) { + throw new Error( + `--tasks ${wanted} cannot be narrowed to this suite's ${allowed}: use a task ` + + `name, or a glob with one "*" and no [...] that starts within one of the suite's ` + + `(${taskNames.join(", ")})` + ); + } + for (const glob of both) kept.add(glob); + } + } + if (kept.size === 0) return undefined; + taskNames = [...kept]; + } + } + const cap = + nTasks === undefined + ? dataset.n_tasks + : dataset.n_tasks === undefined + ? nTasks + : Math.min(dataset.n_tasks, nTasks); + return { + ...dataset, + ...(taskNames ? { task_names: taskNames } : {}), + ...(cap === undefined ? {} : { n_tasks: cap }), + }; +} diff --git a/examples/agent-eval/src/test/atif.test.ts b/examples/agent-eval/src/test/atif.test.ts new file mode 100644 index 0000000000..e22e044524 --- /dev/null +++ b/examples/agent-eval/src/test/atif.test.ts @@ -0,0 +1,136 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AgentStep, ChatMessage } from "@workglow/ai"; +import { describe, expect, it } from "vitest"; +import { toAtifTrajectory } from "../agent/atif"; +import type { AgentRunSummary } from "../agent/runSummary"; +import { summarizeUsage, tallyTools } from "../agent/runSummary"; + +describe("toAtifTrajectory", () => { + it("counts image tokens in a step's prompt and cached figures, as the totals do", () => { + const steps: AgentStep[] = [ + { + round: 1, + startedAt: "2026-10-11T00:00:00.000Z", + modelMs: 10, + durationMs: 10, + attempts: 1, + text: "done", + tools: [], + usage: { + input: 100, + imageInput: 40, + cached: 20, + imageCached: 5, + cacheWrite: 3, + output: 7, + reasoning: undefined, + total: undefined, + extra: undefined, + }, + costUsd: undefined, + }, + ]; + const messages: ChatMessage[] = [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { role: "assistant", content: [{ type: "text", text: "done" }] }, + ]; + const usage = summarizeUsage(steps); + const summary = { model: "m", usage, costUsd: undefined } as unknown as AgentRunSummary; + + const trajectory = toAtifTrajectory("s", "0.1.0", messages, steps, summary); + + expect(trajectory.steps[1]?.metrics).toEqual({ + prompt_tokens: 168, + completion_tokens: 7, + cached_tokens: 25, + cost_usd: null, + }); + expect(trajectory.steps[1]?.metrics?.prompt_tokens).toBe( + trajectory.final_metrics.total_prompt_tokens + ); + expect(trajectory.steps[1]?.metrics?.cached_tokens).toBe( + trajectory.final_metrics.total_cached_tokens + ); + }); +}); + +describe("toAtifTrajectory metrics", () => { + it("gives a round that reported only a cost no token metrics", () => { + const steps = [ + { + round: 1, + startedAt: "2026-10-11T00:00:00.000Z", + modelMs: 1, + durationMs: 1, + attempts: 1, + text: "done", + tools: [], + usage: { costUsd: 0.01 }, + costUsd: 0.01, + }, + ] as unknown as AgentStep[]; + const messages: ChatMessage[] = [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { role: "assistant", content: [{ type: "text", text: "done" }] }, + ]; + const summary = { + model: "m", + usage: summarizeUsage(steps), + costUsd: 0.01, + } as unknown as AgentRunSummary; + const trajectory = toAtifTrajectory("s", "0.1.0", messages, steps, summary); + expect(trajectory.steps[1]).not.toHaveProperty("metrics"); + }); +}); + +describe("summarizeUsage", () => { + it("counts a round whose usage carries no token count as unreported, not as zero", () => { + const steps = [ + { usage: { input: 10, output: 2 } }, + { usage: { costUsd: 0.01 } }, + { usage: undefined }, + ] as unknown as AgentStep[]; + const usage = summarizeUsage(steps); + expect(usage.unreportedRounds).toBe(2); + expect(usage.inputTokens).toBe(10); + }); +}); + +describe("summarizeUsage cache counts", () => { + it("counts a round with tokens but no cache figure as cache-unreported, not as zero", () => { + const steps = [ + { usage: { input: 10, output: 2, cached: 4 } }, + { usage: { input: 10, output: 2 } }, + ] as unknown as AgentStep[]; + const usage = summarizeUsage(steps); + expect(usage.unreportedRounds).toBe(0); + expect(usage.cacheUnreportedRounds).toBe(1); + const summary = { model: "m", usage, costUsd: undefined } as unknown as AgentRunSummary; + expect(toAtifTrajectory("s", "0.1.0", [], [], summary).final_metrics.total_cached_tokens).toBe( + null + ); + }); +}); + +describe("tallyTools", () => { + it("counts a tool named after an Object.prototype key without touching the prototype", () => { + const steps = [ + { + tools: [ + { name: "__proto__", isError: true }, + { name: "constructor", isError: false }, + ], + }, + ] as unknown as AgentStep[]; + const tally = tallyTools(steps); + expect(tally.byName["__proto__"]).toEqual({ calls: 1, errors: 1 }); + expect(tally.byName["constructor"]).toEqual({ calls: 1, errors: 0 }); + expect(({} as Record).calls).toBeUndefined(); + expect(JSON.parse(JSON.stringify(tally)).byName.constructor).toEqual({ calls: 1, errors: 0 }); + }); +}); diff --git a/examples/agent-eval/src/test/compare.test.ts b/examples/agent-eval/src/test/compare.test.ts new file mode 100644 index 0000000000..0c6a6ed952 --- /dev/null +++ b/examples/agent-eval/src/test/compare.test.ts @@ -0,0 +1,444 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import type { TrialRecord } from "../compare/harborJobs"; +import { classifyTrial, labelArms, loadTrials } from "../compare/harborJobs"; +import { buildReport, renderMarkdown } from "../compare/report"; +import { mcnemarExactP, passAtK, wilsonInterval } from "../compare/stats"; + +describe("stats", () => { + it("bounds a Wilson interval inside [0, 1]", () => { + const all = wilsonInterval(5, 5); + expect(all.high).toBe(1); + expect(all.low).toBeGreaterThan(0.5); + expect(wilsonInterval(0, 0)).toEqual({ low: 0, high: 0 }); + }); + + it("computes unbiased pass@k", () => { + expect(passAtK(5, 0, 1)).toBe(0); + expect(passAtK(5, 5, 3)).toBe(1); + expect(passAtK(4, 1, 1)).toBeCloseTo(0.25); + expect(passAtK(4, 1, 2)).toBeCloseTo(0.5); + }); + + it("gives an exact two-sided McNemar p-value", () => { + expect(mcnemarExactP(0, 0)).toBe(1); + expect(mcnemarExactP(5, 5)).toBe(1); + // 8 discordant pairs all one way: 2 * 0.5^8. + expect(mcnemarExactP(8, 0)).toBeCloseTo(2 / 256); + }); +}); + +describe("classifyTrial", () => { + it("reads an API error reporting an over-long prompt as a context overflow", () => { + expect( + classifyTrial(0, "ApiError", "maximum context length exceeded", true, undefined).failure + ).toBe("context-overflow"); + expect( + classifyTrial(0, "ApiError", "rate limit: too many tokens per minute", true, undefined) + .failure + ).toBe("rate-limit"); + }); + + it("believes a reward even after an agent timeout", () => { + expect(classifyTrial(1, "AgentTimeoutError", "", true, undefined).status).toBe("pass"); + }); + + it("calls an exception before the agent ran blocked", () => { + expect( + classifyTrial(undefined, "RuntimeError", "docker compose failed", false, undefined).status + ).toBe("blocked"); + }); + + it("names failure modes", () => { + expect(classifyTrial(0, "AgentTimeoutError", "", true, undefined).failure).toBe( + "agent-timeout" + ); + expect(classifyTrial(0, undefined, undefined, true, undefined).failure).toBe("wrong-answer"); + const overflow = { + outcome: "error", + error: "prompt is too long: 210000 tokens > 200000 maximum", + rounds: 40, + retries: 0, + tools: { calls: 0, errors: 0, byName: {} }, + finalHistoryChars: 0, + }; + expect(classifyTrial(0, "NonZeroAgentExitCodeError", "exit 1", true, overflow).failure).toBe( + "context-overflow" + ); + expect( + classifyTrial(0, undefined, undefined, true, { + ...overflow, + outcome: "max-rounds", + error: undefined, + }).failure + ).toBe("max-rounds"); + expect(classifyTrial(0, "ApiRateLimitError", "429", true, undefined).failure).toBe( + "rate-limit" + ); + }); + + it("reads a Workglow provider failure from its own error, not the exit code", () => { + const failed = (error: string): any => ({ ...base, outcome: "error", error }); + const base = { + rounds: 3, + retries: 2, + tools: { calls: 0, errors: 0, byName: {} }, + finalHistoryChars: 0, + }; + const exit = "NonZeroAgentExitCodeError"; + expect(classifyTrial(0, exit, "exit 1", true, failed("429 Too Many Requests")).failure).toBe( + "rate-limit" + ); + expect(classifyTrial(0, exit, "exit 1", true, failed("503 Service Unavailable")).failure).toBe( + "provider-error" + ); + expect(classifyTrial(0, exit, "exit 1", true, failed("fetch failed")).failure).toBe( + "provider-error" + ); + expect( + classifyTrial(0, exit, "exit 1", true, failed("TypeError: x is undefined")).failure + ).toBe("agent-crash"); + }); +}); + +function trial( + task: string, + agent: string, + pass: boolean, + extra: Partial = {} +): TrialRecord { + return { + job: "job", + trial: `${task}-${agent}-${Math.random()}`, + task, + arm: agent, + kwargs: {}, + agent, + agentVersion: "1", + model: "anthropic/m", + reward: pass ? 1 : 0, + status: pass ? "pass" : "fail", + failure: pass ? undefined : "wrong-answer", + exceptionType: undefined, + exceptionMessage: undefined, + inputTokens: 100, + cacheTokens: 10, + outputTokens: 5, + costUsd: 0.01, + agentSec: 30, + setupSec: 5, + workglow: undefined, + ...extra, + }; +} + +describe("labelArms", () => { + it("shows only the kwargs that differ between configs of one agent", () => { + const labelled = labelArms([ + trial("t", "workglow", true, { kwargs: { max_rounds: 100, effort: "high" } }), + trial("t", "workglow", true, { + kwargs: { max_rounds: 100, effort: "high", tool_concurrency: 8 }, + }), + trial("t", "pi", true, { kwargs: { max_turns: 100, thinking: "high" } }), + ]); + expect(labelled.map((t) => t.arm)).toEqual(["workglow", "workglow[tool_concurrency=8]", "pi"]); + }); + + it("never gives two different settings one label", () => { + const labelled = labelArms([ + trial("t", "workglow", true, { + kwargs: { append_system_prompt: "hello,tool_concurrency=8" }, + }), + trial("t", "workglow", true, { + kwargs: { append_system_prompt: "hello", tool_concurrency: 8 }, + }), + trial("t", "workglow", true, { kwargs: { append_system_prompt: "8" } }), + trial("t", "workglow", true, { kwargs: { append_system_prompt: 8 } }), + ]); + expect(new Set(labelled.map((t) => t.arm)).size).toBe(4); + expect(labelled[1]!.arm).toBe("workglow[append_system_prompt=hello,tool_concurrency=8]"); + }); + + it("keeps two agent builds apart", () => { + const labelled = labelArms([ + trial("t", "workglow", true, { kwargs: { bundle_path: "/a/workglow-agent.mjs" } }), + trial("t", "workglow", false, { kwargs: { bundle_path: "/b/workglow-agent.mjs" } }), + trial("t", "opencode", true, { kwargs: { version: "2.0.0" } }), + trial("t", "opencode", true, { kwargs: { version: "2.1.0" } }), + ]); + expect(new Set(labelled.map((t) => t.arm)).size).toBe(4); + }); + + it("keeps a trial whose model is unknown in its agent's arm", () => { + const labelled = labelArms([ + trial("t", "pi", true), + trial("u", "pi", false, { model: undefined }), + trial("t", "opencode", true, { model: "openai/x" }), + ]); + expect(labelled.map((t) => t.arm)).toEqual([ + "pi@anthropic/m", + "pi@anthropic/m", + "opencode@openai/x", + ]); + }); + + it("adds the model when the comparison spans models", () => { + const labelled = labelArms([ + trial("t", "pi", true), + trial("t", "pi", true, { model: "openai/x" }), + ]); + expect(labelled.map((t) => t.arm)).toEqual(["pi@anthropic/m", "pi@openai/x"]); + }); +}); + +describe("buildReport", () => { + const trials = [ + trial("a", "workglow", true), + trial("b", "workglow", false), + trial("c", "workglow", false, { status: "blocked", failure: undefined, reward: undefined }), + trial("a", "opencode", true), + trial("b", "opencode", true), + trial("c", "opencode", true), + ]; + + it("summarizes each arm with blocked trials kept out of the rate", () => { + const report = buildReport(trials); + const workglow = report.arms.find((arm) => arm.arm === "workglow")!; + expect(workglow.blocked).toBe(1); + expect(workglow.passRate).toBe(0.5); + expect(workglow.costUsd).toBeCloseTo(0.02); + expect(workglow.failures).toEqual({ "wrong-answer": 1 }); + expect(report.arms[0]!.arm).toBe("opencode"); + }); + + it("reports no cost per pass when a trial went unpriced", () => { + const report = buildReport([ + trial("a", "pi", true), + trial("b", "pi", true, { costUsd: undefined }), + trial("a", "opencode", true), + ]); + expect(report.arms.find((arm) => arm.arm === "pi")!.costPerPass).toBeUndefined(); + expect(report.arms.find((arm) => arm.arm === "opencode")!.costPerPass).toBeCloseTo(0.01); + }); + + it("shows no rate for an arm none of whose trials ran, and ranks it last", () => { + const blocked = { status: "blocked" as const, failure: undefined, reward: undefined }; + const report = buildReport([ + trial("a", "pi", false), + trial("a", "opencode", false, blocked), + trial("b", "opencode", false, blocked), + ]); + expect(report.arms.map((arm) => arm.arm)).toEqual(["pi", "opencode"]); + expect(report.arms[1]).toMatchObject({ + passRate: undefined, + passRateCi: undefined, + passAt1: undefined, + }); + expect(renderMarkdown(report)).toMatch( + /\| opencode \| anthropic\/m \| 2 \| 0 \| 2 \| — \| — \|/ + ); + }); + + it("marks token totals that leave out trials which reported none", () => { + const report = buildReport([ + trial("a", "pi", true), + trial("b", "pi", true, { inputTokens: undefined }), + ]); + expect(report.arms[0]!.usageMissingTrials).toBe(1); + expect( + buildReport([trial("a", "pi", true, { cacheTokens: undefined })]).arms[0]!.usageMissingTrials + ).toBe(1); + expect(renderMarkdown(report)).toContain("† some trials reported no token counts"); + }); + + it("pairs arms on the tasks both ran", () => { + const pair = buildReport(trials).pairs[0]!; + expect(pair.commonTasks).toBe(2); + expect(pair.aBetter).toEqual(["b"]); + expect(pair.bBetter).toEqual([]); + }); + + it("counts a tie short of every attempt apart from a task both arms always passed", () => { + const pair = buildReport([ + trial("a", "workglow", true), + trial("a", "workglow", true), + trial("a", "opencode", true), + trial("a", "opencode", true), + trial("b", "workglow", true), + trial("b", "workglow", false), + trial("b", "opencode", false), + trial("b", "opencode", true), + ]).pairs[0]!; + expect(pair.bothPass).toBe(1); + expect(pair.partialTie).toBe(1); + expect(pair.neither).toBe(0); + }); + + it("keeps an arm or a tool named __proto__ as data, touching no prototype", () => { + const report = buildReport([ + trial("t", "__proto__", true), + trial("t", "workglow", false, { + workglow: { + outcome: "answered", + rounds: 1, + retries: 0, + tools: { + calls: 1, + errors: 1, + byName: JSON.parse('{"__proto__":{"calls":1,"errors":1}}'), + }, + maxHistoryChars: 10, + } as any, + }), + ]); + const parsed = JSON.parse(JSON.stringify(report)); + expect(Object.keys(parsed.tasks[0].cells).sort()).toEqual(["__proto__", "workglow"]); + expect(({} as Record).calls).toBeUndefined(); + }); + + it("lists a task one arm has no result for, so the gap shows", () => { + const markdown = renderMarkdown( + buildReport([ + trial("both", "workglow", true), + trial("both", "opencode", true), + trial("only-one", "workglow", true), + ]) + ); + expect(markdown).toMatch(/\| only-one \| · \| ✓ \|/); + expect(markdown).not.toMatch(/\| both \|/); + }); + + it("renders the tables and the disagreements", () => { + const markdown = renderMarkdown(buildReport(trials)); + expect(markdown).toContain("| opencode | anthropic/m |"); + expect(markdown).toContain("**opencode** better on: b"); + expect(markdown).toMatch(/\| b \| ✓ \| ✗ wrong-answer \|/); + }); +}); + +describe("loadTrials", () => { + let dir: string | undefined; + afterEach(() => { + if (dir) rmSync(dir, { recursive: true, force: true }); + }); + + it("reads a trial that failed before it had any agent logs", () => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-jobs-")); + const trialDir = join(dir, "job-1", "hello-world__abc"); + mkdirSync(trialDir, { recursive: true }); + writeFileSync(join(dir, "job-1", "result.json"), JSON.stringify({ n_total_trials: 1 })); + writeFileSync( + join(trialDir, "result.json"), + JSON.stringify({ + trial_name: "hello-world__abc", + task_name: "harbor/hello-world", + config: { + agent: { + name: "workglow", + import_path: "workglow_agent:WorkglowAgent", + model_name: "anthropic/m", + kwargs: {}, + }, + }, + exception_info: { exception_type: "EnvironmentStartTimeoutError", exception_message: "" }, + }) + ); + const trials = loadTrials([dir]); + expect(trials).toHaveLength(1); + expect(trials[0]!.status).toBe("blocked"); + // No agent_info yet: the configured model still names its arm. + expect(trials[0]!.model).toBe("anthropic/m"); + }); + + it("reads Harbor trial results from a jobs directory", () => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-jobs-")); + const trialDir = join(dir, "job-1", "hello-world__abc"); + mkdirSync(join(trialDir, "agent"), { recursive: true }); + writeFileSync(join(dir, "job-1", "result.json"), JSON.stringify({ n_total_trials: 1 })); + writeFileSync( + join(trialDir, "result.json"), + JSON.stringify({ + trial_name: "hello-world__abc", + task_name: "harbor/hello-world", + config: { + agent: { import_path: "workglow_agent:WorkglowAgent", kwargs: { max_rounds: 100 } }, + }, + agent_info: { + name: "workglow", + version: "0.1.0", + model_info: { name: "m", provider: "anthropic" }, + }, + agent_result: { + n_input_tokens: 3062, + n_cache_tokens: 0, + n_output_tokens: 72, + cost_usd: 0.0068, + metadata: { + workglow: { + outcome: "answered", + rounds: 3, + retries: 0, + tools: { calls: 2, errors: 0, byName: { bash: { calls: 2, errors: 0 } } }, + finalHistoryChars: 755, + }, + }, + }, + verifier_result: { rewards: { reward: 1 } }, + exception_info: null, + agent_execution: { + started_at: "2026-10-06T17:59:41Z", + finished_at: "2026-10-06T17:59:43Z", + }, + }) + ); + const [record] = loadTrials([dir]); + expect(record).toMatchObject({ + job: "job-1", + task: "harbor/hello-world", + arm: "workglow", + model: "anthropic/m", + status: "pass", + inputTokens: 3062, + agentSec: 2, + }); + expect(record!.workglow?.rounds).toBe(3); + }); + + it("adds opencode's reasoning tokens to its output, as the other harnesses report them", () => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-jobs-")); + const trialDir = join(dir, "job-2", "regex-log__xyz"); + mkdirSync(join(trialDir, "agent"), { recursive: true }); + writeFileSync( + join(trialDir, "result.json"), + JSON.stringify({ + trial_name: "regex-log__xyz", + task_name: "regex-log", + agent_info: { name: "opencode", model_info: { name: "m", provider: "deepseek" } }, + agent_result: { n_input_tokens: 100, n_output_tokens: 40, cost_usd: 0.01 }, + verifier_result: { rewards: { reward: 1 } }, + agent_execution: { + started_at: "2026-10-07T00:00:00Z", + finished_at: "2026-10-07T00:00:05Z", + }, + }) + ); + writeFileSync( + join(trialDir, "agent", "opencode.txt"), + [ + JSON.stringify({ type: "step_start" }), + JSON.stringify({ type: "step_finish", part: { tokens: { output: 10, reasoning: 25 } } }), + "not json", + JSON.stringify({ type: "step_finish", part: { tokens: { output: 30, reasoning: 5 } } }), + ].join("\n") + ); + expect(loadTrials([dir])[0]!.outputTokens).toBe(70); + }); +}); diff --git a/examples/agent-eval/src/test/jobConfig.test.ts b/examples/agent-eval/src/test/jobConfig.test.ts new file mode 100644 index 0000000000..e819a6a70b --- /dev/null +++ b/examples/agent-eval/src/test/jobConfig.test.ts @@ -0,0 +1,199 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from "vitest"; +import { parseEffort, resolveAgentModel } from "../agent/models"; +import { buildHarborJob, parityKwargs, parseArm } from "../run/jobConfig"; +import type { SuiteDataset } from "../run/suites"; +import { narrowDataset } from "../run/suites"; +import { getSuite } from "../run/suites"; + +const parity = { + maxTurns: 50, + effort: "high" as const, + opencodeVariant: "high", + piModelApi: undefined, +}; + +describe("parseArm", () => { + it("parses a bare harness and one with kwargs", () => { + expect(parseArm("pi")).toEqual({ harness: "pi", kwargs: {} }); + expect(parseArm("workglow:tool_concurrency=8,append_system_prompt=be brief")).toEqual({ + harness: "workglow", + kwargs: { tool_concurrency: 8, append_system_prompt: "be brief" }, + }); + }); + + it("refuses an unknown harness", () => { + expect(() => parseArm("aider")).toThrow(/unknown harness/); + }); + + it("keeps commas inside a JSON value or a sentence", () => { + expect( + parseArm('opencode:opencode_config={"a":{"b":1,"c":2}},append_system_prompt=Use X, then Y') + ).toEqual({ + harness: "opencode", + kwargs: { opencode_config: { a: { b: 1, c: 2 } }, append_system_prompt: "Use X, then Y" }, + }); + }); + + it("keeps a quoted value whole even where a comma and key= fall inside it", () => { + expect( + parseArm('workglow:append_system_prompt="When x=a, set y=b",tool_concurrency=4') + ).toEqual({ + harness: "workglow", + kwargs: { append_system_prompt: "When x=a, set y=b", tool_concurrency: 4 }, + }); + expect(parseArm('opencode:opencode_config={"note":"p=1, q=2"}').kwargs).toEqual({ + opencode_config: { note: "p=1, q=2" }, + }); + }); + + it("refuses a pair without = and text after a quoted value", () => { + expect(() => parseArm("workglow:concise")).toThrow(/must be key=value/); + expect(() => parseArm('workglow:append_system_prompt="a" b')).toThrow(/text after its value/); + expect(() => + parseArm('workglow:append_system_prompt="When x=a, set y=b,tool_concurrency=4') + ).toThrow(/never closed/); + }); +}); + +describe("parityKwargs", () => { + it("holds every harness to the same turn cap and reasoning level", () => { + expect(parityKwargs("workglow", parity)).toEqual({ max_rounds: 50, effort: "high" }); + expect(parityKwargs("pi", parity)).toEqual({ max_turns: 50, thinking: "high" }); + expect(parityKwargs("opencode", parity)).toEqual({ + opencode_config: { agent: { build: { steps: 50 } } }, + variant: "high", + }); + }); +}); + +describe("buildHarborJob", () => { + it("puts every arm in one job, letting an arm's own kwargs win", () => { + const job = buildHarborJob({ + jobName: "j", + jobsDir: "./jobs", + model: "anthropic/claude-sonnet-5-5", + arms: [parseArm("workglow"), parseArm("workglow:max_rounds=10"), parseArm("opencode")], + datasets: getSuite("smoke").datasets, + parity, + attempts: 3, + concurrency: 2, + timeoutMultiplier: undefined, + }); + expect(job).toMatchObject({ job_name: "j", n_attempts: 3, n_concurrent_trials: 2 }); + // Only infrastructure failures are retried; an agent's own failure never is. + expect(job.retry).toEqual({ + max_retries: 2, + include_exceptions: [ + "RuntimeError", + "EnvironmentStartTimeoutError", + "AgentSetupTimeoutError", + ], + }); + const agents = job.agents as Array>; + expect(agents[0]).toMatchObject({ + import_path: "workglow_agent:WorkglowAgent", + kwargs: { max_rounds: 50 }, + }); + expect(agents[1]).toMatchObject({ kwargs: { max_rounds: 10 } }); + expect(agents[2]).toMatchObject({ + name: "opencode", + model_name: "anthropic/claude-sonnet-5-5", + }); + expect(job.datasets).toHaveLength(2); + }); +}); + +describe("resolveAgentModel", () => { + it("maps provider prefixes the way the other harnesses spell them", () => { + expect(resolveAgentModel("anthropic/claude-sonnet-5-5", { env: {} })).toMatchObject({ + provider: "ANTHROPIC", + provider_config: { model_name: "claude-sonnet-5-5" }, + }); + expect(resolveAgentModel("google/gemini-3-pro", { env: {} }).provider).toBe("GOOGLE_GEMINI"); + expect(resolveAgentModel("openrouter/qwen/qwen3-coder", { env: {} })).toMatchObject({ + provider: "OPENROUTER", + provider_config: { model_name: "qwen/qwen3-coder" }, + }); + }); + + it("takes a custom endpoint from the provider's base URL variable", () => { + expect( + resolveAgentModel("anthropic/m", { env: { ANTHROPIC_BASE_URL: "http://127.0.0.1:9" } }) + .provider_config + ).toMatchObject({ base_url: "http://127.0.0.1:9", trustedBaseUrl: true }); + }); + + it("refuses an id without a provider", () => { + expect(() => resolveAgentModel("claude-sonnet-5-5", { env: {} })).toThrow(/provider\/model/); + expect(() => resolveAgentModel("mystery/m", { env: {} })).toThrow(/unsupported provider/); + }); +}); + +describe("parseEffort", () => { + it("accepts pi's names for reasoning levels", () => { + expect(parseEffort("off")).toBe("none"); + expect(parseEffort("xhigh")).toBe("extra"); + expect(parseEffort("high")).toBe("high"); + expect(parseEffort(undefined)).toBeUndefined(); + expect(() => parseEffort("turbo")).toThrow(/unknown effort/); + }); +}); + +describe("narrowDataset", () => { + const lite = { + name: "polyglot", + version: "1", + task_names: ["polyglot_python_*", "polyglot_javascript_*"], + }; + + it("keeps only the tasks both --tasks and the suite allow", () => { + expect(narrowDataset(lite, ["polyglot_python_bank*"], undefined)!.task_names).toEqual([ + "polyglot_python_bank*", + ]); + expect(narrowDataset(lite, ["polyglot_*"], undefined)!.task_names).toEqual(lite.task_names); + expect(narrowDataset(lite, ["polyglot_go_*"], undefined)).toBeUndefined(); + }); + + it("keeps a glob that overlaps the suite's without falling inside it", () => { + expect(narrowDataset(lite, ["*_001"], undefined)!.task_names).toEqual([ + "polyglot_python_*_001", + "polyglot_python_001", + "polyglot_javascript_*_001", + "polyglot_javascript_001", + ]); + expect(narrowDataset(lite, ["polyglot_python_*"], undefined)!.task_names).toEqual([ + "polyglot_python_*", + ]); + expect(narrowDataset(lite, ["polyglot_python_bank"], undefined)!.task_names).toEqual([ + "polyglot_python_bank", + ]); + }); + + it("refuses an overlap it cannot write down rather than dropping it", () => { + expect(() => narrowDataset(lite, ["polyglot_*_bank*"], undefined)).toThrow( + /cannot be narrowed/ + ); + }); + + it("refuses a character class rather than finding no overlap", () => { + expect(() => narrowDataset(lite, ["polyglot_[pj]ython_*"], undefined)).toThrow( + /cannot be narrowed/ + ); + expect(() => narrowDataset(lite, ["polyglot_python_[ab]ank"], undefined)).toThrow( + /cannot be narrowed/ + ); + }); + + it("lowers a suite's cap but never raises it", () => { + const swe = { name: "swebench-verified", version: "1.0", n_tasks: 50 }; + expect(narrowDataset(swe, undefined, 10)!.n_tasks).toBe(10); + expect(narrowDataset(swe, undefined, 100)!.n_tasks).toBe(50); + expect(narrowDataset({ name: "x", version: "1" }, undefined, 7)!.n_tasks).toBe(7); + }); +}); diff --git a/examples/agent-eval/src/test/runCodingAgent.test.ts b/examples/agent-eval/src/test/runCodingAgent.test.ts new file mode 100644 index 0000000000..5355867600 --- /dev/null +++ b/examples/agent-eval/src/test/runCodingAgent.test.ts @@ -0,0 +1,386 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import type { Server } from "node:http"; +import { createServer } from "node:http"; +import type { AddressInfo } from "node:net"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { toAtifTrajectory } from "../agent/atif"; +import { resolveAgentModel } from "../agent/models"; +import { registerAgentProvider } from "../agent/providers"; +import type { AgentRunEvent } from "../agent/runCodingAgent"; +import { runCodingAgent } from "../agent/runCodingAgent"; +import { decideReply, startAnthropicMock } from "../mock/anthropicMock"; + +describe("decideReply", () => { + const options = { port: 0, commands: ["echo one", "echo two"] }; + const tools = [ + { + name: "bash", + input_schema: { + properties: { command: { type: "string" }, description: { type: "string" } }, + required: ["command", "description"], + }, + }, + ]; + + it("runs the next command through the harness's bash tool, filling its required fields", () => { + expect(decideReply({ tools, messages: [{ role: "user", content: "x" }] }, options)).toEqual({ + kind: "tool", + name: "bash", + input: { command: "echo one", description: "Run the next step" }, + }); + const later = decideReply( + { + tools, + messages: [ + { role: "user", content: "x" }, + { role: "assistant", content: [] }, + { role: "user", content: [] }, + ], + }, + options + ); + expect(later).toMatchObject({ kind: "tool", input: { command: "echo two" } }); + }); + + it("answers in text once the script is spent, or when no bash tool is offered", () => { + const spent = Array.from({ length: 4 }, (_, i) => ({ + role: i % 2 ? "assistant" : "user", + content: [], + })); + expect(decideReply({ tools, messages: spent }, options).kind).toBe("text"); + expect(decideReply({ messages: [] }, options).kind).toBe("text"); + }); +}); + +/** + * The whole loop against the scripted Anthropic API: the real provider, the + * real AgentTask and the real tools, with no key and no network. + */ +describe("startAnthropicMock", () => { + it("rejects when its port is already taken, rather than throwing later", async () => { + const first = await startAnthropicMock({ port: 0, commands: [] }); + try { + const { port } = first.address() as AddressInfo; + await expect(startAnthropicMock({ port, commands: [] })).rejects.toThrow(/EADDRINUSE/); + } finally { + first.close(); + } + }); +}); + +describe("runCodingAgent against the mock model", () => { + let server: Server; + let work: string; + const previous = { url: process.env.ANTHROPIC_BASE_URL, key: process.env.ANTHROPIC_API_KEY }; + + beforeAll(async () => { + server = await startAnthropicMock({ + port: 0, + commands: ["printf 'Hello, world!\\n' > hello.txt", "cat hello.txt", "exit 4"], + }); + const { port } = server.address() as AddressInfo; + process.env.ANTHROPIC_BASE_URL = `http://127.0.0.1:${port}`; + process.env.ANTHROPIC_API_KEY = "mock"; + await registerAgentProvider("ANTHROPIC"); + work = mkdtempSync(join(tmpdir(), "agent-eval-run-")); + }); + + afterAll(() => { + server.close(); + rmSync(work, { recursive: true, force: true }); + process.env.ANTHROPIC_BASE_URL = previous.url; + process.env.ANTHROPIC_API_KEY = previous.key; + if (previous.url === undefined) delete process.env.ANTHROPIC_BASE_URL; + if (previous.key === undefined) delete process.env.ANTHROPIC_API_KEY; + }); + + it("runs the tools, records every round, and prices the run", async () => { + const events: AgentRunEvent[] = []; + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: undefined, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + onEvent: (event) => events.push(event), + }); + + expect(readFileSync(join(work, "hello.txt"), "utf8")).toBe("Hello, world!\n"); + const { summary } = result; + expect(summary.outcome).toBe("answered"); + expect(summary.rounds).toBe(4); + expect(summary.tools).toEqual({ + calls: 3, + errors: 1, + byName: { bash: { calls: 3, errors: 1 } }, + }); + expect(summary.usage.promptTokens).toBeGreaterThan(0); + expect(summary.usage.outputTokens).toBe(4 * 24); + expect(summary.costUsd).toBeGreaterThan(0); + expect(events[0]?.type).toBe("start"); + expect(events.at(-1)?.type).toBe("finish"); + expect(events.filter((event) => event.type === "round")).toHaveLength(4); + + const trajectory = toAtifTrajectory("s", "0.1.0", result.messages, result.steps, summary); + expect(trajectory.steps.map((step) => step.source)).toEqual([ + "user", + "agent", + "agent", + "agent", + "agent", + ]); + expect(trajectory.steps[1]?.tool_calls?.[0]?.function_name).toBe("bash"); + expect(trajectory.steps[3]?.observation?.results[0]?.content).toMatch(/exited with code 4/); + expect(trajectory.final_metrics.total_steps).toBe(5); + }); + + it("counts the round and the retries of a call that never succeeded", async () => { + // Every request is refused as overloaded; `x-should-retry: false` keeps the SDK + // from retrying on its own, so each attempt is one the loop made. + const overloaded = createServer((_request, response) => { + response.writeHead(529, { "content-type": "application/json", "x-should-retry": "false" }); + response.end( + JSON.stringify({ type: "error", error: { type: "overloaded_error", message: "busy" } }) + ); + }); + await new Promise((resolve) => overloaded.listen(0, "127.0.0.1", resolve)); + const mockUrl = process.env.ANTHROPIC_BASE_URL; + process.env.ANTHROPIC_BASE_URL = `http://127.0.0.1:${(overloaded.address() as AddressInfo).port}`; + try { + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: 1, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + }); + expect(result.summary.outcome).toBe("error"); + expect(result.summary.rounds).toBe(1); + expect(result.summary.retries).toBe(1); + // The failed round has no step: its usage and cost are unknown, not zero. + expect(result.summary.costUsd).toBeUndefined(); + expect(result.summary.usage.unreportedRounds).toBe(1); + } finally { + process.env.ANTHROPIC_BASE_URL = mockUrl; + overloaded.close(); + } + }); + + it("does not count a retry the run was stopped before starting", async () => { + const overloaded = createServer((_request, response) => { + response.writeHead(529, { "content-type": "application/json", "x-should-retry": "false" }); + response.end( + JSON.stringify({ type: "error", error: { type: "overloaded_error", message: "busy" } }) + ); + }); + await new Promise((resolve) => overloaded.listen(0, "127.0.0.1", resolve)); + const mockUrl = process.env.ANTHROPIC_BASE_URL; + process.env.ANTHROPIC_BASE_URL = `http://127.0.0.1:${(overloaded.address() as AddressInfo).port}`; + try { + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: 5, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + // Inside the first retry's wait, which is at least a second. + stopAfterMs: 400, + }); + expect(result.summary.outcome).toBe("budget"); + expect(result.summary.rounds).toBe(1); + expect(result.summary.retries).toBe(0); + } finally { + process.env.ANTHROPIC_BASE_URL = mockUrl; + overloaded.close(); + } + }); + + it("stops outright at its hard limit, mid-call, and still returns its record", async () => { + // A provider that accepts the request and never answers: the loop's own + // budget would wait on it, since it stops only between rounds. + const silent = createServer(() => {}); + await new Promise((resolve) => silent.listen(0, "127.0.0.1", resolve)); + const mockUrl = process.env.ANTHROPIC_BASE_URL; + process.env.ANTHROPIC_BASE_URL = `http://127.0.0.1:${(silent.address() as AddressInfo).port}`; + const started = Date.now(); + try { + const events: AgentRunEvent[] = []; + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: undefined, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + stopAfterMs: 500, + onEvent: (event) => events.push(event), + }); + expect(Date.now() - started).toBeLessThan(10_000); + expect(result.summary.outcome).toBe("budget"); + expect(result.summary.rounds).toBe(1); + expect(result.summary.error).toMatch(/hard time limit/); + expect(events.at(-1)?.type).toBe("finish"); + } finally { + process.env.ANTHROPIC_BASE_URL = mockUrl; + silent.closeAllConnections(); + silent.close(); + } + }); + + it("stops at the round cap", async () => { + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 2, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: undefined, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + }); + expect(result.summary.outcome).toBe("max-rounds"); + expect(result.summary.rounds).toBe(2); + expect(result.summary.finalText).toBe(""); + }); + + it("asks for one review before finishing when given a review prompt", async () => { + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: undefined, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: "Check your work.", + }, + signal: new AbortController().signal, + }); + expect(result.summary.outcome).toBe("answered"); + expect(result.summary.rounds).toBe(5); + expect( + result.messages.filter( + (m) => m.role === "user" && JSON.stringify(m.content).includes("Check your work.") + ) + ).toHaveLength(1); + }); +}); diff --git a/examples/agent-eval/src/test/runHarbor.test.ts b/examples/agent-eval/src/test/runHarbor.test.ts new file mode 100644 index 0000000000..afa3789703 --- /dev/null +++ b/examples/agent-eval/src/test/runHarbor.test.ts @@ -0,0 +1,50 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { missingBundle } from "../run/runHarbor"; + +describe("missingBundle", () => { + let dir: string; + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-bundle-")); + }); + afterEach(() => { + rmSync(dir, { recursive: true, force: true }); + }); + + const workglow = (kwargs: Record = {}) => ({ + import_path: "workglow_agent:WorkglowAgent", + kwargs, + }); + + it("accepts an arm's own bundle when the shared one was never built", () => { + const own = join(dir, "custom.mjs"); + writeFileSync(own, ""); + expect(missingBundle([workglow({ bundle_path: own })], join(dir, "absent.mjs"))).toBe( + undefined + ); + }); + + it("names an arm's own bundle that does not exist", () => { + const shared = join(dir, "shared.mjs"); + writeFileSync(shared, ""); + expect(missingBundle([workglow({ bundle_path: join(dir, "nope.mjs") })], shared)).toMatch( + /bundle_path .*nope\.mjs does not exist/ + ); + }); + + it("asks for a build when an arm relies on the shared bundle and it is absent", () => { + expect(missingBundle([workglow()], join(dir, "absent.mjs"))).toMatch(/build-agent/); + }); + + it("ignores arms that are not the workglow adapter", () => { + expect(missingBundle([{ import_path: "pi:Pi" }], join(dir, "absent.mjs"))).toBe(undefined); + }); +}); diff --git a/examples/agent-eval/src/test/tools.test.ts b/examples/agent-eval/src/test/tools.test.ts new file mode 100644 index 0000000000..9576786596 --- /dev/null +++ b/examples/agent-eval/src/test/tools.test.ts @@ -0,0 +1,796 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { ToolCallError, ToolResultContent } from "@workglow/ai"; +import { execFileSync } from "node:child_process"; +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + rmSync, + statSync, + symlinkSync, + truncateSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { loadCredentialsFile } from "../agent/credentialsFile"; +import { PROBED_COMMANDS, probeCommands } from "../agent/environment"; +import { + commandTimeoutSec, + createBashTool, + MAX_TIMEOUT_SEC, + runCommand, + shellEnv, +} from "../agent/tools/bash"; +import type { CodingToolContext } from "../agent/tools/context"; +import { applyEdits, createEditTool, normalizeLine } from "../agent/tools/edit"; +import { codingSystemPrompt } from "../agent/systemPrompt"; +import { createReadTool, imageMimeType, MAX_READ_BYTES } from "../agent/tools/read"; +import { countLines, lineAt, truncateHead, truncateTail } from "../agent/tools/truncate"; +import { createWriteTool } from "../agent/tools/write"; + +let dir: string; +let context: CodingToolContext; +const signal = new AbortController().signal; +const call = { toolUseId: "t1", signal, deadline: undefined }; + +beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-tools-")); + context = { + cwd: dir, + spillDir: join(dir, ".spill"), + defaultCommandTimeoutSec: 10, + images: true, + }; +}); + +afterEach(() => { + rmSync(dir, { recursive: true, force: true }); +}); + +describe("truncate", () => { + // The split-based definitions the line walkers replaced, kept as the reference. + const reference = { + lines: (text: string) => (text.endsWith("\n") ? text.slice(0, -1) : text).split("\n"), + }; + + it("charges a newline only where the text has one", () => { + expect(truncateTail("abc", 100, 3)).toMatchObject({ text: "abc", truncated: false }); + expect(truncateTail("abc\n", 100, 3)).toMatchObject({ text: "bc\n", truncated: true }); + expect(truncateHead("abc", 100, 3)).toMatchObject({ text: "abc", truncated: false }); + expect(truncateHead("ab\ncd", 100, 5)).toMatchObject({ text: "ab\ncd", truncated: false }); + }); + + it("walks lines exactly as splitting them would, without building the array", () => { + const pieces = ["", "a", "bb", "\n", "\n\n", "ü", "x".repeat(30)]; + let seed = 7; + const next = () => (seed = (seed * 1103515245 + 12345) % 2 ** 31) / 2 ** 31; + for (let n = 0; n < 500; n++) { + let text = ""; + const parts = Math.floor(next() * 12); + for (let i = 0; i < parts; i++) text += pieces[Math.floor(next() * pieces.length)]; + const lines = reference.lines(text); + expect(countLines(text)).toBe(lines.length); + for (let line = 1; line <= lines.length; line++) + expect(lineAt(text, line)).toBe(lines[line - 1]); + for (const [maxLines, maxBytes] of [ + [3, 1000], + [1000, 20], + [2, 5], + ] as const) { + const head = truncateHead(text, maxLines, maxBytes); + expect(head.totalLines).toBe(lines.length); + expect(head.text.split("\n").slice(0, head.keptLines)).toEqual( + lines.slice(0, head.keptLines) + ); + const tail = truncateTail(text, maxLines, maxBytes); + expect(tail.totalLines).toBe(lines.length); + if (!tail.partialLine) { + expect(reference.lines(tail.text.length === 0 ? "" : tail.text)).toEqual( + lines.slice(lines.length - tail.keptLines) + ); + } + } + } + }); + + it("keeps the head of long text and says how much there was", () => { + const text = Array.from({ length: 10 }, (_, i) => `line ${i}`).join("\n"); + const head = truncateHead(text, 3, 1000); + expect(head.text).toBe("line 0\nline 1\nline 2"); + expect(head.truncated).toBe(true); + expect(head.totalLines).toBe(10); + }); + + it("counts a final newline as the end of the last line", () => { + const text = Array.from({ length: 5 }, (_, i) => `${i + 1}`).join("\n") + "\n"; + const tail = truncateTail(text, 2, 1000); + expect(tail.totalLines).toBe(5); + expect(tail.text).toBe("4\n5\n"); + expect(truncateHead(text, 10, 1000)).toMatchObject({ totalLines: 5, truncated: false }); + }); + + it("keeps the tail when the end is what matters", () => { + const text = Array.from({ length: 10 }, (_, i) => `line ${i}`).join("\n"); + expect(truncateTail(text, 2, 1000).text).toBe("line 8\nline 9"); + }); + + it("stops at the byte limit before the line limit", () => { + const text = ["aaaa", "bbbb", "cccc"].join("\n"); + expect(truncateHead(text, 100, 10).keptLines).toBe(2); + }); + + it("keeps the end of a last line longer than the whole budget", () => { + const tail = truncateTail(`${"x".repeat(100)}ERROR\n`, 100, 21); + expect(tail.text).toBe(`${"x".repeat(15)}ERROR\n`); + expect(tail.truncated).toBe(true); + expect(tail.partialLine).toBe(true); + }); + + it("cuts a long last line on a character boundary", () => { + // Each "é" is two bytes: an odd budget lands mid-character. + const tail = truncateTail("é".repeat(50), 100, 9); + expect(tail.text).toBe("é".repeat(4)); + expect(tail.partialLine).toBe(true); + }); +}); + +describe("edit", () => { + it("replaces a unique exact match", () => { + expect(applyEdits("a\nb\nc\n", [{ oldText: "b", newText: "B" }], "f").content).toBe( + "a\nB\nc\n" + ); + }); + + it("locates every edit against the original text", () => { + const result = applyEdits( + "one\ntwo\nthree\n", + [ + { oldText: "three", newText: "3" }, + { oldText: "one", newText: "1\none and a half" }, + ], + "f" + ); + expect(result.content).toBe("1\none and a half\ntwo\n3\n"); + }); + + it("reports an ambiguous match and applies nothing for it", () => { + const result = applyEdits("x\nx\n", [{ oldText: "x", newText: "y" }], "f"); + expect(result.content).toBe("x\nx\n"); + expect(result.failures).toEqual([expect.stringMatching(/^edits\[0\]: .*2 occurrences/)]); + }); + + it("applies the first of two overlapping edits and reports the second", () => { + const result = applyEdits( + "abcdef", + [ + { oldText: "abcd", newText: "1" }, + { oldText: "cdef", newText: "2" }, + ], + "f" + ); + expect(result.content).toBe("1ef"); + expect(result.applied).toEqual([0]); + expect(result.failures).toEqual([ + expect.stringMatching(/^edits\[1\]: overlaps edits\[0\], which was applied\. Read the file/), + ]); + }); + + it("counts an edit whose text is already in place as unchanged, not as a failure", () => { + const result = applyEdits("a", [{ oldText: "a", newText: "a" }], "f"); + expect(result.unchanged).toEqual([0]); + expect(result.failures).toEqual([]); + }); + + it("finds no blank line in an empty file", () => { + const result = applyEdits("", [{ oldText: "\n", newText: "inserted" }], "f"); + expect(result.content).toBe(""); + expect(result.failures).toEqual([expect.stringMatching(/^edits\[0\]/)]); + }); + + it("counts overlapping fuzzy matches, so an ambiguous edit is refused", () => { + // Trailing spaces defeat the exact match; each "a" line pair matches twice. + const result = applyEdits("a\na\na\n", [{ oldText: "a \na ", newText: "x" }], "f"); + expect(result.failures).toEqual([expect.stringMatching(/found 2 places matching/)]); + }); + + it("searches fuzzily in time proportional to the file, not file × oldText", () => { + const content = "a\n".repeat(200_000); + // Thousands of lines that all match, then one that never does: a search that + // restarts the whole comparison at every line does 400 million of them. + const oldText = `${"a \n".repeat(2_000)}b`; + const started = Date.now(); + const result = applyEdits(content, [{ oldText, newText: "x" }], "f"); + expect(result.failures).toEqual([expect.stringMatching(/could not find oldText/)]); + expect(Date.now() - started).toBeLessThan(3_000); + }); + + it("reports an empty oldText", () => { + expect(applyEdits("a", [{ oldText: "", newText: "b" }], "f").failures).toEqual([ + expect.stringMatching(/empty/), + ]); + }); + + it("matches whole lines that differ only in trailing whitespace and typographic quotes", () => { + const content = "const s = “hi”; \nnext();\n"; + const result = applyEdits( + content, + [{ oldText: 'const s = "hi";\n', newText: 'const s = "bye";\n' }], + "f" + ); + expect(result.content).toBe('const s = "bye";\nnext();\n'); + expect(result.fuzzy).toBe(1); + }); + + it("does not take the end of a file for a blank line the oldText asks for", () => { + const result = applyEdits("foo \n", [{ oldText: "foo\n\n", newText: "bar\n" }], "f"); + expect(result.content).toBe("foo \n"); + expect(result.failures).toHaveLength(1); + // The same edit without the blank line still matches past the trailing spaces. + expect(applyEdits("foo \n", [{ oldText: "foo\n", newText: "bar\n" }], "f").content).toBe( + "bar\n" + ); + }); + + it("normalizes dashes and non-breaking spaces", () => { + expect(normalizeLine("a—b c ")).toBe("a-b c"); + }); + + it("says what to do when nothing matches", () => { + expect(applyEdits("abc", [{ oldText: "zzz", newText: "y" }], "f").failures).toEqual([ + expect.stringMatching(/Read the file again/), + ]); + }); + + it("writes the edits that apply and names the ones that did not", async () => { + writeFileSync(join(dir, "f.txt"), "one\ntwo\ntwo\n"); + await expect( + createEditTool(context).execute!( + { + path: "f.txt", + edits: [ + { oldText: "one", newText: "1" }, + { oldText: "two", newText: "2" }, + ], + }, + call + ) + ).rejects.toThrow(/Applied 1 of 2 edits[\s\S]*edits\[1\]: .*2 occurrences/); + expect(readFileSync(join(dir, "f.txt"), "utf8")).toBe("1\ntwo\ntwo\n"); + }); + + it("applies concurrent edits through a symlink and its target, losing neither", async () => { + writeFileSync(join(dir, "real.txt"), "one\ntwo\n"); + symlinkSync(join(dir, "real.txt"), join(dir, "alias.txt")); + const edit = createEditTool(context); + await Promise.all([ + edit.execute!({ path: "real.txt", edits: [{ oldText: "one", newText: "1" }] }, call), + edit.execute!({ path: "alias.txt", edits: [{ oldText: "two", newText: "2" }] }, call), + ]); + expect(readFileSync(join(dir, "real.txt"), "utf8")).toBe("1\n2\n"); + }); + + it("refuses to edit a file that is not UTF-8, leaving its bytes alone", async () => { + const path = join(dir, "latin1.txt"); + writeFileSync(path, Buffer.from([0x61, 0xff])); + await expect( + createEditTool(context).execute!( + { path: "latin1.txt", edits: [{ oldText: "a", newText: "b" }] }, + call + ) + ).rejects.toThrow(/not UTF-8 text/); + expect([...readFileSync(path)]).toEqual([0x61, 0xff]); + }); + + it("refuses to edit a device, which has no end to read to", async () => { + await expect( + createEditTool(context).execute!( + { path: "/dev/zero", edits: [{ oldText: "a", newText: "b" }] }, + call + ) + ).rejects.toThrow(/not a regular file/); + }); + + it("refuses to edit a file too large to load", async () => { + const big = join(dir, "huge.dat"); + writeFileSync(big, ""); + truncateSync(big, MAX_READ_BYTES + 1); + await expect( + createEditTool(context).execute!( + { path: "huge.dat", edits: [{ oldText: "a", newText: "b" }] }, + call + ) + ).rejects.toThrow(/too large to edit here/); + }); + + it("tells a missing file from one it could not read", async () => { + const edits = [{ oldText: "a", newText: "b" }]; + await expect( + createEditTool(context).execute!({ path: "missing.txt", edits }, call) + ).rejects.toThrow(/File not found: .*missing\.txt\. Use write to create it\./); + // A path through a regular file fails with ENOTDIR: no file to create either. + writeFileSync(join(dir, "a-file"), "x"); + const unreadable = createEditTool(context).execute!({ path: "a-file/child", edits }, call); + await expect(unreadable).rejects.toThrow(/Could not read .*a-file\/child: /); + await expect(unreadable).rejects.not.toThrow(/Use write/); + // A directory is no file to edit at all. + mkdirSync(join(dir, "a-dir")); + await expect(createEditTool(context).execute!({ path: "a-dir", edits }, call)).rejects.toThrow( + /not a regular file/ + ); + }); + + it("says how many edits already matched when others in the call fail", async () => { + writeFileSync(join(dir, "p.txt"), "one\nsame\ntwo\ntwo\n"); + await expect( + createEditTool(context).execute!( + { + path: "p.txt", + edits: [ + { oldText: "one", newText: "1" }, + { oldText: "same", newText: "same" }, + { oldText: "two", newText: "2" }, + ], + }, + call + ) + ).rejects.toThrow(/Applied 1 of 3 edits to .*p\.txt; 1 already matched their newText/); + }); + + it("leaves the file byte-identical when every edit fails, CRLF included", async () => { + const path = join(dir, "all-failed.txt"); + writeFileSync(path, "a\r\nb\r\n"); + await expect( + createEditTool(context).execute!( + { path: "all-failed.txt", edits: [{ oldText: "not there", newText: "x" }] }, + call + ) + ).rejects.toThrow(/No edits applied/); + expect(readFileSync(path)).toEqual(Buffer.from("a\r\nb\r\n")); + }); + + it("keeps CRLF line endings and a BOM through the tool", async () => { + const path = join(dir, "crlf.txt"); + writeFileSync(path, "one\r\ntwo\r\n"); + const tool = createEditTool(context); + await tool.execute!( + { path: "crlf.txt", edits: [{ oldText: "one\ntwo", newText: "1\n2" }] }, + call + ); + expect(readFileSync(path, "utf8")).toBe("1\r\n2\r\n"); + }); + + it("writes a mixed file back in the ending of its first line break, as pi does", async () => { + writeFileSync(join(dir, "mixed.txt"), "a\nb\r\nc\n"); + await createEditTool(context).execute!( + { path: "mixed.txt", edits: [{ oldText: "c", newText: "C" }] }, + call + ); + expect(readFileSync(join(dir, "mixed.txt"), "utf8")).toBe("a\nb\nC\n"); + }); + + it("reports a missing file as a tool error", async () => { + const tool = createEditTool(context); + await expect( + tool.execute!({ path: "missing.txt", edits: [{ oldText: "a", newText: "b" }] }, call) + ).rejects.toBeInstanceOf(ToolCallError); + }); +}); + +describe("edit under concurrency", () => { + it("keeps both of two edits made to one file at the same time", async () => { + writeFileSync(join(dir, "both.txt"), "alpha\nbeta\n"); + const edit = createEditTool(context); + await Promise.all([ + edit.execute!({ path: "both.txt", edits: [{ oldText: "alpha", newText: "ALPHA" }] }, call), + edit.execute!({ path: "both.txt", edits: [{ oldText: "beta", newText: "BETA" }] }, call), + ]); + expect(readFileSync(join(dir, "both.txt"), "utf8")).toBe("ALPHA\nBETA\n"); + }); +}); + +describe("read and write", () => { + it("writes a file under new directories and reads it back", async () => { + await createWriteTool(context).execute!({ path: "a/b/c.txt", content: "hello\n" }, call); + expect(await createReadTool(context).execute!({ path: "a/b/c.txt" }, call)).toBe("hello"); + }); + + it("reports a path it cannot stat as unreadable, not as missing", async () => { + writeFileSync(join(dir, "plain.txt"), "x"); + // A file used as a directory fails with ENOTDIR: there, but not readable as asked. + await expect( + createReadTool(context).execute!({ path: "plain.txt/child" }, call) + ).rejects.toThrow(/Could not read .*plain\.txt\/child: /); + await expect(createReadTool(context).execute!({ path: "absent.txt" }, call)).rejects.toThrow( + /File not found: / + ); + }); + + it("refuses a file too large to load, before reading any of it", async () => { + const big = join(dir, "huge.log"); + writeFileSync(big, ""); + truncateSync(big, MAX_READ_BYTES + 1); + await expect(createReadTool(context).execute!({ path: "huge.log" }, call)).rejects.toThrow( + /too large to read whole\. Use bash/ + ); + }); + + it("refuses a path that is not a regular file", async () => { + await expect(createReadTool(context).execute!({ path: "/dev/zero" }, call)).rejects.toThrow( + "is not a regular file" + ); + }); + + it("refuses to write anything but a regular file, so a FIFO cannot hang it", async () => { + const fifo = join(dir, "pipe"); + execFileSync("mkfifo", [fifo]); + await expect( + createWriteTool(context).execute!({ path: "pipe", content: "x" }, call) + ).rejects.toThrow(/is not a regular file/); + await expect( + createWriteTool(context).execute!({ path: ".", content: "x" }, call) + ).rejects.toThrow(/is not a regular file/); + }); + + it("refuses an offset or limit below 1", async () => { + writeFileSync(join(dir, "two.txt"), "a\nb\n"); + const read = createReadTool(context); + await expect(read.execute!({ path: "two.txt", offset: 0 }, call)).rejects.toThrow( + "offset must be 1 or more" + ); + await expect(read.execute!({ path: "two.txt", limit: 0 }, call)).rejects.toThrow( + "limit must be 1 or more" + ); + }); + + it("counts a final newline as the end of the last line, not a line of its own", async () => { + writeFileSync(join(dir, "one.txt"), "hello\n"); + await expect( + createReadTool(context).execute!({ path: "one.txt", offset: 2 }, call) + ).rejects.toThrow("offset 2 is past the end of"); + }); + + it("cuts a line too long for the budget by bytes, not characters", async () => { + writeFileSync(join(dir, "wide.txt"), "😀".repeat(30_000)); + const out = (await createReadTool(context).execute!({ path: "wide.txt" }, call)) as string; + const head = out.split("\n\n[Line 1")[0]!; + expect(Buffer.byteLength(head, "utf8")).toBeLessThanOrEqual(50 * 1024); + expect(head).toBe("😀".repeat(12_800)); + }); + + it("pages a long file and names the next offset", async () => { + writeFileSync( + join(dir, "long.txt"), + Array.from({ length: 50 }, (_, i) => `L${i + 1}`).join("\n") + ); + const out = (await createReadTool(context).execute!( + { path: "long.txt", offset: 11, limit: 5 }, + call + )) as string; + expect(out.startsWith("L11\nL12\nL13\nL14\nL15")).toBe(true); + expect(out).toContain("Use offset=16 to continue"); + }); + + it("bounds a directory listing like a file", async () => { + mkdirSync(join(dir, "many")); + for (let i = 0; i < 2100; i++) writeFileSync(join(dir, "many", `f${i}`), ""); + const out = (await createReadTool(context).execute!({ path: "many" }, call)) as string; + expect(out).toContain( + "[Showing 2000 of 2100 entries, in directory order rather than by name. Use bash" + ); + expect(out.split("\n").filter((line) => /^f\d+$/.test(line))).toHaveLength(2000); + }); + + it("lists a directory instead of failing", async () => { + writeFileSync(join(dir, "x.txt"), ""); + const out = (await createReadTool(context).execute!({ path: "." }, call)) as string; + expect(out).toContain("x.txt"); + }); + + it("returns an image as an image the model can see", async () => { + const png = Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + Buffer.from([0, 0, 0, 0]), + ]); + writeFileSync(join(dir, "board.png"), png); + const out = await createReadTool(context).execute!({ path: "board.png" }, call); + expect(out).toBeInstanceOf(ToolResultContent); + const content = (out as ToolResultContent).content; + expect(content[0]).toMatchObject({ type: "text" }); + expect(content[1]).toEqual({ + type: "image", + mimeType: "image/png", + data: png.toString("base64"), + }); + }); + + it("refuses an image as binary when images are switched off", async () => { + writeFileSync( + join(dir, "board.png"), + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0]) + ); + await expect( + createReadTool({ ...context, images: false }).execute!({ path: "board.png" }, call) + ).rejects.toThrow(/binary/); + }); + + it("refuses a binary file", async () => { + writeFileSync(join(dir, "bin"), Buffer.from([1, 0, 2, 0])); + await expect(createReadTool(context).execute!({ path: "bin" }, call)).rejects.toThrow(/binary/); + }); +}); + +describe("bash", () => { + it("returns combined output", async () => { + const out = await createBashTool(context).execute!({ command: "echo out; echo err >&2" }, call); + expect(out).toBe("out\nerr\n"); + }); + + it("runs in the working directory", async () => { + expect(await createBashTool(context).execute!({ command: "pwd" }, call)).toBe(`${dir}\n`); + }); + + it("turns a non-zero exit into a tool error carrying the output", async () => { + await expect( + createBashTool(context).execute!({ command: "echo nope; exit 3" }, call) + ).rejects.toThrow(/nope[\s\S]*exited with code 3/); + }); + + it("runs commands without the model's credentials in their environment", async () => { + const saved = { ...process.env }; + process.env.ANTHROPIC_API_KEY = "sk-secret"; + process.env.ANTHROPIC_AUTH_TOKEN = "tok-secret"; + process.env.ANTHROPIC_BASE_URL = "http://127.0.0.1:1"; + try { + const out = String(await createBashTool(context).execute!({ command: "env" }, call)); + expect(out).not.toContain("sk-secret"); + expect(out).not.toContain("tok-secret"); + expect(out).toContain("ANTHROPIC_BASE_URL=http://127.0.0.1:1"); + expect( + shellEnv({ OPENAI_API_KEY: "k", HF_TOKEN: "t", AWS_SECRET_ACCESS_KEY: "s", PATH: "/bin" }) + ).toEqual({ PATH: "/bin" }); + } finally { + process.env = saved; + } + }); + + it.skipIf(process.platform !== "linux")( + "keeps keys loaded from a credentials file out of reach of its commands", + async () => { + const saved = { ...process.env }; + const file = join(dir, "credentials.json"); + writeFileSync(file, JSON.stringify({ WORKGLOW_TEST_API_KEY: "sk-from-file" })); + try { + expect(loadCredentialsFile(file)).toEqual(["WORKGLOW_TEST_API_KEY"]); + expect(process.env.WORKGLOW_TEST_API_KEY).toBe("sk-from-file"); + expect(existsSync(file)).toBe(false); + // The shell's own environment, and its parent's starting environment, which + // a filter on the shell's environment cannot reach. + const out = String( + await createBashTool(context).execute!( + { command: "env; tr '\\0' '\\n' < /proc/$PPID/environ" }, + call + ) + ); + expect(out).toContain("PATH="); + expect(out).not.toContain("sk-from-file"); + } finally { + process.env = saved; + } + } + ); + + it("deletes a credentials file even when it cannot be read as one", () => { + const file = join(dir, "credentials.json"); + writeFileSync(file, "not json"); + expect(() => loadCredentialsFile(file, {})).toThrow(/not valid JSON/); + expect(existsSync(file)).toBe(false); + writeFileSync(file, JSON.stringify({ KEY: 1 })); + expect(() => loadCredentialsFile(file, {})).toThrow(/non-string value for KEY/); + expect(existsSync(file)).toBe(false); + }); + + it("kills a command at its timeout", async () => { + const started = Date.now(); + await expect( + createBashTool(context).execute!({ command: "sleep 30", timeout: 1 }, call) + ).rejects.toThrow(/timed out after 1s/); + expect(Date.now() - started).toBeLessThan(10_000); + }); + + it("returns from a timeout only once a child ignoring SIGTERM is gone too", async () => { + const log = join(dir, "bg.txt"); + await expect( + createBashTool(context).execute!( + { + command: `(trap '' TERM; while true; do echo x >> ${log}; sleep 0.1; done) & sleep 30`, + timeout: 1, + }, + call + ) + ).rejects.toThrow(/timed out after 1s/); + const size = statSync(log).size; + await new Promise((resolve) => setTimeout(resolve, 600)); + expect(statSync(log).size).toBe(size); + }); + + it("refuses a timeout under one second without running the command", async () => { + for (const timeout of [0, -5, 0.5]) { + await expect( + createBashTool(context).execute!({ command: "touch ran-anyway", timeout }, call) + ).rejects.toThrow(/at least 1 second/); + } + expect(existsSync(join(context.cwd, "ran-anyway"))).toBe(false); + }); + + it("refuses a timeout longer than a timer can be set, rather than shortening it", async () => { + await expect( + createBashTool(context).execute!( + { command: "touch too-long", timeout: MAX_TIMEOUT_SEC + 1 }, + call + ) + ).rejects.toThrow(/at most \d+ seconds/); + expect(existsSync(join(context.cwd, "too-long"))).toBe(false); + }); + + it("cuts a timeout to the time left, and only when it is shorter", () => { + expect(commandTimeoutSec(120, undefined, 0)).toEqual({ sec: 120, capped: false }); + expect(commandTimeoutSec(120, 300_000, 0)).toEqual({ sec: 120, capped: false }); + expect(commandTimeoutSec(600, 90_500, 0)).toEqual({ sec: 90, capped: true }); + expect(commandTimeoutSec(600, 100, 0)).toEqual({ sec: 1, capped: true }); + }); + + it("kills a command when the task's time runs out, whatever timeout it asked for", async () => { + const started = Date.now(); + await expect( + createBashTool(context).execute!( + { command: "sleep 30", timeout: 600 }, + { ...call, deadline: Date.now() + 1_500 } + ) + ).rejects.toThrow(/all the time left/); + expect(Date.now() - started).toBeLessThan(10_000); + }); + + it("does not run a command when no time is left", async () => { + await expect( + createBashTool(context).execute!( + { command: "touch ran.txt" }, + { ...call, deadline: Date.now() - 1 } + ) + ).rejects.toThrow( + "No time is left for this task; the command was not run. Finish with what you have." + ); + expect(readdirSync(dir)).not.toContain("ran.txt"); + }); + + it("does not run a command with under a second left, which a 1 s timeout would overrun", async () => { + await expect( + createBashTool(context).execute!( + { command: "touch ran.txt" }, + { ...call, deadline: Date.now() + 500 } + ) + ).rejects.toThrow(/No time is left/); + expect(readdirSync(dir)).not.toContain("ran.txt"); + }); + + it("does not wait for a process the command left in the background", async () => { + const started = Date.now(); + const out = await createBashTool(context).execute!( + { command: "sleep 30 & echo started" }, + call + ); + expect(out).toBe("started\n"); + expect(Date.now() - started).toBeLessThan(5_000); + }); + + it("shows the tail of long output and spills the rest to a file", async () => { + const out = (await createBashTool(context).execute!({ command: "seq 1 5000" }, call)) as string; + expect(out).toMatch(/^\[Output truncated: showing the last 2000 of 5000 lines/); + expect(out.trimEnd().endsWith("5000")).toBe(true); + const spilled = readdirSync(context.spillDir); + expect(spilled).toHaveLength(1); + expect(readFileSync(join(context.spillDir, spilled[0]!), "utf8").startsWith("1\n2\n")).toBe( + true + ); + }); +}); + +describe("runCommand", () => { + it("does not time out a command that finished, while a background child drains", async () => { + const result = await runCommand("sleep 0.85; sleep 30 & echo done", dir, 1_000, signal); + expect(result.timedOut).toBe(false); + expect(result.output).toContain("done"); + }); + + it("never starts a command whose signal is already aborted", async () => { + const controller = new AbortController(); + controller.abort(new Error("stopped")); + const marker = join(dir, "ran"); + await expect(runCommand(`touch ${marker}`, dir, 10_000, controller.signal)).rejects.toThrow( + "stopped" + ); + expect(readdirSync(dir)).not.toContain("ran"); + }); + + it("keeps the end of an output past the capture cap", async () => { + const result = await runCommand( + "head -c 17000000 /dev/zero | tr '\\0' a; echo; echo THE-END", + dir, + 30_000, + signal + ); + expect(result.droppedBytes).toBeGreaterThan(0); + expect(result.output.trimEnd().endsWith("THE-END")).toBe(true); + }); +}); + +describe("imageMimeType", () => { + it("recognises the formats vision providers take, by their leading bytes", () => { + expect(imageMimeType(Buffer.from([0xff, 0xd8, 0xff, 0xe0]))).toBe("image/jpeg"); + expect(imageMimeType(Buffer.from("GIF89a"))).toBe("image/gif"); + expect(imageMimeType(Buffer.from("RIFF\0\0\0\0WEBPVP8 "))).toBe("image/webp"); + expect(imageMimeType(Buffer.from("plain text"))).toBeUndefined(); + }); +}); + +describe("probeCommands", () => { + it("probes the tools coding tasks reach for beyond the language runtimes", () => { + expect(PROBED_COMMANDS).toEqual( + expect.arrayContaining(["file", "xxd", "strings", "7zz", "7z"]) + ); + }); + + it("finds executables on PATH and lists the rest as missing", () => { + const bin = join(dir, "bin"); + mkdirSync(bin); + writeFileSync(join(bin, "python3"), "#!/bin/sh\n"); + chmodSync(join(bin, "python3"), 0o755); + writeFileSync(join(bin, "rg"), "not executable"); + expect(probeCommands(["python3", "python", "rg"], bin)).toEqual({ + found: ["python3"], + missing: ["python", "rg"], + }); + }); +}); + +describe("codingSystemPrompt", () => { + const env = { cwd: "/app", platform: "linux x64", date: "2026-10-07", commands: undefined }; + it("asks for verification once the change is complete, not after every edit", () => { + const prompt = codingSystemPrompt({ ...env, concise: true, images: true }); + expect(prompt).toContain( + "- Verify once the change is complete: run the code or its tests then, not after every edit." + ); + expect(prompt).not.toContain("Verify your work"); + }); + it("asks for concision only when switched on", () => { + expect(codingSystemPrompt({ ...env, concise: true, images: true })).toContain("Be concise"); + expect(codingSystemPrompt({ ...env, concise: false, images: true })).not.toContain( + "Be concise" + ); + }); + it("tells the model read shows images only when it does", () => { + expect(codingSystemPrompt({ ...env, concise: true, images: true })).toContain( + "shows you images" + ); + expect(codingSystemPrompt({ ...env, concise: true, images: false })).not.toContain("images"); + }); + it("names the commands it found and the ones that are missing", () => { + const prompt = codingSystemPrompt({ + ...env, + concise: true, + images: true, + commands: { found: ["python3", "git"], missing: ["python", "rg"] }, + }); + expect(prompt).toContain("commands on PATH: python3, git"); + expect(prompt).toContain("not installed: python, rg"); + }); +}); diff --git a/examples/agent-eval/tsconfig.json b/examples/agent-eval/tsconfig.json new file mode 100644 index 0000000000..fa30b0afe5 --- /dev/null +++ b/examples/agent-eval/tsconfig.json @@ -0,0 +1,13 @@ +{ + // No `references`: with `use-source` stubs, project-reference redirect turns + // `dist/*.d.ts` → `src` and makes stub `export * from "../src/browser"` a + // self-import, so every `@workglow/*` export appears missing. + "extends": "../../tsconfig.json", + "include": ["src/**/*"], + "exclude": ["dist"], + "compilerOptions": { + "composite": true, + "outDir": "dist", + "rootDir": "./src" + } +} diff --git a/examples/agent-eval/vitest.config.ts b/examples/agent-eval/vitest.config.ts new file mode 100644 index 0000000000..3a2df05af0 --- /dev/null +++ b/examples/agent-eval/vitest.config.ts @@ -0,0 +1,13 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + include: ["src/test/**/*.test.ts"], + }, +}); From 259dff5cb386545d5655e308855b4617d8e6c38f Mon Sep 17 00:00:00 2001 From: Steven Roussey Date: Mon, 12 Oct 2026 05:34:45 +0000 Subject: [PATCH 7/7] ci: CodeQL advanced setup, excluding examples/agent-eval The harness's tools read, write and run what a model hands them inside a throwaway container by design, which scanning reports as injection; default setup cannot exclude a path. --- .github/codeql/codeql-config.yml | 8 +++++++ .github/workflows/codeql.yml | 38 ++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 .github/codeql/codeql-config.yml create mode 100644 .github/workflows/codeql.yml diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml new file mode 100644 index 0000000000..b54d8d0907 --- /dev/null +++ b/.github/codeql/codeql-config.yml @@ -0,0 +1,8 @@ +# The benchmark harness gives a model a shell and the file system of a +# throwaway task container: its tools read, write and run whatever paths and +# commands they are handed, by design. Scanning it reports that design as +# injection on every tool, which buries findings in the code this repository +# ships. +name: workglow CodeQL config +paths-ignore: + - examples/agent-eval diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000000..b9ae3cf107 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,38 @@ +# Advanced setup rather than GitHub's default setup, which cannot exclude a +# path: see `.github/codeql/codeql-config.yml` for what is left out and why. +# Default setup must be off in the repository's settings for this to run. +name: CodeQL +permissions: + contents: read + +on: + push: + branches: [main] + pull_request: + branches: [main] + schedule: + - cron: "27 6 * * 1" + workflow_dispatch: + +jobs: + analyze: + name: Analyze (${{ matrix.language }}) + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + strategy: + fail-fast: false + matrix: + language: [actions, javascript-typescript] + steps: + - uses: actions/checkout@v6 + - uses: github/codeql-action/init@v4 + with: + languages: ${{ matrix.language }} + build-mode: none + config-file: ./.github/codeql/codeql-config.yml + - uses: github/codeql-action/analyze@v4 + with: + category: "/language:${{ matrix.language }}"