diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 7c83846fee..3b85dab4ea 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -243,15 +243,52 @@ The rest are controls a batch host needs: `toolConcurrency` (opt-in; a round hol put to a person still runs one at a time, and results return in the order asked), `maxRoundRetries` (default 2, for a `RetryableJobError`, waiting as the provider's retry-after says within 1–60 s), `roundTimeoutMs` (a provider can accept a request and never answer; -past it the round is abandoned as a retryable failure, so the retries cover it; a round's text is -forwarded only once the attempt settles, since a failed attempt's partial cannot be taken back -from the accumulated text port), and budgets — `maxInputTokens`, `maxCostUsd` (refused -without a price card, since a budget it cannot measure never stops anything, and failing closed -for a round that reports no usage: it cannot be priced, so the turn ends `"budget"` after it), +past it the round is abandoned as a retryable failure, so the retries cover it; a round's text and +reasoning are forwarded only once the attempt settles, since a failed attempt's partial cannot be +taken back from the accumulated port; reasoning goes out on the turn's own `reasoning` port, a +blank line between rounds, and is absent from the output when no round returned any), and +budgets — `maxInputTokens`, `maxCostUsd` (refused without a price card, +since a budget it cannot measure never stops anything, and failing closed for a round that reports no usage: it cannot be priced, so the turn ends `"budget"` after it), `maxDurationMs` — each ending the turn `"budget"`, never between a `tool_use` and its result. Every round leaves an `AgentStep` on `steps` (timings, attempts, each tool's outcome and size, usage, cost) and rides on the `snapshot` beside `messages`; `costUsd` totals them when every round could be priced. +A tool with an `execute` function is handed the turn's `deadline` on its `ToolExecuteContext` +(undefined without `maxDurationMs`; a task-backed tool never receives it): the loop stops only +between rounds, so a tool that can bound its own work caps it there. A retry wait that would pass +the deadline ends the turn `"budget"` instead of sleeping. `announceTimeLeft` appends the time +left to the text of each round's last tool result, and the notice stays in the returned +`messages` — the model cannot budget time it is never told about, and a long command started in +the last minutes is how a turn runs past its budget. +`reviewPrompt` is sent once per turn, the first time the model replies without a tool call, so it +checks its answer against what it was asked before the turn ends; the model may go back to its +tools. A chat host that passes it on every turn gets a review each turn. It is skipped with +`outputSchema` (that is `checkSubmission`'s job) and when no round, budget or time is left. The +turn's `text` then carries both replies. + +**A provider can be handed its own turn back.** A run-fn that can replay its own output emits it +once per round on `ToolCallingTask`'s `nativeTurn` port (`{ provider, payload }`), and `AgentTask` +keeps it on the assistant message's `reasoning` block (`provider`, `payload`). That provider's +converter then replays the payload verbatim instead of rebuilding the turn — the OpenAI Responses +items with their encrypted reasoning, Anthropic's content blocks with their signed thinking — and +only while the turn's call ids still match it (by count and order: each call's id, name and arguments, and for Anthropic the turn's text too), so +a host that changed or dropped a call id gets the rebuilt turn. The `provider` is `openai:` +or `anthropic:` (the provider name and its API model name), because encrypted reasoning and +signed thinking are readable only by the model that wrote them, so a turn is replayed only to that +model and rebuilt after a model switch. A reasoning model handed nothing back starts its thinking +over every round, spending output tokens re-deriving it. The payload stays on the message, so a +turn that is not truly replayable must not be emitted at all: OpenAI emits one only when encrypted +reasoning was requested (`include: ["reasoning.encrypted_content"]`, sent only for a reasoning +model, that is when reasoning effort is not `none`) and every reasoning item came back with it, +since a reasoning item replayed without it is looked up by id and fails where nothing is stored. +Anthropic emits none when it cannot capture the turn verbatim — a block or delta type it does not +reproduce, a thinking block without its signature, or an empty text block dropped ahead of a +thinking block. Anthropic binds a thinking block's signature to the conversation before it, so +the replay must be append-only; a request it rejects as edited is retried once with every thinking +block stripped. History trimming measures a message without its payload. +`provider_config.replay_reasoning: false` turns the OpenAI replay off; Anthropic has no such +switch. DeepSeek's `replay_reasoning` is a different mechanism: it controls replaying the +`reasoning_content` text, not a native turn. **The loop and the model call can live in different places.** A host whose tools are closures (they draw on a screen, ask a person, read state only that process holds) but whose model is @@ -422,6 +459,22 @@ Rules for writing a run-fn: discriminate on a field one caller always sends and the other never does (e.g. `Array.isArray(input.messages) && input.messages.length > 0`). +**Reasoning travels between rounds, and each provider carries it its own way.** DeepSeek and +the llama.cpp server replay `reasoning_content`; Ollama replays `thinking`; HF Transformers +hands `reasoning_content` to the chat template; node-llama-cpp replays thought segments. +OpenAI and Anthropic replay a native turn, and OpenRouter carries `reasoning_details` as one +keyed `openrouter:`; all three replay it only while the turn's calls still match (each call's +id, name and arguments, in order), so a host that changed or dropped one gets the rebuilt turn, and OpenRouter emits none for Anthropic +thinking cut off before its signature, or for a turn one of whose calls it filtered out. **A tool-calling run-fn converts `messages` with a +converter that keeps tool calls and prefers `messages` over `prompt`:** `AgentTask` sends both +every round, and a converter that reads `prompt` or flattens to text shows the model its first +turn again and never a tool result. A history holding tool calls or tool results must never go +through `toTextFlatMessages`; HF Transformers still uses it, but only on the path where the +history has no tool messages and no reasoning yet, where it loses nothing. Tool-result images go +in a user message after the tool message (OpenAI, xAI, OpenRouter, HF Inference, Ollama); Gemini 3+ takes them in +`functionResponse.parts`, and older models, which reject that with HTTP 400, get a user turn of +their own — beside the response they leak their thinking into the answer. + ### `examples/cli` — the CLI and its web console `workglow web` (`src/web/`, client in `src/web/client/`) serves the same commands the @@ -539,6 +592,32 @@ The client bundles via `bun run build-web` into `dist/web` and is part of `build deliberately non-render-blocking; a blocking font link blanks the console on any machine that cannot reach the CDN. +### `examples/agent-eval` — the harness comparison + +This example runs `AgentTask`, opencode and pi on public benchmarks (Terminal-Bench 2.0, +Aider Polyglot, SWE-bench Verified) through **Harbor**, with the same model for all three, and +compares them task by task. Harbor is a Python tool and needs Docker; it ships the opencode and +pi adapters. + +What the Workglow arm is: + +- **The agent.** `src/bin/workglow-agent.ts` is `AgentTask` plus four coding tools (`read`, + `bash`, `edit`, `write`). The tools match pi's defaults and limits on purpose, so the + Workglow–pi gap measures the loop and not the tools. +- **How it gets into the container.** It is bundled into one Node file. `harbor/workglow_agent.py` + uploads that bundle into each task container, so a run measures the checkout it was built + from: rebuild with `bun run build-agent` after touching `AgentTask`. + +Two rules hold for every run: + +- **Parity.** `run` sets the turn cap and the reasoning level explicitly for every arm, because + the three harnesses' defaults differ enough to decide the result. +- **Blocked trials.** A trial whose container or install failed is _blocked_, and is excluded + from every rate rather than counted as a failure. + +`mock-model` is a scripted Anthropic API. With it, the whole pipeline runs offline with no key +and no cost. + ### `@workglow/util` `EventEmitter`, `ServiceRegistry` (DI), `DirectedAcyclicGraph`, `DataPortSchema`/`JsonSchema`, diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml new file mode 100644 index 0000000000..b54d8d0907 --- /dev/null +++ b/.github/codeql/codeql-config.yml @@ -0,0 +1,8 @@ +# The benchmark harness gives a model a shell and the file system of a +# throwaway task container: its tools read, write and run whatever paths and +# commands they are handed, by design. Scanning it reports that design as +# injection on every tool, which buries findings in the code this repository +# ships. +name: workglow CodeQL config +paths-ignore: + - examples/agent-eval diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000000..b9ae3cf107 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,38 @@ +# Advanced setup rather than GitHub's default setup, which cannot exclude a +# path: see `.github/codeql/codeql-config.yml` for what is left out and why. +# Default setup must be off in the repository's settings for this to run. +name: CodeQL +permissions: + contents: read + +on: + push: + branches: [main] + pull_request: + branches: [main] + schedule: + - cron: "27 6 * * 1" + workflow_dispatch: + +jobs: + analyze: + name: Analyze (${{ matrix.language }}) + runs-on: ubuntu-latest + permissions: + actions: read + contents: read + security-events: write + strategy: + fail-fast: false + matrix: + language: [actions, javascript-typescript] + steps: + - uses: actions/checkout@v6 + - uses: github/codeql-action/init@v4 + with: + languages: ${{ matrix.language }} + build-mode: none + config-file: ./.github/codeql/codeql-config.yml + - uses: github/codeql-action/analyze@v4 + with: + category: "/language:${{ matrix.language }}" diff --git a/bun.lock b/bun.lock index c87f621e16..100e4224fa 100644 --- a/bun.lock +++ b/bun.lock @@ -34,6 +34,25 @@ "@sqliteai/sqlite-vector-win32-x86_64": "^1.1.2", }, }, + "examples/agent-eval": { + "name": "@workglow/agent-eval", + "version": "0.6.12", + "bin": { + "workglow-agent-eval": "./dist/bin/workglow-agent-eval.js", + }, + "dependencies": { + "@workglow/ai": "workspace:*", + "@workglow/anthropic": "workspace:*", + "@workglow/deepseek": "workspace:*", + "@workglow/google-gemini": "workspace:*", + "@workglow/openai": "workspace:*", + "@workglow/openrouter": "workspace:*", + "@workglow/task-graph": "workspace:*", + "@workglow/util": "workspace:*", + "@workglow/xai": "workspace:*", + "commander": "catalog:", + }, + }, "examples/cli": { "name": "@workglow/cli", "version": "0.6.12", @@ -1763,6 +1782,8 @@ "@workglow/a2ui": ["@workglow/a2ui@workspace:packages/a2ui"], + "@workglow/agent-eval": ["@workglow/agent-eval@workspace:examples/agent-eval"], + "@workglow/ai": ["@workglow/ai@workspace:packages/ai"], "@workglow/anthropic": ["@workglow/anthropic@workspace:providers/anthropic"], diff --git a/examples/agent-eval/.gitignore b/examples/agent-eval/.gitignore new file mode 100644 index 0000000000..cdb6529f54 --- /dev/null +++ b/examples/agent-eval/.gitignore @@ -0,0 +1,2 @@ +jobs/ +__pycache__/ diff --git a/examples/agent-eval/README.md b/examples/agent-eval/README.md new file mode 100644 index 0000000000..13264da8a4 --- /dev/null +++ b/examples/agent-eval/README.md @@ -0,0 +1,206 @@ +# Workglow Agent Harness Eval + +Runs Workglow's `AgentTask` loop, [opencode](https://github.com/sst/opencode) and +[pi](https://github.com/badlogic/pi-mono) on the same public agentic benchmarks, with the +same model, and compares them task by task. + +The point is to steer work on `AgentTask`. A pass rate alone doesn't do that. What does is +seeing which tasks another harness solved and ours didn't, why ours failed, and whether a +change to the loop moves those tasks. + +## How it works + +[Harbor](https://harborframework.com) (the Terminal-Bench harness) runs everything: + +- It builds each task's container and runs the agent inside it. +- It scores the result with the task's own tests. +- It ships maintained adapters for opencode and pi. +- Its registry carries Terminal-Bench 2.0, Aider Polyglot and SWE-bench Verified, among about + seventy others. + +This package adds three pieces: + +| Piece | What it is | +| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `src/bin/workglow-agent.ts` | A headless coding agent: `AgentTask` plus four tools (`read`, `bash`, `edit`, `write`). It runs one instruction and writes a summary, an event log and an ATIF trajectory. It is bundled into one Node file (`dist/workglow-agent.mjs`) so it runs in any task container. | +| `harbor/workglow_agent.py` | The Harbor adapter. It installs Node, uploads the bundle and runs it. Because it uploads the bundle you just built, the eval measures your `libs` checkout. | +| `src/bin/workglow-agent-eval.ts` | The driver. `run` puts every arm in one Harbor job, with parity settings, then writes the comparison. `compare` reads any Harbor job directories. `mock-model` serves a scripted model for offline checks. | + +### What is held equal + +Each harness's defaults differ in ways that would decide the result, so `run` sets them +explicitly: + +| | workglow | opencode | pi | +| -------------------------------------- | -------------------------------------------------- | ------------------------------------------ | ------------------------------------ | +| Turn cap (`--max-turns`, default 100) | `max_rounds` (AgentTask's own default is 8) | `agent.build.steps` (unlimited by default) | `max_turns` (unlimited by default) | +| Reasoning (`--effort`, default `high`) | `effort` | `variant` (`--opencode-variant`) | `thinking` (pi defaults to `medium`) | +| Model | `-m provider/model`, the same string for all three | | | +| Time limit | the task's own agent timeout, enforced by Harbor | | | + +Reasoning defaults to `high` rather than `off` because "off" does not mean the same thing +everywhere. On Claude 5.x, a request without `thinking` runs adaptive thinking. pi's `off` +and opencode's default omit the field, while Workglow sends Sonnet 5.5's real off switch. + +**The tools are not held equal, deliberately.** The Workglow arm gets pi's default four, +with pi's limits: 2000 lines or 50KB per output, the tail for commands, and the head plus an +offset hint for reads. So the gap between the Workglow and pi columns measures the loop +around the tools. opencode's richer toolset (grep, glob, todo, subagents, a fuzzier edit) is +part of what its column measures. + +## Getting started + +```bash +uv tool install harbor # Harbor 0.24+; needs Docker +bun install && bun run build-example # in examples/agent-eval: bundles dist/workglow-agent.mjs + +export ANTHROPIC_API_KEY=... # whichever provider the model needs + +# Does every arm install and run? hello-world + the 10-task Terminal-Bench sample +./dist/bin/workglow-agent-eval.js run -m anthropic/claude-sonnet-5-5 -s smoke + +# The decision set: Terminal-Bench 2.0 + Polyglot (Python/JS) + SWE-bench Verified 50 +./dist/bin/workglow-agent-eval.js run -m anthropic/claude-sonnet-5-5 -s core -n 8 -k 2 +``` + +`run` writes `//comparison.md` and `comparison.json` and prints the Markdown. +To re-compare any Harbor jobs later, including ones run by hand or merged from several +machines: + +```bash +./dist/bin/workglow-agent-eval.js compare jobs/core-2026-10-06T18-00-00 [more job dirs…] +./dist/bin/workglow-agent-eval.js compare jobs/ --format json --out comparison.json +``` + +`harbor view jobs/` shows each trial's trajectory. The Workglow arm writes ATIF, so it is +drawn the same way as the other two. + +### Suites + +`./dist/bin/workglow-agent-eval.js suites` lists them: + +| Suite | Contents | Exercises | +| ---------------------------- | ------------------------------------------------------------ | ---------------------------------------------------------------------- | +| `smoke` | `hello-world` + `terminal-bench-sample@2.0` (10) | install and plumbing | +| `terminal-bench` | Terminal-Bench 2.0 (89) | long-horizon terminal work; context growth, command timeouts, recovery | +| `polyglot` / `polyglot-lite` | Aider Polyglot, all six languages (225) / Python and JS (83) | edit-run-fix cycles; edit reliability, reading test output | +| `swebench-50` | SWE-bench Verified, first 50 | navigating a large repository, minimal patches | +| `core` | the last three together | the set to decide on | + +For any other registry dataset, use `-d name@version` (repeatable). To narrow a run, use +`--tasks 'glob,glob'` and `-l N`. Both only narrow a suite: `--tasks` keeps the tasks that match +both it and the suite's own list (`--tasks '*_001'` on `polyglot-lite` runs each language's +`_001`), and fails if none do or the overlap is not one it can express, and `-l` lowers the suite's cap but never +raises it. opencode's `--opencode-variant` defaults to the `--effort` level. + +To run task folders on disk instead — a downloaded dataset you have patched, or tasks of your +own — use `--tasks-dir ` (repeatable). A host behind a TLS-intercepting proxy needs this: +the registry tasks fetch over HTTPS while their images build, before any compose overlay can +put the proxy's CA in the container, so the CA has to go into each task's Dockerfile. + +### Ablations: testing a change to the loop + +An arm is a harness plus agent kwargs. Extra arms run in the same job, on the same tasks: + +```bash +./dist/bin/workglow-agent-eval.js run -m anthropic/claude-sonnet-5-5 -s polyglot-lite \ + -a workglow --arm workglow:tool_concurrency=8 --arm workglow:max_history_chars=400000 +``` + +The report labels arms by the kwargs that differ (`workglow[tool_concurrency=8]`), and pairs +them task by task like any other arms. For a change to `AgentTask` itself, edit `libs`, run +`bun run build-agent`, and re-run the same suite. The job config is saved beside the results, +so the before and after runs are the same experiment. + +Values are JSON where they parse as JSON. A value that holds a comma followed by `key=` goes in +double quotes: `--arm 'workglow:append_system_prompt="When x=a, set y=b"'`. + +The Workglow kwargs (`--arm workglow:k=v`) are: `effort`, `max_rounds`, +`max_tool_result_chars`, `max_history_chars`, `tool_concurrency`, `max_tokens`, +`temperature`, `round_timeout_sec`, `max_round_retries`, `command_timeout_sec`, +`append_system_prompt`, `concise`, `images`, `replay_reasoning`, `time_budget`, `time_left`, +`review`, `max_duration_sec`. opencode takes `opencode_config` and `variant`. pi takes +`thinking` and `max_turns`. + +- `time_budget` (default on): the adapter reads the trial's agent time limit and passes it, + less a reserve of 10% (30 s to 120 s), as `--max-duration-sec`. The loop then ends cleanly with + `stopReason: "budget"` and writes its summary before Harbor would cancel it. Harbor gives the + opencode and pi arms no such signal; this is a harness capability, not a benchmark setting. +- `time_left` (default on): the model sees `[Time left: 7m 12s]` after each round's tool results. +- `review` (default on): the first time the model tries to finish, it is asked once to check its + work against the instruction's exact requirements. +- `max_duration_sec`: an explicit budget, overriding `time_budget`. + +The budget stops the loop only between rounds, so the adapter also passes `--stop-after-sec` +a little under Harbor's limit: the agent then stops outright, a model call in flight included, +records the run as `budget`, and writes its summary and trajectory before Harbor cancels it. + +`bash` now defaults to a 120 s timeout (was 600 s), and never runs past the deadline. +Its commands run without the agent's credential variables (`*_API_KEY`, `*_TOKEN`, `*_SECRET` +and the like), so the model cannot print its own key into the transcript or the logs. + +Filtering the shell's environment is not enough on its own: a process's starting environment +stays readable at `/proc//environ`, and every command the model runs descends from the +agent. So under Harbor the keys never enter that environment. The adapter writes them to a +`0600` file in an agent-only directory, passes only its path (`--credentials-file`), and the +agent loads them after it starts and deletes the file before any tool can run. + +## Reading the report + +- **Pass rate (95% CI)** uses a Wilson interval. Arms run on tens of tasks, so expect wide + intervals. Use the head-to-head table to tell two arms apart, not the overlap of their + intervals. +- **Head to head** pairs the arms on the tasks both ran. A task solved by both or by neither + says nothing about which arm is better. The p-value is an exact sign test (McNemar) over the + tasks where the arms differ, and those tasks are listed by name. Read their transcripts + first. +- **Blocked** trials are excluded from every rate. These are trials where the container or + the agent's install failed before the agent saw the task. Counting them as failures would + charge a harness for a flaky registry. +- **Why runs failed** sorts failures into modes: `wrong-answer`, `agent-timeout`, + `context-overflow`, `max-rounds`, `budget`, `rate-limit`, `provider-error`, `agent-crash`, + `verifier-error`. The Workglow arm also reports outcomes, rounds, retries and tool error + counts from the loop itself. +- **Cost.** Each harness prices its own runs. For the Workglow arm, that is `AgentTask`'s + per-round step records. When a custom endpoint is configured, pi runs it as a provider with + no price card, so its cost is blank. + +## Offline check (no key, no bill) + +`mock-model` serves a scripted Anthropic Messages API. Each reply calls the harness's `bash` +tool with the next scripted command, and replies in text once the script is used up. All +three harnesses can target an Anthropic base URL, so one mock serves the whole pipeline: + +```bash +./dist/bin/workglow-agent-eval.js mock-model -p 18089 \ + -c "echo 'Hello, world!' > hello.txt" --request-log requests.jsonl & +ANTHROPIC_BASE_URL=http://127.0.0.1:18089 ANTHROPIC_API_KEY=mock \ + ./dist/bin/workglow-agent-eval.js run -m anthropic/claude-sonnet-5-5 -d hello-world@1.0 \ + --harbor-config hostnet.yaml +``` + +The task container has to reach the mock. On Linux, point a Harbor config at a compose +overlay that sets `network_mode: host` for the `main` service: + +```yaml +environment: + extra_docker_compose: [./hostnet.compose.yaml] +``` + +`requests.jsonl` captures what each harness actually sends the model: the system prompt, the +tool schemas, the thinking settings and the cache markers. That is the quickest way to see +how the three differ before spending anything. + +The same mock drives `src/test/runCodingAgent.test.ts`, which runs the real provider, the +real `AgentTask` and the real tools in process. + +## Running the Workglow agent by itself + +```bash +node dist/workglow-agent.mjs -m anthropic/claude-sonnet-5-5 --cwd ./repo \ + --logs-dir ./logs --effort high "Fix the failing test in src/parse.ts" +``` + +Providers: `anthropic`, `openai`, `google`/`gemini`, `deepseek`, `xai`, `openrouter`. Keys +are read from the usual environment variables, and a custom endpoint from `_BASE_URL`; +`--credentials-file ` loads them from a JSON object instead and deletes the file. diff --git a/examples/agent-eval/harbor/workglow_agent.py b/examples/agent-eval/harbor/workglow_agent.py new file mode 100644 index 0000000000..452756b914 --- /dev/null +++ b/examples/agent-eval/harbor/workglow_agent.py @@ -0,0 +1,384 @@ +# @license +# Copyright 2026 Steven Roussey +# SPDX-License-Identifier: Apache-2.0 +"""Harbor adapter for the headless Workglow coding agent (AgentTask). + +Run it beside Harbor's built-in ``opencode`` and ``pi`` agents: + + harbor run -d terminal-bench@2.0 -m anthropic/claude-sonnet-5-5 \\ + -a workglow_agent:WorkglowAgent + +with this directory on ``PYTHONPATH``. It installs Node in the task container, +uploads the single-file agent bundle (``dist/workglow-agent.mjs``, built by +``bun run build-agent``), and runs it on the task instruction. The bundle is +uploaded rather than installed from a registry so a run always measures the +``libs`` checkout it was built from: change AgentTask, rebuild, re-run. +""" + +import json +import os +import shlex +import tempfile +import time +from pathlib import Path, PurePosixPath +from typing import Annotated, Any, override + +from pydantic import Field + +from harbor.agents.capabilities import AgentCapabilities +from harbor.agents.installed.base import BaseInstalledAgent, with_prompt_template +from harbor.agents.installed.node_install import nvm_node_install_snippet +from harbor.agents.model_connection import ModelConnectionSpec +from harbor.agents.options import Cli, InstalledAgentOptions +from harbor.environments.base import BaseEnvironment +from harbor.models.agent.context import AgentContext +from harbor.models.task.task import Task +from harbor.models.trial.config import TrialConfig + +_REMOTE_DIR = PurePosixPath("/installed-agent") +_REMOTE_BUNDLE = _REMOTE_DIR / "workglow-agent.mjs" +_REMOTE_INSTRUCTION = _REMOTE_DIR / "instruction.md" +# Owned by the agent user and closed to everyone else, so the agent can delete +# the credentials file it reads from here. +_REMOTE_SECRETS_DIR = _REMOTE_DIR / "secrets" +_REMOTE_CREDENTIALS = _REMOTE_SECRETS_DIR / "credentials.json" +_DEFAULT_BUNDLE = Path(__file__).resolve().parent.parent / "dist" / "workglow-agent.mjs" +_NODE_PREFIX = "[ -f ~/.nvm/nvm.sh ] && . ~/.nvm/nvm.sh; " +# The longest timer Node can set, in whole seconds; the agent refuses anything longer. +_MAX_TIMER_SEC = 2_147_483 + + +class WorkglowOptions(InstalledAgentOptions): + """Agent kwargs (``--ak name=value``), passed through as CLI flags. + + Every knob that changes AgentTask's behaviour is here so a job can run an + ablation — the same model and tasks with one setting changed — without a + rebuild. + """ + + effort: Annotated[str | None, Cli("--effort")] = Field( + default=None, + description="Reasoning effort: none|low|medium|high|extra|ultra (or off|minimal|xhigh|max).", + ) + max_rounds: Annotated[int | None, Cli("--max-rounds")] = Field( + default=None, ge=1, description="Model calls before the run stops (agent default 100)." + ) + max_tool_result_chars: Annotated[int | None, Cli("--max-tool-result-chars")] = Field( + default=None, ge=1, description="Characters of one tool result the model is shown." + ) + max_history_chars: Annotated[int | None, Cli("--max-history-chars")] = Field( + default=None, ge=1, description="History budget per round (AgentTask default if unset)." + ) + tool_concurrency: Annotated[int | None, Cli("--tool-concurrency")] = Field( + default=None, ge=1, description="Tool calls of one round run at once." + ) + max_tokens: Annotated[int | None, Cli("--max-tokens")] = Field( + default=None, ge=1, description="Output tokens per model call." + ) + temperature: Annotated[float | None, Cli("--temperature")] = Field( + default=None, description="Sampling temperature." + ) + round_timeout_sec: Annotated[int | None, Cli("--round-timeout-sec")] = Field( + default=None, + ge=1, + le=_MAX_TIMER_SEC, + description="Abandon and retry a model call after this long.", + ) + max_round_retries: Annotated[int | None, Cli("--max-round-retries")] = Field( + default=None, ge=0, description="Retries of a failed model call." + ) + command_timeout_sec: Annotated[int | None, Cli("--command-timeout-sec")] = Field( + default=None, ge=1, le=_MAX_TIMER_SEC, description="Default bash timeout." + ) + append_system_prompt: Annotated[str | None, Cli("--append-system-prompt")] = Field( + default=None, description="Text appended to the system prompt." + ) + concise: Annotated[str | None, Cli("--concise")] = Field( + default=None, description="on|off: ask for terse replies between tool calls (default on)." + ) + images: Annotated[str | None, Cli("--images")] = Field( + default=None, description="on|off: let read show images to the model (default on)." + ) + replay_reasoning: Annotated[str | None, Cli("--replay-reasoning")] = Field( + default=None, + description="on|off: send the model's reasoning back on later rounds (default on).", + ) + max_duration_sec: Annotated[int | None, Cli("--max-duration-sec")] = Field( + default=None, + ge=1, + description="Wall-clock budget; set from the trial's time limit unless given.", + ) + time_budget: str | None = Field( + default=None, + pattern="^(on|off)$", + description="on|off: give the agent the trial's time limit, less a reserve (default on).", + ) + time_left: Annotated[str | None, Cli("--time-left")] = Field( + default=None, + pattern="^(on|off)$", + description="on|off: show the model its remaining time (default on)." + ) + review: Annotated[str | None, Cli("--review")] = Field( + default=None, + pattern="^(on|off)$", + description="on|off: one check against the instructions before finishing (default on).", + ) + bundle_path: str | None = Field( + default=None, + description="Local path of workglow-agent.mjs (default: ../dist, or $WORKGLOW_AGENT_BUNDLE).", + ) + + +class WorkglowAgent(BaseInstalledAgent): + capabilities = AgentCapabilities(atif=True) + MODEL_CONNECTION = ModelConnectionSpec(passthrough=True) + + options_model = WorkglowOptions + options: WorkglowOptions + + _OUTPUT_FILENAME = "workglow.txt" + _SUMMARY_FILENAME = "workglow-summary.json" + + @staticmethod + @override + def name() -> str: + return "workglow" + + @override + def get_version_command(self) -> str | None: + return f"{_NODE_PREFIX}node {shlex.quote(_REMOTE_BUNDLE.as_posix())} --version" + + def _agent_timeout_sec(self) -> float | None: + """The time Harbor gives run(), resolved as the trial resolves it. + + Harbor hands an agent no deadline: it cancels run() when the time is + up. The trial's config and task sit next to the logs directory, so the + agent can learn its limit and stop cleanly, with its summary written, + before that. + """ + try: + config = TrialConfig.model_validate_json( + (self.logs_dir.parent / "config.json").read_text() + ) + base = config.agent.override_timeout_sec or Task( + config.task.get_local_path() + ).config.agent.timeout_sec + except (OSError, ValueError) as error: + self.logger.warning( + "Could not resolve the trial's agent time limit (%s); the workglow agent " + "runs without a time budget.", + error, + ) + return None + if base is None: + return None + multiplier = ( + config.agent_timeout_multiplier + if config.agent_timeout_multiplier is not None + else config.timeout_multiplier + ) + return min(base, config.agent.max_timeout_sec or float("inf")) * multiplier + + def _max_duration_sec(self) -> int | None: + if self.options.max_duration_sec is not None: + return self.options.max_duration_sec + if self.options.time_budget == "off": + return None + limit = self._agent_timeout_sec() + if limit is None: + return None + # The loop stops only between rounds; the reserve covers the round in + # flight and writing the summary, and never takes more than half a + # short limit. + # The budget is whole seconds and must end before the limit, so the shortest + # limit that leaves a one-second budget is two seconds. + if limit < 2: + raise ValueError( + f"the trial's agent time limit is {limit:.3f}s; a timed run needs at least 2s" + ) + reserve = min(120.0, max(30.0, limit * 0.1), limit / 2) + return max(1, int(limit - reserve)) + + def _stop_after_sec(self) -> int | None: + """When the agent stops outright, a little before Harbor cancels run(). + + The time budget stops the loop only between rounds, so a model call in + flight at Harbor's limit would be cancelled with it and the summary and + trajectory never written. Stopping first, with time left to write them, + keeps the trial's record whatever round was running. Set whenever the + limit is known, ``time_budget`` or not: it changes no behaviour before + the limit. + """ + limit = self._agent_timeout_sec() + if limit is None or limit < 2: + return None + margin = min(10.0, limit / 4) + return min(_MAX_TIMER_SEC, max(1, int(limit - margin))) + + def _bundle(self) -> Path: + configured = self.options.bundle_path or os.environ.get("WORKGLOW_AGENT_BUNDLE") + bundle = Path(configured).expanduser() if configured else _DEFAULT_BUNDLE + if not bundle.is_file(): + raise FileNotFoundError( + f"workglow agent bundle not found at {bundle}; run `bun run build-agent` in " + "examples/agent-eval, or pass --ak bundle_path=..." + ) + return bundle + + @override + async def install(self, environment: BaseEnvironment) -> None: + bundle = self._bundle() + # Only what nvm needs. Distro Node is not asked for: on glibc images nvm + # installs its own, and asking apt for `nodejs` fails the whole install + # on images whose package index has gone stale. + await self.ensure_system_dependencies(environment, ("curl", "bash", "ca_certificates")) + # The bundle needs Node 22+. On glibc, install it with nvm the way the + # opencode and pi adapters do; on musl nvm's binaries do not run, so + # the packaged Node is all there is. + await self.exec_as_root( + environment, + command=( + "if ldd --version 2>&1 | grep -qi musl || [ -f /etc/alpine-release ]; then " + "command -v node >/dev/null || apk add --no-cache nodejs; fi" + ), + ) + await self.exec_as_agent( + environment, + command=( + "set -euo pipefail; " + "if ldd --version 2>&1 | grep -qi musl || [ -f /etc/alpine-release ]; then " + "node --version; " + # The image's own Node can be older than the bundle accepts; say so + # here, as a setup failure, rather than as every trial crashing. + "node -e 'process.exit(Number(process.versions.node.split(\".\")[0]) >= 22 ? 0 : 1)' " + "|| { echo \"workglow agent needs Node 22+, found $(node --version)\" >&2; exit 1; }; " + f"else {nvm_node_install_snippet()}; fi" + ), + ) + await self.exec_as_root(environment, command=f"mkdir -p {_REMOTE_DIR.as_posix()}") + await self._upload_agent_owned_file(environment, bundle, _REMOTE_BUNDLE.as_posix()) + await self.exec_as_agent( + environment, + command=f"{_NODE_PREFIX}node {shlex.quote(_REMOTE_BUNDLE.as_posix())} --version", + ) + + @override + @with_prompt_template + async def run( + self, + instruction: str, + environment: BaseEnvironment, + context: AgentContext, + ) -> None: + # Harbor's limit runs from here, not from when the agent process starts. + entered = time.monotonic() + if not self.model_name or "/" not in self.model_name: + raise ValueError("Model name must be in the format provider/model_name") + + # Through a file, not argv: a long instruction is past what a shell + # command line safely carries. + self.logs_dir.mkdir(parents=True, exist_ok=True) + local_instruction = self.logs_dir / "instruction.md" + local_instruction.write_text(instruction) + await self._upload_agent_owned_file( + environment, local_instruction, _REMOTE_INSTRUCTION.as_posix() + ) + + await self._upload_credentials(environment) + try: + logs_dir = self.environment_logs_dir.as_posix() + flags = self.build_cli_flags() + # Both are relative to the agent's own start, so the time setup took + # (the uploads above) comes off them. Kept as a float: truncating + # would understate the setup time by up to a second. When setup has + # eaten the whole allowance there is no positive whole second left + # that ends before Harbor's deadline, so the flag is left off rather + # than clamped to 1 and scheduled past it. + spent = time.monotonic() - entered + if self.options.max_duration_sec is None: + budget = self._max_duration_sec() + if budget is not None: + remaining = int(budget - spent) + if remaining >= 1: + flags = f"{flags} --max-duration-sec {remaining}".strip() + stop = self._stop_after_sec() + if stop is not None: + remaining = int(stop - spent) + if remaining >= 1: + flags = f"{flags} --stop-after-sec {remaining}".strip() + await self.exec_as_agent( + environment, + command=( + # Without pipefail the run's status is `tee`'s, and an agent that + # crashed would be recorded as one that finished. + "set -o pipefail; " + f"{_NODE_PREFIX}" + f"node {shlex.quote(_REMOTE_BUNDLE.as_posix())} " + f"--model {shlex.quote(self.model_name)} " + f"--instruction-file {shlex.quote(_REMOTE_INSTRUCTION.as_posix())} " + f"--credentials-file {shlex.quote(_REMOTE_CREDENTIALS.as_posix())} " + f'--cwd "$PWD" --logs-dir {shlex.quote(logs_dir)} ' + f"{flags + ' ' if flags else ''}" + f"2>&1 None: + """Hands the agent its API keys in a file rather than in its environment. + + A process's starting environment stays readable at ``/proc//environ`` + for its whole life, and every command the model runs is a descendant of + the shell and the Node process this adapter starts. Keys passed in the + ``exec`` environment could be read back by the model however the bash + tool filters its own; the agent loads the file into its environment after + it starts, where that file does not show it, and deletes it. + """ + secrets_dir = shlex.quote(_REMOTE_SECRETS_DIR.as_posix()) + user = environment.default_user + owner = f"chown {shlex.quote(str(user))} {secrets_dir} && " if user is not None else "" + await self.exec_as_root( + environment, command=f"mkdir -p {secrets_dir} && {owner}chmod 700 {secrets_dir}" + ) + # Never under logs_dir: that directory is kept with the job's results. + with tempfile.TemporaryDirectory(prefix="workglow-agent-") as temp_dir: + local = Path(temp_dir) / "credentials.json" + fd = os.open(local, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(fd, "w") as handle: + json.dump(dict(self.model_connection.env), handle) + await self._upload_agent_owned_file(environment, local, _REMOTE_CREDENTIALS.as_posix()) + await self.exec_as_root( + environment, command=f"chmod 600 {shlex.quote(_REMOTE_CREDENTIALS.as_posix())}" + ) + + def _summary(self) -> dict[str, Any] | None: + path = self.logs_dir / self._SUMMARY_FILENAME + try: + return json.loads(path.read_text()) + except (OSError, json.JSONDecodeError): + return None + + @override + def populate_context_post_run(self, context: AgentContext) -> None: + summary = self._summary() + if summary is None: + return + usage = summary.get("usage") or {} + # A round the provider reported no usage for makes every sum an + # undercount; leaving them unset is what the report marks as partial. + if not usage.get("unreportedRounds"): + context.n_input_tokens = usage.get("promptTokens") + context.n_output_tokens = usage.get("outputTokens") + # A provider that never reported a cache count did not report zero. + if not usage.get("cacheUnreportedRounds"): + context.n_cache_tokens = usage.get("cacheReadTokens") + context.cost_usd = summary.get("costUsd") + # The comparison reads outcome, rounds and tool tallies from here. + context.metadata = { + "workglow": {key: value for key, value in summary.items() if key != "finalText"} + } diff --git a/examples/agent-eval/package.json b/examples/agent-eval/package.json new file mode 100644 index 0000000000..65075dbb59 --- /dev/null +++ b/examples/agent-eval/package.json @@ -0,0 +1,39 @@ +{ + "name": "@workglow/agent-eval", + "private": true, + "type": "module", + "version": "0.6.12", + "repository": { + "type": "git", + "url": "https://github.com/workglow-dev/libs.git", + "directory": "examples/agent-eval" + }, + "bugs": { + "url": "https://github.com/workglow-dev/libs/issues" + }, + "description": "Agent harness eval: run Workglow's AgentTask, opencode and pi on the same public agentic benchmarks through Harbor, and compare them task by task.", + "homepage": "https://workglow.dev", + "scripts": { + "build-example": "concurrently -c 'auto' -n 'agent,cli,types' 'bun run build-agent' 'bun run build-cli' 'bun run build-types'", + "build-clean": "rm -fr dist/* tsconfig.tsbuildinfo", + "build-agent": "bun build --target=node --format=esm --outfile ./dist/workglow-agent.mjs ./src/bin/workglow-agent.ts", + "build-cli": "bun build --target=bun --packages=external --outdir ./dist/bin ./src/bin/workglow-agent-eval.ts", + "build-types": "rm -f tsconfig.tsbuildinfo && tsc --noEmit", + "test": "vitest run --config ../../vitest.config.ts --project agent-eval" + }, + "bin": { + "workglow-agent-eval": "./dist/bin/workglow-agent-eval.js" + }, + "dependencies": { + "@workglow/ai": "workspace:*", + "@workglow/anthropic": "workspace:*", + "@workglow/deepseek": "workspace:*", + "@workglow/google-gemini": "workspace:*", + "@workglow/openai": "workspace:*", + "@workglow/openrouter": "workspace:*", + "@workglow/task-graph": "workspace:*", + "@workglow/util": "workspace:*", + "@workglow/xai": "workspace:*", + "commander": "catalog:" + } +} diff --git a/examples/agent-eval/src/agent/atif.ts b/examples/agent-eval/src/agent/atif.ts new file mode 100644 index 0000000000..55b90eff92 --- /dev/null +++ b/examples/agent-eval/src/agent/atif.ts @@ -0,0 +1,150 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AgentStep, ChatMessage, ContentBlock } from "@workglow/ai"; +import type { AgentRunSummary } from "./runSummary"; +import { hasTokenCounts } from "./runSummary"; + +/** + * A run as an ATIF trajectory (Harbor's Agent Trajectory Interchange Format), + * so `harbor view` draws a workglow trial the way it draws opencode's and + * pi's. Only the fields Harbor reads are filled. + */ +export interface AtifTrajectory { + readonly schema_version: "ATIF-v1.8"; + readonly session_id: string; + readonly agent: { readonly name: string; readonly version: string; readonly model_name: string }; + readonly steps: readonly AtifStep[]; + readonly final_metrics: { + readonly total_prompt_tokens: number; + readonly total_completion_tokens: number; + readonly total_cached_tokens: number | null; + readonly total_cost_usd: number | null; + readonly total_steps: number; + }; +} + +interface AtifStep { + readonly step_id: number; + readonly timestamp?: string | undefined; + readonly source: "user" | "agent"; + readonly message: string; + readonly model_name?: string | undefined; + readonly tool_calls?: + | ReadonlyArray<{ tool_call_id: string; function_name: string; arguments: unknown }> + | undefined; + readonly observation?: + | { readonly results: ReadonlyArray<{ source_call_id: string; content: string }> } + | undefined; + readonly metrics?: + | { + readonly prompt_tokens: number | null; + readonly completion_tokens: number | null; + readonly cached_tokens: number | null; + readonly cost_usd: number | null; + } + | undefined; +} + +function textOf(blocks: readonly ContentBlock[]): string { + return blocks + .map((block) => (block.type === "text" ? block.text : "")) + .filter((text) => text.length > 0) + .join("\n"); +} + +function resultText(block: Extract): string { + return block.content + .map((part) => (part.type === "text" ? part.text : `[${part.type}]`)) + .join(""); +} + +export function toAtifTrajectory( + sessionId: string, + version: string, + messages: readonly ChatMessage[], + steps: readonly AgentStep[], + summary: AgentRunSummary +): AtifTrajectory { + const out: AtifStep[] = []; + let round = 0; + for (let i = 0; i < messages.length; i++) { + const message = messages[i]!; + if (message.role === "user") { + out.push({ step_id: out.length + 1, source: "user", message: textOf(message.content) }); + continue; + } + if (message.role !== "assistant") continue; + // A round that produced neither text nor a call left no assistant message. + while (round < steps.length && steps[round]!.tools.length === 0 && steps[round]!.text === "") { + round++; + } + const step = steps[round++]; + const calls = message.content.filter( + (block): block is Extract => block.type === "tool_use" + ); + const next = messages[i + 1]; + const results = + next?.role === "tool" + ? next.content + .filter( + (block): block is Extract => + block.type === "tool_result" + ) + .map((block) => ({ source_call_id: block.tool_use_id, content: resultText(block) })) + : []; + const usage = step?.usage; + out.push({ + step_id: out.length + 1, + timestamp: step?.startedAt, + source: "agent", + message: textOf(message.content), + model_name: summary.model, + ...(calls.length > 0 + ? { + tool_calls: calls.map((call) => ({ + tool_call_id: call.id, + function_name: call.name, + arguments: call.input, + })), + } + : {}), + ...(results.length > 0 ? { observation: { results } } : {}), + ...(hasTokenCounts(usage) + ? { + metrics: { + prompt_tokens: + (usage.input ?? 0) + + (usage.imageInput ?? 0) + + (usage.cached ?? 0) + + (usage.imageCached ?? 0) + + (usage.cacheWrite ?? 0), + completion_tokens: usage.output ?? null, + cached_tokens: + usage.cached === undefined && usage.imageCached === undefined + ? null + : (usage.cached ?? 0) + (usage.imageCached ?? 0), + cost_usd: step?.costUsd ?? null, + }, + } + : {}), + }); + } + return { + schema_version: "ATIF-v1.8", + session_id: sessionId, + agent: { name: "workglow", version, model_name: summary.model }, + steps: out, + final_metrics: { + total_prompt_tokens: summary.usage.promptTokens, + total_completion_tokens: summary.usage.outputTokens, + total_cached_tokens: + summary.usage.cacheUnreportedRounds > 0 ? null : summary.usage.cacheReadTokens, + total_cost_usd: summary.costUsd ?? null, + total_steps: out.length, + }, + }; +} diff --git a/examples/agent-eval/src/agent/credentialsFile.ts b/examples/agent-eval/src/agent/credentialsFile.ts new file mode 100644 index 0000000000..de22f74737 --- /dev/null +++ b/examples/agent-eval/src/agent/credentialsFile.ts @@ -0,0 +1,50 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { readFileSync, unlinkSync } from "node:fs"; + +/** + * Moves the variables in a JSON file into `env` and deletes the file, returning + * the names it set. + * + * This is how the Harbor adapter hands the agent its API keys. A process's + * starting environment stays readable at `/proc//environ` for its whole + * life, and every command the model runs through `bash` is a descendant of the + * agent, so a key passed in the environment of the command that starts it can be + * read back by the model however the shell's own environment is filtered. + * Variables set after start are not in that file. The file is deleted before any + * tool can run, and a file that cannot be deleted stops the run: left in place, + * the keys would be one `cat` away. + */ +export function loadCredentialsFile(path: string, env: NodeJS.ProcessEnv = process.env): string[] { + const text = readFileSync(path, "utf8"); + // Deleted before it is parsed, so a malformed file does not outlive the run either. + try { + unlinkSync(path); + } catch (error) { + throw new Error( + `could not delete the credentials file ${path} after reading it: ${(error as Error).message}` + ); + } + let parsed: unknown; + try { + parsed = JSON.parse(text); + } catch { + throw new Error(`the credentials file ${path} is not valid JSON`); + } + if (parsed === null || typeof parsed !== "object" || Array.isArray(parsed)) { + throw new Error(`the credentials file ${path} is not a JSON object`); + } + const names: string[] = []; + for (const [name, value] of Object.entries(parsed)) { + if (typeof value !== "string") { + throw new Error(`the credentials file ${path} has a non-string value for ${name}`); + } + env[name] = value; + names.push(name); + } + return names; +} diff --git a/examples/agent-eval/src/agent/environment.ts b/examples/agent-eval/src/agent/environment.ts new file mode 100644 index 0000000000..662fc28177 --- /dev/null +++ b/examples/agent-eval/src/agent/environment.ts @@ -0,0 +1,61 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { accessSync, constants, statSync } from "node:fs"; +import { delimiter, join } from "node:path"; + +/** + * Commands a coding task commonly reaches for. A benchmark image ships some and + * not others (`python3` without `python`, no `rg`), and a model that guesses + * spends a round on "command not found" each time. + */ +export const PROBED_COMMANDS: readonly string[] = [ + "python3", + "python", + "pip3", + "pip", + "node", + "npm", + "git", + "rg", + "gcc", + "g++", + "make", + "cmake", + "cargo", + "go", + "java", + "file", + "xxd", + "strings", + "7zz", + "7z", +]; + +export interface CommandInventory { + readonly found: readonly string[]; + readonly missing: readonly string[]; +} + +function isExecutableFile(file: string): boolean { + try { + accessSync(file, constants.X_OK); + return statSync(file).isFile(); + } catch { + return false; + } +} + +/** Which of `commands` resolve to an executable on `path` (a PATH-style list). */ +export function probeCommands(commands: readonly string[], path: string): CommandInventory { + const dirs = path.split(delimiter).filter((dir) => dir.length > 0); + const found: string[] = []; + const missing: string[] = []; + for (const command of commands) { + (dirs.some((dir) => isExecutableFile(join(dir, command))) ? found : missing).push(command); + } + return { found, missing }; +} diff --git a/examples/agent-eval/src/agent/models.ts b/examples/agent-eval/src/agent/models.ts new file mode 100644 index 0000000000..1ccf5d24d1 --- /dev/null +++ b/examples/agent-eval/src/agent/models.ts @@ -0,0 +1,106 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ModelConfig, ModelEffort } from "@workglow/ai"; +import { isModelEffort } from "@workglow/ai"; +import { getAnthropicModelPricing } from "@workglow/anthropic/ai"; +import { getDeepSeekModelPricing } from "@workglow/deepseek/ai"; +import { getGeminiModelPricing } from "@workglow/google-gemini/ai"; +import { getOpenAiModelPricing } from "@workglow/openai/ai"; +import { getXaiModelPricing } from "@workglow/xai/ai"; + +interface ProviderSpec { + readonly provider: string; + /** Environment variables naming a custom endpoint, first set wins. */ + readonly baseUrlEnvs: readonly string[]; + readonly pricing?: ((model: string) => ModelConfig["pricing"]) | undefined; +} + +/** + * Provider prefixes as opencode, pi and Harbor spell them, so one `--model` + * string means the same model to all three harnesses. + */ +const PROVIDERS: Readonly> = { + anthropic: { + provider: "ANTHROPIC", + baseUrlEnvs: ["ANTHROPIC_BASE_URL"], + pricing: getAnthropicModelPricing, + }, + openai: { + provider: "OPENAI", + baseUrlEnvs: ["OPENAI_BASE_URL", "OPENAI_API_BASE"], + pricing: getOpenAiModelPricing, + }, + google: { provider: "GOOGLE_GEMINI", baseUrlEnvs: [], pricing: getGeminiModelPricing }, + gemini: { provider: "GOOGLE_GEMINI", baseUrlEnvs: [], pricing: getGeminiModelPricing }, + deepseek: { + provider: "DEEPSEEK", + baseUrlEnvs: ["DEEPSEEK_BASE_URL"], + pricing: getDeepSeekModelPricing, + }, + xai: { provider: "XAI", baseUrlEnvs: ["XAI_BASE_URL"], pricing: getXaiModelPricing }, + openrouter: { provider: "OPENROUTER", baseUrlEnvs: ["OPENROUTER_BASE_URL"] }, +}; + +export const SUPPORTED_PROVIDERS = Object.keys(PROVIDERS); + +/** + * Reasoning levels as pi and opencode name them, mapped onto the coarse + * {@link ModelEffort} dial, so a run configured for one harness reads the same + * here. + */ +const EFFORT_ALIASES: Readonly> = { + off: "none", + minimal: "low", + xhigh: "extra", + max: "ultra", +}; + +export function parseEffort(value: string | undefined): ModelEffort | undefined { + if (value === undefined || value === "") return undefined; + const effort = EFFORT_ALIASES[value] ?? value; + if (!isModelEffort(effort)) { + throw new Error( + `unknown effort "${value}" — use none|low|medium|high|extra|ultra (or off|minimal|xhigh|max)` + ); + } + return effort; +} + +/** + * `provider/model` (Harbor's `-m`) to an inline {@link ModelConfig}. A custom + * endpoint in the provider's `*_BASE_URL` variable is trusted: the operator set + * it for this run, which is the case the allow-list exists to make explicit. + */ +export function resolveAgentModel( + id: string, + options: { readonly effort?: ModelEffort | undefined; readonly env?: NodeJS.ProcessEnv } = {} +): ModelConfig { + const slash = id.indexOf("/"); + if (slash <= 0 || slash === id.length - 1) { + throw new Error(`model must be provider/model (e.g. anthropic/claude-sonnet-4-5), got "${id}"`); + } + const prefix = id.slice(0, slash).toLowerCase(); + const model = id.slice(slash + 1); + const spec = PROVIDERS[prefix]; + if (spec === undefined) { + throw new Error( + `unsupported provider "${prefix}" — use one of ${SUPPORTED_PROVIDERS.join(", ")}` + ); + } + const env = options.env ?? process.env; + const baseUrl = spec.baseUrlEnvs.map((name) => env[name]).find((value) => !!value); + const pricing = spec.pricing?.(model); + return { + provider: spec.provider, + provider_config: { + model_name: model, + ...(baseUrl ? { base_url: baseUrl, trustedBaseUrl: true } : {}), + }, + ...(pricing ? { pricing } : {}), + ...(options.effort ? { effort: options.effort } : {}), + } as ModelConfig; +} diff --git a/examples/agent-eval/src/agent/providers.ts b/examples/agent-eval/src/agent/providers.ts new file mode 100644 index 0000000000..4e27c63d8e --- /dev/null +++ b/examples/agent-eval/src/agent/providers.ts @@ -0,0 +1,33 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { registerAnthropicInline } from "@workglow/anthropic/ai-runtime"; +import { registerDeepSeekInline } from "@workglow/deepseek/ai-runtime"; +import { registerGeminiInline } from "@workglow/google-gemini/ai-runtime"; +import { registerOpenAiInline } from "@workglow/openai/ai-runtime"; +import { registerOpenRouterInline } from "@workglow/openrouter/ai-runtime"; +import { EnvCredentialStore, setGlobalCredentialStore } from "@workglow/util"; +import { registerXaiInline } from "@workglow/xai/ai-runtime"; + +const REGISTRATIONS: Readonly Promise>> = { + ANTHROPIC: () => registerAnthropicInline(), + OPENAI: () => registerOpenAiInline(), + GOOGLE_GEMINI: () => registerGeminiInline(), + DEEPSEEK: () => registerDeepSeekInline(), + XAI: () => registerXaiInline(), + OPENROUTER: () => registerOpenRouterInline(), +}; + +/** + * Registers the one provider a run needs, in process. Keys come from the + * environment, as every harness in the comparison reads them. + */ +export async function registerAgentProvider(provider: string): Promise { + setGlobalCredentialStore(new EnvCredentialStore()); + const register = REGISTRATIONS[provider]; + if (register === undefined) throw new Error(`no registration for provider ${provider}`); + await register(); +} diff --git a/examples/agent-eval/src/agent/runCodingAgent.ts b/examples/agent-eval/src/agent/runCodingAgent.ts new file mode 100644 index 0000000000..9ac8e67535 --- /dev/null +++ b/examples/agent-eval/src/agent/runCodingAgent.ts @@ -0,0 +1,271 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { + AgentStep, + AgentTaskInput, + AgentTaskOutput, + ChatMessage, + ModelConfig, +} from "@workglow/ai"; +import { AGENT_APPROVAL_OPT_OUT, AgentTask, measureHistoryForModel } from "@workglow/ai"; +import type { StreamEvent } from "@workglow/task-graph"; +import { globalServiceRegistry, ServiceRegistry } from "@workglow/util"; +import { PROBED_COMMANDS, probeCommands } from "./environment"; +import type { AgentRunOutcome, AgentRunSummary } from "./runSummary"; +import { summarizeUsage, tallyTools, totalCost } from "./runSummary"; +import { codingSystemPrompt } from "./systemPrompt"; +import type { CodingToolContext } from "./tools/context"; +import { createCodingTools } from "./tools/codingTools"; + +/** + * Rounds a benchmark run gets unless told otherwise. AgentTask's own default + * (8) is sized for a chat turn; a terminal task routinely takes dozens. + */ +export const DEFAULT_BENCH_MAX_ROUNDS = 100; +/** + * Room for one full tool output (50KB) plus its truncation notice, so the + * loop's own clamp never cuts a result the tool already sized. + */ +export const DEFAULT_BENCH_MAX_TOOL_RESULT_CHARS = 60_000; + +/** One line of the run's event log. */ +export type AgentRunEvent = + | { readonly type: "start"; readonly at: string; readonly model: string; readonly cwd: string } + | { + readonly type: "tool-call"; + readonly at: string; + readonly status: string; + readonly id: string; + readonly name: string | undefined; + readonly input?: unknown; + readonly result?: string | undefined; + } + | { readonly type: "round"; readonly at: string; readonly step: AgentStep } + | { readonly type: "finish"; readonly at: string; readonly summary: AgentRunSummary }; + +/** The knobs a Harbor job passes through as agent kwargs, for ablations. */ +export interface CodingAgentSettings { + readonly maxRounds: number; + readonly maxToolResultChars: number; + readonly maxHistoryChars: number | undefined; + readonly toolConcurrency: number | undefined; + readonly maxTokens: number | undefined; + readonly temperature: number | undefined; + readonly maxDurationMs: number | undefined; + readonly roundTimeoutMs: number | undefined; + readonly maxRoundRetries: number | undefined; + readonly systemPromptAppend: string | undefined; + /** Ask for terse replies between tool calls. */ + readonly concise: boolean; + /** Show the model its remaining time each round; needs maxDurationMs. */ + readonly announceTimeLeft: boolean; + /** Sent once when the model first tries to finish; undefined sends none. */ + readonly reviewPrompt: string | undefined; +} + +export interface CodingAgentRun { + readonly instruction: string; + /** `provider/model`, as reported. */ + readonly modelId: string; + readonly model: ModelConfig; + readonly tools: CodingToolContext; + readonly settings: CodingAgentSettings; + readonly signal: AbortSignal; + /** + * Stops the run outright this long after it starts, ending it as `budget`. The + * loop's own `maxDurationMs` stops only between rounds, so a model call in + * flight at the outer limit would run on until whatever runs the agent kills + * it, and the summary and trajectory would never be written. + */ + readonly stopAfterMs?: number | undefined; + readonly onEvent?: ((event: AgentRunEvent) => void) | undefined; +} + +export interface CodingAgentResult { + readonly summary: AgentRunSummary; + readonly messages: readonly ChatMessage[]; + readonly steps: readonly AgentStep[]; +} + +function errorText(error: unknown): string { + return error instanceof Error ? error.message : String(error); +} + +function trimForLog(text: string | undefined, max = 4000): string | undefined { + if (text === undefined || text.length <= max) return text; + return `${text.slice(0, max)}… [${text.length} chars]`; +} + +/** + * One benchmark attempt: the instruction as the only user message, AgentTask + * running the four coding tools until the model stops calling them, a limit + * runs out, or it fails. + * + * The transcript is kept from the loop's own snapshots rather than from its + * final output, so a run that dies mid-way — a context-window overflow is + * the usual one — still leaves every round it completed on the record. + */ +export async function runCodingAgent(run: CodingAgentRun): Promise { + const now = (): string => new Date().toISOString(); + const emit = (event: AgentRunEvent): void => run.onEvent?.(event); + const startedAt = Date.now(); + emit({ type: "start", at: now(), model: run.modelId, cwd: run.tools.cwd }); + + const registry = new ServiceRegistry(globalServiceRegistry.container.createChildContainer()); + // Headless: no person to approve a call, and every tool is a host function. + registry.registerInstance(AGENT_APPROVAL_OPT_OUT, true); + + let messages: readonly ChatMessage[] = []; + let steps: readonly AgentStep[] = []; + let reportedRounds = 0; + const reportRounds = (): void => { + for (; reportedRounds < steps.length; reportedRounds++) { + emit({ type: "round", at: now(), step: steps[reportedRounds]! }); + } + }; + const task = new AgentTask(); + // Counted off the loop's own progress messages, since `steps` holds only the + // rounds that settled: a round that exhausted its retries leaves no step, and + // its attempts would otherwise be missing from the summary. + let roundsStarted = 0; + let retriesSeen = 0; + const offProgress = task.subscribe("progress", (_progress, message) => { + if (message === "Thinking") roundsStarted++; + else if (message?.startsWith("Thinking (attempt ")) retriesSeen++; + }); + const off = task.subscribe("stream_chunk", (event: StreamEvent) => { + if (event.type === "snapshot") { + const data = event.data as { messages?: ChatMessage[]; steps?: AgentStep[] }; + if (data.messages) messages = data.messages; + if (data.steps) { + steps = data.steps; + reportRounds(); + } + } else if (event.type === "tool-call") { + const call = event as unknown as { + status: string; + toolCallId: string; + name?: string; + input?: unknown; + result?: string; + }; + emit({ + type: "tool-call", + at: now(), + status: call.status, + id: call.toolCallId, + name: call.name, + ...(call.input === undefined ? {} : { input: call.input }), + ...(call.result === undefined ? {} : { result: trimForLog(call.result) }), + }); + } + }); + + const settings = run.settings; + const input: AgentTaskInput = { + model: run.model, + prompt: run.instruction, + systemPrompt: + codingSystemPrompt({ + cwd: run.tools.cwd, + platform: `${process.platform} ${process.arch}`, + date: new Date().toISOString().slice(0, 10), + concise: settings.concise, + images: run.tools.images, + commands: probeCommands(PROBED_COMMANDS, process.env.PATH ?? ""), + }) + (settings.systemPromptAppend ? `\n\n${settings.systemPromptAppend}` : ""), + tools: createCodingTools(run.tools), + maxRounds: settings.maxRounds, + maxToolResultChars: settings.maxToolResultChars, + approval: "never", + ...(settings.maxHistoryChars === undefined + ? {} + : { maxHistoryChars: settings.maxHistoryChars }), + ...(settings.toolConcurrency === undefined + ? {} + : { toolConcurrency: settings.toolConcurrency }), + ...(settings.maxTokens === undefined ? {} : { maxTokens: settings.maxTokens }), + ...(settings.temperature === undefined ? {} : { temperature: settings.temperature }), + ...(settings.maxDurationMs === undefined ? {} : { maxDurationMs: settings.maxDurationMs }), + ...(settings.announceTimeLeft && settings.maxDurationMs !== undefined + ? { announceTimeLeft: true } + : {}), + ...(settings.reviewPrompt === undefined ? {} : { reviewPrompt: settings.reviewPrompt }), + ...(settings.roundTimeoutMs === undefined ? {} : { roundTimeoutMs: settings.roundTimeoutMs }), + ...(settings.maxRoundRetries === undefined + ? {} + : { maxRoundRetries: settings.maxRoundRetries }), + }; + + const stopper = new AbortController(); + let stoppedAtLimit = false; + const stopAfterMs = run.stopAfterMs; + const stopTimer = + stopAfterMs === undefined + ? undefined + : setTimeout(() => { + stoppedAtLimit = true; + stopper.abort(new Error(`stopped at the hard time limit of ${stopAfterMs} ms`)); + }, stopAfterMs); + const signal = AbortSignal.any([run.signal, stopper.signal]); + + let outcome: AgentRunOutcome; + let error: string | undefined; + let finalText = ""; + let rounds = 0; + try { + const output = (await task.run(input, { registry, signal })) as AgentTaskOutput; + messages = output.messages; + steps = output.steps; + outcome = output.stopReason; + rounds = output.rounds; + // Only a round that called no tool answered; the narration beside a call is + // not an answer when the turn ran out of rounds or budget after it. + const last = steps.at(-1); + finalText = last !== undefined && last.tools.length === 0 ? last.text : ""; + } catch (caught) { + outcome = stoppedAtLimit ? "budget" : run.signal.aborted ? "aborted" : "error"; + // The provider reports its own cancellation; the limit is the reason it ran into. + error = errorText(stoppedAtLimit ? stopper.signal.reason : caught); + rounds = Math.max(steps.length, roundsStarted); + } finally { + clearTimeout(stopTimer); + off(); + offProgress(); + } + // The last round, when it ends the turn without a call, is in the output + // but was never in a snapshot. + reportRounds(); + + // A round that started and never settled (it failed, was stopped, or ended the + // turn while waiting to retry) has no step, though its request may have been + // billed. Its usage and cost are unknown, so the totals are reported as + // incomplete rather than as the settled rounds' subtotal. + const unsettled = rounds > steps.length; + const usage = summarizeUsage(steps); + const summary: AgentRunSummary = { + harness: "workglow", + model: run.modelId, + outcome, + error, + rounds, + retries: Math.max( + retriesSeen, + steps.reduce((sum, step) => sum + Math.max(0, step.attempts - 1), 0) + ), + durationMs: Date.now() - startedAt, + modelMs: steps.reduce((sum, step) => sum + step.modelMs, 0), + usage: unsettled ? { ...usage, unreportedRounds: usage.unreportedRounds + 1 } : usage, + costUsd: unsettled ? undefined : totalCost(steps), + tools: tallyTools(steps), + finalHistoryChars: measureHistoryForModel(messages), + finalText, + settings: { ...settings }, + }; + emit({ type: "finish", at: now(), summary }); + return { summary, messages, steps }; +} diff --git a/examples/agent-eval/src/agent/runSummary.ts b/examples/agent-eval/src/agent/runSummary.ts new file mode 100644 index 0000000000..25bbe148cc --- /dev/null +++ b/examples/agent-eval/src/agent/runSummary.ts @@ -0,0 +1,148 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AgentStep } from "@workglow/ai"; + +/** How a run ended: the loop's own stop reason, or the error that ended it. */ +export type AgentRunOutcome = + | "answered" + | "submitted" + | "budget" + | "max-rounds" + | "error" + | "aborted"; + +export interface AgentRunUsage { + /** Prompt tokens not served from cache. */ + readonly inputTokens: number; + readonly cacheReadTokens: number; + readonly cacheWriteTokens: number; + readonly outputTokens: number; + /** Every prompt token, cached or not — what Harbor reports as input. */ + readonly promptTokens: number; + /** Rounds whose provider reported no usage at all. */ + readonly unreportedRounds: number; + /** + * Rounds that reported token counts but no cache-read count. Such a provider + * did not say it read nothing from cache, so `cacheReadTokens` is a floor. + */ + readonly cacheUnreportedRounds: number; +} + +export interface AgentToolTally { + readonly calls: number; + readonly errors: number; + readonly byName: Readonly>; +} + +/** + * Everything the comparison reads about one workglow run, written to + * `summary.json` beside the transcript. The Harbor adapter copies the token + * and cost totals into its trial result; the rest rides along as metadata. + */ +export interface AgentRunSummary { + readonly harness: "workglow"; + readonly model: string; + readonly outcome: AgentRunOutcome; + readonly error: string | undefined; + readonly rounds: number; + /** Model calls that failed and were retried. */ + readonly retries: number; + readonly durationMs: number; + readonly modelMs: number; + readonly usage: AgentRunUsage; + /** Undefined when any round could not be priced. */ + readonly costUsd: number | undefined; + readonly tools: AgentToolTally; + /** + * Size of the transcript, in characters, as AgentTask's history budget + * measures it: a reasoning block's replay `payload` is left out, so the + * figure compares with `maxHistoryChars` rather than with encrypted reasoning. + */ + readonly finalHistoryChars: number; + readonly finalText: string; + readonly settings: Readonly>; +} + +/** + * Whether a round's usage carries any token count. One with none (a provider + * that reported only a cost) is as unreported as no usage at all: summing it + * would count the round as zero tokens. + */ +export function hasTokenCounts( + usage: AgentStep["usage"] +): usage is NonNullable { + return ( + usage !== undefined && + [ + usage.input, + usage.output, + usage.cached, + usage.cacheWrite, + usage.imageInput, + usage.imageCached, + ].some((n) => n !== undefined) + ); +} + +export function summarizeUsage(steps: readonly AgentStep[]): AgentRunUsage { + let inputTokens = 0; + let cacheReadTokens = 0; + let cacheWriteTokens = 0; + let outputTokens = 0; + let unreportedRounds = 0; + let cacheUnreportedRounds = 0; + for (const step of steps) { + const usage = step.usage; + if (!hasTokenCounts(usage)) { + unreportedRounds++; + continue; + } + if (usage.cached === undefined && usage.imageCached === undefined) cacheUnreportedRounds++; + inputTokens += (usage.input ?? 0) + (usage.imageInput ?? 0); + cacheReadTokens += (usage.cached ?? 0) + (usage.imageCached ?? 0); + cacheWriteTokens += usage.cacheWrite ?? 0; + outputTokens += usage.output ?? 0; + } + return { + inputTokens, + cacheReadTokens, + cacheWriteTokens, + outputTokens, + promptTokens: inputTokens + cacheReadTokens + cacheWriteTokens, + unreportedRounds, + cacheUnreportedRounds, + }; +} + +export function tallyTools(steps: readonly AgentStep[]): AgentToolTally { + // Keyed by names the model chose, so no prototype: `__proto__` is a name too. + const byName: Record = Object.create(null); + let calls = 0; + let errors = 0; + for (const step of steps) { + for (const tool of step.tools) { + const entry = (byName[tool.name] ??= { calls: 0, errors: 0 }); + entry.calls++; + calls++; + if (tool.isError) { + entry.errors++; + errors++; + } + } + } + return { calls, errors, byName }; +} + +/** Sum of round costs, or undefined as soon as one round has none. */ +export function totalCost(steps: readonly AgentStep[]): number | undefined { + let total = 0; + for (const step of steps) { + if (step.costUsd === undefined) return undefined; + total += step.costUsd; + } + return steps.length > 0 ? total : undefined; +} diff --git a/examples/agent-eval/src/agent/systemPrompt.ts b/examples/agent-eval/src/agent/systemPrompt.ts new file mode 100644 index 0000000000..26cced5d8d --- /dev/null +++ b/examples/agent-eval/src/agent/systemPrompt.ts @@ -0,0 +1,79 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { CommandInventory } from "./environment"; + +export interface SystemPromptEnvironment { + readonly cwd: string; + readonly platform: string; + readonly date: string; + /** Ask for terse replies between tool calls; off reproduces the earlier prompt. */ + readonly concise: boolean; + /** Whether `read` shows images, which the tools line then says. */ + readonly images: boolean; + /** Commands probed on PATH; undefined leaves the line out. */ + readonly commands: CommandInventory | undefined; +} + +/** + * A deliberately small prompt in the shape of pi's: who the agent is, one line + * per tool, the rules that keep tool calls from failing, and where it is. It + * says nothing a benchmark could game; it is the prompt a host would ship. + */ +export function codingSystemPrompt(env: SystemPromptEnvironment): string { + return [ + "You are an expert software engineer working autonomously in a terminal. You complete the " + + "task you are given by reading files, running commands, editing code and writing files. " + + "No one will answer questions: make reasonable decisions and finish the task.", + "", + "", + env.images + ? "- read: read file contents (use offset/limit to page through long files); shows you images (png, jpg, gif, webp)" + : "- read: read file contents (use offset/limit to page through long files)", + "- bash: run shell commands (ls, grep, find, tests, builds, package managers)", + "- edit: replace exact text in a file; several disjoint edits per call", + "- write: create a file or overwrite it completely", + "", + "", + "", + "- Read a file before editing it; oldText must match the current file exactly.", + "- Prefer edit for changes to existing files; use write for new files or full rewrites.", + "- Each bash call is a fresh shell; chain dependent commands with &&.", + "- Verify once the change is complete: run the code or its tests then, not after every edit.", + "- When the task is complete, reply with a short summary and no tool calls.", + ...(env.concise + ? [ + // Narration between calls is output the model pays for and nobody reads. + "- Be concise: do not narrate between tool calls; say only what the next step needs.", + ] + : []), + "", + "", + "", + `cwd: ${env.cwd}`, + `platform: ${env.platform}`, + `date: ${env.date}`, + ...(env.commands === undefined + ? [] + : [ + `commands on PATH: ${env.commands.found.join(", ") || "(none of the usual ones)"}`, + ...(env.commands.missing.length > 0 + ? [`not installed: ${env.commands.missing.join(", ")}`] + : []), + ]), + "", + ].join("\n"); +} + +/** + * Sent once when the model first tries to finish. The tests are often hidden, + * and the requirement a correct-looking answer misses is an exact string the + * instruction named (an error message, an output format, a file name). + */ +export const REVIEW_PROMPT = + "Before you finish, reread the task instructions. Check each exact requirement they state " + + "(names, messages, output formats, file paths, values) against what you built, and fix " + + "any mismatch. Then reply with your summary."; diff --git a/examples/agent-eval/src/agent/tools/bash.ts b/examples/agent-eval/src/agent/tools/bash.ts new file mode 100644 index 0000000000..4a28171272 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/bash.ts @@ -0,0 +1,324 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { ToolCallError } from "@workglow/ai"; +import { uuid4 } from "@workglow/util"; +import { spawn } from "node:child_process"; +import { mkdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import type { CodingToolContext } from "./context"; +import { optionalInteger, requireString } from "./context"; +import { truncateTail } from "./truncate"; + +/** Output held in memory per command; the rest is counted, not kept. */ +const MAX_CAPTURE_BYTES = 16 * 1024 * 1024; +/** Grace between SIGTERM and SIGKILL for a command being stopped. */ +const KILL_GRACE_MS = 2000; +/** Wait after the shell exits for output still in the pipes. */ +const DRAIN_MS = 250; +/** The longest delay a Node timer holds; a longer one fires at once. */ +const MAX_TIMER_MS = 2_147_483_647; +/** The longest timeout a command can ask for: the longest timer Node can set. */ +export const MAX_TIMEOUT_SEC = Math.floor(MAX_TIMER_MS / 1000); + +export interface CommandResult { + readonly output: string; + readonly exitCode: number | null; + readonly timedOut: boolean; + readonly droppedBytes: number; +} + +/** + * Names of variables holding a credential: `*_API_KEY`, `*_TOKEN` (`HF_TOKEN`, + * `ANTHROPIC_AUTH_TOKEN`), `*_SECRET`, `*_SECRET_KEY`, `*_ACCESS_KEY`, `*_PASSWORD`. + */ +const CREDENTIAL_ENV = /(?:_API_KEY|_TOKEN|_SECRET|_SECRET_KEY|_ACCESS_KEY|_PASSWORD)$/i; + +/** + * The environment a command runs in: the agent's own, minus the credentials it + * reads its model's key from. The model writes the commands, so a key left here + * is one `printenv` away from the transcript and the logs. + */ +export function shellEnv(env: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv { + return Object.fromEntries(Object.entries(env).filter(([name]) => !CREDENTIAL_ENV.test(name))); +} + +/** + * Values that must never reach the transcript, read live so a test that sets + * them late still scrubs. Only non-trivial values: redacting a one-character + * value would mangle innocent output sharing it. + */ +function credentialValues(env: NodeJS.ProcessEnv = process.env): string[] { + const values = Object.entries(env) + .filter( + ([name, value]) => CREDENTIAL_ENV.test(name) && value !== undefined && value.length >= 8 + ) + .map(([, value]) => value as string) + .sort((a, b) => b.length - a.length); + return [...new Set(values)]; +} + +/** + * Removes credential values from command output before it reaches the + * transcript, the event log, or the spill file. Filtering the child's + * environment is not enough on its own: the agent starts with the provider + * keys in its own environment, and on Linux a same-UID command can read that + * back out of `/proc/$PPID/environ` no matter what the child was given, so a + * `printenv` and a `/proc` read exfiltrate alike. Scrubbing the output covers + * both, whatever path the value took to get there. + */ +export function scrubCredentialValues( + output: string, + env: NodeJS.ProcessEnv = process.env +): string { + let scrubbed = output; + for (const secret of credentialValues(env)) { + if (scrubbed.includes(secret)) scrubbed = scrubbed.split(secret).join("[REDACTED]"); + } + return scrubbed; +} + +/** + * Runs `command` under `bash -c` in its own process group, so a timeout or an + * abort stops everything it started — a test runner's workers, a server it + * backgrounded — rather than only the shell. + */ +export function runCommand( + command: string, + cwd: string, + timeoutMs: number, + signal: AbortSignal +): Promise { + return new Promise((resolve, reject) => { + // An abort that landed before the call would never fire the listener below. + if (signal.aborted) { + reject(signal.reason ?? new Error("aborted")); + return; + } + const child = spawn("bash", ["-c", command], { + cwd, + detached: true, + stdio: ["ignore", "pipe", "pipe"], + env: shellEnv(), + }); + const chunks: Buffer[] = []; + let held = 0; + let dropped = 0; + // The end of a command's output is where its failures and summaries are, so + // past the cap the oldest bytes go, never the newest. + const take = (chunk: Buffer): void => { + chunks.push(chunk); + held += chunk.length; + while (held > MAX_CAPTURE_BYTES) { + const excess = held - MAX_CAPTURE_BYTES; + const first = chunks[0]!; + if (first.length <= excess) { + chunks.shift(); + held -= first.length; + dropped += first.length; + } else { + chunks[0] = first.subarray(excess); + held -= excess; + dropped += excess; + } + } + }; + child.stdout.on("data", take); + child.stderr.on("data", take); + + let timedOut = false; + // Set once the command is told to stop. The command is over only when its + // whole group is: a process it started can ignore SIGTERM and outlive the + // shell, and returning while it still runs would let the next tool call + // race it. So the result waits for the group to go, and sends SIGKILL to + // whatever is left once the grace period is up. + let stopped: Promise | undefined; + const stop = (): void => { + if (child.pid === undefined || stopped !== undefined) return; + const group = child.pid; + try { + process.kill(-group, "SIGTERM"); + } catch { + // Already gone. + } + stopped = new Promise((done) => { + const since = Date.now(); + const check = (): void => { + if (!groupAlive(group)) return done(); + if (Date.now() - since >= KILL_GRACE_MS) { + try { + process.kill(-group, "SIGKILL"); + } catch { + // Already gone. + } + return done(); + } + setTimeout(check, 50); + }; + check(); + }); + }; + const timer = setTimeout( + () => { + timedOut = true; + stop(); + }, + Math.min(timeoutMs, MAX_TIMER_MS) + ); + const onAbort = (): void => stop(); + signal.addEventListener("abort", onAbort, { once: true }); + + let settled = false; + const finish = (code: number | null): void => { + if (settled) return; + settled = true; + clearTimeout(timer); + signal.removeEventListener("abort", onAbort); + child.stdout.destroy(); + child.stderr.destroy(); + if (signal.aborted) { + reject(signal.reason ?? new Error("aborted")); + return; + } + resolve({ + output: Buffer.concat(chunks).toString("utf8"), + exitCode: code, + timedOut, + droppedBytes: dropped, + }); + }; + child.on("error", (error) => { + if (settled) return; + settled = true; + clearTimeout(timer); + signal.removeEventListener("abort", onAbort); + reject(error); + }); + // A command that backgrounds a process (`server &`) leaves it holding the + // pipes, and "close" would wait for it. The shell exiting is the command + // finishing; give the pipes a moment to drain, then stop listening. + // The command is over once the shell exits, so the timeout is too: it must + // not fire during the drain and report a finished command as timed out. + const finishOnceStopped = (code: number | null): void => { + if (stopped === undefined) finish(code); + else void stopped.then(() => finish(code)); + }; + child.on("exit", (code) => { + clearTimeout(timer); + setTimeout(() => finishOnceStopped(code), DRAIN_MS); + }); + child.on("close", (code) => finishOnceStopped(code)); + }); +} + +/** Whether any process in a group is still there (a zombie counts until reaped). */ +function groupAlive(group: number): boolean { + try { + process.kill(-group, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code === "EPERM"; + } +} + +/** + * The timeout a command runs under: what was asked for, cut to the whole + * seconds the task has left. The tool runs nothing with under a second left, + * so the floor of one second never reaches past the deadline. + */ +export function commandTimeoutSec( + requested: number, + deadline: number | undefined, + now: number +): { readonly sec: number; readonly capped: boolean } { + if (deadline === undefined) return { sec: requested, capped: false }; + const left = Math.max(1, Math.floor((deadline - now) / 1000)); + return left < requested ? { sec: left, capped: true } : { sec: requested, capped: false }; +} + +export function createBashTool(context: CodingToolContext): ToolDefinition { + return { + name: "bash", + description: + "Run a bash command in the working directory and return its combined stdout and stderr. " + + "Each call is a fresh shell: cd and exported variables do not carry over, so chain " + + "commands with && when they depend on each other. Output is truncated to the last 2000 " + + `lines or 50KB; the full output is saved to a file named in the result. Commands time out ` + + `after ${context.defaultCommandTimeoutSec}s unless you pass a larger timeout (seconds), ` + + "and never run past the time left for the task: run long jobs in the background and poll. " + + "Do not start interactive programs.", + inputSchema: { + type: "object", + properties: { + command: { type: "string", description: "The command to run" }, + timeout: { + type: "integer", + minimum: 1, + maximum: MAX_TIMEOUT_SEC, + description: "Timeout in seconds", + }, + }, + required: ["command"], + additionalProperties: false, + }, + requiresApproval: false, + execute: async (input, { signal, deadline }) => { + const command = requireString(input, "command"); + const requested = optionalInteger(input, "timeout") ?? context.defaultCommandTimeoutSec; + if (requested < 1) { + throw new ToolCallError(`"timeout" must be at least 1 second; the command was not run.`); + } + // A longer timer cannot be set; refused rather than silently shortened. + if (requested > MAX_TIMEOUT_SEC) { + throw new ToolCallError( + `"timeout" can be at most ${MAX_TIMEOUT_SEC} seconds; the command was not run.` + ); + } + if (deadline !== undefined && deadline - Date.now() < 1000) { + throw new ToolCallError( + "No time is left for this task; the command was not run. Finish with what you have." + ); + } + const { sec: timeoutSec, capped } = commandTimeoutSec(requested, deadline, Date.now()); + const result = await runCommand(command, context.cwd, timeoutSec * 1000, signal); + + // Scrubbed before anything persists it: the transcript, the event log, + // and the spill file below all derive from this string. + const output = scrubCredentialValues(result.output); + const tail = truncateTail(output); + let text = tail.text; + if (tail.truncated || result.droppedBytes > 0) { + await mkdir(context.spillDir, { recursive: true }); + const spill = join(context.spillDir, `bash-${uuid4()}.log`); + await writeFile(spill, output, "utf8"); + const lost = + result.droppedBytes > 0 ? `, ${result.droppedBytes} earlier bytes discarded` : ""; + text = + `[Output truncated: showing the last ${tail.keptLines} of ${tail.totalLines} lines` + + `${tail.partialLine ? ", the first of them only its end" : ""}. ` + + `Full output (${result.droppedBytes > 0 ? "last " : ""}${output.length} characters${lost}) saved to ${spill}]\n` + + text; + } + if (text.length === 0) text = "(no output)"; + if (result.timedOut) { + throw new ToolCallError( + capped + ? `${text}\n\nCommand was killed after ${timeoutSec}s, all the time left for this ` + + "task. Finish with what you have." + : `${text}\n\nCommand timed out after ${timeoutSec}s and was killed. ` + + "Pass a larger timeout, or run it in the background and poll." + ); + } + if (result.exitCode !== 0) { + throw new ToolCallError( + `${text}\n\nCommand exited with code ${result.exitCode ?? "unknown"}` + ); + } + return text; + }, + }; +} diff --git a/examples/agent-eval/src/agent/tools/codingTools.ts b/examples/agent-eval/src/agent/tools/codingTools.ts new file mode 100644 index 0000000000..0763c84323 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/codingTools.ts @@ -0,0 +1,28 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { createBashTool } from "./bash"; +import type { CodingToolContext } from "./context"; +import { createEditTool } from "./edit"; +import { createReadTool } from "./read"; +import { createWriteTool } from "./write"; + +/** + * The toolset the eval hands AgentTask: read, bash, edit, write — pi's default + * four, with its limits. Holding the tools to pi's means a gap between the + * Workglow and pi columns is a gap in the loop around them; opencode's richer + * set (grep, glob, todo, subagents, fuzzier edit) is part of what its column + * measures. + */ +export function createCodingTools(context: CodingToolContext): ToolDefinition[] { + return [ + createReadTool(context), + createBashTool(context), + createEditTool(context), + createWriteTool(context), + ]; +} diff --git a/examples/agent-eval/src/agent/tools/context.ts b/examples/agent-eval/src/agent/tools/context.ts new file mode 100644 index 0000000000..d865dcf06b --- /dev/null +++ b/examples/agent-eval/src/agent/tools/context.ts @@ -0,0 +1,46 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { isAbsolute, resolve } from "node:path"; + +/** What every coding tool is bound to. */ +export interface CodingToolContext { + /** Directory relative paths and commands resolve against. */ + readonly cwd: string; + /** Directory a command's full output is spilled to when the model sees only its tail. */ + readonly spillDir: string; + /** Seconds a command may run when the model does not say. */ + readonly defaultCommandTimeoutSec: number; + /** Whether `read` returns images to the model, or refuses them as binary. */ + readonly images: boolean; +} + +/** + * Where a path the model wrote points. Absolute paths are honoured: benchmark + * instructions routinely name `/app/…`, and the sandbox is the container, not + * the working directory. + */ +export function resolvePath(context: CodingToolContext, path: string): string { + const trimmed = path.trim(); + const expanded = trimmed.startsWith("~/") + ? `${process.env.HOME ?? "/root"}${trimmed.slice(1)}` + : trimmed; + return isAbsolute(expanded) ? expanded : resolve(context.cwd, expanded); +} + +export function requireString(input: Record, key: string): string { + const value = input[key]; + if (typeof value !== "string") throw new Error(`"${key}" must be a string`); + return value; +} + +export function optionalInteger(input: Record, key: string): number | undefined { + const value = input[key]; + if (value === undefined || value === null) return undefined; + const n = typeof value === "string" ? Number(value) : value; + if (typeof n !== "number" || !Number.isFinite(n)) throw new Error(`"${key}" must be a number`); + return Math.trunc(n); +} diff --git a/examples/agent-eval/src/agent/tools/edit.ts b/examples/agent-eval/src/agent/tools/edit.ts new file mode 100644 index 0000000000..d745655a42 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/edit.ts @@ -0,0 +1,329 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { ToolCallError } from "@workglow/ai"; +import { readFile, stat, writeFile } from "node:fs/promises"; +import type { CodingToolContext } from "./context"; +import { requireString, resolvePath } from "./context"; +import { withFileLock } from "./fileLock"; +import { MAX_READ_BYTES } from "./read"; + +export interface TextEdit { + readonly oldText: string; + readonly newText: string; +} + +interface Range { + readonly start: number; + readonly end: number; + readonly fuzzy: boolean; +} + +/** + * One line as the fuzzy pass compares it: compatibility-normalized, with + * typographic quotes, dashes and spaces folded to ASCII and trailing + * whitespace dropped. These are the differences a model introduces when it + * retypes a line it read, and none of them is a difference the author meant. + */ +export function normalizeLine(line: string): string { + return line + .normalize("NFKC") + .replace(/[‘’‚‛]/g, "'") + .replace(/[“”„‟]/g, '"') + .replace(/[‐-―−]/g, "-") + .replace(/[  -​  　]/g, " ") + .trimEnd(); +} + +function exactRanges(content: string, needle: string): number[] { + const found: number[] = []; + let at = content.indexOf(needle); + // Only uniqueness matters to the caller; stop once it is disproven instead + // of retaining tens of millions of hits on a repetitive file. + while (at !== -1 && found.length < 2) { + found.push(at); + at = content.indexOf(needle, at + 1); + } + return found; +} + +/** + * Whole-line matches of `oldText` once both sides are normalized. Matching + * whole lines keeps the mapping back to the original text exact: the range is + * the original lines, whatever the normalization changed inside them. + */ +function fuzzyRanges(content: string, oldText: string): Range[] { + const trailingNewline = oldText.endsWith("\n"); + const body = trailingNewline ? oldText.slice(0, -1) : oldText; + const wanted = body.split("\n").map(normalizeLine); + // Knuth–Morris–Pratt over normalized lines: one pass, each file line + // normalized once, so a needle of many repeated lines against a file of them + // costs file + needle rather than file × needle. The file is walked in place + // (a file within the size limit can hold millions of lines); only `wanted`, + // and where its last few candidate lines start, are kept. + const fallback = new Array(wanted.length).fill(0); + for (let i = 1, k = 0; i < wanted.length; i++) { + while (k > 0 && wanted[i] !== wanted[k]) k = fallback[k - 1]!; + if (wanted[i] === wanted[k]) k++; + fallback[i] = k; + } + const starts = new Array(wanted.length).fill(0); + const ranges: Range[] = []; + let matched = 0; + // Lines are the fields `content.split("\n")` would give, the empty one after + // a final newline included. + for (let at = 0, index = 0; at <= content.length; index++) { + const newline = content.indexOf("\n", at); + const lineEnd = newline === -1 ? content.length : newline; + const line = normalizeLine(content.slice(at, lineEnd)); + starts[index % wanted.length] = at; + while (matched > 0 && line !== wanted[matched]) matched = fallback[matched - 1]!; + if (line === wanted[matched]) matched++; + if (matched === wanted.length) { + const start = starts[(index - wanted.length + 1) % wanted.length]!; + // A file ending in a newline splits into a last empty field that is no + // line at all; a match reaching it claimed a blank line the file does not have. + const pastLastLine = content.endsWith("\n") && at === content.length; + let end = lineEnd; + if (trailingNewline && end < content.length) end += 1; + // Blank lines matched against the end of a file that has none (an empty + // file asked for "\n") consume nothing: the edit would insert, not replace. + if (!pastLastLine && end !== start) { + ranges.push({ start, end, fuzzy: true }); + // As above: stop once uniqueness is disproven. + if (ranges.length === 2) return ranges; + } + matched = fallback[matched - 1]!; + } + at = newline === -1 ? content.length + 1 : newline + 1; + } + return ranges; +} + +/** + * Where one edit applies in the original text. Exact first, then the fuzzy + * line match; either way the match must be unique, because guessing which of + * two identical blocks the model meant is how an edit lands in the wrong place. + */ +function locate(content: string, edit: TextEdit, path: string, index: number): Range { + const label = `edits[${index}]`; + if (edit.oldText.length === 0) { + throw new ToolCallError(`${label}: oldText is empty. Use write to create a file.`); + } + const exact = exactRanges(content, edit.oldText); + if (exact.length === 1) { + return { start: exact[0]!, end: exact[0]! + edit.oldText.length, fuzzy: false }; + } + if (exact.length > 1) { + throw new ToolCallError( + `${label}: found multiple occurrences of oldText in ${path}. ` + + "Include more surrounding lines so it matches exactly one place." + ); + } + const fuzzy = fuzzyRanges(content, edit.oldText); + if (fuzzy.length === 1) return fuzzy[0]!; + if (fuzzy.length > 1) { + throw new ToolCallError( + `${label}: found multiple places matching oldText in ${path} (ignoring trailing ` + + "whitespace and typographic quotes). Include more surrounding lines so it matches one." + ); + } + throw new ToolCallError( + `${label}: could not find oldText in ${path}. It must match the file exactly, including ` + + "indentation. Read the file again to see its current content." + ); +} + +export interface EditOutcome { + readonly content: string; + /** Applied edits that matched only after normalizing whitespace and quotes. */ + readonly fuzzy: number; + /** Indices of the edits written. */ + readonly applied: readonly number[]; + /** Indices of edits whose newText equals their oldText: nothing to do. */ + readonly unchanged: readonly number[]; + /** Why each remaining edit was not applied, as `edits[i]: reason`. */ + readonly failures: readonly string[]; +} + +/** + * Applies every edit that can be applied, each located against the ORIGINAL + * text so one edit cannot move another's match, and reports the rest. Failing + * the whole call for one bad entry costs the model a round resending the good + * ones; this way it resends only what failed. An edit overlapping one earlier + * in the list is refused rather than resolved in some order. + */ +export function applyEdits(content: string, edits: readonly TextEdit[], path: string): EditOutcome { + const unchanged: number[] = []; + const failures: string[] = []; + const located: Array<{ + readonly index: number; + readonly edit: TextEdit; + readonly range: Range; + }> = []; + edits.forEach((edit, index) => { + if (edit.oldText.length > 0 && edit.oldText === edit.newText) { + unchanged.push(index); + return; + } + try { + const range = locate(content, edit, path, index); + const clash = located.find( + (other) => range.start < other.range.end && other.range.start < range.end + ); + if (clash) { + failures.push( + `edits[${index}]: overlaps edits[${clash.index}], which was applied. Read the file and ` + + "resend this change against its current text." + ); + return; + } + located.push({ index, edit, range }); + } catch (error) { + if (!(error instanceof ToolCallError)) throw error; + failures.push(error.message); + } + }); + let next = content; + for (const { edit, range } of [...located].sort((a, b) => b.range.start - a.range.start)) { + next = next.slice(0, range.start) + edit.newText + next.slice(range.end); + } + return { + content: next, + fuzzy: located.filter((entry) => entry.range.fuzzy).length, + applied: located.map((entry) => entry.index).sort((a, b) => a - b), + unchanged, + failures, + }; +} + +/** + * The edits, as the schema already checked them. A JSON-encoded array or the + * single `oldText`/`newText` shape models also send never get here: AgentTask + * validates against the schema before the tool runs, and has no hook to repair + * arguments first. + */ +function readEdits(input: Record): TextEdit[] { + const raw = input.edits; + if (!Array.isArray(raw) || raw.length === 0) { + throw new ToolCallError("edits must be a non-empty array of {oldText, newText}"); + } + return raw.map((entry, index) => { + const edit = entry as Record; + if (typeof edit?.oldText !== "string" || typeof edit?.newText !== "string") { + throw new ToolCallError(`edits[${index}] needs string oldText and newText`); + } + return { oldText: edit.oldText, newText: edit.newText }; + }); +} + +export function createEditTool(context: CodingToolContext): ToolDefinition { + return { + name: "edit", + description: + "Replace text in an existing file. Each edit's oldText must match exactly one place in " + + "the file as it was before this call (a match that differs only in trailing whitespace " + + "or typographic quotes is accepted). Put several disjoint changes to one file in one call. " + + "Edits that match are applied even when others in the call do not; the result names " + + "the ones that were not. Keep oldText as short as uniqueness allows.", + inputSchema: { + type: "object", + properties: { + path: { type: "string", description: "Path to the file, absolute or relative to the cwd" }, + edits: { + type: "array", + minItems: 1, + items: { + type: "object", + properties: { + oldText: { type: "string", description: "Text to replace" }, + newText: { type: "string", description: "Replacement text" }, + }, + required: ["oldText", "newText"], + additionalProperties: false, + }, + }, + }, + required: ["path", "edits"], + additionalProperties: false, + }, + requiresApproval: false, + execute: async (input) => { + const path = resolvePath(context, requireString(input, "path")); + return withFileLock(path, async () => { + const edits = readEdits(input); + // A file `read` would not load is no file to load here either: a device or + // a pipe has no end to read to (/dev/zero would fill memory), and a huge + // file is not held whole. + const info = await stat(path).catch(() => undefined); + if (info !== undefined && !info.isFile()) { + throw new ToolCallError(`${path} is not a regular file`); + } + if (info !== undefined && info.size > MAX_READ_BYTES) { + throw new ToolCallError( + `${path} is ${info.size} bytes, too large to edit here. Use bash (sed -i, a script) instead.` + ); + } + const bytes = await readFile(path).catch((err: NodeJS.ErrnoException) => { + // Only a missing file is one to create: telling the model to `write` a + // file it merely could not read would have it overwrite the file. + if (err.code === "ENOENT") { + throw new ToolCallError(`File not found: ${path}. Use write to create it.`); + } + throw new ToolCallError(`Could not read ${path}: ${err.message}`); + }); + // Decoding replaces an invalid byte with U+FFFD, and writing the text back + // would replace it in the file too, so a file that is not UTF-8 is not edited. + let original: string; + try { + original = new TextDecoder("utf-8", { fatal: true, ignoreBOM: true }).decode(bytes); + } catch { + throw new ToolCallError( + `${path} is not UTF-8 text, so editing it here would corrupt it. Use bash instead.` + ); + } + // Matching runs on LF text; the file keeps its BOM and its line ending, + // read from its first line break as pi's edit does (a file mixing the two + // comes back in that one). + const bom = original.startsWith("") ? "" : ""; + const firstBreak = original.indexOf("\n"); + const crlf = firstBreak > 0 && original[firstBreak - 1] === "\r"; + const lf = (text: string): string => text.replace(/\r\n/g, "\n"); + const body = lf(bom ? original.slice(1) : original); + const result = applyEdits( + body, + edits.map((edit) => ({ oldText: lf(edit.oldText), newText: lf(edit.newText) })), + path + ); + if (result.applied.length > 0) { + const out = bom + (crlf ? result.content.replace(/\n/g, "\r\n") : result.content); + await writeFile(path, out, "utf8"); + } + const note = + result.fuzzy > 0 ? ` (${result.fuzzy} matched ignoring whitespace/quotes)` : ""; + const same = + result.unchanged.length > 0 + ? `; ${result.unchanged.length} already matched their newText` + : ""; + if (result.failures.length === 0) { + return result.applied.length > 0 + ? `Applied ${result.applied.length} edit(s) to ${path}${note}${same}` + : `No changes to ${path}: every edit's oldText equals its newText.`; + } + const head = + result.applied.length > 0 + ? `Applied ${result.applied.length} of ${edits.length} edits to ${path}${note}${same}; ` + + "the file now contains them. Not applied:" + : `No edits applied to ${path}:`; + throw new ToolCallError( + `${head}\n${result.failures.join("\n")}\n` + + "Retry only these, against the file as it is now." + ); + }); + }, + }; +} diff --git a/examples/agent-eval/src/agent/tools/fileLock.ts b/examples/agent-eval/src/agent/tools/fileLock.ts new file mode 100644 index 0000000000..c1b157e1cd --- /dev/null +++ b/examples/agent-eval/src/agent/tools/fileLock.ts @@ -0,0 +1,45 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { realpath } from "node:fs/promises"; +import { basename, dirname, join } from "node:path"; + +const tails = new Map>(); + +/** The file a path names, so a symlink and its target share one lock. */ +async function lockKey(path: string): Promise { + try { + return await realpath(path); + } catch { + // Not there yet (a write creating it): the file it will be, in its real directory. + try { + return join(await realpath(dirname(path)), basename(path)); + } catch { + return path; + } + } +} + +/** + * Runs `work` once every earlier holder of `path` has finished. With + * `toolConcurrency` above one, two edits of one file would otherwise both read + * it, each apply its own change, and the later write would silently drop the + * other's. + */ +export async function withFileLock(path: string, work: () => Promise): Promise { + const key = await lockKey(path); + const previous = tails.get(key) ?? Promise.resolve(); + let release!: () => void; + const tail = previous.then(() => new Promise((resolve) => (release = resolve))); + tails.set(key, tail); + await previous; + try { + return await work(); + } finally { + release(); + if (tails.get(key) === tail) tails.delete(key); + } +} diff --git a/examples/agent-eval/src/agent/tools/read.ts b/examples/agent-eval/src/agent/tools/read.ts new file mode 100644 index 0000000000..b117950d14 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/read.ts @@ -0,0 +1,197 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { ToolCallError, ToolResultContent } from "@workglow/ai"; +import { opendir, readFile, stat } from "node:fs/promises"; +import type { CodingToolContext } from "./context"; +import { optionalInteger, requireString, resolvePath } from "./context"; +import { + countLines, + lineAt, + lineStart, + MAX_OUTPUT_BYTES, + MAX_OUTPUT_LINES, + truncateHead, +} from "./truncate"; + +/** Bytes sampled to decide a file is binary. */ +const BINARY_SNIFF_BYTES = 8000; + +/** + * Largest image handed to the model. Providers cap an image at about 5 MB and + * bill it on every later round of the turn, so a bigger one is refused with a + * way to shrink it rather than sent to fail. + */ +const MAX_IMAGE_BYTES = 4 * 1024 * 1024; +/** + * Past this a file is not loaded at all: a page is cut to 50KB anyway, but + * finding it means holding the whole file, and a multi-gigabyte log would take + * the agent's memory with it. + */ +export const MAX_READ_BYTES = 64 * 1024 * 1024; + +/** The image formats every vision provider takes, recognised by their leading bytes. */ +export function imageMimeType(buffer: Buffer): string | undefined { + if (buffer.subarray(0, 8).equals(Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))) { + return "image/png"; + } + if (buffer[0] === 0xff && buffer[1] === 0xd8 && buffer[2] === 0xff) return "image/jpeg"; + if (buffer.subarray(0, 6).toString("latin1").startsWith("GIF8")) return "image/gif"; + if ( + buffer.subarray(0, 4).toString("latin1") === "RIFF" && + buffer.subarray(8, 12).toString("latin1") === "WEBP" + ) { + return "image/webp"; + } + return undefined; +} + +function looksBinary(buffer: Buffer): boolean { + const sample = buffer.subarray(0, BINARY_SNIFF_BYTES); + return sample.includes(0); +} + +/** The longest start of `text` that fits `maxBytes` of UTF-8, never splitting a character. */ +function headWithinBytes(text: string, maxBytes: number): string { + let bytes = 0; + let end = 0; + for (const char of text) { + const size = Buffer.byteLength(char, "utf8"); + if (bytes + size > maxBytes) break; + bytes += size; + end += char.length; + } + return text.slice(0, end); +} + +export function createReadTool(context: CodingToolContext): ToolDefinition { + return { + name: "read", + description: + "Read a text file. Output is truncated to 2000 lines or 50KB, whichever comes first; " + + "use offset and limit to page through a longer file. Lines are shown as they are, " + + "without line numbers." + + (context.images + ? " An image (png, jpg, gif, webp) is shown to you as the image itself." + : ""), + inputSchema: { + type: "object", + properties: { + path: { type: "string", description: "Path to the file, absolute or relative to the cwd" }, + offset: { type: "integer", minimum: 1, description: "First line to show (1-based)" }, + limit: { type: "integer", minimum: 1, description: "Maximum lines to show" }, + }, + required: ["path"], + additionalProperties: false, + }, + requiresApproval: false, + execute: async (input) => { + const path = resolvePath(context, requireString(input, "path")); + const offset = optionalInteger(input, "offset") ?? 1; + const limit = optionalInteger(input, "limit"); + // The schema says so, but a tool's arguments reach it unchecked, and a zero + // offset would slice from the end of the file. + if (offset < 1) throw new ToolCallError(`offset must be 1 or more, got ${offset}`); + if (limit !== undefined && limit < 1) { + throw new ToolCallError(`limit must be 1 or more, got ${limit}`); + } + + const info = await stat(path).catch((err: NodeJS.ErrnoException) => { + // Only a missing path is "not found"; anything else (no permission, an + // I/O error) is reported as it is, so the model does not take an + // unreadable file for one it may create. + if (err.code === "ENOENT") throw new ToolCallError(`File not found: ${path}`); + throw new ToolCallError(`Could not read ${path}: ${err.message}`); + }); + if (info.isDirectory()) { + // Kept as they are read, never whole: a directory can hold millions of + // entries and only the first MAX_OUTPUT_LINES are ever shown, so the rest + // are only counted. + const names: string[] = []; + let total = 0; + for await (const entry of await opendir(path)) { + total++; + if (names.length < MAX_OUTPUT_LINES) { + names.push(entry.isDirectory() ? `${entry.name}/` : entry.name); + } + } + names.sort(); + const listing = truncateHead(names.join("\n")); + // Past the cap the names kept are the first the directory returned, not + // the first by name, and the model is told so. + const order = total > names.length ? ", in directory order rather than by name" : ""; + const more = + listing.keptLines < total + ? `\n\n[Showing ${listing.keptLines} of ${total} entries${order}. Use bash (ls, find) to see the rest.]` + : ""; + return `${path} is a directory:\n${listing.text}${more}`; + } + // A device or a pipe has no end to read to (/dev/zero would fill memory). + if (!info.isFile()) throw new ToolCallError(`${path} is not a regular file`); + if (info.size > MAX_READ_BYTES) { + throw new ToolCallError( + `${path} is ${info.size} bytes, too large to read whole. ` + + "Use bash to look at part of it (head, tail, sed -n 'X,Yp', grep)." + ); + } + + const buffer = await readFile(path); + const mimeType = context.images ? imageMimeType(buffer) : undefined; + if (mimeType !== undefined) { + if (buffer.length > MAX_IMAGE_BYTES) { + throw new ToolCallError( + `${path} is a ${buffer.length}-byte image, over the ${MAX_IMAGE_BYTES}-byte limit. ` + + "Shrink it first (e.g. with Python's PIL) and read the smaller copy." + ); + } + return new ToolResultContent([ + { type: "text", text: `Image: ${path} (${mimeType}, ${buffer.length} bytes)` }, + { type: "image", mimeType, data: buffer.toString("base64") }, + ]); + } + if (looksBinary(buffer)) { + throw new ToolCallError(`${path} is a binary file (${buffer.length} bytes)`); + } + const text = buffer.toString("utf8"); + // The window is cut out of the text by line position rather than by + // splitting the file: a file of short lines can hold millions of them. + const totalLines = countLines(text); + if (offset > totalLines) { + throw new ToolCallError( + `offset ${offset} is past the end of ${path} (${totalLines} lines)` + ); + } + const after = limit === undefined ? totalLines + 1 : Math.min(offset + limit, totalLines + 1); + const from = lineStart(text, offset); + const to = + after > totalLines + ? text.endsWith("\n") + ? text.length - 1 + : text.length + : lineStart(text, after) - 1; + const shown = truncateHead(text.slice(from, to)); + + if (shown.keptLines === 0) { + // One line larger than the whole budget: show its start and say how to see the rest. + const first = lineAt(text, offset); + const head = headWithinBytes(first, MAX_OUTPUT_BYTES); + return ( + `${head}\n\n[Line ${offset} is ${first.length} characters; showing the first ` + + `${head.length}. Use bash (e.g. cut -c or head -c) to read the rest.]` + ); + } + const last = offset + shown.keptLines - 1; + if (shown.truncated || (limit !== undefined && last < totalLines)) { + return ( + `${shown.text}\n\n[Showing lines ${offset}-${last} of ${totalLines}. ` + + `Use offset=${last + 1} to continue.]` + ); + } + return shown.text; + }, + }; +} diff --git a/examples/agent-eval/src/agent/tools/truncate.ts b/examples/agent-eval/src/agent/tools/truncate.ts new file mode 100644 index 0000000000..b51e6bb4c8 --- /dev/null +++ b/examples/agent-eval/src/agent/tools/truncate.ts @@ -0,0 +1,150 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * Output limits shared by every tool. They are pi's and opencode's numbers on + * purpose: the eval compares loops, and a tool that showed the model five + * times more output than the other harnesses' would be measuring the tool. + */ +export const MAX_OUTPUT_LINES = 2000; +export const MAX_OUTPUT_BYTES = 50 * 1024; + +export interface Truncated { + readonly text: string; + readonly truncated: boolean; + /** Lines in the input, before truncation. */ + readonly totalLines: number; + /** Lines kept. */ + readonly keptLines: number; + /** The first kept line is only the end of a line too long for the budget. */ + readonly partialLine: boolean; +} + +// Lines are walked with indexOf rather than split: a capped output or an allowed +// file of short lines can hold millions of them, and an array of every line is +// many times the text's own size, while only the first or last few are kept. + +/** Where the text's lines end: a final newline ends the last line rather than starting another. */ +function bodyEnd(text: string): number { + return text.endsWith("\n") ? text.length - 1 : text.length; +} + +/** The number of lines in `text`, counted without materializing them. */ +export function countLines(text: string): number { + const end = bodyEnd(text); + let lines = 1; + for (let i = text.indexOf("\n"); i !== -1 && i < end; i = text.indexOf("\n", i + 1)) lines++; + return lines; +} + +/** The index where 1-based line `n` starts; `n` must be at most {@link countLines}. */ +export function lineStart(text: string, n: number): number { + let at = 0; + for (let line = 1; line < n; line++) at = text.indexOf("\n", at) + 1; + return at; +} + +/** Line `n` (1-based) of `text`, without its newline. */ +export function lineAt(text: string, n: number): string { + const start = lineStart(text, n); + const next = text.indexOf("\n", start); + return text.slice(start, next === -1 || next > bodyEnd(text) ? bodyEnd(text) : next); +} + +function byteLength(text: string): number { + return Buffer.byteLength(text, "utf8"); +} + +/** + * The first lines of `text` that fit both limits. A read wants the head: the + * model continues from where it stopped with `offset`. + */ +export function truncateHead( + text: string, + maxLines: number = MAX_OUTPUT_LINES, + maxBytes: number = MAX_OUTPUT_BYTES +): Truncated { + const end = bodyEnd(text); + const totalLines = countLines(text); + const kept: string[] = []; + let bytes = 0; + for (let at = 0; kept.length < totalLines;) { + const next = text.indexOf("\n", at); + const lineEnd = next === -1 || next > end ? end : next; + const line = text.slice(at, lineEnd); + // Each line costs its newline, except a last line the text does not end with. + const size = byteLength(line) + (lineEnd === end && !text.endsWith("\n") ? 0 : 1); + if (kept.length >= maxLines || bytes + size > maxBytes) break; + kept.push(line); + bytes += size; + at = lineEnd + 1; + } + const truncated = kept.length < totalLines; + return { + // The final newline stays when the last line is shown. + text: kept.join("\n") + (!truncated && text.endsWith("\n") ? "\n" : ""), + truncated, + totalLines, + keptLines: kept.length, + partialLine: false, + }; +} + +/** The end of `line` within `maxBytes` of UTF-8, cut on a character boundary. */ +function tailWithinBytes(line: string, maxBytes: number): string { + if (maxBytes <= 0) return ""; + const buffer = Buffer.from(line, "utf8"); + if (buffer.length <= maxBytes) return line; + let start = buffer.length - maxBytes; + // A UTF-8 continuation byte is 10xxxxxx; start on the next character instead. + while (start < buffer.length && (buffer[start]! & 0xc0) === 0x80) start++; + return buffer.subarray(start).toString("utf8"); +} + +/** + * The last lines of `text` that fit both limits. A command wants the tail: + * the error, the test summary and the exit status are at the end. A last line + * longer than the whole budget (minified JSON, a progress bar redrawn with + * carriage returns) keeps its end, rather than leaving nothing to show. + */ +export function truncateTail( + text: string, + maxLines: number = MAX_OUTPUT_LINES, + maxBytes: number = MAX_OUTPUT_BYTES +): Truncated { + const totalLines = countLines(text); + const kept: string[] = []; + let bytes = 0; + let partial = false; + // Walked from the end: `end` is where the current line stops. + for (let end = bodyEnd(text); kept.length < totalLines;) { + if (kept.length >= maxLines) break; + const previous = end === 0 ? -1 : text.lastIndexOf("\n", end - 1); + const line = text.slice(previous + 1, end); + // Each line costs its newline, except a last line the text does not end with. + const newline = kept.length === 0 && !text.endsWith("\n") ? 0 : 1; + const size = byteLength(line) + newline; + if (bytes + size > maxBytes) { + if (kept.every((shown) => shown.length === 0)) { + kept.push(tailWithinBytes(line, maxBytes - bytes - newline)); + partial = true; + } + break; + } + kept.push(line); + bytes += size; + end = previous; + } + kept.reverse(); + return { + // The tail always ends on the last line, so its final newline stays. + text: kept.join("\n") + (kept.length > 0 && text.endsWith("\n") ? "\n" : ""), + truncated: partial || kept.length < totalLines, + totalLines, + keptLines: kept.length, + partialLine: partial, + }; +} diff --git a/examples/agent-eval/src/agent/tools/write.ts b/examples/agent-eval/src/agent/tools/write.ts new file mode 100644 index 0000000000..c391ce8a2d --- /dev/null +++ b/examples/agent-eval/src/agent/tools/write.ts @@ -0,0 +1,50 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolDefinition } from "@workglow/ai"; +import { ToolCallError } from "@workglow/ai"; +import { mkdir, stat, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; +import type { CodingToolContext } from "./context"; +import { requireString, resolvePath } from "./context"; +import { withFileLock } from "./fileLock"; + +export function createWriteTool(context: CodingToolContext): ToolDefinition { + return { + name: "write", + description: + "Create a file or overwrite it with new content, creating parent directories as needed. " + + "Use edit for changes to an existing file; use write for new files or complete rewrites.", + inputSchema: { + type: "object", + properties: { + path: { type: "string", description: "Path to the file, absolute or relative to the cwd" }, + content: { type: "string", description: "The complete file content" }, + }, + required: ["path", "content"], + additionalProperties: false, + }, + requiresApproval: false, + execute: async (input) => { + const path = resolvePath(context, requireString(input, "path")); + const content = requireString(input, "content"); + await mkdir(dirname(path), { recursive: true }); + await withFileLock(path, async () => { + // Opening a FIFO for writing blocks until something reads it, and a device + // is not a file to overwrite; only a regular file, or none yet, is written. + const existing = await stat(path).catch((error: NodeJS.ErrnoException) => { + if (error.code === "ENOENT") return undefined; + throw error; + }); + if (existing && !existing.isFile()) { + throw new ToolCallError(`${path} is not a regular file; write only writes files.`); + } + await writeFile(path, content, "utf8"); + }); + return `Wrote ${Buffer.byteLength(content, "utf8")} bytes to ${path}`; + }, + }; +} diff --git a/examples/agent-eval/src/bin/workglow-agent-eval.ts b/examples/agent-eval/src/bin/workglow-agent-eval.ts new file mode 100644 index 0000000000..f265b19ee8 --- /dev/null +++ b/examples/agent-eval/src/bin/workglow-agent-eval.ts @@ -0,0 +1,275 @@ +#!/usr/bin/env bun + +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { existsSync, writeFileSync } from "node:fs"; +import { resolve } from "node:path"; +import { Command, Option } from "commander"; +import { loadTrials } from "../compare/harborJobs"; +import { buildReport, renderMarkdown } from "../compare/report"; +import { startAnthropicMock } from "../mock/anthropicMock"; +import type { ArmSpec, Effort } from "../run/jobConfig"; +import { buildHarborJob, EFFORTS, HARNESSES, parseArm } from "../run/jobConfig"; +import { runHarbor } from "../run/runHarbor"; +import type { JobDataset } from "../run/suites"; +import { getSuite, narrowDataset, SUITES } from "../run/suites"; + +function positiveInteger(value: string): number { + const n = Number(value); + if (!Number.isInteger(n) || n < 1) throw new Error(`expected a positive integer, got ${value}`); + return n; +} + +function positiveNumber(value: string): number { + const n = Number(value); + if (!Number.isFinite(n) || n <= 0) throw new Error(`expected a positive number, got ${value}`); + return n; +} + +function collect(value: string, previous: string[] = []): string[] { + return [...previous, value]; +} + +function timestamp(): string { + return new Date().toISOString().replace(/[:.]/g, "-").slice(0, 19); +} + +function writeReport( + paths: readonly string[], + format: string, + out: string | undefined, + allTasks: boolean +): void { + const trials = loadTrials(paths); + if (trials.length === 0) throw new Error(`no Harbor trial results under ${paths.join(", ")}`); + const report = buildReport(trials); + const text = + format === "json" + ? `${JSON.stringify(report, null, 2)}\n` + : renderMarkdown(report, { allTasks }); + if (out) { + writeFileSync(out, text); + process.stderr.write(`wrote ${out}\n`); + } else process.stdout.write(text); +} + +const program = new Command() + .name("workglow-agent-eval") + .version("0.1.0") + .description( + "Compare Workglow's AgentTask with opencode and pi on public agentic benchmarks, through Harbor" + ); + +program + .command("suites") + .description("List the named benchmark suites") + .action(() => { + for (const [name, suite] of Object.entries(SUITES)) { + const datasets = suite.datasets + .map( + (d) => + `${d.name}@${d.version}${d.task_names ? ` ${d.task_names.join(" ")}` : ""}${d.n_tasks ? ` (first ${d.n_tasks})` : ""}` + ) + .join(" + "); + process.stdout.write( + `${name.padEnd(16)} ${suite.description}\n${"".padEnd(16)} ${datasets}\n` + ); + } + }); + +program + .command("run") + .description("Run every arm on a suite in one Harbor job, then write the comparison") + .requiredOption( + "-m, --model ", + "the model every arm uses, e.g. anthropic/claude-sonnet-5-5" + ) + .option("-s, --suite ", "benchmark suite (see `suites`)", "smoke") + .option( + "-d, --dataset ", + "run this registry dataset instead of a suite (repeatable)", + collect + ) + .option( + "--tasks-dir ", + "run the Harbor task folders in this directory instead of a suite (repeatable)", + collect + ) + .option( + "-a, --agents ", + `comma-separated harnesses: ${HARNESSES.join(",")}`, + HARNESSES.join(",") + ) + .option( + "--arm ", + "an extra arm, e.g. workglow:tool_concurrency=8 (repeatable; agent kwargs as key=value)", + collect + ) + .addOption( + new Option("--effort ", "reasoning level for every arm") + .choices([...EFFORTS]) + .default("high") + ) + .option( + "--opencode-variant ", + "opencode's name for that reasoning level (default: the --effort level)" + ) + .option("--max-turns ", "model calls each arm may make", positiveInteger, 100) + .option("-k, --attempts ", "attempts per task per arm", positiveInteger, 1) + .option("-n, --concurrency ", "trials run at once", positiveInteger, 4) + .option("--tasks ", "only tasks matching these comma-separated globs") + .option("-l, --n-tasks ", "at most this many tasks per dataset", positiveInteger) + .option( + "--timeout-multiplier ", + "scale every task's agent and verifier timeouts", + positiveNumber + ) + .option( + "--pi-model-api ", + "pi's wire API when the provider base URL is overridden (e.g. anthropic-messages)" + ) + .option( + "--harbor-config ", + "extra Harbor config layered on the generated one (repeatable)", + collect + ) + .option("--jobs-dir ", "where Harbor writes results", "./jobs") + .option("--job-name ", "Harbor job name (default -)") + .option("--harbor ", "the harbor executable", "harbor") + .option("--dry-run", "print the Harbor job config and exit") + .action(async (opts) => { + const custom = opts.dataset !== undefined || opts.tasksDir !== undefined; + const suite = custom ? undefined : getSuite(opts.suite); + const suiteDatasets: JobDataset[] = suite + ? [...suite.datasets] + : [ + ...((opts.dataset as string[] | undefined) ?? []).map((spec) => { + const at = spec.lastIndexOf("@"); + if (at <= 0) throw new Error(`dataset must be name@version, got "${spec}"`); + return { name: spec.slice(0, at), version: spec.slice(at + 1) }; + }), + ...((opts.tasksDir as string[] | undefined) ?? []).map((dir) => ({ + path: resolve(dir), + })), + ]; + const arms: ArmSpec[] = [ + ...String(opts.agents) + .split(",") + .map((name) => name.trim()) + .filter((name) => name.length > 0) + .map(parseArm), + ...((opts.arm as string[] | undefined) ?? []).map(parseArm), + ]; + if (arms.length === 0) throw new Error("no arms to run"); + const taskFilter = opts.tasks + ? String(opts.tasks) + .split(",") + .map((glob) => glob.trim()) + : undefined; + // A filter may name tasks of only some of a suite's datasets; the rest drop out. + const datasets = suiteDatasets + .map((dataset) => narrowDataset(dataset, taskFilter, opts.nTasks as number | undefined)) + .filter((dataset): dataset is JobDataset => dataset !== undefined); + if (datasets.length === 0) { + throw new Error(`--tasks ${String(opts.tasks)} matches none of the tasks this suite runs`); + } + const piModelApi = + opts.piModelApi ?? + (String(opts.model).startsWith("anthropic/") && process.env.ANTHROPIC_BASE_URL + ? "anthropic-messages" + : undefined); + const jobName = opts.jobName ?? `${suite ? opts.suite : "custom"}-${timestamp()}`; + const config = buildHarborJob({ + jobName, + jobsDir: opts.jobsDir, + model: opts.model, + arms, + datasets, + parity: { + maxTurns: opts.maxTurns, + effort: opts.effort as Effort, + // opencode names its levels after the effort itself, so the arms run at + // the same one unless the caller names a variant of their own. + opencodeVariant: opts.opencodeVariant || (opts.effort === "off" ? undefined : opts.effort), + piModelApi, + }, + attempts: opts.attempts, + concurrency: opts.concurrency, + timeoutMultiplier: opts.timeoutMultiplier, + }); + if (opts.dryRun) { + process.stdout.write(`${JSON.stringify(config, null, 2)}\n`); + return; + } + // Results already there would be read back as this run's if Harbor then + // wrote none of its own. + const existing = resolve(String(opts.jobsDir), jobName); + if (existsSync(existing)) { + throw new Error(`${existing} already exists; pass another --job-name or move it away`); + } + const { jobDir, exitCode } = await runHarbor({ + config, + extraConfigs: (opts.harborConfig as string[] | undefined) ?? [], + harbor: opts.harbor, + }); + if (!existsSync(jobDir)) throw new Error(`harbor exited with ${exitCode} and wrote no results`); + if (exitCode !== 0) + process.stderr.write(`harbor exited with ${exitCode}; comparing what it wrote\n`); + const trials = loadTrials([jobDir]); + if (trials.length === 0) + throw new Error(`harbor exited with ${exitCode} and wrote no trial results to ${jobDir}`); + const report = buildReport(trials); + const markdown = renderMarkdown(report); + writeFileSync(`${jobDir}/comparison.md`, markdown); + writeFileSync(`${jobDir}/comparison.json`, `${JSON.stringify(report, null, 2)}\n`); + process.stdout.write(`\n${markdown}\nwrote ${jobDir}/comparison.md and comparison.json\n`); + // The partial comparison is still worth having, but a run Harbor failed is + // not a run that succeeded. + if (exitCode !== 0) process.exitCode = exitCode; + }); + +program + .command("compare") + .description("Compare the trials in one or more Harbor job directories") + .argument("", "Harbor job directories, or a jobs directory holding several") + .addOption( + new Option("-f, --format ", "output format").choices(["md", "json"]).default("md") + ) + .option("-o, --out ", "write to a file instead of stdout") + .option("--all-tasks", "list every task, not only those where the arms differ") + .action((paths: string[], opts) => writeReport(paths, opts.format, opts.out, !!opts.allTasks)); + +program + .command("mock-model") + .description("Serve a scripted Anthropic Messages API for an offline end-to-end check") + .option("-p, --port ", "port", positiveInteger, 18089) + .option("--host ", "interface to listen on", "127.0.0.1") + .option("-c, --command ", "shell command the model runs, in order (repeatable)", collect) + .option("--request-log ", "append every request body to this JSONL file") + .action(async (opts) => { + const commands = (opts.command as string[] | undefined) ?? ["echo 'Hello, world!' > hello.txt"]; + await startAnthropicMock({ + port: opts.port, + host: opts.host, + commands, + requestLog: opts.requestLog, + }); + process.stderr.write( + `mock Anthropic API on http://${opts.host}:${opts.port} running ${commands.length} command(s)\n` + + `point the arms at it with ANTHROPIC_BASE_URL=http://${opts.host}:${opts.port} ANTHROPIC_API_KEY=mock\n` + ); + await new Promise(() => {}); + }); + +try { + await program.parseAsync(process.argv); +} catch (error) { + process.stderr.write( + `workglow-agent-eval: ${error instanceof Error ? error.message : String(error)}\n` + ); + process.exitCode = 1; +} diff --git a/examples/agent-eval/src/bin/workglow-agent.ts b/examples/agent-eval/src/bin/workglow-agent.ts new file mode 100644 index 0000000000..9d56dd1df6 --- /dev/null +++ b/examples/agent-eval/src/bin/workglow-agent.ts @@ -0,0 +1,243 @@ +#!/usr/bin/env node + +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * The headless Workglow coding agent: AgentTask plus the four coding tools, + * run once on one instruction. It is what the Harbor adapter installs into a + * task container — the counterpart of `opencode run` and `pi --print` — so it + * is bundled into one file with every dependency and needs only Node. + */ + +import { appendFileSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { Command } from "commander"; +import { toAtifTrajectory } from "../agent/atif"; +import { loadCredentialsFile } from "../agent/credentialsFile"; +import { parseEffort, resolveAgentModel } from "../agent/models"; +import { registerAgentProvider } from "../agent/providers"; +import type { AgentRunEvent } from "../agent/runCodingAgent"; +import { + DEFAULT_BENCH_MAX_ROUNDS, + DEFAULT_BENCH_MAX_TOOL_RESULT_CHARS, + runCodingAgent, +} from "../agent/runCodingAgent"; +import { REVIEW_PROMPT } from "../agent/systemPrompt"; +import { MAX_TIMEOUT_SEC } from "../agent/tools/bash"; + +const VERSION = "0.1.0"; + +function integer(value: string): number { + const n = Number(value); + if (!Number.isInteger(n) || n < 1) throw new Error(`expected a positive integer, got ${value}`); + return n; +} + +/** `on`/`off` switches, spelled so Harbor agent kwargs pass them through as strings. */ +function onOff(value: string): boolean { + if (value === "on" || value === "true") return true; + if (value === "off" || value === "false") return false; + throw new Error(`expected on or off, got ${value}`); +} + +/** Seconds for a timer: past MAX_TIMEOUT_SEC Node would fire it at once. */ +function timerSeconds(value: string): number { + const n = integer(value); + if (n > MAX_TIMEOUT_SEC) + throw new Error(`expected at most ${MAX_TIMEOUT_SEC} seconds, got ${value}`); + return n; +} + +function nonNegativeInteger(value: string): number { + const n = Number(value); + if (!Number.isInteger(n) || n < 0) { + throw new Error(`expected a non-negative integer, got ${value}`); + } + return n; +} + +function number(value: string): number { + const n = Number(value); + if (!Number.isFinite(n)) throw new Error(`expected a number, got ${value}`); + return n; +} + +function describe(event: AgentRunEvent): string | undefined { + switch (event.type) { + case "start": + return `workglow-agent ${VERSION} · ${event.model} · ${event.cwd}`; + case "tool-call": + return event.status === "running" ? ` → ${event.name ?? "?"}` : undefined; + case "round": { + const usage = event.step.usage; + const tokens = usage + ? ` · ${usage.input ?? 0}+${usage.cached ?? 0} in / ${usage.output ?? 0} out` + : ""; + return `round ${event.step.round} · ${event.step.tools.length} tool(s)${tokens}`; + } + case "finish": + return `finished: ${event.summary.outcome}${event.summary.error ? ` (${event.summary.error})` : ""}`; + } +} + +const program = new Command() + .name("workglow-agent") + .version(VERSION) + .description("Run Workglow's AgentTask with coding tools on one instruction") + .argument("[instruction...]", "the task; or pass --instruction-file") + .requiredOption("-m, --model ", "model, e.g. anthropic/claude-sonnet-4-5") + .option("--instruction-file ", "read the instruction from a file") + .option( + "--credentials-file ", + "JSON object of environment variables (API keys) to load, then delete" + ) + .option("--cwd ", "working directory", process.cwd()) + .option("--logs-dir ", "where the events, summary and trajectory are written") + .option("--effort ", "reasoning effort: none|low|medium|high|extra|ultra") + .option("--max-rounds ", "model calls before the run stops", integer, DEFAULT_BENCH_MAX_ROUNDS) + .option( + "--max-tool-result-chars ", + "characters of one tool result the model is shown", + integer, + DEFAULT_BENCH_MAX_TOOL_RESULT_CHARS + ) + .option( + "--max-history-chars ", + "history budget sent per round (AgentTask default if unset)", + integer + ) + .option("--tool-concurrency ", "tool calls of one round run at once", integer) + .option("--max-tokens ", "output tokens per model call", integer) + .option("--temperature ", "sampling temperature", number) + .option("--max-duration-sec ", "wall-clock budget; no round starts after it", integer) + .option( + "--stop-after-sec ", + "stop outright after this long, even mid-call, and still write the logs", + timerSeconds + ) + .option("--round-timeout-sec ", "abandon and retry a model call after this long", timerSeconds) + .option("--max-round-retries ", "retries of a failed model call", nonNegativeInteger) + .option("--command-timeout-sec ", "default bash timeout", timerSeconds, 120) + .option("--append-system-prompt ", "text appended to the system prompt") + .option("--concise ", "ask for terse replies between tool calls", onOff, true) + .option("--images ", "let read show images to the model", onOff, true) + .option( + "--replay-reasoning ", + "send the model's reasoning back on later rounds (providers that take it)", + onOff, + true + ) + .option( + "--time-left ", + "show the model its remaining time each round (needs --max-duration-sec)", + onOff, + true + ) + .option("--review ", "ask the model to check its work once before finishing", onOff, true) + .action(async (words: string[], opts) => { + // First, before anything can start a child process. + if (opts.credentialsFile) loadCredentialsFile(opts.credentialsFile); + const instruction = opts.instructionFile + ? readFileSync(opts.instructionFile, "utf8") + : words.join(" "); + if (instruction.trim() === "") throw new Error("no instruction given"); + + const resolved = resolveAgentModel(opts.model, { effort: parseEffort(opts.effort) }); + // An ablation switch for the provider's reasoning replay; DeepSeek and OpenAI read it, + // and Anthropic has no such switch. + const model = opts.replayReasoning + ? resolved + : ({ + ...resolved, + provider_config: { ...resolved.provider_config, replay_reasoning: false }, + } as typeof resolved); + await registerAgentProvider(model.provider); + + const cwd = resolve(opts.cwd); + const logsDir = opts.logsDir ? resolve(opts.logsDir) : undefined; + if (logsDir) mkdirSync(logsDir, { recursive: true }); + const eventsPath = logsDir ? join(logsDir, "workglow-events.jsonl") : undefined; + // One run per log, like the summary and trajectory beside it. + if (eventsPath) writeFileSync(eventsPath, ""); + + const controller = new AbortController(); + for (const signal of ["SIGINT", "SIGTERM"] as const) { + process.once(signal, () => controller.abort(new Error(`received ${signal}`))); + } + + const result = await runCodingAgent({ + instruction, + modelId: opts.model, + model, + tools: { + cwd, + spillDir: join(logsDir ?? tmpdir(), "spill"), + defaultCommandTimeoutSec: opts.commandTimeoutSec, + images: opts.images, + }, + settings: { + maxRounds: opts.maxRounds, + maxToolResultChars: opts.maxToolResultChars, + maxHistoryChars: opts.maxHistoryChars, + toolConcurrency: opts.toolConcurrency, + maxTokens: opts.maxTokens, + temperature: opts.temperature, + maxDurationMs: opts.maxDurationSec === undefined ? undefined : opts.maxDurationSec * 1000, + roundTimeoutMs: + opts.roundTimeoutSec === undefined ? undefined : opts.roundTimeoutSec * 1000, + maxRoundRetries: opts.maxRoundRetries, + systemPromptAppend: opts.appendSystemPrompt, + concise: opts.concise, + announceTimeLeft: opts.timeLeft, + reviewPrompt: opts.review ? REVIEW_PROMPT : undefined, + }, + signal: controller.signal, + stopAfterMs: opts.stopAfterSec === undefined ? undefined : opts.stopAfterSec * 1000, + onEvent: (event) => { + const line = describe(event); + if (line) process.stderr.write(`${line}\n`); + if (eventsPath) appendFileSync(eventsPath, `${JSON.stringify(event)}\n`); + }, + }); + + if (logsDir) { + writeFileSync( + join(logsDir, "workglow-summary.json"), + JSON.stringify(result.summary, null, 2) + ); + const trajectory = toAtifTrajectory( + `workglow-${Date.now()}`, + VERSION, + result.messages, + result.steps, + result.summary + ); + writeFileSync(join(logsDir, "trajectory.json"), JSON.stringify(trajectory, null, 2)); + } + if (result.summary.finalText) process.stdout.write(`${result.summary.finalText}\n`); + process.exitCode = + result.summary.outcome === "error" || result.summary.outcome === "aborted" ? 1 : 0; + }); + +try { + await program.parseAsync(process.argv); +} catch (error) { + process.stderr.write( + `workglow-agent: ${error instanceof Error ? error.message : String(error)}\n` + ); + process.exitCode = 2; +} +// Provider SDKs can hold sockets open, so the process is ended outright — once +// what was written has drained, since a write to a pipe can still be queued +// where pipes are asynchronous (macOS), and exiting would cut it off. +await Promise.all( + [process.stdout, process.stderr].map( + (stream) => new Promise((resolve) => stream.write("", () => resolve())) + ) +); +process.exit(process.exitCode ?? 0); diff --git a/examples/agent-eval/src/compare/harborJobs.ts b/examples/agent-eval/src/compare/harborJobs.ts new file mode 100644 index 0000000000..7a420500d3 --- /dev/null +++ b/examples/agent-eval/src/compare/harborJobs.ts @@ -0,0 +1,377 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { existsSync, readdirSync, readFileSync, statSync } from "node:fs"; +import { basename, dirname, join } from "node:path"; + +/** + * How a trial ended, from the comparison's point of view. + * + * `blocked` is the one that never reaches a pass rate: the environment or the + * agent's install failed before the agent saw the task, so the trial measured + * the infrastructure. Counting it as a failure would charge a harness for a + * flaky registry; dropping it silently would shrink one arm's denominator. + * It is excluded and reported. + */ +export type TrialStatus = "pass" | "fail" | "blocked"; + +/** Why a trial that ran did not pass, coarse enough to tally across harnesses. */ +export type FailureMode = + | "wrong-answer" + | "agent-timeout" + | "context-overflow" + | "max-rounds" + | "budget" + | "rate-limit" + | "provider-error" + | "agent-crash" + | "verifier-error"; + +export interface TrialRecord { + readonly job: string; + readonly trial: string; + readonly task: string; + /** + * The comparison column: the agent, plus whatever tells this configuration + * from the agent's others in the same comparison — e.g. + * `workglow[tool_concurrency=8]`, or `pi@openai/gpt-5.5` when models differ. + */ + readonly arm: string; + /** The agent kwargs this trial ran with. */ + readonly kwargs: Readonly>; + readonly agent: string; + readonly agentVersion: string | undefined; + readonly model: string | undefined; + readonly reward: number | undefined; + readonly status: TrialStatus; + readonly failure: FailureMode | undefined; + readonly exceptionType: string | undefined; + readonly exceptionMessage: string | undefined; + readonly inputTokens: number | undefined; + readonly cacheTokens: number | undefined; + readonly outputTokens: number | undefined; + readonly costUsd: number | undefined; + readonly agentSec: number | undefined; + readonly setupSec: number | undefined; + /** The workglow summary the adapter stores on the trial, when this arm is workglow. */ + readonly workglow: WorkglowTrialMetadata | undefined; +} + +export interface WorkglowTrialMetadata { + readonly outcome: string; + readonly error?: string | undefined; + readonly rounds: number; + readonly retries: number; + readonly tools: { + readonly calls: number; + readonly errors: number; + readonly byName: Readonly>; + }; + readonly finalHistoryChars: number; +} + +interface HarborTrialJson { + readonly trial_name?: string; + readonly task_name?: string; + readonly config?: { + readonly agent?: { + name?: string; + import_path?: string; + model_name?: string | null; + kwargs?: Record; + }; + }; + readonly agent_info?: { + readonly name?: string; + readonly version?: string; + readonly model_info?: { readonly name?: string; readonly provider?: string } | null; + }; + readonly agent_result?: { + readonly n_input_tokens?: number | null; + readonly n_cache_tokens?: number | null; + readonly n_output_tokens?: number | null; + readonly cost_usd?: number | null; + readonly metadata?: { readonly workglow?: WorkglowTrialMetadata } | null; + } | null; + readonly verifier_result?: { readonly rewards?: Record | null } | null; + readonly exception_info?: { + readonly exception_type?: string; + readonly exception_message?: string; + } | null; + readonly agent_setup?: Span | null; + readonly agent_execution?: Span | null; + readonly environment_setup?: Span | null; +} + +interface Span { + readonly started_at?: string | null; + readonly finished_at?: string | null; +} + +function seconds(span: Span | null | undefined): number | undefined { + if (!span?.started_at || !span.finished_at) return undefined; + const ms = Date.parse(span.finished_at) - Date.parse(span.started_at); + return Number.isFinite(ms) ? ms / 1000 : undefined; +} + +/** + * A kwarg value as a label shows it. A string goes bare only when it can neither + * split the label (a comma, `=`, a bracket) nor read as another value (`8`, + * `true`): two settings must never share a label, or they are counted as one arm. + */ +function kwargValue(value: unknown): string { + if (typeof value !== "string") return JSON.stringify(value); + if (/[,=[\]"]/.test(value)) return JSON.stringify(value); + try { + JSON.parse(value); + return JSON.stringify(value); + } catch { + return value; + } +} + +/** + * Names each trial's arm. A kwarg appears in a label only when the agent ran + * with more than one value for it in this comparison: the parity settings every + * arm carries would otherwise make every label unreadable, and an ablation's + * one changed setting is exactly what should show. That includes `bundle_path` + * and `version`: they choose the build that ran, so two of them are two agents. + */ +export function labelArms(trials: readonly TrialRecord[]): TrialRecord[] { + // A trial whose model is unknown joins its agent's arm when that agent ran one + // model; left unlabelled it would be an arm of its own, and a blocked trial + // there is missing from the arm it belongs to. + const agentModels = new Map>(); + for (const trial of trials) { + if (trial.model === undefined) continue; + const known = agentModels.get(trial.agent) ?? new Set(); + known.add(trial.model); + agentModels.set(trial.agent, known); + } + const modelOf = (trial: TrialRecord): string | undefined => { + if (trial.model !== undefined) return trial.model; + const known = agentModels.get(trial.agent); + return known?.size === 1 ? [...known][0] : undefined; + }; + const models = new Set(trials.map(modelOf)); + const varying = new Map>(); + const byAgent = new Map(); + for (const trial of trials) { + const list = byAgent.get(trial.agent) ?? []; + list.push(trial); + byAgent.set(trial.agent, list); + } + for (const [agent, list] of byAgent) { + const keys = new Set(list.flatMap((trial) => Object.keys(trial.kwargs))); + const differ = new Set(); + for (const key of keys) { + const values = new Set(list.map((trial) => kwargValue(trial.kwargs[key]))); + if (values.size > 1) differ.add(key); + } + varying.set(agent, differ); + } + return trials.map((trial) => { + const keys = [...(varying.get(trial.agent) ?? [])].sort(); + const parts = keys + .filter((key) => trial.kwargs[key] !== undefined) + .map((key) => `${key}=${kwargValue(trial.kwargs[key])}`); + const resolved = modelOf(trial); + const model = models.size > 1 && resolved ? `@${resolved}` : ""; + return { + ...trial, + arm: `${trial.agent}${model}${parts.length > 0 ? `[${parts.join(",")}]` : ""}`, + }; + }); +} + +const OVERFLOW = + /context(?:[ _](?:length|window))|maximum context|prompt is too long|too many tokens|exceeds the (?:context|maximum)|input token count exceeds/i; +const RATE = /rate.?limit|429|too many requests|overloaded|quota/i; +/** A model API that failed rather than the agent: a 5xx, an API error, a dropped connection. */ +const PROVIDER = + /\b5\d\d\b|api ?error|internal server error|service unavailable|bad gateway|gateway timeout|ECONNRESET|ETIMEDOUT|ENOTFOUND|ECONNREFUSED|fetch failed|socket hang up/i; + +/** + * Classifies a trial. A reward is believed whenever the verifier produced + * one, even after an agent exception: an agent that timed out after writing + * the right answer did solve the task, which is how Harbor scores it too. + */ +export function classifyTrial( + reward: number | undefined, + exceptionType: string | undefined, + exceptionMessage: string | undefined, + agentRan: boolean, + workglow: WorkglowTrialMetadata | undefined +): { status: TrialStatus; failure: FailureMode | undefined } { + if (reward !== undefined && reward >= 1 - 1e-9) return { status: "pass", failure: undefined }; + if (!agentRan && exceptionType !== undefined) return { status: "blocked", failure: undefined }; + + const text = `${exceptionType ?? ""} ${exceptionMessage ?? ""} ${workglow?.error ?? ""}`; + let failure: FailureMode; + if (exceptionType === "AgentTimeoutError") failure = "agent-timeout"; + // Ahead of the API-error branch: a provider reports an over-long prompt as an + // API error too, and that is the overflow it describes. A rate-limit message + // ("too many tokens per minute") is not one. + else if ( + exceptionType === "ContextWindowExceededError" || + (OVERFLOW.test(text) && !RATE.test(text)) + ) + failure = "context-overflow"; + else if (workglow?.outcome === "max-rounds") failure = "max-rounds"; + else if (workglow?.outcome === "budget") failure = "budget"; + else if (exceptionType !== undefined && /RateLimit|UsageLimit|Overloaded/.test(exceptionType)) + failure = "rate-limit"; + else if ( + exceptionType !== undefined && + /Api|Network|Authentication|ModelNotFound/.test(exceptionType) + ) + failure = RATE.test(text) ? "rate-limit" : "provider-error"; + else if (exceptionType !== undefined && /Verifier|RewardFile|AddTests/.test(exceptionType)) + failure = "verifier-error"; + // The Workglow agent exits non-zero on a provider failure, which Harbor reports + // as a generic exit-code error; its own recorded error says what failed. + else if (workglow?.error !== undefined && RATE.test(workglow.error)) failure = "rate-limit"; + else if (workglow?.error !== undefined && PROVIDER.test(workglow.error)) + failure = "provider-error"; + else if (exceptionType !== undefined) + failure = OVERFLOW.test(text) ? "context-overflow" : "agent-crash"; + else if (workglow?.outcome === "error") + failure = OVERFLOW.test(text) ? "context-overflow" : "agent-crash"; + else failure = "wrong-answer"; + return { status: "fail", failure }; +} + +export function parseTrial(json: HarborTrialJson, job: string): TrialRecord | undefined { + if (!json.trial_name || !json.task_name) return undefined; + const configured = json.config?.agent; + const agent = json.agent_info?.name ?? configured?.name ?? configured?.import_path ?? "unknown"; + const rewards = json.verifier_result?.rewards ?? undefined; + const reward = rewards ? (rewards.reward ?? Object.values(rewards)[0]) : undefined; + const exceptionType = json.exception_info?.exception_type ?? undefined; + const exceptionMessage = json.exception_info?.exception_message ?? undefined; + const agentRan = !!json.agent_execution?.started_at; + const workglow = json.agent_result?.metadata?.workglow ?? undefined; + const { status, failure } = classifyTrial( + reward, + exceptionType, + exceptionMessage, + agentRan, + workglow + ); + // A trial that failed in setup has no agent_info yet; the model it was + // configured with still says which arm it belongs to. + const model = json.agent_info?.model_info + ? `${json.agent_info.model_info.provider ?? "?"}/${json.agent_info.model_info.name ?? "?"}` + : (json.config?.agent?.model_name ?? undefined); + const result = json.agent_result ?? undefined; + return { + job, + trial: json.trial_name, + task: json.task_name, + arm: agent, + kwargs: configured?.kwargs ?? {}, + agent, + agentVersion: json.agent_info?.version, + model, + reward, + status, + failure, + exceptionType, + exceptionMessage, + inputTokens: result?.n_input_tokens ?? undefined, + cacheTokens: result?.n_cache_tokens ?? undefined, + outputTokens: result?.n_output_tokens ?? undefined, + costUsd: result?.cost_usd ?? undefined, + agentSec: seconds(json.agent_execution), + setupSec: seconds(json.agent_setup), + workglow, + }; +} + +/** + * Whether a `result.json` is a trial's rather than its job's. Told apart by the + * trial's own fields, not by an `agent/` directory beside it: a trial that + * failed in setup can leave its result before it has any agent logs, and is the + * blocked trial every rate has to exclude. + */ +function isTrialResult(file: string): boolean { + try { + const json = JSON.parse(readFileSync(file, "utf8")) as Record | null; + return ( + json !== null && + typeof json === "object" && + (typeof json.trial_name === "string" || typeof json.task_name === "string") + ); + } catch { + return false; + } +} + +/** Every trial `result.json` at or below `path` (a trial dir, a job dir, or a jobs dir). */ +function findTrialResults(path: string, depth = 0): string[] { + if (!existsSync(path)) throw new Error(`no such path: ${path}`); + if (statSync(path).isFile()) return [path]; + const own = join(path, "result.json"); + const found: string[] = []; + if (existsSync(own) && isTrialResult(own)) return [own]; + if (depth >= 3) return found; + for (const entry of readdirSync(path, { withFileTypes: true })) { + if (entry.isDirectory()) found.push(...findTrialResults(join(path, entry.name), depth + 1)); + } + return found; +} + +/** + * Reasoning tokens opencode reported for a trial, from its own step records. + * + * Harbor's opencode adapter counts `tokens.output` alone, while opencode + * reports reasoning beside it as `tokens.reasoning`. Providers bill reasoning + * as output, and the other two arms' counts already include it. Without this, + * opencode's output column reads a third of what it generated. Its cost is + * unaffected: opencode prices both. + */ +export function opencodeReasoningTokens(trialDir: string): number | undefined { + const path = join(trialDir, "agent", "opencode.txt"); + if (!existsSync(path)) return undefined; + let total = 0; + for (const line of readFileSync(path, "utf8").split("\n")) { + if (!line.includes('"step_finish"')) continue; + try { + const event = JSON.parse(line) as { + type?: string; + part?: { tokens?: { reasoning?: number } }; + }; + if (event.type === "step_finish") total += event.part?.tokens?.reasoning ?? 0; + } catch { + // A line that is not an event. + } + } + return total; +} + +export function loadTrials(paths: readonly string[]): TrialRecord[] { + const trials: TrialRecord[] = []; + for (const path of paths) { + for (const file of findTrialResults(path)) { + let json: HarborTrialJson; + try { + json = JSON.parse(readFileSync(file, "utf8")) as HarborTrialJson; + } catch { + continue; + } + const record = parseTrial(json, basename(dirname(dirname(file)))); + if (!record) continue; + const reasoning = + record.agent === "opencode" && record.outputTokens !== undefined + ? opencodeReasoningTokens(dirname(file)) + : undefined; + trials.push( + reasoning ? { ...record, outputTokens: record.outputTokens! + reasoning } : record + ); + } + } + return labelArms(trials); +} diff --git a/examples/agent-eval/src/compare/report.ts b/examples/agent-eval/src/compare/report.ts new file mode 100644 index 0000000000..dae9b17407 --- /dev/null +++ b/examples/agent-eval/src/compare/report.ts @@ -0,0 +1,400 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { FailureMode, TrialRecord } from "./harborJobs"; +import type { Interval } from "./stats"; +import { mcnemarExactP, mean, median, passAtK, wilsonInterval } from "./stats"; + +export interface ArmSummary { + readonly arm: string; + readonly models: readonly string[]; + readonly versions: readonly string[]; + readonly tasks: number; + readonly trials: number; + /** Trials the infrastructure lost before the agent ran; excluded from every rate. */ + readonly blocked: number; + readonly passed: number; + /** Passes over trials that ran; undefined when none ran, which is no measurement, not zero. */ + readonly passRate: number | undefined; + readonly passRateCi: Interval | undefined; + /** Mean over tasks of the task's pass fraction: pass@1, unbiased with several attempts. */ + readonly passAt1: number | undefined; + /** pass@k at the largest k every task in this arm has attempts for, when above 1. */ + readonly passAtK: { readonly k: number; readonly value: number } | undefined; + readonly costUsd: number | undefined; + readonly costPerPass: number | undefined; + /** Trials that reported no cost; a total over the rest would understate. */ + readonly unpricedTrials: number; + /** Trials that reported no token counts; the token totals cover the rest. */ + readonly usageMissingTrials: number; + readonly inputTokens: number; + readonly cacheTokens: number; + readonly outputTokens: number; + readonly medianAgentSec: number | undefined; + readonly failures: Readonly>>; + readonly workglow: WorkglowArmDetail | undefined; +} + +export interface WorkglowArmDetail { + readonly outcomes: Readonly>; + readonly meanRounds: number | undefined; + readonly retries: number; + readonly toolCalls: number; + readonly toolErrors: number; + readonly byTool: Readonly>; + readonly maxHistoryChars: number; +} + +export interface PairComparison { + readonly a: string; + readonly b: string; + readonly commonTasks: number; + /** Tasks every attempt of both arms passed. */ + readonly bothPass: number; + readonly neither: number; + /** Tasks both arms passed on the same share of attempts, short of all of them. */ + readonly partialTie: number; + /** Tasks where `a`'s pass fraction is higher. */ + readonly aBetter: readonly string[]; + readonly bBetter: readonly string[]; + /** Exact sign / McNemar p-value over the tasks where the arms differ. */ + readonly pValue: number; +} + +export interface TaskCell { + readonly passes: number; + readonly runs: number; + readonly blocked: number; + readonly failures: readonly FailureMode[]; +} + +export interface ComparisonReport { + readonly generatedAt: string; + readonly arms: readonly ArmSummary[]; + readonly pairs: readonly PairComparison[]; + readonly tasks: ReadonlyArray<{ + readonly task: string; + readonly cells: Readonly>; + }>; +} + +function groupBy(items: readonly T[], key: (item: T) => string): Map { + const groups = new Map(); + for (const item of items) { + const k = key(item); + const list = groups.get(k); + if (list) list.push(item); + else groups.set(k, [item]); + } + return groups; +} + +function sum(values: ReadonlyArray): number { + return values.reduce((total, value) => total + (value ?? 0), 0); +} + +function workglowDetail(trials: readonly TrialRecord[]): WorkglowArmDetail | undefined { + const runs = trials.flatMap((trial) => (trial.workglow ? [trial.workglow] : [])); + if (runs.length === 0) return undefined; + const outcomes: Record = {}; + // Tool names are the model's to choose, `__proto__` included. + const byTool: Record = Object.create(null); + for (const run of runs) { + outcomes[run.outcome] = (outcomes[run.outcome] ?? 0) + 1; + for (const [name, tally] of Object.entries(run.tools.byName)) { + const entry = (byTool[name] ??= { calls: 0, errors: 0 }); + entry.calls += tally.calls; + entry.errors += tally.errors; + } + } + return { + outcomes, + meanRounds: mean(runs.map((run) => run.rounds)), + retries: sum(runs.map((run) => run.retries)), + toolCalls: sum(runs.map((run) => run.tools.calls)), + toolErrors: sum(runs.map((run) => run.tools.errors)), + byTool, + maxHistoryChars: Math.max(...runs.map((run) => run.finalHistoryChars)), + }; +} + +function summarizeArm(arm: string, trials: readonly TrialRecord[]): ArmSummary { + const ran = trials.filter((trial) => trial.status !== "blocked"); + const passed = ran.filter((trial) => trial.status === "pass").length; + const byTask = groupBy(ran, (trial) => trial.task); + const fractions = [...byTask.values()].map( + (list) => list.filter((trial) => trial.status === "pass").length / list.length + ); + const minAttempts = Math.min(...[...byTask.values()].map((list) => list.length)); + const passAtKValue = + byTask.size > 0 && minAttempts > 1 + ? { + k: minAttempts, + value: + mean( + [...byTask.values()].map((list) => + passAtK( + list.length, + list.filter((trial) => trial.status === "pass").length, + minAttempts + ) + ) + ) ?? 0, + } + : undefined; + const priced = ran.filter((trial) => trial.costUsd !== undefined); + const costUsd = priced.length > 0 ? sum(priced.map((trial) => trial.costUsd)) : undefined; + const failures: Partial> = {}; + for (const trial of ran) { + if (trial.failure) failures[trial.failure] = (failures[trial.failure] ?? 0) + 1; + } + return { + arm, + models: [...new Set(trials.flatMap((trial) => (trial.model ? [trial.model] : [])))], + versions: [ + ...new Set(trials.flatMap((trial) => (trial.agentVersion ? [trial.agentVersion] : []))), + ], + tasks: new Set(trials.map((trial) => trial.task)).size, + trials: trials.length, + blocked: trials.length - ran.length, + passed, + passRate: ran.length > 0 ? passed / ran.length : undefined, + passRateCi: ran.length > 0 ? wilsonInterval(passed, ran.length) : undefined, + passAt1: mean(fractions), + passAtK: passAtKValue, + costUsd, + // Only when every trial was priced: a total missing some trials, divided by + // every pass, would understate what a pass costs. + costPerPass: + costUsd !== undefined && passed > 0 && priced.length === ran.length + ? costUsd / passed + : undefined, + unpricedTrials: ran.length - priced.length, + usageMissingTrials: ran.filter( + (trial) => + trial.inputTokens === undefined || + trial.cacheTokens === undefined || + trial.outputTokens === undefined + ).length, + inputTokens: sum(ran.map((trial) => trial.inputTokens)), + cacheTokens: sum(ran.map((trial) => trial.cacheTokens)), + outputTokens: sum(ran.map((trial) => trial.outputTokens)), + medianAgentSec: median( + ran.flatMap((trial) => (trial.agentSec === undefined ? [] : [trial.agentSec])) + ), + failures, + workglow: workglowDetail(ran), + }; +} + +function cell(trials: readonly TrialRecord[]): TaskCell { + const ran = trials.filter((trial) => trial.status !== "blocked"); + return { + passes: ran.filter((trial) => trial.status === "pass").length, + runs: ran.length, + blocked: trials.length - ran.length, + failures: ran.flatMap((trial) => (trial.failure ? [trial.failure] : [])), + }; +} + +function fraction(c: TaskCell | undefined): number | undefined { + return c === undefined || c.runs === 0 ? undefined : c.passes / c.runs; +} + +/** + * The paired comparison is the one that answers "is A better than B": both + * arms ran the same tasks, so a task solved by both or by neither says + * nothing, and the tasks where they differ are both the evidence and the + * list worth reading transcripts for. + */ +function comparePair(a: string, b: string, tasks: ComparisonReport["tasks"]): PairComparison { + let commonTasks = 0; + let bothPass = 0; + let neither = 0; + let partialTie = 0; + const aBetter: string[] = []; + const bBetter: string[] = []; + for (const { task, cells } of tasks) { + const fa = fraction(cells[a]); + const fb = fraction(cells[b]); + if (fa === undefined || fb === undefined) continue; + commonTasks++; + if (fa > fb) aBetter.push(task); + else if (fb > fa) bBetter.push(task); + else if (fa === 0) neither++; + else if (fa === 1) bothPass++; + else partialTie++; + } + return { + a, + b, + commonTasks, + bothPass, + neither, + partialTie, + aBetter, + bBetter, + pValue: mcnemarExactP(aBetter.length, bBetter.length), + }; +} + +/** Whether any of the arm's trials ran: an arm whose every trial was blocked has no rate. */ +function ranAny(arm: ArmSummary): boolean { + return arm.trials > arm.blocked; +} + +export function buildReport(trials: readonly TrialRecord[]): ComparisonReport { + const byArm = groupBy(trials, (trial) => trial.arm); + const arms = [...byArm.entries()] + .map(([arm, list]) => summarizeArm(arm, list)) + // An arm none of whose trials ran has no rate to rank by, which is not a + // rate of zero; it goes last. + .sort( + (x, y) => + Number(ranAny(y)) - Number(ranAny(x)) || + (y.passAt1 ?? 0) - (x.passAt1 ?? 0) || + x.arm.localeCompare(y.arm) + ); + const armNames = arms.map((arm) => arm.arm); + + const tasks = [...groupBy(trials, (trial) => trial.task).entries()] + .map(([task, list]) => { + // Arm names come from job directories, so no inherited keys to collide with. + const cells: Record = Object.create(null); + for (const [arm, armTrials] of groupBy(list, (trial) => trial.arm)) + cells[arm] = cell(armTrials); + return { task, cells }; + }) + .sort((x, y) => x.task.localeCompare(y.task)); + + const pairs: PairComparison[] = []; + for (let i = 0; i < armNames.length; i++) { + for (let j = i + 1; j < armNames.length; j++) { + pairs.push(comparePair(armNames[i]!, armNames[j]!, tasks)); + } + } + return { generatedAt: new Date().toISOString(), arms, pairs, tasks }; +} + +const pct = (value: number): string => `${(value * 100).toFixed(1)}%`; +const usd = (value: number | undefined): string => + value === undefined ? "—" : `$${value.toFixed(value < 1 ? 4 : 2)}`; +const int = (value: number): string => value.toLocaleString("en-US"); +const secs = (value: number | undefined): string => + value === undefined ? "—" : `${value.toFixed(0)}s`; + +function markSymbol(c: TaskCell | undefined): string { + if (c === undefined) return "·"; + if (c.runs === 0) return c.blocked > 0 ? "B" : "·"; + if (c.runs === 1) return c.passes === 1 ? "✓" : `✗ ${c.failures[0] ?? ""}`.trim(); + return `${c.passes}/${c.runs}`; +} + +export function renderMarkdown( + report: ComparisonReport, + options: { readonly allTasks?: boolean } = {} +): string { + const lines: string[] = []; + lines.push(`# Agent harness comparison`, "", `Generated ${report.generatedAt}.`, ""); + + lines.push( + "| Arm | Model | Tasks | Ran | Blocked | Pass rate (95% CI) | pass@1 | pass@k | Cost | $/pass | In tok | Cache tok | Out tok | Median time |", + "|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|" + ); + for (const arm of report.arms) { + const cost = + arm.unpricedTrials > 0 && arm.costUsd !== undefined + ? `${usd(arm.costUsd)}*` + : usd(arm.costUsd); + const partial = arm.usageMissingTrials > 0 ? "†" : ""; + const rates = + arm.passRate !== undefined && arm.passRateCi !== undefined && arm.passAt1 !== undefined + ? `${pct(arm.passRate)} (${pct(arm.passRateCi.low)}–${pct(arm.passRateCi.high)}) | ${pct(arm.passAt1)}` + : "— | —"; + lines.push( + `| ${arm.arm} | ${arm.models.join(", ")} | ${arm.tasks} | ${arm.trials - arm.blocked} | ${arm.blocked} | ` + + `${rates} | ` + + `${arm.passAtK ? `${pct(arm.passAtK.value)} (k=${arm.passAtK.k})` : "—"} | ${cost} | ${usd(arm.costPerPass)} | ` + + `${int(arm.inputTokens)}${partial} | ${int(arm.cacheTokens)}${partial} | ${int(arm.outputTokens)}${partial} | ${secs(arm.medianAgentSec)} |` + ); + } + if (report.arms.some((arm) => arm.unpricedTrials > 0)) { + lines.push("", "\\* some trials reported no cost; the total covers the rest."); + } + if (report.arms.some((arm) => arm.usageMissingTrials > 0)) { + lines.push("", "† some trials reported no token counts; the totals cover the rest."); + } + + lines.push("", "## Head to head", ""); + lines.push( + "| A | B | Tasks | Both ✓ | Neither | Tied partial | A better | B better | p (exact sign) |", + "|---|---|---:|---:|---:|---:|---:|---:|---:|" + ); + for (const pair of report.pairs) { + lines.push( + `| ${pair.a} | ${pair.b} | ${pair.commonTasks} | ${pair.bothPass} | ${pair.neither} | ${pair.partialTie} | ` + + `${pair.aBetter.length} | ${pair.bBetter.length} | ${pair.pValue.toFixed(3)} |` + ); + } + for (const pair of report.pairs) { + if (pair.aBetter.length + pair.bBetter.length === 0) continue; + lines.push("", `### ${pair.a} vs ${pair.b}`, ""); + if (pair.aBetter.length > 0) + lines.push(`- **${pair.a}** better on: ${pair.aBetter.join(", ")}`); + if (pair.bBetter.length > 0) + lines.push(`- **${pair.b}** better on: ${pair.bBetter.join(", ")}`); + } + + lines.push("", "## Why runs failed", ""); + const modes = [ + ...new Set(report.arms.flatMap((arm) => Object.keys(arm.failures))), + ].sort() as FailureMode[]; + if (modes.length === 0) lines.push("No failures."); + else { + lines.push(`| Arm | ${modes.join(" | ")} |`, `|---|${modes.map(() => "---:").join("|")}|`); + for (const arm of report.arms) { + lines.push(`| ${arm.arm} | ${modes.map((mode) => arm.failures[mode] ?? 0).join(" | ")} |`); + } + } + + const workglowArms = report.arms.filter((arm) => arm.workglow); + if (workglowArms.length > 0) { + lines.push("", "## Workglow loop detail", ""); + for (const arm of workglowArms) { + const detail = arm.workglow!; + const outcomes = Object.entries(detail.outcomes) + .map(([outcome, n]) => `${outcome} ${n}`) + .join(", "); + const tools = Object.entries(detail.byTool) + .sort(([, x], [, y]) => y.calls - x.calls) + .map(([name, t]) => `${name} ${t.calls} (${t.errors} err)`) + .join(", "); + lines.push( + `- **${arm.arm}**: outcomes ${outcomes}; mean rounds ${detail.meanRounds?.toFixed(1) ?? "—"}; ` + + `retries ${detail.retries}; tool calls ${detail.toolCalls} with ${detail.toolErrors} errors ` + + `(${tools}); largest transcript ${int(detail.maxHistoryChars)} chars` + ); + } + } + + const arms = report.arms.map((arm) => arm.arm); + const rows = report.tasks.filter((row) => { + if (options.allTasks) return true; + const values = arms.map((arm) => fraction(row.cells[arm])).filter((v) => v !== undefined); + // An arm with no result for the task (blocked, or never run) is a gap in the + // comparison, and the row is how the reader finds it. + return new Set(values).size > 1 || values.length < arms.length; + }); + lines.push("", options.allTasks ? "## Every task" : "## Tasks where the arms differ", ""); + if (rows.length === 0) lines.push("None."); + else { + lines.push(`| Task | ${arms.join(" | ")} |`, `|---|${arms.map(() => "---").join("|")}|`); + for (const row of rows) + lines.push(`| ${row.task} | ${arms.map((arm) => markSymbol(row.cells[arm])).join(" | ")} |`); + } + lines.push(""); + return lines.join("\n"); +} diff --git a/examples/agent-eval/src/compare/stats.ts b/examples/agent-eval/src/compare/stats.ts new file mode 100644 index 0000000000..29b612fa90 --- /dev/null +++ b/examples/agent-eval/src/compare/stats.ts @@ -0,0 +1,70 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +export interface Interval { + readonly low: number; + readonly high: number; +} + +/** + * Wilson score interval for a pass rate. Benchmark arms are small (tens of + * tasks) and rates near 0 or 1 are common, which is where the normal + * approximation gives intervals outside [0, 1]. + */ +export function wilsonInterval(successes: number, trials: number, z = 1.96): Interval { + if (trials === 0) return { low: 0, high: 0 }; + const p = successes / trials; + const z2 = z * z; + const denominator = 1 + z2 / trials; + const center = (p + z2 / (2 * trials)) / denominator; + const margin = (z * Math.sqrt((p * (1 - p)) / trials + z2 / (4 * trials * trials))) / denominator; + return { low: Math.max(0, center - margin), high: Math.min(1, center + margin) }; +} + +function logChoose(n: number, k: number): number { + let sum = 0; + for (let i = 1; i <= k; i++) sum += Math.log(n - k + i) - Math.log(i); + return sum; +} + +/** + * Unbiased pass@k for one task from n attempts with c passes (Chen et al., + * 2021): the chance that at least one of k attempts drawn without replacement + * passes. + */ +export function passAtK(n: number, c: number, k: number): number { + if (k > n) throw new Error(`pass@${k} needs at least ${k} attempts, got ${n}`); + if (n - c < k) return 1; + return 1 - Math.exp(logChoose(n - c, k) - logChoose(n, k)); +} + +/** + * Exact two-sided McNemar test on the discordant pairs of a paired + * comparison: `onlyA` tasks the first arm solved and the second did not, and + * the reverse. The concordant tasks carry no information about which arm is + * better, which is why a paired test needs far fewer tasks than comparing two + * pass rates as if they were independent samples. + */ +export function mcnemarExactP(onlyA: number, onlyB: number): number { + const n = onlyA + onlyB; + if (n === 0) return 1; + const k = Math.min(onlyA, onlyB); + let tail = 0; + for (let i = 0; i <= k; i++) tail += Math.exp(logChoose(n, i) - n * Math.LN2); + return Math.min(1, 2 * tail); +} + +export function median(values: readonly number[]): number | undefined { + if (values.length === 0) return undefined; + const sorted = [...values].sort((a, b) => a - b); + const mid = Math.floor(sorted.length / 2); + return sorted.length % 2 === 1 ? sorted[mid]! : (sorted[mid - 1]! + sorted[mid]!) / 2; +} + +export function mean(values: readonly number[]): number | undefined { + if (values.length === 0) return undefined; + return values.reduce((sum, value) => sum + value, 0) / values.length; +} diff --git a/examples/agent-eval/src/mock/anthropicMock.ts b/examples/agent-eval/src/mock/anthropicMock.ts new file mode 100644 index 0000000000..6e4364d7d9 --- /dev/null +++ b/examples/agent-eval/src/mock/anthropicMock.ts @@ -0,0 +1,224 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { appendFileSync } from "node:fs"; +import type { IncomingMessage, Server, ServerResponse } from "node:http"; +import { createServer } from "node:http"; + +/** + * A scripted stand-in for the Anthropic Messages API, for checking the whole + * pipeline — install, run, usage capture, verification, comparison — without a + * key or a bill. Every harness in the comparison can be pointed at an + * Anthropic base URL, so one fake serves all three. + * + * The script is a list of shell commands. Each request is answered with a call + * to whatever tool the harness named `bash`, running the next command — the + * position is the number of assistant turns already in the conversation — + * and once the commands run out, with a short text answer. That makes the + * behaviour a function of the conversation alone, so it is the same for every + * harness and survives the retries and side requests (titles, summaries) a + * harness makes. + */ +export interface AnthropicMockOptions { + readonly port: number; + readonly host?: string | undefined; + readonly commands: readonly string[]; + /** Append every request body here as JSON lines, for inspection. */ + readonly requestLog?: string | undefined; + readonly finalText?: string | undefined; +} + +interface MessagesRequest { + readonly model?: string; + readonly stream?: boolean; + readonly system?: unknown; + readonly messages?: ReadonlyArray<{ role: string; content: unknown }>; + readonly tools?: ReadonlyArray<{ + name: string; + input_schema?: { properties?: Record; required?: string[] }; + }>; +} + +type Reply = + | { readonly kind: "text"; readonly text: string } + | { readonly kind: "tool"; readonly name: string; readonly input: Record }; + +let nextId = 0; +const id = (prefix: string): string => `${prefix}_mock${(++nextId).toString().padStart(6, "0")}`; + +/** Fills a tool's required arguments: the command where it wants one, a placeholder elsewhere. */ +function bashInput( + tool: NonNullable[number], + command: string +): Record { + const input: Record = { command }; + const properties = tool.input_schema?.properties ?? {}; + for (const name of tool.input_schema?.required ?? []) { + if (name in input) continue; + const type = properties[name]?.type; + input[name] = + type === "number" || type === "integer" + ? 60 + : type === "boolean" + ? false + : "Run the next step"; + } + return input; +} + +export function decideReply(request: MessagesRequest, options: AnthropicMockOptions): Reply { + const finalText = options.finalText ?? "Done. The task is complete."; + const bash = request.tools?.find((tool) => tool.name.toLowerCase() === "bash"); + if (bash === undefined) return { kind: "text", text: finalText }; + const step = (request.messages ?? []).filter((message) => message.role === "assistant").length; + const command = options.commands[step]; + if (command === undefined) return { kind: "text", text: finalText }; + return { kind: "tool", name: bash.name, input: bashInput(bash, command) }; +} + +function promptSize(request: MessagesRequest): number { + return Math.ceil(JSON.stringify([request.system, request.messages, request.tools]).length / 4); +} + +function sse(res: ServerResponse, event: string, data: unknown): void { + res.write(`event: ${event}\ndata: ${JSON.stringify(data)}\n\n`); +} + +function respond(res: ServerResponse, request: MessagesRequest, reply: Reply): void { + const messageId = id("msg"); + const model = request.model ?? "mock"; + const usage = { + input_tokens: promptSize(request), + output_tokens: 24, + cache_creation_input_tokens: 0, + cache_read_input_tokens: 0, + }; + const stopReason = reply.kind === "tool" ? "tool_use" : "end_turn"; + const block = + reply.kind === "tool" + ? { type: "tool_use", id: id("toolu"), name: reply.name, input: reply.input } + : { type: "text", text: reply.text }; + + if (!request.stream) { + res.writeHead(200, { "content-type": "application/json" }); + res.end( + JSON.stringify({ + id: messageId, + type: "message", + role: "assistant", + model, + content: [block], + stop_reason: stopReason, + stop_sequence: null, + usage, + }) + ); + return; + } + + res.writeHead(200, { + "content-type": "text/event-stream", + "cache-control": "no-cache", + connection: "keep-alive", + }); + sse(res, "message_start", { + type: "message_start", + message: { + id: messageId, + type: "message", + role: "assistant", + model, + content: [], + stop_reason: null, + stop_sequence: null, + usage: { ...usage, output_tokens: 1 }, + }, + }); + if (block.type === "tool_use") { + sse(res, "content_block_start", { + type: "content_block_start", + index: 0, + content_block: { type: "tool_use", id: block.id, name: block.name, input: {} }, + }); + sse(res, "content_block_delta", { + type: "content_block_delta", + index: 0, + delta: { type: "input_json_delta", partial_json: JSON.stringify(block.input) }, + }); + } else { + sse(res, "content_block_start", { + type: "content_block_start", + index: 0, + content_block: { type: "text", text: "" }, + }); + sse(res, "content_block_delta", { + type: "content_block_delta", + index: 0, + delta: { type: "text_delta", text: block.text }, + }); + } + sse(res, "content_block_stop", { type: "content_block_stop", index: 0 }); + sse(res, "message_delta", { + type: "message_delta", + delta: { stop_reason: stopReason, stop_sequence: null }, + usage: { output_tokens: usage.output_tokens }, + }); + sse(res, "message_stop", { type: "message_stop" }); + res.end(); +} + +async function readBody(req: IncomingMessage): Promise { + const chunks: Buffer[] = []; + for await (const chunk of req) chunks.push(chunk as Buffer); + return Buffer.concat(chunks).toString("utf8"); +} + +export function startAnthropicMock(options: AnthropicMockOptions): Promise { + const server = createServer(async (req, res) => { + try { + const path = (req.url ?? "").split("?")[0] ?? ""; + if (req.method === "POST" && path.endsWith("/messages/count_tokens")) { + const request = JSON.parse(await readBody(req)) as MessagesRequest; + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify({ input_tokens: promptSize(request) })); + return; + } + if (req.method === "POST" && path.endsWith("/messages")) { + const body = await readBody(req); + const request = JSON.parse(body) as MessagesRequest; + if (options.requestLog) { + appendFileSync( + options.requestLog, + `${JSON.stringify({ at: new Date().toISOString(), request })}\n` + ); + } + respond(res, request, decideReply(request, options)); + return; + } + res.writeHead(404, { "content-type": "application/json" }); + res.end(JSON.stringify({ type: "error", error: { type: "not_found_error", message: path } })); + } catch (error) { + res.writeHead(400, { "content-type": "application/json" }); + res.end( + JSON.stringify({ + type: "error", + error: { + type: "invalid_request_error", + message: error instanceof SyntaxError ? "request body is not JSON" : "invalid request", + }, + }) + ); + } + }); + return new Promise((resolve, reject) => { + // A port already in use is reported as an error event, not thrown. + server.once("error", reject); + server.listen(options.port, options.host ?? "127.0.0.1", () => { + server.off("error", reject); + resolve(server); + }); + }); +} diff --git a/examples/agent-eval/src/run/jobConfig.ts b/examples/agent-eval/src/run/jobConfig.ts new file mode 100644 index 0000000000..09d80f928f --- /dev/null +++ b/examples/agent-eval/src/run/jobConfig.ts @@ -0,0 +1,189 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { JobDataset } from "./suites"; + +export const HARNESSES = ["workglow", "opencode", "pi"] as const; +export type Harness = (typeof HARNESSES)[number]; + +/** One column of the comparison: a harness and the kwargs that make it this arm. */ +export interface ArmSpec { + readonly harness: Harness; + readonly kwargs: Readonly>; +} + +/** Reasoning levels every harness can be set to, in pi's vocabulary. */ +export const EFFORTS = ["off", "minimal", "low", "medium", "high", "xhigh", "max"] as const; +export type Effort = (typeof EFFORTS)[number]; + +export interface ParityOptions { + /** Model calls each arm may make. Every harness gets the same cap. */ + readonly maxTurns: number; + readonly effort: Effort; + /** opencode names reasoning levels per model ("high", "max", …); unset leaves it at its default. */ + readonly opencodeVariant: string | undefined; + /** + * pi needs the wire API spelled out when the provider endpoint is + * overridden (a proxy or a mock): `anthropic-messages`, `openai-responses`, … + */ + readonly piModelApi: string | undefined; +} + +/** + * The kwargs that hold each harness to the same budget and reasoning level. + * + * Without them the arms are not comparable: opencode and pi run until the + * model stops (no turn cap at all), pi thinks at `medium` by default while the + * other two do not think unless asked, and AgentTask stops after 8 rounds. + * An arm's own kwargs win, so an ablation can still move one of these. + */ +export function parityKwargs(harness: Harness, options: ParityOptions): Record { + switch (harness) { + case "workglow": + return { max_rounds: options.maxTurns, effort: options.effort }; + case "opencode": + return { + opencode_config: { agent: { build: { steps: options.maxTurns } } }, + ...(options.opencodeVariant ? { variant: options.opencodeVariant } : {}), + }; + case "pi": + return { + max_turns: options.maxTurns, + thinking: options.effort, + ...(options.piModelApi ? { model_api: options.piModelApi } : {}), + }; + } +} + +/** + * Where the value starting at `start` ends. A JSON string, array or object runs to + * its closing quote or bracket, so it may hold anything; a bare value runs to the + * next comma that a `key=` follows, so a sentence or a list can carry its own commas. + */ +function kwargValueEnd(text: string, start: number): number { + let i = start; + while (i < text.length && /\s/.test(text[i]!)) i++; + const open = text[i]; + if (open === '"' || open === "{" || open === "[") { + let depth = 0; + let inString = false; + for (; i < text.length; i++) { + const c = text[i]; + if (inString) { + if (c === "\\") i++; + else if (c === '"') { + inString = false; + if (depth === 0) return i + 1; + } + } else if (c === '"') { + inString = true; + } else if (c === "{" || c === "[") { + depth++; + } else if (c === "}" || c === "]") { + depth--; + if (depth === 0) return i + 1; + } + } + // An opened value that never closes would otherwise swallow every pair after it. + throw new Error(`arm kwarg value starting ${text.slice(start).trim()} is never closed`); + } + const next = text.slice(start).search(/,(?=\s*[A-Za-z_][\w-]*=)/); + return next === -1 ? text.length : start + next; +} + +/** + * `workglow`, `pi:thinking=high`, `workglow:tool_concurrency=8,max_history_chars=400000`. + * Values are JSON where they parse as JSON, strings otherwise — Harbor's own + * `--ak` rule. A value holding a comma followed by `key=` is written as a JSON + * string: `workglow:append_system_prompt="When x=a, set y=b"`. + */ +export function parseArm(spec: string): ArmSpec { + const colon = spec.indexOf(":"); + const name = (colon === -1 ? spec : spec.slice(0, colon)).trim(); + if (!(HARNESSES as readonly string[]).includes(name)) { + throw new Error(`unknown harness "${name}" — one of ${HARNESSES.join(", ")}`); + } + const kwargs: Record = {}; + const body = colon === -1 ? "" : spec.slice(colon + 1); + let pos = 0; + while (pos < body.length) { + const eq = body.indexOf("=", pos); + const comma = body.indexOf(",", pos); + if (eq === -1 || (comma !== -1 && comma < eq)) { + const pair = body.slice(pos, comma === -1 ? undefined : comma).trim(); + if (pair !== "") throw new Error(`arm kwarg "${pair}" must be key=value`); + pos = comma === -1 ? body.length : comma + 1; + continue; + } + const key = body.slice(pos, eq).trim(); + const end = kwargValueEnd(body, eq + 1); + const raw = body.slice(eq + 1, end).trim(); + let value: unknown = raw; + try { + value = JSON.parse(raw); + } catch { + // A bare string. + } + kwargs[key] = value; + const after = body.indexOf(",", end); + const trailing = body.slice(end, after === -1 ? undefined : after).trim(); + if (trailing !== "") + throw new Error(`arm kwarg "${key}" has text after its value: ${trailing}`); + pos = after === -1 ? body.length : after + 1; + } + return { harness: name as Harness, kwargs }; +} + +export interface HarborJobOptions { + readonly jobName: string; + readonly jobsDir: string; + readonly model: string; + readonly arms: readonly ArmSpec[]; + readonly datasets: readonly JobDataset[]; + readonly parity: ParityOptions; + readonly attempts: number; + readonly concurrency: number; + readonly timeoutMultiplier: number | undefined; +} + +/** + * Exceptions that mean the infrastructure failed before or around the agent: + * a container that would not build or start (Docker reports those as + * `RuntimeError`), or an install that timed out. Retrying them is free of + * bias. Nothing the agent itself did is listed — retrying an agent failure + * would turn one attempt into best-of-n for whichever arm failed more. + */ +export const INFRASTRUCTURE_RETRY_EXCEPTIONS = [ + "RuntimeError", + "EnvironmentStartTimeoutError", + "AgentSetupTimeoutError", +] as const; + +/** + * One Harbor job holding every arm, so all of them run the same task list + * under the same environment settings; the comparison then pairs trials by + * task. The workglow arm is loaded by import path, which needs this + * package's `harbor/` directory on PYTHONPATH. + */ +export function buildHarborJob(options: HarborJobOptions): Record { + return { + job_name: options.jobName, + jobs_dir: options.jobsDir, + n_attempts: options.attempts, + n_concurrent_trials: options.concurrency, + retry: { max_retries: 2, include_exceptions: [...INFRASTRUCTURE_RETRY_EXCEPTIONS] }, + ...(options.timeoutMultiplier === undefined + ? {} + : { timeout_multiplier: options.timeoutMultiplier }), + agents: options.arms.map((arm) => { + const kwargs = { ...parityKwargs(arm.harness, options.parity), ...arm.kwargs }; + return arm.harness === "workglow" + ? { import_path: "workglow_agent:WorkglowAgent", model_name: options.model, kwargs } + : { name: arm.harness, model_name: options.model, kwargs }; + }), + datasets: options.datasets.map((dataset) => ({ ...dataset })), + }; +} diff --git a/examples/agent-eval/src/run/runHarbor.ts b/examples/agent-eval/src/run/runHarbor.ts new file mode 100644 index 0000000000..781f05bdeb --- /dev/null +++ b/examples/agent-eval/src/run/runHarbor.ts @@ -0,0 +1,97 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { spawn } from "node:child_process"; +import { existsSync, mkdirSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { delimiter, join, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; + +/** This package's root, found from this module whether it runs from src or a bundle in dist. */ +export function packageRoot(): string { + let dir = fileURLToPath(new URL(".", import.meta.url)); + for (let i = 0; i < 4; i++) { + if (existsSync(join(dir, "harbor", "workglow_agent.py"))) return dir; + dir = resolve(dir, ".."); + } + throw new Error("cannot find the agent-eval package root (harbor/workglow_agent.py)"); +} + +export interface HarborRun { + readonly config: Record; + /** Extra Harbor config files layered after the generated one (environment overlays and the like). */ + readonly extraConfigs: readonly string[]; + readonly harbor: string; +} + +interface HarborAgentConfig { + readonly import_path?: string; + readonly kwargs?: Record; +} + +/** + * What stops a workglow arm from finding the agent bundle it would upload, if + * anything. An arm's own `bundle_path` wins over the shared one, as it does in + * the adapter, which expands `~` and resolves it from the same working directory. + */ +export function missingBundle( + agents: readonly HarborAgentConfig[], + bundle: string +): string | undefined { + for (const agent of agents) { + if (!agent.import_path?.startsWith("workglow_agent")) continue; + const own = agent.kwargs?.bundle_path; + if (typeof own === "string" && own.length > 0) { + const path = resolve(own.replace(/^~(?=$|\/)/, homedir())); + if (!existsSync(path)) return `the workglow arm's bundle_path ${path} does not exist`; + } else if (!existsSync(bundle)) { + return `the workglow arm needs ${bundle}; run \`bun run build-agent\` first`; + } + } + return undefined; +} + +/** + * Writes the job config beside the job's results and runs `harbor run` on it, + * with the workglow adapter importable and the bundle it uploads located. + * Harbor's own progress display goes straight to the terminal. + */ +export async function runHarbor(run: HarborRun): Promise<{ jobDir: string; exitCode: number }> { + const root = packageRoot(); + const bundle = process.env.WORKGLOW_AGENT_BUNDLE ?? join(root, "dist", "workglow-agent.mjs"); + const jobsDir = resolve(String(run.config.jobs_dir)); + const jobDir = join(jobsDir, String(run.config.job_name)); + mkdirSync(jobsDir, { recursive: true }); + const configPath = join(jobsDir, `${String(run.config.job_name)}.config.json`); + writeFileSync(configPath, JSON.stringify({ ...run.config, jobs_dir: jobsDir }, null, 2)); + + const problem = missingBundle(run.config.agents as readonly HarborAgentConfig[], bundle); + if (problem !== undefined) throw new Error(problem); + + const args = [ + "run", + "-c", + configPath, + ...run.extraConfigs.flatMap((path) => ["-c", resolve(path)]), + ]; + const env = { + ...process.env, + PYTHONPATH: [join(root, "harbor"), process.env.PYTHONPATH].filter(Boolean).join(delimiter), + WORKGLOW_AGENT_BUNDLE: bundle, + }; + const exitCode = await new Promise((resolvePromise, reject) => { + const child = spawn(run.harbor, args, { stdio: "inherit", env }); + child.on("error", (error) => + reject( + new Error( + `could not start ${run.harbor} (${error.message}); install Harbor with \`uv tool install harbor\`` + ) + ) + ); + child.on("exit", (code) => resolvePromise(code ?? 1)); + }); + return { jobDir, exitCode }; +} diff --git a/examples/agent-eval/src/run/suites.ts b/examples/agent-eval/src/run/suites.ts new file mode 100644 index 0000000000..473493c245 --- /dev/null +++ b/examples/agent-eval/src/run/suites.ts @@ -0,0 +1,243 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** One Harbor registry dataset, optionally narrowed to some of its tasks. */ +export interface SuiteDataset { + readonly name: string; + readonly version: string; + /** Glob patterns over task names. */ + readonly task_names?: readonly string[] | undefined; + /** Cap applied after `task_names`, in registry order. */ + readonly n_tasks?: number | undefined; +} + +/** + * A directory of Harbor task folders on disk — a custom benchmark, or a local + * copy of a registry dataset prepared for an environment the registry copy + * cannot build in. + */ +export interface LocalDataset { + readonly path: string; + readonly task_names?: readonly string[] | undefined; + readonly n_tasks?: number | undefined; +} + +export type JobDataset = SuiteDataset | LocalDataset; + +export interface Suite { + readonly description: string; + readonly datasets: readonly SuiteDataset[]; +} + +/** + * Named slices of public benchmarks in Harbor's registry. Each is chosen for + * what it exercises in a loop, not only for its name: + * + * - Terminal-Bench 2.0 is long-horizon terminal work — builds, services, + * debugging — where context growth, command timeouts and recovery from tool + * errors decide the outcome. + * - Aider Polyglot is short edit-run-fix cycles in six languages, where edit + * reliability and reading test output dominate. + * - SWE-bench Verified is repository navigation and a minimal patch in a + * large Python codebase: search, reading at scale, and not breaking + * neighbouring tests. + */ +export const SUITES: Readonly> = { + smoke: { + description: + "hello-world plus the 10-task Terminal-Bench sample: does every arm install and run", + datasets: [ + { name: "hello-world", version: "1.0" }, + { name: "terminal-bench-sample", version: "2.0" }, + ], + }, + "terminal-bench": { + description: "Terminal-Bench 2.0, all 89 tasks", + datasets: [{ name: "terminal-bench", version: "2.0" }], + }, + polyglot: { + description: "Aider Polyglot, all 225 exercises in six languages", + datasets: [{ name: "aider-polyglot", version: "1.0" }], + }, + "polyglot-lite": { + description: "Aider Polyglot, Python and JavaScript only (83 exercises)", + datasets: [ + { + name: "aider-polyglot", + version: "1.0", + task_names: ["polyglot_python_*", "polyglot_javascript_*"], + }, + ], + }, + "swebench-50": { + description: "SWE-bench Verified, the first 50 tasks in registry order", + datasets: [{ name: "swebench-verified", version: "1.0", n_tasks: 50 }], + }, + core: { + description: + "Terminal-Bench 2.0 + Polyglot (Python/JS) + SWE-bench Verified 50: the decision set", + datasets: [ + { name: "terminal-bench", version: "2.0" }, + { + name: "aider-polyglot", + version: "1.0", + task_names: ["polyglot_python_*", "polyglot_javascript_*"], + }, + { name: "swebench-verified", version: "1.0", n_tasks: 50 }, + ], + }, +}; + +export function getSuite(name: string): Suite { + const suite = SUITES[name]; + if (suite === undefined) { + throw new Error(`unknown suite "${name}" — one of ${Object.keys(SUITES).join(", ")}`); + } + return suite; +} + +function globRegExp(glob: string): RegExp { + const body = glob + .replace(/[.+^${}()|[\]\\]/g, "\\$&") + .replace(/\*/g, ".*") + .replace(/\?/g, "."); + return new RegExp(`^${body}$`); +} + +/** A glob with exactly one `*` and no `?`, split at the star. */ +function splitStar(glob: string): { readonly head: string; readonly tail: string } | undefined { + const star = glob.indexOf("*"); + if (star < 0 || glob.indexOf("*", star + 1) >= 0 || glob.includes("?")) return undefined; + return { head: glob.slice(0, star), tail: glob.slice(star + 1) }; +} + +/** + * Globs matching exactly the names that start with `prefix` and end with `suffix`. + * `prefix*suffix` alone misses the names where the two overlap (`ab` and `bc` both + * match `abc`), so each such overlap is listed as a literal name beside it. + */ +function prefixSuffixGlobs(prefix: string, suffix: string): string[] { + if (suffix === "") return [`${prefix}*`]; + const globs = [`${prefix}*${suffix}`]; + for (let k = Math.min(prefix.length, suffix.length); k > 0; k--) { + if (prefix.endsWith(suffix.slice(0, k))) globs.push(prefix + suffix.slice(k)); + } + return globs; +} + +/** + * `glob` itself when every name it can match already falls under `prefixGlob` + * (`P*`), `[]` when none can, `undefined` when that cannot be told apart here. + * Decided when `glob` has no `*` (each character matches exactly one, so a + * fixed-length match either covers the prefix or contradicts it), and when it + * holds a single `*` whose head already covers the prefix. This is what lets a + * `?`-filter narrower than a suite's `prefix_*` pattern through. + */ +function subsetOfPrefix(glob: string, prefixGlob: string): string[] | undefined { + const split = splitStar(prefixGlob); + if (!split || split.tail !== "") return undefined; + const prefix = split.head; + if (!glob.includes("*")) { + if (glob.length < prefix.length) return []; + for (let i = 0; i < prefix.length; i++) { + if (glob[i] !== prefix[i] && glob[i] !== "?") return []; + } + return [glob]; + } + // A single-`*` glob (which may also hold `?`) whose head already covers the + // prefix matches only inside it (`polyglot_python_?*` needs one more + // character after the suite's `polyglot_python_*` prefix). A `?` inside the + // prefix range is only a partial overlap, and a head short of the prefix + // reaches into the star, so neither can be written down here. + const star = glob.indexOf("*"); + if (star < 0 || glob.indexOf("*", star + 1) >= 0) return undefined; + const head = glob.slice(0, star); + // A contradiction inside the overlap means no name matches both, even when + // the head is short of the prefix. Past the overlap a short head reaches + // into the star, so only a head covering the whole prefix proves the subset. + const compared = Math.min(head.length, prefix.length); + for (let i = 0; i < compared; i++) { + if (head[i] === prefix[i] || prefix[i] === "?") continue; + return head[i] === "?" ? undefined : []; + } + if (head.length < prefix.length) return undefined; + return [glob]; +} + +/** + * The task names `a` and `b` both match, as globs Harbor takes as one list: `[]` when + * no name can match both, `undefined` when the overlap is not one this can write down. + * Decided for literals, and for a single-`*` glob against a prefix glob (`P*`), which + * is the shape every suite here restricts itself with; never for a `[...]` class. + */ +function globIntersection(a: string, b: string): string[] | undefined { + if (a === b) return [a]; + // Harbor matches with fnmatch, where `[...]` is a character class; nothing here + // reasons about classes, so an overlap involving one is not written down. + if (a.includes("[") || b.includes("[")) return undefined; + if (!/[*?]/.test(a)) return globRegExp(b).test(a) ? [a] : []; + if (!/[*?]/.test(b)) return globRegExp(a).test(b) ? [b] : []; + const sa = splitStar(a); + const sb = splitStar(b); + if (!sa || !sb) { + // One side may be a `?`-glob with no `*` decided against the other's + // prefix above; anything wider stays unwritable. + return subsetOfPrefix(a, b) ?? subsetOfPrefix(b, a); + } + const [glob, prefix] = sb.tail === "" ? [sa, sb.head] : sa.tail === "" ? [sb, sa.head] : []; + if (!glob || prefix === undefined) return undefined; + if (glob.head.startsWith(prefix)) return [`${glob.head}*${glob.tail}`]; + if (prefix.startsWith(glob.head)) return prefixSuffixGlobs(prefix, glob.tail); + return []; +} + +/** + * A dataset narrowed by `--tasks` and `--n-tasks`. Both only ever narrow: the + * filter is intersected with the tasks the suite already restricts itself to + * (Harbor takes one list, so the intersection is computed here), and the cap + * replaces the suite's only when it is lower. Undefined when the filter shares + * no task with the dataset, which then has nothing to run. + */ +export function narrowDataset( + dataset: T, + taskFilter: readonly string[] | undefined, + nTasks: number | undefined +): T | undefined { + let taskNames = dataset.task_names; + if (taskFilter && taskFilter.length > 0) { + if (!taskNames) { + taskNames = [...taskFilter]; + } else { + const kept = new Set(); + for (const wanted of taskFilter) { + for (const allowed of taskNames) { + const both = globIntersection(wanted, allowed); + if (both === undefined) { + throw new Error( + `--tasks ${wanted} cannot be narrowed to this suite's ${allowed}: use a task ` + + `name, or a glob with at most one "*" and no [...] that stays within one of the suite's ` + + `(${taskNames.join(", ")})` + ); + } + for (const glob of both) kept.add(glob); + } + } + if (kept.size === 0) return undefined; + taskNames = [...kept]; + } + } + const cap = + nTasks === undefined + ? dataset.n_tasks + : dataset.n_tasks === undefined + ? nTasks + : Math.min(dataset.n_tasks, nTasks); + return { + ...dataset, + ...(taskNames ? { task_names: taskNames } : {}), + ...(cap === undefined ? {} : { n_tasks: cap }), + }; +} diff --git a/examples/agent-eval/src/test/atif.test.ts b/examples/agent-eval/src/test/atif.test.ts new file mode 100644 index 0000000000..e22e044524 --- /dev/null +++ b/examples/agent-eval/src/test/atif.test.ts @@ -0,0 +1,136 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AgentStep, ChatMessage } from "@workglow/ai"; +import { describe, expect, it } from "vitest"; +import { toAtifTrajectory } from "../agent/atif"; +import type { AgentRunSummary } from "../agent/runSummary"; +import { summarizeUsage, tallyTools } from "../agent/runSummary"; + +describe("toAtifTrajectory", () => { + it("counts image tokens in a step's prompt and cached figures, as the totals do", () => { + const steps: AgentStep[] = [ + { + round: 1, + startedAt: "2026-10-11T00:00:00.000Z", + modelMs: 10, + durationMs: 10, + attempts: 1, + text: "done", + tools: [], + usage: { + input: 100, + imageInput: 40, + cached: 20, + imageCached: 5, + cacheWrite: 3, + output: 7, + reasoning: undefined, + total: undefined, + extra: undefined, + }, + costUsd: undefined, + }, + ]; + const messages: ChatMessage[] = [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { role: "assistant", content: [{ type: "text", text: "done" }] }, + ]; + const usage = summarizeUsage(steps); + const summary = { model: "m", usage, costUsd: undefined } as unknown as AgentRunSummary; + + const trajectory = toAtifTrajectory("s", "0.1.0", messages, steps, summary); + + expect(trajectory.steps[1]?.metrics).toEqual({ + prompt_tokens: 168, + completion_tokens: 7, + cached_tokens: 25, + cost_usd: null, + }); + expect(trajectory.steps[1]?.metrics?.prompt_tokens).toBe( + trajectory.final_metrics.total_prompt_tokens + ); + expect(trajectory.steps[1]?.metrics?.cached_tokens).toBe( + trajectory.final_metrics.total_cached_tokens + ); + }); +}); + +describe("toAtifTrajectory metrics", () => { + it("gives a round that reported only a cost no token metrics", () => { + const steps = [ + { + round: 1, + startedAt: "2026-10-11T00:00:00.000Z", + modelMs: 1, + durationMs: 1, + attempts: 1, + text: "done", + tools: [], + usage: { costUsd: 0.01 }, + costUsd: 0.01, + }, + ] as unknown as AgentStep[]; + const messages: ChatMessage[] = [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { role: "assistant", content: [{ type: "text", text: "done" }] }, + ]; + const summary = { + model: "m", + usage: summarizeUsage(steps), + costUsd: 0.01, + } as unknown as AgentRunSummary; + const trajectory = toAtifTrajectory("s", "0.1.0", messages, steps, summary); + expect(trajectory.steps[1]).not.toHaveProperty("metrics"); + }); +}); + +describe("summarizeUsage", () => { + it("counts a round whose usage carries no token count as unreported, not as zero", () => { + const steps = [ + { usage: { input: 10, output: 2 } }, + { usage: { costUsd: 0.01 } }, + { usage: undefined }, + ] as unknown as AgentStep[]; + const usage = summarizeUsage(steps); + expect(usage.unreportedRounds).toBe(2); + expect(usage.inputTokens).toBe(10); + }); +}); + +describe("summarizeUsage cache counts", () => { + it("counts a round with tokens but no cache figure as cache-unreported, not as zero", () => { + const steps = [ + { usage: { input: 10, output: 2, cached: 4 } }, + { usage: { input: 10, output: 2 } }, + ] as unknown as AgentStep[]; + const usage = summarizeUsage(steps); + expect(usage.unreportedRounds).toBe(0); + expect(usage.cacheUnreportedRounds).toBe(1); + const summary = { model: "m", usage, costUsd: undefined } as unknown as AgentRunSummary; + expect(toAtifTrajectory("s", "0.1.0", [], [], summary).final_metrics.total_cached_tokens).toBe( + null + ); + }); +}); + +describe("tallyTools", () => { + it("counts a tool named after an Object.prototype key without touching the prototype", () => { + const steps = [ + { + tools: [ + { name: "__proto__", isError: true }, + { name: "constructor", isError: false }, + ], + }, + ] as unknown as AgentStep[]; + const tally = tallyTools(steps); + expect(tally.byName["__proto__"]).toEqual({ calls: 1, errors: 1 }); + expect(tally.byName["constructor"]).toEqual({ calls: 1, errors: 0 }); + expect(({} as Record).calls).toBeUndefined(); + expect(JSON.parse(JSON.stringify(tally)).byName.constructor).toEqual({ calls: 1, errors: 0 }); + }); +}); diff --git a/examples/agent-eval/src/test/compare.test.ts b/examples/agent-eval/src/test/compare.test.ts new file mode 100644 index 0000000000..0c6a6ed952 --- /dev/null +++ b/examples/agent-eval/src/test/compare.test.ts @@ -0,0 +1,444 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import type { TrialRecord } from "../compare/harborJobs"; +import { classifyTrial, labelArms, loadTrials } from "../compare/harborJobs"; +import { buildReport, renderMarkdown } from "../compare/report"; +import { mcnemarExactP, passAtK, wilsonInterval } from "../compare/stats"; + +describe("stats", () => { + it("bounds a Wilson interval inside [0, 1]", () => { + const all = wilsonInterval(5, 5); + expect(all.high).toBe(1); + expect(all.low).toBeGreaterThan(0.5); + expect(wilsonInterval(0, 0)).toEqual({ low: 0, high: 0 }); + }); + + it("computes unbiased pass@k", () => { + expect(passAtK(5, 0, 1)).toBe(0); + expect(passAtK(5, 5, 3)).toBe(1); + expect(passAtK(4, 1, 1)).toBeCloseTo(0.25); + expect(passAtK(4, 1, 2)).toBeCloseTo(0.5); + }); + + it("gives an exact two-sided McNemar p-value", () => { + expect(mcnemarExactP(0, 0)).toBe(1); + expect(mcnemarExactP(5, 5)).toBe(1); + // 8 discordant pairs all one way: 2 * 0.5^8. + expect(mcnemarExactP(8, 0)).toBeCloseTo(2 / 256); + }); +}); + +describe("classifyTrial", () => { + it("reads an API error reporting an over-long prompt as a context overflow", () => { + expect( + classifyTrial(0, "ApiError", "maximum context length exceeded", true, undefined).failure + ).toBe("context-overflow"); + expect( + classifyTrial(0, "ApiError", "rate limit: too many tokens per minute", true, undefined) + .failure + ).toBe("rate-limit"); + }); + + it("believes a reward even after an agent timeout", () => { + expect(classifyTrial(1, "AgentTimeoutError", "", true, undefined).status).toBe("pass"); + }); + + it("calls an exception before the agent ran blocked", () => { + expect( + classifyTrial(undefined, "RuntimeError", "docker compose failed", false, undefined).status + ).toBe("blocked"); + }); + + it("names failure modes", () => { + expect(classifyTrial(0, "AgentTimeoutError", "", true, undefined).failure).toBe( + "agent-timeout" + ); + expect(classifyTrial(0, undefined, undefined, true, undefined).failure).toBe("wrong-answer"); + const overflow = { + outcome: "error", + error: "prompt is too long: 210000 tokens > 200000 maximum", + rounds: 40, + retries: 0, + tools: { calls: 0, errors: 0, byName: {} }, + finalHistoryChars: 0, + }; + expect(classifyTrial(0, "NonZeroAgentExitCodeError", "exit 1", true, overflow).failure).toBe( + "context-overflow" + ); + expect( + classifyTrial(0, undefined, undefined, true, { + ...overflow, + outcome: "max-rounds", + error: undefined, + }).failure + ).toBe("max-rounds"); + expect(classifyTrial(0, "ApiRateLimitError", "429", true, undefined).failure).toBe( + "rate-limit" + ); + }); + + it("reads a Workglow provider failure from its own error, not the exit code", () => { + const failed = (error: string): any => ({ ...base, outcome: "error", error }); + const base = { + rounds: 3, + retries: 2, + tools: { calls: 0, errors: 0, byName: {} }, + finalHistoryChars: 0, + }; + const exit = "NonZeroAgentExitCodeError"; + expect(classifyTrial(0, exit, "exit 1", true, failed("429 Too Many Requests")).failure).toBe( + "rate-limit" + ); + expect(classifyTrial(0, exit, "exit 1", true, failed("503 Service Unavailable")).failure).toBe( + "provider-error" + ); + expect(classifyTrial(0, exit, "exit 1", true, failed("fetch failed")).failure).toBe( + "provider-error" + ); + expect( + classifyTrial(0, exit, "exit 1", true, failed("TypeError: x is undefined")).failure + ).toBe("agent-crash"); + }); +}); + +function trial( + task: string, + agent: string, + pass: boolean, + extra: Partial = {} +): TrialRecord { + return { + job: "job", + trial: `${task}-${agent}-${Math.random()}`, + task, + arm: agent, + kwargs: {}, + agent, + agentVersion: "1", + model: "anthropic/m", + reward: pass ? 1 : 0, + status: pass ? "pass" : "fail", + failure: pass ? undefined : "wrong-answer", + exceptionType: undefined, + exceptionMessage: undefined, + inputTokens: 100, + cacheTokens: 10, + outputTokens: 5, + costUsd: 0.01, + agentSec: 30, + setupSec: 5, + workglow: undefined, + ...extra, + }; +} + +describe("labelArms", () => { + it("shows only the kwargs that differ between configs of one agent", () => { + const labelled = labelArms([ + trial("t", "workglow", true, { kwargs: { max_rounds: 100, effort: "high" } }), + trial("t", "workglow", true, { + kwargs: { max_rounds: 100, effort: "high", tool_concurrency: 8 }, + }), + trial("t", "pi", true, { kwargs: { max_turns: 100, thinking: "high" } }), + ]); + expect(labelled.map((t) => t.arm)).toEqual(["workglow", "workglow[tool_concurrency=8]", "pi"]); + }); + + it("never gives two different settings one label", () => { + const labelled = labelArms([ + trial("t", "workglow", true, { + kwargs: { append_system_prompt: "hello,tool_concurrency=8" }, + }), + trial("t", "workglow", true, { + kwargs: { append_system_prompt: "hello", tool_concurrency: 8 }, + }), + trial("t", "workglow", true, { kwargs: { append_system_prompt: "8" } }), + trial("t", "workglow", true, { kwargs: { append_system_prompt: 8 } }), + ]); + expect(new Set(labelled.map((t) => t.arm)).size).toBe(4); + expect(labelled[1]!.arm).toBe("workglow[append_system_prompt=hello,tool_concurrency=8]"); + }); + + it("keeps two agent builds apart", () => { + const labelled = labelArms([ + trial("t", "workglow", true, { kwargs: { bundle_path: "/a/workglow-agent.mjs" } }), + trial("t", "workglow", false, { kwargs: { bundle_path: "/b/workglow-agent.mjs" } }), + trial("t", "opencode", true, { kwargs: { version: "2.0.0" } }), + trial("t", "opencode", true, { kwargs: { version: "2.1.0" } }), + ]); + expect(new Set(labelled.map((t) => t.arm)).size).toBe(4); + }); + + it("keeps a trial whose model is unknown in its agent's arm", () => { + const labelled = labelArms([ + trial("t", "pi", true), + trial("u", "pi", false, { model: undefined }), + trial("t", "opencode", true, { model: "openai/x" }), + ]); + expect(labelled.map((t) => t.arm)).toEqual([ + "pi@anthropic/m", + "pi@anthropic/m", + "opencode@openai/x", + ]); + }); + + it("adds the model when the comparison spans models", () => { + const labelled = labelArms([ + trial("t", "pi", true), + trial("t", "pi", true, { model: "openai/x" }), + ]); + expect(labelled.map((t) => t.arm)).toEqual(["pi@anthropic/m", "pi@openai/x"]); + }); +}); + +describe("buildReport", () => { + const trials = [ + trial("a", "workglow", true), + trial("b", "workglow", false), + trial("c", "workglow", false, { status: "blocked", failure: undefined, reward: undefined }), + trial("a", "opencode", true), + trial("b", "opencode", true), + trial("c", "opencode", true), + ]; + + it("summarizes each arm with blocked trials kept out of the rate", () => { + const report = buildReport(trials); + const workglow = report.arms.find((arm) => arm.arm === "workglow")!; + expect(workglow.blocked).toBe(1); + expect(workglow.passRate).toBe(0.5); + expect(workglow.costUsd).toBeCloseTo(0.02); + expect(workglow.failures).toEqual({ "wrong-answer": 1 }); + expect(report.arms[0]!.arm).toBe("opencode"); + }); + + it("reports no cost per pass when a trial went unpriced", () => { + const report = buildReport([ + trial("a", "pi", true), + trial("b", "pi", true, { costUsd: undefined }), + trial("a", "opencode", true), + ]); + expect(report.arms.find((arm) => arm.arm === "pi")!.costPerPass).toBeUndefined(); + expect(report.arms.find((arm) => arm.arm === "opencode")!.costPerPass).toBeCloseTo(0.01); + }); + + it("shows no rate for an arm none of whose trials ran, and ranks it last", () => { + const blocked = { status: "blocked" as const, failure: undefined, reward: undefined }; + const report = buildReport([ + trial("a", "pi", false), + trial("a", "opencode", false, blocked), + trial("b", "opencode", false, blocked), + ]); + expect(report.arms.map((arm) => arm.arm)).toEqual(["pi", "opencode"]); + expect(report.arms[1]).toMatchObject({ + passRate: undefined, + passRateCi: undefined, + passAt1: undefined, + }); + expect(renderMarkdown(report)).toMatch( + /\| opencode \| anthropic\/m \| 2 \| 0 \| 2 \| — \| — \|/ + ); + }); + + it("marks token totals that leave out trials which reported none", () => { + const report = buildReport([ + trial("a", "pi", true), + trial("b", "pi", true, { inputTokens: undefined }), + ]); + expect(report.arms[0]!.usageMissingTrials).toBe(1); + expect( + buildReport([trial("a", "pi", true, { cacheTokens: undefined })]).arms[0]!.usageMissingTrials + ).toBe(1); + expect(renderMarkdown(report)).toContain("† some trials reported no token counts"); + }); + + it("pairs arms on the tasks both ran", () => { + const pair = buildReport(trials).pairs[0]!; + expect(pair.commonTasks).toBe(2); + expect(pair.aBetter).toEqual(["b"]); + expect(pair.bBetter).toEqual([]); + }); + + it("counts a tie short of every attempt apart from a task both arms always passed", () => { + const pair = buildReport([ + trial("a", "workglow", true), + trial("a", "workglow", true), + trial("a", "opencode", true), + trial("a", "opencode", true), + trial("b", "workglow", true), + trial("b", "workglow", false), + trial("b", "opencode", false), + trial("b", "opencode", true), + ]).pairs[0]!; + expect(pair.bothPass).toBe(1); + expect(pair.partialTie).toBe(1); + expect(pair.neither).toBe(0); + }); + + it("keeps an arm or a tool named __proto__ as data, touching no prototype", () => { + const report = buildReport([ + trial("t", "__proto__", true), + trial("t", "workglow", false, { + workglow: { + outcome: "answered", + rounds: 1, + retries: 0, + tools: { + calls: 1, + errors: 1, + byName: JSON.parse('{"__proto__":{"calls":1,"errors":1}}'), + }, + maxHistoryChars: 10, + } as any, + }), + ]); + const parsed = JSON.parse(JSON.stringify(report)); + expect(Object.keys(parsed.tasks[0].cells).sort()).toEqual(["__proto__", "workglow"]); + expect(({} as Record).calls).toBeUndefined(); + }); + + it("lists a task one arm has no result for, so the gap shows", () => { + const markdown = renderMarkdown( + buildReport([ + trial("both", "workglow", true), + trial("both", "opencode", true), + trial("only-one", "workglow", true), + ]) + ); + expect(markdown).toMatch(/\| only-one \| · \| ✓ \|/); + expect(markdown).not.toMatch(/\| both \|/); + }); + + it("renders the tables and the disagreements", () => { + const markdown = renderMarkdown(buildReport(trials)); + expect(markdown).toContain("| opencode | anthropic/m |"); + expect(markdown).toContain("**opencode** better on: b"); + expect(markdown).toMatch(/\| b \| ✓ \| ✗ wrong-answer \|/); + }); +}); + +describe("loadTrials", () => { + let dir: string | undefined; + afterEach(() => { + if (dir) rmSync(dir, { recursive: true, force: true }); + }); + + it("reads a trial that failed before it had any agent logs", () => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-jobs-")); + const trialDir = join(dir, "job-1", "hello-world__abc"); + mkdirSync(trialDir, { recursive: true }); + writeFileSync(join(dir, "job-1", "result.json"), JSON.stringify({ n_total_trials: 1 })); + writeFileSync( + join(trialDir, "result.json"), + JSON.stringify({ + trial_name: "hello-world__abc", + task_name: "harbor/hello-world", + config: { + agent: { + name: "workglow", + import_path: "workglow_agent:WorkglowAgent", + model_name: "anthropic/m", + kwargs: {}, + }, + }, + exception_info: { exception_type: "EnvironmentStartTimeoutError", exception_message: "" }, + }) + ); + const trials = loadTrials([dir]); + expect(trials).toHaveLength(1); + expect(trials[0]!.status).toBe("blocked"); + // No agent_info yet: the configured model still names its arm. + expect(trials[0]!.model).toBe("anthropic/m"); + }); + + it("reads Harbor trial results from a jobs directory", () => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-jobs-")); + const trialDir = join(dir, "job-1", "hello-world__abc"); + mkdirSync(join(trialDir, "agent"), { recursive: true }); + writeFileSync(join(dir, "job-1", "result.json"), JSON.stringify({ n_total_trials: 1 })); + writeFileSync( + join(trialDir, "result.json"), + JSON.stringify({ + trial_name: "hello-world__abc", + task_name: "harbor/hello-world", + config: { + agent: { import_path: "workglow_agent:WorkglowAgent", kwargs: { max_rounds: 100 } }, + }, + agent_info: { + name: "workglow", + version: "0.1.0", + model_info: { name: "m", provider: "anthropic" }, + }, + agent_result: { + n_input_tokens: 3062, + n_cache_tokens: 0, + n_output_tokens: 72, + cost_usd: 0.0068, + metadata: { + workglow: { + outcome: "answered", + rounds: 3, + retries: 0, + tools: { calls: 2, errors: 0, byName: { bash: { calls: 2, errors: 0 } } }, + finalHistoryChars: 755, + }, + }, + }, + verifier_result: { rewards: { reward: 1 } }, + exception_info: null, + agent_execution: { + started_at: "2026-10-06T17:59:41Z", + finished_at: "2026-10-06T17:59:43Z", + }, + }) + ); + const [record] = loadTrials([dir]); + expect(record).toMatchObject({ + job: "job-1", + task: "harbor/hello-world", + arm: "workglow", + model: "anthropic/m", + status: "pass", + inputTokens: 3062, + agentSec: 2, + }); + expect(record!.workglow?.rounds).toBe(3); + }); + + it("adds opencode's reasoning tokens to its output, as the other harnesses report them", () => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-jobs-")); + const trialDir = join(dir, "job-2", "regex-log__xyz"); + mkdirSync(join(trialDir, "agent"), { recursive: true }); + writeFileSync( + join(trialDir, "result.json"), + JSON.stringify({ + trial_name: "regex-log__xyz", + task_name: "regex-log", + agent_info: { name: "opencode", model_info: { name: "m", provider: "deepseek" } }, + agent_result: { n_input_tokens: 100, n_output_tokens: 40, cost_usd: 0.01 }, + verifier_result: { rewards: { reward: 1 } }, + agent_execution: { + started_at: "2026-10-07T00:00:00Z", + finished_at: "2026-10-07T00:00:05Z", + }, + }) + ); + writeFileSync( + join(trialDir, "agent", "opencode.txt"), + [ + JSON.stringify({ type: "step_start" }), + JSON.stringify({ type: "step_finish", part: { tokens: { output: 10, reasoning: 25 } } }), + "not json", + JSON.stringify({ type: "step_finish", part: { tokens: { output: 30, reasoning: 5 } } }), + ].join("\n") + ); + expect(loadTrials([dir])[0]!.outputTokens).toBe(70); + }); +}); diff --git a/examples/agent-eval/src/test/jobConfig.test.ts b/examples/agent-eval/src/test/jobConfig.test.ts new file mode 100644 index 0000000000..2b15c239ed --- /dev/null +++ b/examples/agent-eval/src/test/jobConfig.test.ts @@ -0,0 +1,206 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from "vitest"; +import { parseEffort, resolveAgentModel } from "../agent/models"; +import { buildHarborJob, parityKwargs, parseArm } from "../run/jobConfig"; +import type { SuiteDataset } from "../run/suites"; +import { narrowDataset } from "../run/suites"; +import { getSuite } from "../run/suites"; + +const parity = { + maxTurns: 50, + effort: "high" as const, + opencodeVariant: "high", + piModelApi: undefined, +}; + +describe("parseArm", () => { + it("parses a bare harness and one with kwargs", () => { + expect(parseArm("pi")).toEqual({ harness: "pi", kwargs: {} }); + expect(parseArm("workglow:tool_concurrency=8,append_system_prompt=be brief")).toEqual({ + harness: "workglow", + kwargs: { tool_concurrency: 8, append_system_prompt: "be brief" }, + }); + }); + + it("refuses an unknown harness", () => { + expect(() => parseArm("aider")).toThrow(/unknown harness/); + }); + + it("keeps commas inside a JSON value or a sentence", () => { + expect( + parseArm('opencode:opencode_config={"a":{"b":1,"c":2}},append_system_prompt=Use X, then Y') + ).toEqual({ + harness: "opencode", + kwargs: { opencode_config: { a: { b: 1, c: 2 } }, append_system_prompt: "Use X, then Y" }, + }); + }); + + it("keeps a quoted value whole even where a comma and key= fall inside it", () => { + expect( + parseArm('workglow:append_system_prompt="When x=a, set y=b",tool_concurrency=4') + ).toEqual({ + harness: "workglow", + kwargs: { append_system_prompt: "When x=a, set y=b", tool_concurrency: 4 }, + }); + expect(parseArm('opencode:opencode_config={"note":"p=1, q=2"}').kwargs).toEqual({ + opencode_config: { note: "p=1, q=2" }, + }); + }); + + it("refuses a pair without = and text after a quoted value", () => { + expect(() => parseArm("workglow:concise")).toThrow(/must be key=value/); + expect(() => parseArm('workglow:append_system_prompt="a" b')).toThrow(/text after its value/); + expect(() => + parseArm('workglow:append_system_prompt="When x=a, set y=b,tool_concurrency=4') + ).toThrow(/never closed/); + }); +}); + +describe("parityKwargs", () => { + it("holds every harness to the same turn cap and reasoning level", () => { + expect(parityKwargs("workglow", parity)).toEqual({ max_rounds: 50, effort: "high" }); + expect(parityKwargs("pi", parity)).toEqual({ max_turns: 50, thinking: "high" }); + expect(parityKwargs("opencode", parity)).toEqual({ + opencode_config: { agent: { build: { steps: 50 } } }, + variant: "high", + }); + }); +}); + +describe("buildHarborJob", () => { + it("puts every arm in one job, letting an arm's own kwargs win", () => { + const job = buildHarborJob({ + jobName: "j", + jobsDir: "./jobs", + model: "anthropic/claude-sonnet-5-5", + arms: [parseArm("workglow"), parseArm("workglow:max_rounds=10"), parseArm("opencode")], + datasets: getSuite("smoke").datasets, + parity, + attempts: 3, + concurrency: 2, + timeoutMultiplier: undefined, + }); + expect(job).toMatchObject({ job_name: "j", n_attempts: 3, n_concurrent_trials: 2 }); + // Only infrastructure failures are retried; an agent's own failure never is. + expect(job.retry).toEqual({ + max_retries: 2, + include_exceptions: [ + "RuntimeError", + "EnvironmentStartTimeoutError", + "AgentSetupTimeoutError", + ], + }); + const agents = job.agents as Array>; + expect(agents[0]).toMatchObject({ + import_path: "workglow_agent:WorkglowAgent", + kwargs: { max_rounds: 50 }, + }); + expect(agents[1]).toMatchObject({ kwargs: { max_rounds: 10 } }); + expect(agents[2]).toMatchObject({ + name: "opencode", + model_name: "anthropic/claude-sonnet-5-5", + }); + expect(job.datasets).toHaveLength(2); + }); +}); + +describe("resolveAgentModel", () => { + it("maps provider prefixes the way the other harnesses spell them", () => { + expect(resolveAgentModel("anthropic/claude-sonnet-5-5", { env: {} })).toMatchObject({ + provider: "ANTHROPIC", + provider_config: { model_name: "claude-sonnet-5-5" }, + }); + expect(resolveAgentModel("google/gemini-3-pro", { env: {} }).provider).toBe("GOOGLE_GEMINI"); + expect(resolveAgentModel("openrouter/qwen/qwen3-coder", { env: {} })).toMatchObject({ + provider: "OPENROUTER", + provider_config: { model_name: "qwen/qwen3-coder" }, + }); + }); + + it("takes a custom endpoint from the provider's base URL variable", () => { + expect( + resolveAgentModel("anthropic/m", { env: { ANTHROPIC_BASE_URL: "http://127.0.0.1:9" } }) + .provider_config + ).toMatchObject({ base_url: "http://127.0.0.1:9", trustedBaseUrl: true }); + }); + + it("refuses an id without a provider", () => { + expect(() => resolveAgentModel("claude-sonnet-5-5", { env: {} })).toThrow(/provider\/model/); + expect(() => resolveAgentModel("mystery/m", { env: {} })).toThrow(/unsupported provider/); + }); +}); + +describe("parseEffort", () => { + it("accepts pi's names for reasoning levels", () => { + expect(parseEffort("off")).toBe("none"); + expect(parseEffort("xhigh")).toBe("extra"); + expect(parseEffort("high")).toBe("high"); + expect(parseEffort(undefined)).toBeUndefined(); + expect(() => parseEffort("turbo")).toThrow(/unknown effort/); + }); +}); + +describe("narrowDataset", () => { + const lite = { + name: "polyglot", + version: "1", + task_names: ["polyglot_python_*", "polyglot_javascript_*"], + }; + + it("keeps only the tasks both --tasks and the suite allow", () => { + expect(narrowDataset(lite, ["polyglot_python_bank*"], undefined)!.task_names).toEqual([ + "polyglot_python_bank*", + ]); + expect(narrowDataset(lite, ["polyglot_*"], undefined)!.task_names).toEqual(lite.task_names); + expect(narrowDataset(lite, ["polyglot_go_*"], undefined)).toBeUndefined(); + }); + + it("keeps a glob that overlaps the suite's without falling inside it", () => { + expect(narrowDataset(lite, ["*_001"], undefined)!.task_names).toEqual([ + "polyglot_python_*_001", + "polyglot_python_001", + "polyglot_javascript_*_001", + "polyglot_javascript_001", + ]); + expect(narrowDataset(lite, ["polyglot_python_*"], undefined)!.task_names).toEqual([ + "polyglot_python_*", + ]); + expect(narrowDataset(lite, ["polyglot_python_bank"], undefined)!.task_names).toEqual([ + "polyglot_python_bank", + ]); + }); + + it("keeps a ?-filter already inside one of the suite's prefixes", () => { + expect(narrowDataset(lite, ["polyglot_python_?ank"], undefined)!.task_names).toEqual([ + "polyglot_python_?ank", + ]); + expect(narrowDataset(lite, ["polyglot_go_???"], undefined)).toBeUndefined(); + }); + + it("refuses an overlap it cannot write down rather than dropping it", () => { + expect(() => narrowDataset(lite, ["polyglot_*_bank*"], undefined)).toThrow( + /cannot be narrowed/ + ); + }); + + it("refuses a character class rather than finding no overlap", () => { + expect(() => narrowDataset(lite, ["polyglot_[pj]ython_*"], undefined)).toThrow( + /cannot be narrowed/ + ); + expect(() => narrowDataset(lite, ["polyglot_python_[ab]ank"], undefined)).toThrow( + /cannot be narrowed/ + ); + }); + + it("lowers a suite's cap but never raises it", () => { + const swe = { name: "swebench-verified", version: "1.0", n_tasks: 50 }; + expect(narrowDataset(swe, undefined, 10)!.n_tasks).toBe(10); + expect(narrowDataset(swe, undefined, 100)!.n_tasks).toBe(50); + expect(narrowDataset({ name: "x", version: "1" }, undefined, 7)!.n_tasks).toBe(7); + }); +}); diff --git a/examples/agent-eval/src/test/runCodingAgent.test.ts b/examples/agent-eval/src/test/runCodingAgent.test.ts new file mode 100644 index 0000000000..5355867600 --- /dev/null +++ b/examples/agent-eval/src/test/runCodingAgent.test.ts @@ -0,0 +1,386 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdtempSync, readFileSync, rmSync } from "node:fs"; +import type { Server } from "node:http"; +import { createServer } from "node:http"; +import type { AddressInfo } from "node:net"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterAll, beforeAll, describe, expect, it } from "vitest"; +import { toAtifTrajectory } from "../agent/atif"; +import { resolveAgentModel } from "../agent/models"; +import { registerAgentProvider } from "../agent/providers"; +import type { AgentRunEvent } from "../agent/runCodingAgent"; +import { runCodingAgent } from "../agent/runCodingAgent"; +import { decideReply, startAnthropicMock } from "../mock/anthropicMock"; + +describe("decideReply", () => { + const options = { port: 0, commands: ["echo one", "echo two"] }; + const tools = [ + { + name: "bash", + input_schema: { + properties: { command: { type: "string" }, description: { type: "string" } }, + required: ["command", "description"], + }, + }, + ]; + + it("runs the next command through the harness's bash tool, filling its required fields", () => { + expect(decideReply({ tools, messages: [{ role: "user", content: "x" }] }, options)).toEqual({ + kind: "tool", + name: "bash", + input: { command: "echo one", description: "Run the next step" }, + }); + const later = decideReply( + { + tools, + messages: [ + { role: "user", content: "x" }, + { role: "assistant", content: [] }, + { role: "user", content: [] }, + ], + }, + options + ); + expect(later).toMatchObject({ kind: "tool", input: { command: "echo two" } }); + }); + + it("answers in text once the script is spent, or when no bash tool is offered", () => { + const spent = Array.from({ length: 4 }, (_, i) => ({ + role: i % 2 ? "assistant" : "user", + content: [], + })); + expect(decideReply({ tools, messages: spent }, options).kind).toBe("text"); + expect(decideReply({ messages: [] }, options).kind).toBe("text"); + }); +}); + +/** + * The whole loop against the scripted Anthropic API: the real provider, the + * real AgentTask and the real tools, with no key and no network. + */ +describe("startAnthropicMock", () => { + it("rejects when its port is already taken, rather than throwing later", async () => { + const first = await startAnthropicMock({ port: 0, commands: [] }); + try { + const { port } = first.address() as AddressInfo; + await expect(startAnthropicMock({ port, commands: [] })).rejects.toThrow(/EADDRINUSE/); + } finally { + first.close(); + } + }); +}); + +describe("runCodingAgent against the mock model", () => { + let server: Server; + let work: string; + const previous = { url: process.env.ANTHROPIC_BASE_URL, key: process.env.ANTHROPIC_API_KEY }; + + beforeAll(async () => { + server = await startAnthropicMock({ + port: 0, + commands: ["printf 'Hello, world!\\n' > hello.txt", "cat hello.txt", "exit 4"], + }); + const { port } = server.address() as AddressInfo; + process.env.ANTHROPIC_BASE_URL = `http://127.0.0.1:${port}`; + process.env.ANTHROPIC_API_KEY = "mock"; + await registerAgentProvider("ANTHROPIC"); + work = mkdtempSync(join(tmpdir(), "agent-eval-run-")); + }); + + afterAll(() => { + server.close(); + rmSync(work, { recursive: true, force: true }); + process.env.ANTHROPIC_BASE_URL = previous.url; + process.env.ANTHROPIC_API_KEY = previous.key; + if (previous.url === undefined) delete process.env.ANTHROPIC_BASE_URL; + if (previous.key === undefined) delete process.env.ANTHROPIC_API_KEY; + }); + + it("runs the tools, records every round, and prices the run", async () => { + const events: AgentRunEvent[] = []; + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: undefined, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + onEvent: (event) => events.push(event), + }); + + expect(readFileSync(join(work, "hello.txt"), "utf8")).toBe("Hello, world!\n"); + const { summary } = result; + expect(summary.outcome).toBe("answered"); + expect(summary.rounds).toBe(4); + expect(summary.tools).toEqual({ + calls: 3, + errors: 1, + byName: { bash: { calls: 3, errors: 1 } }, + }); + expect(summary.usage.promptTokens).toBeGreaterThan(0); + expect(summary.usage.outputTokens).toBe(4 * 24); + expect(summary.costUsd).toBeGreaterThan(0); + expect(events[0]?.type).toBe("start"); + expect(events.at(-1)?.type).toBe("finish"); + expect(events.filter((event) => event.type === "round")).toHaveLength(4); + + const trajectory = toAtifTrajectory("s", "0.1.0", result.messages, result.steps, summary); + expect(trajectory.steps.map((step) => step.source)).toEqual([ + "user", + "agent", + "agent", + "agent", + "agent", + ]); + expect(trajectory.steps[1]?.tool_calls?.[0]?.function_name).toBe("bash"); + expect(trajectory.steps[3]?.observation?.results[0]?.content).toMatch(/exited with code 4/); + expect(trajectory.final_metrics.total_steps).toBe(5); + }); + + it("counts the round and the retries of a call that never succeeded", async () => { + // Every request is refused as overloaded; `x-should-retry: false` keeps the SDK + // from retrying on its own, so each attempt is one the loop made. + const overloaded = createServer((_request, response) => { + response.writeHead(529, { "content-type": "application/json", "x-should-retry": "false" }); + response.end( + JSON.stringify({ type: "error", error: { type: "overloaded_error", message: "busy" } }) + ); + }); + await new Promise((resolve) => overloaded.listen(0, "127.0.0.1", resolve)); + const mockUrl = process.env.ANTHROPIC_BASE_URL; + process.env.ANTHROPIC_BASE_URL = `http://127.0.0.1:${(overloaded.address() as AddressInfo).port}`; + try { + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: 1, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + }); + expect(result.summary.outcome).toBe("error"); + expect(result.summary.rounds).toBe(1); + expect(result.summary.retries).toBe(1); + // The failed round has no step: its usage and cost are unknown, not zero. + expect(result.summary.costUsd).toBeUndefined(); + expect(result.summary.usage.unreportedRounds).toBe(1); + } finally { + process.env.ANTHROPIC_BASE_URL = mockUrl; + overloaded.close(); + } + }); + + it("does not count a retry the run was stopped before starting", async () => { + const overloaded = createServer((_request, response) => { + response.writeHead(529, { "content-type": "application/json", "x-should-retry": "false" }); + response.end( + JSON.stringify({ type: "error", error: { type: "overloaded_error", message: "busy" } }) + ); + }); + await new Promise((resolve) => overloaded.listen(0, "127.0.0.1", resolve)); + const mockUrl = process.env.ANTHROPIC_BASE_URL; + process.env.ANTHROPIC_BASE_URL = `http://127.0.0.1:${(overloaded.address() as AddressInfo).port}`; + try { + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: 5, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + // Inside the first retry's wait, which is at least a second. + stopAfterMs: 400, + }); + expect(result.summary.outcome).toBe("budget"); + expect(result.summary.rounds).toBe(1); + expect(result.summary.retries).toBe(0); + } finally { + process.env.ANTHROPIC_BASE_URL = mockUrl; + overloaded.close(); + } + }); + + it("stops outright at its hard limit, mid-call, and still returns its record", async () => { + // A provider that accepts the request and never answers: the loop's own + // budget would wait on it, since it stops only between rounds. + const silent = createServer(() => {}); + await new Promise((resolve) => silent.listen(0, "127.0.0.1", resolve)); + const mockUrl = process.env.ANTHROPIC_BASE_URL; + process.env.ANTHROPIC_BASE_URL = `http://127.0.0.1:${(silent.address() as AddressInfo).port}`; + const started = Date.now(); + try { + const events: AgentRunEvent[] = []; + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: undefined, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + stopAfterMs: 500, + onEvent: (event) => events.push(event), + }); + expect(Date.now() - started).toBeLessThan(10_000); + expect(result.summary.outcome).toBe("budget"); + expect(result.summary.rounds).toBe(1); + expect(result.summary.error).toMatch(/hard time limit/); + expect(events.at(-1)?.type).toBe("finish"); + } finally { + process.env.ANTHROPIC_BASE_URL = mockUrl; + silent.closeAllConnections(); + silent.close(); + } + }); + + it("stops at the round cap", async () => { + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 2, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: undefined, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: undefined, + }, + signal: new AbortController().signal, + }); + expect(result.summary.outcome).toBe("max-rounds"); + expect(result.summary.rounds).toBe(2); + expect(result.summary.finalText).toBe(""); + }); + + it("asks for one review before finishing when given a review prompt", async () => { + const result = await runCodingAgent({ + instruction: "Create hello.txt", + modelId: "anthropic/claude-sonnet-5-5", + model: resolveAgentModel("anthropic/claude-sonnet-5-5"), + tools: { + cwd: work, + spillDir: join(work, ".spill"), + defaultCommandTimeoutSec: 30, + images: true, + }, + settings: { + maxRounds: 10, + maxToolResultChars: 60_000, + maxHistoryChars: undefined, + toolConcurrency: undefined, + maxTokens: undefined, + temperature: undefined, + maxDurationMs: undefined, + roundTimeoutMs: undefined, + maxRoundRetries: undefined, + systemPromptAppend: undefined, + concise: true, + announceTimeLeft: false, + reviewPrompt: "Check your work.", + }, + signal: new AbortController().signal, + }); + expect(result.summary.outcome).toBe("answered"); + expect(result.summary.rounds).toBe(5); + expect( + result.messages.filter( + (m) => m.role === "user" && JSON.stringify(m.content).includes("Check your work.") + ) + ).toHaveLength(1); + }); +}); diff --git a/examples/agent-eval/src/test/runHarbor.test.ts b/examples/agent-eval/src/test/runHarbor.test.ts new file mode 100644 index 0000000000..afa3789703 --- /dev/null +++ b/examples/agent-eval/src/test/runHarbor.test.ts @@ -0,0 +1,50 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { missingBundle } from "../run/runHarbor"; + +describe("missingBundle", () => { + let dir: string; + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-bundle-")); + }); + afterEach(() => { + rmSync(dir, { recursive: true, force: true }); + }); + + const workglow = (kwargs: Record = {}) => ({ + import_path: "workglow_agent:WorkglowAgent", + kwargs, + }); + + it("accepts an arm's own bundle when the shared one was never built", () => { + const own = join(dir, "custom.mjs"); + writeFileSync(own, ""); + expect(missingBundle([workglow({ bundle_path: own })], join(dir, "absent.mjs"))).toBe( + undefined + ); + }); + + it("names an arm's own bundle that does not exist", () => { + const shared = join(dir, "shared.mjs"); + writeFileSync(shared, ""); + expect(missingBundle([workglow({ bundle_path: join(dir, "nope.mjs") })], shared)).toMatch( + /bundle_path .*nope\.mjs does not exist/ + ); + }); + + it("asks for a build when an arm relies on the shared bundle and it is absent", () => { + expect(missingBundle([workglow()], join(dir, "absent.mjs"))).toMatch(/build-agent/); + }); + + it("ignores arms that are not the workglow adapter", () => { + expect(missingBundle([{ import_path: "pi:Pi" }], join(dir, "absent.mjs"))).toBe(undefined); + }); +}); diff --git a/examples/agent-eval/src/test/tools.test.ts b/examples/agent-eval/src/test/tools.test.ts new file mode 100644 index 0000000000..3968d94821 --- /dev/null +++ b/examples/agent-eval/src/test/tools.test.ts @@ -0,0 +1,827 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { ToolCallError, ToolResultContent } from "@workglow/ai"; +import { execFileSync } from "node:child_process"; +import { + chmodSync, + existsSync, + mkdirSync, + mkdtempSync, + readdirSync, + readFileSync, + rmSync, + statSync, + symlinkSync, + truncateSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; +import { loadCredentialsFile } from "../agent/credentialsFile"; +import { PROBED_COMMANDS, probeCommands } from "../agent/environment"; +import { + commandTimeoutSec, + createBashTool, + MAX_TIMEOUT_SEC, + runCommand, + shellEnv, +} from "../agent/tools/bash"; +import type { CodingToolContext } from "../agent/tools/context"; +import { applyEdits, createEditTool, normalizeLine } from "../agent/tools/edit"; +import { codingSystemPrompt } from "../agent/systemPrompt"; +import { createReadTool, imageMimeType, MAX_READ_BYTES } from "../agent/tools/read"; +import { countLines, lineAt, truncateHead, truncateTail } from "../agent/tools/truncate"; +import { createWriteTool } from "../agent/tools/write"; + +let dir: string; +let context: CodingToolContext; +const signal = new AbortController().signal; +const call = { toolUseId: "t1", signal, deadline: undefined }; + +beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "agent-eval-tools-")); + context = { + cwd: dir, + spillDir: join(dir, ".spill"), + defaultCommandTimeoutSec: 10, + images: true, + }; +}); + +afterEach(() => { + rmSync(dir, { recursive: true, force: true }); +}); + +describe("truncate", () => { + // The split-based definitions the line walkers replaced, kept as the reference. + const reference = { + lines: (text: string) => (text.endsWith("\n") ? text.slice(0, -1) : text).split("\n"), + }; + + it("charges a newline only where the text has one", () => { + expect(truncateTail("abc", 100, 3)).toMatchObject({ text: "abc", truncated: false }); + expect(truncateTail("abc\n", 100, 3)).toMatchObject({ text: "bc\n", truncated: true }); + expect(truncateHead("abc", 100, 3)).toMatchObject({ text: "abc", truncated: false }); + expect(truncateHead("ab\ncd", 100, 5)).toMatchObject({ text: "ab\ncd", truncated: false }); + }); + + it("walks lines exactly as splitting them would, without building the array", () => { + const pieces = ["", "a", "bb", "\n", "\n\n", "ü", "x".repeat(30)]; + let seed = 7; + const next = () => (seed = (seed * 1103515245 + 12345) % 2 ** 31) / 2 ** 31; + for (let n = 0; n < 500; n++) { + let text = ""; + const parts = Math.floor(next() * 12); + for (let i = 0; i < parts; i++) text += pieces[Math.floor(next() * pieces.length)]; + const lines = reference.lines(text); + expect(countLines(text)).toBe(lines.length); + for (let line = 1; line <= lines.length; line++) + expect(lineAt(text, line)).toBe(lines[line - 1]); + for (const [maxLines, maxBytes] of [ + [3, 1000], + [1000, 20], + [2, 5], + ] as const) { + const head = truncateHead(text, maxLines, maxBytes); + expect(head.totalLines).toBe(lines.length); + expect(head.text.split("\n").slice(0, head.keptLines)).toEqual( + lines.slice(0, head.keptLines) + ); + const tail = truncateTail(text, maxLines, maxBytes); + expect(tail.totalLines).toBe(lines.length); + if (!tail.partialLine) { + expect(reference.lines(tail.text.length === 0 ? "" : tail.text)).toEqual( + lines.slice(lines.length - tail.keptLines) + ); + } + } + } + }); + + it("keeps the head of long text and says how much there was", () => { + const text = Array.from({ length: 10 }, (_, i) => `line ${i}`).join("\n"); + const head = truncateHead(text, 3, 1000); + expect(head.text).toBe("line 0\nline 1\nline 2"); + expect(head.truncated).toBe(true); + expect(head.totalLines).toBe(10); + }); + + it("counts a final newline as the end of the last line", () => { + const text = Array.from({ length: 5 }, (_, i) => `${i + 1}`).join("\n") + "\n"; + const tail = truncateTail(text, 2, 1000); + expect(tail.totalLines).toBe(5); + expect(tail.text).toBe("4\n5\n"); + expect(truncateHead(text, 10, 1000)).toMatchObject({ totalLines: 5, truncated: false }); + }); + + it("keeps the tail when the end is what matters", () => { + const text = Array.from({ length: 10 }, (_, i) => `line ${i}`).join("\n"); + expect(truncateTail(text, 2, 1000).text).toBe("line 8\nline 9"); + }); + + it("stops at the byte limit before the line limit", () => { + const text = ["aaaa", "bbbb", "cccc"].join("\n"); + expect(truncateHead(text, 100, 10).keptLines).toBe(2); + }); + + it("keeps the end of a last line longer than the whole budget", () => { + const tail = truncateTail(`${"x".repeat(100)}ERROR\n`, 100, 21); + expect(tail.text).toBe(`${"x".repeat(15)}ERROR\n`); + expect(tail.truncated).toBe(true); + expect(tail.partialLine).toBe(true); + }); + + it("cuts a long last line on a character boundary", () => { + // Each "é" is two bytes: an odd budget lands mid-character. + const tail = truncateTail("é".repeat(50), 100, 9); + expect(tail.text).toBe("é".repeat(4)); + expect(tail.partialLine).toBe(true); + }); +}); + +describe("edit", () => { + it("replaces a unique exact match", () => { + expect(applyEdits("a\nb\nc\n", [{ oldText: "b", newText: "B" }], "f").content).toBe( + "a\nB\nc\n" + ); + }); + + it("locates every edit against the original text", () => { + const result = applyEdits( + "one\ntwo\nthree\n", + [ + { oldText: "three", newText: "3" }, + { oldText: "one", newText: "1\none and a half" }, + ], + "f" + ); + expect(result.content).toBe("1\none and a half\ntwo\n3\n"); + }); + + it("reports an ambiguous match and applies nothing for it", () => { + const result = applyEdits("x\nx\n", [{ oldText: "x", newText: "y" }], "f"); + expect(result.content).toBe("x\nx\n"); + expect(result.failures).toEqual([expect.stringMatching(/^edits\[0\]: .*multiple occurrences/)]); + }); + + it("applies the first of two overlapping edits and reports the second", () => { + const result = applyEdits( + "abcdef", + [ + { oldText: "abcd", newText: "1" }, + { oldText: "cdef", newText: "2" }, + ], + "f" + ); + expect(result.content).toBe("1ef"); + expect(result.applied).toEqual([0]); + expect(result.failures).toEqual([ + expect.stringMatching(/^edits\[1\]: overlaps edits\[0\], which was applied\. Read the file/), + ]); + }); + + it("counts an edit whose text is already in place as unchanged, not as a failure", () => { + const result = applyEdits("a", [{ oldText: "a", newText: "a" }], "f"); + expect(result.unchanged).toEqual([0]); + expect(result.failures).toEqual([]); + }); + + it("finds no blank line in an empty file", () => { + const result = applyEdits("", [{ oldText: "\n", newText: "inserted" }], "f"); + expect(result.content).toBe(""); + expect(result.failures).toEqual([expect.stringMatching(/^edits\[0\]/)]); + }); + + it("counts overlapping fuzzy matches, so an ambiguous edit is refused", () => { + // Trailing spaces defeat the exact match; each "a" line pair matches twice. + const result = applyEdits("a\na\na\n", [{ oldText: "a \na ", newText: "x" }], "f"); + expect(result.failures).toEqual([expect.stringMatching(/found multiple places matching/)]); + }); + + it("searches fuzzily in time proportional to the file, not file × oldText", () => { + const content = "a\n".repeat(200_000); + // Thousands of lines that all match, then one that never does: a search that + // restarts the whole comparison at every line does 400 million of them. + const oldText = `${"a \n".repeat(2_000)}b`; + const started = Date.now(); + const result = applyEdits(content, [{ oldText, newText: "x" }], "f"); + expect(result.failures).toEqual([expect.stringMatching(/could not find oldText/)]); + expect(Date.now() - started).toBeLessThan(3_000); + }); + + it("reports an empty oldText", () => { + expect(applyEdits("a", [{ oldText: "", newText: "b" }], "f").failures).toEqual([ + expect.stringMatching(/empty/), + ]); + }); + + it("matches whole lines that differ only in trailing whitespace and typographic quotes", () => { + const content = "const s = “hi”; \nnext();\n"; + const result = applyEdits( + content, + [{ oldText: 'const s = "hi";\n', newText: 'const s = "bye";\n' }], + "f" + ); + expect(result.content).toBe('const s = "bye";\nnext();\n'); + expect(result.fuzzy).toBe(1); + }); + + it("does not take the end of a file for a blank line the oldText asks for", () => { + const result = applyEdits("foo \n", [{ oldText: "foo\n\n", newText: "bar\n" }], "f"); + expect(result.content).toBe("foo \n"); + expect(result.failures).toHaveLength(1); + // The same edit without the blank line still matches past the trailing spaces. + expect(applyEdits("foo \n", [{ oldText: "foo\n", newText: "bar\n" }], "f").content).toBe( + "bar\n" + ); + }); + + it("normalizes dashes and non-breaking spaces", () => { + expect(normalizeLine("a—b c ")).toBe("a-b c"); + }); + + it("says what to do when nothing matches", () => { + expect(applyEdits("abc", [{ oldText: "zzz", newText: "y" }], "f").failures).toEqual([ + expect.stringMatching(/Read the file again/), + ]); + }); + + it("writes the edits that apply and names the ones that did not", async () => { + writeFileSync(join(dir, "f.txt"), "one\ntwo\ntwo\n"); + await expect( + createEditTool(context).execute!( + { + path: "f.txt", + edits: [ + { oldText: "one", newText: "1" }, + { oldText: "two", newText: "2" }, + ], + }, + call + ) + ).rejects.toThrow(/Applied 1 of 2 edits[\s\S]*edits\[1\]: .*multiple occurrences/); + expect(readFileSync(join(dir, "f.txt"), "utf8")).toBe("1\ntwo\ntwo\n"); + }); + + it("applies concurrent edits through a symlink and its target, losing neither", async () => { + writeFileSync(join(dir, "real.txt"), "one\ntwo\n"); + symlinkSync(join(dir, "real.txt"), join(dir, "alias.txt")); + const edit = createEditTool(context); + await Promise.all([ + edit.execute!({ path: "real.txt", edits: [{ oldText: "one", newText: "1" }] }, call), + edit.execute!({ path: "alias.txt", edits: [{ oldText: "two", newText: "2" }] }, call), + ]); + expect(readFileSync(join(dir, "real.txt"), "utf8")).toBe("1\n2\n"); + }); + + it("refuses to edit a file that is not UTF-8, leaving its bytes alone", async () => { + const path = join(dir, "latin1.txt"); + writeFileSync(path, Buffer.from([0x61, 0xff])); + await expect( + createEditTool(context).execute!( + { path: "latin1.txt", edits: [{ oldText: "a", newText: "b" }] }, + call + ) + ).rejects.toThrow(/not UTF-8 text/); + expect([...readFileSync(path)]).toEqual([0x61, 0xff]); + }); + + it("refuses to edit a device, which has no end to read to", async () => { + await expect( + createEditTool(context).execute!( + { path: "/dev/zero", edits: [{ oldText: "a", newText: "b" }] }, + call + ) + ).rejects.toThrow(/not a regular file/); + }); + + it("refuses to edit a file too large to load", async () => { + const big = join(dir, "huge.dat"); + writeFileSync(big, ""); + truncateSync(big, MAX_READ_BYTES + 1); + await expect( + createEditTool(context).execute!( + { path: "huge.dat", edits: [{ oldText: "a", newText: "b" }] }, + call + ) + ).rejects.toThrow(/too large to edit here/); + }); + + it("tells a missing file from one it could not read", async () => { + const edits = [{ oldText: "a", newText: "b" }]; + await expect( + createEditTool(context).execute!({ path: "missing.txt", edits }, call) + ).rejects.toThrow(/File not found: .*missing\.txt\. Use write to create it\./); + // A path through a regular file fails with ENOTDIR: no file to create either. + writeFileSync(join(dir, "a-file"), "x"); + const unreadable = createEditTool(context).execute!({ path: "a-file/child", edits }, call); + await expect(unreadable).rejects.toThrow(/Could not read .*a-file\/child: /); + await expect(unreadable).rejects.not.toThrow(/Use write/); + // A directory is no file to edit at all. + mkdirSync(join(dir, "a-dir")); + await expect(createEditTool(context).execute!({ path: "a-dir", edits }, call)).rejects.toThrow( + /not a regular file/ + ); + }); + + it("says how many edits already matched when others in the call fail", async () => { + writeFileSync(join(dir, "p.txt"), "one\nsame\ntwo\ntwo\n"); + await expect( + createEditTool(context).execute!( + { + path: "p.txt", + edits: [ + { oldText: "one", newText: "1" }, + { oldText: "same", newText: "same" }, + { oldText: "two", newText: "2" }, + ], + }, + call + ) + ).rejects.toThrow(/Applied 1 of 3 edits to .*p\.txt; 1 already matched their newText/); + }); + + it("leaves the file byte-identical when every edit fails, CRLF included", async () => { + const path = join(dir, "all-failed.txt"); + writeFileSync(path, "a\r\nb\r\n"); + await expect( + createEditTool(context).execute!( + { path: "all-failed.txt", edits: [{ oldText: "not there", newText: "x" }] }, + call + ) + ).rejects.toThrow(/No edits applied/); + expect(readFileSync(path)).toEqual(Buffer.from("a\r\nb\r\n")); + }); + + it("keeps CRLF line endings and a BOM through the tool", async () => { + const path = join(dir, "crlf.txt"); + writeFileSync(path, "one\r\ntwo\r\n"); + const tool = createEditTool(context); + await tool.execute!( + { path: "crlf.txt", edits: [{ oldText: "one\ntwo", newText: "1\n2" }] }, + call + ); + expect(readFileSync(path, "utf8")).toBe("1\r\n2\r\n"); + }); + + it("writes a mixed file back in the ending of its first line break, as pi does", async () => { + writeFileSync(join(dir, "mixed.txt"), "a\nb\r\nc\n"); + await createEditTool(context).execute!( + { path: "mixed.txt", edits: [{ oldText: "c", newText: "C" }] }, + call + ); + expect(readFileSync(join(dir, "mixed.txt"), "utf8")).toBe("a\nb\nC\n"); + }); + + it("reports a missing file as a tool error", async () => { + const tool = createEditTool(context); + await expect( + tool.execute!({ path: "missing.txt", edits: [{ oldText: "a", newText: "b" }] }, call) + ).rejects.toBeInstanceOf(ToolCallError); + }); +}); + +describe("edit under concurrency", () => { + it("keeps both of two edits made to one file at the same time", async () => { + writeFileSync(join(dir, "both.txt"), "alpha\nbeta\n"); + const edit = createEditTool(context); + await Promise.all([ + edit.execute!({ path: "both.txt", edits: [{ oldText: "alpha", newText: "ALPHA" }] }, call), + edit.execute!({ path: "both.txt", edits: [{ oldText: "beta", newText: "BETA" }] }, call), + ]); + expect(readFileSync(join(dir, "both.txt"), "utf8")).toBe("ALPHA\nBETA\n"); + }); +}); + +describe("read and write", () => { + it("writes a file under new directories and reads it back", async () => { + await createWriteTool(context).execute!({ path: "a/b/c.txt", content: "hello\n" }, call); + expect(await createReadTool(context).execute!({ path: "a/b/c.txt" }, call)).toBe("hello"); + }); + + it("reports a path it cannot stat as unreadable, not as missing", async () => { + writeFileSync(join(dir, "plain.txt"), "x"); + // A file used as a directory fails with ENOTDIR: there, but not readable as asked. + await expect( + createReadTool(context).execute!({ path: "plain.txt/child" }, call) + ).rejects.toThrow(/Could not read .*plain\.txt\/child: /); + await expect(createReadTool(context).execute!({ path: "absent.txt" }, call)).rejects.toThrow( + /File not found: / + ); + }); + + it("refuses a file too large to load, before reading any of it", async () => { + const big = join(dir, "huge.log"); + writeFileSync(big, ""); + truncateSync(big, MAX_READ_BYTES + 1); + await expect(createReadTool(context).execute!({ path: "huge.log" }, call)).rejects.toThrow( + /too large to read whole\. Use bash/ + ); + }); + + it("refuses a path that is not a regular file", async () => { + await expect(createReadTool(context).execute!({ path: "/dev/zero" }, call)).rejects.toThrow( + "is not a regular file" + ); + }); + + it("refuses to write anything but a regular file, so a FIFO cannot hang it", async () => { + const fifo = join(dir, "pipe"); + execFileSync("mkfifo", [fifo]); + await expect( + createWriteTool(context).execute!({ path: "pipe", content: "x" }, call) + ).rejects.toThrow(/is not a regular file/); + await expect( + createWriteTool(context).execute!({ path: ".", content: "x" }, call) + ).rejects.toThrow(/is not a regular file/); + }); + + it("refuses an offset or limit below 1", async () => { + writeFileSync(join(dir, "two.txt"), "a\nb\n"); + const read = createReadTool(context); + await expect(read.execute!({ path: "two.txt", offset: 0 }, call)).rejects.toThrow( + "offset must be 1 or more" + ); + await expect(read.execute!({ path: "two.txt", limit: 0 }, call)).rejects.toThrow( + "limit must be 1 or more" + ); + }); + + it("counts a final newline as the end of the last line, not a line of its own", async () => { + writeFileSync(join(dir, "one.txt"), "hello\n"); + await expect( + createReadTool(context).execute!({ path: "one.txt", offset: 2 }, call) + ).rejects.toThrow("offset 2 is past the end of"); + }); + + it("cuts a line too long for the budget by bytes, not characters", async () => { + writeFileSync(join(dir, "wide.txt"), "😀".repeat(30_000)); + const out = (await createReadTool(context).execute!({ path: "wide.txt" }, call)) as string; + const head = out.split("\n\n[Line 1")[0]!; + expect(Buffer.byteLength(head, "utf8")).toBeLessThanOrEqual(50 * 1024); + expect(head).toBe("😀".repeat(12_800)); + }); + + it("pages a long file and names the next offset", async () => { + writeFileSync( + join(dir, "long.txt"), + Array.from({ length: 50 }, (_, i) => `L${i + 1}`).join("\n") + ); + const out = (await createReadTool(context).execute!( + { path: "long.txt", offset: 11, limit: 5 }, + call + )) as string; + expect(out.startsWith("L11\nL12\nL13\nL14\nL15")).toBe(true); + expect(out).toContain("Use offset=16 to continue"); + }); + + it("bounds a directory listing like a file", async () => { + mkdirSync(join(dir, "many")); + for (let i = 0; i < 2100; i++) writeFileSync(join(dir, "many", `f${i}`), ""); + const out = (await createReadTool(context).execute!({ path: "many" }, call)) as string; + expect(out).toContain( + "[Showing 2000 of 2100 entries, in directory order rather than by name. Use bash" + ); + expect(out.split("\n").filter((line) => /^f\d+$/.test(line))).toHaveLength(2000); + }); + + it("lists a directory instead of failing", async () => { + writeFileSync(join(dir, "x.txt"), ""); + const out = (await createReadTool(context).execute!({ path: "." }, call)) as string; + expect(out).toContain("x.txt"); + }); + + it("returns an image as an image the model can see", async () => { + const png = Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + Buffer.from([0, 0, 0, 0]), + ]); + writeFileSync(join(dir, "board.png"), png); + const out = await createReadTool(context).execute!({ path: "board.png" }, call); + expect(out).toBeInstanceOf(ToolResultContent); + const content = (out as ToolResultContent).content; + expect(content[0]).toMatchObject({ type: "text" }); + expect(content[1]).toEqual({ + type: "image", + mimeType: "image/png", + data: png.toString("base64"), + }); + }); + + it("refuses an image as binary when images are switched off", async () => { + writeFileSync( + join(dir, "board.png"), + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0]) + ); + await expect( + createReadTool({ ...context, images: false }).execute!({ path: "board.png" }, call) + ).rejects.toThrow(/binary/); + }); + + it("refuses a binary file", async () => { + writeFileSync(join(dir, "bin"), Buffer.from([1, 0, 2, 0])); + await expect(createReadTool(context).execute!({ path: "bin" }, call)).rejects.toThrow(/binary/); + }); +}); + +describe("bash", () => { + it("returns combined output", async () => { + const out = await createBashTool(context).execute!({ command: "echo out; echo err >&2" }, call); + expect(out).toBe("out\nerr\n"); + }); + + it("runs in the working directory", async () => { + expect(await createBashTool(context).execute!({ command: "pwd" }, call)).toBe(`${dir}\n`); + }); + + it("turns a non-zero exit into a tool error carrying the output", async () => { + await expect( + createBashTool(context).execute!({ command: "echo nope; exit 3" }, call) + ).rejects.toThrow(/nope[\s\S]*exited with code 3/); + }); + + it("runs commands without the model's credentials in their environment", async () => { + const saved = { ...process.env }; + process.env.ANTHROPIC_API_KEY = "sk-secret"; + process.env.ANTHROPIC_AUTH_TOKEN = "tok-secret"; + process.env.ANTHROPIC_BASE_URL = "http://127.0.0.1:1"; + try { + const out = String(await createBashTool(context).execute!({ command: "env" }, call)); + expect(out).not.toContain("sk-secret"); + expect(out).not.toContain("tok-secret"); + expect(out).toContain("ANTHROPIC_BASE_URL=http://127.0.0.1:1"); + expect( + shellEnv({ OPENAI_API_KEY: "k", HF_TOKEN: "t", AWS_SECRET_ACCESS_KEY: "s", PATH: "/bin" }) + ).toEqual({ PATH: "/bin" }); + } finally { + process.env = saved; + } + }); + + it("loads a credentials file into the environment and deletes it", () => { + const file = join(dir, "credentials.json"); + writeFileSync(file, JSON.stringify({ WORKGLOW_TEST_API_KEY: "sk-from-file" })); + expect(loadCredentialsFile(file)).toEqual(["WORKGLOW_TEST_API_KEY"]); + expect(process.env.WORKGLOW_TEST_API_KEY).toBe("sk-from-file"); + expect(existsSync(file)).toBe(false); + delete process.env.WORKGLOW_TEST_API_KEY; + }); + + it.skipIf(process.platform !== "linux")( + "keeps an exec-time credential out of command output however it is read", + () => { + // /proc//environ shows the exec-time environment: a key set after + // this process started could never appear there, so asserting on it here + // would pass even if an inherited key leaked. The exfiltration has to run + // in a child launched with the key already in its environment — where a + // filter on the shell's environment alone cannot hide it from + // /proc/$PPID/environ. + const bashSrc = fileURLToPath(new URL("../agent/tools/bash.ts", import.meta.url)); + const script = join(dir, "exfil-check.ts"); + writeFileSync( + script, + [ + `import { readFileSync } from "node:fs";`, + `import { createBashTool } from ${JSON.stringify(bashSrc)};`, + `const tool = createBashTool({`, + ` cwd: ${JSON.stringify(dir)},`, + ` spillDir: ${JSON.stringify(join(dir, ".spill"))},`, + ` defaultCommandTimeoutSec: 10,`, + ` images: false,`, + `});`, + `if (!readFileSync("/proc/self/environ", "utf8").includes("sk-from-file")) {`, + ` process.stderr.write("key missing from exec-time environ; test setup broken\\n");`, + ` process.exit(2);`, + `}`, + `const out = String(`, + ` await tool.execute!(`, + ` { command: "env; tr '\\\\0' '\\\\n' < /proc/$PPID/environ" },`, + ` { toolUseId: "t1", signal: new AbortController().signal, deadline: undefined }`, + ` )`, + `);`, + `process.stdout.write(out);`, + ``, + ].join("\n") + ); + const out = execFileSync("bun", [script], { + cwd: dir, + env: { ...process.env, WORKGLOW_TEST_API_KEY: "sk-from-file" }, + encoding: "utf8", + timeout: 60_000, + }); + expect(out).toContain("PATH="); + expect(out).not.toContain("sk-from-file"); + } + ); + + it("deletes a credentials file even when it cannot be read as one", () => { + const file = join(dir, "credentials.json"); + writeFileSync(file, "not json"); + expect(() => loadCredentialsFile(file, {})).toThrow(/not valid JSON/); + expect(existsSync(file)).toBe(false); + writeFileSync(file, JSON.stringify({ KEY: 1 })); + expect(() => loadCredentialsFile(file, {})).toThrow(/non-string value for KEY/); + expect(existsSync(file)).toBe(false); + }); + + it("kills a command at its timeout", async () => { + const started = Date.now(); + await expect( + createBashTool(context).execute!({ command: "sleep 30", timeout: 1 }, call) + ).rejects.toThrow(/timed out after 1s/); + expect(Date.now() - started).toBeLessThan(10_000); + }); + + it("returns from a timeout only once a child ignoring SIGTERM is gone too", async () => { + const log = join(dir, "bg.txt"); + await expect( + createBashTool(context).execute!( + { + command: `(trap '' TERM; while true; do echo x >> ${log}; sleep 0.1; done) & sleep 30`, + timeout: 1, + }, + call + ) + ).rejects.toThrow(/timed out after 1s/); + const size = statSync(log).size; + await new Promise((resolve) => setTimeout(resolve, 600)); + expect(statSync(log).size).toBe(size); + }); + + it("refuses a timeout under one second without running the command", async () => { + for (const timeout of [0, -5, 0.5]) { + await expect( + createBashTool(context).execute!({ command: "touch ran-anyway", timeout }, call) + ).rejects.toThrow(/at least 1 second/); + } + expect(existsSync(join(context.cwd, "ran-anyway"))).toBe(false); + }); + + it("refuses a timeout longer than a timer can be set, rather than shortening it", async () => { + await expect( + createBashTool(context).execute!( + { command: "touch too-long", timeout: MAX_TIMEOUT_SEC + 1 }, + call + ) + ).rejects.toThrow(/at most \d+ seconds/); + expect(existsSync(join(context.cwd, "too-long"))).toBe(false); + }); + + it("cuts a timeout to the time left, and only when it is shorter", () => { + expect(commandTimeoutSec(120, undefined, 0)).toEqual({ sec: 120, capped: false }); + expect(commandTimeoutSec(120, 300_000, 0)).toEqual({ sec: 120, capped: false }); + expect(commandTimeoutSec(600, 90_500, 0)).toEqual({ sec: 90, capped: true }); + expect(commandTimeoutSec(600, 100, 0)).toEqual({ sec: 1, capped: true }); + }); + + it("kills a command when the task's time runs out, whatever timeout it asked for", async () => { + const started = Date.now(); + await expect( + createBashTool(context).execute!( + { command: "sleep 30", timeout: 600 }, + { ...call, deadline: Date.now() + 1_500 } + ) + ).rejects.toThrow(/all the time left/); + expect(Date.now() - started).toBeLessThan(10_000); + }); + + it("does not run a command when no time is left", async () => { + await expect( + createBashTool(context).execute!( + { command: "touch ran.txt" }, + { ...call, deadline: Date.now() - 1 } + ) + ).rejects.toThrow( + "No time is left for this task; the command was not run. Finish with what you have." + ); + expect(readdirSync(dir)).not.toContain("ran.txt"); + }); + + it("does not run a command with under a second left, which a 1 s timeout would overrun", async () => { + await expect( + createBashTool(context).execute!( + { command: "touch ran.txt" }, + { ...call, deadline: Date.now() + 500 } + ) + ).rejects.toThrow(/No time is left/); + expect(readdirSync(dir)).not.toContain("ran.txt"); + }); + + it("does not wait for a process the command left in the background", async () => { + const started = Date.now(); + const out = await createBashTool(context).execute!( + { command: "sleep 30 & echo started" }, + call + ); + expect(out).toBe("started\n"); + expect(Date.now() - started).toBeLessThan(5_000); + }); + + it("shows the tail of long output and spills the rest to a file", async () => { + const out = (await createBashTool(context).execute!({ command: "seq 1 5000" }, call)) as string; + expect(out).toMatch(/^\[Output truncated: showing the last 2000 of 5000 lines/); + expect(out.trimEnd().endsWith("5000")).toBe(true); + const spilled = readdirSync(context.spillDir); + expect(spilled).toHaveLength(1); + expect(readFileSync(join(context.spillDir, spilled[0]!), "utf8").startsWith("1\n2\n")).toBe( + true + ); + }); +}); + +describe("runCommand", () => { + it("does not time out a command that finished, while a background child drains", async () => { + const result = await runCommand("sleep 0.85; sleep 30 & echo done", dir, 1_000, signal); + expect(result.timedOut).toBe(false); + expect(result.output).toContain("done"); + }); + + it("never starts a command whose signal is already aborted", async () => { + const controller = new AbortController(); + controller.abort(new Error("stopped")); + const marker = join(dir, "ran"); + await expect(runCommand(`touch ${marker}`, dir, 10_000, controller.signal)).rejects.toThrow( + "stopped" + ); + expect(readdirSync(dir)).not.toContain("ran"); + }); + + it("keeps the end of an output past the capture cap", async () => { + const result = await runCommand( + "head -c 17000000 /dev/zero | tr '\\0' a; echo; echo THE-END", + dir, + 30_000, + signal + ); + expect(result.droppedBytes).toBeGreaterThan(0); + expect(result.output.trimEnd().endsWith("THE-END")).toBe(true); + }); +}); + +describe("imageMimeType", () => { + it("recognises the formats vision providers take, by their leading bytes", () => { + expect(imageMimeType(Buffer.from([0xff, 0xd8, 0xff, 0xe0]))).toBe("image/jpeg"); + expect(imageMimeType(Buffer.from("GIF89a"))).toBe("image/gif"); + expect(imageMimeType(Buffer.from("RIFF\0\0\0\0WEBPVP8 "))).toBe("image/webp"); + expect(imageMimeType(Buffer.from("plain text"))).toBeUndefined(); + }); +}); + +describe("probeCommands", () => { + it("probes the tools coding tasks reach for beyond the language runtimes", () => { + expect(PROBED_COMMANDS).toEqual( + expect.arrayContaining(["file", "xxd", "strings", "7zz", "7z"]) + ); + }); + + it("finds executables on PATH and lists the rest as missing", () => { + const bin = join(dir, "bin"); + mkdirSync(bin); + writeFileSync(join(bin, "python3"), "#!/bin/sh\n"); + chmodSync(join(bin, "python3"), 0o755); + writeFileSync(join(bin, "rg"), "not executable"); + expect(probeCommands(["python3", "python", "rg"], bin)).toEqual({ + found: ["python3"], + missing: ["python", "rg"], + }); + }); +}); + +describe("codingSystemPrompt", () => { + const env = { cwd: "/app", platform: "linux x64", date: "2026-10-07", commands: undefined }; + it("asks for verification once the change is complete, not after every edit", () => { + const prompt = codingSystemPrompt({ ...env, concise: true, images: true }); + expect(prompt).toContain( + "- Verify once the change is complete: run the code or its tests then, not after every edit." + ); + expect(prompt).not.toContain("Verify your work"); + }); + it("asks for concision only when switched on", () => { + expect(codingSystemPrompt({ ...env, concise: true, images: true })).toContain("Be concise"); + expect(codingSystemPrompt({ ...env, concise: false, images: true })).not.toContain( + "Be concise" + ); + }); + it("tells the model read shows images only when it does", () => { + expect(codingSystemPrompt({ ...env, concise: true, images: true })).toContain( + "shows you images" + ); + expect(codingSystemPrompt({ ...env, concise: true, images: false })).not.toContain("images"); + }); + it("names the commands it found and the ones that are missing", () => { + const prompt = codingSystemPrompt({ + ...env, + concise: true, + images: true, + commands: { found: ["python3", "git"], missing: ["python", "rg"] }, + }); + expect(prompt).toContain("commands on PATH: python3, git"); + expect(prompt).toContain("not installed: python, rg"); + }); +}); diff --git a/examples/agent-eval/tsconfig.json b/examples/agent-eval/tsconfig.json new file mode 100644 index 0000000000..fa30b0afe5 --- /dev/null +++ b/examples/agent-eval/tsconfig.json @@ -0,0 +1,13 @@ +{ + // No `references`: with `use-source` stubs, project-reference redirect turns + // `dist/*.d.ts` → `src` and makes stub `export * from "../src/browser"` a + // self-import, so every `@workglow/*` export appears missing. + "extends": "../../tsconfig.json", + "include": ["src/**/*"], + "exclude": ["dist"], + "compilerOptions": { + "composite": true, + "outDir": "dist", + "rootDir": "./src" + } +} diff --git a/examples/agent-eval/vitest.config.ts b/examples/agent-eval/vitest.config.ts new file mode 100644 index 0000000000..3a2df05af0 --- /dev/null +++ b/examples/agent-eval/vitest.config.ts @@ -0,0 +1,13 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { defineConfig } from "vitest/config"; + +export default defineConfig({ + test: { + include: ["src/test/**/*.test.ts"], + }, +}); diff --git a/examples/cli/src/agent/runAgentChat.ts b/examples/cli/src/agent/runAgentChat.ts index 25ba06907f..498c57c87b 100644 --- a/examples/cli/src/agent/runAgentChat.ts +++ b/examples/cli/src/agent/runAgentChat.ts @@ -198,7 +198,11 @@ async function runTurn( let phase: string | undefined; const offStream = task.subscribe("stream_chunk", (event: StreamEvent) => { - if (event.type === "text-delta") transcript.delta(event.textDelta); + // The turn streams its reasoning beside its reply on its own port; printed + // here it would read as part of the answer. + if (event.type === "text-delta" && (event.port ?? "text") === "text") { + transcript.delta(event.textDelta); + } }); const offProgress = task.subscribe("progress", (_progress, message) => { // Only the tool lines are worth a row: "Thinking" is what the blank space diff --git a/examples/cli/src/run-events/RunEventTypes.ts b/examples/cli/src/run-events/RunEventTypes.ts index 1484e18859..2daba2d195 100644 --- a/examples/cli/src/run-events/RunEventTypes.ts +++ b/examples/cli/src/run-events/RunEventTypes.ts @@ -59,6 +59,8 @@ export type RunEvent = readonly message?: string; } | { readonly k: "text"; readonly id: string; readonly delta: string } + /** A model's thinking, streamed beside its answer; never folded into `text`. */ + | { readonly k: "reasoning"; readonly id: string; readonly delta: string } | { readonly k: "messages"; readonly id: string; readonly messages: unknown } | { readonly k: "human_request"; diff --git a/examples/cli/src/run-events/projectRunEvents.test.ts b/examples/cli/src/run-events/projectRunEvents.test.ts index 2d39e5596a..71d07142f0 100644 --- a/examples/cli/src/run-events/projectRunEvents.test.ts +++ b/examples/cli/src/run-events/projectRunEvents.test.ts @@ -66,6 +66,50 @@ describe("projectRunEvents", () => { }); }); +const STREAMED_PORTS = { + type: "object", + properties: { + text: { type: "string", "x-stream": "append" }, + reasoning: { type: "string", "x-stream": "append" }, + }, + additionalProperties: false, +} as const satisfies DataPortSchema; + +/** Streams a reply beside its reasoning, as `AgentTask` does. */ +class ThinkingTask extends Task, { text: string; reasoning: string }> { + static override readonly type = "ThinkingTask"; + static override readonly title = "Thinking task"; + static override readonly category = "Test"; + static override readonly cacheable = false; + static override inputSchema(): DataPortSchema { + return EMPTY; + } + static override outputSchema(): DataPortSchema { + return STREAMED_PORTS; + } + async *executeStream() { + yield { type: "text-delta" as const, port: "reasoning", textDelta: "Let me think." }; + yield { type: "text-delta" as const, port: "text", textDelta: "Hello." }; + yield { type: "finish" as const, data: {} as { text: string; reasoning: string } }; + } +} + +describe("projectRunEvents streamed text", () => { + it("reports a task's text and reasoning as separate kinds", async () => { + const { events, sink } = collector(); + const workflow = new Workflow(); + workflow.pipe(new ThinkingTask() as never); + const stop = projectRunEvents(workflow.graph, sink); + await workflow.run({}); + stop(); + + const deltas = events.flatMap((e) => (e.k === "text" ? [e.delta] : [])); + expect(deltas).toEqual(["Hello."]); + const thinking = events.flatMap((e) => (e.k === "reasoning" ? [e.delta] : [])); + expect(thinking).toEqual(["Let me think."]); + }); +}); + const ITEM_SCHEMA = { type: "object", properties: { item: { type: "number" } }, diff --git a/examples/cli/src/run-events/projectRunEvents.ts b/examples/cli/src/run-events/projectRunEvents.ts index 50dd9a88a4..0ec1111a6d 100644 --- a/examples/cli/src/run-events/projectRunEvents.ts +++ b/examples/cli/src/run-events/projectRunEvents.ts @@ -5,6 +5,7 @@ */ import type { ITask, StreamEvent, TaskGraph, TaskIdType, Usage } from "@workglow/task-graph"; +import { classifyTextDelta } from "../ui/model/streamedText"; import { cliTaskLabel } from "../ui/taskGraphCliSubscriptions"; import { projectTaskSubtree } from "./projectSubtrees"; import type { RunEventSink } from "./runEventChannel"; @@ -119,8 +120,16 @@ export function projectRunEvents( }); const onChunk = (taskId: TaskIdType, event: StreamEvent): void => { if (event.type === "text-delta") { - const delta = (event as { textDelta?: string; text?: string }).textDelta ?? ""; - if (delta) sink.emit({ k: "text", id: String(taskId), delta }); + // A task's reasoning streams beside its reply on its own port; folded into + // the row's text it would read as part of the answer, so it is its own kind. + const classified = classifyTextDelta(event); + if (classified) { + sink.emit({ + k: classified.port === "text" ? "text" : "reasoning", + id: String(taskId), + delta: classified.delta, + }); + } } else if (event.type === "object-delta" && (event as { port?: string }).port === "messages") { const task = graph.getTask(taskId); sink.emit({ diff --git a/examples/cli/src/test/agentChat.test.ts b/examples/cli/src/test/agentChat.test.ts index 52c8ffa9fe..c729f2ad7e 100644 --- a/examples/cli/src/test/agentChat.test.ts +++ b/examples/cli/src/test/agentChat.test.ts @@ -99,6 +99,21 @@ describe("agent chat loop", () => { expect(out()).toContain("Hello back."); }); + it("prints the reply and not the reasoning streamed beside it", async () => { + const runFn: AiProviderRunFn = async (_input, _model, _signal, emit) => { + emit({ type: "text-delta", port: "reasoning", textDelta: "Pondering quietly." }); + emit({ type: "text-delta", port: "text", textDelta: "Hello back." }); + emit({ type: "finish", data: {} }); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + const { io, out } = scriptedIo(["hello", "/exit"]); + + await runAgentChat(baseOptions, io); + + expect(out()).toContain("Hello back."); + expect(out()).not.toContain("Pondering"); + }); + it("ends at end of input as well as on /exit", async () => { scriptModel(["…"]); const { io } = scriptedIo(["hello", undefined as unknown as string]); diff --git a/examples/cli/src/test/streaming-reasoning-row.test.ts b/examples/cli/src/test/streaming-reasoning-row.test.ts new file mode 100644 index 0000000000..140cfa294a --- /dev/null +++ b/examples/cli/src/test/streaming-reasoning-row.test.ts @@ -0,0 +1,109 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { StreamEvent } from "@workglow/task-graph"; +import { Task, Workflow } from "@workglow/task-graph"; +import type { DataPortSchema } from "@workglow/util/schema"; +import { render } from "ink"; +import { EventEmitter } from "node:events"; +import React from "react"; +import { describe, expect, it } from "vitest"; +import { WorkflowRunApp } from "../ui/WorkflowRunApp"; + +const OUTPUT = { + type: "object", + properties: { + text: { type: "string", "x-stream": "append" }, + reasoning: { type: "string", "x-stream": "append" }, + }, +} as const satisfies DataPortSchema; + +const EMPTY = { type: "object", properties: {} } as const satisfies DataPortSchema; + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +/** Streams its thinking, pauses, then streams its answer, pausing after each. */ +class ThinkingThenAnswerTask extends Task, { text: string }> { + static override readonly type = "ThinkingThenAnswerTask"; + static override readonly title = "Thinking then answer"; + static override readonly category = "Test"; + static override readonly cacheable = false; + static override inputSchema(): DataPortSchema { + return EMPTY; + } + static override outputSchema(): DataPortSchema { + return OUTPUT; + } + async *executeStream(): AsyncIterable> { + // Give the mounted row time to subscribe before the first delta. + await sleep(300); + yield { type: "text-delta", port: "reasoning", textDelta: "Weighing the options." }; + await sleep(400); + yield { type: "text-delta", port: "text", textDelta: "The answer is 42." }; + await sleep(400); + yield { type: "finish", data: {} as { text: string } }; + } +} + +class CapturingStdout extends EventEmitter { + readonly columns = 120; + readonly rows = 40; + readonly isTTY = true; + readonly frames: string[] = []; + write(data: string): boolean { + this.frames.push(data); + return true; + } +} + +// eslint-disable-next-line no-control-regex +const stripAnsi = (s: string): string => s.replace(/\x1b\[[0-9;]*[A-Z]/gi, ""); + +async function renderFrames(): Promise { + const workflow = new Workflow(); + workflow.pipe(new ThinkingThenAnswerTask() as never); + const stdout = new CapturingStdout(); + let finished = false; + const instance = render( + React.createElement(WorkflowRunApp, { + graph: workflow.graph, + input: {}, + runExecutor: () => workflow.run({}), + onComplete: () => { + finished = true; + }, + onError: () => { + finished = true; + }, + }), + // Under a CI runner Ink would otherwise write only the final frame, and the + // thinking this reads is drawn mid-run. + { stdout: stdout as never, patchConsole: false, exitOnCtrlC: false, interactive: true } + ); + const deadline = Date.now() + 6000; + while (!finished && Date.now() < deadline) await sleep(20); + await sleep(100); + instance.unmount(); + return stdout.frames.map(stripAnsi); +} + +describe("streaming row with a reasoning port", () => { + it("draws thinking apart from the answer, and drops it once the answer starts", async () => { + const frames = await renderFrames(); + + const thinkingFrame = frames.find((f) => f.includes("Weighing the options.")); + expect(thinkingFrame, "reasoning was never drawn").toBeDefined(); + expect(thinkingFrame).toContain("thinking"); + expect(thinkingFrame).not.toContain("The answer is 42."); + + const answerFrame = frames.find((f) => f.includes("The answer is 42.")); + expect(answerFrame, "the answer was never drawn").toBeDefined(); + expect(answerFrame).not.toContain("Weighing the options."); + expect(answerFrame).not.toContain("thinking"); + }); +}); diff --git a/examples/cli/src/ui/TaskRunApp.tsx b/examples/cli/src/ui/TaskRunApp.tsx index d93e7711db..e397addb3d 100644 --- a/examples/cli/src/ui/TaskRunApp.tsx +++ b/examples/cli/src/ui/TaskRunApp.tsx @@ -4,7 +4,7 @@ * SPDX-License-Identifier: Apache-2.0 */ -import type { IRunConfig, ITask } from "@workglow/task-graph"; +import type { IRunConfig, ITask, StreamEvent } from "@workglow/task-graph"; import { Box, Static, Text, useWindowSize } from "ink"; import path from "node:path"; import React, { useEffect, useMemo, useRef, useState } from "react"; @@ -15,11 +15,18 @@ import { ProgressBar } from "./components/ProgressBar"; import { deriveRunState, RunStatusBar } from "./components/RunStatusBar"; import { RunViewportProvider } from "./components/RunViewport"; import { ScrollRegion } from "./components/ScrollRegion"; +import { StreamedReasoning } from "./components/StreamedReasoning"; import { StreamOutput } from "./components/StreamOutput"; import { TaskErrorDetail } from "./components/TaskErrorDetail"; import { TaskStatusProgressRow } from "./components/TaskStatusProgressRow"; import { HumanInteractionHost } from "./HumanInteractionHost"; import { planRunViewport } from "./model/runViewport"; +import { + EMPTY_STREAMED_TEXT, + foldTextDelta, + visibleReasoning, + type StreamedText, +} from "./model/streamedText"; import { isRedundantSubgraph, SubtaskRows } from "./rows/SubtaskRows"; import { settledTaskDurationMs } from "./rows/taskDuration"; import { useSubtaskRows } from "./rows/useSubtaskRows"; @@ -99,7 +106,7 @@ export function TaskRunApp({ message: undefined, }); const [downloadFiles, setDownloadFiles] = useState([]); - const [streamText, setStreamText] = useState(""); + const [streamed, setStreamed] = useState(EMPTY_STREAMED_TEXT); const [logs, setLogs] = useState([]); const runUsageLine = useGraphUsageLine(task.subGraph); const subtasks = useSubtaskRows(task); @@ -152,21 +159,18 @@ export function TaskRunApp({ if (msg) progressRef.current.msg = msg; }); - // `textDelta`, not `text`: that is the field name on StreamTextDelta, and - // the loose local type here is why reading the wrong one showed as an - // empty panel rather than as a compile error. - task.events.on("stream_chunk", (event: { type: string; textDelta?: string }) => { - if (event.type === "text-delta" && event.textDelta) { - setStreamText((prev) => prev + event.textDelta); - } + task.events.on("stream_chunk", (event: StreamEvent) => { + setStreamed((prev) => foldTextDelta(prev, event)); }); + // Only the answer is kept in the scrollback; the thinking behind it is + // shown while it streams and then dropped. task.events.on("stream_end", () => { - setStreamText((prev) => { - if (prev) { - setLogs((logs) => [...logs, { id: logCounter++, text: prev }]); + setStreamed((prev) => { + if (prev.text) { + setLogs((logs) => [...logs, { id: logCounter++, text: prev.text }]); } - return ""; + return EMPTY_STREAMED_TEXT; }); }); @@ -188,6 +192,7 @@ export function TaskRunApp({ }, [onComplete, onError, task, taskType, overrides, runConfig]); /** Job queue may register multiple subgraph tasks of the same type (mirrors); hide when all match the parent. */ + const reasoning = visibleReasoning(streamed); const subgraphIsAllSameTypeMirror = isModelDownloadTask && subtasks.rows.length > 1 && @@ -242,7 +247,8 @@ export function TaskRunApp({ ))} )} - {streamText && } + {streamed.text && } + {reasoning && } {showSubtasksSection && ( + * SPDX-License-Identifier: Apache-2.0 + */ + +import { Box, Text } from "ink"; +import React from "react"; +import { STREAM_PANEL_MAX_LINES, tailLines } from "../model/streamedText"; + +interface StreamedReasoningProps { + readonly reasoning: string; + /** Total height including the label; the thinking itself gets one line fewer. */ + readonly maxLines?: number; +} + +/** + * A model's thinking, drawn dim under a "thinking" label so it cannot be + * mistaken for the answer. Bounded to `maxLines` rows like the answer panel. + */ +export function StreamedReasoning({ + reasoning, + maxLines = STREAM_PANEL_MAX_LINES, +}: StreamedReasoningProps): React.ReactElement | null { + if (!reasoning) return null; + return ( + + + thinking + + {tailLines(reasoning, Math.max(1, maxLines - 1))} + + ); +} diff --git a/examples/cli/src/ui/model/streamedText.test.ts b/examples/cli/src/ui/model/streamedText.test.ts new file mode 100644 index 0000000000..1c7bcddced --- /dev/null +++ b/examples/cli/src/ui/model/streamedText.test.ts @@ -0,0 +1,63 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from "vitest"; +import { + EMPTY_STREAMED_TEXT, + foldTextDelta, + tailLines, + visibleReasoning, + type StreamedText, +} from "./streamedText"; + +const fold = (events: ReadonlyArray[1]>): StreamedText => + events.reduce(foldTextDelta, EMPTY_STREAMED_TEXT); + +describe("foldTextDelta", () => { + it("keeps text and reasoning apart", () => { + const out = fold([ + { type: "text-delta", port: "reasoning", textDelta: "Let me " }, + { type: "text-delta", port: "reasoning", textDelta: "think." }, + { type: "text-delta", port: "text", textDelta: "Hel" }, + { type: "text-delta", port: "text", textDelta: "lo" }, + ]); + expect(out).toEqual({ text: "Hello", reasoning: "Let me think." }); + }); + + it("treats a missing port as text", () => { + expect(fold([{ type: "text-delta", textDelta: "hi" }])).toEqual({ + text: "hi", + reasoning: "", + }); + }); + + it("reads a task's own output port as its text", () => { + expect( + foldTextDelta(EMPTY_STREAMED_TEXT, { type: "text-delta", port: "code", textDelta: "x" }) + ).toEqual({ text: "x", reasoning: "" }); + }); + + it("ignores other event types and empty deltas, returning the same object", () => { + const prev: StreamedText = { text: "a", reasoning: "b" }; + expect(foldTextDelta(prev, { type: "finish" })).toBe(prev); + expect(foldTextDelta(prev, { type: "text-delta", port: "text", textDelta: "" })).toBe(prev); + expect(foldTextDelta(prev, { type: "text-delta", port: "text" })).toBe(prev); + }); +}); + +describe("visibleReasoning", () => { + it("shows thinking until the answer starts", () => { + expect(visibleReasoning({ text: "", reasoning: "hmm" })).toBe("hmm"); + expect(visibleReasoning({ text: "A", reasoning: "hmm" })).toBe(""); + }); +}); + +describe("tailLines", () => { + it("keeps the last n lines and leaves a short text alone", () => { + expect(tailLines("a\nb\nc\nd", 2)).toBe("c\nd"); + expect(tailLines("a\nb", 5)).toBe("a\nb"); + }); +}); diff --git a/examples/cli/src/ui/model/streamedText.ts b/examples/cli/src/ui/model/streamedText.ts new file mode 100644 index 0000000000..755e568671 --- /dev/null +++ b/examples/cli/src/ui/model/streamedText.ts @@ -0,0 +1,77 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** + * What a streaming task has said so far, split by the port it said it on. + * + * A task streams its answer on port `text` and a reasoning model's thinking on + * port `reasoning`. Folded into one string the thinking reads as part of the + * answer, so every surface that draws a streaming row folds through here and + * decides for itself how to show the two apart. + */ +export interface StreamedText { + readonly text: string; + readonly reasoning: string; +} + +export const EMPTY_STREAMED_TEXT: StreamedText = { text: "", reasoning: "" }; + +/** The part of a stream event the fold reads; any `StreamEvent` is assignable. */ +export interface TextDeltaLike { + readonly type: string; + readonly port?: string; + readonly textDelta?: string; +} + +/** Lines of a streaming panel shown at once; the oldest scroll off the top. */ +export const STREAM_PANEL_MAX_LINES = 8; + +/** A text delta on a port worth showing: which one, and what it said. */ +export interface ClassifiedDelta { + readonly port: "text" | "reasoning"; + readonly delta: string; +} + +/** + * Reads a stream event as a text delta, or `undefined` when it is not one worth + * showing. Port `reasoning` is the model's thinking; every other port, a missing + * one included, is what the task is saying, since a task may stream its output + * on a port of its own name (`code`, `summary`). An empty delta is ignored. + */ +export function classifyTextDelta(event: TextDeltaLike): ClassifiedDelta | undefined { + if (event.type !== "text-delta") return undefined; + const delta = event.textDelta ?? ""; + if (!delta) return undefined; + return { port: event.port === "reasoning" ? "reasoning" : "text", delta }; +} + +/** + * Folds one stream event into the accumulated text. Returns `prev` itself when + * nothing changed so a state setter can bail out of a re-render. + */ +export function foldTextDelta(prev: StreamedText, event: TextDeltaLike): StreamedText { + const classified = classifyTextDelta(event); + if (!classified) return prev; + return classified.port === "text" + ? { text: prev.text + classified.delta, reasoning: prev.reasoning } + : { text: prev.text, reasoning: prev.reasoning + classified.delta }; +} + +/** The last `n` lines of `text`, so a long stream never grows its panel. */ +export function tailLines(text: string, n: number): string { + const lines = text.split("\n"); + if (lines.length <= n) return text; + return lines.slice(lines.length - n).join("\n"); +} + +/** + * The thinking worth drawing, or "" when there is none. Reasoning is shown only + * until the answer starts: once text is flowing it is the thing being read, and + * the thinking behind it is no longer news. + */ +export function visibleReasoning(streamed: StreamedText): string { + return streamed.text.length > 0 ? "" : streamed.reasoning; +} diff --git a/examples/cli/src/ui/rows/ChatTaskRow.tsx b/examples/cli/src/ui/rows/ChatTaskRow.tsx index eb05416779..6e8ce603b6 100644 --- a/examples/cli/src/ui/rows/ChatTaskRow.tsx +++ b/examples/cli/src/ui/rows/ChatTaskRow.tsx @@ -9,7 +9,14 @@ import type { ITask, StreamEvent } from "@workglow/task-graph"; import { Box, Text } from "ink"; import React, { useEffect, useState, useSyncExternalStore } from "react"; import { TaskErrorDetail } from "../components/TaskErrorDetail"; +import { StreamedReasoning } from "../components/StreamedReasoning"; import { TaskStatusProgressRow } from "../components/TaskStatusProgressRow"; +import { + EMPTY_STREAMED_TEXT, + foldTextDelta, + visibleReasoning, + type StreamedText, +} from "../model/streamedText"; import type { TaskRowProps } from "./pickRenderer"; import { settledTaskDurationMs } from "./taskDuration"; import { useTaskUsageLine } from "./useTaskUsageLine"; @@ -39,6 +46,9 @@ function blockToText(block: ContentBlock): string { return `[tool_use: ${block.name}]`; case "tool_result": return "[tool_result]"; + case "reasoning": + // Kept for the model, not shown as part of the reply. + return ""; } } @@ -72,25 +82,24 @@ export function ChatTaskRow({ task, line }: TaskRowProps): React.ReactElement { () => messagesFromTask(task), () => messagesFromTask(task) ); - // `streamText` is the in-progress assistant text for the *current* turn. - // Appends on every text-delta; resets whenever history grows (meaning the - // previous turn's assistant message just landed, or a new user turn has - // started), and on stream_end. - const [streamText, setStreamText] = useState(""); + // `streamed` is the in-progress assistant text (and thinking, kept apart by + // port) for the *current* turn. Appends on every text-delta; resets whenever + // history grows (meaning the previous turn's assistant message just landed, + // or a new user turn has started), and on stream_end. + const [streamed, setStreamed] = useState(EMPTY_STREAMED_TEXT); const usageLine = useTaskUsageLine(task); useEffect(() => { const onChunk = (event: StreamEvent): void => { if (event.type === "text-delta") { - const delta = (event as { textDelta?: string; text?: string }).textDelta ?? ""; - if (delta) setStreamText((prev) => prev + delta); + setStreamed((prev) => foldTextDelta(prev, event)); } else if (event.type === "object-delta" && event.port === "messages") { // Each object-delta marks a turn boundary — clear the per-turn // streaming buffer so the next turn's text-deltas start fresh. - setStreamText(""); + setStreamed(EMPTY_STREAMED_TEXT); } }; - const onEnd = (): void => setStreamText(""); + const onEnd = (): void => setStreamed(EMPTY_STREAMED_TEXT); task.events.on("stream_chunk", onChunk); task.events.on("stream_end", onEnd); return () => { @@ -148,7 +157,8 @@ export function ChatTaskRow({ task, line }: TaskRowProps): React.ReactElement { // PROCESSING const tail = messages.slice(Math.max(0, messages.length - HISTORY_TAIL)); - const showStreamingBubble = streamText.length > 0; + const showStreamingBubble = streamed.text.length > 0; + const reasoning = visibleReasoning(streamed); return ( @@ -179,7 +189,15 @@ export function ChatTaskRow({ task, line }: TaskRowProps): React.ReactElement { assistant:{" "} - {streamText} + {streamed.text} + + )} + {reasoning.length > 0 && ( + + + assistant:{" "} + + )} diff --git a/examples/cli/src/ui/rows/StreamingTextRow.tsx b/examples/cli/src/ui/rows/StreamingTextRow.tsx index f8c5df4ffb..cf2fae78fc 100644 --- a/examples/cli/src/ui/rows/StreamingTextRow.tsx +++ b/examples/cli/src/ui/rows/StreamingTextRow.tsx @@ -7,25 +7,22 @@ import { Box, Text } from "ink"; import React from "react"; import { TaskErrorDetail } from "../components/TaskErrorDetail"; +import { StreamedReasoning } from "../components/StreamedReasoning"; import { TaskStatusProgressRow } from "../components/TaskStatusProgressRow"; +import { STREAM_PANEL_MAX_LINES, tailLines, visibleReasoning } from "../model/streamedText"; import type { TaskRowProps } from "./pickRenderer"; import { settledTaskDurationMs } from "./taskDuration"; import { useTaskStreamText } from "./useTaskStreamText"; import { useTaskUsageLine } from "./useTaskUsageLine"; -const MAX_LINES = 8; - -function tailLines(text: string, n: number): string { - const lines = text.split("\n"); - if (lines.length <= n) return text; - return lines.slice(lines.length - n).join("\n"); -} - export function StreamingTextRow({ task, line }: TaskRowProps): React.ReactElement { - const streamText = useTaskStreamText(task); + const streamed = useTaskStreamText(task); + const reasoning = visibleReasoning(streamed); const usageLine = useTaskUsageLine(task); - const isActive = line.status === "PROCESSING"; - const showPanel = isActive && streamText.length > 0; + // The first text-delta flips a task to STREAMING, so PROCESSING alone would + // hide the panel for exactly the tasks that have something to show in it. + const isActive = line.status === "PROCESSING" || line.status === "STREAMING"; + const showPanel = isActive && (streamed.text.length > 0 || reasoning.length > 0); return ( @@ -47,7 +44,11 @@ export function StreamingTextRow({ task, line }: TaskRowProps): React.ReactEleme borderColor="gray" paddingX={1} > - {tailLines(streamText, MAX_LINES)} + {streamed.text.length > 0 ? ( + {tailLines(streamed.text, STREAM_PANEL_MAX_LINES)} + ) : ( + + )} )} diff --git a/examples/cli/src/ui/rows/useTaskStreamText.ts b/examples/cli/src/ui/rows/useTaskStreamText.ts index 1f7a8b5c8f..c4292e2d7e 100644 --- a/examples/cli/src/ui/rows/useTaskStreamText.ts +++ b/examples/cli/src/ui/rows/useTaskStreamText.ts @@ -6,27 +6,25 @@ import type { ITask, StreamEvent } from "@workglow/task-graph"; import { useEffect, useState } from "react"; +import { EMPTY_STREAMED_TEXT, foldTextDelta, type StreamedText } from "../model/streamedText"; /** * Subscribes to `stream_chunk` text-delta events on a task and returns the - * accumulated text. Clears on `stream_end` or when the task status transitions - * to COMPLETED. + * accumulated answer and reasoning, kept apart by port. Clears on `stream_end` + * or when the task status transitions to COMPLETED. */ -export function useTaskStreamText(task: ITask): string { - const [text, setText] = useState(""); +export function useTaskStreamText(task: ITask): StreamedText { + const [text, setText] = useState(EMPTY_STREAMED_TEXT); useEffect(() => { const onChunk = (event: StreamEvent): void => { - if (event.type !== "text-delta") return; - const delta = event.textDelta ?? ""; - if (!delta) return; - setText((prev) => prev + delta); + setText((prev) => foldTextDelta(prev, event)); }; const onEnd = (): void => { - setText(""); + setText(EMPTY_STREAMED_TEXT); }; const onStatus = (status: string): void => { - if (status === "COMPLETED") setText(""); + if (status === "COMPLETED") setText(EMPTY_STREAMED_TEXT); }; task.events.on("stream_chunk", onChunk); diff --git a/examples/cli/src/web/client/app.css b/examples/cli/src/web/client/app.css index f9e0ac8c9b..5fe8193f88 100644 --- a/examples/cli/src/web/client/app.css +++ b/examples/cli/src/web/client/app.css @@ -1303,6 +1303,20 @@ details.adv > summary:hover { .bubble .txt { color: var(--scr-fg); } +.bubble.thinking { + border-left-color: var(--scr-faint); + background: none; +} +.bubble.thinking .r { + margin-right: 8px; + font-style: italic; + font-weight: 400; + color: var(--scr-faint); +} +.bubble.thinking .txt { + color: var(--scr-dim); + font-style: italic; +} .bubble .caret { display: inline-block; width: 7px; diff --git a/examples/cli/src/web/client/state.test.ts b/examples/cli/src/web/client/state.test.ts index 25b1b61672..ff485bf397 100644 --- a/examples/cli/src/web/client/state.test.ts +++ b/examples/cli/src/web/client/state.test.ts @@ -61,6 +61,17 @@ describe("reduceRunEvent", () => { expect(state.rows.get("t1")!.streamText).toBe("Hello"); }); + it("keeps a row's reasoning apart from its streamed text", () => { + const state = apply([ + added("t1"), + { k: "reasoning", id: "t1", delta: "Let me " }, + { k: "reasoning", id: "t1", delta: "think." }, + { k: "text", id: "t1", delta: "Hello" }, + ]); + expect(state.rows.get("t1")!.reasoningText).toBe("Let me think."); + expect(state.rows.get("t1")!.streamText).toBe("Hello"); + }); + it("drops a row the run released", () => { const state = apply([added("t1"), { k: "task_removed", id: "t1" }]); expect(state.rows.has("t1")).toBe(false); diff --git a/examples/cli/src/web/client/state.ts b/examples/cli/src/web/client/state.ts index 3921e23e3f..abac520685 100644 --- a/examples/cli/src/web/client/state.ts +++ b/examples/cli/src/web/client/state.ts @@ -26,6 +26,7 @@ export interface RunRow { readonly progress: number | undefined; readonly message: string | undefined; readonly streamText: string; + readonly reasoningText: string; readonly messages: unknown; readonly usage: { input: number; output: number; cached: number } | undefined; readonly startedAt: number | undefined; @@ -192,6 +193,7 @@ export function reduceRunEvent( progress: undefined, message: undefined, streamText: "", + reasoningText: "", messages: undefined, usage: undefined, startedAt: undefined, @@ -222,6 +224,11 @@ export function reduceRunEvent( if (!row) return state; return withRow(state, event.id, { streamText: row.streamText + event.delta }); } + case "reasoning": { + const row = state.rows.get(event.id); + if (!row) return state; + return withRow(state, event.id, { reasoningText: row.reasoningText + event.delta }); + } case "messages": return withRow(state, event.id, { messages: event.messages }); case "usage": diff --git a/examples/cli/src/web/client/views/RunConsole.tsx b/examples/cli/src/web/client/views/RunConsole.tsx index 2664e41a61..f2892dc3b9 100644 --- a/examples/cli/src/web/client/views/RunConsole.tsx +++ b/examples/cli/src/web/client/views/RunConsole.tsx @@ -21,6 +21,9 @@ import { MapView } from "./MapView"; /** Braille frames, the same family the terminal spins. */ const SPINNER = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"] as const; +/** The most of a model's thinking a row keeps on screen; older text scrolls off. */ +const REASONING_TAIL_CHARS = 600; + const RUNNING = new Set(["PROCESSING", "STREAMING", "ABORTING"]); function durationOf(row: RunRow): number | undefined { @@ -51,7 +54,10 @@ function TaskRow({ const iteration = state.iterations.get(row.id); const detail = taskDetailText(row.progress, durationOf(row), running); const showBar = running && row.progress !== undefined; - const indeterminate = running && row.progress === undefined && row.streamText.length > 0; + const indeterminate = + running && + row.progress === undefined && + (row.streamText.length > 0 || row.reasoningText.length > 0); return (
@@ -84,6 +90,15 @@ function TaskRow({
) : null} {iteration ? : null} + {row.reasoningText && !row.streamText && running ? ( +
+
+ thinking + {row.reasoningText.slice(-REASONING_TAIL_CHARS)} + +
+
+ ) : null} {row.streamText && running ? (
diff --git a/examples/web/src/graph/TaskNode.test.ts b/examples/web/src/graph/TaskNode.test.ts index ef84517535..9a36112d10 100644 --- a/examples/web/src/graph/TaskNode.test.ts +++ b/examples/web/src/graph/TaskNode.test.ts @@ -57,8 +57,15 @@ afterEach(() => { container.remove(); }); -function renderTaskNode(usageValue: Usage | undefined): void { - const task = new NoopTask(); +function renderTaskNode(usageValue: Usage | undefined, task: Task = new NoopTask()): void { + act(() => { + root = createRoot(container); + }); + rerenderTaskNode(usageValue, task); +} + +/** Renders into the existing root, as ReactFlow does when it reuses a node for another task. */ +function rerenderTaskNode(usageValue: Usage | undefined, task: Task): void { const nodeProps = { id: String(task.id), type: "task", @@ -79,7 +86,6 @@ function renderTaskNode(usageValue: Usage | undefined): void { }; act(() => { - root = createRoot(container); root.render( createElement(ReactFlowProvider, null, createElement(TaskNode, nodeProps as never)) ); @@ -98,3 +104,66 @@ describe("TaskNode per-node usage badge (the node actually rendered on the graph expect(container.querySelector(".task-node-usage")).toBeNull(); }); }); + +describe("TaskNode streaming panel", () => { + const delta = (port: string | undefined, textDelta: string) => + port === undefined + ? { type: "text-delta" as const, textDelta } + : { type: "text-delta" as const, port, textDelta }; + + it("shows reasoning in its own subdued block, above the answer", () => { + const task = new NoopTask(); + renderTaskNode(undefined, task); + act(() => { + task.emit("stream_start"); + task.emit("stream_chunk", delta("reasoning", "Weighing it.") as never); + task.emit("stream_chunk", delta("text", "The answer.") as never); + }); + + const reasoning = container.querySelector("details.task-node-reasoning"); + expect(reasoning?.textContent).toContain("Weighing it."); + expect(reasoning?.textContent).not.toContain("The answer."); + + const answer = [...container.querySelectorAll("pre")].find((pre) => !reasoning?.contains(pre)); + expect(answer?.textContent).toBe("The answer."); + expect( + reasoning!.compareDocumentPosition(answer!) & Node.DOCUMENT_POSITION_FOLLOWING + ).toBeTruthy(); + }); + + it("drops the previous task's stream when the node is reused for another task", () => { + const first = new NoopTask(); + renderTaskNode(undefined, first); + act(() => { + first.emit("stream_start"); + first.emit("stream_chunk", delta("reasoning", "First task's thinking.") as never); + first.emit("stream_chunk", delta("text", "First answer.") as never); + }); + expect(container.textContent).toContain("First task's thinking."); + + rerenderTaskNode(undefined, new NoopTask()); + expect(container.textContent).not.toContain("First task's thinking."); + expect(container.textContent).not.toContain("First answer."); + }); + + it("treats a delta with no port as the answer", () => { + const task = new NoopTask(); + renderTaskNode(undefined, task); + act(() => { + task.emit("stream_start"); + task.emit("stream_chunk", delta(undefined, "plain") as never); + }); + expect(container.querySelector("details.task-node-reasoning")).toBeNull(); + expect(container.querySelector("pre")?.textContent).toBe("plain"); + }); + + it("draws no reasoning block for a task that sends none", () => { + const task = new NoopTask(); + renderTaskNode(undefined, task); + act(() => { + task.emit("stream_start"); + task.emit("stream_chunk", delta("text", "only answer") as never); + }); + expect(container.querySelector("details.task-node-reasoning")).toBeNull(); + }); +}); diff --git a/examples/web/src/graph/TaskNode.tsx b/examples/web/src/graph/TaskNode.tsx index 7d6bafe746..228be6a064 100644 --- a/examples/web/src/graph/TaskNode.tsx +++ b/examples/web/src/graph/TaskNode.tsx @@ -49,6 +49,8 @@ export function TaskNode(props: NodeProps>) { const [streamingText, setStreamingText] = useState(""); const [isStreaming, setIsStreaming] = useState(false); const streamingTextRef = useRef(""); + const [reasoningText, setReasoningText] = useState(""); + const reasoningTextRef = useRef(""); const task = data.task; const [trackedTask, setTrackedTask] = useState(task); if (trackedTask !== task) { @@ -58,10 +60,17 @@ export function TaskNode(props: NodeProps>) { setProgressMessage(undefined); setSubTasks(task.hasChildren() ? task.subGraph.getTasks() : []); setIsExpandable(task.hasChildren()); + // What the previous task streamed is not this one's, and this one may have + // started streaming before the effect below subscribes to it. + setIsStreaming(false); + setStreamingText(""); + setReasoningText(""); } useEffect(() => { const unsubscribes: (() => void)[] = []; + streamingTextRef.current = ""; + reasoningTextRef.current = ""; // Helper to update consolidated progress const updateConsolidatedProgress = () => { @@ -93,14 +102,23 @@ export function TaskNode(props: NodeProps>) { setIsStreaming(true); setStreamingText(""); streamingTextRef.current = ""; + setReasoningText(""); + reasoningTextRef.current = ""; }) ); unsubscribes.push( task.subscribe("stream_chunk", (event: StreamEvent) => { if (event.type === "text-delta") { - streamingTextRef.current += event.textDelta; - setStreamingText(streamingTextRef.current); + // A model's thinking streams beside its answer on its own port; every + // other port, a missing one included, is what the task is saying. + if (event.port === "reasoning") { + reasoningTextRef.current += event.textDelta; + setReasoningText(reasoningTextRef.current); + } else { + streamingTextRef.current += event.textDelta; + setStreamingText(streamingTextRef.current); + } } else if (event.type === "snapshot") { const text = typeof event.data === "string" ? event.data : JSON.stringify(event.data); streamingTextRef.current = text; @@ -203,12 +221,20 @@ export function TaskNode(props: NodeProps>) { message={progressMessage} /> - {(isStreaming || (status === TaskStatus.STREAMING && streamingText)) && ( + {(isStreaming || (status === TaskStatus.STREAMING && (streamingText || reasoningText))) && (
Streaming
+ {reasoningText && ( +
+ Thinking + 
+                  {reasoningText}
+
+
+ )} 
{streamingText}
)} diff --git a/packages/ai/src/provider-utils.ts b/packages/ai/src/provider-utils.ts index 11b2387e5d..64382898e5 100644 --- a/packages/ai/src/provider-utils.ts +++ b/packages/ai/src/provider-utils.ts @@ -18,6 +18,7 @@ export * from "./provider-utils/registerProvider"; export * from "./provider-utils/modelSearchQuery"; export * from "./provider-utils/ToolCallParsers"; export * from "./provider-utils/ToolCallMarkupFilter"; +export * from "./provider-utils/ThinkTagFilter"; export * from "./provider-utils/HfModelSearch"; export * from "./provider-utils/PipelineTaskMapping"; export * from "./provider-utils/imageOutputHelpers"; diff --git a/packages/ai/src/provider-utils/OpenAIShapedChat.ts b/packages/ai/src/provider-utils/OpenAIShapedChat.ts index f938968c3c..f5745bb919 100644 --- a/packages/ai/src/provider-utils/OpenAIShapedChat.ts +++ b/packages/ai/src/provider-utils/OpenAIShapedChat.ts @@ -182,6 +182,17 @@ export async function accumulateOpenAIChatStream( const choice = chunk?.choices?.[0]; if (!choice) continue; + // Thinking models on OpenAI-compatible APIs stream their reasoning beside + // the content: DeepSeek as `reasoning_content`, OpenRouter as `reasoning`. + // A chunk can carry both with one empty, so the first non-empty one wins. + const reasoningDelta = [choice.delta?.reasoning_content, choice.delta?.reasoning].find( + (value: unknown): value is string => typeof value === "string" && value.length > 0 + ); + if (reasoningDelta !== undefined) { + provisionalUsage?.onText(reasoningDelta); + emit({ type: "text-delta", port: "reasoning", textDelta: reasoningDelta }); + } + const contentDelta: string = choice.delta?.content ?? ""; if (contentDelta) { provisionalUsage?.onText(contentDelta); diff --git a/packages/ai/src/provider-utils/OpenAIShapedResponses.ts b/packages/ai/src/provider-utils/OpenAIShapedResponses.ts index c4e83e86bb..3e1187a68d 100644 --- a/packages/ai/src/provider-utils/OpenAIShapedResponses.ts +++ b/packages/ai/src/provider-utils/OpenAIShapedResponses.ts @@ -43,6 +43,8 @@ export interface OpenAIResponsesMessageSource { readonly function?: { readonly name?: string; readonly arguments?: string }; }[]; readonly tool_call_id?: string; + /** The provider's own output items for this assistant turn, replayed verbatim. */ + readonly native_items?: readonly unknown[] | undefined; } /** A Responses request `input` item — a role message or a tool item. */ @@ -143,6 +145,18 @@ export function buildResponsesInput(args: { } if (message.role === "assistant") { + // The provider's own turn, replayed as it produced it: its reasoning only + // continues from items it can decrypt, and only while the calls still + // match — a host that renamed or dropped a call gets the rebuilt turn. + const native = message.native_items; + if ( + native && + sameCalls(native, message.tool_calls ?? []) && + nativeText(native) === contentToString(message.content) + ) { + items.push(...(native as OpenAIResponsesInputItem[])); + continue; + } // Preserve any assistant text, then each tool call as a `function_call` // item (Responses represents tool calls as items, not message fields). const text = contentToString(message.content); @@ -171,6 +185,66 @@ export function buildResponsesInput(args: { }; } +/** + * The text the native items said, run together as the turn records it: an + * edited or removed reply means the items are no longer the turn, so it is + * rebuilt rather than replayed. + */ +function nativeText(native: readonly unknown[]): string { + return native + .filter((item) => (item as { type?: unknown } | null)?.type === "message") + .flatMap((item) => { + const content = (item as { content?: unknown }).content; + return Array.isArray(content) ? content : []; + }) + .map((part) => { + const p = part as { type?: unknown; text?: unknown }; + return p?.type === "output_text" && typeof p.text === "string" ? p.text : ""; + }) + .join(""); +} + +/** A call's arguments as one canonical string: the API's own spacing is not part of it. */ +function canonicalArguments(args: unknown): string | undefined { + if (typeof args !== "string") return undefined; + try { + return JSON.stringify(JSON.parse(args === "" ? "{}" : args)); + } catch { + return undefined; + } +} + +/** + * Whether the native items' calls are the message's calls exactly: the same + * ids, names and arguments, in order. A host that changed any of them gets the + * rebuilt turn, since replaying the items would send the call it replaced. + */ +function sameCalls( + native: readonly unknown[], + calls: readonly { + readonly id?: string; + readonly function?: { readonly name?: string; readonly arguments?: string }; + }[] +): boolean { + const nativeCalls = native.filter( + (item): item is { type: string; call_id: string; name?: unknown; arguments?: unknown } => + (item as { type?: string } | null)?.type === "function_call" + ); + return ( + nativeCalls.length === calls.length && + nativeCalls.every((item, i) => { + const call = calls[i]!; + const args = canonicalArguments(item.arguments); + return ( + item.call_id === (call.id ?? "") && + item.name === call.function?.name && + args !== undefined && + args === canonicalArguments(call.function?.arguments ?? "") + ); + }) + ); +} + export function buildResponsesTools(tools: readonly ToolDefinition[]): OpenAIResponsesTool[] { return tools.map((t) => ({ type: "function" as const, @@ -246,11 +320,21 @@ interface ResponsesToolCallEntry { * * Pass `promptText` to emit a provisional ↑ estimate before the first delta — * Responses only attaches billed usage to the terminal lifecycle event. + * + * Pass `nativeTurnProvider` to also emit the turn's finished output items, + * verbatim and in order, as one `nativeTurn` `object-delta` keyed by that + * provider, so the next round can replay them. Nothing is emitted when the turn + * has no items, or when any reasoning item came back without + * `encrypted_content`: replaying that item makes the API look it up by id, + * which fails under `store: false`, zero data retention, or once it expired. */ export async function accumulateOpenAIResponsesStream>( stream: AsyncIterable, emit: (event: StreamEvent) => void, - options: { readonly promptText?: string | undefined } = {} + options: { + readonly promptText?: string | undefined; + readonly nativeTurnProvider?: string | undefined; + } = {} ): Promise { // Keyed by the output item's stable id (`item.id` on `output_item.{added,done}`, // `item_id` on `function_call_arguments.delta`). Falls back to `output_index` @@ -258,6 +342,9 @@ export async function accumulateOpenAIResponsesStream(); + // Every finished output item by position, for replay on the next round; kept + // only when the caller asked for a native turn. + const outputItems = new Map(); // Responses only reports billed usage on the terminal lifecycle event; when the // caller supplies promptText, estimate ↑ before the first delta and ↓ from // content / tool-arg deltas so the CLI counter moves during the call. The @@ -343,6 +430,12 @@ export async function accumulateOpenAIResponsesStream 0) { + const items = [...outputItems.entries()].sort((a, b) => a[0] - b[0]).map(([, item]) => item); + const replayable = items.every((item) => { + const { type, encrypted_content } = item as { type?: unknown; encrypted_content?: unknown }; + return ( + type !== "reasoning" || (typeof encrypted_content === "string" && encrypted_content !== "") + ); + }); + if (replayable) { + emit({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: options.nativeTurnProvider, payload: JSON.stringify(items) }, + } as StreamEvent); + } + } + provisionalUsage?.flush(); return usage; } diff --git a/packages/ai/src/provider-utils/ThinkTagFilter.ts b/packages/ai/src/provider-utils/ThinkTagFilter.ts new file mode 100644 index 0000000000..d834ad868c --- /dev/null +++ b/packages/ai/src/provider-utils/ThinkTagFilter.ts @@ -0,0 +1,84 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +/** One routed piece of a stream: which output port it belongs to, and its text. */ +export interface ThinkTagPart { + readonly port: "text" | "reasoning"; + readonly text: string; +} + +export interface ThinkTagFilter { + /** Routes one streamed delta; returns the parts to emit, in order. */ + push(delta: string): ReadonlyArray; + /** Releases anything held back and returns to the starting state. */ + flush(): ReadonlyArray; +} + +const OPEN_TAG = ""; +const CLOSE_TAG = ""; + +/** Length of the longest suffix of `text` that is a proper prefix of `tag`. */ +function danglingPrefixLength(text: string, tag: string): number { + for (let len = Math.min(text.length, tag.length - 1); len >= 1; len--) { + if (text.endsWith(tag.slice(0, len))) return len; + } + return 0; +} + +/** + * Splits a stream of text deltas into reasoning and answer text, using the + * `…` convention of Qwen3-style chat templates. + * + * A tag can arrive split across deltas, so a suffix that might still become + * the tag being looked for is held back until it resolves. A block the stream + * never closes stays on `reasoning` to the end: a generation cut off at the + * token limit loses its closing tag, and what follows the opening one is still + * thinking. + * + * `startInside` is for a template that opens the tag in the prompt, so the + * generation begins mid-block and only the closing tag appears in the stream. + */ +export function createThinkTagFilter(options?: { + readonly startInside?: boolean | undefined; +}): ThinkTagFilter { + const startInside = options?.startInside === true; + let inside = startInside; + let held = ""; + + function part(port: "text" | "reasoning", text: string, into: ThinkTagPart[]): void { + if (text.length > 0) into.push({ port, text }); + } + + function push(delta: string): ReadonlyArray { + const out: ThinkTagPart[] = []; + let buffer = held + delta; + held = ""; + for (;;) { + const tag = inside ? CLOSE_TAG : OPEN_TAG; + const port = inside ? "reasoning" : "text"; + const at = buffer.indexOf(tag); + if (at === -1) { + const keep = danglingPrefixLength(buffer, tag); + part(port, buffer.slice(0, buffer.length - keep), out); + held = buffer.slice(buffer.length - keep); + return out; + } + part(port, buffer.slice(0, at), out); + buffer = buffer.slice(at + tag.length); + inside = !inside; + } + } + + function flush(): ReadonlyArray { + const out: ThinkTagPart[] = []; + part(inside ? "reasoning" : "text", held, out); + held = ""; + inside = startInside; + return out; + } + + return { push, flush }; +} diff --git a/packages/ai/src/task/AgentRoundRunner.ts b/packages/ai/src/task/AgentRoundRunner.ts index 8707ce924a..5d64e482d7 100644 --- a/packages/ai/src/task/AgentRoundRunner.ts +++ b/packages/ai/src/task/AgentRoundRunner.ts @@ -13,6 +13,12 @@ export interface AgentRoundContext { readonly signal: AbortSignal; /** The round's text as it streams, re-emitted as the turn's own. */ readonly onTextDelta: (delta: string) => void; + /** + * The round's reasoning as it streams, re-emitted as the turn's own on its + * `reasoning` port. Held with the round's text until the attempt settles, so + * a round that fails and is retried contributes neither. + */ + readonly onReasoningDelta: (delta: string) => void; /** * Progress while the round has nothing to say yet — a model being downloaded * or loaded. `progress` is 0..100, or undefined when there is no denominator. diff --git a/packages/ai/src/task/AgentTask.ts b/packages/ai/src/task/AgentTask.ts index c025327b2f..d0f60bd7e0 100644 --- a/packages/ai/src/task/AgentTask.ts +++ b/packages/ai/src/task/AgentTask.ts @@ -48,7 +48,9 @@ import { import type { ChatMessage, ContentBlock, + ContentBlockImage, ContentBlockInToolResultBody, + ContentBlockText, ContentBlockToolResult, } from "./ChatMessage"; import { ChatMessageSchema } from "./ChatMessage"; @@ -85,6 +87,9 @@ const DEFAULT_MAX_TOOL_RESULT_CHARS = 20_000; */ const DEFAULT_MAX_ROUND_RETRIES = 2; +/** Between two rounds' reasoning, which would otherwise run together into one thought. */ +const REASONING_ROUND_SEPARATOR = "\n\n"; + /** * Times a turn with an `outputSchema` reminds a model that replied in text to * submit instead. Some models narrate a final answer out of habit and submit @@ -158,6 +163,8 @@ export const AgentInputSchema = { description: "Time one model call may take before it is abandoned and retried as a retryable failure", minimum: 1, + // The longest timer that can be set; past it the round would be abandoned at once. + maximum: 2_147_483_647, "x-ui-group": "Configuration", }, toolConcurrency: { @@ -200,6 +207,20 @@ export const AgentInputSchema = { minimum: 1, "x-ui-group": "Configuration", }, + announceTimeLeft: { + type: "boolean", + title: "Announce Time Left", + description: + "Append the time left before maxDurationMs to the last tool result of every round, so the model can plan the work it has time for. Needs maxDurationMs", + "x-ui-group": "Configuration", + }, + reviewPrompt: { + type: "string", + title: "Review Prompt", + description: + "Sent once, as a user message, the first time the model replies without calling a tool, so it checks its work before the turn ends. Not sent on a turn with outputSchema, or when no round, budget or time is left to act on it", + "x-ui-group": "Configuration", + }, approval: { type: "string", title: "Approval", @@ -223,6 +244,12 @@ export const AgentOutputSchema = { "Everything the assistant said this turn, including what it narrated between tool calls", "x-stream": "append", }, + reasoning: { + type: "string", + title: "Reasoning", + description: "The model's reasoning this turn, as the provider streamed it", + "x-stream": "append", + }, messages: { type: "array", items: ChatMessageSchema, @@ -296,6 +323,8 @@ export type AgentTaskInput = { readonly maxInputTokens?: number | undefined; readonly maxCostUsd?: number | undefined; readonly maxDurationMs?: number | undefined; + readonly announceTimeLeft?: boolean | undefined; + readonly reviewPrompt?: string | undefined; readonly approval?: AgentApprovalMode | undefined; }; @@ -303,6 +332,12 @@ export type AgentStopReason = "answered" | "submitted" | "budget" | "max-rounds" export type AgentTaskOutput = { text: string; + /** + * Each round's reasoning, a blank line between rounds that reasoned. Absent + * when no round returned any, so a turn against a model that does not think + * has the output it always had. + */ + reasoning?: string | undefined; messages: ChatMessage[]; rounds: number; stopReason: AgentStopReason; @@ -329,8 +364,27 @@ function isAnswerable(call: ToolCall): boolean { return typeof call.id === "string" && call.id.length > 0 && typeof call.name === "string"; } -function assistantMessage(text: string, calls: readonly ToolCall[]): ChatMessage { +function assistantMessage( + text: string, + calls: readonly ToolCall[], + reasoning: string | undefined, + nativeTurn: { readonly provider: string; readonly payload: string } | undefined +): ChatMessage { const content: ContentBlock[] = []; + // Reasoning rides only on a reply that has something else: on its own it is + // not a reply, and an assistant turn with no text and no call is rejected. + if (text.length > 0 || calls.length > 0) { + if (nativeTurn) { + content.push({ + type: "reasoning", + text: reasoning ?? "", + provider: nativeTurn.provider, + payload: nativeTurn.payload, + }); + } else if (reasoning) { + content.push({ type: "reasoning", text: reasoning }); + } + } if (text.length > 0) content.push({ type: "text", text }); for (const call of calls) { content.push({ @@ -362,24 +416,83 @@ function toolResult( call: ToolCall, text: string, isError: boolean, - maxChars: number + maxChars: number, + content?: ReadonlyArray ): ContentBlockToolResult { - const body: ContentBlockInToolResultBody[] = [ - { type: "text", text: clampToolText(text, maxChars) }, - ]; return { type: "tool_result", tool_use_id: call.id, - content: body, + content: content ? orderedToolBody(content, maxChars) : [textBlock(text, maxChars)], is_error: isError ? true : undefined, }; } +function textBlock(text: string, maxChars: number): ContentBlockText { + return { type: "text", text: clampToolText(text, maxChars) }; +} + +/** + * A tool's own blocks in the order it returned them, so text describing an image + * stays beside it. Adjacent text joins into one block, which keeps a result of + * text alone a single block, as providers send it as a plain string. The text + * shares one budget: past it a block is cut and later text dropped, while images + * stay. A result of images alone carries no text block: an empty one is not a + * block Anthropic accepts, and there is nothing in it for the model to read. + */ +function orderedToolBody( + blocks: ReadonlyArray, + maxChars: number +): ContentBlockInToolResultBody[] { + const runs: Array = []; + for (const block of blocks) { + const previous = runs[runs.length - 1]; + if (block.type === "text" && previous?.type === "text") { + runs[runs.length - 1] = { type: "text", text: `${previous.text}\n${block.text}` }; + } else { + runs.push(block); + } + } + const total = runs.reduce((n, block) => n + (block.type === "text" ? block.text.length : 0), 0); + const note = `[truncated, ${total} chars total]`; + let left = maxChars; + let noted = false; + const body: ContentBlockInToolResultBody[] = []; + for (const block of runs) { + if (block.type === "image") { + body.push(block); + } else if (block.text.length === 0) { + continue; + } else if (block.text.length <= left) { + body.push(block); + left -= block.text.length; + } else if (left > 0) { + body.push({ type: "text", text: `${block.text.slice(0, left)}… ${note}` }); + left = 0; + noted = true; + } else if (!noted) { + // The budget ran out exactly at an earlier block: the model is still told + // that text was left out, where it was. + body.push({ type: "text", text: `… ${note}` }); + noted = true; + } + } + return body.length > 0 ? body : [textBlock("", maxChars)]; +} + +/** The time a turn has left, as the model is shown it: whole seconds, rounded down. */ +function timeLeftNotice(ms: number): string { + const total = Math.max(0, Math.floor(ms / 1000)); + const minutes = Math.floor(total / 60); + return `\n\n[Time left: ${minutes > 0 ? `${minutes}m ` : ""}${total % 60}s]`; +} + interface RunCallOptions { readonly byName: ReadonlyMap; readonly validators: ReturnType; readonly approval: AgentApprovalMode; readonly maxResultChars: number; + /** When the turn's time budget runs out; see {@link ToolExecuteContext.deadline}. */ + readonly deadline: number | undefined; /** The tool that submits the answer, when the turn has an `outputSchema`. */ readonly submitToolName: string | undefined; } @@ -540,8 +653,15 @@ export class AgentTask extends Task ({ messages, steps }); /** * The transcript so far, for a host drawing one while the turn is still @@ -571,11 +694,13 @@ export class AgentTask extends Task; let text = ""; + let reasoning = ""; let rounds = 0; const finish = (stopReason: AgentStopReason): StreamEvent => ({ type: "finish", data: { text, + ...(reasoning.length > 0 ? { reasoning } : {}), messages, rounds, stopReason, @@ -596,8 +721,14 @@ export class AgentTask extends Task= deadline) { + yield finish("budget"); + return; + } + rounds = round + 1; const roundStart = Date.now(); const captured: { output: ToolCallingTaskOutput | undefined } = { output: undefined }; @@ -610,9 +741,11 @@ export class AgentTask extends Task 0 )) { yield event; } @@ -621,8 +754,23 @@ export class AgentTask extends Task maxRoundRetries || !isRetryableRoundError(err)) throw err; context.signal.throwIfAborted(); const wait = retryDelayMs(err, attempts); + // Nothing from the failed attempt has been recorded, so the turn can end + // here as cleanly as between rounds. + if (deadline !== undefined && Date.now() + wait >= deadline) { + yield finish("budget"); + return; + } await context.updateProgress(undefined, `Retrying in ${Math.ceil(wait / 1000)}s`); await sleepUnlessAborted(wait, context.signal); + // The wait can overrun (a slow progress write, a late timer); an attempt + // started past the deadline is one the budget no longer covers. + if (deadline !== undefined && Date.now() >= deadline) { + yield finish("budget"); + return; + } + // Said as the attempt starts, not before the wait: a wait cut short + // (the run stopped during it) is no attempt made. + await context.updateProgress(undefined, `Thinking (attempt ${attempts + 1})`); } } const modelMs = Date.now() - roundStart; @@ -634,6 +782,10 @@ export class AgentTask extends Task 0) { + reasoning += (reasoning.length > 0 ? REASONING_ROUND_SEPARATOR : "") + roundReasoning; + } const record = (toolRecords: readonly AgentToolRecord[]): void => { steps.push({ @@ -661,13 +813,33 @@ export class AgentTask extends Task 0) { messages.push(reply); yield transcript(); } if (calls.length === 0) { record([]); + // A turn with an outputSchema has checkSubmission for this, and a review + // sent with no round, budget or time left could never be acted on. + if ( + input.reviewPrompt && + !reviewed && + submitTool === undefined && + round + 1 < maxRounds && + !overBudget() && + (deadline === undefined || Date.now() < deadline) + ) { + reviewed = true; + messages.push({ role: "user", content: [{ type: "text", text: input.reviewPrompt }] }); + yield transcript(); + continue; + } if (submitTool !== undefined && reminders < MAX_SUBMIT_REMINDERS && !overBudget()) { reminders++; messages.push({ role: "user", content: [{ type: "text", text: SUBMIT_REMINDER }] }); @@ -707,10 +879,34 @@ export class AgentTask extends Task 0) { + const last = results[results.length - 1]!; + const notice = timeLeftNotice(deadline - Date.now()); + // Joined onto the last text block, not added as a second one: a result of + // one text block stays one, which providers send as a plain string (or + // parse as a whole) and a second block would turn into a list of parts. + // A result of images alone has no text to join, so the notice leads them. + const content = [...last.content]; + const at = content.findLastIndex((block) => block.type === "text"); + const block = content[at]; + if (block?.type === "text") content[at] = { ...block, text: block.text + notice }; + else content.unshift({ type: "text", text: notice }); + results[results.length - 1] = { ...last, content }; + // The record counts what the model is shown, which now includes the notice. + const recordAt = toolRecords.findIndex((r) => r.id === last.tool_use_id); + if (recordAt !== -1) { + const counted = toolRecords[recordAt]!; + toolRecords[recordAt] = { ...counted, chars: counted.chars + notice.length }; + } + } messages.push({ role: "tool", content: results }); record(toolRecords); yield transcript(); @@ -793,7 +989,7 @@ export class AgentTask extends Task> { const roundInput: ToolCallingTaskInput = { model: input.model, prompt: input.prompt, systemPrompt: input.systemPrompt, - messages: normalizeHistoryForModel(trimHistoryForModel(messages, maxHistoryChars)), + // The current turn is held whole: a reminder or review prompt is a user + // message, and a cut there would drop the prompt the model is asked to + // check its answer against and every tool result it gathered. Earlier + // turns are still cut to fit. + messages: normalizeHistoryForModel(trimHistoryForModel(messages, maxHistoryChars, turnStart)), tools: input.tools, temperature: input.temperature, maxTokens: input.maxTokens, @@ -844,6 +1046,8 @@ export class AgentTask extends Task queue.push({ type: "text-delta", port: "text", textDelta: delta }), + onReasoningDelta: (delta) => + queue.push({ type: "text-delta", port: "reasoning", textDelta: delta }), // Progress is advisory: a failed update must not surface as an // unhandled rejection, nor fail a round that is otherwise fine. onProgress: (progress, message) => { @@ -856,8 +1060,9 @@ export class AgentTask extends Task turn.abort(); off = turn.subscribe("stream_chunk", (event: StreamEvent) => { if (event.type !== "text-delta") return; - if ((event.port ?? "text") !== "text") return; - queue.push({ type: "text-delta", port: "text", textDelta: event.textDelta }); + const port = event.port ?? "text"; + if (port !== "text" && port !== "reasoning") return; + queue.push({ type: "text-delta", port, textDelta: event.textDelta }); }); runRound = () => turn.run(roundInput); } @@ -887,13 +1092,25 @@ export class AgentTask extends Task {}); // Held until the attempt settles: the stream accumulates every delta it is - // handed into the task's text port and offers no way to take one back, so - // text forwarded from an attempt that then fails and is retried would be - // joined to the retry's. A failed attempt throws here and its text is dropped. + // handed into the task's text and reasoning ports and offers no way to take + // one back, so either forwarded from an attempt that then fails and is + // retried would be joined to the retry's. A failed attempt throws here and + // its text and reasoning are dropped together. const held: StreamEvent[] = []; for await (const event of queue.iterable) held.push(event); await run; - for (const event of held) yield event; + let separated = !reasonedBefore; + for (const event of held) { + // Rounds are told apart on the wire exactly as in the settled output, and + // only once this round has something to say: a separator ahead of nothing + // would leave the streamed reasoning ending differently than the final one. + if (!separated && event.type === "text-delta" && event.port === "reasoning") { + if (event.textDelta.length === 0) continue; + yield { type: "text-delta", port: "reasoning", textDelta: REASONING_ROUND_SEPARATOR }; + separated = true; + } + yield event; + } } /** @@ -936,8 +1153,9 @@ export class AgentTask extends Task | undefined; } /** @@ -160,13 +166,15 @@ export function toolCallNeedsApproval( async function askToApprove( context: IExecuteContext, toolName: string, - request: IHumanRequest + request: IHumanRequest, + signal: AbortSignal ): Promise { try { - return await context.registry.get(HUMAN_CONNECTOR).send(request, context.signal); + return await context.registry.get(HUMAN_CONNECTOR).send(request, signal); } catch (error) { if (context.signal.aborted) throw error; - getLogger().warn(`Could not ask for approval of "${toolName}"`, { error }); + // Withdrawn at the turn's deadline: the caller reports that, not a failure to ask. + if (!signal.aborted) getLogger().warn(`Could not ask for approval of "${toolName}"`, { error }); return undefined; } } @@ -198,7 +206,8 @@ function describeToolReach( async function approveToolCall( tool: ToolDefinition, call: ToolCall, - context: IExecuteContext + context: IExecuteContext, + signal: AbortSignal ): Promise { if (!context.registry.has(HUMAN_CONNECTOR)) { return { @@ -209,21 +218,26 @@ async function approveToolCall( }; } const ctor = getTaskConstructors(context.registry).get(backingTaskType(tool)); - const response = await askToApprove(context, tool.name, { - requestId: uuid4(), - targetHumanId: "default", - kind: "confirm", - message: `Run "${tool.name}"?`, - contentSchema: APPROVAL_SCHEMA as DataPortSchema, - contentData: { - tool: tool.name, - reach: describeToolReach(tool, ctor), - arguments: clampToolText(stringifyForModel(call.input), MAX_APPROVAL_ARGUMENT_CHARS), + const response = await askToApprove( + context, + tool.name, + { + requestId: uuid4(), + targetHumanId: "default", + kind: "confirm", + message: `Run "${tool.name}"?`, + contentSchema: APPROVAL_SCHEMA as DataPortSchema, + contentData: { + tool: tool.name, + reach: describeToolReach(tool, ctor), + arguments: clampToolText(stringifyForModel(call.input), MAX_APPROVAL_ARGUMENT_CHARS), + }, + expectsResponse: true, + mode: "single", + metadata: { toolUseId: call.id, toolName: tool.name }, }, - expectsResponse: true, - mode: "single", - metadata: { toolUseId: call.id, toolName: tool.name }, - }); + signal + ); if (response === undefined) { return { text: @@ -243,6 +257,51 @@ async function approveToolCall( }; } +const MAX_TIMER_MS = 2_147_483_647; + +/** + * A signal that aborts at `deadline`, however far off. A timer past 2^31-1 ms + * fires at once, so a deadline further away than that is reached in steps, each + * re-armed for what is left. + */ +export function deadlineSignal(deadline: number): { + readonly signal: AbortSignal; + readonly cancel: () => void; +} { + const expiry = new AbortController(); + let timer: ReturnType | undefined; + const arm = (): void => { + const left = deadline - Date.now(); + timer = + left > MAX_TIMER_MS + ? setTimeout(arm, MAX_TIMER_MS) + : setTimeout(() => expiry.abort(), Math.max(0, left)); + }; + arm(); + return { signal: expiry.signal, cancel: () => clearTimeout(timer) }; +} + +/** Asks for approval, withdrawing the request when the turn's deadline passes. */ +async function approveWithinDeadline( + tool: ToolDefinition, + call: ToolCall, + context: IExecuteContext, + deadline: number | undefined +): Promise { + if (deadline === undefined) return approveToolCall(tool, call, context, context.signal); + const expiry = deadlineSignal(deadline); + try { + return await approveToolCall( + tool, + call, + context, + AbortSignal.any([context.signal, expiry.signal]) + ); + } finally { + expiry.cancel(); + } +} + /** * Runs one tool call and returns what the model reads back. * @@ -255,15 +314,40 @@ export async function runAgentTool( tool: ToolDefinition, call: ToolCall, context: IExecuteContext, - options: { readonly approval: AgentApprovalMode; readonly maxResultChars: number } + options: { + readonly approval: AgentApprovalMode; + readonly maxResultChars: number; + readonly deadline: number | undefined; + } ): Promise { - const refusal = toolCallNeedsApproval(tool, options.approval, context.registry) - ? await approveToolCall(tool, call, context) + const needsApproval = toolCallNeedsApproval(tool, options.approval, context.registry); + const refusal = needsApproval + ? await approveWithinDeadline(tool, call, context, options.deadline) : undefined; + // A person can take longer to answer than the turn has left: the request is + // withdrawn at the deadline, and an approval landing after it does not start the tool. + if (needsApproval && options.deadline !== undefined && Date.now() >= options.deadline) { + return { + text: + `The turn's time ran out while "${tool.name}" waited for approval, so it was not run. ` + + `Do not call it again this turn.`, + isError: true, + }; + } if (refusal) return refusal; try { - const output = await invokeTool(tool, call, context); + const output = await invokeTool(tool, call, context, options.deadline); + if (output instanceof ToolResultContent) { + const text = output.content + .flatMap((block) => (block.type === "text" ? [block.text] : [])) + .join("\n"); + return { + text: clampToolText(text, options.maxResultChars), + isError: false, + content: output.content, + }; + } return { text: clampToolText(stringifyForModel(output), options.maxResultChars), isError: false, @@ -287,13 +371,14 @@ export async function runAgentTool( async function invokeTool( tool: ToolDefinition, call: ToolCall, - context: IExecuteContext + context: IExecuteContext, + deadline: number | undefined ): Promise { if (tool.type === "function" || (tool.type === undefined && tool.execute)) { if (!tool.execute) { throw new Error(`Tool "${tool.name}" is declared type "function" but supplies no execute()`); } - return await tool.execute(call.input, { toolUseId: call.id, signal: context.signal }); + return await tool.execute(call.input, { toolUseId: call.id, signal: context.signal, deadline }); } const ctor = getTaskConstructors(context.registry).get(backingTaskType(tool)); if (!ctor) { diff --git a/packages/ai/src/task/AiChatTask.ts b/packages/ai/src/task/AiChatTask.ts index 97dc2603af..2e482b1845 100644 --- a/packages/ai/src/task/AiChatTask.ts +++ b/packages/ai/src/task/AiChatTask.ts @@ -208,6 +208,7 @@ export type AiChatTaskInput = Omit< )[]; tool_use_id: string; } + | { type: "reasoning"; text: string; provider?: string; payload?: string } )[]; }, "messages" diff --git a/packages/ai/src/task/ChatHistory.ts b/packages/ai/src/task/ChatHistory.ts index de73eb4311..ef60e4aae6 100644 --- a/packages/ai/src/task/ChatHistory.ts +++ b/packages/ai/src/task/ChatHistory.ts @@ -62,14 +62,29 @@ export function normalizeHistoryForModel(history: readonly ChatMessage[]): ChatM */ export const DEFAULT_MAX_HISTORY_CHARS = 120_000; +/** + * A message's size as the model reads it. A reasoning block's `payload` is a + * provider's own turn kept for replay, not text this budget is protecting the + * context window from, so it is measured without it. + */ function messageChars(message: ChatMessage): number { try { - return JSON.stringify(message).length; + const content = message.content.map((block) => { + if (block.type !== "reasoning" || block.payload === undefined) return block; + const { payload: _payload, ...rest } = block; + return rest; + }); + return JSON.stringify({ ...message, content }).length; } catch { return 0; } } +/** `history`'s size as {@link trimHistoryForModel} measures it against its budget. */ +export function measureHistoryForModel(history: readonly ChatMessage[]): number { + return history.reduce((sum, message) => sum + messageChars(message), 0); +} + /** * Caps `history` at `max` characters by dropping whole turns from the front. * @@ -82,14 +97,24 @@ function messageChars(message: ChatMessage): number { * otherwise be nothing for the model to answer, and silently returning an empty * list turns "this turn is too long" into "the conversation is gone". * - * A leading run of `system` messages belongs to no turn and is never cut: it - * carries the instructions every later turn is answered under, and dropping it - * for being old changes how the model behaves with nothing to show for it. Its - * size still counts against the budget, so the turns cut to fit around it. + * A `system` message is never cut, leading or not: it carries instructions + * every later turn is answered under, and dropping one for being old changes + * how the model behaves with nothing to show for it. A later one stays in its + * place ahead of the turns that survive. Their size still counts against the + * budget, so the turns cut to fit around them. + * + * `keepFrom` holds everything from that index on, for a caller whose newest + * turn holds user messages of its own (a reminder, a review prompt): cutting at + * one of those would keep the prompt and drop the question it is about. + * + * A reasoning block's `payload` is not counted, so a provider replaying its own + * turns (encrypted reasoning, signed thinking) sends more than `max`. The budget + * caps the conversation's text, not those opaque payloads. */ export function trimHistoryForModel( history: readonly ChatMessage[], - max: number = DEFAULT_MAX_HISTORY_CHARS + max: number = DEFAULT_MAX_HISTORY_CHARS, + keepFrom: number = history.length ): ChatMessage[] { const sizes = history.map(messageChars); let total = sizes.reduce((sum, n) => sum + n, 0); @@ -99,15 +124,25 @@ export function trimHistoryForModel( while (history[prefixEnd]?.role === "system") prefixEnd++; const turnStarts: number[] = []; + // A run of user messages is one pending turn (as normalizeHistoryForModel + // merges it), so only its first message is a place to cut: cutting inside it + // would keep the newest prompt and drop the question it follows. for (let i = prefixEnd; i < history.length; i++) { - if (history[i]?.role === "user") turnStarts.push(i); + if (history[i]?.role === "user" && history[i - 1]?.role !== "user") turnStarts.push(i); } + // A system message past the leading run is an instruction too, and every + // provider lifts it into the system prompt wherever it sits, so a cut keeps it + // and it stays counted. let cut = prefixEnd; for (const start of turnStarts.slice(1)) { - for (let i = cut; i < start; i++) total -= sizes[i] ?? 0; + if (start > keepFrom) break; + for (let i = cut; i < start; i++) { + if (history[i]?.role !== "system") total -= sizes[i] ?? 0; + } cut = start; if (total <= max) break; } - return [...history.slice(0, prefixEnd), ...history.slice(cut)]; + const keptSystem = history.slice(prefixEnd, cut).filter((message) => message.role === "system"); + return [...history.slice(0, prefixEnd), ...keptSystem, ...history.slice(cut)]; } diff --git a/packages/ai/src/task/ChatMessage.ts b/packages/ai/src/task/ChatMessage.ts index a9d379494a..4202535f55 100644 --- a/packages/ai/src/task/ChatMessage.ts +++ b/packages/ai/src/task/ChatMessage.ts @@ -32,6 +32,28 @@ export type ContentBlockToolUse = { readonly providerSignature?: string; }; +/** + * What a thinking model reasoned before the rest of its reply, kept so it can + * be handed back on later rounds of the same turn. Some providers ask for it + * back (DeepSeek's `reasoning_content` on an assistant turn that called tools), + * and a model given its own reasoning continues from it rather than working the + * plan out again each round. A provider that has no use for it skips the block. + * `payload` is a provider's whole turn, kept so that provider can resume its own + * reasoning, which it returns only in its own encoded form. + */ +export type ContentBlockReasoning = { + readonly type: "reasoning"; + readonly text: string; + /** + * The provider whose assistant turn `payload` is, when the round kept it. That + * provider replays the payload verbatim on later rounds; every other provider + * reads `text` or skips the block. + */ + readonly provider?: string; + /** The provider's own assistant output for this turn, serialized in its wire format and order. */ + readonly payload?: string; +}; + /** * Blocks that may appear in a `tool_result`'s `content` array. Provider payloads * typically use text, image, and tool_use; nested `tool_result` is not modeled @@ -55,7 +77,8 @@ export type ContentBlock = | ContentBlockText | ContentBlockImage | ContentBlockToolUse - | ContentBlockToolResult; + | ContentBlockToolResult + | ContentBlockReasoning; export type ChatRole = "user" | "assistant" | "tool" | "system"; @@ -98,6 +121,18 @@ const ContentBlockToolUseSchema = { additionalProperties: false, } as const; +const ContentBlockReasoningSchema = { + type: "object", + properties: { + type: { type: "string", enum: ["reasoning"] }, + text: { type: "string" }, + provider: { type: "string" }, + payload: { type: "string" }, + }, + required: ["type", "text"], + additionalProperties: false, +} as const; + /** `tool_result.content` — text, image, and tool_use only (no nested `tool_result`). */ const ContentBlockInToolResultBodySchema = { oneOf: [ContentBlockTextSchema, ContentBlockImageSchema, ContentBlockToolUseSchema], @@ -126,6 +161,7 @@ export const ContentBlockSchema = { ContentBlockImageSchema, ContentBlockToolUseSchema, ContentBlockToolResultSchema, + ContentBlockReasoningSchema, ], title: "ContentBlock", description: "A single content block within a chat message", @@ -189,6 +225,12 @@ export function isContentBlock(value: unknown): value is ContentBlock { Array.isArray(v.content) && v.content.every(isContentBlockInToolResultBody) ); + case "reasoning": + return ( + typeof v.text === "string" && + (v.provider === undefined || typeof v.provider === "string") && + (v.payload === undefined || typeof v.payload === "string") + ); default: return false; } diff --git a/packages/ai/src/task/MessageConversion.ts b/packages/ai/src/task/MessageConversion.ts index 5e22b19d1c..a81b363101 100644 --- a/packages/ai/src/task/MessageConversion.ts +++ b/packages/ai/src/task/MessageConversion.ts @@ -53,6 +53,32 @@ export interface OpenAICompatMessage { function: { name: string; arguments: string }; }>; tool_call_id?: string; + /** An assistant turn's reasoning, under the name the provider reads it back by. */ + reasoning_content?: string; + /** The provider's own output items for this turn, for the Responses converter to replay. */ + native_items?: unknown[]; +} + +export interface OpenAIMessageOptions { + /** + * Send each assistant turn's reasoning back as `reasoning_content`. DeepSeek's + * thinking models read it on the turns of a tool-calling loop; other + * OpenAI-compatible APIs do not know the field, so it is opt-in and reasoning + * blocks are otherwise dropped. + */ + readonly replayReasoning?: boolean | undefined; + /** + * Chat-completions accepts only text in a `tool` message. With this set, the + * images a turn's tools returned are sent in one user message after that + * turn's tool results, and each tool message says an image follows. Without + * it they stay inline in the tool message, for APIs that accept that. + */ + readonly toolImagesInUserMessage?: boolean | undefined; + /** + * Attach an assistant turn's native items (`native_items`) when its reasoning + * block was produced by this provider, for the Responses converter to replay. + */ + readonly nativeTurnProvider?: string | undefined; } /** @@ -61,14 +87,24 @@ export interface OpenAICompatMessage { * * Multi-turn capable: preserves full tool call metadata across turns. */ -export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessage[] { +export function toOpenAIMessages( + input: ToolCallingTaskInput, + options: OpenAIMessageOptions = {} +): OpenAICompatMessage[] { const messages: OpenAICompatMessage[] = []; - if (input.systemPrompt) { - messages.push({ role: "system", content: input.systemPrompt }); + // System turns in the history join the system prompt; the loop below has no + // place for them, and a caller's instructions must not be dropped. + const history = getInputMessages(input); + const lifted = history ? liftSystemMessages(history, input.systemPrompt) : undefined; + const systemPrompt = lifted ? lifted.systemPrompt : input.systemPrompt; + if (systemPrompt) { + messages.push({ role: "system", content: systemPrompt }); } - const inputMessages = getInputMessages(input); + // A history that held only system turns is still the history: the prompt is + // what a caller with no history sends, never a stand-in for one. + const inputMessages = lifted?.messages; if (!inputMessages) { if (!Array.isArray(input.prompt)) { messages.push({ role: "user", content: input.prompt }); @@ -102,7 +138,26 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa return messages; } + // Images a tool returned wait until every result of the run of tool messages + // is out: a user turn between two results of one assistant turn would leave a + // call unanswered before it. + let deferredImages: Array<{ type: string; [key: string]: unknown }> = []; + let deferredImageCount = 0; + const flushDeferredImages = (): void => { + if (deferredImages.length === 0) return; + messages.push({ + role: "user", + content: [ + { type: "text", text: "Images returned by the tool calls above:" }, + ...deferredImages, + ], + }); + deferredImages = []; + deferredImageCount = 0; + }; + for (const msg of inputMessages) { + if (msg.role !== "tool") flushDeferredImages(); if (msg.role === "user") { const parts: Array<{ type: string; [key: string]: unknown }> = []; for (const block of msg.content) { @@ -113,6 +168,9 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa type: "image_url", image_url: { url: `data:${block.mimeType};base64,${block.data}` }, }); + } else if ((block as { type: string }).type === "image_url") { + // Already in this format: a caller written against chat-completions. + parts.push(block as unknown as { type: string; [key: string]: unknown }); } // tool_use / tool_result not valid in a user message — skip } @@ -134,18 +192,86 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa })); const entry: OpenAICompatMessage = { role: "assistant", - content: textParts.length > 0 ? textParts : null, + // A turn with neither text nor calls (the model only reasoned) still needs a + // content string: chat-completions APIs reject `null` without `tool_calls`. + content: textParts.length > 0 ? textParts : toolCalls.length > 0 ? null : "", }; if (toolCalls.length > 0) { entry.tool_calls = toolCalls; } + if (options.replayReasoning) { + // Sent even when empty: DeepSeek returns an empty reasoning on some + // turns and still expects the field on every assistant turn it reads. + entry.reasoning_content = msg.content + .filter((b): b is Extract => b.type === "reasoning") + .map((b) => b.text) + .join(""); + } + if (options.nativeTurnProvider !== undefined) { + const own = msg.content.find( + (b): b is Extract => + b.type === "reasoning" && + b.provider === options.nativeTurnProvider && + b.payload !== undefined + ); + if (own) { + try { + const items = JSON.parse(own.payload!) as unknown; + // Each item must at least be a typed object; `[null]` or a bare value + // would be sent as an input item and refused, where a rebuilt turn + // is not. + if ( + Array.isArray(items) && + items.length > 0 && + items.every( + (item) => + item !== null && + typeof item === "object" && + !Array.isArray(item) && + typeof (item as { type?: unknown }).type === "string" + ) + ) { + entry.native_items = items; + } + } catch { + // A payload this provider cannot read is not replayed; the turn is rebuilt. + } + } + } messages.push(entry); } else if (msg.role === "tool") { for (const block of msg.content) { if (block.type !== "tool_result") continue; let content: string | Array<{ type: string; [key: string]: unknown }>; - if (block.content.length === 1 && block.content[0].type === "text") { - content = block.content[0].text; + const onlyText = block.content.every((inner) => inner.type === "text"); + if (onlyText) { + // Several text blocks, or none, still become one string: some chat-completions + // APIs reject a parts array in a tool message. + content = block.content.map((inner) => (inner as { text: string }).text).join("\n"); + } else if (options.toolImagesInUserMessage) { + // Each image leaves a numbered marker where it stood, and the user message + // labels it with the same number, so text around an image still points at + // the right one. A block that is neither text nor an image (a nested + // tool_use) is data the tool returned, and goes as JSON rather than dropped. + const lines: string[] = []; + for (const inner of block.content) { + if (inner.type === "text") { + lines.push(inner.text); + } else if (inner.type === "image") { + const label = `[image ${++deferredImageCount}]`; + lines.push(`${label} follows in the next message`); + deferredImages.push( + { type: "text", text: label }, + { + type: "image_url", + image_url: { url: `data:${inner.mimeType};base64,${inner.data}` }, + } + ); + } else { + lines.push(JSON.stringify(inner)); + } + } + content = lines.join("\n"); } else { const parts: Array<{ type: string; [key: string]: unknown }> = []; for (const inner of block.content) { @@ -156,6 +282,8 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa type: "image_url", image_url: { url: `data:${inner.mimeType};base64,${inner.data}` }, }); + } else { + parts.push({ type: "text", text: JSON.stringify(inner) }); } } content = parts; @@ -164,10 +292,56 @@ export function toOpenAIMessages(input: ToolCallingTaskInput): OpenAICompatMessa } } } + flushDeferredImages(); return messages; } +/** A message whose content is either workglow content blocks or a bare string. */ +export interface LooseChatMessage { + readonly role: string; + readonly content: string | ReadonlyArray; +} + +/** + * Chat history as content blocks, accepting the bare-string `content` that + * text-generation callers have always been allowed to pass. A provider runs + * this before its converter, so a string message and a block message reach the + * wire the same way instead of the array being forwarded unconverted. + */ +export function normalizeChatMessages(messages: ReadonlyArray): ChatMessage[] { + return messages.map((message) => ({ + role: message.role as ChatMessage["role"], + content: + typeof message.content === "string" + ? [{ type: "text", text: message.content }] + : message.content, + })); +} + +/** + * Moves system-role messages out of a history and into the system prompt. Most + * converters take the system prompt as its own input and skip system turns in + * `messages`, so a caller that put its instructions in the history would + * otherwise lose them; `toOpenAIMessages` lifts them itself. + */ +export function liftSystemMessages( + history: ReadonlyArray, + systemPrompt: string | undefined +): { readonly messages: ChatMessage[]; readonly systemPrompt: string | undefined } { + const lifted = history + .filter((message) => message.role === "system") + .map((message) => + message.content.flatMap((block) => (block.type === "text" ? [block.text] : [])).join("") + ) + .filter((text) => text.length > 0); + const parts = [systemPrompt, ...lifted].filter((text): text is string => !!text); + return { + messages: history.filter((message) => message.role !== "system"), + systemPrompt: parts.length > 0 ? parts.join("\n\n") : undefined, + }; +} + export interface TextFlatMessage { role: string; content: string; @@ -175,8 +349,8 @@ export interface TextFlatMessage { /** * Converts ToolCallingTaskInput to a simplified text-only message format. - * Used by providers that don't natively support structured multi-turn - * tool calling (Ollama, HuggingFace Transformers). + * Used for single-turn prompts by providers whose template takes only role and + * text. A tool-calling loop must not use it: * * NOTE: This format discards tool_use blocks from assistant messages. * The LLM will not see what tools it previously called. Multi-turn tool diff --git a/packages/ai/src/task/ToolCallingTask.ts b/packages/ai/src/task/ToolCallingTask.ts index 3d0a5706bc..c10180f70b 100644 --- a/packages/ai/src/task/ToolCallingTask.ts +++ b/packages/ai/src/task/ToolCallingTask.ts @@ -258,6 +258,26 @@ export const ToolCallingOutputSchema = { description: "Tool calls requested by the model", "x-stream": "object", }, + reasoning: { + type: "string", + title: "Reasoning", + description: + "What a thinking model reasoned before replying, when its provider returns it as text", + "x-stream": "append", + }, + nativeTurn: { + type: "object", + title: "Native Turn", + description: + "The provider's own assistant output for this round, kept so the provider can be handed it back verbatim on the next round", + properties: { + provider: { type: "string" }, + payload: { type: "string" }, + }, + required: ["provider", "payload"], + additionalProperties: false, + "x-stream": "object", + }, ...CheckpointOutputProperty, }, required: ["text", "toolCalls"], @@ -320,6 +340,10 @@ export type ToolCallingTaskOutput = { input: { [x: string]: unknown }; providerSignature?: string; }[]; + /** Present when the provider returned the model's reasoning as text. */ + reasoning?: string | undefined; + /** Present when the provider handed back its own turn for replay. */ + nativeTurn?: { readonly provider: string; readonly payload: string } | undefined; checkpoint?: string | undefined; }; export type ToolCallingTaskConfig = TaskConfig; @@ -450,7 +474,12 @@ export class ToolCallingTask extends StreamingAiTask< private async finalizeCheckpoint( input: ToolCallingTaskInput, - out: { text: string; toolCalls: ToolCallingTaskOutput["toolCalls"] } + out: { + text: string; + toolCalls: ToolCallingTaskOutput["toolCalls"]; + reasoning?: string | undefined; + nativeTurn?: ToolCallingTaskOutput["nativeTurn"]; + } ): Promise { const resolved = this._resolvedCheckpoint; if (!resolved?.emitCheckpointId) return; @@ -459,12 +488,41 @@ export class ToolCallingTask extends StreamingAiTask< input.messages && input.messages.length > 0 ? [...input.messages] : [promptToUserMessage(input.prompt)]; - const assistantContent = [ + const replyContent = [ ...(out.text ? ([{ type: "text", text: out.text }] as const) : []), + // A call's provider signature (Gemini's thought signature) rides with it, + // as the agent loop records it: Gemini refuses a replayed call without it. ...out.toolCalls.map( - (tc) => ({ type: "tool_use", id: tc.id, name: tc.name, input: tc.input }) as const + (tc) => + ({ + type: "tool_use", + id: tc.id, + name: tc.name, + input: tc.input, + ...(tc.providerSignature === undefined + ? {} + : { providerSignature: tc.providerSignature }), + }) as const ), ]; + // The turn is recorded with its reasoning and the provider's own output, as + // the agent loop records it, so a provider that replays its turn (signed + // thinking, encrypted reasoning) is handed it back from the checkpoint too. + // Reasoning alone is not a reply, so it rides only beside one. + const reasoningBlock = + replyContent.length === 0 + ? undefined + : out.nativeTurn + ? ({ + type: "reasoning", + text: out.reasoning ?? "", + provider: out.nativeTurn.provider, + payload: out.nativeTurn.payload, + } as const) + : out.reasoning + ? ({ type: "reasoning", text: out.reasoning } as const) + : undefined; + const assistantContent = reasoningBlock ? [reasoningBlock, ...replyContent] : replyContent; await finalizeEmittedCheckpoint({ model, resolved, @@ -547,12 +605,18 @@ export class ToolCallingTask extends StreamingAiTask< } let text = ""; + let reasoning = ""; + let nativeTurn: ToolCallingTaskOutput["nativeTurn"]; let toolCalls: ToolCallingTaskOutput["toolCalls"] = []; let finalized = false; try { for await (const event of super.executeStream(input, context)) { if (event.type === "text-delta" && (event.port ?? "text") === "text") { text += event.textDelta; + } else if (event.type === "text-delta" && event.port === "reasoning") { + reasoning += event.textDelta; + } else if (event.type === "object-delta" && event.port === "nativeTurn") { + nativeTurn = event.objectDelta as unknown as ToolCallingTaskOutput["nativeTurn"]; } else if (event.type === "object-delta" && event.port === "toolCalls") { // Mirror StreamProcessor's canonical accumulation: array deltas are // upserts by id (OpenAI-shaped providers emit one single-element @@ -571,7 +635,12 @@ export class ToolCallingTask extends StreamingAiTask< } } if (event.type === "finish") { - await this.finalizeCheckpoint(input, { text, toolCalls }); + await this.finalizeCheckpoint(input, { + text, + toolCalls, + reasoning: reasoning || undefined, + nativeTurn, + }); finalized = true; yield { type: "text-delta", diff --git a/packages/ai/src/task/ToolCallingUtils.ts b/packages/ai/src/task/ToolCallingUtils.ts index d31c492bc9..0d1d597924 100644 --- a/packages/ai/src/task/ToolCallingUtils.ts +++ b/packages/ai/src/task/ToolCallingUtils.ts @@ -8,6 +8,7 @@ import type { SchemaNode } from "@workglow/util/schema"; import { compileSchema } from "@workglow/util/schema"; import type { JsonSchema } from "@workglow/util/worker"; import { getLogger } from "@workglow/util/worker"; +import type { ContentBlockImage, ContentBlockText } from "./ChatMessage"; /** * A tool definition that can be passed to an LLM for tool calling. @@ -77,6 +78,23 @@ export interface ToolDefinition { export interface ToolExecuteContext { readonly toolUseId: string; readonly signal: AbortSignal; + /** + * When the turn's `maxDurationMs` runs out, in epoch milliseconds; undefined + * when the turn has no time budget. The loop stops only between rounds, so a + * tool that can bound its own work (a command timeout, a poll) caps it here, + * or one call can run the turn past its budget. + */ + readonly deadline: number | undefined; +} + +/** + * A tool result made of content blocks rather than one string — what a tool's + * `execute` returns to hand the model an image. Anything else a tool returns is + * shown to the model as text, so without this an image would arrive as a + * base64 string, clamped. + */ +export class ToolResultContent { + constructor(readonly content: ReadonlyArray) {} } /** diff --git a/packages/ai/src/task/__tests__/AgentDeadlineSignal.test.ts b/packages/ai/src/task/__tests__/AgentDeadlineSignal.test.ts new file mode 100644 index 0000000000..61cf61c5f8 --- /dev/null +++ b/packages/ai/src/task/__tests__/AgentDeadlineSignal.test.ts @@ -0,0 +1,39 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { afterEach, describe, expect, it, vi } from "vitest"; +import { deadlineSignal } from "../AgentToolExecution"; + +const MAX_TIMER_MS = 2_147_483_647; + +describe("deadlineSignal", () => { + afterEach(() => { + vi.useRealTimers(); + }); + + it("aborts at a deadline too far off for one timer, and not before", () => { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] }); + const days = 30 * 24 * 60 * 60 * 1000; + const { signal } = deadlineSignal(Date.now() + days); + // An overflowed timer would have fired at once. + vi.advanceTimersByTime(1); + expect(signal.aborted).toBe(false); + vi.advanceTimersByTime(MAX_TIMER_MS); + expect(signal.aborted).toBe(false); + vi.advanceTimersByTime(days - MAX_TIMER_MS - 2); + expect(signal.aborted).toBe(false); + vi.advanceTimersByTime(1); + expect(signal.aborted).toBe(true); + }); + + it("stops when cancelled", () => { + vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout", "Date"] }); + const { signal, cancel } = deadlineSignal(Date.now() + 100); + cancel(); + vi.advanceTimersByTime(1_000); + expect(signal.aborted).toBe(false); + }); +}); diff --git a/packages/ai/src/task/__tests__/CacheCheckpoint.test.ts b/packages/ai/src/task/__tests__/CacheCheckpoint.test.ts index 31aa5db9a0..27eaa953a9 100644 --- a/packages/ai/src/task/__tests__/CacheCheckpoint.test.ts +++ b/packages/ai/src/task/__tests__/CacheCheckpoint.test.ts @@ -519,9 +519,18 @@ describe("CacheCheckpointTask storage cost inside a graph", () => { describe("ToolCallingTask checkpoint ports", () => { let toolCalls: { session: AiSessionContext | undefined }[]; + let withNativeTurn = false; const toolUseFn: AiProviderRunFn = async (_input, _model, _signal, emit, _schema, session) => { toolCalls.push({ session }); + if (withNativeTurn) { + emit({ type: "text-delta", port: "reasoning", textDelta: "thought" } as any); + emit({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: "p:m", payload: "SIGNED" }, + } as any); + } emit({ type: "text-delta", port: "text", textDelta: "done" } as any); emit({ type: "finish", data: { text: "", toolCalls: [] } } as any); }; @@ -545,6 +554,7 @@ describe("ToolCallingTask checkpoint ports", () => { setAiProviderRegistry(new AiProviderRegistry()); clearCheckpointsForTesting(); toolCalls = []; + withNativeTurn = false; const provider = new CheckpointTestProvider([ { serves: ["text.generation", "tool-use"] as Capability[], runFn: toolUseFn }, { serves: ["cache.checkpoint"] as Capability[], runFn: ckptWarmFn }, @@ -598,6 +608,48 @@ describe("ToolCallingTask checkpoint ports", () => { expect(entry?.prefix.messages?.at(-2)?.role).toBe("user"); }); + it("records the turn's reasoning and native turn in the emitted checkpoint", async () => { + withNativeTurn = true; + const scope = new ResourceScope(); + const out = await new ToolCallingTask().run( + { model: toolModel(), prompt: "hi", tools: [aTool], emitCheckpoint: true }, + { resourceScope: scope } + ); + const entry = getCheckpoint((out as { checkpoint?: string }).checkpoint!); + expect(entry?.prefix.messages?.at(-1)).toEqual({ + role: "assistant", + content: [ + { type: "reasoning", text: "thought", provider: "p:m", payload: "SIGNED" }, + { type: "text", text: "done" }, + ], + }); + }); + + it("records a tool call's provider signature in the emitted checkpoint", async () => { + const signedCall = async (_i: any, _m: any, _s: any, emit: any) => { + emit({ + type: "object-delta", + port: "toolCalls", + objectDelta: [{ id: "c1", name: "a", input: {}, providerSignature: "SIG" }], + }); + emit({ type: "finish", data: { text: "", toolCalls: [] } }); + }; + setAiProviderRegistry(new AiProviderRegistry()); + clearCheckpointsForTesting(); + await new CheckpointTestProvider([ + { serves: ["text.generation", "tool-use"] as Capability[], runFn: signedCall as any }, + { serves: ["cache.checkpoint"] as Capability[], runFn: ckptWarmFn }, + ]).register({ queue: { autoCreate: false } }); + const out = await new ToolCallingTask().run( + { model: toolModel(), prompt: "hi", tools: [aTool], emitCheckpoint: true }, + { resourceScope: new ResourceScope() } + ); + const entry = getCheckpoint((out as { checkpoint?: string }).checkpoint!); + expect(entry?.prefix.messages?.at(-1)?.content).toEqual([ + { type: "tool_use", id: "c1", name: "a", input: {}, providerSignature: "SIG" }, + ]); + }); + it("keepParentCheckpoint preserves the parent", async () => { registerCheckpoint("ckpt-parent", { provider: CKPT_PROVIDER, diff --git a/packages/ai/src/task/__tests__/ChatHistory.test.ts b/packages/ai/src/task/__tests__/ChatHistory.test.ts index 69ee679c43..e2a5eb23e4 100644 --- a/packages/ai/src/task/__tests__/ChatHistory.test.ts +++ b/packages/ai/src/task/__tests__/ChatHistory.test.ts @@ -7,6 +7,7 @@ import type { ChatMessage } from "@workglow/ai"; import { DEFAULT_MAX_HISTORY_CHARS, + measureHistoryForModel, normalizeHistoryForModel, trimHistoryForModel, } from "@workglow/ai"; @@ -67,6 +68,28 @@ describe("trimHistoryForModel", () => { expect(trimHistoryForModel(history)).toEqual(history); }); + it("keeps a system message past the leading run when it cuts the turn around it", () => { + const filler = "z".repeat(400); + const system: ChatMessage = { role: "system", content: [{ type: "text", text: "be terse" }] }; + const history = [user(`old ${filler}`), system, assistant(`reply ${filler}`), user("now")]; + const trimmed = trimHistoryForModel(history, 100, 3); + expect(trimmed).toEqual([system, user("now")]); + }); + + it("keeps a run of user messages whole, never cutting between them", () => { + const filler = "z".repeat(400); + const history = [ + user(`old ${filler}`), + assistant(`reply ${filler}`), + user("the question"), + user("the new prompt"), + ]; + // Too small for everything: the old turn goes, the pending user turn stays whole, + // even with keepFrom at the newest prompt. + const trimmed = trimHistoryForModel(history, 100, 3); + expect(trimmed).toEqual([user("the question"), user("the new prompt")]); + }); + it("cuts only at a user message, so no tool_result outlives its tool_use", () => { // The invariant is not "the first message is a user message" — it is that // every surviving `tool_result` still has the `tool_use` it answers, which @@ -99,6 +122,32 @@ describe("trimHistoryForModel", () => { for (const id of results) expect(uses.has(id)).toBe(true); }); + it("does not count a reasoning block's payload against the budget", () => { + // The payload is a provider's own turn, replayed verbatim and never shown + // to the model as text, so a long one must not push earlier turns out. + const build = (payload: string | undefined): ChatMessage[] => { + const history: ChatMessage[] = []; + for (let i = 0; i < 6; i++) { + history.push(user(`ask ${i} ${"q".repeat(200)}`)); + history.push({ + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: "openai:m", ...(payload ? { payload } : {}) }, + { type: "text", text: `done ${i} ${"a".repeat(200)}` }, + ], + }); + } + return history; + }; + const budget = 1500; + const plain = trimHistoryForModel(build(undefined), budget); + expect(plain.length).toBeLessThan(12); + expect(plain.length).toBeGreaterThan(0); + const withPayload = trimHistoryForModel(build("p".repeat(5000)), budget); + expect(withPayload).toHaveLength(plain.length); + expect(withPayload[0]).toEqual(plain[0]); + }); + it("keeps the newest turn even when it alone exceeds the budget", () => { // Discarding here is how a conversation gets erased for being too long. const history = [user("old"), assistant("old reply"), user("x".repeat(5000))]; @@ -153,6 +202,36 @@ describe("trimHistoryForModel", () => { expect(trimmed.map((m) => m.role)).toEqual(["system", "system", "user"]); }); + it("never cuts past keepFrom", () => { + // A turn's own reminder is a user message; cutting there would keep the + // reminder and drop the question it is about. + const history = [ + user("old".repeat(200)), + assistant("old reply"), + user("the question"), + assistant("x".repeat(1000)), + user("check your work"), + ]; + expect(trimHistoryForModel(history, 500)).toEqual([history[4]]); + expect(trimHistoryForModel(history, 500, 2)).toEqual(history.slice(2)); + }); + + it("measures history as the trim does, payload left out", () => { + const plain: ChatMessage[] = [user("a"), assistant("b")]; + const withPayload: ChatMessage[] = [ + user("a"), + { + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: "openai:m", payload: "p".repeat(5000) }, + { type: "text", text: "b" }, + ], + }, + ]; + expect(measureHistoryForModel(plain)).toBeGreaterThan(0); + expect(measureHistoryForModel(withPayload)).toBeLessThan(5000); + }); + it("exposes a default budget callers can reason about", () => { expect(DEFAULT_MAX_HISTORY_CHARS).toBeGreaterThan(0); }); diff --git a/packages/test/src/test/ai-provider-api/DeepSeekProvider.test.ts b/packages/test/src/test/ai-provider-api/DeepSeekProvider.test.ts index ab68797b1a..b3c7ba55d8 100644 --- a/packages/test/src/test/ai-provider-api/DeepSeekProvider.test.ts +++ b/packages/test/src/test/ai-provider-api/DeepSeekProvider.test.ts @@ -11,6 +11,7 @@ import { DEEPSEEK_ALLOWED_HOSTS, DEEPSEEK_DEFAULT_BASE_URL, DEEPSEEK_DEFAULT_REASONING_ALLOWANCE, + reasoningParams, resolveMaxTokens, } from "@workglow/deepseek/ai"; import { describe, expect, it } from "vitest"; @@ -208,6 +209,37 @@ describe("resolveMaxTokens", () => { }); }); +describe("reasoningParams", () => { + const flash = (effort?: string) => + ({ + provider: "DEEPSEEK", + provider_config: { model_name: "deepseek-flash" }, + ...(effort ? { effort } : {}), + }) as never; + + it("sends the effort a caller set, in DeepSeek's names", () => { + expect(reasoningParams(flash("low"))).toEqual({ reasoning_effort: "low" }); + expect(reasoningParams(flash("high"))).toEqual({ reasoning_effort: "high" }); + expect(reasoningParams(flash("extra"))).toEqual({ reasoning_effort: "xhigh" }); + expect(reasoningParams(flash("ultra"))).toEqual({ reasoning_effort: "max" }); + }); + + it("turns thinking off for none, which DeepSeek spells as its own switch", () => { + expect(reasoningParams(flash("none"))).toEqual({ thinking: { type: "disabled" } }); + }); + + it("sends nothing when no effort is set, or the model does not reason", () => { + expect(reasoningParams(flash())).toEqual({}); + expect( + reasoningParams({ + provider: "DEEPSEEK", + provider_config: { model_name: "deepseek-chat" }, + effort: "high", + } as never) + ).toEqual({}); + }); +}); + describe("assertNotTruncatedByReasoning", () => { it("throws when the budget was exhausted before any content was emitted", () => { expect(() => assertNotTruncatedByReasoning("length", "", 4096)).toThrow( diff --git a/packages/test/src/test/ai-provider-api/GoogleGeminiProvider.test.ts b/packages/test/src/test/ai-provider-api/GoogleGeminiProvider.test.ts index a77112e242..c42a880323 100644 --- a/packages/test/src/test/ai-provider-api/GoogleGeminiProvider.test.ts +++ b/packages/test/src/test/ai-provider-api/GoogleGeminiProvider.test.ts @@ -6,7 +6,7 @@ import type { ChatMessage, ModelRecord } from "@workglow/ai"; import { GEMINI_FALLBACK_MODELS, _testOnly } from "@workglow/google-gemini/ai"; -import { describe, expect, it } from "vitest"; +import { afterEach, describe, expect, it } from "vitest"; const { GoogleGeminiQueuedProvider, GEMINI_RUN_FN_SPECS, GEMINI_RUN_FNS } = _testOnly; @@ -296,3 +296,227 @@ describe("buildGeminiContents multi-turn", () => { expect("thoughtSignature" in fnPart).toBe(false); }); }); + +describe("buildGeminiContents tool results", () => { + const { buildGeminiContents } = _testOnly; + + const call = (id: string, name: string) => ({ + role: "assistant", + content: [{ type: "tool_use", id, name, input: {} }], + }); + const result = (id: string, content: any[], is_error?: boolean) => ({ + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: id, + content, + ...(is_error ? { is_error } : {}), + }, + ], + }); + + it.each([ + ["42", { result: 42 }], + ["null", { result: null }], + ["[1,2]", { result: [1, 2] }], + ['"s"', { result: "s" }], + ['{"a":1}', { a: 1 }], + ["plain words", { result: "plain words" }], + ])("wraps a %s tool result as an object", (text, response) => { + const contents = buildGeminiContents( + [call("c1", "t"), result("c1", [{ type: "text", text }])] as any, + "x" + ); + expect(contents.at(-1).parts[0].functionResponse.response).toEqual(response); + }); + + it("holds tool images until every separate tool message of the turn is out", () => { + const contents = buildGeminiContents( + [ + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + result("c1", [{ type: "image", mimeType: "image/png", data: "AAAA" }]), + result("c2", [{ type: "text", text: "ok" }]), + ] as any, + "x" + ); + const responses = contents.filter((c: any) => c.parts[0]?.functionResponse); + expect(responses).toHaveLength(2); + expect(contents.indexOf(responses[1])).toBeLessThan(contents.length - 1); + expect(contents.at(-1)).toEqual({ + role: "user", + parts: [ + { text: "Images returned by the tool calls above:" }, + { inlineData: { mimeType: "image/png", data: "AAAA" } }, + ], + }); + }); + + it("reports an error result under `error`", () => { + const contents = buildGeminiContents( + [call("c1", "t"), result("c1", [{ type: "text", text: "boom" }], true)] as any, + "x" + ); + expect(contents.at(-1).parts[0].functionResponse.response).toEqual({ error: "boom" }); + }); + + it("reports an is_error result whose text is JSON under `error`, unparsed", () => { + const contents = buildGeminiContents( + [call("c1", "t"), result("c1", [{ type: "text", text: '{"code":1}' }], true)] as any, + "x" + ); + expect(contents.at(-1).parts[0].functionResponse.response).toEqual({ + error: '{"code":1}', + }); + }); + + it("returns images in functionResponse.parts when the model takes them there", () => { + const contents = buildGeminiContents( + [ + call("c1", "t"), + result("c1", [ + { type: "text", text: "here" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ]), + ] as any, + "x", + { multimodalFunctionResponses: true } + ); + expect(contents.at(-1).parts[0].functionResponse).toEqual({ + name: "t", + response: { result: "here" }, + parts: [{ inlineData: { mimeType: "image/png", data: "AAAA" } }], + }); + }); + + it("sends images in a separate user turn after the function responses by default", () => { + const contents = buildGeminiContents( + [ + call("c1", "t"), + result("c1", [ + { type: "text", text: "here" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ]), + ] as any, + "x" + ); + expect(contents).toHaveLength(3); + expect(contents[1].parts).toEqual([ + { functionResponse: { name: "t", response: { result: "here" } } }, + ]); + expect(contents[2]).toEqual({ + role: "user", + parts: [ + { text: "Images returned by the tool calls above:" }, + { inlineData: { mimeType: "image/png", data: "AAAA" } }, + ], + }); + }); + + it("puts every image of a tool message in one labelled turn after all of its function responses", () => { + const toolMessage = { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + { + type: "tool_result", + tool_use_id: "c2", + content: [{ type: "text", text: "two" }], + }, + ], + }; + const contents = buildGeminiContents( + [call("c1", "a"), call("c2", "b"), toolMessage] as any, + "x" + ); + const [responses, images] = contents.slice(-2); + expect(responses.parts).toEqual([ + { functionResponse: { name: "a", response: { result: "" } } }, + { functionResponse: { name: "b", response: { result: "two" } } }, + ]); + expect(images.parts).toEqual([ + { text: "Images returned by the tool calls above:" }, + { inlineData: { mimeType: "image/png", data: "AAAA" } }, + ]); + }); +}); + +describe("geminiSupportsMultimodalFunctionResponses", () => { + const { geminiSupportsMultimodalFunctionResponses } = _testOnly; + + it.each(["gemini-3-flash-preview", "gemini-3.5-flash", "gemini-10-pro"])( + "accepts multimodal function responses for %s", + (name) => { + expect(geminiSupportsMultimodalFunctionResponses(name)).toBe(true); + } + ); + + it.each(["gemini-2.5-flash", "gemini-flash-latest", "gemini-2.0-flash"])( + "rejects multimodal function responses for %s", + (name) => { + expect(geminiSupportsMultimodalFunctionResponses(name)).toBe(false); + } + ); +}); + +describe("Gemini tool calling thought summaries", () => { + afterEach(() => { + _testOnly.setGeminiClientForTests(undefined); + }); + + const run = async (model_name: string, parts: readonly Record[]) => { + const requests: Array> = []; + _testOnly.setGeminiClientForTests({ + models: { + generateContentStream: async (request: Record) => { + requests.push(request); + return { + async *[Symbol.asyncIterator]() { + yield { candidates: [{ content: { parts } }] }; + }, + }; + }, + }, + } as never); + const registration = GEMINI_RUN_FNS.find(({ serves }) => + (serves as readonly string[]).includes("tool-use") + ); + const events: any[] = []; + await (registration!.runFn as any)( + { prompt: "hi", tools: [] }, + { provider: "GOOGLE_GEMINI", provider_config: { api_key: "k", model_name } }, + undefined, + (event: unknown) => events.push(event), + undefined + ); + return { requests, events }; + }; + + it("emits thought parts on reasoning and the rest on text", async () => { + const { requests, events } = await run("gemini-2.5-flash", [ + { text: "pondering", thought: true }, + { text: "answer" }, + ]); + const deltas = events.filter((e) => e.type === "text-delta"); + expect(deltas).toEqual([ + { type: "text-delta", port: "reasoning", textDelta: "pondering" }, + { type: "text-delta", port: "text", textDelta: "answer" }, + ]); + expect(requests[0].config.thinkingConfig.includeThoughts).toBe(true); + }); + + it("sends no thinkingConfig to a model that does not think", async () => { + const { requests } = await run("gemini-embedding-001", [{ text: "answer" }]); + expect(requests[0].config.thinkingConfig).toBeUndefined(); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/HFI_Effort.test.ts b/packages/test/src/test/ai-provider-api/HFI_Effort.test.ts new file mode 100644 index 0000000000..d11a26af5b --- /dev/null +++ b/packages/test/src/test/ai-provider-api/HFI_Effort.test.ts @@ -0,0 +1,81 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { _testOnly, hfiEffortPolicy, hfiReasoningParams } from "@workglow/huggingface-inference/ai"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const model = (effort?: string, model_name = "openai/gpt-oss-120b") => + ({ provider_config: { model_name }, ...(effort ? { effort } : {}) }) as any; + +describe("Hugging Face Inference effort", () => { + it("sends reasoning_effort for a set effort", () => { + expect(hfiReasoningParams(model("high"))).toEqual({ reasoning_effort: "high" }); + expect(hfiReasoningParams(model("low"))).toEqual({ reasoning_effort: "low" }); + }); + + it("maps extra and ultra to high and sends nothing for none or unset", () => { + expect(hfiReasoningParams(model("ultra"))).toEqual({ reasoning_effort: "high" }); + expect(hfiReasoningParams(model("extra"))).toEqual({ reasoning_effort: "high" }); + expect(hfiReasoningParams(model("none"))).toEqual({}); + expect(hfiReasoningParams(model())).toEqual({}); + }); + + it("gives embedding models no effort", () => { + expect(hfiEffortPolicy(model(undefined, "BAAI/bge-small-en-v1.5")).supported).toEqual([]); + expect(hfiReasoningParams(model("high", "BAAI/bge-small-en-v1.5"))).toEqual({}); + }); + + it("excludes whisper and tts ids by name, not by substring", () => { + expect(hfiEffortPolicy(model(undefined, "openai/whisper-large-v3")).supported).toEqual([]); + expect(hfiEffortPolicy(model(undefined, "hexgrad/tts-1")).supported).toEqual([]); + expect(hfiEffortPolicy(model(undefined, "org/matts-chat")).supported).toContain("high"); + expect(hfiReasoningParams(model("high", "org/matts-chat"))).toEqual({ + reasoning_effort: "high", + }); + }); + + it("is what the provider reports for a model", () => { + const provider = new _testOnly.HfInferenceQueuedProvider() as any; + expect(provider.effortPolicy(model("high"))?.supported).toContain("high"); + expect(provider.effortPolicy(model(undefined, "BAAI/bge-small-en-v1.5"))?.supported).toEqual( + [] + ); + }); +}); + +describe("Hugging Face Inference text generation effort", () => { + let sent: Record[] = []; + class CapturingClient { + chatCompletionStream(params: Record) { + sent.push(params); + return (async function* () { + yield { choices: [{ delta: { content: "ok" } }] }; + })(); + } + } + + beforeEach(() => { + sent = []; + _testOnly._setHfInferenceSDKForTesting({ InferenceClient: CapturingClient } as any); + }); + + afterEach(() => { + _testOnly._setHfInferenceSDKForTesting(undefined); + }); + + it("sends the model's effort on a plain text-generation request", async () => { + const runFn = _testOnly.HFI_RUN_FNS.find( + (r) => [...r.serves].sort().join(",") === "text.generation" + )!.runFn; + const withKey = (effort?: string) => ({ + ...model(effort), + provider_config: { model_name: "openai/gpt-oss-120b", api_key: "k" }, + }); + await runFn({ prompt: "hi" } as any, withKey("high"), new AbortController().signal, () => {}); + await runFn({ prompt: "hi" } as any, withKey(), new AbortController().signal, () => {}); + expect(sent.map((params) => params.reasoning_effort)).toEqual(["high", undefined]); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/LlamaCppServer_TextGenerationStream.test.ts b/packages/test/src/test/ai-provider-api/LlamaCppServer_TextGenerationStream.test.ts index e1ab72b9ad..0f7862b801 100644 --- a/packages/test/src/test/ai-provider-api/LlamaCppServer_TextGenerationStream.test.ts +++ b/packages/test/src/test/ai-provider-api/LlamaCppServer_TextGenerationStream.test.ts @@ -73,6 +73,63 @@ describe("createLlamaCppServerTextGenerationStream", () => { ]); }); + it("sends a history of system turns alone with no invented user turn", async () => { + const fetchSpy = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(sseResponse([dataLine("ok"), "data: [DONE]\n"])); + const fn = createLlamaCppServerTextGenerationStream({}); + await fn( + { prompt: "", messages: [{ role: "system", content: "be brief" }] } as any, + model, + undefined as any, + (_e: any) => undefined + ); + const body = JSON.parse(asText((fetchSpy.mock.calls[0]![1] as RequestInit).body)); + expect(body.messages).toEqual([{ role: "system", content: "be brief" }]); + }); + + it("converts block history to chat-completions parts and drops blocks it has no form for", async () => { + const fetchSpy = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(sseResponse([dataLine("ok"), "data: [DONE]\n"])); + const fn = createLlamaCppServerTextGenerationStream({}); + await fn( + { + prompt: "", + messages: [ + { + role: "user", + content: [ + { type: "text", text: "What is this?" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "It looks red." }, + { type: "text", text: "A red square." }, + ], + }, + ], + } as any, + model, + undefined as any, + () => undefined + ); + const body = JSON.parse(asText((fetchSpy.mock.calls[0]![1] as RequestInit).body)); + expect(body.messages).toEqual([ + { + role: "user", + content: [ + { type: "text", text: "What is this?" }, + { type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } }, + ], + }, + { role: "assistant", content: "A red square." }, + ]); + }); + it("throws on non-2xx with informative message", async () => { vi.spyOn(globalThis, "fetch").mockResolvedValue(new Response("nope", { status: 500 })); const fn = createLlamaCppServerTextGenerationStream({}); diff --git a/packages/test/src/test/ai-provider-api/LlamaCppServer_ToolCalling.test.ts b/packages/test/src/test/ai-provider-api/LlamaCppServer_ToolCalling.test.ts index a4bc78031b..7de871624b 100644 --- a/packages/test/src/test/ai-provider-api/LlamaCppServer_ToolCalling.test.ts +++ b/packages/test/src/test/ai-provider-api/LlamaCppServer_ToolCalling.test.ts @@ -120,4 +120,149 @@ describe("createLlamaCppServerToolCallingStream", () => { const body = JSON.parse(asText((fetchSpy.mock.calls[0]![1] as RequestInit).body)); expect(body.tools).toBeUndefined(); }); + describe("conversation and reasoning", () => { + async function requestBodyFor(input: object): Promise { + const fetchSpy = vi + .spyOn(globalThis, "fetch") + .mockResolvedValue(sseChunks([{ choices: [{ delta: { content: "ok" } }] }])); + const fn = createLlamaCppServerToolCallingStream({}); + await fn( + { tools: TOOLS, toolChoice: "auto", ...input } as any, + model, + undefined as any, + () => undefined + ); + return JSON.parse(asText((fetchSpy.mock.calls[0]![1] as RequestInit).body)); + } + + it("sends the whole conversation, including its own tool calls and their results", async () => { + const body = await requestBodyFor({ + prompt: "p", + messages: [ + { role: "user", content: [{ type: "text", text: "p" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "lookup", input: {} }], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "text", text: "42" }], + }, + ], + }, + ], + }); + expect(body.messages).toEqual([ + { role: "user", content: [{ type: "text", text: "p" }] }, + { + role: "assistant", + content: null, + tool_calls: [ + { id: "c1", type: "function", function: { name: "lookup", arguments: "{}" } }, + ], + reasoning_content: "", + }, + { role: "tool", content: "42", tool_call_id: "c1" }, + ]); + }); + + it("counts a parts-array user message in the provisional prompt estimate", async () => { + vi.spyOn(globalThis, "fetch").mockResolvedValue( + sseChunks([{ choices: [{ delta: { content: "ok" } }] }]) + ); + const events: any[] = []; + await createLlamaCppServerToolCallingStream({})( + { + tools: TOOLS, + toolChoice: "auto", + prompt: "", + messages: [{ role: "user", content: [{ type: "text", text: "word ".repeat(400) }] }], + } as any, + model, + undefined as any, + (event: any) => events.push(event) + ); + const first = events.find((event) => event.type === "usage"); + expect(first?.usage?.input).toBeGreaterThan(0); + }); + + it("replays an assistant turn's reasoning as reasoning_content", async () => { + const body = await requestBodyFor({ + prompt: "p", + messages: [ + { role: "user", content: [{ type: "text", text: "p" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "I should look it up" }, + { type: "tool_use", id: "c1", name: "lookup", input: {} }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "c1", content: [{ type: "text", text: "42" }] }, + ], + }, + ], + }); + const assistant = body.messages.find((m: any) => m.role === "assistant"); + expect(assistant.reasoning_content).toBe("I should look it up"); + }); + + it("sends a tool-result image as a user message after the string tool message", async () => { + const body = await requestBodyFor({ + prompt: "p", + messages: [ + { role: "user", content: [{ type: "text", text: "p" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "shot", input: {} }], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + ], + }); + const toolIdx = body.messages.findIndex((m: any) => m.role === "tool"); + expect(typeof body.messages[toolIdx].content).toBe("string"); + const after = body.messages[toolIdx + 1]; + expect(after.role).toBe("user"); + expect(after.content).toContainEqual({ + type: "image_url", + image_url: { url: "data:image/png;base64,AAAA" }, + }); + }); + + it("emits delta.reasoning_content on the reasoning port, not on text", async () => { + vi.spyOn(globalThis, "fetch").mockResolvedValue( + sseChunks([ + { choices: [{ delta: { reasoning_content: "hmm" } }] }, + { choices: [{ delta: { content: "answer" } }] }, + ]) + ); + const fn = createLlamaCppServerToolCallingStream({}); + const events: any[] = []; + await fn( + { prompt: "p", tools: TOOLS, toolChoice: "auto" } as any, + model, + undefined as any, + (e: any) => events.push(e) + ); + expect(events).toContainEqual({ type: "text-delta", port: "reasoning", textDelta: "hmm" }); + const textDeltas = events.filter((e) => e.type === "text-delta" && e.port === "text"); + expect(textDeltas.map((e) => e.textDelta)).toEqual(["answer"]); + }); + }); }); diff --git a/packages/test/src/test/ai-provider-api/Ollama_TextGenerationStream.test.ts b/packages/test/src/test/ai-provider-api/Ollama_TextGenerationStream.test.ts index 0fdb41b6b8..1203f8f7e7 100644 --- a/packages/test/src/test/ai-provider-api/Ollama_TextGenerationStream.test.ts +++ b/packages/test/src/test/ai-provider-api/Ollama_TextGenerationStream.test.ts @@ -207,3 +207,105 @@ describe("createOllamaTextGenerationStream usage", () => { expect(finish.usage.output).toBe(3); }); }); + +describe("createOllamaTextGenerationStream chat history", () => { + const model = { + model_id: "ollama:test", + provider_config: { model_name: "llava" }, + } as any; + + it("sends block history as Ollama messages: string content, images beside it, thinking replayed", async () => { + const chat = vi.fn().mockResolvedValue(makeFakeStream(["ok"])); + const streamFn = createOllamaTextGenerationStream(vi.fn().mockResolvedValue({ chat })); + await streamFn( + { + prompt: "", + systemPrompt: "be brief", + messages: [ + { role: "system", content: [{ type: "text", text: "answer in English" }] }, + { + role: "user", + content: [ + { type: "text", text: "What is this?" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "It looks red." }, + { type: "text", text: "A red square." }, + ], + }, + { role: "user", content: "Thanks" }, + ], + } as any, + model, + new AbortController().signal, + () => {} + ); + + expect(chat).toHaveBeenCalledTimes(1); + expect(chat.mock.calls[0]![0].messages).toEqual([ + { role: "system", content: "be brief\n\nanswer in English" }, + { role: "user", content: "What is this?", images: ["AAAA"] }, + { role: "assistant", content: "A red square.", thinking: "It looks red." }, + { role: "user", content: "Thanks" }, + ]); + }); + + it("keeps a tool exchange in the history whole: the call and the result it answers", async () => { + const chat = vi.fn().mockResolvedValue(makeFakeStream(["ok"])); + const streamFn = createOllamaTextGenerationStream(vi.fn().mockResolvedValue({ chat })); + await streamFn( + { + prompt: "", + messages: [ + { role: "user", content: [{ type: "text", text: "weather?" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "weather", input: { city: "Paris" } }], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "text", text: "sunny" }], + }, + ], + }, + { role: "user", content: "and tomorrow?" }, + ], + } as any, + model, + new AbortController().signal, + () => {} + ); + expect(chat.mock.calls[0]![0].messages).toEqual([ + { role: "user", content: "weather?" }, + { + role: "assistant", + content: "", + tool_calls: [{ function: { name: "weather", arguments: { city: "Paris" } } }], + }, + { role: "tool", content: "sunny", tool_name: "weather" }, + { role: "user", content: "and tomorrow?" }, + ]); + }); + it("sends the model's effort as think, and nothing for a model with no effort set", async () => { + const chat = vi.fn().mockResolvedValue(makeFakeStream(["ok"])); + const streamFn = createOllamaTextGenerationStream(vi.fn().mockResolvedValue({ chat })); + const qwen = (effort?: string): any => ({ + model_id: "ollama:qwen3", + provider: "OLLAMA", + provider_config: { model_name: "qwen3:8b" }, + ...(effort === undefined ? {} : { effort }), + }); + await streamFn({ prompt: "hi" } as any, qwen("high"), new AbortController().signal, () => {}); + await streamFn({ prompt: "hi" } as any, qwen("none"), new AbortController().signal, () => {}); + await streamFn({ prompt: "hi" } as any, qwen(), new AbortController().signal, () => {}); + expect(chat.mock.calls.map((call) => call[0].think)).toEqual([true, false, undefined]); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/Ollama_ToolCalling.test.ts b/packages/test/src/test/ai-provider-api/Ollama_ToolCalling.test.ts new file mode 100644 index 0000000000..1c1b9a727f --- /dev/null +++ b/packages/test/src/test/ai-provider-api/Ollama_ToolCalling.test.ts @@ -0,0 +1,441 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { ollamaEffortPolicy } from "@workglow/ollama/ai"; +import { readFileSync } from "node:fs"; +import { describe, expect, it, vi } from "vitest"; +import { ollamaThinkParam } from "../../../../../providers/ollama/src/ai/common/Ollama_EffortPolicy"; +import { toOllamaMessages } from "../../../../../providers/ollama/src/ai/common/Ollama_Messages"; +import { createOllamaToolCallingStream } from "../../../../../providers/ollama/src/ai/common/Ollama_ToolCalling"; + +const model = (name: string, effort?: string): any => ({ + provider: "OLLAMA", + provider_config: { model_name: name }, + ...(effort === undefined ? {} : { effort }), +}); + +describe("toOllamaMessages", () => { + it("refuses audio rather than sending the turn without it", () => { + const audio = { type: "audio", mimeType: "audio/wav", data: "AAAA" }; + expect(() => toOllamaMessages({ prompt: [audio], tools: [] } as never)).toThrow(/audio/); + expect(() => + toOllamaMessages({ + prompt: "", + tools: [], + messages: [{ role: "user", content: [{ type: "text", text: "hear this" }, audio] }], + } as never) + ).toThrow(/audio/); + }); + + it("keeps tool calls, tool names, thinking and images across a turn", () => { + const input = { + systemPrompt: "be brief", + prompt: "weather?", + tools: [], + messages: [ + { + role: "user", + content: [ + { type: "text", text: "weather?" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "need the tool" }, + { type: "text", text: "Checking." }, + { type: "tool_use", id: "c1", name: "weather", input: { city: "Paris" } }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "sunny" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + ], + }, + ], + }, + ], + } as any; + expect(toOllamaMessages(input)).toEqual([ + { role: "system", content: "be brief" }, + { role: "user", content: "weather?", images: ["AAAA"] }, + { + role: "assistant", + content: "Checking.", + thinking: "need the tool", + tool_calls: [{ function: { name: "weather", arguments: { city: "Paris" } } }], + }, + { + role: "tool", + content: "sunny\n[image 1] follows in the next message", + tool_name: "weather", + }, + { + role: "user", + content: "Images returned by the tool calls above, in order: [image 1]", + images: ["BBBB"], + }, + ]); + }); + + it("marks each moved image where it stood, numbered across the turn's results", () => { + const image = (data: string) => ({ type: "image", mimeType: "image/png", data }); + const out = toOllamaMessages({ + prompt: "", + tools: [], + messages: [ + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "before:" }, + image("AAAA"), + { type: "text", text: "after:" }, + image("BBBB"), + ], + }, + { type: "tool_result", tool_use_id: "c2", content: [image("CCCC")] }, + ], + }, + ], + } as never); + expect(out.map((message) => message.content)).toEqual([ + "", + "before:\n[image 1] follows in the next message\nafter:\n[image 2] follows in the next message", + "[image 3] follows in the next message", + "Images returned by the tool calls above, in order: [image 1], [image 2], [image 3]", + ]); + expect(out.at(-1)!.images).toEqual(["AAAA", "BBBB", "CCCC"]); + }); + + it("holds tool images until every separate tool message of the turn is out", () => { + const out = toOllamaMessages({ + prompt: "", + tools: [], + messages: [ + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "c2", content: [{ type: "text", text: "ok" }] }, + ], + }, + ], + } as any); + expect(out.map((m) => m.role)).toEqual(["assistant", "tool", "tool", "user"]); + expect(out[3]!.images).toEqual(["AAAA"]); + }); + + it("keeps a non-text block a tool returned, as JSON", () => { + const nested = { type: "tool_use", id: "x", name: "inner", input: { a: 1 } }; + const out = toOllamaMessages({ + prompt: "", + tools: [], + messages: [ + { role: "assistant", content: [{ type: "tool_use", id: "c1", name: "t", input: {} }] }, + { role: "tool", content: [{ type: "tool_result", tool_use_id: "c1", content: [nested] }] }, + ], + } as any); + expect(out[1]).toEqual({ role: "tool", content: JSON.stringify(nested), tool_name: "t" }); + }); + + it("lifts system turns in the history into the system prompt", () => { + const out = toOllamaMessages({ + systemPrompt: "be brief", + prompt: "", + tools: [], + messages: [ + { role: "system", content: [{ type: "text", text: "answer in French" }] }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ], + } as any); + expect(out).toEqual([ + { role: "system", content: "be brief\n\nanswer in French" }, + { role: "user", content: "hi" }, + ]); + }); + + it("sends no prompt for a history of system turns alone", () => { + const out = toOllamaMessages({ + prompt: "stale", + tools: [], + messages: [{ role: "system", content: [{ type: "text", text: "rule" }] }], + } as any); + expect(out).toEqual([{ role: "system", content: "rule" }]); + }); + + it("prefers messages over prompt when both are present", () => { + const out = toOllamaMessages({ + prompt: "first", + tools: [], + messages: [ + { role: "user", content: [{ type: "text", text: "first" }] }, + { role: "assistant", content: [{ type: "text", text: "second" }] }, + ], + } as any); + expect(out).toEqual([ + { role: "user", content: "first" }, + { role: "assistant", content: "second" }, + ]); + }); + + it("uses a string prompt when there are no messages", () => { + expect(toOllamaMessages({ systemPrompt: "s", prompt: "hi", tools: [] } as any)).toEqual([ + { role: "system", content: "s" }, + { role: "user", content: "hi" }, + ]); + expect(toOllamaMessages({ prompt: "hi", tools: [], messages: [] } as any)).toEqual([ + { role: "user", content: "hi" }, + ]); + }); + + it("joins an array prompt's text with newlines and keeps its images", () => { + const out = toOllamaMessages({ + prompt: [ + "one", + { type: "text", text: "two" }, + { type: "image", mimeType: "image/png", data: "IMG" }, + ], + tools: [], + } as any); + expect(out).toEqual([{ role: "user", content: "one\ntwo", images: ["IMG"] }]); + }); + + it("gives an assistant turn holding only a tool call empty content", () => { + const out = toOllamaMessages({ + prompt: "x", + tools: [], + messages: [ + { role: "user", content: [{ type: "text", text: "x" }] }, + { role: "assistant", content: [{ type: "tool_use", id: "a", name: "t", input: {} }] }, + ], + } as any); + expect(out[1]).toEqual({ + role: "assistant", + content: "", + tool_calls: [{ function: { name: "t", arguments: {} } }], + }); + }); + + it("omits tool_name for a result whose call id is unknown", () => { + const out = toOllamaMessages({ + prompt: "x", + tools: [], + messages: [ + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "missing", + content: [{ type: "text", text: "r" }], + }, + ], + }, + ], + } as any); + expect(out).toEqual([{ role: "tool", content: "r" }]); + }); + + it("prefixes an error result's content with Error:", () => { + const out = toOllamaMessages({ + prompt: "x", + tools: [], + messages: [ + { role: "assistant", content: [{ type: "tool_use", id: "a", name: "t", input: {} }] }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "a", + is_error: true, + content: [{ type: "text", text: "boom" }], + }, + ], + }, + ], + } as any); + expect(out[1]).toEqual({ role: "tool", content: "Error: boom", tool_name: "t" }); + }); + + it("resolves a repeated call id against the most recent preceding call", () => { + const result = (text: string): any => ({ + role: "tool", + content: [{ type: "tool_result", tool_use_id: "call_0", content: [{ type: "text", text }] }], + }); + const out = toOllamaMessages({ + prompt: "x", + tools: [], + messages: [ + { role: "assistant", content: [{ type: "tool_use", id: "call_0", name: "a", input: {} }] }, + result("ra"), + { role: "assistant", content: [{ type: "tool_use", id: "call_0", name: "b", input: {} }] }, + result("rb"), + ], + } as any); + expect(out.filter((m) => m.role === "tool").map((m) => m.tool_name)).toEqual(["a", "b"]); + }); +}); + +describe("ollamaThinkParam", () => { + it("maps gpt-oss effort onto low/medium/high", () => { + expect(ollamaThinkParam(model("gpt-oss:20b", "high"))).toEqual({ think: "high" }); + expect(ollamaThinkParam(model("gpt-oss:20b", "low"))).toEqual({ think: "low" }); + expect(ollamaThinkParam(model("gpt-oss:20b", "medium"))).toEqual({ think: "medium" }); + expect(ollamaThinkParam(model("gpt-oss:20b", "extra"))).toEqual({ think: "high" }); + expect(ollamaThinkParam(model("gpt-oss:20b", "ultra"))).toEqual({ think: "high" }); + }); + + it("sends nothing for gpt-oss at effort none, which its policy does not list", () => { + expect(ollamaEffortPolicy(model("gpt-oss:20b")).supported).not.toContain("none"); + expect(ollamaThinkParam(model("gpt-oss:20b", "none"))).toEqual({}); + }); + + it("maps boolean-thinking families onto true/false", () => { + expect(ollamaThinkParam(model("qwen3:8b", "none"))).toEqual({ think: false }); + expect(ollamaThinkParam(model("qwen3:8b", "medium"))).toEqual({ think: true }); + expect(ollamaThinkParam(model("deepseek-r1:7b", "high"))).toEqual({ think: true }); + }); + + it("sends nothing for qwen3 variants that cannot think", () => { + expect(ollamaThinkParam(model("qwen3-coder:30b", "high"))).toEqual({}); + expect(ollamaThinkParam(model("qwen3-embedding:8b", "high"))).toEqual({}); + expect(ollamaThinkParam(model("qwen3:30b-a3b-instruct-2507", "high"))).toEqual({}); + expect(ollamaEffortPolicy(model("qwen3-coder:30b")).supported).toEqual([]); + expect(ollamaEffortPolicy(model("qwen3:30b-a3b-thinking-2507")).supported).toContain("high"); + expect(ollamaThinkParam(model("qwen3:8b", "high"))).toEqual({ think: true }); + }); + + it("reads the model name the way the policy does, trimmed", () => { + expect(ollamaThinkParam(model(" gpt-oss:20b ", "high"))).toEqual({ think: "high" }); + }); + + it("sends nothing for a model that cannot think, or when no effort is set", () => { + expect(ollamaThinkParam(model("llama3.2", "high"))).toEqual({}); + expect(ollamaThinkParam(model("qwen3:8b"))).toEqual({}); + expect(ollamaThinkParam(undefined)).toEqual({}); + }); +}); + +function chunks(items: unknown[]): any { + async function* gen(): AsyncGenerator { + for (const item of items) yield item; + } + return Object.assign(gen(), { abort() {} }); +} + +async function runStream(m: any): Promise<{ events: any[]; request: any }> { + let request: any; + const chat = vi.fn(async (req: any) => { + request = req; + return chunks([ + { message: { thinking: "hm", content: "" } }, + { message: { content: "ok" } }, + { + message: { + content: "", + tool_calls: [{ function: { name: "weather", arguments: { city: "Paris" } } }], + }, + done: true, + }, + ]); + }); + const run = createOllamaToolCallingStream(async () => ({ chat }), toOllamaMessages); + const events: any[] = []; + await run( + { + prompt: "weather?", + tools: [ + { + name: "weather", + description: "d", + inputSchema: { type: "object", properties: { city: { type: "string" } } }, + }, + ], + } as any, + m, + new AbortController().signal, + (e: any) => events.push(e) + ); + return { events, request }; +} + +describe("createOllamaToolCallingStream", () => { + it("streams thinking on the reasoning port, text on text, and sends think for a thinking model", async () => { + const { events, request } = await runStream(model("qwen3:8b", "medium")); + expect(events).toContainEqual({ type: "text-delta", port: "reasoning", textDelta: "hm" }); + expect(events).toContainEqual({ type: "text-delta", port: "text", textDelta: "ok" }); + expect(events).toContainEqual({ + type: "object-delta", + port: "toolCalls", + objectDelta: [{ id: "call_0", name: "weather", input: { city: "Paris" } }], + }); + expect(request.think).toBe(true); + expect(request.messages).toEqual([{ role: "user", content: "weather?" }]); + }); + + it("omits think for a model that cannot think", async () => { + const { request } = await runStream(model("llama3.2", "medium")); + expect("think" in request).toBe(false); + }); +}); + +describe("browser build", () => { + it("passes toOllamaMessages and no longer carries its own prompt-only converter", () => { + const source = readFileSync( + new URL( + "../../../../../providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts", + import.meta.url + ), + "utf8" + ); + expect(source).not.toContain("buildBrowserToolCallingMessages"); + expect(source).toContain("toOllamaMessages"); + }); + + it("node build no longer uses the text-flat converter", () => { + const source = readFileSync( + new URL("../../../../../providers/ollama/src/ai/common/Ollama_JobRunFns.ts", import.meta.url), + "utf8" + ); + expect(source).not.toContain("toTextFlatMessages"); + expect(source).toContain("toOllamaMessages"); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/OpenAICompatToolImages.test.ts b/packages/test/src/test/ai-provider-api/OpenAICompatToolImages.test.ts new file mode 100644 index 0000000000..2588cc6c2f --- /dev/null +++ b/packages/test/src/test/ai-provider-api/OpenAICompatToolImages.test.ts @@ -0,0 +1,151 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { _testOnly as hfiTestOnly } from "@workglow/huggingface-inference/ai"; +import { _testOnly as openRouterTestOnly } from "@workglow/openrouter/ai"; +import { _testOnly as xaiTestOnly } from "@workglow/xai/ai"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +function findRunFn( + fns: readonly { readonly serves: readonly string[]; readonly runFn: any }[], + label: string +) { + const reg = fns.find((r) => [...r.serves].sort().join(",") === "text.generation,tool-use"); + if (!reg) throw new Error(`no ${label} tool-calling run-fn registered`); + return reg.runFn; +} + +const input = { + prompt: "look", + tools: [{ name: "shot", description: "", inputSchema: { type: "object", properties: {} } }], + messages: [ + { role: "user", content: [{ type: "text", text: "look" }] }, + { role: "assistant", content: [{ type: "tool_use", id: "c1", name: "shot", input: {} }] }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "here" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + ], + }, + ], +} as any; + +function expectImageInFollowingUserMessage(messages: any[]): void { + const toolIndex = messages.findIndex((m) => m.role === "tool"); + expect(toolIndex).toBeGreaterThan(-1); + const toolMessage = messages[toolIndex]; + expect(typeof toolMessage.content).toBe("string"); + expect(toolMessage.content).toContain("[image 1] follows in the next message"); + + const next = messages[toolIndex + 1]; + expect(next.role).toBe("user"); + expect(next.content).toContainEqual({ + type: "image_url", + image_url: { url: "data:image/png;base64,AAAA" }, + }); +} + +function sseEmptyResponse(): Response { + const encoder = new TextEncoder(); + const stream = new ReadableStream({ + start(controller) { + controller.enqueue(encoder.encode(`data: [DONE]\n\n`)); + controller.close(); + }, + }); + return new Response(stream, { + status: 200, + headers: { "content-type": "text/event-stream" }, + }); +} + +describe("tool-result images reach the model in a user message", () => { + describe("over the openai SDK (xAI, OpenRouter)", () => { + let fetchSpy: ReturnType; + + beforeEach(() => { + fetchSpy = vi.spyOn(globalThis, "fetch"); + fetchSpy.mockImplementation(async () => sseEmptyResponse()); + }); + + afterEach(() => { + fetchSpy.mockRestore(); + }); + + function sentMessages(): any[] { + const init = fetchSpy.mock.calls[0]![1] as RequestInit; + return JSON.parse(init.body as string).messages; + } + + it("xAI", async () => { + const runFn = findRunFn(xaiTestOnly.XAI_RUN_FNS, "xAI"); + await runFn( + input, + { + model_id: "grok-x", + provider_config: { model_name: "grok-4", api_key: "test-key" }, + } as any, + new AbortController().signal, + () => {} + ); + expectImageInFollowingUserMessage(sentMessages()); + }); + + it("OpenRouter", async () => { + const runFn = findRunFn(openRouterTestOnly.OPENROUTER_RUN_FNS, "OpenRouter"); + await runFn( + input, + { + model_id: "openrouter-x", + provider_config: { model_name: "anthropic/claude-sonnet-4", api_key: "test-key" }, + } as any, + new AbortController().signal, + () => {} + ); + expectImageInFollowingUserMessage(sentMessages()); + }); + }); + + describe("Hugging Face Inference", () => { + let captured: any; + + beforeEach(() => { + captured = undefined; + class FakeInferenceClient { + chatCompletionStream(params: unknown) { + captured = params; + return (async function* () {})(); + } + } + hfiTestOnly._setHfInferenceSDKForTesting({ InferenceClient: FakeInferenceClient } as any); + }); + + afterEach(() => { + hfiTestOnly._setHfInferenceSDKForTesting(undefined); + }); + + it("sends the image in a user message", async () => { + const runFn = findRunFn(hfiTestOnly.HFI_RUN_FNS, "Hugging Face Inference"); + await runFn( + input, + { + model_id: "hfi-x", + provider_config: { model_name: "meta-llama/Llama-3.3-70B-Instruct", api_key: "test-key" }, + } as any, + new AbortController().signal, + () => {} + ); + expectImageInFollowingUserMessage(captured.messages); + }); + }); +}); diff --git a/packages/test/src/test/ai-provider-api/OpenRouter_ReasoningDetails.test.ts b/packages/test/src/test/ai-provider-api/OpenRouter_ReasoningDetails.test.ts new file mode 100644 index 0000000000..99f8612c3e --- /dev/null +++ b/packages/test/src/test/ai-provider-api/OpenRouter_ReasoningDetails.test.ts @@ -0,0 +1,466 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { _testOnly as openRouterTestOnly } from "@workglow/openrouter/ai"; +import { + createReasoningDetailAccumulator, + encodeReasoningTurn, + mergeReasoningDetailDelta, + withReasoningDetails, +} from "../../../../../providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const MODEL = "anthropic/claude-sonnet-5-5"; +const KEY = `openrouter:${MODEL}`; + +describe("reasoning detail accumulator", () => { + it("joins pieces by index and sorts the result", () => { + const acc = createReasoningDetailAccumulator(); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.text", index: 0, text: "Let me " }]); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.text", index: 0, text: "think." }]); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.text", index: 0, signature: "sig" }]); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.encrypted", index: 1, data: "xyz" }]); + expect(acc.items()).toEqual([ + { type: "reasoning.text", index: 0, text: "Let me think.", signature: "sig" }, + { type: "reasoning.encrypted", index: 1, data: "xyz" }, + ]); + }); + + it("is undefined when nothing arrived", () => { + expect(createReasoningDetailAccumulator().items()).toBeUndefined(); + }); + + it("appends a piece with no index after the last one", () => { + const acc = createReasoningDetailAccumulator(); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.summary", index: 0, summary: "a" }]); + mergeReasoningDetailDelta(acc, [{ type: "reasoning.summary", summary: "b" }]); + expect(acc.items()).toEqual([ + { type: "reasoning.summary", index: 0, summary: "a" }, + { type: "reasoning.summary", summary: "b" }, + ]); + }); + + it("ignores a delta that is not an array of objects", () => { + const acc = createReasoningDetailAccumulator(); + mergeReasoningDetailDelta(acc, undefined); + mergeReasoningDetailDelta(acc, "x"); + mergeReasoningDetailDelta(acc, [null, 3]); + expect(acc.items()).toBeUndefined(); + }); +}); + +describe("withReasoningDetails", () => { + const detail = { type: "reasoning.text", text: "x", signature: "sig" }; + const written = (ids: string[]) => ids.map((id) => ({ id, name: "shot", arguments: "{}" })); + const turn = ( + callIds: string[], + payloadOf: unknown = { calls: written(callIds), details: [detail] } + ): any => ({ + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: KEY, payload: JSON.stringify(payloadOf) }, + ...callIds.map((id) => ({ type: "tool_use", id, name: "shot", input: {} })), + ], + }); + const converted = (ids: string[]): any => ({ + role: "assistant", + content: null, + ...(ids.length > 0 + ? { + tool_calls: ids.map((id) => ({ + id, + type: "function", + function: { name: "shot", arguments: "{}" }, + })), + } + : {}), + }); + + it("attaches the details to the turn whose call ids match, without mutating", () => { + const input = [{ role: "user", content: "hi" }, converted(["a", "b"])]; + const out = withReasoningDetails( + input, + [{ role: "user", content: [{ type: "text", text: "hi" }] }, turn(["a", "b"])], + KEY + ); + expect(out[0]).toEqual({ role: "user", content: "hi" }); + expect(out[1]).toMatchObject({ reasoning_details: [detail] }); + expect(input[1]).not.toHaveProperty("reasoning_details"); + }); + + it("matches a turn with no tool calls against an empty id list", () => { + const out = withReasoningDetails([converted([])], [turn([])], KEY); + expect(out[0]).toMatchObject({ reasoning_details: [detail] }); + }); + + it("does not replay when an id changed, was dropped or was added", () => { + const history = [turn(["a", "b"])]; + expect(withReasoningDetails([converted(["a", "z"])], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + expect(withReasoningDetails([converted(["a"])], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + expect(withReasoningDetails([converted(["a", "b", "c"])], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + expect(withReasoningDetails([converted(["b", "a"])], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + }); + + it("does not replay when a kept id now carries another name or arguments", () => { + const history = [turn(["a"])]; + const renamed = converted(["a"]); + renamed.tool_calls[0].function.name = "other"; + expect(withReasoningDetails([renamed], history, KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + const edited = converted(["a"]); + edited.tool_calls[0].function.arguments = '{"x":1}'; + expect(withReasoningDetails([edited], history, KEY)[0]).not.toHaveProperty("reasoning_details"); + }); + + it("does not replay a payload that is not a turn, or another model's", () => { + expect(withReasoningDetails([converted([])], [turn([], [detail])], KEY)[0]).not.toHaveProperty( + "reasoning_details" + ); + expect( + withReasoningDetails([converted([])], [turn([], "not json{")], KEY)[0] + ).not.toHaveProperty("reasoning_details"); + expect( + withReasoningDetails([converted([])], [turn([])], "openrouter:other/model")[0] + ).not.toHaveProperty("reasoning_details"); + }); + + it("does not replay details that are not typed objects", () => { + for (const details of [[null], [7], [{ text: "untyped" }], [detail, null]]) { + expect( + withReasoningDetails([converted([])], [turn([], { calls: [], details })], KEY)[0] + ).not.toHaveProperty("reasoning_details"); + } + }); +}); + +describe("encodeReasoningTurn", () => { + it("carries the calls beside the details, arguments as the converter writes them", () => { + expect( + JSON.parse(encodeReasoningTurn([{ id: "c1", name: "shot", input: { a: 1 } }], details)!) + ).toEqual({ calls: [{ id: "c1", name: "shot", arguments: '{"a":1}' }], details }); + }); + + it("refuses an anthropic thinking block cut off before its signature", () => { + const cut = [{ type: "reasoning.text", format: "anthropic-claude-v1", text: "hm" }]; + expect(encodeReasoningTurn([], cut)).toBeUndefined(); + expect(encodeReasoningTurn([], [{ ...cut[0], signature: "" }])).toBeUndefined(); + expect(encodeReasoningTurn([], [{ ...cut[0], signature: "sig" }])).toBeDefined(); + }); + + it("keeps a signature-less text detail from another format", () => { + const plain = [{ type: "reasoning.text", format: "unknown", text: "hm" }]; + expect(encodeReasoningTurn([], plain)).toBeDefined(); + }); +}); + +function sseResponse(chunks: readonly unknown[]): Response { + const encoder = new TextEncoder(); + const body = new ReadableStream({ + start(controller) { + for (const chunk of chunks) { + controller.enqueue(encoder.encode(`data: ${JSON.stringify(chunk)}\n\n`)); + } + controller.enqueue(encoder.encode(`data: [DONE]\n\n`)); + controller.close(); + }, + }); + return new Response(body, { status: 200, headers: { "content-type": "text/event-stream" } }); +} + +function chunk(delta: Record, finish: string | null = null): unknown { + return { + id: "c", + object: "chat.completion.chunk", + choices: [{ index: 0, delta, finish_reason: finish }], + }; +} + +const details = [ + { type: "reasoning.text", index: 0, text: "Let me think.", signature: "sig" }, + { type: "reasoning.encrypted", index: 1, data: "xyz" }, +]; + +const tools = [{ name: "shot", description: "", inputSchema: { type: "object", properties: {} } }]; + +const modelConfig = { + model_id: "openrouter-x", + provider_config: { model_name: MODEL, api_key: "test-key" }, +} as any; + +function findRunFn() { + const reg = openRouterTestOnly.OPENROUTER_RUN_FNS.find( + (r: any) => + [...r.serves].sort((a: string, b: string) => a.localeCompare(b)).join(",") === + "text.generation,tool-use" + ); + if (!reg) throw new Error("no OpenRouter tool-calling run-fn registered"); + return reg.runFn as any; +} + +describe("OpenRouter tool calling replays reasoning_details", () => { + let fetchSpy: ReturnType; + + beforeEach(() => { + fetchSpy = vi.spyOn(globalThis, "fetch"); + }); + + afterEach(() => { + fetchSpy.mockRestore(); + }); + + function sentMessages(): any[] { + const init = fetchSpy.mock.calls[0]![1] as RequestInit; + return JSON.parse(init.body as string).messages; + } + + function replayInput(provider: string, callId = "c1", payloadCallId = callId): any { + return { + prompt: "look", + tools, + messages: [ + { role: "user", content: [{ type: "text", text: "look" }] }, + { + role: "assistant", + content: [ + { + type: "reasoning", + text: "", + provider, + payload: JSON.stringify({ + calls: [{ id: payloadCallId, name: "shot", arguments: "{}" }], + details, + }), + }, + { type: "tool_use", id: callId, name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: callId, content: [{ type: "text", text: "ok" }] }, + ], + }, + ], + }; + } + + it("forwards thinking carried only in reasoning_details to the reasoning port, once", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + // A null entry is malformed and skipped, not a reason to fail the run. + chunk({ + reasoning_details: [null, { type: "reasoning.text", index: 0, text: "only here" }], + }), + // An empty legacy field must not hide the details' text. + chunk({ + reasoning_content: "", + reasoning_details: [{ type: "reasoning.text", index: 0, text: ", and here" }], + }), + chunk({ + reasoning: "legacy", + reasoning_details: [{ type: "reasoning.text", index: 0, text: "legacy" }], + }), + // Nor an empty alias hide the other one's text, with or without details. + chunk({ + reasoning: " too", + reasoning_content: "", + reasoning_details: [{ type: "reasoning.text", index: 0, text: " too" }], + }), + chunk({ reasoning: "!", reasoning_content: "" }), + chunk({ content: "done" }), + chunk({}, "stop"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + const reasoning = events + .filter((e) => e.type === "text-delta" && e.port === "reasoning") + .map((e) => e.textDelta) + .join(""); + expect(reasoning).toBe("only here, and herelegacy too!"); + }); + + it("emits the streamed details as the native turn before finish", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + chunk({ reasoning_details: [{ type: "reasoning.text", index: 0, text: "Let me " }] }), + chunk({ + reasoning_details: [ + { type: "reasoning.text", index: 0, text: "think.", signature: "sig" }, + { type: "reasoning.encrypted", index: 1, data: "xyz" }, + ], + }), + chunk({ + tool_calls: [ + { index: 0, id: "c1", type: "function", function: { name: "shot", arguments: "{}" } }, + ], + }), + chunk({}, "tool_calls"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + const nativeIndex = events.findIndex( + (e) => e.type === "object-delta" && e.port === "nativeTurn" + ); + expect(nativeIndex).toBeGreaterThan(-1); + expect(events[nativeIndex]).toEqual({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { + provider: KEY, + payload: JSON.stringify({ calls: [{ id: "c1", name: "shot", arguments: "{}" }], details }), + }, + }); + expect(nativeIndex).toBeLessThan(events.findIndex((e) => e.type === "finish")); + }); + + it("emits an empty call list for a round that made no tool calls", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + chunk({ reasoning_details: [{ type: "reasoning.encrypted", index: 0, data: "xyz" }] }), + chunk({ content: "done" }, "stop"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + const native = events.find((e) => e.port === "nativeTurn"); + expect(JSON.parse(native.objectDelta.payload)).toEqual({ + calls: [], + details: [{ type: "reasoning.encrypted", index: 0, data: "xyz" }], + }); + }); + + it("emits no native turn when a call it made was filtered out", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + chunk({ reasoning_details: [{ type: "reasoning.encrypted", index: 0, data: "xyz" }] }), + chunk({ + tool_calls: [ + { index: 0, id: "bad", type: "function", function: { name: "nope", arguments: "{}" } }, + { index: 1, id: "c2", type: "function", function: { name: "shot", arguments: "{}" } }, + ], + }), + chunk({}, "tool_calls"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + expect( + events.find((e) => e.type === "object-delta" && e.port === "toolCalls")?.objectDelta + ).toEqual([expect.objectContaining({ id: "c2", name: "shot" })]); + expect(events.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("emits no native turn when thinking was cut off before its signature", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([ + chunk({ + reasoning_details: [ + { type: "reasoning.text", index: 0, format: "anthropic-claude-v1", text: "hm" }, + ], + }), + chunk({ content: "partial" }, "length"), + ]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + expect(events.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("emits no native turn when the stream carried no details", async () => { + fetchSpy.mockImplementation(async () => + sseResponse([chunk({ content: "hi" }), chunk({}, "stop")]) + ); + const events: any[] = []; + await findRunFn()( + { prompt: "look", tools }, + modelConfig, + new AbortController().signal, + (e: any) => events.push(e) + ); + expect(events.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("sends the turn's own details back as reasoning_details", async () => { + fetchSpy.mockImplementation(async () => sseResponse([])); + await findRunFn()(replayInput(KEY), modelConfig, new AbortController().signal, () => {}); + const assistant = sentMessages().find((m) => m.role === "assistant"); + expect(assistant.reasoning_details).toEqual(details); + expect(assistant).not.toHaveProperty("native_items"); + expect(assistant.tool_calls).toHaveLength(1); + }); + + it("does not replay when the host changed the call id", async () => { + fetchSpy.mockImplementation(async () => sseResponse([])); + await findRunFn()( + replayInput(KEY, "c1-renamed", "c1"), + modelConfig, + new AbortController().signal, + () => {} + ); + const assistant = sentMessages().find((m) => m.role === "assistant"); + expect(assistant).not.toHaveProperty("reasoning_details"); + expect(assistant.tool_calls).toHaveLength(1); + }); + + it("does not replay when the host dropped the call", async () => { + fetchSpy.mockImplementation(async () => sseResponse([])); + const input = replayInput(KEY); + input.messages[1].content = input.messages[1].content.filter((b: any) => b.type !== "tool_use"); + input.messages.pop(); + await findRunFn()(input, modelConfig, new AbortController().signal, () => {}); + const assistant = sentMessages().find((m) => m.role === "assistant"); + expect(assistant).not.toHaveProperty("reasoning_details"); + }); + + it("does not replay details another model produced", async () => { + fetchSpy.mockImplementation(async () => sseResponse([])); + await findRunFn()( + replayInput("openrouter:openai/gpt-6-luna"), + modelConfig, + new AbortController().signal, + () => {} + ); + const assistant = sentMessages().find((m) => m.role === "assistant"); + expect(assistant).not.toHaveProperty("reasoning_details"); + expect(assistant).not.toHaveProperty("native_items"); + }); +}); diff --git a/packages/test/src/test/ai-provider-cactus/Cactus_ToolCalling.test.ts b/packages/test/src/test/ai-provider-cactus/Cactus_ToolCalling.test.ts index 5b18207816..a8f616527f 100644 --- a/packages/test/src/test/ai-provider-cactus/Cactus_ToolCalling.test.ts +++ b/packages/test/src/test/ai-provider-cactus/Cactus_ToolCalling.test.ts @@ -273,9 +273,9 @@ describe("Cactus_ToolCalling v3 reasoning on the text port", () => { expect(text).toContain("Paris is sunny."); }); - it("releases an unterminated rather than swallowing the generation", async () => { - // Cut off at the token limit: there is no closing tag, and the parser - // deliberately leaves such a block alone, so the stream must too. + it("keeps an unterminated off the text port", async () => { + // Cut off at the token limit: there is no closing tag, and what follows the + // opening one is still reasoning, so it goes to the reasoning port. const { text } = await runToolCalling({ run: () => "unused", run_stream: async (_q, _t, cb) => { @@ -283,7 +283,7 @@ describe("Cactus_ToolCalling v3 reasoning on the text port", () => { return "reasoning that never closes"; }, }); - expect(text).toBe("reasoning that never closes"); + expect(text).toBe(""); }); it("does not hold back text that merely starts like a tag", async () => { @@ -580,3 +580,144 @@ describe("Cactus_ToolCalling tool-call markup on the text port", () => { expect(text).toBe("ends mid-tag { + const toolUse = { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "lookup_weather", input: { city: "Paris" } }], + }; + const toolResult = { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "text", text: "sunny" }], + }, + ], + }; + const user = (text: string) => ({ role: "user", content: [{ type: "text", text }] }); + + async function queryFor(extra: Record): Promise { + let seen = ""; + cactusEngines.set("needle-26m", { + run: (q: string) => { + seen = q; + return "[]"; + }, + } as never); + await Cactus_ToolCalling( + { prompt: "first", tools, toolChoice: "auto", ...extra } as never, + model as never, + new AbortController().signal, + () => {} + ); + return seen; + } + + it("sends the most recent user message after the prompt, not the prompt twice", async () => { + const query = await queryFor({ + messages: [user("first"), toolUse, toolResult, user("second")], + }); + expect(query).toBe("first\n\nsecond"); + }); + + it("sends the prompt alone while it is the latest user message", async () => { + expect(await queryFor({ messages: [user("first"), toolUse, toolResult] })).toBe("first"); + }); + + it("leaves an earlier request in the chat history out of the query", async () => { + const query = await queryFor({ + prompt: "and in Rome?", + messages: [ + user("weather in Paris?"), + { role: "assistant", content: [{ type: "text", text: "sunny" }] }, + user("and in Rome?"), + ], + }); + expect(query).toBe("and in Rome?"); + }); + + it("keeps the question a review prompt asks the model to check against", async () => { + const query = await queryFor({ + prompt: "weather in Paris?", + messages: [ + user("weather in Paris?"), + toolUse, + toolResult, + { role: "assistant", content: [{ type: "text", text: "sunny" }] }, + user("Check your answer against what you were asked."), + ], + }); + expect(query).toBe("weather in Paris?\n\nCheck your answer against what you were asked."); + }); + + it("uses the prompt when there are no messages", async () => { + expect(await queryFor({})).toBe("first"); + expect(await queryFor({ messages: [] })).toBe("first"); + }); + + it("prepends the system prompt and a blank line", async () => { + const query = await queryFor({ + systemPrompt: "rules", + messages: [user("first"), toolUse, toolResult, user("second")], + }); + expect(query).toBe("rules\n\nfirst\n\nsecond"); + expect(await queryFor({ systemPrompt: "rules" })).toBe("rules\n\nfirst"); + }); +}); + +describe("Cactus_ToolCalling emits text on the reasoning port", () => { + const reasoningOf = (events: StreamEvent[]): string => + events + .filter((e) => e.type === "text-delta" && e.port === "reasoning") + .map((e) => (e as { textDelta: string }).textDelta) + .join(""); + + it("sends the think contents of a one-shot generation to reasoning", async () => { + const { names, events, text } = await runToolCalling({ + run: () => + `pick weather` + + `[{"name":"lookup_weather","arguments":{"city":"Paris"}}]`, + }); + expect(names).toEqual(["lookup_weather"]); + expect(events).toContainEqual({ + type: "text-delta", + port: "reasoning", + textDelta: "pick weather", + }); + expect(text).toBe(""); + }); + + it("streams think contents as they arrive, whatever the tag boundaries", async () => { + const { events, text } = await runToolCalling({ + run: () => "unused", + run_stream: async (_q, _t, cb) => { + for (const piece of ["pick ", "wea", "therDone."]) cb(piece); + return "pick weatherDone."; + }, + }); + expect(reasoningOf(events)).toBe("pick weather"); + expect(text).toBe("Done."); + }); + + it("sends an unterminated think block to reasoning, not text", async () => { + const { events, text } = await runToolCalling({ + run: () => "unused", + run_stream: async (_q, _t, cb) => { + cb("reasoning that never "); + cb("closesreasoning that never closes { + const { events } = await runToolCalling({ + run: () => `[{"name":"lookup_weather","arguments":{"city":"Paris"}}]`, + }); + expect(events.some((e) => e.type === "text-delta" && e.port === "reasoning")).toBe(false); + }); +}); diff --git a/packages/test/src/test/ai-provider-hft/HFT_ToolCalling.test.ts b/packages/test/src/test/ai-provider-hft/HFT_ToolCalling.test.ts index 84a29bc6f8..3d85182597 100644 --- a/packages/test/src/test/ai-provider-hft/HFT_ToolCalling.test.ts +++ b/packages/test/src/test/ai-provider-hft/HFT_ToolCalling.test.ts @@ -15,7 +15,11 @@ import { loadTransformersSDK, pipelines, } from "../../../../../providers/huggingface-transformers/src/ai/common/HFT_Pipeline"; -import { HFT_ToolCalling } from "../../../../../providers/huggingface-transformers/src/ai/common/HFT_ToolCalling"; +import { + buildHFTMessages, + createHftDeltaRouter, + HFT_ToolCalling, +} from "../../../../../providers/huggingface-transformers/src/ai/common/HFT_ToolCalling"; // "qwen" in the path selects the Hermes-style `` parser family. const model = { @@ -31,22 +35,27 @@ const PREAMBLE = "Let me check the weather."; * The open tag is split across pieces so the markup filter has to hold a * partial tag back rather than leak it onto the text port. */ -const PIECES: readonly string[] = [ +const TOOL_PIECES: readonly string[] = [ "Let me check ", "the weather.{"name": "get_weather", ', '"arguments": {"city": "Paris"}}', ]; -function makeFakePipeline(): any { +const RENDERED_PROMPT = "What is the weather?\n"; + +function makeFakePipeline( + pieces: readonly string[] = TOOL_PIECES, + rendered: string = RENDERED_PROMPT +): any { const tokenizer = Object.assign((text: string) => ({ input_ids: { dims: [1, text.length] } }), { all_special_ids: [] as number[], - apply_chat_template: () => "What is the weather?\n", + apply_chat_template: () => rendered, decode: () => "", }); const hfModel = { generate: async (args: Record) => { - for (const piece of PIECES) args.streamer?.callback_function?.(piece); + for (const piece of pieces) args.streamer?.callback_function?.(piece); return { dims: [1, args.input_ids.dims[1]] }; }, }; @@ -133,3 +142,207 @@ describe("HFT_ToolCalling stream output", () => { ]); }); }); + +function portText(events: StreamEvent[], port: string): string { + return events + .filter((e) => e.type === "text-delta" && e.port === port) + .map((e) => (e as { textDelta: string }).textDelta) + .join(""); +} + +describe("HFT_ToolCalling thinking", () => { + it("routes a block to the reasoning port and keeps it out of the answer", async () => { + pipelines.set( + cacheKey, + makeFakePipeline(["I should check", " the map.It is sunny."]) + ); + const { events, output } = await run(undefined); + + expect(portText(events, "reasoning")).toBe("I should check the map."); + expect(portText(events, "text")).toBe("It is sunny."); + expect(output.text).toBe("It is sunny."); + expect(output.toolCalls).toEqual([]); + expectNoFinishAccumulation(events); + }); + + it("treats generation as already inside when the prompt opens the tag", async () => { + pipelines.set( + cacheKey, + makeFakePipeline(["planning\n\nDone."], "\n\n") + ); + const { events } = await run(undefined); + + expect(portText(events, "reasoning")).toBe("planning"); + expect(portText(events, "text")).toBe("\n\nDone."); + }); + + it("parses the tool call after the thinking, and ignores one quoted inside it", async () => { + pipelines.set( + cacheKey, + makeFakePipeline([ + 'Maybe {"name": "get_weather", "arguments": {"city": "Rome"}}?', + "", + '{"name": "get_weather", "arguments": {"city": "Paris"}}', + ]) + ); + const { events, output } = await run(undefined); + + expect(portText(events, "text")).toBe(""); + expect(output.toolCalls).toEqual([ + { id: expect.any(String), name: "get_weather", input: { city: "Paris" } }, + ]); + }); +}); + +describe("HFT_ToolCalling chat history", () => { + it("hands earlier reasoning to the template when no tool has run yet", async () => { + const pipeline = makeFakePipeline(["It is still sunny."]); + const rendered: unknown[] = []; + pipeline.tokenizer.apply_chat_template = (messages: unknown) => { + rendered.push(messages); + return RENDERED_PROMPT; + }; + pipelines.set(cacheKey, pipeline); + + await HFT_ToolCalling( + { + ...input, + messages: [ + { role: "user", content: [{ type: "text", text: "Weather?" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "No tool needed." }, + { type: "text", text: "Sunny." }, + ], + }, + { role: "user", content: [{ type: "text", text: "And now?" }] }, + ], + } as unknown as ToolCallingTaskInput, + model, + new AbortController().signal, + () => {} + ); + + expect(rendered[0]).toEqual([ + { role: "user", content: "Weather?" }, + { role: "assistant", content: "Sunny.", reasoning_content: "No tool needed." }, + { role: "user", content: "And now?" }, + ]); + }); +}); + +describe("createHftDeltaRouter", () => { + it("sends content to reasoning and the answer to text", () => { + const seen: Array<{ port: string; textDelta: string }> = []; + const router = createHftDeltaRouter((e) => seen.push(e), false); + router.feed("ab"); + router.flush(); + expect(seen).toEqual([ + { type: "text-delta", port: "reasoning", textDelta: "a" }, + { type: "text-delta", port: "text", textDelta: "b" }, + ]); + expect(router.answerText()).toBe("b"); + }); + + it("starts on reasoning when told the prompt opened the tag", () => { + const seen: Array<{ port: string; textDelta: string }> = []; + const router = createHftDeltaRouter((e) => seen.push(e), true); + router.feed("ab"); + router.flush(); + expect(seen.map((e) => [e.port, e.textDelta])).toEqual([ + ["reasoning", "a"], + ["text", "b"], + ]); + }); +}); + +describe("buildHFTMessages", () => { + const history = [ + { role: "user", content: [{ type: "text", text: "Weather in Paris?" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "Need the weather tool." }, + { type: "text", text: "Checking." }, + { type: "tool_use", id: "call_1", name: "get_weather", input: { city: "Paris" } }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "call_1", + content: [{ type: "text", text: "sunny" }], + }, + ], + }, + ] as never; + + it("replays reasoning as reasoning_content on the assistant turn", () => { + const out = buildHFTMessages(history, undefined, undefined, undefined); + expect(out[1]).toEqual({ + role: "assistant", + content: "Checking.", + reasoning_content: "Need the weather tool.", + tool_calls: [{ id: "call_1", name: "get_weather", arguments: { city: "Paris" } }], + }); + }); + + it("omits reasoning_content when the turn has no reasoning", () => { + const out = buildHFTMessages( + [{ role: "assistant", content: [{ type: "text", text: "Hi" }] }] as never, + undefined, + undefined, + undefined + ); + expect(out[0]).toEqual({ role: "assistant", content: "Hi" }); + }); + + it("names a tool message after the tool_use it answers", () => { + const out = buildHFTMessages(history, undefined, undefined, undefined); + expect(out[2]).toEqual({ + role: "tool", + content: "sunny", + tool_call_id: "call_1", + name: "get_weather", + }); + }); + + it("resolves a repeated id to the most recent tool_use", () => { + const out = buildHFTMessages( + [ + { role: "assistant", content: [{ type: "tool_use", id: "x", name: "first", input: {} }] }, + { role: "assistant", content: [{ type: "tool_use", id: "x", name: "second", input: {} }] }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "x", content: [{ type: "text", text: "r" }] }, + ], + }, + ] as never, + undefined, + undefined, + undefined + ); + expect(out[2]).toMatchObject({ role: "tool", name: "second" }); + }); + + it("leaves the name off a result with no matching tool_use", () => { + const out = buildHFTMessages( + [ + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "z", content: [{ type: "text", text: "r" }] }, + ], + }, + ] as never, + undefined, + undefined, + undefined + ); + expect(out[0]).toEqual({ role: "tool", content: "r", tool_call_id: "z" }); + }); +}); diff --git a/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCalling.test.ts b/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCalling.test.ts new file mode 100644 index 0000000000..ae651ea818 --- /dev/null +++ b/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCalling.test.ts @@ -0,0 +1,233 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { describe, expect, it } from "vitest"; +import { + llamaCppEffortPolicy, + llamaCppThoughtBudget, + llamaCppThoughtBudgetOption, +} from "../../../../../providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy"; +import { + convertMessagesToChatHistory, + messagesToPureChatHistoryForPrefix, + routeLlamaResponseChunk, +} from "../../../../../providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling"; + +const IMAGE_MARKER = "[image/png image returned; this model reads text only]"; + +describe("convertMessagesToChatHistory", () => { + it("replays a reasoning block as an ended thought segment ahead of the calls", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "let me call it" }, + { type: "tool_use", id: "c1", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0]).toEqual({ + type: "segment", + segmentType: "thought", + text: "let me call it", + ended: true, + }); + expect(history[1].response[1].result).toBe(IMAGE_MARKER); + expect(history[1].response[1].result).not.toContain("AAAA"); + }); + + it("skips a reasoning block with no text", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "" }, + { type: "text", text: "hello" }, + ], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response).toEqual(["hello"]); + }); + + it("keeps the text of a mixed result and puts the image marker on its own line", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "shot", input: {} }], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "captured" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + ], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0].result).toBe(`captured\n${IMAGE_MARKER}`); + }); + + it("keeps each block of a result in the place it held", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { role: "assistant", content: [{ type: "tool_use", id: "c1", name: "shot", input: {} }] }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "before" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "after" }, + ], + }, + ], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0].result).toBe(`before\n${IMAGE_MARKER}\nafter`); + }); + + it("leaves an empty result empty instead of serializing the blocks", () => { + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "noop", input: {} }], + }, + { + role: "tool", + content: [{ type: "tool_result", tool_use_id: "c1", content: [] }], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0].result).toBe(""); + }); + + it("keeps a non-text block a tool returned, as JSON", () => { + const nested = { type: "tool_use", id: "x", name: "inner", input: { a: 1 } }; + const history = convertMessagesToChatHistory( + [ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "noop", input: {} }], + }, + { + role: "tool", + content: [{ type: "tool_result", tool_use_id: "c1", content: [nested] }], + }, + ] as any, + undefined, + undefined + ); + expect(history[1].response[0].result).toBe(JSON.stringify(nested)); + }); + + it("renders the same thought segment for a checkpoint prefix", () => { + const history = messagesToPureChatHistoryForPrefix( + [{ role: "assistant", content: [{ type: "reasoning", text: "thinking" }] }] as any, + undefined + ); + expect(history[0].response[0]).toMatchObject({ type: "segment", segmentType: "thought" }); + }); +}); + +describe("routeLlamaResponseChunk", () => { + it("routes a thought segment to the reasoning port", () => { + expect( + routeLlamaResponseChunk({ type: "segment", segmentType: "thought", text: "hm" }) + ).toEqual({ port: "reasoning", text: "hm" }); + }); + + it("drops any other segment type", () => { + expect( + routeLlamaResponseChunk({ type: "segment", segmentType: "comment", text: "note" }) + ).toBeUndefined(); + }); + + it("routes a main-response chunk to the text port, untyped or typed text", () => { + for (const type of [undefined, "text"]) { + expect(routeLlamaResponseChunk({ type, text: "answer" })).toEqual({ + port: "text", + text: "answer", + }); + } + }); +}); + +describe("llamaCppThoughtBudget", () => { + const budget = (effort?: string): number | undefined => + llamaCppThoughtBudget(effort === undefined ? {} : ({ effort } as object)); + + it("is undefined when no effort is set", () => { + expect(budget()).toBeUndefined(); + }); + + it.each([ + ["none", 0], + ["low", 512], + ["medium", 1024], + ["high", 2048], + ["extra", 4096], + ["ultra", 8192], + ])("maps %s to %i thought tokens", (effort, tokens) => { + expect(budget(effort)).toBe(tokens); + }); + + it("is the prompt option text generation and chat send, empty when no effort is set", () => { + expect(llamaCppThoughtBudgetOption({ effort: "high" } as object)).toEqual({ + budgets: { thoughtTokens: 2048 }, + }); + expect(llamaCppThoughtBudgetOption({})).toEqual({}); + }); +}); + +describe("llamaCppEffortPolicy", () => { + it("offers every level on any local model", () => { + expect( + llamaCppEffortPolicy({ provider_config: { model_path: "models/x.gguf" } }).supported + ).toEqual(["none", "low", "medium", "high", "extra", "ultra"]); + }); +}); diff --git a/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCallingCheckpoint.test.ts b/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCallingCheckpoint.test.ts index a0c8569d74..a39f8af438 100644 --- a/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCallingCheckpoint.test.ts +++ b/packages/test/src/test/ai-provider-nodellama/LlamaCpp_ToolCallingCheckpoint.test.ts @@ -65,7 +65,11 @@ vi.mock("node-llama-cpp", () => ({ generateResponse( history: any[], options: { - readonly onTextChunk: (chunk: string) => void; + readonly onResponseChunk: (chunk: { + type: undefined; + segmentType: undefined; + text: string; + }) => void; readonly signal: AbortSignal; } ): Promise<{ @@ -84,7 +88,7 @@ vi.mock("node-llama-cpp", () => ({ return Promise.reject(options.signal.reason); } advanceFakeSequence(this.sequence, TURN_TOKENS); - options.onTextChunk("calling"); + options.onResponseChunk({ type: undefined, segmentType: undefined, text: "calling" }); const cleanHistory = [ ...history, { diff --git a/packages/test/src/test/ai-provider/AnthropicCheckpointParams.test.ts b/packages/test/src/test/ai-provider/AnthropicCheckpointParams.test.ts index a143081088..c45993049d 100644 --- a/packages/test/src/test/ai-provider/AnthropicCheckpointParams.test.ts +++ b/packages/test/src/test/ai-provider/AnthropicCheckpointParams.test.ts @@ -102,6 +102,44 @@ describe("applyAnthropicPrefixReplay", () => { expect((params.system as any[])[0].text).toBe("sys"); }); + it("replays a prefix turn's signed thinking for the model that wrote it", () => { + const thinking = { type: "thinking", thinking: "plan", signature: "SIG" }; + const turn = (provider: string) => ({ + ...prefix, + messages: [ + prefix.messages[0]!, + { + role: "assistant" as const, + content: [ + { + type: "reasoning" as const, + text: "plan", + provider, + payload: JSON.stringify([thinking, { type: "text", text: "hi" }]), + }, + { type: "text" as const, text: "hi" }, + ], + }, + ], + }); + const replay = (provider: string): any[] => { + const params: Record = { + model: "claude-x", + messages: [{ role: "user", content: [{ type: "text", text: "tail" }] }], + }; + applyAnthropicPrefixReplay(params, { sessionId: "ckpt", prefix: turn(provider) }); + return (params.messages as any[])[1].content; + }; + expect(replay("anthropic:claude-x")[0]).toEqual(thinking); + expect(replay("anthropic:other")[0]).toEqual({ + type: "text", + text: "hi", + cache_control: { type: "ephemeral" }, + }); + const warm = buildAnthropicCheckpointParams(turn("anthropic:claude-x") as any, "claude-x"); + expect((warm.messages as any[])[1].content[0]).toEqual(thinking); + }); + it("annotates the final turn when emitting a chained checkpoint", () => { const session: AiSessionContext = { sessionId: "ckpt", emitCheckpointId: "next", prefix }; const params: Record = { diff --git a/packages/test/src/test/ai-provider/Anthropic_NativeTurn.test.ts b/packages/test/src/test/ai-provider/Anthropic_NativeTurn.test.ts new file mode 100644 index 0000000000..26a539095b --- /dev/null +++ b/packages/test/src/test/ai-provider/Anthropic_NativeTurn.test.ts @@ -0,0 +1,508 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn } from "@workglow/ai"; +import { ANTHROPIC, _testOnly } from "@workglow/anthropic/ai"; +import { _testOnly as runtimeTestOnly } from "@workglow/anthropic/ai-runtime"; +import { getLogger } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const { ANTHROPIC_RUN_FNS, setAnthropicClientForTests } = _testOnly; + +const MODEL_NAME = "claude-sonnet-5-5"; +const NATIVE_KEY = `anthropic:${MODEL_NAME}`; + +function toolCallingRunFn(): AiProviderRunFn { + const registration = ANTHROPIC_RUN_FNS.find(({ serves }) => + (serves as readonly string[]).includes("tool-use") + ); + expect(registration).toBeDefined(); + return registration!.runFn as AiProviderRunFn; +} + +const model = { + model_id: MODEL_NAME, + title: "", + description: "", + provider: ANTHROPIC, + provider_config: { model_name: MODEL_NAME, api_key: "test-key" }, + capabilities: ["text.generation", "tool-use"], + metadata: {}, +} as never; + +type Scripted = readonly unknown[] | Error; + +const thinkingThenCall = [ + { type: "content_block_start", index: 0, content_block: { type: "thinking", thinking: "" } }, + { + type: "content_block_delta", + index: 0, + delta: { type: "thinking_delta", thinking: "Plan" }, + }, + { + type: "content_block_delta", + index: 0, + delta: { type: "signature_delta", signature: "SIG" }, + }, + { type: "content_block_stop", index: 0 }, + { + type: "content_block_start", + index: 1, + content_block: { type: "tool_use", id: "t1", name: "look", input: {} }, + }, + { + type: "content_block_delta", + index: 1, + delta: { type: "input_json_delta", partial_json: '{"a":' }, + }, + { + type: "content_block_delta", + index: 1, + delta: { type: "input_json_delta", partial_json: "1}" }, + }, + { type: "content_block_stop", index: 1 }, +]; + +const signedBlocks = [ + { type: "thinking", thinking: "Plan", signature: "SIG" }, + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, +]; + +const turnMessages = (provider: string, toolUseId: string) => [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "", provider, payload: JSON.stringify(signedBlocks) }, + { type: "tool_use", id: toolUseId, name: "look", input: { a: 1 } }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: toolUseId, content: [{ type: "text", text: "ok" }] }, + ], + }, +]; + +const bindingError = () => + Object.assign( + new Error( + "messages.1.content.0: Invalid `signature` in `thinking` block. The block is bound to a different conversation." + ), + { status: 400 } + ); + +describe("Anthropic native turn", () => { + let calls: Array>; + let script: Scripted[]; + + const install = () => { + const fakeClient = { + messages: { + stream: (params: Record) => { + calls.push(structuredClone(params)); + const next = script.shift() ?? []; + if (next instanceof Error) throw next; + return { + async *[Symbol.asyncIterator]() { + for (const event of next) { + if (event instanceof Error) throw event; + yield event; + } + }, + }; + }, + }, + }; + setAnthropicClientForTests(fakeClient); + runtimeTestOnly.setAnthropicClientForTests?.(fakeClient); + }; + + beforeEach(() => { + calls = []; + script = []; + vi.spyOn(getLogger(), "warn").mockImplementation(() => {}); + install(); + }); + + afterEach(() => { + setAnthropicClientForTests(undefined); + runtimeTestOnly.setAnthropicClientForTests?.(undefined); + vi.restoreAllMocks(); + }); + + async function run(messages: unknown[]): Promise>> { + const events: Array> = []; + await toolCallingRunFn()( + { model, prompt: "", messages, tools: [] } as never, + model, + undefined as never, + ((event: Record) => events.push(event)) as never + ); + return events; + } + + const nativeTurns = (events: Array>) => + events.filter((e) => e.type === "object-delta" && e.port === "nativeTurn"); + + const sentAssistant = (call: Record) => + (call.messages as Array<{ role: string; content: any[] }>).find((m) => m.role === "assistant")!; + + it("captures the reply's blocks in order, signature included", async () => { + script = [thinkingThenCall]; + const events = await run([{ role: "user", content: [{ type: "text", text: "go" }] }]); + const native = nativeTurns(events); + expect(native).toHaveLength(1); + expect(native[0]!.objectDelta).toEqual({ + provider: NATIVE_KEY, + payload: JSON.stringify(signedBlocks), + }); + expect(events.indexOf(native[0]!)).toBeLessThan(events.findIndex((e) => e.type === "finish")); + }); + + it("streams the thinking on the reasoning port as well as keeping it for replay", async () => { + script = [thinkingThenCall]; + const events = await run([{ role: "user", content: [{ type: "text", text: "go" }] }]); + expect(events).toContainEqual({ type: "text-delta", port: "reasoning", textDelta: "Plan" }); + }); + + it("rebuilds rather than replays a stored tool_use block with no input", async () => { + script = [[]]; + const blocks = [{ type: "tool_use", id: "t1", name: "look" }]; + await run([ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: NATIVE_KEY, payload: JSON.stringify(blocks) }, + { type: "tool_use", id: "t1", name: "look", input: {} }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "t1", content: [{ type: "text", text: "ok" }] }, + ], + }, + ]); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: {} }, + ]); + }); + + it("rebuilds rather than replays a stored turn holding an empty text block", async () => { + script = [[]]; + const blocks = [{ type: "text", text: "" }, ...signedBlocks]; + await run([ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "", provider: NATIVE_KEY, payload: JSON.stringify(blocks) }, + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "t1", content: [{ type: "text", text: "ok" }] }, + ], + }, + ]); + expect(sentAssistant(calls[0]!).content).not.toContainEqual({ type: "text", text: "" }); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + }); + + it.each([ + ["a null entry", [null]], + ["a block of a type it never writes", [{ type: "image", source: {} }]], + ["thinking without its signature", [{ type: "thinking", thinking: "hm" }]], + ["redacted thinking without its data", [{ type: "redacted_thinking" }]], + ["text that is not a string", [{ type: "text", text: 7 }]], + ])("rebuilds rather than replays a stored turn holding %s", async (_label, extra) => { + script = [[]]; + await run([ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { + type: "reasoning", + text: "", + provider: NATIVE_KEY, + payload: JSON.stringify([...(extra as unknown[]), ...signedBlocks]), + }, + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "t1", content: [{ type: "text", text: "ok" }] }, + ], + }, + ]); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + }); + + it("captures redacted thinking as it arrived", async () => { + script = [ + [ + { + type: "content_block_start", + index: 0, + content_block: { type: "redacted_thinking", data: "R" }, + }, + { type: "content_block_stop", index: 0 }, + ], + ]; + const events = await run([{ role: "user", content: [{ type: "text", text: "go" }] }]); + expect(nativeTurns(events)[0]!.objectDelta.payload).toBe( + JSON.stringify([{ type: "redacted_thinking", data: "R" }]) + ); + }); + + const textBlock = (index: number, text: string) => [ + { type: "content_block_start", index, content_block: { type: "text", text: "" } }, + { type: "content_block_delta", index, delta: { type: "text_delta", text } }, + { type: "content_block_stop", index }, + ]; + + const thinkingBlock = (index: number, signature: string | undefined) => [ + { type: "content_block_start", index, content_block: { type: "thinking", thinking: "" } }, + { type: "content_block_delta", index, delta: { type: "thinking_delta", thinking: "Plan" } }, + ...(signature === undefined + ? [] + : [{ type: "content_block_delta", index, delta: { type: "signature_delta", signature } }]), + { type: "content_block_stop", index }, + ]; + + const toolBlock = (index: number) => [ + { + type: "content_block_start", + index, + content_block: { type: "tool_use", id: "t1", name: "look", input: {} }, + }, + { + type: "content_block_delta", + index, + delta: { type: "input_json_delta", partial_json: '{"a":1}' }, + }, + { type: "content_block_stop", index }, + ]; + + const lookTool = [ + { name: "look", description: "", inputSchema: { type: "object", properties: {} } }, + ]; + + /** Runs one reply and asserts it carries no native turn but its text and call came through. */ + async function expectNoNativeTurn( + reply: readonly unknown[], + expectedText: string | undefined + ): Promise { + script = [reply]; + const events: Array> = []; + await toolCallingRunFn()( + { model, prompt: "", messages: [], tools: lookTool } as never, + model, + undefined as never, + ((event: Record) => events.push(event)) as never + ); + expect(nativeTurns(events)).toHaveLength(0); + expect(events.some((e) => e.type === "finish")).toBe(true); + const text = events + .filter((e) => e.type === "text-delta" && e.port === "text") + .map((e) => e.textDelta) + .join(""); + expect(text).toBe(expectedText ?? ""); + const lastCalls = events.filter((e) => e.type === "object-delta" && e.port === "toolCalls"); + expect(lastCalls.at(-1)!.objectDelta).toEqual([{ id: "t1", name: "look", input: { a: 1 } }]); + } + + it("emits no native turn when a block of an unknown type arrived", async () => { + await expectNoNativeTurn( + [ + ...thinkingBlock(0, "SIG"), + { + type: "content_block_start", + index: 1, + content_block: { type: "server_tool_use", id: "s1", name: "web_search", input: {} }, + }, + { type: "content_block_stop", index: 1 }, + ...textBlock(2, "hello"), + ...toolBlock(3), + ], + "hello" + ); + }); + + it("emits no native turn when a delta of an unknown type arrived", async () => { + await expectNoNativeTurn( + [ + ...thinkingBlock(0, "SIG"), + { type: "content_block_start", index: 1, content_block: { type: "text", text: "" } }, + { + type: "content_block_delta", + index: 1, + delta: { type: "citations_delta", citation: { type: "char_location" } }, + }, + { type: "content_block_delta", index: 1, delta: { type: "text_delta", text: "hello" } }, + { type: "content_block_stop", index: 1 }, + ...toolBlock(2), + ], + "hello" + ); + }); + + it("emits no native turn when a thinking block never received its signature", async () => { + await expectNoNativeTurn([...thinkingBlock(0, undefined), ...toolBlock(1)], undefined); + }); + + it("emits no native turn when an empty text block precedes thinking", async () => { + await expectNoNativeTurn( + [ + { type: "content_block_start", index: 0, content_block: { type: "text", text: "" } }, + { type: "content_block_stop", index: 0 }, + ...thinkingBlock(1, "SIG"), + ...toolBlock(2), + ], + undefined + ); + }); + + it("still emits the native turn when the empty text block follows the last thinking", async () => { + script = [ + [ + ...thinkingBlock(0, "SIG"), + { type: "content_block_start", index: 1, content_block: { type: "text", text: "" } }, + { type: "content_block_stop", index: 1 }, + ...toolBlock(2), + ], + ]; + const events = await run([{ role: "user", content: [{ type: "text", text: "go" }] }]); + expect(nativeTurns(events)[0]!.objectDelta.payload).toBe(JSON.stringify(signedBlocks)); + }); + + it("replays the captured turn verbatim while the tool calls still match", async () => { + await run(turnMessages(NATIVE_KEY, "t1")); + expect(sentAssistant(calls[0]!).content).toEqual(signedBlocks); + }); + + it("rebuilds the turn without thinking when a tool call id changed", async () => { + await run(turnMessages(NATIVE_KEY, "t1-2")); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1-2", name: "look", input: { a: 1 } }, + ]); + }); + + it("rebuilds the turn without thinking when a call kept its id but changed its input", async () => { + const edited = turnMessages(NATIVE_KEY, "t1").map((message) => + message.role === "assistant" + ? { + ...message, + content: message.content.map((block) => + block.type === "tool_use" ? { ...block, input: { a: 2 } } : block + ), + } + : message + ); + await run(edited as never); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 2 } }, + ]); + }); + + it("rebuilds the turn without thinking when the host added text the model never wrote", async () => { + const edited = turnMessages(NATIVE_KEY, "t1").map((message) => + message.role === "assistant" + ? { ...message, content: [...message.content, { type: "text", text: "edited" }] } + : message + ); + await run(edited as never); + expect(sentAssistant(calls[0]!).content).not.toContainEqual(signedBlocks[0]); + }); + + it("rebuilds the turn without thinking when another model produced it", async () => { + await run(turnMessages("anthropic:some-other-model", "t1")); + expect(sentAssistant(calls[0]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + }); + + it("strips thinking and retries once when Anthropic rejects the replayed signature", async () => { + script = [bindingError(), thinkingThenCall]; + const events = await run(turnMessages(NATIVE_KEY, "t1")); + expect(calls).toHaveLength(2); + expect(sentAssistant(calls[0]!).content).toEqual(signedBlocks); + expect(sentAssistant(calls[1]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + expect(events.some((e) => e.type === "finish")).toBe(true); + expect(getLogger().warn).toHaveBeenCalledWith( + "Anthropic rejected replayed thinking; retrying without it.", + { model: MODEL_NAME } + ); + }); + + it("strips redacted thinking on the retry too", async () => { + script = [bindingError(), []]; + const messages = turnMessages(NATIVE_KEY, "t1"); + (messages[1]!.content as unknown[])[0] = { + type: "reasoning", + text: "", + provider: NATIVE_KEY, + payload: JSON.stringify([ + { type: "redacted_thinking", data: "R" }, + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]), + }; + await run(messages); + expect(sentAssistant(calls[1]!).content).toEqual([ + { type: "tool_use", id: "t1", name: "look", input: { a: 1 } }, + ]); + }); + + it("does not loop: a second rejection is rethrown", async () => { + script = [bindingError(), bindingError()]; + await expect(run(turnMessages(NATIVE_KEY, "t1"))).rejects.toThrow(/Invalid `signature`/); + expect(calls).toHaveLength(2); + }); + + it("does not retry an unrelated 400", async () => { + script = [Object.assign(new Error("max_tokens too large"), { status: 400 })]; + await expect(run(turnMessages(NATIVE_KEY, "t1"))).rejects.toThrow(/max_tokens/); + expect(calls).toHaveLength(1); + }); + + it("rethrows a binding 400 when the request carried no thinking, after one call", async () => { + script = [bindingError()]; + await expect(run([{ role: "user", content: [{ type: "text", text: "go" }] }])).rejects.toThrow( + /Invalid `signature`/ + ); + expect(calls).toHaveLength(1); + }); + + it("does not retry after output reached the consumer", async () => { + script = [[...textBlock(0, "partial"), bindingError()]]; + const events: Array> = []; + await expect( + toolCallingRunFn()( + { model, prompt: "", messages: turnMessages(NATIVE_KEY, "t1"), tools: [] } as never, + model, + undefined as never, + ((event: Record) => events.push(event)) as never + ) + ).rejects.toThrow(/Invalid `signature`/); + expect(calls).toHaveLength(1); + expect(events.filter((e) => e.type === "text-delta")).toHaveLength(1); + expect(events.find((e) => e.type === "text-delta")!.textDelta).toBe("partial"); + }); +}); diff --git a/packages/test/src/test/ai-provider/Anthropic_ReasoningAndToolImages.test.ts b/packages/test/src/test/ai-provider/Anthropic_ReasoningAndToolImages.test.ts new file mode 100644 index 0000000000..ead4394a25 --- /dev/null +++ b/packages/test/src/test/ai-provider/Anthropic_ReasoningAndToolImages.test.ts @@ -0,0 +1,152 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn } from "@workglow/ai"; +import { ANTHROPIC, _testOnly } from "@workglow/anthropic/ai"; +import { _testOnly as runtimeTestOnly } from "@workglow/anthropic/ai-runtime"; +import { getLogger } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; + +const { ANTHROPIC_RUN_FNS, setAnthropicClientForTests } = _testOnly; + +function toolCallingRunFn(): AiProviderRunFn { + const registration = ANTHROPIC_RUN_FNS.find(({ serves }) => + (serves as readonly string[]).includes("tool-use") + ); + expect(registration).toBeDefined(); + return registration!.runFn as AiProviderRunFn; +} + +const model = { + model_id: "claude-sonnet-5-5", + title: "", + description: "", + provider: ANTHROPIC, + provider_config: { model_name: "claude-sonnet-5-5", api_key: "test-key" }, + capabilities: ["text.generation", "tool-use"], + metadata: {}, +} as never; + +/** + * A conversation another provider produced: its reasoning was streamed as text + * and kept on the assistant turn, and a tool returned an image. + */ +const messages = [ + { role: "user", content: [{ type: "text", text: "What is on the board?" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "I should look at the image." }, + { type: "tool_use", id: "c1", name: "read", input: { path: "board.png" } }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "Image: board.png" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + is_error: undefined, + }, + ], + }, +]; + +describe("Anthropic tool-calling request with reasoning and tool images", () => { + let captured: Record[]; + + beforeEach(() => { + captured = []; + vi.spyOn(getLogger(), "warn").mockImplementation(() => {}); + const fakeClient = { + messages: { + stream: (params: Record) => { + captured.push(params); + return { + async *[Symbol.asyncIterator]() { + // No events: the run-fn finishes on an empty stream. + }, + }; + }, + }, + }; + setAnthropicClientForTests(fakeClient); + runtimeTestOnly.setAnthropicClientForTests?.(fakeClient); + }); + + afterEach(() => { + setAnthropicClientForTests(undefined); + runtimeTestOnly.setAnthropicClientForTests?.(undefined); + vi.restoreAllMocks(); + }); + + it("drops reasoning blocks, which have no unsigned Anthropic form, and keeps tool images", async () => { + await toolCallingRunFn()( + { model, prompt: "", messages, tools: [] } as never, + model, + undefined as never, + (() => {}) as never + ); + expect(captured).toHaveLength(1); + const sent = captured[0]!.messages as Array<{ role: string; content: unknown[] }>; + const assistant = sent.find((message) => message.role === "assistant")!; + expect(assistant.content).toEqual([ + { type: "tool_use", id: "c1", name: "read", input: { path: "board.png" } }, + ]); + expect(JSON.stringify(sent)).not.toContain('"reasoning"'); + const toolResult = sent.at(-1)!.content[0] as { content: unknown[] }; + expect(toolResult.content[1]).toEqual({ + type: "image", + source: { type: "base64", media_type: "image/png", data: "AAAA" }, + }); + }); + + it("sends no empty text block inside a tool result", async () => { + const results = [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + }, + { type: "tool_result", tool_use_id: "c2", content: [{ type: "text", text: "" }] }, + ]; + const history = [ + messages[0], + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "read", input: { path: "board.png" } }, + { type: "tool_use", id: "c2", name: "touch", input: { path: "x" } }, + ], + }, + { role: "tool", content: results }, + ]; + await toolCallingRunFn()( + { model, prompt: "", messages: history, tools: [] } as never, + model, + undefined as never, + (() => {}) as never + ); + const sent = captured[0]!.messages as Array<{ role: string; content: unknown[] }>; + expect(sent.at(-1)!.content).toEqual([ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "image", source: { type: "base64", media_type: "image/png", data: "AAAA" } }, + ], + }, + { type: "tool_result", tool_use_id: "c2" }, + ]); + }); +}); diff --git a/packages/test/src/test/ai-provider/Gemini_ThinkingBudget.test.ts b/packages/test/src/test/ai-provider/Gemini_ThinkingBudget.test.ts index 824622b8eb..0e3f004e74 100644 --- a/packages/test/src/test/ai-provider/Gemini_ThinkingBudget.test.ts +++ b/packages/test/src/test/ai-provider/Gemini_ThinkingBudget.test.ts @@ -90,4 +90,71 @@ describe("resolveThinkingConfig", () => { maxOutputTokens: 1000, }); }); + describe("includeThoughts option", () => { + const flash = (extra: Record = {}) => + model({ provider_config: { model_name: "gemini-2.5-flash" }, ...extra }); + + it("asks for thoughts with no budget when no effort is set", () => { + expect(resolveThinkingConfig(flash(), 100, undefined, { includeThoughts: true })).toEqual({ + thinkingConfig: { includeThoughts: true }, + maxOutputTokens: 100, + }); + }); + + it("adds includeThoughts beside the effort budget and pads the output cap", () => { + expect( + resolveThinkingConfig(flash({ effort: "high" }), 100, undefined, { includeThoughts: true }) + ).toEqual({ + thinkingConfig: { thinkingBudget: 2048, includeThoughts: true }, + maxOutputTokens: 2148, + }); + }); + + it("sends nothing for a model that does not think", () => { + for (const model_name of ["gemini-embedding-001", "imagen-4.0-generate-001"]) { + expect( + resolveThinkingConfig(model({ provider_config: { model_name } }), 100, undefined, { + includeThoughts: true, + }) + ).toEqual({ thinkingConfig: undefined, maxOutputTokens: 100 }); + } + }); + + it("does not ask a model from before Gemini 2.5 for thoughts", () => { + for (const model_name of ["gemini-1.5-pro", "gemini-2.0-flash", "gemini-pro"]) { + expect( + resolveThinkingConfig(model({ provider_config: { model_name } }), 100, undefined, { + includeThoughts: true, + }) + ).toEqual({ thinkingConfig: undefined, maxOutputTokens: 100 }); + } + }); + + it("asks a -latest alias and Gemini 3 for thoughts", () => { + for (const model_name of ["gemini-flash-latest", "gemini-3.5-pro"]) { + expect( + resolveThinkingConfig(model({ provider_config: { model_name } }), 100, undefined, { + includeThoughts: true, + }) + ).toEqual({ thinkingConfig: { includeThoughts: true }, maxOutputTokens: 100 }); + } + }); + + it("does not ask for thoughts when the budget is zero", () => { + expect( + resolveThinkingConfig(flash({ effort: "none" }), 100, undefined, { includeThoughts: true }) + ).toEqual({ thinkingConfig: { thinkingBudget: 0 }, maxOutputTokens: 100 }); + }); + + it("leaves calls without the option unchanged", () => { + expect(resolveThinkingConfig(flash(), 100)).toEqual({ + thinkingConfig: undefined, + maxOutputTokens: 100, + }); + expect(resolveThinkingConfig(flash({ effort: "high" }), 100, undefined, {})).toEqual({ + thinkingConfig: { thinkingBudget: 2048 }, + maxOutputTokens: 2148, + }); + }); + }); }); diff --git a/packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts b/packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts index 50c780fa57..8847706f12 100644 --- a/packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts +++ b/packages/test/src/test/ai-provider/OpenAIGeminiCheckpointParams.test.ts @@ -217,6 +217,121 @@ describe("OpenAI checkpoint prompt_cache_key parity", () => { expect(consumerRequest.tool_choice).toBeUndefined(); }); + it.each([ + ["a reasoning model replays", "gpt-5", true], + ["a model without reasoning rebuilds", "gpt-4.1", false], + ])( + "writes a native turn and a tool image the same way in warm-up and tool calling: %s it", + async (_label, modelName, replays) => { + const model = { provider_config: { model_name: modelName } } as never; + const nativeItems = [ + { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "ENC" }, + { type: "function_call", call_id: "c1", name: "a", arguments: "{}" }, + ]; + const replayPrefix = { + systemPrompt: "sys", + tools: prefix.tools, + messages: [ + { role: "user" as const, content: [{ type: "text" as const, text: "look" }] }, + { + role: "assistant" as const, + content: [ + { + type: "reasoning" as const, + text: "", + provider: `openai:${modelName}`, + payload: JSON.stringify(nativeItems), + }, + { type: "tool_use" as const, id: "c1", name: "a", input: {} }, + ], + }, + { + role: "tool" as const, + content: [ + { + type: "tool_result" as const, + tool_use_id: "c1", + content: [{ type: "image" as const, mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + ], + }; + const session = { sessionId: `ckpt-${modelName}`, prefix: replayPrefix } as AiSessionContext; + await findOpenAiRunFn("cache.checkpoint")( + { model: modelName } as CacheCheckpointTaskInput, + model, + new AbortController().signal, + () => {}, + undefined, + session + ); + await findOpenAiRunFn("tool-use")( + { model: modelName, prompt: "tail", tools: [] } as never, + model, + new AbortController().signal, + () => {}, + undefined, + session + ); + + expect(openAiRequests).toHaveLength(2); + const [warmUpRequest, consumerRequest] = openAiRequests; + const warmed = warmUpRequest.input as unknown[]; + const sent = consumerRequest.input as unknown[]; + expect(sent.slice(0, warmed.length)).toEqual(warmed); + expect(sent.length).toBeGreaterThan(warmed.length); + if (replays) expect(warmed).toContainEqual(nativeItems[0]); + else expect(warmed).not.toContainEqual(nativeItems[0]); + } + ); + + it("writes a tool image the same way in warm-up and text generation", async () => { + const model = { provider_config: { model_name: "gpt-4.1" } } as never; + const imagePrefix = { + systemPrompt: "sys", + messages: [ + { role: "user" as const, content: [{ type: "text" as const, text: "look" }] }, + { + role: "assistant" as const, + content: [{ type: "tool_use" as const, id: "c1", name: "a", input: {} }], + }, + { + role: "tool" as const, + content: [ + { + type: "tool_result" as const, + tool_use_id: "c1", + content: [{ type: "image" as const, mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + ], + }; + const session = { sessionId: "ckpt-text-image", prefix: imagePrefix } as AiSessionContext; + await findOpenAiRunFn("cache.checkpoint")( + { model: "gpt-4.1" } as CacheCheckpointTaskInput, + model, + new AbortController().signal, + () => {}, + undefined, + session + ); + await findOpenAiRunFn("text.generation")( + { model: "gpt-4.1", prompt: "tail" } as TextGenerationTaskInput, + model, + new AbortController().signal, + () => {}, + undefined, + session + ); + const [warmUpRequest, consumerRequest] = openAiRequests; + const warmed = warmUpRequest!.input as unknown[]; + const sent = consumerRequest!.input as unknown[]; + expect(sent.slice(0, warmed.length)).toEqual(warmed); + expect(JSON.stringify(warmed)).toContain("data:image/png;base64,AAAA"); + }); + it("an empty-string system prompt keeps the warmed instructions and cache key", async () => { const session: AiSessionContext = { sessionId: "ckpt-oai-2", prefix }; await runWarmUp(session); diff --git a/packages/test/src/test/ai-provider/OpenAIShapedChatToolArgs.test.ts b/packages/test/src/test/ai-provider/OpenAIShapedChatToolArgs.test.ts index 5f16c4700b..ed16fb6cfc 100644 --- a/packages/test/src/test/ai-provider/OpenAIShapedChatToolArgs.test.ts +++ b/packages/test/src/test/ai-provider/OpenAIShapedChatToolArgs.test.ts @@ -98,3 +98,20 @@ describe("accumulateOpenAIChatStream tool-call arguments", () => { expect(deltas.at(-1)).toEqual([{ id: "c0", name: "now", input: {} }]); }); }); + +describe("accumulateOpenAIChatStream reasoning", () => { + it("streams reasoning_content and reasoning deltas to the reasoning port", async () => { + const out: string[] = []; + await accumulateOpenAIChatStream( + chunks([ + { choices: [{ delta: { reasoning_content: "think " } }] }, + { choices: [{ delta: { reasoning: "more" } }] }, + { choices: [{ delta: { content: "answer" } }] }, + ]), + (e) => { + if (e.type === "text-delta") out.push(`${e.port}:${e.textDelta}`); + } + ); + expect(out).toEqual(["reasoning:think ", "reasoning:more", "text:answer"]); + }); +}); diff --git a/packages/test/src/test/ai-provider/OpenAIShapedResponses.test.ts b/packages/test/src/test/ai-provider/OpenAIShapedResponses.test.ts index da8339e45f..4e85d5b66b 100644 --- a/packages/test/src/test/ai-provider/OpenAIShapedResponses.test.ts +++ b/packages/test/src/test/ai-provider/OpenAIShapedResponses.test.ts @@ -29,6 +29,69 @@ async function collect(list: readonly unknown[]): Promise { } describe("accumulateOpenAIResponsesStream", () => { + it("hands back every output item in order when asked for the native turn", async () => { + const out: StreamEvent>[] = []; + const reasoning = { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "ENC" }; + const call = { + type: "function_call", + id: "fc_1", + call_id: "c1", + name: "look", + arguments: "{}", + }; + await accumulateOpenAIResponsesStream( + events([ + { type: "response.output_item.done", output_index: 1, item: call }, + { type: "response.output_item.done", output_index: 0, item: reasoning }, + { type: "response.completed", response: { usage: undefined } }, + ]), + (e) => out.push(e), + { nativeTurnProvider: "openai" } + ); + const native = out.find((e) => e.type === "object-delta" && e.port === "nativeTurn"); + expect(native).toEqual({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: "openai", payload: JSON.stringify([reasoning, call]) }, + }); + }); + + it("emits no native turn when a reasoning item came back without encrypted content", async () => { + for (const reasoning of [ + { type: "reasoning", id: "rs_1", summary: [] }, + { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "" }, + { type: "reasoning", id: "rs_1", summary: [], encrypted_content: null }, + ]) { + const out: StreamEvent>[] = []; + await accumulateOpenAIResponsesStream( + events([ + { type: "response.output_item.done", output_index: 0, item: reasoning }, + { + type: "response.output_item.done", + output_index: 1, + item: { + type: "function_call", + id: "fc_1", + call_id: "c1", + name: "look", + arguments: "{}", + }, + }, + ]), + (e) => out.push(e), + { nativeTurnProvider: "openai" } + ); + expect(out.some((e) => e.type === "object-delta" && e.port === "nativeTurn")).toBe(false); + } + }); + + it("emits no native turn unless asked", async () => { + const out = await collect([ + { type: "response.output_item.done", output_index: 0, item: { type: "reasoning", id: "r" } }, + ]); + expect(out.some((e) => e.type === "object-delta" && e.port === "nativeTurn")).toBe(false); + }); + it("maps output_text deltas to text-delta events on the text port", async () => { const out = await collect([ { type: "response.created" }, @@ -340,6 +403,84 @@ describe("mapResponsesToolChoice", () => { }); describe("buildResponsesInput", () => { + it("replays an assistant turn's native items verbatim instead of rebuilding it", () => { + const items = [ + { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "ENC" }, + { type: "function_call", id: "fc_1", call_id: "c1", name: "look", arguments: "{}" }, + ]; + const { input } = buildResponsesInput({ + messages: [ + { role: "user", content: "go" }, + { + role: "assistant", + content: null, + tool_calls: [{ id: "c1", type: "function", function: { name: "look", arguments: "{}" } }], + native_items: items, + }, + { role: "tool", tool_call_id: "c1", content: "looked" }, + ] as never, + }); + expect(input).toEqual([ + { role: "user", content: "go" }, + ...items, + { type: "function_call_output", call_id: "c1", output: "looked" }, + ]); + }); + + it("rebuilds the turn when its calls no longer match the native items", () => { + const { input } = buildResponsesInput({ + messages: [ + { + role: "assistant", + content: null, + tool_calls: [ + { id: "c1-2", type: "function", function: { name: "look", arguments: "{}" } }, + ], + native_items: [{ type: "function_call", call_id: "c1", name: "look", arguments: "{}" }], + }, + ] as never, + }); + expect(input).toEqual([ + { type: "function_call", call_id: "c1-2", name: "look", arguments: "{}" }, + ]); + }); + + it("replays despite the API's spacing, and rebuilds when a call's arguments changed", () => { + const turn = (args: string) => + buildResponsesInput({ + messages: [ + { + role: "assistant", + content: null, + tool_calls: [ + { id: "c1", type: "function", function: { name: "look", arguments: args } }, + ], + native_items: [ + { type: "function_call", call_id: "c1", name: "look", arguments: '{"a": 1}' }, + ], + }, + ] as never, + }).input; + expect(turn('{"a":1}')).toEqual([ + { type: "function_call", call_id: "c1", name: "look", arguments: '{"a": 1}' }, + ]); + expect(turn('{"a":2}')).toEqual([ + { type: "function_call", call_id: "c1", name: "look", arguments: '{"a":2}' }, + ]); + }); + + it("rebuilds the turn when its text no longer matches the native items", () => { + const items = [ + { type: "message", role: "assistant", content: [{ type: "output_text", text: "Hi" }] }, + ]; + const turn = (content: string) => + buildResponsesInput({ + messages: [{ role: "assistant", content, native_items: items }] as never, + }).input; + expect(turn("Hi")).toEqual(items); + expect(turn("Edited")).toEqual([{ role: "assistant", content: "Edited" }]); + }); + it("uses the prompt string as input when there are no messages", () => { expect(buildResponsesInput({ prompt: "hello" })).toEqual({ input: "hello", diff --git a/packages/test/src/test/ai-provider/OpenAI_NativeTurn.test.ts b/packages/test/src/test/ai-provider/OpenAI_NativeTurn.test.ts new file mode 100644 index 0000000000..87d9cd7496 --- /dev/null +++ b/packages/test/src/test/ai-provider/OpenAI_NativeTurn.test.ts @@ -0,0 +1,250 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn } from "@workglow/ai"; +import { OPENAI, _testOnly } from "@workglow/openai/ai"; +import { _testOnly as runtimeTestOnly } from "@workglow/openai/ai-runtime"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const { OPENAI_RUN_FNS, setOpenAIClientForTests } = _testOnly; + +function toolCallingRunFn(): AiProviderRunFn { + const registration = OPENAI_RUN_FNS.find(({ serves }) => + (serves as readonly string[]).includes("tool-use") + ); + expect(registration).toBeDefined(); + return registration!.runFn as AiProviderRunFn; +} + +const modelWith = (providerConfig: Record) => + ({ + model_id: "gpt-5.5", + title: "gpt-5.5", + description: "", + provider: OPENAI, + provider_config: { + model_name: MODEL_NAME, + api_key: "test-key", + reasoning: { effort: "high" }, + ...providerConfig, + }, + capabilities: ["text.generation", "tool-use"], + metadata: {}, + }) as never; + +const reasoningItem = { type: "reasoning", id: "rs_1", summary: [], encrypted_content: "ENC" }; +const callItem = { + type: "function_call", + id: "fc_1", + call_id: "c1", + name: "look", + arguments: "{}", +}; + +const MODEL_NAME = "gpt-5.5"; + +const messagesFrom = (provider: string) => [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [ + { + type: "reasoning", + text: "", + provider, + payload: JSON.stringify([reasoningItem, callItem]), + }, + { type: "tool_use", id: "c1", name: "look", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "text", text: "looked" }], + is_error: undefined, + }, + ], + }, +]; + +const messages = messagesFrom(`openai:${MODEL_NAME}`); + +describe("OpenAI tool-calling native turn", () => { + let captured: Record[]; + let streamEvents: unknown[]; + + beforeEach(() => { + captured = []; + streamEvents = []; + const fakeClient = { + responses: { + create: async (params: Record) => { + captured.push(params); + return { + async *[Symbol.asyncIterator]() { + for (const event of streamEvents) yield event; + }, + }; + }, + }, + }; + setOpenAIClientForTests(fakeClient); + runtimeTestOnly.setOpenAIClientForTests(fakeClient); + }); + + afterEach(() => { + setOpenAIClientForTests(undefined); + runtimeTestOnly.setOpenAIClientForTests(undefined); + }); + + const run = async ( + model: never, + emitted: unknown[] = [], + history: unknown[] = messages + ): Promise> => { + await toolCallingRunFn()( + { + model, + prompt: "", + messages: history, + tools: [{ name: "look", description: "", inputSchema: {} }], + } as never, + model, + undefined as never, + ((event: unknown) => emitted.push(event)) as never + ); + expect(captured).toHaveLength(1); + return captured[0]!; + }; + + it("asks for the encrypted reasoning when the model reasons", async () => { + const params = await run(modelWith({})); + expect(params.include).toEqual(["reasoning.encrypted_content"]); + }); + + it("asks for nothing extra when replay is switched off", async () => { + const params = await run(modelWith({ replay_reasoning: false })); + expect(params).not.toHaveProperty("include"); + expect(JSON.stringify(params.input)).not.toContain("ENC"); + }); + + it("asks for nothing extra when reasoning is off", async () => { + const params = await run(modelWith({ reasoning: { effort: "none" } })); + expect(params).not.toHaveProperty("include"); + }); + + it("rebuilds an earlier native turn when reasoning is off", async () => { + const params = await run(modelWith({ reasoning: { effort: "none" } })); + expect(params.input).toEqual([ + { role: "user", content: [{ type: "input_text", text: "go" }] }, + { type: "function_call", call_id: "c1", name: "look", arguments: "{}" }, + { type: "function_call_output", call_id: "c1", output: "looked" }, + ]); + expect(JSON.stringify(params.input)).not.toContain("ENC"); + }); + + it("replays the native items it produced earlier", async () => { + const params = await run(modelWith({})); + expect(params.input).toEqual([ + { role: "user", content: [{ type: "input_text", text: "go" }] }, + reasoningItem, + callItem, + { type: "function_call_output", call_id: "c1", output: "looked" }, + ]); + }); + + it("sends a tool result's images on a user turn after the results", async () => { + const withImage = messages.map((message) => + message.role === "tool" + ? { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "looked" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + is_error: undefined, + }, + ], + } + : message + ); + const params = await run(modelWith({}), [], withImage); + const input = params.input as Record[]; + expect(input.at(-2)).toMatchObject({ type: "function_call_output", call_id: "c1" }); + expect(input.at(-1)).toEqual({ + role: "user", + content: [ + { type: "input_text", text: "Images returned by the tool calls above:" }, + { type: "input_text", text: "[image 1]" }, + { type: "input_image", image_url: "data:image/png;base64,AAAA" }, + ], + }); + }); + + it("rebuilds a turn another model produced instead of replaying it", async () => { + const params = await run(modelWith({}), [], messagesFrom("openai:some-other-model")); + expect(params.input).toEqual([ + { role: "user", content: [{ type: "input_text", text: "go" }] }, + { type: "function_call", call_id: "c1", name: "look", arguments: "{}" }, + { type: "function_call_output", call_id: "c1", output: "looked" }, + ]); + expect(JSON.stringify(params.input)).not.toContain("ENC"); + }); + + it("emits the turn's items as the native turn", async () => { + streamEvents = [ + { type: "response.output_item.done", output_index: 0, item: reasoningItem }, + { type: "response.output_item.done", output_index: 1, item: callItem }, + { type: "response.completed", response: {} }, + ]; + const emitted: Array<{ type: string; port?: string }> = []; + await run(modelWith({}), emitted); + expect(emitted.find((e) => e.port === "nativeTurn")).toEqual({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { + provider: `openai:${MODEL_NAME}`, + payload: JSON.stringify([reasoningItem, callItem]), + }, + }); + }); + + it("emits no native turn when replay is switched off", async () => { + streamEvents = [{ type: "response.output_item.done", output_index: 0, item: reasoningItem }]; + const emitted: Array<{ type: string; port?: string }> = []; + await run(modelWith({ replay_reasoning: false }), emitted); + expect(emitted.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("emits no native turn when the model takes no reasoning", async () => { + streamEvents = [ + { type: "response.output_item.done", output_index: 0, item: reasoningItem }, + { type: "response.output_item.done", output_index: 1, item: callItem }, + ]; + const emitted: Array<{ type: string; port?: string }> = []; + const params = await run(modelWith({ model_name: "gpt-4o", reasoning: undefined }), emitted); + expect(params).not.toHaveProperty("include"); + expect(emitted.some((e) => e.port === "nativeTurn")).toBe(false); + }); + + it("emits no native turn when reasoning effort is none", async () => { + streamEvents = [ + { type: "response.output_item.done", output_index: 0, item: reasoningItem }, + { type: "response.output_item.done", output_index: 1, item: callItem }, + ]; + const emitted: Array<{ type: string; port?: string }> = []; + const params = await run(modelWith({ reasoning: { effort: "none" } }), emitted); + expect(params).not.toHaveProperty("include"); + expect(emitted.some((e) => e.port === "nativeTurn")).toBe(false); + }); +}); diff --git a/packages/test/src/test/ai-provider/ThinkTagFilter.test.ts b/packages/test/src/test/ai-provider/ThinkTagFilter.test.ts new file mode 100644 index 0000000000..80736b01b2 --- /dev/null +++ b/packages/test/src/test/ai-provider/ThinkTagFilter.test.ts @@ -0,0 +1,119 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import { createThinkTagFilter } from "@workglow/ai/provider-utils"; +import { describe, expect, it } from "vitest"; + +type Part = { readonly port: "text" | "reasoning"; readonly text: string }; + +/** Merges adjacent same-port parts so assertions do not depend on chunking. */ +function merged(parts: ReadonlyArray): Part[] { + const out: Part[] = []; + for (const part of parts) { + if (part.text === "") continue; + const last = out[out.length - 1]; + if (last && last.port === part.port) { + out[out.length - 1] = { port: last.port, text: last.text + part.text }; + } else { + out.push(part); + } + } + return out; +} + +describe("createThinkTagFilter", () => { + it("routes a think block to reasoning and the rest to text, across split tags", () => { + const f = createThinkTagFilter(); + const parts = [ + ...f.push("plan"), + ...f.push(" more\n\nAnswer"), + ...f.flush(), + ]; + expect(merged(parts)).toEqual([ + { port: "reasoning", text: "plan more" }, + { port: "text", text: "\n\nAnswer" }, + ]); + expect(parts.filter((p) => p.port === "reasoning").map((p) => p.text)).toEqual([ + "plan", + " more", + ]); + }); + + it("starts inside a think block when asked", () => { + const g = createThinkTagFilter({ startInside: true }); + expect(merged([...g.push("whyok"), ...g.flush()])).toEqual([ + { port: "reasoning", text: "why" }, + { port: "text", text: "ok" }, + ]); + }); + + it("passes text containing other tags through untouched", () => { + const h = createThinkTagFilter(); + expect(merged([...h.push("plain text"), ...h.flush()])).toEqual([ + { port: "text", text: "plain text" }, + ]); + }); + + it("releases a held partial open tag as text once it cannot be a tag", () => { + const f = createThinkTagFilter(); + const parts = [...f.push("a { + const outside = createThinkTagFilter(); + expect(merged([...outside.push("done { + const f = createThinkTagFilter(); + expect(merged([...f.push("cut off"), ...f.push(" here"), ...f.flush()])).toEqual([ + { port: "reasoning", text: "cut off here" }, + ]); + }); + + it("handles several blocks and a whole stream in one push", () => { + const f = createThinkTagFilter(); + expect(merged([...f.push("abcd"), ...f.flush()])).toEqual([ + { port: "reasoning", text: "a" }, + { port: "text", text: "b" }, + { port: "reasoning", text: "c" }, + { port: "text", text: "d" }, + ]); + }); + + it("gives the same result however the stream is chunked", () => { + const whole = "xwhy\nbecause\nanswer again end"; + const expected = merged( + (() => { + const f = createThinkTagFilter(); + return [...f.push(whole), ...f.flush()]; + })() + ); + for (let size = 1; size <= 8; size++) { + const f = createThinkTagFilter(); + const parts: Part[] = []; + for (let i = 0; i < whole.length; i += size) parts.push(...f.push(whole.slice(i, i + size))); + parts.push(...f.flush()); + expect(merged(parts)).toEqual(expected); + } + }); + + it("is usable again after a flush", () => { + const f = createThinkTagFilter(); + f.push("a"); + f.flush(); + expect(merged([...f.push("b"), ...f.flush()])).toEqual([{ port: "text", text: "b" }]); + }); +}); diff --git a/packages/test/src/test/ai-provider/WebBrowserProvider.test.ts b/packages/test/src/test/ai-provider/WebBrowserProvider.test.ts index 549bb60630..69b7d5faf7 100644 --- a/packages/test/src/test/ai-provider/WebBrowserProvider.test.ts +++ b/packages/test/src/test/ai-provider/WebBrowserProvider.test.ts @@ -1106,6 +1106,28 @@ describe("WebBrowser_ToolCalling session lifecycle", () => { restore(); } }); + + it("finishes with the full empty scaffold, leaving accumulation to the deltas", async () => { + const { factory } = makeFakeToolCallingModel(); + const restore = installLanguageModelGlobal(factory); + try { + const events: Array<{ type?: string; data?: unknown }> = []; + await WebBrowser_ToolCalling( + asTCI({ prompt: "do it", tools: [toolA, toolB] }), + undefined, + new AbortController().signal, + (e: unknown) => { + events.push(e as { type?: string; data?: unknown }); + }, + undefined, + { sessionId: sid } + ); + const finish = events.find((e) => e.type === "finish"); + expect(finish?.data).toEqual({ text: "", toolCalls: [] }); + } finally { + restore(); + } + }); }); // -------------------------------------------------------------------------- diff --git a/packages/test/src/test/ai/AgentTask.test.ts b/packages/test/src/test/ai/AgentTask.test.ts index 0fe42ccf82..1bc6e195c4 100644 --- a/packages/test/src/test/ai/AgentTask.test.ts +++ b/packages/test/src/test/ai/AgentTask.test.ts @@ -43,6 +43,8 @@ const MODEL: ModelConfig = { interface ScriptedRound { readonly text?: string; + /** Used by the host-runner script only: handed to `onReasoningDelta`. */ + readonly reasoning?: string; /** Report the text on the finish event only, as a non-streaming provider does. */ readonly withoutDeltas?: boolean; readonly calls?: ReadonlyArray<{ @@ -648,6 +650,85 @@ describe("AgentTask", () => { expect(JSON.stringify(toolResults(output.messages)[0])).toContain("asking for one failed"); }); + it("withdraws an unanswered approval at the turn's deadline instead of waiting on", async () => { + scriptModel([ + { + calls: [ + { id: "c1", name: "AgentTest_FetchTask", input: { url: "https://example.test" } }, + ], + }, + { text: "ok" }, + ]); + let withdrawn = false; + // A person who never answers: the request settles only when it is withdrawn. + registry.registerInstance(HUMAN_CONNECTOR, { + send: (_request: unknown, signal: AbortSignal) => + new Promise((_resolve, reject) => { + signal.addEventListener("abort", () => { + withdrawn = true; + reject(signal.reason); + }); + }), + }); + + const started = Date.now(); + const output = await new AgentTask().run( + { model: MODEL, prompt: "fetch", tools: [FETCH_TOOL], maxDurationMs: 300 }, + { registry } + ); + + expect(Date.now() - started).toBeLessThan(5_000); + expect(withdrawn).toBe(true); + expect(fetchRuns).toBe(0); + expect(output.stopReason).toBe("budget"); + expect(JSON.stringify(toolResults(output.messages)[0])).toContain("time ran out"); + }); + + it("waits on an approval normally under a deadline too far off for a timer", async () => { + scriptModel([ + { + calls: [ + { id: "c1", name: "AgentTest_FetchTask", input: { url: "https://example.test" } }, + ], + }, + { text: "ok" }, + ]); + // Answers after a moment, and withdraws on abort as a real connector does: a + // timer that overflowed to 1 ms would withdraw it before the answer lands. + registry.registerInstance(HUMAN_CONNECTOR, { + send: (request: IHumanRequest, signal: AbortSignal): Promise => + new Promise((resolve, reject) => { + const timer = setTimeout( + () => + resolve({ + requestId: request.requestId, + action: "accept", + content: undefined, + done: true, + }), + 50 + ); + signal.addEventListener("abort", () => { + clearTimeout(timer); + reject(signal.reason); + }); + }), + }); + + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "fetch", + tools: [FETCH_TOOL], + maxDurationMs: 30 * 24 * 60 * 60 * 1000, + }, + { registry } + ); + + expect(fetchRuns).toBe(1); + expect(output.stopReason).toBe("answered"); + }); + it("refuses rather than runs when there is nobody to ask", async () => { scriptModel([ { @@ -1133,8 +1214,13 @@ describe("AgentTask", () => { inputs.push(input); signals.push(context.signal); context.onProgress(40, "Loading model"); + if (round.reasoning) context.onReasoningDelta(round.reasoning); if (round.text) context.onTextDelta(round.text); - return { text: round.text ?? "", toolCalls: [...(round.calls ?? [])] }; + return { + text: round.text ?? "", + toolCalls: [...(round.calls ?? [])], + ...(round.reasoning ? { reasoning: round.reasoning } : {}), + }; }; return { runner, inputs, signals }; } @@ -1196,6 +1282,54 @@ describe("AgentTask", () => { expect(signals.every((signal) => signal instanceof AbortSignal)).toBe(true); }); + it("forwards the reasoning a runner reports on the turn's own port, apart from the text", async () => { + const { runner } = scriptRunner([ + { + reasoning: "Need the tool.", + text: "Looking. ", + calls: [{ id: "c1", name: "AgentTest_EchoTask", input: { text: "hi" } }], + }, + { reasoning: "Got it.", text: "It said HI." }, + ]); + registry.registerInstance(AGENT_ROUND_RUNNER, runner); + const task = new AgentTask(); + const events: Array<{ port: string; delta: string }> = []; + task.subscribe("stream_chunk", (event) => { + if (event.type === "text-delta") { + events.push({ port: event.port ?? "text", delta: event.textDelta }); + } + }); + + const output = await task.run( + { model: MODEL, prompt: "echo hi", tools: [ECHO_TOOL], approval: "never" }, + { registry } + ); + + expect(events.filter((e) => e.port === "reasoning").map((e) => e.delta)).toEqual([ + "Need the tool.", + "\n\n", + "Got it.", + ]); + expect(events.filter((e) => e.port === "text").map((e) => e.delta)).toEqual([ + "Looking. ", + "It said HI.", + ]); + expect(output.reasoning).toBe("Need the tool.\n\nGot it."); + expect(output.text).toBe("Looking. It said HI."); + }); + + it("leaves reasoning off the output when the runner reported none", async () => { + const { runner } = scriptRunner([{ text: "Plain." }]); + registry.registerInstance(AGENT_ROUND_RUNNER, runner); + + const output = await new AgentTask().run( + { model: MODEL, prompt: "hi", tools: [ECHO_TOOL], approval: "never" }, + { registry } + ); + + expect("reasoning" in output).toBe(false); + }); + it("still answers every call the model made through the runner", async () => { const { runner } = scriptRunner([ { calls: [{ id: "c1", name: "nope", input: {} }] }, diff --git a/packages/test/src/test/ai/AgentTaskDeadlineAndReview.test.ts b/packages/test/src/test/ai/AgentTaskDeadlineAndReview.test.ts new file mode 100644 index 0000000000..9ab4d3a0c1 --- /dev/null +++ b/packages/test/src/test/ai/AgentTaskDeadlineAndReview.test.ts @@ -0,0 +1,352 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { + AiProviderRunFn, + ChatMessage, + ModelConfig, + ToolCallingTaskInput, + ToolDefinition, +} from "@workglow/ai"; +import { + AgentTask, + AiProviderRegistry, + DirectExecutionStrategy, + getAiProviderRegistry, + setAiProviderRegistry, +} from "@workglow/ai"; +import { RetryableJobError } from "@workglow/job-queue"; +import { TaskConfigurationError } from "@workglow/task-graph"; +import { Container, ServiceRegistry } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const PROVIDER = "mock-deadline-provider"; + +const MODEL: ModelConfig = { + model_id: "mock/deadline-1", + provider: PROVIDER, + title: "", + description: "", + capabilities: ["tool-use"], + provider_config: {}, + metadata: {}, +}; + +interface Round { + readonly text?: string; + readonly calls?: ReadonlyArray<{ id: string; name: string; input: Record }>; +} + +/** A scripted model, one entry per call with the last repeating; returns what each call was sent. */ +function script(rounds: readonly Round[]): ToolCallingTaskInput[] { + const seen: ToolCallingTaskInput[] = []; + const runFn: AiProviderRunFn = async (input, _model, _signal, emit) => { + const round = rounds[Math.min(seen.length, rounds.length - 1)]!; + seen.push(input as ToolCallingTaskInput); + if (round.text) emit({ type: "text-delta", port: "text", textDelta: round.text }); + if (round.calls?.length) { + emit({ type: "object-delta", port: "toolCalls", objectDelta: [...round.calls] }); + } + emit({ type: "finish", data: {} }); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + return seen; +} + +const EMPTY = { type: "object", properties: {}, additionalProperties: false } as const; + +function tool(name: string, execute: ToolDefinition["execute"]): ToolDefinition { + return { name, description: name, inputSchema: EMPTY, execute }; +} + +describe("AgentTask deadline", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + it("hands a tool the moment the turn's time budget runs out", async () => { + script([{ calls: [{ id: "c1", name: "clock", input: {} }] }, { text: "done" }]); + const seen: Array = []; + const before = Date.now(); + await new AgentTask().run( + { + model: MODEL, + prompt: "?", + maxDurationMs: 60_000, + tools: [ + tool("clock", async (_input, context) => { + seen.push(context.deadline); + return "ok"; + }), + ], + }, + { registry } + ); + expect(seen).toHaveLength(1); + expect(seen[0]).toBeGreaterThanOrEqual(before + 60_000); + expect(seen[0]).toBeLessThanOrEqual(Date.now() + 60_000); + }); + + it("hands no deadline on a turn with no time budget", async () => { + script([{ calls: [{ id: "c1", name: "clock", input: {} }] }, { text: "done" }]); + const seen: Array = []; + await new AgentTask().run( + { + model: MODEL, + prompt: "?", + tools: [ + tool("clock", async (_input, context) => { + seen.push(context.deadline); + return "ok"; + }), + ], + }, + { registry } + ); + expect(seen).toEqual([undefined]); + }); +}); + +describe("AgentTask announceTimeLeft", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + const look = tool("look", async () => "looked"); + + it("appends the time left to the round's last tool result", async () => { + const seen = script([ + { + calls: [ + { id: "c1", name: "look", input: {} }, + { id: "c2", name: "look", input: {} }, + ], + }, + { text: "done" }, + ]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look], maxDurationMs: 600_000, announceTimeLeft: true }, + { registry } + ); + const sent = (seen[1]!.messages as ChatMessage[]).find((m) => m.role === "tool")!; + const [first, last] = sent.content as unknown as Array<{ + content: Array<{ type: string; text?: string }>; + }>; + expect(first!.content.map((block) => block.text)).toEqual(["looked"]); + expect(last!.content).toHaveLength(1); + expect(last!.content[0]!.type).toBe("text"); + expect(last!.content[0]!.text).toMatch(/^looked\n\n\[Time left: (9m [0-5]?\d|10m 0)s\]$/); + // The step records what the model was shown, notice included. + expect(output.steps![0]!.tools.map((t) => t.chars)).toEqual([ + "looked".length, + last!.content[0]!.text!.length, + ]); + }); + + it("says nothing about time unless asked", async () => { + const seen = script([{ calls: [{ id: "c1", name: "look", input: {} }] }, { text: "done" }]); + await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look], maxDurationMs: 600_000 }, + { registry } + ); + expect(JSON.stringify(seen[1]!.messages)).not.toContain("Time left"); + }); + + it("refuses announceTimeLeft without a time budget to announce", async () => { + script([{ text: "done" }]); + await expect( + new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look], announceTimeLeft: true }, + { registry } + ) + ).rejects.toBeInstanceOf(TaskConfigurationError); + }); +}); + +describe("AgentTask deadline and round retries", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + it("ends the turn instead of sleeping through a retry wait that passes the deadline", async () => { + let calls = 0; + const runFn: AiProviderRunFn = async () => { + calls++; + throw new RetryableJobError("429", new Date(Date.now() + 30_000)); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + const started = Date.now(); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [], maxDurationMs: 5_000 }, + { registry } + ); + expect(output.stopReason).toBe("budget"); + expect(Date.now() - started).toBeLessThan(5_000); + expect(calls).toBe(1); + }); + + it("does not start a round a slow progress write carried past the deadline", async () => { + let calls = 0; + const runFn: AiProviderRunFn = async () => { + calls++; + throw new Error("the model must not be called"); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + const task = new AgentTask(); + task.events.on("progress", (_progress, message) => { + if (message !== "Thinking") return; + const until = Date.now() + 300; + while (Date.now() < until) { + // A slow progress listener holds the turn past its deadline. + } + }); + const output = await task.run( + { model: MODEL, prompt: "?", tools: [], maxDurationMs: 200 }, + { registry } + ); + expect(output.stopReason).toBe("budget"); + expect(output.rounds).toBe(0); + expect(calls).toBe(0); + }); + + it("does not start a retry the wait carried past the deadline", async () => { + let calls = 0; + const runFn: AiProviderRunFn = async () => { + calls++; + // Asks for a 1 s wait with 1.5 s left; the progress write then runs long. + throw new RetryableJobError("429", new Date(Date.now() + 1_000)); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + const task = new AgentTask(); + task.events.on("progress", (_progress, message) => { + if (!message?.startsWith("Retrying")) return; + const until = Date.now() + 700; + while (Date.now() < until) { + // A slow progress listener holds the turn past its deadline. + } + }); + const output = await task.run( + { model: MODEL, prompt: "?", tools: [], maxDurationMs: 1_500, maxRoundRetries: 5 }, + { registry } + ); + expect(output.stopReason).toBe("budget"); + expect(calls).toBe(1); + }); +}); + +describe("AgentTask reviewPrompt", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + const reviewText = (messages: readonly ChatMessage[]): number => + messages.filter( + (m) => m.role === "user" && JSON.stringify(m.content).includes("Check your work.") + ).length; + + it("asks once for a review when the model first tries to finish", async () => { + const seen = script([{ text: "done" }, { text: "checked" }]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [], reviewPrompt: "Check your work." }, + { registry } + ); + expect(seen).toHaveLength(2); + expect(reviewText(output.messages)).toBe(1); + expect(output.stopReason).toBe("answered"); + }); + + it("lets the model go back to its tools after the review", async () => { + let looked = 0; + const seen = script([ + { text: "done" }, + { calls: [{ id: "c1", name: "look", input: {} }] }, + { text: "fixed" }, + ]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "?", + tools: [ + tool("look", async () => { + looked++; + return "looked"; + }), + ], + reviewPrompt: "Check your work.", + }, + { registry } + ); + expect(seen).toHaveLength(3); + expect(looked).toBe(1); + expect(reviewText(output.messages)).toBe(1); + }); + + it("sends no review when no round is left to act on it", async () => { + const seen = script([{ text: "done" }]); + await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [], reviewPrompt: "Check your work.", maxRounds: 1 }, + { registry } + ); + expect(seen).toHaveLength(1); + }); + + it("keeps the turn's prompt and work in the review round when history is over budget", async () => { + const seen = script([{ calls: [{ id: "c1", name: "look", input: {} }] }, { text: "done" }]); + await new AgentTask().run( + { + model: MODEL, + prompt: "Write the report.", + messages: [ + { role: "user", content: [{ type: "text", text: "an earlier question" }] }, + { role: "assistant", content: [{ type: "text", text: "an earlier answer" }] }, + ], + tools: [tool("look", async () => "x".repeat(2_000))], + reviewPrompt: "Check your work.", + maxHistoryChars: 500, + }, + { registry } + ); + expect(seen).toHaveLength(3); + const review = seen[2]!.messages!; + // The earlier turn is cut to fit; the review is not a turn of its own. + expect(JSON.stringify(review)).not.toContain("an earlier question"); + expect(JSON.stringify(review)).toContain("Write the report."); + expect(review.some((m) => m.role === "tool")).toBe(true); + expect(reviewText(review)).toBe(1); + }); +}); diff --git a/packages/test/src/test/ai/AgentTaskNativeTurn.test.ts b/packages/test/src/test/ai/AgentTaskNativeTurn.test.ts new file mode 100644 index 0000000000..415fd11f11 --- /dev/null +++ b/packages/test/src/test/ai/AgentTaskNativeTurn.test.ts @@ -0,0 +1,133 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn, ChatMessage, ModelConfig, ToolCallingTaskInput } from "@workglow/ai"; +import { + AgentTask, + AiProviderRegistry, + DirectExecutionStrategy, + getAiProviderRegistry, + setAiProviderRegistry, +} from "@workglow/ai"; +import { Container, ServiceRegistry } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const PROVIDER = "mock-native-turn-provider"; + +const MODEL: ModelConfig = { + model_id: "mock/native-1", + provider: PROVIDER, + title: "", + description: "", + capabilities: ["tool-use"], + provider_config: {}, + metadata: {}, +}; + +interface Round { + readonly reasoning?: string; + readonly text?: string; + readonly calls?: ReadonlyArray<{ id: string; name: string; input: Record }>; + readonly native?: { readonly provider: string; readonly payload: string }; +} + +/** A scripted model that may hand back its own turn, and records what each round was sent. */ +function script(rounds: readonly Round[]): ToolCallingTaskInput[] { + const seen: ToolCallingTaskInput[] = []; + const runFn: AiProviderRunFn = async (input, _model, _signal, emit) => { + const round = rounds[Math.min(seen.length, rounds.length - 1)]!; + seen.push(input as ToolCallingTaskInput); + if (round.reasoning) + emit({ type: "text-delta", port: "reasoning", textDelta: round.reasoning }); + if (round.text) emit({ type: "text-delta", port: "text", textDelta: round.text }); + if (round.calls?.length) { + emit({ type: "object-delta", port: "toolCalls", objectDelta: [...round.calls] }); + } + if (round.native) { + emit({ type: "object-delta", port: "nativeTurn", objectDelta: round.native }); + } + emit({ type: "finish", data: {} }); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + return seen; +} + +const look = { + name: "look", + description: "Look", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => "looked", +} as const; + +describe("AgentTask native turn", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + it("keeps a round's native turn on its reply and sends it back next round", async () => { + const seen = script([ + { + calls: [{ id: "c1", name: "look", input: {} }], + native: { provider: "mock", payload: '[{"kind":"reasoning","enc":"xyz"}]' }, + }, + { text: "done" }, + ]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look] }, + { registry } + ); + const reply = output.messages.find((m) => m.role === "assistant")!; + expect(reply.content[0]).toEqual({ + type: "reasoning", + text: "", + provider: "mock", + payload: '[{"kind":"reasoning","enc":"xyz"}]', + }); + const sent = (seen[1]!.messages as ChatMessage[]).find((m) => m.role === "assistant")!; + expect(sent.content[0]).toEqual({ + type: "reasoning", + text: "", + provider: "mock", + payload: '[{"kind":"reasoning","enc":"xyz"}]', + }); + }); + + it("carries text reasoning and the native turn on one block", async () => { + script([ + { + reasoning: "Plan.", + calls: [{ id: "c1", name: "look", input: {} }], + native: { provider: "mock", payload: "[]" }, + }, + { text: "done" }, + ]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [look] }, + { registry } + ); + const blocks = output.messages + .find((m) => m.role === "assistant")! + .content.filter((b) => b.type === "reasoning"); + expect(blocks).toEqual([{ type: "reasoning", text: "Plan.", provider: "mock", payload: "[]" }]); + }); + + it("records no native turn on a reply that has neither text nor calls", async () => { + script([{ native: { provider: "mock", payload: "[]" } }]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "?", tools: [] }, + { registry } + ); + expect(output.messages.filter((m) => m.role === "assistant")).toEqual([]); + }); +}); diff --git a/packages/test/src/test/ai/AgentTaskReasoningAndImages.test.ts b/packages/test/src/test/ai/AgentTaskReasoningAndImages.test.ts new file mode 100644 index 0000000000..d1b7c93755 --- /dev/null +++ b/packages/test/src/test/ai/AgentTaskReasoningAndImages.test.ts @@ -0,0 +1,379 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { AiProviderRunFn, ChatMessage, ModelConfig, ToolCallingTaskInput } from "@workglow/ai"; +import { + AgentTask, + AiProviderRegistry, + DirectExecutionStrategy, + getAiProviderRegistry, + setAiProviderRegistry, + ToolResultContent, +} from "@workglow/ai"; +import { RetryableJobError } from "@workglow/job-queue"; +import { Container, ServiceRegistry } from "@workglow/util"; +import { afterEach, beforeEach, describe, expect, it } from "vitest"; + +const PROVIDER = "mock-reasoning-provider"; + +const MODEL: ModelConfig = { + model_id: "mock/reasoning-1", + provider: PROVIDER, + title: "", + description: "", + capabilities: ["tool-use"], + provider_config: {}, + metadata: {}, +}; + +interface Round { + readonly reasoning?: string; + readonly text?: string; + readonly calls?: ReadonlyArray<{ id: string; name: string; input: Record }>; + /** Streamed, then thrown instead of answering, as a provider failing mid-stream does. */ + readonly failWith?: { + readonly error: Error; + readonly reasoning?: string; + readonly text?: string; + }; +} + +/** A scripted model that streams reasoning beside its reply, and records what each round was sent. */ +function script(rounds: readonly Round[]): ToolCallingTaskInput[] { + const seen: ToolCallingTaskInput[] = []; + const runFn: AiProviderRunFn = async (input, _model, _signal, emit) => { + const round = rounds[Math.min(seen.length, rounds.length - 1)]!; + seen.push(input as ToolCallingTaskInput); + if (round.failWith) { + const { error, reasoning, text } = round.failWith; + if (reasoning) emit({ type: "text-delta", port: "reasoning", textDelta: reasoning }); + if (text) emit({ type: "text-delta", port: "text", textDelta: text }); + throw error; + } + if (round.reasoning) + emit({ type: "text-delta", port: "reasoning", textDelta: round.reasoning }); + if (round.text) emit({ type: "text-delta", port: "text", textDelta: round.text }); + if (round.calls?.length) { + emit({ type: "object-delta", port: "toolCalls", objectDelta: [...round.calls] }); + } + emit({ type: "finish", data: {} }); + }; + getAiProviderRegistry().registerRunFn(PROVIDER, { serves: ["tool-use"], runFn }); + return seen; +} + +describe("AgentTask reasoning and tool images", () => { + let registry: ServiceRegistry; + + beforeEach(() => { + setAiProviderRegistry(new AiProviderRegistry()); + getAiProviderRegistry().setDefaultStrategy(new DirectExecutionStrategy()); + registry = new ServiceRegistry(new Container()); + }); + + afterEach(() => { + getAiProviderRegistry().unregisterProvider(PROVIDER); + }); + + it("keeps a round's reasoning on its reply, so the next round is sent it", async () => { + const seen = script([ + { + reasoning: "The file is the place to start.", + calls: [{ id: "c1", name: "look", input: {} }], + }, + { reasoning: "Done thinking.", text: "It is fine." }, + ]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "Check it", + tools: [ + { + name: "look", + description: "Look", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => "looked", + }, + ], + }, + { registry } + ); + + const replies = output.messages.filter((message) => message.role === "assistant"); + expect(replies[0]!.content[0]).toEqual({ + type: "reasoning", + text: "The file is the place to start.", + }); + expect(replies[1]!.content).toEqual([ + { type: "reasoning", text: "Done thinking." }, + { type: "text", text: "It is fine." }, + ]); + // The second round was sent the first round's reasoning with its call. + const sentReply = (seen[1]!.messages as ChatMessage[]).find((m) => m.role === "assistant"); + expect(sentReply!.content.map((block) => block.type)).toEqual(["reasoning", "tool_use"]); + // Reasoning is not part of what the turn said. + expect(output.text).toBe("It is fine."); + }); + + it("records no reply for a round that only reasoned", async () => { + script([{ reasoning: "Hmm." }]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "Say nothing", tools: [] }, + { registry } + ); + expect(output.messages.filter((message) => message.role === "assistant")).toEqual([]); + }); + + it("hands the model an image a tool returned through ToolResultContent", async () => { + const seen = script([ + { calls: [{ id: "c1", name: "snap", input: {} }] }, + { text: "I see a board." }, + ]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "What is in the picture?", + tools: [ + { + name: "snap", + description: "Take a picture", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => + new ToolResultContent([ + { type: "text", text: "Image: board.png" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ]), + }, + ], + }, + { registry } + ); + + const result = output.messages.find((message) => message.role === "tool")!.content[0]!; + expect(result).toEqual({ + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "Image: board.png" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + is_error: undefined, + }); + const sentTool = (seen[1]!.messages as ChatMessage[]).find((m) => m.role === "tool")!; + expect(JSON.stringify(sentTool)).toContain('"type":"image"'); + // The step record counts the text the model read, not the image bytes. + expect(output.steps[0]!.tools[0]!.chars).toBe("Image: board.png".length); + }); + + it("keeps a tool's text beside the image it describes, and the text within budget", async () => { + script([{ calls: [{ id: "c1", name: "pair", input: {} }] }, { text: "done" }]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "Compare them", + maxToolResultChars: 12, + tools: [ + { + name: "pair", + description: "Two pictures", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => + new ToolResultContent([ + { type: "text", text: "before:" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "after:" }, + { type: "text", text: "second" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + ]), + }, + ], + }, + { registry } + ); + + const result = output.messages.find((message) => message.role === "tool")!.content[0]!; + expect(result.type === "tool_result" && result.content).toEqual([ + { type: "text", text: "before:" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "after… [truncated, 20 chars total]" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + ]); + }); + + it("counts the separator joining a tool's adjacent text blocks, as the model reads them", async () => { + script([{ calls: [{ id: "c1", name: "two", input: {} }] }, { text: "done" }]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "go", + tools: [ + { + name: "two", + description: "Two blocks", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => + new ToolResultContent([ + { type: "text", text: "a" }, + { type: "text", text: "b" }, + ]), + }, + ], + }, + { registry } + ); + const result = output.messages.find((message) => message.role === "tool")!.content[0]!; + expect(result.type === "tool_result" && result.content).toEqual([ + { type: "text", text: "a\nb" }, + ]); + expect(output.steps[0]!.tools[0]!.chars).toBe(3); + }); + + it("says text was left out when the budget ran out exactly at an earlier block", async () => { + script([{ calls: [{ id: "c1", name: "pair", input: {} }] }, { text: "done" }]); + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "Compare them", + maxToolResultChars: 7, + tools: [ + { + name: "pair", + description: "Two pictures", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => + new ToolResultContent([ + { type: "text", text: "before:" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "after" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + { type: "text", text: "end" }, + ]), + }, + ], + }, + { registry } + ); + + const result = output.messages.find((message) => message.role === "tool")!.content[0]!; + expect(result.type === "tool_result" && result.content).toEqual([ + { type: "text", text: "before:" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + { type: "text", text: "… [truncated, 15 chars total]" }, + { type: "image", mimeType: "image/png", data: "BBBB" }, + ]); + }); + + it("refuses a round timeout longer than a timer can be set", async () => { + script([{ text: "done" }]); + await expect( + new AgentTask().run( + { model: MODEL, prompt: "hi", tools: [], roundTimeoutMs: 2_147_483_648 }, + { registry } + ) + ).rejects.toThrow(/roundTimeoutMs|2147483647|maximum/); + }); + + describe("reasoning on the turn's own port", () => { + const LOOK = { + name: "look", + description: "Look", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + execute: async () => "looked", + } as const; + + function listen(task: AgentTask): Array<{ port: string; delta: string }> { + const events: Array<{ port: string; delta: string }> = []; + task.subscribe("stream_chunk", (event) => { + if (event.type === "text-delta") { + events.push({ port: event.port ?? "text", delta: event.textDelta }); + } + }); + return events; + } + + it("streams a round's reasoning on its own port, apart from the text", async () => { + script([ + { reasoning: "Check the file first.", calls: [{ id: "c1", name: "look", input: {} }] }, + { reasoning: "It looks fine.", text: "All good." }, + ]); + const task = new AgentTask(); + const events = listen(task); + + const output = await task.run( + { model: MODEL, prompt: "Check it", tools: [LOOK], approval: "never" }, + { registry } + ); + + expect(events.filter((event) => event.port === "reasoning").map((e) => e.delta)).toEqual([ + "Check the file first.", + // A round's reasoning is told from the one before it. + "\n\n", + "It looks fine.", + ]); + expect(events.filter((event) => event.port === "text").map((e) => e.delta)).toEqual([ + "All good.", + ]); + expect(output.reasoning).toBe("Check the file first.\n\nIt looks fine."); + expect(output.text).toBe("All good."); + }); + + it("leaves reasoning off the output when the model streamed none", async () => { + script([{ text: "Plain answer." }]); + const task = new AgentTask(); + const events = listen(task); + + const output = await task.run( + { model: MODEL, prompt: "Hi", tools: [], approval: "never" }, + { registry } + ); + + expect(events.some((event) => event.port === "reasoning")).toBe(false); + expect(output.reasoning).toBeUndefined(); + expect("reasoning" in output).toBe(false); + expect(output.text).toBe("Plain answer."); + }); + + it("does not separate a round that reasoned from one before it that did not", async () => { + script([ + { calls: [{ id: "c1", name: "look", input: {} }] }, + { reasoning: "Only now.", text: "Done." }, + ]); + const output = await new AgentTask().run( + { model: MODEL, prompt: "Check it", tools: [LOOK], approval: "never" }, + { registry } + ); + expect(output.reasoning).toBe("Only now."); + }); + + it("drops a failed attempt's reasoning and text, so the retry's are not joined to them", async () => { + script([ + { + failWith: { + error: new RetryableJobError("overloaded", new Date(Date.now())), + reasoning: "Half a thought", + text: "Half an ans", + }, + }, + { reasoning: "A whole thought.", text: "A whole answer." }, + ]); + const task = new AgentTask(); + const events = listen(task); + + const output = await task.run( + { model: MODEL, prompt: "?", tools: [], approval: "never" }, + { registry } + ); + + expect(output.steps[0]!.attempts).toBe(2); + expect(output.reasoning).toBe("A whole thought."); + expect(output.text).toBe("A whole answer."); + expect(events).toEqual([ + { port: "reasoning", delta: "A whole thought." }, + { port: "text", delta: "A whole answer." }, + ]); + }); + }); +}); diff --git a/packages/test/src/test/ai/AgentTaskTurnControls.test.ts b/packages/test/src/test/ai/AgentTaskTurnControls.test.ts index e0e0861e60..77e3e13ab8 100644 --- a/packages/test/src/test/ai/AgentTaskTurnControls.test.ts +++ b/packages/test/src/test/ai/AgentTaskTurnControls.test.ts @@ -333,6 +333,42 @@ describe("AgentTask turn controls", () => { ]); expect(overlap.peak).toBe(1); }); + + it("does not start a tool whose approval arrives after the deadline", async () => { + script([{ calls: [{ id: "a", name: "gated", input: {} }] }, { text: "done" }]); + const slowApproval: IHumanConnector = { + send: async (request) => { + await new Promise((resolve) => setTimeout(resolve, 120)); + return { requestId: request.requestId, action: "accept", content: undefined, done: true }; + }, + }; + registry.registerInstance(HUMAN_CONNECTOR, slowApproval); + let ran = false; + const output = await new AgentTask().run( + { + model: MODEL, + prompt: "?", + tools: [ + { + name: "gated", + description: "gated", + inputSchema: { type: "object", properties: {}, additionalProperties: false }, + requiresApproval: true, + execute: async () => { + ran = true; + return "ran"; + }, + }, + ], + maxDurationMs: 50, + }, + { registry } + ); + expect(ran).toBe(false); + const [result] = toolResults(output.messages); + expect(result!.is_error).toBe(true); + expect(JSON.stringify(result!.content)).toContain("time ran out"); + }); }); describe("steps", () => { diff --git a/packages/test/src/test/task/MessageConversion.test.ts b/packages/test/src/test/task/MessageConversion.test.ts index 8c4def1bc8..89ba5318af 100644 --- a/packages/test/src/test/task/MessageConversion.test.ts +++ b/packages/test/src/test/task/MessageConversion.test.ts @@ -5,7 +5,12 @@ */ import type { ToolCallingTaskInput, ToolDefinition } from "@workglow/ai"; -import { toOpenAIMessages, toTextFlatMessages } from "@workglow/ai"; +import { + liftSystemMessages, + normalizeChatMessages, + toOpenAIMessages, + toTextFlatMessages, +} from "@workglow/ai"; import { describe, expect, test } from "vitest"; const dummyTools: ToolDefinition[] = [ @@ -45,6 +50,72 @@ describe("toOpenAIMessages", () => { expect(msgs[1].role).toBe("user"); }); + test("keeps a nested tool_use a tool returned, as JSON, whichever way images go", () => { + const nested = { type: "tool_use", id: "x", name: "inner", input: { a: 1 } }; + const input = makeInput({ + messages: [ + { role: "assistant", content: [{ type: "tool_use", id: "c1", name: "t", input: {} }] }, + { + role: "tool", + content: [{ type: "tool_result", tool_use_id: "c1", content: [nested] }], + }, + ], + } as any); + expect(toOpenAIMessages(input, { toolImagesInUserMessage: true })[1]).toEqual({ + role: "tool", + content: JSON.stringify(nested), + tool_call_id: "c1", + }); + expect(toOpenAIMessages(input)[1]!.content).toEqual([ + { type: "text", text: JSON.stringify(nested) }, + ]); + }); + + test("passes an image_url part a caller wrote for chat-completions through", () => { + const part = { type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } }; + const msgs = toOpenAIMessages( + makeInput({ + messages: [{ role: "user", content: [{ type: "text", text: "what is this" }, part] }], + } as any) + ); + expect(msgs).toEqual([ + { role: "user", content: [{ type: "text", text: "what is this" }, part] }, + ]); + }); + + test("joins system turns in the history to the system prompt", () => { + const msgs = toOpenAIMessages( + makeInput({ + systemPrompt: "base", + messages: [ + { role: "system", content: [{ type: "text", text: "rule one" }] }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ], + } as any) + ); + expect(msgs).toEqual([ + { role: "system", content: "base\n\nrule one" }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ]); + const only = toOpenAIMessages( + makeInput({ + messages: [ + { role: "system", content: [{ type: "text", text: "rule" }] }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ], + } as any) + ); + expect(only[0]).toEqual({ role: "system", content: "rule" }); + // A history of system turns alone is still the history: the prompt is not sent. + const systemOnly = toOpenAIMessages( + makeInput({ + prompt: "stale", + messages: [{ role: "system", content: [{ type: "text", text: "rule" }] }], + } as any) + ); + expect(systemOnly).toEqual([{ role: "system", content: "rule" }]); + }); + test("should convert multi-turn messages with user and assistant", () => { const input = makeInput({ messages: [ @@ -125,7 +196,7 @@ describe("toOpenAIMessages", () => { }); }); - test("should set content to null for empty assistant text", () => { + test("should send empty assistant text as an empty string when there are no tool calls", () => { const input = makeInput({ messages: [ { role: "user", content: [{ type: "text", text: "Go" }] }, @@ -134,7 +205,7 @@ describe("toOpenAIMessages", () => { }); const msgs = toOpenAIMessages(input); - expect(msgs[1].content).toBeNull(); + expect(msgs[1].content).toBe(""); }); test("should convert user message with image content blocks to OpenAI format", () => { @@ -217,6 +288,279 @@ describe("toOpenAIMessages", () => { expect(parts[0]).toEqual({ type: "text", text: "Plain text" }); expect(parts[1].type).toBe("image_url"); }); + + test("sends an empty tool result as an empty string, not an empty parts array", () => { + const out = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "t", input: {} }], + }, + { role: "tool", content: [{ type: "tool_result", tool_use_id: "c1", content: [] }] }, + ] as any, + }) + ); + expect(out.at(-1)).toEqual({ role: "tool", content: "", tool_call_id: "c1" }); + }); + + test("sends a reasoning-only assistant turn with empty-string content", () => { + const out = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { role: "assistant", content: [{ type: "reasoning", text: "thinking" }] }, + { role: "user", content: [{ type: "text", text: "again" }] }, + ] as any, + }) + ); + expect(out[1]).toEqual({ role: "assistant", content: "" }); + }); + + test("keeps null content for an assistant turn that only calls tools", () => { + const out = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "go" }] }, + { + role: "assistant", + content: [{ type: "tool_use", id: "c1", name: "t", input: {} }], + }, + ] as any, + }) + ); + expect(out[1]!.content).toBeNull(); + expect(out[1]!.tool_calls).toHaveLength(1); + }); + + describe("reasoning and tool images", () => { + const history = makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "Look" }] }, + { + role: "assistant", + content: [ + { type: "reasoning", text: "I should read the file." }, + { type: "tool_use", id: "c1", name: "read", input: { path: "board.png" } }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "Image: board.png" }, + { type: "image", mimeType: "image/png", data: "AAAA" }, + ], + is_error: undefined, + }, + ], + }, + ], + }); + + test("drops reasoning blocks unless asked to replay them", () => { + const msgs = toOpenAIMessages(history); + expect(msgs[1]).not.toHaveProperty("reasoning_content"); + expect(msgs[1]!.content).toBeNull(); + }); + + test("replays reasoning as reasoning_content when asked", () => { + const msgs = toOpenAIMessages(history, { replayReasoning: true }); + expect(msgs[1]).toMatchObject({ + role: "assistant", + reasoning_content: "I should read the file.", + tool_calls: [{ id: "c1" }], + }); + }); + + test("keeps tool images inline by default", () => { + const msgs = toOpenAIMessages(history); + expect(msgs[2]).toMatchObject({ role: "tool", tool_call_id: "c1" }); + expect(msgs[2]!.content).toEqual([ + { type: "text", text: "Image: board.png" }, + { type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } }, + ]); + }); + + test("moves tool images into a user message after the turn's results when asked", () => { + const msgs = toOpenAIMessages(history, { toolImagesInUserMessage: true }); + expect(msgs[2]).toEqual({ + role: "tool", + tool_call_id: "c1", + content: "Image: board.png\n[image 1] follows in the next message", + }); + expect(msgs[3]).toEqual({ + role: "user", + content: [ + { type: "text", text: "Images returned by the tool calls above:" }, + { type: "text", text: "[image 1]" }, + { type: "image_url", image_url: { url: "data:image/png;base64,AAAA" } }, + ], + }); + }); + + test("marks each moved image where it stood, numbered across the turn's results", () => { + const image = (data: string) => ({ type: "image", mimeType: "image/png", data }); + const msgs = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "Compare" }] }, + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [ + { type: "text", text: "before:" }, + image("AAAA"), + { type: "text", text: "after:" }, + image("BBBB"), + ], + }, + { type: "tool_result", tool_use_id: "c2", content: [image("CCCC")] }, + ], + }, + ], + } as never), + { toolImagesInUserMessage: true } + ); + expect(msgs[2]!.content).toBe( + "before:\n[image 1] follows in the next message\nafter:\n[image 2] follows in the next message" + ); + expect(msgs[3]!.content).toBe("[image 3] follows in the next message"); + const labels = ( + msgs[4]!.content as Array<{ type: string; text?: string; image_url?: { url: string } }> + ) + .slice(1) + .map((part) => part.text ?? part.image_url!.url.slice(-4)); + expect(labels).toEqual(["[image 1]", "AAAA", "[image 2]", "BBBB", "[image 3]", "CCCC"]); + }); + }); + + test("holds tool images until every separate tool message of the turn is out", () => { + const msgs = toOpenAIMessages( + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "Look" }] }, + { + role: "assistant", + content: [ + { type: "tool_use", id: "c1", name: "shot", input: {} }, + { type: "tool_use", id: "c2", name: "shot", input: {} }, + ], + }, + { + role: "tool", + content: [ + { + type: "tool_result", + tool_use_id: "c1", + content: [{ type: "image", mimeType: "image/png", data: "AAAA" }], + }, + ], + }, + { + role: "tool", + content: [ + { type: "tool_result", tool_use_id: "c2", content: [{ type: "text", text: "ok" }] }, + ], + }, + { role: "user", content: [{ type: "text", text: "Next" }] }, + ], + } as never), + { toolImagesInUserMessage: true } + ); + expect(msgs.map((m) => (m.role === "tool" ? `tool:${m.tool_call_id}` : m.role))).toEqual([ + "user", + "assistant", + "tool:c1", + "tool:c2", + "user", + "user", + ]); + expect(JSON.stringify(msgs[4]!.content)).toContain("Images returned by the tool calls above"); + }); + + describe("native turn replay", () => { + const withBlock = (block: Record) => + makeInput({ + messages: [ + { role: "user", content: [{ type: "text", text: "Look" }] }, + { + role: "assistant", + content: [block, { type: "tool_use", id: "c1", name: "read", input: {} }], + }, + ], + } as never); + const items = [{ type: "reasoning", id: "rs_1", encrypted_content: "ENC" }]; + + test("attaches the items of a reasoning block this provider produced", () => { + const msgs = toOpenAIMessages( + withBlock({ + type: "reasoning", + text: "", + provider: "openai", + payload: JSON.stringify(items), + }), + { nativeTurnProvider: "openai" } + ); + expect(msgs[1]!.native_items).toEqual(items); + }); + + test("ignores a block another provider produced", () => { + const msgs = toOpenAIMessages( + withBlock({ + type: "reasoning", + text: "", + provider: "anthropic", + payload: JSON.stringify(items), + }), + { nativeTurnProvider: "openai" } + ); + expect(msgs[1]).not.toHaveProperty("native_items"); + }); + + test("ignores a payload that is not JSON, not an array, or not typed items", () => { + for (const payload of [ + "{not json", + JSON.stringify({ a: 1 }), + "[]", + "[null]", + JSON.stringify([{ type: "reasoning" }, 7]), + JSON.stringify([{ id: "rs_1" }]), + ]) { + const msgs = toOpenAIMessages( + withBlock({ type: "reasoning", text: "", provider: "openai", payload }), + { nativeTurnProvider: "openai" } + ); + expect(msgs[1]).not.toHaveProperty("native_items"); + } + }); + + test("attaches nothing unless asked", () => { + const msgs = toOpenAIMessages( + withBlock({ + type: "reasoning", + text: "", + provider: "openai", + payload: JSON.stringify(items), + }) + ); + expect(msgs[1]).not.toHaveProperty("native_items"); + }); + }); }); // ======================================================================== @@ -367,3 +711,36 @@ describe("toTextFlatMessages", () => { expect(msgs[0]).toEqual({ role: "user", content: "Look at this" }); }); }); + +describe("normalizeChatMessages", () => { + test("turns bare-string content into a text block and leaves blocks alone", () => { + const blocks = [{ type: "image" as const, mimeType: "image/png", data: "AAAA" }]; + expect( + normalizeChatMessages([ + { role: "user", content: "hi" }, + { role: "user", content: blocks }, + ]) + ).toEqual([ + { role: "user", content: [{ type: "text", text: "hi" }] }, + { role: "user", content: blocks }, + ]); + }); +}); + +describe("liftSystemMessages", () => { + test("moves system turns into the system prompt, after the one given", () => { + const lifted = liftSystemMessages( + [ + { role: "system", content: [{ type: "text", text: "rule one" }] }, + { role: "user", content: [{ type: "text", text: "hi" }] }, + ], + "base" + ); + expect(lifted.systemPrompt).toBe("base\n\nrule one"); + expect(lifted.messages).toEqual([{ role: "user", content: [{ type: "text", text: "hi" }] }]); + }); + + test("leaves the system prompt undefined when there is none anywhere", () => { + expect(liftSystemMessages([], undefined).systemPrompt).toBeUndefined(); + }); +}); diff --git a/providers/anthropic/src/ai/common/Anthropic_CacheCheckpoint.ts b/providers/anthropic/src/ai/common/Anthropic_CacheCheckpoint.ts index 28fe1d36ea..40fb90c056 100644 --- a/providers/anthropic/src/ai/common/Anthropic_CacheCheckpoint.ts +++ b/providers/anthropic/src/ai/common/Anthropic_CacheCheckpoint.ts @@ -16,6 +16,7 @@ import { buildToolDescription } from "@workglow/ai/worker"; import { getClient, getModelName } from "./Anthropic_Client"; import type { AnthropicModelConfig } from "./Anthropic_ModelSchema"; import { buildAnthropicMessages } from "./Anthropic_ToolCalling"; +import { applyAnthropicThinkingParams } from "./Anthropic_Thinking"; import { mapAnthropicUsage } from "./Anthropic_Usage"; /** @@ -72,11 +73,42 @@ export function annotateLastBlock(message: { content: unknown }): void { * with no messages gets a throwaway user message (not annotated — the * system/tools breakpoints cover the cached content). */ +const CHECKPOINT_THINKING_BLOCK_TYPES: ReadonlySet = new Set([ + "thinking", + "redacted_thinking", +]); + +/** + * The messages with every thinking block removed (a message left empty is + * dropped), or `undefined` when there was none to remove. Local copy of the + * tool-calling retry helper: this module cannot import it back without + * widening the existing import cycle with `Anthropic_ToolCalling`. + */ +function stripCheckpointThinking(messages: readonly unknown[]): unknown[] | undefined { + const out: unknown[] = []; + let removed = false; + for (const msg of messages) { + const m = msg as { role?: unknown; content?: unknown }; + if (m?.role !== "assistant" || !Array.isArray(m.content)) { + out.push(msg); + continue; + } + const content = (m.content as unknown[]).filter( + (b) => !CHECKPOINT_THINKING_BLOCK_TYPES.has((b as { type?: unknown })?.type as string) + ); + if (content.length !== (m.content as unknown[]).length) removed = true; + if (content.length > 0) out.push({ ...(m as Record), content }); + } + return removed ? out : undefined; +} + export function buildAnthropicCheckpointParams( prefix: CheckpointPrefix, - modelName: string + modelName: string, + model?: AnthropicModelConfig | undefined ): Record { const params: Record = { model: modelName, max_tokens: 1 }; + if (model !== undefined) applyAnthropicThinkingParams(params, model); if (prefix.systemPrompt) { params.system = wrapSystemWithCacheControl(prefix.systemPrompt); } @@ -90,8 +122,24 @@ export function buildAnthropicCheckpointParams( // entries, so a non-empty prefix.messages can still convert to []. const messages = prefix.messages && prefix.messages.length > 0 - ? buildAnthropicMessages(prefix.messages, "") + ? buildAnthropicMessages(prefix.messages, "", `anthropic:${modelName}`) : []; + // A replayed signed thinking block without thinking enabled is a 400, + // so when a known model left no thinking on the request, strip the blocks + // rather than fail the warm-up. Without a model the thinking params are + // unknown and the replay is preserved. + if ( + model !== undefined && + messages.length > 0 && + params.thinking === undefined && + (params.output_config as { effort?: unknown } | undefined)?.effort === undefined + ) { + const stripped = stripCheckpointThinking(messages); + if (stripped !== undefined) { + messages.length = 0; + messages.push(...stripped); + } + } if (messages.length > 0) { annotateLastBlock(messages[messages.length - 1] as { content: unknown }); params.messages = messages; @@ -138,9 +186,12 @@ export function applyAnthropicPrefixReplay( // Guard on the CONVERTED array: buildAnthropicMessages skips system-role // entries, so a non-empty prefix.messages can still convert to []. + // The prefix's own turns replay with their signed thinking, keyed as tool + // calling keys them, so a checkpointed turn reaches the model as it wrote it. + const nativeKey = typeof params.model === "string" ? `anthropic:${params.model}` : undefined; const prefixMessages = prefix.messages && prefix.messages.length > 0 - ? buildAnthropicMessages(prefix.messages, "") + ? buildAnthropicMessages(prefix.messages, "", nativeKey) : []; if (prefixMessages.length > 0) { annotateLastBlock(prefixMessages[prefixMessages.length - 1] as { content: unknown }); @@ -163,7 +214,7 @@ export const Anthropic_CacheCheckpoint_Stream: AiProviderRunFn< > = async (_input, model, signal, emit, _outputSchema, session) => { const prefix = session?.prefix ?? {}; const client = await getClient(model); - const params = buildAnthropicCheckpointParams(prefix, getModelName(model)); + const params = buildAnthropicCheckpointParams(prefix, getModelName(model), model); const response = await (client.messages.create as (p: unknown, o: unknown) => Promise)( params, { signal } diff --git a/providers/anthropic/src/ai/common/Anthropic_TextGeneration.ts b/providers/anthropic/src/ai/common/Anthropic_TextGeneration.ts index 2613b8e5a0..5b71abec19 100644 --- a/providers/anthropic/src/ai/common/Anthropic_TextGeneration.ts +++ b/providers/anthropic/src/ai/common/Anthropic_TextGeneration.ts @@ -65,7 +65,8 @@ export const Anthropic_TextGeneration_Stream: AiProviderRunFn< const messages = hasMessages ? buildAnthropicMessages( unified.messages as Parameters[0], - unified.prompt ?? "" + unified.prompt ?? "", + undefined ) : [{ role: "user" as const, content: input.prompt }]; diff --git a/providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts b/providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts index 0c6f67c23e..ba1652136b 100644 --- a/providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts +++ b/providers/anthropic/src/ai/common/Anthropic_ToolCalling.ts @@ -43,9 +43,91 @@ function isEmptyTextBlock(block: { type?: unknown; text?: unknown }): boolean { return block.type === "text" && (block.text === undefined || block.text === ""); } +/** + * Whether a stored block is one of the kinds this provider captures, with the + * fields Anthropic requires of it. An empty text block is refused outright, and + * one cannot be dropped from a signed turn, so a payload holding one (never + * emitted here) is rebuilt instead; so is a tool_use without an input object. + */ +function isReplayableBlock(block: unknown): boolean { + if (block === null || typeof block !== "object" || Array.isArray(block)) return false; + const b = block as Record; + switch (b.type) { + case "text": + return typeof b.text === "string" && !isEmptyTextBlock(b); + case "thinking": + return ( + typeof b.thinking === "string" && typeof b.signature === "string" && b.signature !== "" + ); + case "redacted_thinking": + return typeof b.data === "string" && b.data !== ""; + case "tool_use": + return ( + typeof b.id === "string" && + typeof b.name === "string" && + b.input !== null && + typeof b.input === "object" && + !Array.isArray(b.input) + ); + default: + return false; + } +} + +/** + * The turn a model produced, as the blocks the API returned in the order it + * returned them — or `undefined` when the payload is unusable or the host + * changed the turn's tool calls (an id, a name or an input) or its text, in + * which case it is rebuilt from the message. + */ +function parseOwnTurn(payload: string, content: ChatMessage["content"]): any[] | undefined { + let blocks: unknown; + try { + blocks = JSON.parse(payload); + } catch { + return undefined; + } + // Every block must be one this provider writes, carrying what Anthropic + // requires of it; anything else would be a 400 rather than a rebuilt turn. + if (!Array.isArray(blocks) || !blocks.every(isReplayableBlock)) return undefined; + // The payload's calls must be the message's calls exactly — id, name and + // input — or the replay would send a call the host has since changed. + const sent = blocks.filter((b) => b?.type === "tool_use") as Array<{ + id?: unknown; + name?: unknown; + input?: unknown; + }>; + const calls = content.filter((b) => b.type === "tool_use"); + if (sent.length !== calls.length) return undefined; + const same = sent.every( + (block, i) => + block.id === calls[i]!.id && + block.name === calls[i]!.name && + JSON.stringify(block.input) === JSON.stringify(calls[i]!.input ?? {}) + ); + if (!same) return undefined; + // And its text must be the message's text: the turn's text arrives as its + // text blocks run together, so an edited reply rebuilds rather than replays. + const sentText = blocks + .filter((b) => b?.type === "text" && typeof b.text === "string") + .map((b) => (b as { text: string }).text) + .join(""); + const messageText = content + .filter((b) => b.type === "text") + .map((b) => (b as { text: string }).text) + .join(""); + if (sentText !== messageText) return undefined; + return blocks.length > 0 ? blocks : undefined; +} + +/** + * `nativeTurnProvider` is the key a reasoning block must carry to be replayed; + * without it every assistant turn is rebuilt from its text and tool calls. + */ export function buildAnthropicMessages( messages: ReadonlyArray | undefined, - prompt: unknown + prompt: unknown, + nativeTurnProvider: string | undefined ): any[] { if (!messages || messages.length === 0) { return [{ role: "user", content: prompt }]; @@ -54,6 +136,7 @@ export function buildAnthropicMessages( for (const msg of messages) { if (msg.role === "user") { const blocks = msg.content + .filter((b) => b.type !== "reasoning") .map((b) => { if (b.type === "text") return { type: "text", text: b.text }; if (b.type === "image") { @@ -68,7 +151,28 @@ export function buildAnthropicMessages( if (blocks.length === 0) continue; out.push({ role: "user", content: blocks }); } else if (msg.role === "assistant") { + // The model's own turn, replayed exactly as it produced it: a thinking + // block's signature binds it to the conversation before it, so its text, + // the tool inputs and the block order are sent unchanged — and only while + // the calls still match, since a host that renamed one changed the turn. + const own = + nativeTurnProvider === undefined + ? undefined + : msg.content.find( + (b): b is Extract => + b.type === "reasoning" && + b.provider === nativeTurnProvider && + b.payload !== undefined + ); + const replayed = own ? parseOwnTurn(own.payload!, msg.content) : undefined; + if (replayed) { + out.push({ role: "assistant", content: replayed }); + continue; + } + // Reasoning another provider streamed as text has no Anthropic form (its + // thinking blocks are signed), and an unknown block is a 400. const blocks = msg.content + .filter((b) => b.type !== "reasoning") .map((b) => { if (b.type === "text") return { type: "text", text: b.text }; if (b.type === "tool_use") { @@ -86,20 +190,24 @@ export function buildAnthropicMessages( b.type === "tool_result" ) .map((b) => { - const content = b.content.map((inner) => { - if (inner.type === "text") return { type: "text", text: inner.text }; - if (inner.type === "image") { - return { - type: "image", - source: { type: "base64", media_type: inner.mimeType, data: inner.data }, - }; - } - return inner; - }); + const content = b.content + .map((inner) => { + if (inner.type === "text") return { type: "text", text: inner.text }; + if (inner.type === "image") { + return { + type: "image", + source: { type: "base64", media_type: inner.mimeType, data: inner.data }, + }; + } + return inner; + }) + .filter((inner) => !isEmptyTextBlock(inner)); + // A tool that returned nothing still answers its call: `content` is + // optional on a tool_result, where an empty text block is a 400. return { type: "tool_result", tool_use_id: b.tool_use_id, - content, + ...(content.length > 0 ? { content } : {}), ...(b.is_error ? { is_error: true } : {}), }; }); @@ -127,6 +235,84 @@ type AnthropicBlockMeta = } | { readonly type: "text" }; +/** A content block as the API produced it, accumulated for replay. */ +type NativeBlock = + | { type: "thinking"; thinking: string; signature: string } + | { type: "redacted_thinking"; data: string } + | { type: "text"; text: string } + | { type: "tool_use"; id: string; name: string; raw: string }; + +const THINKING_BLOCK_TYPES: ReadonlySet = new Set(["thinking", "redacted_thinking"]); + +/** Anthropic's 400 for a replayed thinking block whose signature no longer matches. */ +function isThinkingSignatureError(err: unknown): boolean { + const e = err as { status?: unknown; message?: unknown } | undefined; + return ( + e?.status === 400 && + typeof e.message === "string" && + e.message.includes("Invalid `signature` in `thinking` block") + ); +} + +/** + * The messages with every thinking block removed (a message left empty is + * dropped), or `undefined` when there was none to remove — a request without + * thinking cannot have been rejected for it, so there is nothing to retry. + */ +function withoutThinking(messages: readonly any[]): any[] | undefined { + const out: any[] = []; + let removed = false; + for (const msg of messages) { + if (msg?.role !== "assistant" || !Array.isArray(msg.content)) { + out.push(msg); + continue; + } + const content = msg.content.filter((b: any) => !THINKING_BLOCK_TYPES.has(b?.type)); + if (content.length !== msg.content.length) removed = true; + if (content.length > 0) out.push({ ...msg, content }); + } + return removed ? out : undefined; +} + +/** + * The reply's blocks in index order as the API produced them, or `undefined` + * when there is nothing to replay or the turn cannot be replayed verbatim: a + * tool input that cannot be parsed from its raw JSON, a thinking block that + * never received its signature, or an empty text block that must be dropped + * ahead of a thinking block (the signature binds the blocks before it). The + * replay must carry exactly what the model wrote, so a guess is worse than + * rebuilding the turn. + */ +function serializeNativeTurn(native: ReadonlyMap): string | undefined { + const ordered = [...native.entries()].sort((a, b) => a[0] - b[0]).map(([, block]) => block); + let lastThinking = -1; + ordered.forEach((block, i) => { + if (THINKING_BLOCK_TYPES.has(block.type)) lastThinking = i; + }); + const blocks: unknown[] = []; + for (const [i, block] of ordered.entries()) { + if (block.type === "tool_use") { + let input: unknown; + try { + input = block.raw ? JSON.parse(block.raw) : {}; + } catch { + return undefined; + } + blocks.push({ type: "tool_use", id: block.id, name: block.name, input }); + } else if (block.type === "text") { + // The API rejects an empty text block, so one is dropped — which is only + // faithful when no thinking block follows it. + if (block.text !== "") blocks.push(block); + else if (i < lastThinking) return undefined; + } else if (block.type === "thinking" && block.signature === "") { + return undefined; + } else { + blocks.push(block); + } + } + return blocks.length > 0 ? JSON.stringify(blocks) : undefined; +} + function mapAnthropicToolChoice( toolChoice: string | undefined ): { type: "auto" } | { type: "any" } | { type: "tool"; name: string } | undefined { @@ -165,7 +351,10 @@ export const Anthropic_ToolCalling_Stream: AiProviderRunFn< toolChoice = { type: "auto" }; } - const messages = buildAnthropicMessages(input.messages, input.prompt); + // Thinking blocks are bound to the model that produced them, so the native + // turn is keyed by model and another model gets the rebuilt turn. + const nativeKey = `anthropic:${modelName}`; + const messages = buildAnthropicMessages(input.messages, input.prompt, nativeKey); const params: any = { model: modelName, @@ -183,6 +372,23 @@ export const Anthropic_ToolCalling_Stream: AiProviderRunFn< applyAnthropicThinkingParams(params, model); applyAnthropicSamplingParams(params, input, model); + // A forced tool choice suppresses legacy extended thinking (Anthropic rejects + // the combination), but the history may hold signed thinking blocks from an + // earlier auto-choice turn for this same model. Sending those blocks without + // thinking enabled is a 400 the signature retry below does not handle, so + // strip them up front. Scoped to the suppression case — thinking absent with + // no effort configured either — so an adaptive run (thinking or + // output_config.effort present) keeps replaying its turn verbatim. + if ( + (toolChoice?.type === "any" || toolChoice?.type === "tool") && + params.thinking === undefined && + (params.output_config as { effort?: unknown } | undefined)?.effort === undefined && + Array.isArray(params.messages) + ) { + const stripped = withoutThinking(params.messages); + if (stripped !== undefined) params.messages = stripped; + } + if (input.systemPrompt) { params.system = input.systemPrompt; } @@ -216,90 +422,160 @@ export const Anthropic_ToolCalling_Stream: AiProviderRunFn< } } - const stream = client.messages.stream(params, { signal }); - - const blockMeta = new Map(); - /** Keyed by Anthropic content block index — avoids collisions when `id` is missing during early deltas. */ - const toolCallsByBlockIndex = new Map(); - - const toolCallsInStreamOrder = (): ToolCall[] => - [...toolCallsByBlockIndex.entries()].sort((a, b) => a[0] - b[0]).map(([, tc]) => tc); + /** Whether anything has reached the consumer, since a retry after output would repeat it. */ + let emitted = false; + const emitTracked: typeof emit = (event) => { + emitted = true; + emit(event); + }; /** - * Defence-in-depth: drop tool calls whose name isn't in the effective - * declared tool set (the caller's tools, or the prefix's on fallback) - * before yielding to the consumer. Anthropic's tool API normally respects - * the declarations, but a stray response from a model that hallucinates a - * function name would otherwise propagate to dispatch. + * One request and the consumption of its stream. All streaming state lives + * inside, so a retry starts clean. */ - const validatedToolCallsInStreamOrder = (): ToolCall[] => - filterValidToolCalls(toolCallsInStreamOrder(), toolDefinitions); - - const usageCollector = createAnthropicUsageCollector(); - const snapshotUsage = createUsageSnapshotEmitter(emit); - for await (const event of stream) { - usageCollector.observe(event); - snapshotUsage(usageCollector.result()); - maybeEmitAnthropicRefusal(event, emit); - if (event.type === "content_block_start") { - const block = event.content_block; - const index = event.index as number; - if (block.type === "tool_use") { - blockMeta.set(index, { - type: "tool_use", - id: block.id ?? "", - name: block.name ?? "", - json: createPartialJsonStream(), - }); - } else if (block.type === "text") { - blockMeta.set(index, { type: "text" }); - } - } else if (event.type === "content_block_delta") { - const index = event.index as number; - const delta = event.delta as any; - if (delta.type === "text_delta") { - emit({ type: "text-delta", port: "text", textDelta: delta.text }); - } else if (delta.type === "input_json_delta") { + const consume = async (requestParams: any): Promise => { + const stream = client.messages.stream(requestParams, { signal }); + + const blockMeta = new Map(); + const native = new Map(); + /** Keyed by Anthropic content block index — avoids collisions when `id` is missing during early deltas. */ + const toolCallsByBlockIndex = new Map(); + + const toolCallsInStreamOrder = (): ToolCall[] => + [...toolCallsByBlockIndex.entries()].sort((a, b) => a[0] - b[0]).map(([, tc]) => tc); + + /** + * Defence-in-depth: drop tool calls whose name isn't in the effective + * declared tool set (the caller's tools, or the prefix's on fallback) + * before yielding to the consumer. Anthropic's tool API normally respects + * the declarations, but a stray response from a model that hallucinates a + * function name would otherwise propagate to dispatch. + */ + const validatedToolCallsInStreamOrder = (): ToolCall[] => + filterValidToolCalls(toolCallsInStreamOrder(), toolDefinitions); + + /** Cleared by any block or delta the replay cannot reproduce exactly. */ + let verbatim = true; + + const usageCollector = createAnthropicUsageCollector(); + const snapshotUsage = createUsageSnapshotEmitter(emitTracked); + for await (const event of stream) { + usageCollector.observe(event); + snapshotUsage(usageCollector.result()); + maybeEmitAnthropicRefusal(event, emitTracked); + if (event.type === "content_block_start") { + const block = event.content_block; + const index = event.index as number; + if (block.type === "tool_use") { + blockMeta.set(index, { + type: "tool_use", + id: block.id ?? "", + name: block.name ?? "", + json: createPartialJsonStream(), + }); + native.set(index, { + type: "tool_use", + id: block.id ?? "", + name: block.name ?? "", + raw: "", + }); + } else if (block.type === "text") { + blockMeta.set(index, { type: "text" }); + native.set(index, { type: "text", text: "" }); + } else if (block.type === "thinking") { + native.set(index, { + type: "thinking", + thinking: block.thinking ?? "", + signature: block.signature ?? "", + }); + } else if (block.type === "redacted_thinking") { + native.set(index, { type: "redacted_thinking", data: block.data ?? "" }); + } else { + verbatim = false; + } + } else if (event.type === "content_block_delta") { + const index = event.index as number; + const delta = event.delta as any; + const nativeBlock = native.get(index); + if (delta.type === "text_delta") { + if (nativeBlock?.type === "text") nativeBlock.text += delta.text ?? ""; + emitTracked({ type: "text-delta", port: "text", textDelta: delta.text }); + } else if (delta.type === "thinking_delta") { + if (nativeBlock?.type === "thinking") nativeBlock.thinking += delta.thinking ?? ""; + if (delta.thinking) { + emitTracked({ type: "text-delta", port: "reasoning", textDelta: delta.thinking }); + } + } else if (delta.type === "signature_delta") { + if (nativeBlock?.type === "thinking") nativeBlock.signature += delta.signature ?? ""; + } else if (delta.type === "input_json_delta") { + if (nativeBlock?.type === "tool_use") nativeBlock.raw += delta.partial_json ?? ""; + const meta = blockMeta.get(index); + if (meta?.type === "tool_use") { + // `push` returns the parser's live root; `sanitizeToolArgs` copies it, + // so the emitted call is detached from later mutations. + const parsedInput = meta.json.push(delta.partial_json ?? "") ?? {}; + toolCallsByBlockIndex.set(index, { + id: meta.id, + name: meta.name, + input: sanitizeToolArgs(parsedInput) as Record, + }); + emitTracked({ + type: "object-delta", + port: "toolCalls", + objectDelta: validatedToolCallsInStreamOrder(), + }); + } + } else { + verbatim = false; + } + } else if (event.type === "content_block_stop") { + const index = event.index as number; const meta = blockMeta.get(index); if (meta?.type === "tool_use") { - // `push` returns the parser's live root; `sanitizeToolArgs` copies it, - // so the emitted call is detached from later mutations. - const parsedInput = meta.json.push(delta.partial_json ?? "") ?? {}; + const finalInput = meta.json.finishObject(); toolCallsByBlockIndex.set(index, { id: meta.id, name: meta.name, - input: sanitizeToolArgs(parsedInput) as Record, + input: sanitizeToolArgs(finalInput) as Record, }); - emit({ + emitTracked({ type: "object-delta", port: "toolCalls", objectDelta: validatedToolCallsInStreamOrder(), }); } + blockMeta.delete(index); } - } else if (event.type === "content_block_stop") { - const index = event.index as number; - const meta = blockMeta.get(index); - if (meta?.type === "tool_use") { - const finalInput = meta.json.finishObject(); - toolCallsByBlockIndex.set(index, { - id: meta.id, - name: meta.name, - input: sanitizeToolArgs(finalInput) as Record, - }); - emit({ - type: "object-delta", - port: "toolCalls", - objectDelta: validatedToolCallsInStreamOrder(), - }); - } - blockMeta.delete(index); } - } - emit({ - type: "finish", - data: { text: "", toolCalls: [] } as ToolCallingTaskOutput, - usage: usageCollector.result(), - }); + const nativeTurn = verbatim ? serializeNativeTurn(native) : undefined; + if (nativeTurn !== undefined) { + emitTracked({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: nativeKey, payload: nativeTurn }, + }); + } + + emitTracked({ + type: "finish", + data: { text: "", toolCalls: [] } as ToolCallingTaskOutput, + usage: usageCollector.result(), + }); + }; + + try { + await consume(params); + } catch (err) { + // A replayed thinking block whose signature no longer matches is a 400 + // decided before any output; the documented recovery is to send the turn + // without its thinking, once. + if (emitted || !isThinkingSignatureError(err) || !Array.isArray(params.messages)) throw err; + const stripped = withoutThinking(params.messages); + if (stripped === undefined) throw err; + getLogger().warn("Anthropic rejected replayed thinking; retrying without it.", { + model: modelName, + }); + await consume({ ...params, messages: stripped }); + } }; diff --git a/providers/cactus/src/ai/common/Cactus_ParseToolCalls.ts b/providers/cactus/src/ai/common/Cactus_ParseToolCalls.ts index 6b8bbb0ca2..72d1318fef 100644 --- a/providers/cactus/src/ai/common/Cactus_ParseToolCalls.ts +++ b/providers/cactus/src/ai/common/Cactus_ParseToolCalls.ts @@ -55,22 +55,33 @@ function danglingPrefixLength(text: string, tag: string): number { * would leave a host rendering the chain of thought beside the tool cards, and * a host logging `text` persisting it. * + * What the filter drops is handed to `onReasoning` as it arrives, so the + * caller can put it on the `reasoning` port instead of losing it. The tags + * themselves are not part of it. + * * Streaming makes this more than a `replace`: a tag can straddle a delta * boundary, so the filter holds back any trailing run that could still turn - * into `` and releases it once it cannot. + * into `` (or ``, inside a block) and releases it once it + * cannot. * * `flush` exists for the case the fence never closes. A generation cut short - * at the token limit has no ``, and `stripNeedleReasoning` deliberately - * leaves such a block alone — so the filter releases what it held rather than - * swallowing the whole generation. + * at the token limit has no ``: everything after the opening tag is + * still reasoning, so it goes to `onReasoning` — including a partial closing + * tag held back at the end — and none of it reaches the visible text. */ -export function createNeedleReasoningFilter(): NeedleTextFilter { +export function createNeedleReasoningFilter( + onReasoning?: (text: string) => void +): NeedleTextFilter { const OPEN = ""; const CLOSE = ""; - /** Text not yet classifiable: a partial tag, or a fence still open. */ + /** Text not yet classifiable: a partial tag. */ let held = ""; let inside = false; + const think = (text: string): void => { + if (text.length > 0) onReasoning?.(text); + }; + return { push(delta: string): string { held += delta; @@ -78,8 +89,13 @@ export function createNeedleReasoningFilter(): NeedleTextFilter { for (;;) { if (inside) { const close = held.indexOf(CLOSE); - // No closing tag yet: keep holding, in case the stream ends here. - if (close === -1) return visible; + if (close === -1) { + const dangling = danglingPrefixLength(held, CLOSE); + think(held.slice(0, held.length - dangling)); + held = held.slice(held.length - dangling); + return visible; + } + think(held.slice(0, close)); held = held.slice(close + CLOSE.length); inside = false; continue; @@ -92,15 +108,18 @@ export function createNeedleReasoningFilter(): NeedleTextFilter { return visible; } visible += held.slice(0, open); - // Keep the opening tag, so an unterminated block flushes intact. - held = held.slice(open); + held = held.slice(open + OPEN.length); inside = true; } }, flush(): string { const rest = held; held = ""; - inside = false; + if (inside) { + inside = false; + think(rest); + return ""; + } return rest; }, }; @@ -136,8 +155,9 @@ function isBareToolCallPayload(text: string): boolean { * * Three stages, in the order the parser reads the generation: * - * 1. {@link createNeedleReasoningFilter} drops `` blocks, so a fence the - * model only talked about counts for nothing below. + * 1. {@link createNeedleReasoningFilter} drops `` blocks, handing them to + * `onReasoning`, so a fence the model only talked about counts for nothing + * below. * 2. {@link createToolCallMarkupFilter} drops each `` block, holding * back a partial opening tag until it is disambiguated and suppressing an * unclosed block to the end — the parser recovers that tail as a payload. @@ -151,8 +171,10 @@ function isBareToolCallPayload(text: string): boolean { * Output that is nothing but whitespace is dropped at the end too: it is the * newline between `` and ``, not an answer. */ -export function createNeedleVisibleTextFilter(): NeedleTextFilter { - const reasoning = createNeedleReasoningFilter(); +export function createNeedleVisibleTextFilter( + onReasoning?: (text: string) => void +): NeedleTextFilter { + const reasoning = createNeedleReasoningFilter(onReasoning); /** A `` opened outside reasoning, so no bare payload is read. */ let sawFence = false; /** The end of the last routed delta, for an opening tag split across two. */ diff --git a/providers/cactus/src/ai/common/Cactus_ToolCallingCore.ts b/providers/cactus/src/ai/common/Cactus_ToolCallingCore.ts index 5af31e85b0..4d40720218 100644 --- a/providers/cactus/src/ai/common/Cactus_ToolCallingCore.ts +++ b/providers/cactus/src/ai/common/Cactus_ToolCallingCore.ts @@ -31,12 +31,38 @@ function buildToolsJson(tools: ReadonlyArray): string { ); } +/** + * What the engine is asked this round. Needle answers one query with no + * conversation of its own, and each round's input carries the whole history + * beside the turn's prompt, so the question is the latest user turn: sending + * the prompt every round would repeat it after a tool result. A later user turn + * within the same request can lean on the prompt (a review asking the model to + * check its answer against what it was asked), so the prompt goes ahead of it + * whenever the two differ. Earlier requests in the history are left out. + */ function promptText(input: ToolCallingTaskInput): string { - if (typeof input.prompt === "string") return input.prompt; - if (input.prompt) return extractMessageText(input.prompt); - if (input.messages && input.messages.length > 0) { - const last = input.messages[input.messages.length - 1]; - return extractMessageText(last.content); + const query = queryText(input); + return input.systemPrompt ? `${input.systemPrompt}\n\n${query}` : query; +} + +function queryText(input: ToolCallingTaskInput): string { + const prompt = + typeof input.prompt === "string" + ? input.prompt + : input.prompt + ? extractMessageText(input.prompt) + : ""; + const messages = input.messages; + if (messages && messages.length > 0) { + for (let i = messages.length - 1; i >= 0; i--) { + if (messages[i].role !== "user") continue; + const latest = extractMessageText(messages[i].content); + return prompt !== "" && latest !== prompt ? `${prompt}\n\n${latest}` : latest; + } + } + if (prompt !== "") return prompt; + if (messages && messages.length > 0) { + return extractMessageText(messages[messages.length - 1].content); } return ""; } @@ -76,7 +102,9 @@ export function createCactusToolCalling( // One definition of "what the caller sees as text", so the streamed and // one-shot paths cannot disagree about it. - const visible = createNeedleVisibleTextFilter(); + const visible = createNeedleVisibleTextFilter((chunk) => { + emit({ type: "text-delta", port: "reasoning", textDelta: chunk }); + }); const emitVisible = (chunk: string): void => { if (chunk.length > 0) emit({ type: "text-delta", port: "text", textDelta: chunk }); }; diff --git a/providers/chrome-ai/src/ai/common/WebBrowser_ToolCalling.ts b/providers/chrome-ai/src/ai/common/WebBrowser_ToolCalling.ts index de4ab0041d..aac1e4f55e 100644 --- a/providers/chrome-ai/src/ai/common/WebBrowser_ToolCalling.ts +++ b/providers/chrome-ai/src/ai/common/WebBrowser_ToolCalling.ts @@ -197,7 +197,7 @@ export const WebBrowser_ToolCalling: AiProviderRunFn< if (validated.length > 0) { emit({ type: "object-delta", port: "toolCalls", objectDelta: validated }); } - emit({ type: "finish", data: {} as ToolCallingTaskOutput }); + emit({ type: "finish", data: { text: "", toolCalls: [] } as ToolCallingTaskOutput }); } finally { try { session.destroy(); diff --git a/providers/deepseek/src/ai/common/DeepSeek_Client.ts b/providers/deepseek/src/ai/common/DeepSeek_Client.ts index 0616792054..b3f7ac855d 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_Client.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_Client.ts @@ -122,6 +122,34 @@ export function getModelName(model: DeepSeekModelConfig | undefined): string { return name; } +/** + * DeepSeek's names for the shared effort levels. `none` is not a level here: + * it turns thinking off, which DeepSeek spells as a separate switch. + */ +const EFFORT_TO_REASONING_EFFORT: Record, string> = { + low: "low", + medium: "medium", + high: "high", + extra: "xhigh", + ultra: "max", +}; + +/** + * The request fields that carry `model.effort` to DeepSeek. Without them the + * effort a caller set only sized `max_tokens`, and the model reasoned at its own + * default whatever was asked. Nothing is sent for a model whose policy has no + * reasoning, or when no effort is set. + */ +export function reasoningParams(model: DeepSeekModelConfig | undefined): { + reasoning_effort?: string; + thinking?: { type: "disabled" }; +} { + const effort = resolveEnabledEffort(model, deepseekEffortPolicy(model)); + if (effort === undefined) return {}; + if (effort === "none") return { thinking: { type: "disabled" } }; + return { reasoning_effort: EFFORT_TO_REASONING_EFFORT[effort] }; +} + /** * Reasoning headroom added when the model sets no `reasoning_allowance`. Sized * above the ~4k reasoning tokens the thinking models typically spend. diff --git a/providers/deepseek/src/ai/common/DeepSeek_ModelSchema.ts b/providers/deepseek/src/ai/common/DeepSeek_ModelSchema.ts index d2e57d2161..476bbfe7be 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_ModelSchema.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_ModelSchema.ts @@ -48,6 +48,12 @@ export const DeepSeekModelSchema = { description: "Tokens added to the answer budget to leave room for reasoning. Set 0 for a non-thinking model; omit for the default.", }, + replay_reasoning: { + type: "boolean", + description: + "Send each assistant turn's reasoning back on later rounds of a tool-calling loop (default true). Off saves the prompt tokens it costs, at the price of the model working its plan out again each round.", + "x-ui-hidden": true, + }, }, required: ["model_name"], additionalProperties: false, diff --git a/providers/deepseek/src/ai/common/DeepSeek_StructuredGeneration.ts b/providers/deepseek/src/ai/common/DeepSeek_StructuredGeneration.ts index 4054aaae50..4d9c9defc0 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_StructuredGeneration.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_StructuredGeneration.ts @@ -21,6 +21,7 @@ import { getClient, getModelName, resolveMaxTokens, + reasoningParams, } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -62,6 +63,7 @@ export const DeepSeek_StructuredGeneration_Stream: AiProviderRunFn< messages: [{ role: "user", content: userContent }], response_format: responseFormat as never, max_tokens: resolveMaxTokens(model, input.maxTokens), + ...reasoningParams(model), temperature: input.temperature, stream: true, ...OPENAI_STREAM_USAGE_OPTIONS, diff --git a/providers/deepseek/src/ai/common/DeepSeek_TextGeneration.ts b/providers/deepseek/src/ai/common/DeepSeek_TextGeneration.ts index adc2282381..d06c951b20 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_TextGeneration.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_TextGeneration.ts @@ -21,6 +21,7 @@ import { getClient, getModelName, resolveMaxTokens, + reasoningParams, } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -62,6 +63,7 @@ function buildChatParams( }; const resolvedMaxTokens = resolveMaxTokens(model, input.maxTokens); if (resolvedMaxTokens !== undefined) params.max_tokens = resolvedMaxTokens; + Object.assign(params, reasoningParams(model)); if (input.temperature !== undefined) params.temperature = input.temperature; if ((input as { topP?: number }).topP !== undefined) params.top_p = (input as { topP?: number }).topP; diff --git a/providers/deepseek/src/ai/common/DeepSeek_TextRewriter.ts b/providers/deepseek/src/ai/common/DeepSeek_TextRewriter.ts index 8ab702687a..3fbc2bd757 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_TextRewriter.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_TextRewriter.ts @@ -14,7 +14,7 @@ import { createEstimatedOutputUsageReporter, OPENAI_STREAM_USAGE_OPTIONS, } from "@workglow/ai/provider-utils"; -import { getClient, getModelName } from "./DeepSeek_Client"; +import { getClient, getModelName, reasoningParams } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -42,6 +42,7 @@ export const DeepSeek_TextRewriter_Stream: AiProviderRunFn< { role: "system", content: input.prompt }, { role: "user", content: input.text }, ], + ...reasoningParams(model), stream: true, ...OPENAI_STREAM_USAGE_OPTIONS, }, diff --git a/providers/deepseek/src/ai/common/DeepSeek_TextSummary.ts b/providers/deepseek/src/ai/common/DeepSeek_TextSummary.ts index 923c48ad2b..08834f2ccb 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_TextSummary.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_TextSummary.ts @@ -14,7 +14,7 @@ import { createEstimatedOutputUsageReporter, OPENAI_STREAM_USAGE_OPTIONS, } from "@workglow/ai/provider-utils"; -import { getClient, getModelName } from "./DeepSeek_Client"; +import { getClient, getModelName, reasoningParams } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -42,6 +42,7 @@ export const DeepSeek_TextSummary_Stream: AiProviderRunFn< { role: "system", content: "Summarize the following text concisely." }, { role: "user", content: input.text }, ], + ...reasoningParams(model), stream: true, ...OPENAI_STREAM_USAGE_OPTIONS, }, diff --git a/providers/deepseek/src/ai/common/DeepSeek_ToolCalling.ts b/providers/deepseek/src/ai/common/DeepSeek_ToolCalling.ts index ba84fcf971..946a0db542 100644 --- a/providers/deepseek/src/ai/common/DeepSeek_ToolCalling.ts +++ b/providers/deepseek/src/ai/common/DeepSeek_ToolCalling.ts @@ -17,7 +17,7 @@ import { } from "@workglow/ai/provider-utils"; import { filterValidToolCalls, toOpenAIMessages } from "@workglow/ai/worker"; import { RetryableJobError } from "@workglow/job-queue"; -import { getClient, getModelName, resolveMaxTokens } from "./DeepSeek_Client"; +import { getClient, getModelName, reasoningParams, resolveMaxTokens } from "./DeepSeek_Client"; import type { DeepSeekModelConfig } from "./DeepSeek_ModelSchema"; import { mapDeepSeekUsage } from "./DeepSeek_Usage"; @@ -125,7 +125,12 @@ export const DeepSeek_ToolCalling_Stream: AiProviderRunFn< const modelName = getModelName(model); const tools = buildOpenAITools(input.tools); - const messages = toOpenAIMessages(input); + // DeepSeek's thinking models read their reasoning back on each turn of a + // tool-calling loop, and an image a tool returned has to ride a user message. + const messages = toOpenAIMessages(input, { + replayReasoning: model?.provider_config?.replay_reasoning !== false, + toolImagesInUserMessage: true, + }); const toolChoice = mapDeepSeekToolChoice(input.toolChoice); const stream = await client.chat.completions.create( @@ -133,6 +138,7 @@ export const DeepSeek_ToolCalling_Stream: AiProviderRunFn< model: modelName, messages, max_tokens: resolveMaxTokens(model, input.maxTokens), + ...reasoningParams(model), temperature: input.temperature, stream: true, tools, diff --git a/providers/deepseek/src/ai/index.ts b/providers/deepseek/src/ai/index.ts index 2134f94893..37ebca6f6b 100644 --- a/providers/deepseek/src/ai/index.ts +++ b/providers/deepseek/src/ai/index.ts @@ -11,6 +11,7 @@ export { DEEPSEEK_ALLOWED_HOSTS, DEEPSEEK_DEFAULT_BASE_URL, DEEPSEEK_DEFAULT_REASONING_ALLOWANCE, + reasoningParams, resolveMaxTokens, } from "./common/DeepSeek_Client"; export * from "./common/DeepSeek_Constants"; diff --git a/providers/google-gemini/src/ai/common/Gemini_CacheCheckpoint.ts b/providers/google-gemini/src/ai/common/Gemini_CacheCheckpoint.ts index 88d3d1d5b0..8c9bd8b492 100644 --- a/providers/google-gemini/src/ai/common/Gemini_CacheCheckpoint.ts +++ b/providers/google-gemini/src/ai/common/Gemini_CacheCheckpoint.ts @@ -21,6 +21,7 @@ import { createGeminiClient, getModelName } from "./Gemini_Client"; import type { GeminiModelConfig } from "./Gemini_ModelSchema"; import { sanitizeSchemaForGemini } from "./Gemini_Schema"; import { buildGeminiContents } from "./Gemini_ToolCalling"; +import type { IGeminiContentsOptions } from "./Gemini_ToolCalling"; /** Default TTL for explicit cached content — Gemini bills storage per token-hour. */ const GEMINI_CACHE_TTL = "3600s"; @@ -229,11 +230,12 @@ function canonicalSortKey(value: unknown): string { export function buildGeminiPrefixedContents( prefix: CheckpointPrefix, messages: ReadonlyArray | undefined, - prompt: unknown + prompt: unknown, + options: IGeminiContentsOptions = {} ): any[] { const tail: ChatMessage[] = messages && messages.length > 0 ? [...messages] : promptToTailMessages(prompt); - return buildGeminiContents([...(prefix.messages ?? []), ...tail], ""); + return buildGeminiContents([...(prefix.messages ?? []), ...tail], "", options); } /** diff --git a/providers/google-gemini/src/ai/common/Gemini_Client.ts b/providers/google-gemini/src/ai/common/Gemini_Client.ts index 6fddf42433..6329d5c544 100644 --- a/providers/google-gemini/src/ai/common/Gemini_Client.ts +++ b/providers/google-gemini/src/ai/common/Gemini_Client.ts @@ -117,6 +117,28 @@ export function getModelName(model: GeminiModelConfig | undefined): string { return name; } +/** + * Whether the model accepts images inside `functionResponse.parts`. Only the + * Gemini 3 series and later do; older ids answer such a request with a 400. + * Aliases such as `gemini-flash-latest` do not match and take the portable shape. + */ +export function geminiSupportsMultimodalFunctionResponses(modelName: string): boolean { + return /^gemini-(?:[3-9]|\d{2,})(?:[.-]|$)/i.test(modelName); +} + +/** + * Whether the model returns thought summaries: Gemini 2.5 and later, and the + * `-latest` aliases, which point at them. Earlier ids do not think and reject + * `includeThoughts`. + */ +export function geminiReturnsThoughts(modelName: string | undefined): boolean { + if (!modelName) return false; + return ( + /^gemini-(?:2\.5|[3-9]|\d{2,})(?:[.-]|$)/i.test(modelName) || + /^gemini-[a-z-]*-latest$/i.test(modelName) + ); +} + /** * Reasoning-token budget for thinking models. Native * `provider_config.thinking_budget` wins; otherwise map `model.effort`. @@ -158,15 +180,30 @@ export function getGeminiSeed(model: GeminiModelConfig | undefined): number | un export function resolveThinkingConfig( model: GeminiModelConfig | undefined, maxTokens: number | undefined, - defaultBudget?: number + defaultBudget?: number, + options?: { readonly includeThoughts?: boolean | undefined } ): { - thinkingConfig: { thinkingBudget: number } | undefined; + thinkingConfig: { thinkingBudget?: number; includeThoughts?: boolean } | undefined; maxOutputTokens: number | undefined; } { const budget = getThinkingBudget(model) ?? defaultBudget; - if (budget === undefined) { + // Thought summaries exist only when the model thinks: a zero budget turns + // thinking off, and models before Gemini 2.5 reject the config. + const thinks = + budget !== 0 && + geminiEffortPolicy(model).supported.length > 0 && + geminiReturnsThoughts(model?.provider_config?.model_name); + const includeThoughts = options?.includeThoughts === true && thinks; + if (budget === undefined && !includeThoughts) { return { thinkingConfig: undefined, maxOutputTokens: maxTokens }; } - const maxOutputTokens = maxTokens !== undefined && budget > 0 ? maxTokens + budget : maxTokens; - return { thinkingConfig: { thinkingBudget: budget }, maxOutputTokens }; + const maxOutputTokens = + maxTokens !== undefined && budget !== undefined && budget > 0 ? maxTokens + budget : maxTokens; + return { + thinkingConfig: { + ...(budget !== undefined ? { thinkingBudget: budget } : {}), + ...(includeThoughts ? { includeThoughts: true } : {}), + }, + maxOutputTokens, + }; } diff --git a/providers/google-gemini/src/ai/common/Gemini_ToolCalling.ts b/providers/google-gemini/src/ai/common/Gemini_ToolCalling.ts index 3664aecb3a..feef096799 100644 --- a/providers/google-gemini/src/ai/common/Gemini_ToolCalling.ts +++ b/providers/google-gemini/src/ai/common/Gemini_ToolCalling.ts @@ -24,6 +24,7 @@ import { generateGeminiStreamWithCacheFallback } from "./Gemini_CachedContentFal import { evictIfStaleGeminiCachedContent, getGeminiCachedContent } from "./Gemini_CacheStore"; import { createGeminiClient, + geminiSupportsMultimodalFunctionResponses, getGeminiSeed, getModelName, resolveThinkingConfig, @@ -32,9 +33,20 @@ import type { GeminiModelConfig } from "./Gemini_ModelSchema"; import { emitGeminiRefusal, geminiRefusalCategory } from "./Gemini_Refusal"; import { mapGeminiUsage } from "./Gemini_Usage"; +export interface IGeminiContentsOptions { + /** + * Whether the model takes images inside `functionResponse.parts`. When it does + * not, a tool message's images go in their own user turn after its function + * responses, behind a label. Images beside the response in the same turn made + * gemini-2.5-flash write its thinking into the answer text. + */ + readonly multimodalFunctionResponses?: boolean | undefined; +} + export function buildGeminiContents( messages: ReadonlyArray | undefined, - prompt: unknown + prompt: unknown, + options: IGeminiContentsOptions = {} ): any[] { if (!messages || messages.length === 0) { return [{ role: "user", parts: [{ text: prompt }] }]; @@ -50,7 +62,20 @@ export function buildGeminiContents( const toolUseNames = new Map(); const contents: any[] = []; + // Images a tool returned wait until every result of the run of tool messages + // is out: a user turn between two results of one model turn would leave a + // call unanswered before it. + let deferredImages: any[] = []; + const flushDeferredImages = (): void => { + if (deferredImages.length === 0) return; + contents.push({ + role: "user", + parts: [{ text: "Images returned by the tool calls above:" }, ...deferredImages], + }); + deferredImages = []; + }; for (const msg of messages) { + if (msg.role !== "tool") flushDeferredImages(); if (msg.role === "user") { const parts: any[] = []; for (const block of msg.content) { @@ -87,21 +112,46 @@ export function buildGeminiContents( for (const block of msg.content) { if (block.type !== "tool_result") continue; const name = toolUseNames.get(block.tool_use_id) ?? "unknown"; - const textContent = block.content + const text = block.content .filter((b) => b.type === "text") .map((b) => (b as { type: "text"; text: string }).text) .join(""); + const images = block.content.filter( + (b): b is Extract => b.type === "image" + ); let response: Record; - try { - response = JSON.parse(textContent); - } catch { - response = { result: textContent }; + if (block.is_error) { + response = { error: text }; + } else { + let parsed: unknown = text; + try { + parsed = JSON.parse(text); + } catch { + // Not JSON: sent as the text it is. + } + // `response` must be an object; a scalar, array or null result is wrapped. + response = + parsed !== null && typeof parsed === "object" && !Array.isArray(parsed) + ? (parsed as Record) + : { result: parsed }; } - parts.push({ functionResponse: { name, response } }); + const functionResponse: Record = { name, response }; + const imageParts = images.map((image) => ({ + inlineData: { mimeType: image.mimeType, data: image.data }, + })); + if (imageParts.length > 0) { + if (options.multimodalFunctionResponses === true) { + functionResponse.parts = imageParts; + } else { + deferredImages.push(...imageParts); + } + } + parts.push({ functionResponse }); } if (parts.length > 0) contents.push({ role: "user", parts }); } } + flushDeferredImages(); return contents; } @@ -134,6 +184,10 @@ export const Gemini_ToolCalling_Stream: AiProviderRunFn< > = async (input, model, signal, emit, _outputSchema, sessionContext) => { const ai = await createGeminiClient(model); + const contentOptions: IGeminiContentsOptions = { + multimodalFunctionResponses: geminiSupportsMultimodalFunctionResponses(getModelName(model)), + }; + const functionDeclarations = buildGeminiFunctionDeclarations(input.tools); const toolConfig = mapGeminiToolConfig(input.toolChoice); @@ -186,12 +240,17 @@ export const Gemini_ToolCalling_Stream: AiProviderRunFn< // Thinking is opt-in here (no default budget): the model uses its own default // reasoning unless `provider_config.thinking_budget` is set, in which case the // output cap is padded so reasoning can't starve the tool call / answer. - const { thinkingConfig, maxOutputTokens } = resolveThinkingConfig(model, input.maxTokens); + const { thinkingConfig, maxOutputTokens } = resolveThinkingConfig( + model, + input.maxTokens, + undefined, + { includeThoughts: true } + ); /** Build the tail-only request that references the CachedContent handle. */ const buildCachedRequest = (): Record => ({ model: getModelName(model), - contents: buildGeminiContents(input.messages, input.prompt), + contents: buildGeminiContents(input.messages, input.prompt, contentOptions), config: { abortSignal: signal ?? undefined, systemInstruction: undefined, @@ -206,8 +265,8 @@ export const Gemini_ToolCalling_Stream: AiProviderRunFn< /** Build the full inline-replay request (prefix messages + tail + tools). */ const buildInlineReplayRequest = (): Record => { const contents = prefix - ? buildGeminiPrefixedContents(prefix, input.messages, input.prompt) - : buildGeminiContents(input.messages, input.prompt); + ? buildGeminiPrefixedContents(prefix, input.messages, input.prompt, contentOptions) + : buildGeminiContents(input.messages, input.prompt, contentOptions); const systemInstruction = input.systemPrompt || (prefix ? prefix.systemPrompt : undefined); return { model: getModelName(model), @@ -246,7 +305,9 @@ export const Gemini_ToolCalling_Stream: AiProviderRunFn< refusalCategory = refusalCategory ?? geminiRefusalCategory(chunk); const parts = chunk.candidates?.[0]?.content?.parts ?? []; for (const part of parts) { - if (part.text && !part.thought) { + if (part.text && part.thought) { + emit({ type: "text-delta", port: "reasoning", textDelta: part.text }); + } else if (part.text) { emit({ type: "text-delta", port: "text", textDelta: part.text }); } if (part.functionCall) { diff --git a/providers/google-gemini/src/ai/index.ts b/providers/google-gemini/src/ai/index.ts index cc0f12be70..21650bd9e7 100644 --- a/providers/google-gemini/src/ai/index.ts +++ b/providers/google-gemini/src/ai/index.ts @@ -19,7 +19,11 @@ import { generateGeminiStreamWithCacheFallback, isGeminiCachedContentNotFoundError, } from "./common/Gemini_CachedContentFallback"; -import { _testOnly as clientTestOnly, resolveThinkingConfig } from "./common/Gemini_Client"; +import { + _testOnly as clientTestOnly, + geminiSupportsMultimodalFunctionResponses, + resolveThinkingConfig, +} from "./common/Gemini_Client"; import { _cacheStoreTestOnly, getGeminiCachedContent, @@ -42,6 +46,7 @@ export const _testOnly = { GEMINI_RUN_FN_SPECS, GEMINI_RUN_FNS, buildGeminiContents, + geminiSupportsMultimodalFunctionResponses, geminiRefusalCategory, emitGeminiRefusal, setGeminiClientForTests: clientTestOnly.setGeminiClientForTests, diff --git a/providers/huggingface-inference/src/ai/HfInferenceProvider.ts b/providers/huggingface-inference/src/ai/HfInferenceProvider.ts index 7302329b21..b9ad353a71 100644 --- a/providers/huggingface-inference/src/ai/HfInferenceProvider.ts +++ b/providers/huggingface-inference/src/ai/HfInferenceProvider.ts @@ -5,13 +5,14 @@ */ import { createCloudProviderClass } from "@workglow/ai/provider-utils"; -import type { Capability, ModelRecord } from "@workglow/ai/worker"; +import type { Capability, ModelEffortPolicy, ModelRecord } from "@workglow/ai/worker"; import { AiProvider } from "@workglow/ai/worker"; import { hfInferenceWorkerRunFnSpecs, inferHfInferenceCapabilities, } from "./common/HFI_Capabilities"; import { HF_INFERENCE } from "./common/HFI_Constants"; +import { hfiEffortPolicy } from "./common/HFI_EffortPolicy"; import type { HfInferenceModelConfig } from "./common/HFI_ModelSchema"; /** @@ -29,6 +30,10 @@ export class HfInferenceProvider extends createCloudProviderClass + * SPDX-License-Identifier: Apache-2.0 + */ + +import { + EFFORT_POLICY_ALL, + EFFORT_POLICY_NONE, + makeEffortPolicy, + resolveEnabledEffort, + type ModelEffort, + type ModelEffortPolicyFn, +} from "@workglow/ai/worker"; +import type { HfInferenceModelConfig } from "./HFI_ModelSchema"; + +/** Ids on the router that are not chat models, so take no reasoning dial. */ +const NON_TEXT = [ + /embed/i, + /(?:^|\/)(?:bge|gte|e5)-/i, + /rerank/i, + /stable-diffusion|flux|sdxl/i, + /(?:^|[/-])(?:whisper|tts)/i, +]; + +/** + * The router forwards to many backends and drops nothing on its own, so an id + * no rule recognizes keeps the dial; the field is sent only when a caller set + * an effort. + */ +export const hfiEffortPolicy: ModelEffortPolicyFn = makeEffortPolicy({ + rules: [{ when: NON_TEXT, policy: EFFORT_POLICY_NONE }], + fallback: EFFORT_POLICY_ALL, +}); + +const EFFORT_TO_REASONING_EFFORT: Record, string> = { + low: "low", + medium: "medium", + high: "high", + extra: "high", + ultra: "high", +}; + +/** The chat-completions `reasoning_effort` field for a model's effort, or nothing. */ +export function hfiReasoningParams(model: HfInferenceModelConfig | undefined): { + reasoning_effort?: string; +} { + const effort = resolveEnabledEffort(model, hfiEffortPolicy(model)); + if (effort === undefined || effort === "none") return {}; + return { reasoning_effort: EFFORT_TO_REASONING_EFFORT[effort] }; +} diff --git a/providers/huggingface-inference/src/ai/common/HFI_TextGeneration.ts b/providers/huggingface-inference/src/ai/common/HFI_TextGeneration.ts index fd61b539aa..12fad774f7 100644 --- a/providers/huggingface-inference/src/ai/common/HFI_TextGeneration.ts +++ b/providers/huggingface-inference/src/ai/common/HFI_TextGeneration.ts @@ -16,6 +16,7 @@ import { } from "@workglow/ai/provider-utils"; import { toOpenAIMessages } from "@workglow/ai/worker"; import { getClient, getModelName, getProvider } from "./HFI_Client"; +import { hfiReasoningParams } from "./HFI_EffortPolicy"; import type { HfInferenceModelConfig } from "./HFI_ModelSchema"; /** @@ -76,6 +77,7 @@ export const HFI_TextGeneration_Stream: AiProviderRunFn< top_p: input.topP, frequency_penalty: input.frequencyPenalty, provider, + ...hfiReasoningParams(model), }, { signal } ); diff --git a/providers/huggingface-inference/src/ai/common/HFI_ToolCalling.ts b/providers/huggingface-inference/src/ai/common/HFI_ToolCalling.ts index becef88d9a..26563eda1f 100644 --- a/providers/huggingface-inference/src/ai/common/HFI_ToolCalling.ts +++ b/providers/huggingface-inference/src/ai/common/HFI_ToolCalling.ts @@ -14,6 +14,7 @@ import { } from "@workglow/ai/provider-utils"; import { toOpenAIMessages } from "@workglow/ai/worker"; import { getClient, getModelName, getProvider } from "./HFI_Client"; +import { hfiReasoningParams } from "./HFI_EffortPolicy"; import type { HfInferenceModelConfig } from "./HFI_ModelSchema"; export const HFI_ToolCalling_Stream: AiProviderRunFn< @@ -26,7 +27,7 @@ export const HFI_ToolCalling_Stream: AiProviderRunFn< const provider = getProvider(model); const tools = buildOpenAITools(input.tools); - const messages = toOpenAIMessages(input); + const messages = toOpenAIMessages(input, { toolImagesInUserMessage: true }); const toolChoice = mapOpenAIToolChoice(input.toolChoice, false); const promptText = messages .map((m) => (typeof m.content === "string" ? m.content : "")) @@ -40,6 +41,7 @@ export const HFI_ToolCalling_Stream: AiProviderRunFn< temperature: input.temperature, provider, stream: true, + ...hfiReasoningParams(model), }; if (toolChoice !== "none") { params.tools = tools; diff --git a/providers/huggingface-inference/src/ai/index.ts b/providers/huggingface-inference/src/ai/index.ts index 0793d28295..8b7e812ec6 100644 --- a/providers/huggingface-inference/src/ai/index.ts +++ b/providers/huggingface-inference/src/ai/index.ts @@ -8,6 +8,7 @@ export * from "./common/HFI_AspectRatio"; export * from "./common/HFI_Constants"; +export * from "./common/HFI_EffortPolicy"; export * from "./common/HFI_ImageValidation"; export * from "./common/HFI_ModelSchema"; export * from "./common/HFI_ModelSearch"; diff --git a/providers/huggingface-inference/src/ai/runtime.ts b/providers/huggingface-inference/src/ai/runtime.ts index 1d96fa057b..6bf8d23573 100644 --- a/providers/huggingface-inference/src/ai/runtime.ts +++ b/providers/huggingface-inference/src/ai/runtime.ts @@ -14,5 +14,6 @@ // organize-imports-ignore export * from "./common/HFI_Client"; +export * from "./common/HFI_EffortPolicy"; export * from "./registerHfInferenceInline"; export * from "./registerHfInferenceWorker"; diff --git a/providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts b/providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts index a6de439372..2b36942ee1 100644 --- a/providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts +++ b/providers/huggingface-transformers/src/ai/common/HFT_ToolCalling.ts @@ -16,6 +16,7 @@ import type { } from "@workglow/ai"; import { adaptParserResult, + createThinkTagFilter, createToolCallMarkupFilter, forcedToolSelection, getAvailableParsers, @@ -156,7 +157,9 @@ function resolveHFTToolsAndMessages( * Unlike `toTextFlatMessages` (which flattens everything to `{role, content}` * strings), this preserves tool_calls on assistant messages and the tool name * on tool-result messages — both required by HFT chat templates that support - * tool calling. + * tool calling. An assistant turn's reasoning rides as `reasoning_content`, + * which Qwen3-style templates read to keep the thinking of the current tool + * loop in the rendered history; templates that do not know it ignore it. */ export function buildHFTMessages( messages: ReadonlyArray | undefined, @@ -178,6 +181,9 @@ export function buildHFTMessages( out.push({ role: "user", content: extractPromptText(prompt) }); return out; } + // A tool_result names only the call it answers, so the tool's name is + // resolved from the tool_use that raised it (most recent id wins). + const toolNames = new Map(); for (const msg of messages) { if (msg.role === "user") { const text = msg.content @@ -201,7 +207,13 @@ export function buildHFTMessages( }; return { id: tu.id, name: tu.name, arguments: tu.input }; }); + for (const call of toolCalls) toolNames.set(call.id, call.name); + const reasoning = msg.content + .filter((b) => b.type === "reasoning") + .map((b) => (b as { type: "reasoning"; text: string }).text) + .join(""); const entry: Record = { role: "assistant", content: text }; + if (reasoning.length > 0) entry.reasoning_content = reasoning; if (toolCalls.length > 0) entry.tool_calls = toolCalls; out.push(entry); } else if (msg.role === "tool") { @@ -211,10 +223,12 @@ export function buildHFTMessages( .filter((inner) => inner.type === "text") .map((inner) => (inner as { type: "text"; text: string }).text) .join(""); + const name = toolNames.get(b.tool_use_id); out.push({ role: "tool", content: text, tool_call_id: b.tool_use_id, + ...(name !== undefined ? { name } : {}), }); } } @@ -267,6 +281,21 @@ function hasToolMessages(input: ToolCallingTaskInput): boolean { return input.messages?.some((m) => m.role === "tool") ?? false; } +/** + * Whether the history needs {@link buildHFTMessages} rather than flat text: + * tool messages, or an assistant turn carrying reasoning, which only the + * structured messages hand to the template as `reasoning_content`. + */ +function needsStructuredMessages(input: ToolCallingTaskInput): boolean { + return ( + hasToolMessages(input) || + (input.messages?.some( + (m) => m.role === "assistant" && m.content.some((b) => b.type === "reasoning" && b.text) + ) ?? + false) + ); +} + function buildPromptAndPrefix( tokenizer: TextGenerationPipeline["tokenizer"], input: ToolCallingTaskInput, @@ -274,9 +303,9 @@ function buildPromptAndPrefix( ): { prompt: string; responsePrefix: string | undefined } { let basePrompt: string; - if (hasToolMessages(input)) { - // Multi-turn with tool results: use structured messages so the tokenizer - // can format tool_calls and tool responses correctly. + if (needsStructuredMessages(input)) { + // Multi-turn with tool results or reasoning: use structured messages so the + // tokenizer can format tool_calls, tool responses and reasoning correctly. const messages = buildHFTMessages( input.messages, input.systemPrompt, @@ -290,7 +319,7 @@ function buildPromptAndPrefix( add_generation_prompt: true, }) as string; } else { - // Single-turn or no tool results: flat messages work fine. + // No tool results and no reasoning: flat messages lose nothing. const messages = toTextFlatMessages(input); const tools = resolveHFTToolsAndMessages(input, messages); basePrompt = tokenizer.apply_chat_template(messages, { @@ -374,6 +403,52 @@ function buildCheckpointPromptAndPrefix( }; } +// ============================================================================ +// Streamed delta routing +// ============================================================================ + +/** + * Routes decoded deltas to the ports: `` content goes to `reasoning`, + * and what remains goes through the `` markup filter to `text`. + * Thinking is separated first, so a tool-call tag quoted inside it is not + * mistaken for markup and neither is it recorded as the answer. + * + * `answerText` is the text-port content before the markup filter — what the + * tool-call parser must read, without the model's thinking. + */ +export function createHftDeltaRouter( + emit: (event: { type: "text-delta"; port: "text" | "reasoning"; textDelta: string }) => void, + startInside: boolean +): { + readonly feed: (delta: string) => void; + readonly flush: () => void; + readonly answerText: () => string; +} { + let answer = ""; + const markup = createToolCallMarkupFilter((text) => { + emit({ type: "text-delta", port: "text", textDelta: text }); + }); + const think = createThinkTagFilter({ startInside }); + const route = (parts: ReturnType): void => { + for (const part of parts) { + if (part.port === "reasoning") { + emit({ type: "text-delta", port: "reasoning", textDelta: part.text }); + } else { + answer += part.text; + markup.feed(part.text); + } + } + }; + return { + feed: (delta) => route(think.push(delta)), + flush: () => { + route(think.flush()); + markup.flush(); + }, + answerText: () => answer, + }; +} + // ============================================================================ // Provider run functions // ============================================================================ @@ -425,12 +500,15 @@ export const HFT_ToolCalling: AiProviderRunFn< } const { prompt, responsePrefix } = promptParts; - // Accumulate raw tokens for post-hoc tool-call parsing, and feed each - // delta through a markup filter that emits cleaned text-delta events. + // Accumulate raw tokens (the KV cache encodes them verbatim), and route + // each delta: thinking to `reasoning`, the rest through the tool-call + // markup filter to `text`. A template that opens `` in the prompt + // leaves the generation starting inside the block. let fullText = ""; - const filter = createToolCallMarkupFilter((text) => { - emit({ type: "text-delta", port: "text", textDelta: text }); - }); + const filter = createHftDeltaRouter( + (event) => emit(event), + prompt.trimEnd().endsWith("") + ); const streamer = createStreamingTextStreamer( generateText.tokenizer, @@ -517,7 +595,9 @@ export const HFT_ToolCalling: AiProviderRunFn< // Parse the accumulated text for tool calls using the model-family-aware parser. // For models that use a generation prefix, prepend it so the parser sees the // full markup pattern. - const parseableFullText = responsePrefix ? `${responsePrefix}${fullText}` : fullText; + // Thinking is left out: a call quoted inside it was never made. + const answerText = filter.answerText(); + const parseableFullText = responsePrefix ? `${responsePrefix}${answerText}` : answerText; // Only the calls are taken: the parser's cleaned text is the same text the // markup filter already streamed, and the stream is the one copy. const { toolCalls } = adaptParserResult( diff --git a/providers/llamacpp-server/src/ai/common/LlamaCppServer_Client.ts b/providers/llamacpp-server/src/ai/common/LlamaCppServer_Client.ts index 1e4af70e1e..97370e3a83 100644 --- a/providers/llamacpp-server/src/ai/common/LlamaCppServer_Client.ts +++ b/providers/llamacpp-server/src/ai/common/LlamaCppServer_Client.ts @@ -102,6 +102,8 @@ const noopRelease = async (): Promise => {}; /** One parsed delta from an OpenAI-compatible `/v1/chat/completions` stream. */ export interface IChatCompletionDelta { readonly contentDelta?: string; + /** The model's thinking, streamed as `delta.reasoning_content` by a reasoning model. */ + readonly reasoningDelta?: string | undefined; readonly toolCallDeltas?: ReadonlyArray<{ readonly index?: number; readonly id?: string; @@ -159,6 +161,7 @@ export async function* readChatCompletionDeltas( choices?: Array<{ delta?: { content?: string; + reasoning_content?: string; tool_calls?: IChatCompletionDelta["toolCallDeltas"]; }; finish_reason?: string; @@ -172,6 +175,7 @@ export async function* readChatCompletionDeltas( } const choice = chunk.choices?.[0]; const contentDelta = choice?.delta?.content; + const reasoningDelta = choice?.delta?.reasoning_content; const toolCallDeltas = choice?.delta?.tool_calls; const finishReason = choice?.finish_reason; // The usage-bearing chunk carries an empty `choices` array, so it must @@ -179,11 +183,12 @@ export async function* readChatCompletionDeltas( const usage = chunk.usage ?? undefined; if ( contentDelta !== undefined || + reasoningDelta !== undefined || toolCallDeltas !== undefined || finishReason || usage !== undefined ) { - yield { contentDelta, toolCallDeltas, finishReason, usage }; + yield { contentDelta, reasoningDelta, toolCallDeltas, finishReason, usage }; } } } diff --git a/providers/llamacpp-server/src/ai/common/LlamaCppServer_TextGeneration.ts b/providers/llamacpp-server/src/ai/common/LlamaCppServer_TextGeneration.ts index b0654e4539..97209468df 100644 --- a/providers/llamacpp-server/src/ai/common/LlamaCppServer_TextGeneration.ts +++ b/providers/llamacpp-server/src/ai/common/LlamaCppServer_TextGeneration.ts @@ -6,10 +6,13 @@ import type { AiProviderRunFn, + LooseChatMessage, + OpenAICompatMessage, TextGenerationTaskInput, TextGenerationTaskOutput, Usage, } from "@workglow/ai"; +import { normalizeChatMessages, toOpenAIMessages } from "@workglow/ai/worker"; import { createEstimatedOutputUsageReporter, localOnlyFetch, @@ -28,15 +31,7 @@ import { getLlamaCppServerModelName } from "./LlamaCppServer_ModelUtil"; type AcquireFn = typeof acquireBaseUrl; interface UnifiedTextGenerationInput extends TextGenerationTaskInput { - readonly messages?: ReadonlyArray<{ - readonly role: string; - readonly content: - | string - | ReadonlyArray< - | { readonly type: "text"; readonly text: string } - | { readonly type: "image_url"; readonly image_url: { readonly url: string } } - >; - }>; + readonly messages?: ReadonlyArray; readonly systemPrompt?: string; } @@ -62,11 +57,18 @@ export function createLlamaCppServerTextGenerationStream( const unified = input as UnifiedTextGenerationInput; const hasMessages = Array.isArray(unified.messages) && unified.messages.length > 0; + // llama-server speaks chat-completions, so history goes through the shared + // OpenAI converter: workglow image blocks become `image_url` parts (llava), + // and blocks the endpoint has no form for are dropped rather than sent. const messages = hasMessages - ? [ - ...(unified.systemPrompt ? [{ role: "system", content: unified.systemPrompt }] : []), - ...unified.messages!.map((m) => ({ role: m.role, content: m.content })), - ] + ? toOpenAIMessages({ + // The converter lifts system turns into the system prompt itself, and + // keeps a history of system turns alone from becoming an empty user turn. + messages: normalizeChatMessages(unified.messages!), + systemPrompt: unified.systemPrompt, + prompt: "", + tools: [], + } as never).map(collapseTextOnlyContent) : [{ role: "user", content: input.prompt }]; const body = JSON.stringify({ @@ -124,6 +126,20 @@ export function createLlamaCppServerTextGenerationStream( }; } +/** + * A user turn that is text alone goes out as a plain string, as it always has: + * only a turn carrying an image needs the parts array, and a server built + * without multimodal support is the one that may not accept one. + */ +function collapseTextOnlyContent(message: OpenAICompatMessage): OpenAICompatMessage { + const content = message.content; + if (!Array.isArray(content) || !content.every((part) => part.type === "text")) return message; + return { + ...message, + content: content.map((part) => (typeof part.text === "string" ? part.text : "")).join(""), + }; +} + function promptTextFromMessages(messages: ReadonlyArray<{ readonly content: unknown }>): string { const parts: string[] = []; for (const message of messages) { diff --git a/providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts b/providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts index a0707f5fc0..89cbe65367 100644 --- a/providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts +++ b/providers/llamacpp-server/src/ai/common/LlamaCppServer_ToolCalling.ts @@ -23,7 +23,7 @@ import { filterValidToolCalls, isAllowedToolName, sanitizeToolArgs, - toTextFlatMessages, + toOpenAIMessages, } from "@workglow/ai/worker"; import type { PartialJsonStream } from "@workglow/util/worker"; import { createPartialJsonStream } from "@workglow/util/worker"; @@ -62,13 +62,28 @@ function mapTools(tools: readonly ToolDefinition[]) { })); } +/** The text of a message's content, string or parts, for the prompt-size estimate. */ +function promptText(content: unknown): string { + if (typeof content === "string") return content; + if (!Array.isArray(content)) return ""; + return content + .map((part: { type?: unknown; text?: unknown } | null) => + part?.type === "text" && typeof part.text === "string" ? part.text : "" + ) + .join("\n"); +} + export function createLlamaCppServerToolCallingStream( opts: ILlamaCppServerProviderOptions, acquire: AcquireFn = acquireBaseUrl ): AiProviderRunFn { return async (input, model, signal, emit) => { signal?.throwIfAborted?.(); - const messages = toTextFlatMessages(input); + // llama.cpp's server reads `reasoning_content` back on assistant messages. + const messages = toOpenAIMessages(input, { + replayReasoning: true, + toolImagesInUserMessage: true, + }); const tools = input.toolChoice === "none" ? undefined : mapTools(input.tools); const body = JSON.stringify({ model: getLlamaCppServerModelName(model), @@ -84,7 +99,7 @@ export function createLlamaCppServerToolCallingStream( const provisionalUsage = createEstimatedOutputUsageReporter(emit); provisionalUsage.onPrompt( messages - .map((m) => (typeof m.content === "string" ? m.content : "")) + .map((m) => promptText(m.content)) .filter(Boolean) .join("\n") ); @@ -113,6 +128,10 @@ export function createLlamaCppServerToolCallingStream( for await (const delta of readChatCompletionDeltas(response, signal)) { if (delta.done) break; usage = mapOpenAIChatUsage(delta.usage) ?? usage; + if (delta.reasoningDelta) { + provisionalUsage.onText(delta.reasoningDelta); + emit({ type: "text-delta", port: "reasoning", textDelta: delta.reasoningDelta }); + } if (delta.contentDelta) { provisionalUsage.onText(delta.contentDelta); emit({ type: "text-delta", port: "text", textDelta: delta.contentDelta }); diff --git a/providers/node-llama-cpp/src/ai/LlamaCppProvider.ts b/providers/node-llama-cpp/src/ai/LlamaCppProvider.ts index 5b10bbdb9d..39f742411c 100644 --- a/providers/node-llama-cpp/src/ai/LlamaCppProvider.ts +++ b/providers/node-llama-cpp/src/ai/LlamaCppProvider.ts @@ -9,6 +9,7 @@ import type { AiProviderRunFnRegistration, Capability, ModelConfig, + ModelEffortPolicy, ModelRecord, } from "@workglow/ai/worker"; import { AiProvider } from "@workglow/ai/worker"; @@ -17,6 +18,7 @@ import { llamaCppWorkerRunFnSpecs, } from "./common/LlamaCpp_Capabilities"; import { LOCAL_LLAMACPP } from "./common/LlamaCpp_Constants"; +import { llamaCppEffortPolicy } from "./common/LlamaCpp_EffortPolicy"; import type { LlamaCppModelConfig } from "./common/LlamaCpp_ModelSchema"; import { deleteLlamaCppSession } from "./common/LlamaCpp_Runtime"; @@ -39,6 +41,10 @@ export class LlamaCppProvider extends AiProvider { return inferLlamaCppCapabilities(model); } + override effortPolicy(model: LlamaCppModelConfig): ModelEffortPolicy | undefined { + return llamaCppEffortPolicy(model); + } + protected override workerRunFnSpecs(): readonly { serves: readonly Capability[] }[] { return llamaCppWorkerRunFnSpecs(); } diff --git a/providers/node-llama-cpp/src/ai/LlamaCppQueuedProvider.ts b/providers/node-llama-cpp/src/ai/LlamaCppQueuedProvider.ts index 03bdad6819..3cc8ee9015 100644 --- a/providers/node-llama-cpp/src/ai/LlamaCppQueuedProvider.ts +++ b/providers/node-llama-cpp/src/ai/LlamaCppQueuedProvider.ts @@ -9,6 +9,7 @@ import type { AiProviderRunFnRegistration, Capability, ModelConfig, + ModelEffortPolicy, ModelPricing, ModelRecord, } from "@workglow/ai"; @@ -24,6 +25,7 @@ import { llamaCppWorkerRunFnSpecs, } from "./common/LlamaCpp_Capabilities"; import { LOCAL_LLAMACPP } from "./common/LlamaCpp_Constants"; +import { llamaCppEffortPolicy } from "./common/LlamaCpp_EffortPolicy"; import type { LlamaCppModelConfig } from "./common/LlamaCpp_ModelSchema"; import { deleteLlamaCppSession } from "./common/LlamaCpp_Runtime"; @@ -58,6 +60,10 @@ export class LlamaCppQueuedProvider extends QueuedAiProvider { queue.push(chunk); provisionalUsage.onText(chunk); diff --git a/providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy.ts b/providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy.ts new file mode 100644 index 0000000000..98c12901d1 --- /dev/null +++ b/providers/node-llama-cpp/src/ai/common/LlamaCpp_EffortPolicy.ts @@ -0,0 +1,45 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ModelEffort, ModelEffortPolicyFn } from "@workglow/ai/worker"; +import { EFFORT_POLICY_ALL, makeEffortPolicy, resolveEnabledEffort } from "@workglow/ai/worker"; + +/** + * A local GGUF's ability to think is not knowable from its path, so every + * level is offered. A thought budget on a model that does not think is ignored + * by the chat wrapper, which makes offering the dial harmless where it does + * not apply. + */ +export const llamaCppEffortPolicy: ModelEffortPolicyFn = makeEffortPolicy({ + rules: [], + fallback: EFFORT_POLICY_ALL, +}); + +const THOUGHT_TOKENS: Readonly> = { + none: 0, + low: 512, + medium: 1024, + high: 2048, + extra: 4096, + ultra: 8192, +}; + +/** + * The `budgets.thoughtTokens` a caller's effort asks for, or `undefined` when + * no effort is set so the SDK keeps its own default (a share of the context). + */ +export function llamaCppThoughtBudget(model: object | undefined): number | undefined { + const effort = resolveEnabledEffort(model, llamaCppEffortPolicy(model)); + return effort === undefined ? undefined : THOUGHT_TOKENS[effort]; +} + +/** `session.prompt`'s thought budget for the effort a caller set; nothing when unset. */ +export function llamaCppThoughtBudgetOption(model: object | undefined): { + budgets?: { thoughtTokens: number }; +} { + const thoughtTokens = llamaCppThoughtBudget(model); + return thoughtTokens === undefined ? {} : { budgets: { thoughtTokens } }; +} diff --git a/providers/node-llama-cpp/src/ai/common/LlamaCpp_TextGeneration.ts b/providers/node-llama-cpp/src/ai/common/LlamaCpp_TextGeneration.ts index bbef17f1ff..43de24f1a7 100644 --- a/providers/node-llama-cpp/src/ai/common/LlamaCpp_TextGeneration.ts +++ b/providers/node-llama-cpp/src/ai/common/LlamaCpp_TextGeneration.ts @@ -32,6 +32,7 @@ import { withModelInUse, withSessionLock, } from "./LlamaCpp_Runtime"; +import { llamaCppThoughtBudgetOption } from "./LlamaCpp_EffortPolicy"; export const LlamaCpp_TextGeneration_Stream: AiProviderRunFn< TextGenerationTaskInput, @@ -175,6 +176,7 @@ export const LlamaCpp_TextGeneration_Stream: AiProviderRunFn< ...(input.temperature !== undefined && { temperature: input.temperature }), ...(input.maxTokens !== undefined && { maxTokens: input.maxTokens }), ...(input.topP !== undefined && { topP: input.topP }), + ...llamaCppThoughtBudgetOption(model), }); }, signal, diff --git a/providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling.ts b/providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling.ts index d994477531..efa787c081 100644 --- a/providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling.ts +++ b/providers/node-llama-cpp/src/ai/common/LlamaCpp_ToolCalling.ts @@ -24,6 +24,7 @@ import { renderLlamaCppPrefixChatHistory, renderLlamaCppPrefixFunctions, } from "./LlamaCpp_CacheCheckpoint"; +import { llamaCppThoughtBudget } from "./LlamaCpp_EffortPolicy"; import type { LlamaCppModelConfig } from "./LlamaCpp_ModelSchema"; import type { LlamaCppSessionState } from "./LlamaCpp_Runtime"; import { @@ -111,7 +112,16 @@ function messagesToPureChatHistory( const response: any[] = []; for (const block of msg.content) { - if (block.type === "text" && block.text) { + if (block.type === "reasoning") { + if (block.text) { + response.push({ + type: "segment", + segmentType: "thought", + text: block.text, + ended: true, + }); + } + } else if (block.type === "text" && block.text) { response.push(block.text); } else if (block.type === "tool_use") { // Create functionCall entry — result will be filled by subsequent tool message @@ -156,12 +166,21 @@ function messagesToPureChatHistory( item.result === undefined ); if (fnCall) { - // Flatten nested text blocks from tool_result content into a string - let resultText = ""; + // The result is a string, so an image cannot ride along. Say it was + // dropped rather than inlining its base64 into the prompt, in the place + // it held among the result's blocks. + const lines: string[] = []; for (const inner of block.content) { - if (inner.type === "text") resultText += inner.text; + if (inner.type === "text") { + if (inner.text) lines.push(inner.text); + } else if (inner.type === "image") { + lines.push(`[${inner.mimeType} image returned; this model reads text only]`); + } else { + // Any other block is data the tool returned; it goes as JSON. + lines.push(JSON.stringify(inner)); + } } - fnCall.result = resultText || JSON.stringify(block.content); + fnCall.result = lines.join("\n"); } } continue; @@ -246,6 +265,10 @@ function llamaCppChatGenerateOptions( } else if (toolChoiceForcesToolCall(input.toolChoice)) { opts.temperature = 0.2; } + const thoughtTokens = llamaCppThoughtBudget(model); + if (thoughtTokens !== undefined) { + opts.budgets = { thoughtTokens }; + } return opts; } @@ -263,18 +286,43 @@ function extractNativeFunctionCalls( })); } +interface LlamaResponseChunk { + readonly type?: string | undefined; + readonly segmentType?: string | undefined; + readonly text: string; +} + +/** + * Decides which output port a streamed chunk belongs to. `onTextChunk` leaves + * thought segments out entirely, so a run-fn that wants the model's thinking + * has to read `onResponseChunk` and sort the chunks itself: a thought goes to + * `reasoning`, the main response to `text`, and any other segment (a comment + * the model wrote about its own turn) is not part of the answer at all. + */ +export function routeLlamaResponseChunk( + chunk: LlamaResponseChunk +): { readonly port: "text" | "reasoning"; readonly text: string } | undefined { + // The SDK marks a main-response chunk with no type; "text" is taken too, so + // the answer still arrives should a release name it. + if (chunk.type === undefined || chunk.type === "text") return { port: "text", text: chunk.text }; + if (chunk.type === "segment" && chunk.segmentType === "thought") { + return { port: "reasoning", text: chunk.text }; + } + return undefined; +} + /** - * Drives an async generation call that pushes text chunks via `onTextChunk`, - * yielding `text-delta` events (and provisional `usage` snapshots) as they - * arrive. Returns accumulated text and the generation result (if any) once - * complete. + * Drives an async generation call that pushes response chunks via + * `onResponseChunk`, yielding `text-delta` events (and provisional `usage` + * snapshots) as they arrive. Returns the answer text (thoughts excluded) and + * the generation result (if any) once complete. */ async function* streamTextChunks( - startGeneration: (onTextChunk: (chunk: string) => void) => Promise, + startGeneration: (onResponseChunk: (chunk: LlamaResponseChunk) => void) => Promise, signal: AbortSignal, options: { readonly promptText?: string | undefined } = {} ): AsyncGenerator, { text: string; result: T | undefined }> { - const queue: string[] = []; + const queue: { readonly port: "text" | "reasoning"; readonly text: string }[] = []; const pendingUsage: StreamEvent[] = []; let isComplete = false; let completionError: unknown; @@ -295,9 +343,11 @@ async function* streamTextChunks( provisionalUsage.onPrompt(options.promptText); } - const generationPromise = startGeneration((chunk: string) => { - queue.push(chunk); - provisionalUsage.onText(chunk); + const generationPromise = startGeneration((chunk) => { + const routed = routeLlamaResponseChunk(chunk); + if (routed === undefined || routed.text === "") return; + queue.push(routed); + provisionalUsage.onText(routed.text); notifyWaiter(); }) .then((res) => { @@ -317,8 +367,8 @@ async function* streamTextChunks( } while (queue.length > 0) { const chunk = queue.shift()!; - accumulatedText += chunk; - yield { type: "text-delta", port: "text", textDelta: chunk }; + if (chunk.port === "text") accumulatedText += chunk.text; + yield { type: "text-delta", port: chunk.port, textDelta: chunk.text }; } }; @@ -377,13 +427,13 @@ async function generateToolResponse( .join("\n"); gen = streamTextChunks( - (onTextChunk) => + (onResponseChunk) => llamaChat.generateResponse(chatHistory, { signal, ...llamaCppChatGenerateOptions(input, model), functions, ...(toolChoiceForcesToolCall(input.toolChoice) && { documentFunctionParams: true }), - onTextChunk, + onResponseChunk, }), signal, { promptText } diff --git a/providers/node-llama-cpp/src/ai/index.ts b/providers/node-llama-cpp/src/ai/index.ts index fa357bef89..c47be21443 100644 --- a/providers/node-llama-cpp/src/ai/index.ts +++ b/providers/node-llama-cpp/src/ai/index.ts @@ -8,6 +8,7 @@ export * from "./common/LlamaCpp_Constants"; export * from "./common/LlamaCpp_ModelSchema"; +export { llamaCppEffortPolicy } from "./common/LlamaCpp_EffortPolicy"; // Mutable runtime state (e.g. llamaCppSessions) is intentionally NOT re-exported // here. The `ai-provider` and `ai-runtime` entry points are bundled // separately, so re-exporting from both creates two distinct module instances diff --git a/providers/node-llama-cpp/src/ai/runtime.ts b/providers/node-llama-cpp/src/ai/runtime.ts index b546871ee7..5fb86a1292 100644 --- a/providers/node-llama-cpp/src/ai/runtime.ts +++ b/providers/node-llama-cpp/src/ai/runtime.ts @@ -18,6 +18,7 @@ export { renderLlamaCppPrefixChatHistory, renderLlamaCppPrefixFunctions, } from "./common/LlamaCpp_CacheCheckpoint"; +export { llamaCppEffortPolicy, llamaCppThoughtBudget } from "./common/LlamaCpp_EffortPolicy"; export { LlamaCpp_SessionDispose } from "./common/LlamaCpp_SessionDispose"; export * from "./registerLlamaCppInline"; export * from "./registerLlamaCppWorker"; diff --git a/providers/ollama/src/ai/OllamaProvider.ts b/providers/ollama/src/ai/OllamaProvider.ts index c93e435251..beb9a2de67 100644 --- a/providers/ollama/src/ai/OllamaProvider.ts +++ b/providers/ollama/src/ai/OllamaProvider.ts @@ -5,10 +5,11 @@ */ import { createCloudProviderClass } from "@workglow/ai/provider-utils"; -import type { Capability, ModelRecord } from "@workglow/ai/worker"; +import type { Capability, ModelEffortPolicy, ModelRecord } from "@workglow/ai/worker"; import { AiProvider } from "@workglow/ai/worker"; import { inferOllamaCapabilities, ollamaWorkerRunFnSpecs } from "./common/Ollama_Capabilities"; import { OLLAMA } from "./common/Ollama_Constants"; +import { ollamaEffortPolicy } from "./common/Ollama_EffortPolicy"; import type { OllamaModelConfig } from "./common/Ollama_ModelSchema"; /** @@ -27,6 +28,10 @@ export class OllamaProvider extends createCloudProviderClass( return inferOllamaCapabilities(model); } + override effortPolicy(model: OllamaModelConfig): ModelEffortPolicy | undefined { + return ollamaEffortPolicy(model); + } + protected override workerRunFnSpecs(): readonly { serves: readonly Capability[] }[] { return ollamaWorkerRunFnSpecs(); } diff --git a/providers/ollama/src/ai/OllamaQueuedProvider.ts b/providers/ollama/src/ai/OllamaQueuedProvider.ts index c487458009..79e2a513e0 100644 --- a/providers/ollama/src/ai/OllamaQueuedProvider.ts +++ b/providers/ollama/src/ai/OllamaQueuedProvider.ts @@ -4,11 +4,18 @@ * SPDX-License-Identifier: Apache-2.0 */ -import type { Capability, ModelConfig, ModelPricing, ModelRecord } from "@workglow/ai"; +import type { + Capability, + ModelConfig, + ModelEffortPolicy, + ModelPricing, + ModelRecord, +} from "@workglow/ai"; import { AiProvider, FREE_LOCAL_PRICING } from "@workglow/ai"; import { createCloudProviderClass } from "@workglow/ai/provider-utils"; import { inferOllamaCapabilities, ollamaWorkerRunFnSpecs } from "./common/Ollama_Capabilities"; import { OLLAMA } from "./common/Ollama_Constants"; +import { ollamaEffortPolicy } from "./common/Ollama_EffortPolicy"; import type { OllamaModelConfig } from "./common/Ollama_ModelSchema"; /** Main-thread registration (inline or worker-backed). No queue — uses direct execution. */ @@ -28,6 +35,10 @@ export class OllamaQueuedProvider extends createCloudProviderClass + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ModelEffort, ModelEffortPolicyFn } from "@workglow/ai/worker"; +import { + EFFORT_POLICY_ALL, + EFFORT_POLICY_NONE, + makeEffortPolicy, + readModelName, + resolveEnabledEffort, +} from "@workglow/ai/worker"; +import type { OllamaModelConfig } from "./Ollama_ModelSchema"; + +const GPT_OSS = /^gpt-oss/i; + +/** + * The qwen3 tags that cannot think: the coder and embedding models, and the + * `-instruct` releases. Ollama answers `think: true` on them with HTTP 400. + */ +const QWEN3_NON_THINKING = /^qwen3(?:-(?:coder|embedding)|.*instruct)/i; + +function isThinkingQwen3(id: string): boolean { + return /^qwen3/i.test(id) && !QWEN3_NON_THINKING.test(id); +} + +/** Families Ollama marks as thinking-capable; they take `think: true | false`. */ +const THINKING = [ + isThinkingQwen3, + /^deepseek-r1/i, + /^deepseek-v3\.[1-9]/i, + /^magistral/i, + /^qwq/i, + /^phi4-reasoning/i, + /^cogito/i, + /^granite3\.[2-9]/i, +]; + +/** + * gpt-oss takes a level and cannot turn thinking off, so `none` is not listed: + * a level the policy omits resolves to no effort and nothing is sent. + */ +export const ollamaEffortPolicy: ModelEffortPolicyFn = makeEffortPolicy({ + rules: [ + { + when: GPT_OSS, + policy: { supported: ["low", "medium", "high", "extra", "ultra"], default: undefined }, + }, + { when: THINKING, policy: EFFORT_POLICY_ALL }, + ], + fallback: EFFORT_POLICY_NONE, +}); + +type OllamaThink = boolean | "low" | "medium" | "high"; + +function gptOssLevel(effort: ModelEffort): "low" | "medium" | "high" | undefined { + switch (effort) { + case "low": + case "medium": + case "high": + return effort; + case "extra": + case "ultra": + return "high"; + default: + return undefined; + } +} + +/** + * The request's `think` field for the effort a caller set. A model that cannot + * think rejects `think: true`, so nothing is sent unless the policy says the + * model thinks and an effort is set. + */ +export function ollamaThinkParam(model: OllamaModelConfig | undefined): { + think?: OllamaThink; +} { + const effort = resolveEnabledEffort(model, ollamaEffortPolicy(model)); + if (effort === undefined) return {}; + const name = readModelName(model); + if (GPT_OSS.test(name)) { + const level = gptOssLevel(effort); + return level === undefined ? {} : { think: level }; + } + return { think: effort !== "none" }; +} diff --git a/providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts b/providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts index cb630b5596..0f7b8468d9 100644 --- a/providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts +++ b/providers/ollama/src/ai/common/Ollama_JobRunFns.browser.ts @@ -4,7 +4,7 @@ * SPDX-License-Identifier: Apache-2.0 */ -import type { AiProviderRunFnRegistration, ToolCallingTaskInput } from "@workglow/ai"; +import type { AiProviderRunFnRegistration } from "@workglow/ai"; import { OLLAMA_JSON_MODE, OLLAMA_MODEL_INFO, @@ -16,6 +16,7 @@ import { OLLAMA_TOOL_USE, } from "./Ollama_CapabilitySets"; import { getClient } from "./Ollama_Client.browser"; +import { toOllamaMessages } from "./Ollama_Messages"; import { createOllamaModelInfoStream } from "./Ollama_ModelInfo"; import type { OllamaModelConfig } from "./Ollama_ModelSchema"; import { createOllamaModelSearchStream } from "./Ollama_ModelSearch"; @@ -28,27 +29,12 @@ import { createOllamaToolCallingStream } from "./Ollama_ToolCalling"; export { getClient, getModelName, loadOllamaSDK } from "./Ollama_Client.browser"; -function buildBrowserToolCallingMessages(input: ToolCallingTaskInput): Array<{ - role: string; - content: string; -}> { - const messages: Array<{ role: string; content: string }> = []; - if (input.systemPrompt) { - messages.push({ role: "system", content: input.systemPrompt as string }); - } - messages.push({ role: "user", content: input.prompt as string }); - return messages; -} - export const Ollama_TextGeneration_Stream = createOllamaTextGenerationStream(getClient); export const Ollama_StructuredGeneration_Stream = createOllamaStructuredGenerationStream(getClient); export const Ollama_TextRewriter_Stream = createOllamaTextRewriterStream(getClient); export const Ollama_TextSummary_Stream = createOllamaTextSummaryStream(getClient); export const Ollama_TextEmbedding_Stream = createOllamaTextEmbeddingStream(getClient); -export const Ollama_ToolCalling_Stream = createOllamaToolCallingStream( - getClient, - buildBrowserToolCallingMessages -); +export const Ollama_ToolCalling_Stream = createOllamaToolCallingStream(getClient, toOllamaMessages); export const Ollama_ModelInfo_Stream = createOllamaModelInfoStream(getClient); export const Ollama_ModelSearch_Stream = createOllamaModelSearchStream(getClient); diff --git a/providers/ollama/src/ai/common/Ollama_JobRunFns.ts b/providers/ollama/src/ai/common/Ollama_JobRunFns.ts index 54c6cff1fa..e0d40e8e7e 100644 --- a/providers/ollama/src/ai/common/Ollama_JobRunFns.ts +++ b/providers/ollama/src/ai/common/Ollama_JobRunFns.ts @@ -5,7 +5,6 @@ */ import type { AiProviderRunFnRegistration } from "@workglow/ai"; -import { toTextFlatMessages } from "@workglow/ai/worker"; import { OLLAMA_JSON_MODE, OLLAMA_MODEL_INFO, @@ -17,6 +16,7 @@ import { OLLAMA_TOOL_USE, } from "./Ollama_CapabilitySets"; import { getClient } from "./Ollama_Client"; +import { toOllamaMessages } from "./Ollama_Messages"; import { createOllamaModelInfoStream } from "./Ollama_ModelInfo"; import type { OllamaModelConfig } from "./Ollama_ModelSchema"; import { createOllamaModelSearchStream } from "./Ollama_ModelSearch"; @@ -32,10 +32,7 @@ export const Ollama_StructuredGeneration_Stream = createOllamaStructuredGenerati export const Ollama_TextRewriter_Stream = createOllamaTextRewriterStream(getClient); export const Ollama_TextSummary_Stream = createOllamaTextSummaryStream(getClient); export const Ollama_TextEmbedding_Stream = createOllamaTextEmbeddingStream(getClient); -export const Ollama_ToolCalling_Stream = createOllamaToolCallingStream( - getClient, - toTextFlatMessages -); +export const Ollama_ToolCalling_Stream = createOllamaToolCallingStream(getClient, toOllamaMessages); export const Ollama_ModelInfo_Stream = createOllamaModelInfoStream(getClient); export const Ollama_ModelSearch_Stream = createOllamaModelSearchStream(getClient); diff --git a/providers/ollama/src/ai/common/Ollama_Messages.ts b/providers/ollama/src/ai/common/Ollama_Messages.ts new file mode 100644 index 0000000000..093be4f9d6 --- /dev/null +++ b/providers/ollama/src/ai/common/Ollama_Messages.ts @@ -0,0 +1,173 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ToolCallingTaskInput } from "@workglow/ai"; +import { liftSystemMessages } from "@workglow/ai/worker"; +import { PermanentJobError } from "@workglow/job-queue"; + +/** + * One message on Ollama's `/api/chat`. `tool_calls` carry no ids — Ollama pairs + * a result to its call by `tool_name` on the `tool` message instead. + */ +export interface OllamaChatMessage { + readonly role: string; + readonly content: string; + readonly thinking?: string; + readonly images?: readonly string[]; + readonly tool_calls?: ReadonlyArray<{ + readonly function: { + readonly name: string; + readonly arguments: Record; + }; + }>; + readonly tool_name?: string; +} + +interface TextOrImage { + readonly type: string; + readonly text?: unknown; + readonly data?: unknown; +} + +/** + * Splits text/image blocks into the joined text and the base64 image list. With + * `keepOther`, any other block (data a tool returned) is kept as JSON text. + * With `markImagesAfter`, each image leaves a numbered marker where it stood, + * counting on from that many images already set aside, so text around an image + * still points at it once the image itself has moved to a later message. + * Audio is refused wherever it appears. + */ +function flattenBlocks( + blocks: ReadonlyArray, + separator: string, + keepOther = false, + markImagesAfter?: number +): { readonly text: string; readonly images: string[] } { + const texts: string[] = []; + const images: string[] = []; + for (const block of blocks) { + if (block.type === "audio") { + // Ollama's chat API has no audio field, and sending the turn without it + // would have the model answer as if it had heard nothing. + throw new PermanentJobError("Ollama's chat API does not accept audio input."); + } + if (block.type === "text" && typeof block.text === "string") { + texts.push(block.text); + } else if (block.type === "image" && typeof block.data === "string") { + images.push(block.data); + if (markImagesAfter !== undefined) { + texts.push(`[image ${markImagesAfter + images.length}] follows in the next message`); + } + } else if (keepOther && block.type !== "text" && block.type !== "image") { + texts.push(JSON.stringify(block)); + } + } + return { text: texts.join(separator), images }; +} + +function withImages(message: OllamaChatMessage, images: readonly string[]): OllamaChatMessage { + return images.length > 0 ? { ...message, images } : message; +} + +/** + * Converts a tool-calling request into Ollama chat messages, keeping the model's + * own tool calls, the tool name each result answers, thinking, and images. + * + * `messages` wins over `prompt` whenever it is non-empty: the agent loop sends + * both, and `messages` already starts with that prompt. + */ +export function toOllamaMessages(input: ToolCallingTaskInput): OllamaChatMessage[] { + const out: OllamaChatMessage[] = []; + // System turns in the history join the system prompt, as on the + // text-generation path, rather than being dropped. + const lifted = liftSystemMessages(input.messages ?? [], input.systemPrompt); + + if (lifted.systemPrompt) { + out.push({ role: "system", content: lifted.systemPrompt }); + } + + // A history of system turns alone is still the history: the prompt is what a + // caller with no history sends, never a stand-in for one. + const history = lifted.messages; + if (!input.messages || input.messages.length === 0) { + const prompt = input.prompt as unknown; + if (typeof prompt === "string") { + out.push({ role: "user", content: prompt }); + } else if (Array.isArray(prompt)) { + const blocks = prompt.map((item): TextOrImage => + typeof item === "string" ? { type: "text", text: item } : (item as TextOrImage) + ); + const { text, images } = flattenBlocks(blocks, "\n"); + out.push(withImages({ role: "user", content: text }, images)); + } + return out; + } + + // Provider-assigned ids restart at `call_0` each run, so they repeat across + // turns. The map is updated as each assistant turn is read, so a result + // resolves against the most recent preceding call with its id. + const toolNames = new Map(); + // Ollama's chat templates render a tool message's text only, so images a tool + // returned ride on a user turn — after the whole run of tool messages, so it + // never sits between two results of one assistant turn. + let toolImages: string[] = []; + const flushToolImages = (): void => { + if (toolImages.length === 0) return; + const labels = toolImages.map((_, i) => `[image ${i + 1}]`).join(", "); + out.push({ + role: "user", + content: `Images returned by the tool calls above, in order: ${labels}`, + images: toolImages, + }); + toolImages = []; + }; + + for (const msg of history) { + if (msg.role !== "tool") flushToolImages(); + if (msg.role === "user") { + const { text, images } = flattenBlocks(msg.content, "\n"); + out.push(withImages({ role: "user", content: text }, images)); + } else if (msg.role === "assistant") { + const texts: string[] = []; + const thoughts: string[] = []; + const calls: Array<{ + function: { name: string; arguments: Record }; + }> = []; + for (const block of msg.content) { + if (block.type === "text") { + texts.push(block.text); + } else if (block.type === "reasoning") { + if (block.text) thoughts.push(block.text); + } else if (block.type === "tool_use") { + toolNames.set(block.id, block.name); + calls.push({ function: { name: block.name, arguments: block.input } }); + } + } + const content = texts.join(""); + if (content === "" && thoughts.length === 0 && calls.length === 0) continue; + let message: OllamaChatMessage = { role: "assistant", content }; + if (thoughts.length > 0) message = { ...message, thinking: thoughts.join("") }; + if (calls.length > 0) message = { ...message, tool_calls: calls }; + out.push(message); + } else if (msg.role === "tool") { + for (const block of msg.content) { + if (block.type !== "tool_result") continue; + const { text, images } = flattenBlocks(block.content, "\n", true, toolImages.length); + let message: OllamaChatMessage = { + role: "tool", + content: block.is_error ? `Error: ${text}` : text, + }; + const name = toolNames.get(block.tool_use_id); + if (name !== undefined) message = { ...message, tool_name: name }; + out.push(message); + toolImages.push(...images); + } + } + } + flushToolImages(); + + return out; +} diff --git a/providers/ollama/src/ai/common/Ollama_TextGeneration.ts b/providers/ollama/src/ai/common/Ollama_TextGeneration.ts index baf77452cd..33234218ff 100644 --- a/providers/ollama/src/ai/common/Ollama_TextGeneration.ts +++ b/providers/ollama/src/ai/common/Ollama_TextGeneration.ts @@ -6,11 +6,16 @@ import type { AiProviderRunFn, + LooseChatMessage, TextGenerationTaskInput, TextGenerationTaskOutput, Usage, } from "@workglow/ai"; +import { normalizeChatMessages } from "@workglow/ai/worker"; import { createEstimatedOutputUsageReporter } from "@workglow/ai/provider-utils"; +import { ollamaThinkParam } from "./Ollama_EffortPolicy"; +import type { OllamaChatMessage } from "./Ollama_Messages"; +import { toOllamaMessages } from "./Ollama_Messages"; import type { OllamaModelConfig } from "./Ollama_ModelSchema"; import { getOllamaModelName } from "./Ollama_ModelUtil"; import { mapOllamaUsage } from "./Ollama_Usage"; @@ -18,10 +23,29 @@ import { mapOllamaUsage } from "./Ollama_Usage"; type GetClient = (model: OllamaModelConfig | undefined) => Promise; interface UnifiedTextGenerationInput extends TextGenerationTaskInput { - readonly messages?: readonly { readonly role: string; readonly content: string }[]; + readonly messages?: ReadonlyArray; readonly systemPrompt?: string; } +/** + * Chat history in Ollama's shape, through the same converter tool calling uses: + * an assistant turn keeps its tool calls and each tool result keeps the call it + * answers, so a history seeded with a tool exchange is not sent half of it. + * System turns join the system prompt, and a user turn's images ride in its own + * `images` field. + */ +export function toOllamaChatMessages( + messages: ReadonlyArray, + systemPrompt: string | undefined +): OllamaChatMessage[] { + return toOllamaMessages({ + messages: normalizeChatMessages(messages), + systemPrompt, + prompt: "", + tools: [], + } as never); +} + /** * Streaming run-fn factory for the `["text.generation"]` capability. Returns * an async generator that yields `text-delta` events and a final empty @@ -41,11 +65,8 @@ export function createOllamaTextGenerationStream( const unified = input as UnifiedTextGenerationInput; const hasMessages = Array.isArray(unified.messages) && unified.messages.length > 0; - const messages = hasMessages - ? [ - ...(unified.systemPrompt ? [{ role: "system", content: unified.systemPrompt }] : []), - ...unified.messages!.map((m) => ({ role: m.role, content: m.content })), - ] + const messages: OllamaChatMessage[] = hasMessages + ? toOllamaChatMessages(unified.messages!, unified.systemPrompt) : [{ role: "user", content: input.prompt }]; // Ollama only reports counts on the terminal `done: true` chunk; estimate ↑ @@ -61,6 +82,7 @@ export function createOllamaTextGenerationStream( const stream = await client.chat({ model: modelName, messages, + ...ollamaThinkParam(model), options: { temperature: input.temperature, top_p: input.topP, diff --git a/providers/ollama/src/ai/common/Ollama_ToolCalling.ts b/providers/ollama/src/ai/common/Ollama_ToolCalling.ts index effbbfde65..cee852ae92 100644 --- a/providers/ollama/src/ai/common/Ollama_ToolCalling.ts +++ b/providers/ollama/src/ai/common/Ollama_ToolCalling.ts @@ -17,13 +17,13 @@ import { buildToolDescription, filterValidToolCalls, sanitizeToolArgs } from "@w import { parsePartialJson } from "@workglow/util/worker"; import type { OllamaModelConfig } from "./Ollama_ModelSchema"; import { getOllamaModelName } from "./Ollama_ModelUtil"; +import { ollamaThinkParam } from "./Ollama_EffortPolicy"; +import type { OllamaChatMessage } from "./Ollama_Messages"; import { mapOllamaUsage } from "./Ollama_Usage"; type GetClient = (model: OllamaModelConfig | undefined) => Promise; -export type OllamaToolCallingMessagesFn = ( - input: ToolCallingTaskInput -) => Array<{ role: string; content: string }>; +export type OllamaToolCallingMessagesFn = (input: ToolCallingTaskInput) => OllamaChatMessage[]; function mapOllamaTools(tools: ReadonlyArray) { return tools.map((t) => ({ @@ -60,6 +60,7 @@ export function createOllamaToolCallingStream( model: modelName, messages, tools, + ...ollamaThinkParam(model), options: { temperature: input.temperature, num_predict: input.maxTokens, @@ -76,6 +77,11 @@ export function createOllamaToolCallingStream( try { for await (const chunk of stream) { usage = mapOllamaUsage(chunk) ?? usage; + const thinking = chunk.message.thinking; + if (thinking) { + provisionalUsage.onText(thinking); + emit({ type: "text-delta", port: "reasoning", textDelta: thinking }); + } const delta = chunk.message.content; if (delta) { provisionalUsage.onText(delta); diff --git a/providers/ollama/src/ai/index.browser.ts b/providers/ollama/src/ai/index.browser.ts index cd6a8e6d38..82dfa4fb86 100644 --- a/providers/ollama/src/ai/index.browser.ts +++ b/providers/ollama/src/ai/index.browser.ts @@ -7,5 +7,6 @@ // organize-imports-ignore export * from "./common/Ollama_Constants"; +export * from "./common/Ollama_EffortPolicy"; export * from "./common/Ollama_ModelSchema"; export * from "./registerOllama"; diff --git a/providers/ollama/src/ai/index.ts b/providers/ollama/src/ai/index.ts index 42ba9df194..ac73c45b45 100644 --- a/providers/ollama/src/ai/index.ts +++ b/providers/ollama/src/ai/index.ts @@ -7,6 +7,7 @@ // organize-imports-ignore export * from "./common/Ollama_Constants"; +export * from "./common/Ollama_EffortPolicy"; export * from "./common/Ollama_ModelSchema"; export * from "./registerOllama"; diff --git a/providers/ollama/src/ai/runtime.browser.ts b/providers/ollama/src/ai/runtime.browser.ts index 59877518ad..452971a482 100644 --- a/providers/ollama/src/ai/runtime.browser.ts +++ b/providers/ollama/src/ai/runtime.browser.ts @@ -14,5 +14,6 @@ // organize-imports-ignore export * from "./common/Ollama_Client.browser"; +export * from "./common/Ollama_EffortPolicy"; export * from "./registerOllamaInline.browser"; export * from "./registerOllamaWorker.browser"; diff --git a/providers/ollama/src/ai/runtime.ts b/providers/ollama/src/ai/runtime.ts index f394dee74b..dc699e8d15 100644 --- a/providers/ollama/src/ai/runtime.ts +++ b/providers/ollama/src/ai/runtime.ts @@ -14,6 +14,7 @@ // organize-imports-ignore export * from "./common/Ollama_Client"; +export * from "./common/Ollama_EffortPolicy"; export * from "./common/Ollama_TextGeneration"; export * from "./common/Ollama_StructuredGeneration"; export * from "./registerOllamaInline"; diff --git a/providers/openai/src/ai/common/OpenAI_CacheCheckpoint.ts b/providers/openai/src/ai/common/OpenAI_CacheCheckpoint.ts index ae65639721..d3967ecdc0 100644 --- a/providers/openai/src/ai/common/OpenAI_CacheCheckpoint.ts +++ b/providers/openai/src/ai/common/OpenAI_CacheCheckpoint.ts @@ -21,6 +21,34 @@ import { promptToTailMessages, toOpenAIMessages } from "@workglow/ai/worker"; import { finalizeResponsesRequest, getClient, getModelName } from "./OpenAI_Client"; import type { OpenAiModelConfig } from "./OpenAI_ModelSchema"; +/** + * How a conversation is turned into chat messages for a finalized Responses + * request. Warming a checkpoint and sending a turn over it must agree, or the + * request diverges from the warmed prefix at the first turn they write + * differently and the prompt cache stops matching there. + * + * A native turn is replayed only into a request that asks for encrypted + * reasoning back: with reasoning off its items would be looked up by id and + * fail where nothing is stored, so the turn is rebuilt instead. It is keyed by + * model, because encrypted reasoning is readable only by the model that wrote it. + */ +export function openAIConversationOptions( + model: OpenAiModelConfig | undefined, + params: Record +): { readonly toolImagesInUserMessage: true; readonly nativeTurnProvider?: string | undefined } { + const replay = + (model?.provider_config as { replay_reasoning?: boolean } | undefined)?.replay_reasoning !== + false; + const reasoning = params.reasoning as { effort?: string } | undefined; + const wantsEncryptedReasoning = replay && reasoning !== undefined && reasoning.effort !== "none"; + return { + // A function_call_output carries text only, so images a tool returned go on + // a user turn after the results. + toolImagesInUserMessage: true, + ...(wantsEncryptedReasoning ? { nativeTurnProvider: `openai:${getModelName(model)}` } : {}), + }; +} + /** * Merges a checkpoint prefix into the unified generation input: the prefix's * messages come first, the caller's tail follows (its `messages`, or its @@ -89,13 +117,15 @@ export const OpenAI_CacheCheckpoint_Stream: AiProviderRunFn< // The "." prompt is a throwaway user turn used only when the prefix has no // messages of its own (toOpenAIMessages ignores it otherwise); the cached // value is the system/tools region ahead of it. - const messages = toOpenAIMessages({ + const conversation = { messages: prefix.messages ?? [], systemPrompt: prefix.systemPrompt, prompt: ".", tools: [], - } as never); - const { input: responsesInput, instructions } = buildResponsesInput({ messages }); + } as never; + const { input: responsesInput, instructions } = buildResponsesInput({ + messages: toOpenAIMessages(conversation, { toolImagesInUserMessage: true }), + }); const params: Record = { model: getModelName(model), @@ -107,6 +137,11 @@ export const OpenAI_CacheCheckpoint_Stream: AiProviderRunFn< params.tools = buildResponsesTools(prefix.tools); } finalizeResponsesRequest(model, params); + // Written the way a turn over this checkpoint writes it, now that the + // request's reasoning is settled. + params.input = buildResponsesInput({ + messages: toOpenAIMessages(conversation, openAIConversationOptions(model, params)), + }).input; const response = await (client.responses.create as (p: unknown, o: unknown) => Promise)( params, diff --git a/providers/openai/src/ai/common/OpenAI_ModelSchema.ts b/providers/openai/src/ai/common/OpenAI_ModelSchema.ts index 3bde7e83cf..38a2ab020e 100644 --- a/providers/openai/src/ai/common/OpenAI_ModelSchema.ts +++ b/providers/openai/src/ai/common/OpenAI_ModelSchema.ts @@ -77,6 +77,12 @@ export const OpenAiModelSchema = { "Override whether the model accepts a pinned temperature with reasoning effort 'none'. Absent means 'decide from the model id'. A wrong `true` is an HTTP 400.", "x-ui-hidden": true, }, + replay_reasoning: { + type: "boolean", + description: + "Hand the model its own reasoning back on later rounds (default true); false is the ablation switch.", + "x-ui-hidden": true, + }, }, required: ["model_name"], additionalProperties: false, diff --git a/providers/openai/src/ai/common/OpenAI_TextGeneration.ts b/providers/openai/src/ai/common/OpenAI_TextGeneration.ts index 641334fb1d..df99be3e43 100644 --- a/providers/openai/src/ai/common/OpenAI_TextGeneration.ts +++ b/providers/openai/src/ai/common/OpenAI_TextGeneration.ts @@ -49,12 +49,17 @@ function buildResponsesParams( ): Record { const hasMessages = Array.isArray(input.messages) && input.messages.length > 0; const messages = hasMessages - ? toOpenAIMessages({ - messages: input.messages, - systemPrompt: input.systemPrompt, - prompt: "", - tools: [], - } as never) + ? toOpenAIMessages( + { + messages: input.messages, + systemPrompt: input.systemPrompt, + prompt: "", + tools: [], + } as never, + // A function_call_output carries text only: images a tool returned go on a + // user turn after the results, as the checkpoint warm-up writes them. + { toolImagesInUserMessage: true } + ) : undefined; const { input: responsesInput, instructions } = buildResponsesInput({ diff --git a/providers/openai/src/ai/common/OpenAI_ToolCalling.ts b/providers/openai/src/ai/common/OpenAI_ToolCalling.ts index 37cc044e64..c05682448a 100644 --- a/providers/openai/src/ai/common/OpenAI_ToolCalling.ts +++ b/providers/openai/src/ai/common/OpenAI_ToolCalling.ts @@ -19,7 +19,7 @@ import { promptTextForResponsesUsageEstimate, } from "@workglow/ai/provider-utils"; import { filterValidToolCalls, toOpenAIMessages } from "@workglow/ai/worker"; -import { mergeOpenAICheckpointPrefix } from "./OpenAI_CacheCheckpoint"; +import { mergeOpenAICheckpointPrefix, openAIConversationOptions } from "./OpenAI_CacheCheckpoint"; import { finalizeResponsesRequest, getClient, getModelName } from "./OpenAI_Client"; import type { OpenAiModelConfig } from "./OpenAI_ModelSchema"; @@ -27,7 +27,10 @@ import type { OpenAiModelConfig } from "./OpenAI_ModelSchema"; * Streaming run-fn for `["text.generation", "tool-use"]`. Calls the OpenAI * Responses endpoint with `stream: true` and forwards delta events via * {@link accumulateOpenAIResponsesStream}, which emits `text-delta` and tool-call - * `object-delta` events plus a final empty `finish`. + * `object-delta` events plus a final empty `finish`. The turn's own output + * items go back on the `nativeTurn` port only when the request asked for + * encrypted reasoning, since a replayed reasoning item without it is looked up + * by id and fails where the response is not stored. * * Defence-in-depth: each tool-call `object-delta` is filtered against the * effective tool declarations (the caller's tools, or the checkpoint prefix's @@ -49,17 +52,16 @@ export const OpenAI_ToolCalling_Stream: AiProviderRunFn< const merged = mergeOpenAICheckpointPrefix(sessionContext, input); const toolDefinitions = merged?.tools ?? input.tools; const tools = buildResponsesTools(toolDefinitions); + const conversation = merged + ? ({ + ...input, + messages: merged.messages, + systemPrompt: merged.systemPrompt, + prompt: "", + } as ToolCallingTaskInput) + : input; const { input: responsesInput, instructions } = buildResponsesInput({ - messages: toOpenAIMessages( - merged - ? ({ - ...input, - messages: merged.messages, - systemPrompt: merged.systemPrompt, - prompt: "", - } as ToolCallingTaskInput) - : input - ), + messages: toOpenAIMessages(conversation, { toolImagesInUserMessage: true }), }); const toolChoice = mapResponsesToolChoice(input.toolChoice); @@ -73,6 +75,18 @@ export const OpenAI_ToolCalling_Stream: AiProviderRunFn< if (input.maxTokens !== undefined) params.max_output_tokens = input.maxTokens; if (input.temperature !== undefined) params.temperature = input.temperature; finalizeResponsesRequest(model, params); + // A reasoning model starts its thinking over every round unless it is handed + // its encrypted reasoning back; the items come back on `nativeTurn`. The + // input is written the way the checkpoint warm-up writes it, so a prefix + // still matches. + const conversationOptions = openAIConversationOptions(model, params); + const nativeKey = conversationOptions.nativeTurnProvider; + if (nativeKey !== undefined) { + params.include = ["reasoning.encrypted_content"]; + params.input = buildResponsesInput({ + messages: toOpenAIMessages(conversation, conversationOptions), + }).input; + } const promptText = promptTextForResponsesUsageEstimate(params); createEstimatedOutputUsageReporter(emit).onPrompt(promptText); @@ -94,7 +108,7 @@ export const OpenAI_ToolCalling_Stream: AiProviderRunFn< } emit(event); }, - { promptText } + { promptText, nativeTurnProvider: nativeKey } ); emit({ type: "finish", data: { text: "", toolCalls: [] } as ToolCallingTaskOutput, usage }); }; diff --git a/providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails.ts b/providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails.ts new file mode 100644 index 0000000000..3f617f32b0 --- /dev/null +++ b/providers/openrouter/src/ai/common/OpenRouter_ReasoningDetails.ts @@ -0,0 +1,188 @@ +/** + * @license + * Copyright 2026 Steven Roussey + * SPDX-License-Identifier: Apache-2.0 + */ + +import type { ChatMessage, ContentBlockReasoning } from "@workglow/ai"; +import type { OpenAICompatMessage } from "@workglow/ai/worker"; + +/** Fields of a reasoning detail whose streamed pieces are text to be joined. */ +const CONCATENATED_KEYS: ReadonlySet = new Set(["text", "summary", "data"]); + +/** + * Joins the pieces of `reasoning_details` OpenRouter streams on + * `choices[0].delta`. Each piece carries the `index` of the item it belongs to. + */ +export interface ReasoningDetailAccumulator { + readonly entries: Map>; + /** The joined items by index, or `undefined` when no piece arrived. */ + items(): unknown[] | undefined; +} + +export function createReasoningDetailAccumulator(): ReasoningDetailAccumulator { + const entries = new Map>(); + return { + entries, + items() { + if (entries.size === 0) return undefined; + return [...entries.entries()].sort(([a], [b]) => a - b).map(([, item]) => item); + }, + }; +} + +/** Folds one streamed `reasoning_details` array into the accumulator. */ +export function mergeReasoningDetailDelta(acc: ReasoningDetailAccumulator, delta: unknown): void { + if (!Array.isArray(delta)) return; + for (const piece of delta) { + if (piece === null || typeof piece !== "object" || Array.isArray(piece)) continue; + const incoming = piece as Record; + const index = + typeof incoming.index === "number" && Number.isFinite(incoming.index) + ? incoming.index + : nextIndex(acc); + const existing = acc.entries.get(index); + if (!existing) { + acc.entries.set(index, { ...incoming }); + continue; + } + for (const [key, value] of Object.entries(incoming)) { + const current = existing[key]; + existing[key] = + CONCATENATED_KEYS.has(key) && typeof current === "string" && typeof value === "string" + ? current + value + : value; + } + } +} + +function nextIndex(acc: ReasoningDetailAccumulator): number { + let max = -1; + for (const index of acc.entries.keys()) max = Math.max(max, index); + return max + 1; +} + +/** A call as the replay check compares it: its arguments as the converter writes them. */ +export interface ReasoningTurnCall { + readonly id: string; + readonly name: string; + readonly arguments: string; +} + +/** What one round's native turn carries: its reasoning and the calls it was written beside. */ +export interface ReasoningTurn { + readonly calls: readonly ReasoningTurnCall[]; + readonly details: readonly unknown[]; +} + +/** + * Whether the details stand as the model wrote them. An Anthropic thinking + * block streamed through OpenRouter is only replayable with its signature, and + * the stream can end before that arrives. + */ +function isReplayable(details: readonly unknown[]): boolean { + return !details.some((detail) => { + if (detail === null || typeof detail !== "object") return false; + const { type, format, signature } = detail as Record; + return ( + type === "reasoning.text" && + typeof format === "string" && + format.startsWith("anthropic-") && + !(typeof signature === "string" && signature.length > 0) + ); + }); +} + +/** + * The native-turn payload for a round, or `undefined` when the details cannot + * be replayed and the turn is better rebuilt. + */ +export function encodeReasoningTurn( + calls: ReadonlyArray<{ readonly id: string; readonly name: string; readonly input: unknown }>, + details: readonly unknown[] +): string | undefined { + if (!isReplayable(details)) return undefined; + return JSON.stringify({ + calls: calls.map((call) => ({ + id: call.id, + name: call.name, + arguments: JSON.stringify(call.input ?? {}), + })), + details, + }); +} + +function isTurnCall(value: unknown): value is ReasoningTurnCall { + if (value === null || typeof value !== "object") return false; + const { id, name, arguments: args } = value as Record; + return typeof id === "string" && typeof name === "string" && typeof args === "string"; +} + +function decodeReasoningTurn(payload: string): ReasoningTurn | undefined { + let parsed: unknown; + try { + parsed = JSON.parse(payload); + } catch { + return undefined; + } + if (parsed === null || typeof parsed !== "object") return undefined; + const { calls, details } = parsed as Record; + // Each detail must at least be a typed object: `[null]` or a bare value would + // be sent back as `reasoning_details` and refused, where a rebuilt turn is not. + if ( + !Array.isArray(details) || + !details.every( + (detail) => + detail !== null && + typeof detail === "object" && + !Array.isArray(detail) && + typeof (detail as { type?: unknown }).type === "string" + ) + ) { + return undefined; + } + if (!Array.isArray(calls) || !calls.every(isTurnCall)) return undefined; + return { calls, details }; +} + +/** + * OpenRouter reads an assistant turn's reasoning back as `reasoning_details`. + * `converted` is the shared converter's output for `history`, which keeps one + * assistant message per assistant turn, in order. Each turn's own reasoning + * block, written by `nativeKey`, is attached to its message only while the + * turn's tool calls still equal the ones the details were written beside, by + * id, name and arguments in order: a host that changed, dropped or added a call + * gets the rebuilt turn. The inputs are not mutated. + */ +export function withReasoningDetails( + converted: readonly OpenAICompatMessage[], + history: ReadonlyArray | undefined, + nativeKey: string +): Array { + const turns = (history ?? []).filter((message) => message.role === "assistant"); + const aligned = + turns.length === converted.filter((message) => message.role === "assistant").length; + let next = 0; + return converted.map((message) => { + if (message.role !== "assistant" || !aligned) return message; + const turn = turns[next++]!; + const block = turn.content.find( + (b): b is ContentBlockReasoning => + b.type === "reasoning" && b.provider === nativeKey && b.payload !== undefined + ); + const decoded = block === undefined ? undefined : decodeReasoningTurn(block.payload!); + if (decoded === undefined) return message; + const sent = message.tool_calls ?? []; + const matches = + sent.length === decoded.calls.length && + sent.every((call, i) => { + const written = decoded.calls[i]!; + return ( + call.id === written.id && + call.function.name === written.name && + call.function.arguments === written.arguments + ); + }); + return matches ? { ...message, reasoning_details: decoded.details } : message; + }); +} diff --git a/providers/openrouter/src/ai/common/OpenRouter_ToolCalling.ts b/providers/openrouter/src/ai/common/OpenRouter_ToolCalling.ts index bf8186180c..8a0884f87b 100644 --- a/providers/openrouter/src/ai/common/OpenRouter_ToolCalling.ts +++ b/providers/openrouter/src/ai/common/OpenRouter_ToolCalling.ts @@ -20,8 +20,62 @@ import { filterValidToolCalls, toOpenAIMessages } from "@workglow/ai/worker"; import { getClient, getModelName } from "./OpenRouter_Client"; import type { OpenRouterModelConfig } from "./OpenRouter_ModelSchema"; import { buildOpenRouterExtras } from "./OpenRouter_RequestParams"; +import { + createReasoningDetailAccumulator, + encodeReasoningTurn, + mergeReasoningDetailDelta, + withReasoningDetails, +} from "./OpenRouter_ReasoningDetails"; +import type { ReasoningDetailAccumulator } from "./OpenRouter_ReasoningDetails"; import { mapOpenRouterUsage } from "./OpenRouter_Usage"; +/** The readable text of a chunk's reasoning details: `reasoning.text` and summaries. */ +function detailText(details: readonly unknown[]): string { + return details + .map((detail) => { + // A malformed entry is skipped, as the accumulator skips it. + if (detail === null || typeof detail !== "object") return ""; + const d = detail as { type?: unknown; text?: unknown; summary?: unknown }; + if (d.type === "reasoning.text" && typeof d.text === "string") return d.text; + if (d.type === "reasoning.summary" && typeof d.summary === "string") return d.summary; + return ""; + }) + .join(""); +} + +/** + * Passes the stream through while collecting each chunk's `reasoning_details`. + * A chunk whose thinking arrives only there, with no `reasoning` of its own, + * gets that text as `reasoning`, so it reaches the reasoning port as well as + * the replay; one that carries both is left alone, so nothing is said twice. + */ +async function* tapReasoningDetails( + stream: AsyncIterable, + acc: ReasoningDetailAccumulator +): AsyncGenerator { + for await (const chunk of stream) { + const delta = chunk?.choices?.[0]?.delta; + const details = delta?.reasoning_details; + if (!Array.isArray(details)) { + yield chunk; + continue; + } + mergeReasoningDetailDelta(acc, details); + const text = detailText(details); + if (text && !delta.reasoning && !delta.reasoning_content) { + yield { + ...chunk, + choices: [ + { ...chunk.choices[0], delta: { ...delta, reasoning: text } }, + ...chunk.choices.slice(1), + ], + }; + continue; + } + yield chunk; + } +} + /** * Streaming run-fn for `["text.generation", "tool-use"]`. Forwards deltas via * {@link accumulateOpenAIChatStream}; each tool-call delta is filtered against @@ -36,7 +90,14 @@ export const OpenRouter_ToolCalling_Stream: AiProviderRunFn< const modelName = getModelName(model); const tools = buildOpenAITools(input.tools); - const messages = toOpenAIMessages(input); + // The details are readable only by the model that wrote them, so the native + // turn is keyed by model and another model gets the rebuilt turn. + const nativeKey = `openrouter:${modelName}`; + const messages = withReasoningDetails( + toOpenAIMessages(input, { toolImagesInUserMessage: true }), + input.messages, + nativeKey + ); const toolChoice = mapOpenAIToolChoice(input.toolChoice, true); const stream = await client.chat.completions.create( @@ -54,12 +115,24 @@ export const OpenRouter_ToolCalling_Stream: AiProviderRunFn< { signal } ); + const details = createReasoningDetailAccumulator(); + // The calls this round actually emitted, in order, each as last streamed: the + // replay is valid only beside the same calls. + const emittedCalls = new Map(); + // Whether each call the model made survived validation, as last streamed. The + // details were produced beside every call, so a turn missing one of them is + // not the turn they belong to and is not replayed. + const callKept = new Map(); const usage = await accumulateOpenAIChatStream( - stream, + tapReasoningDetails(stream, details), (event) => { if (event.type === "object-delta" && event.port === "toolCalls") { - const validated = filterValidToolCalls(event.objectDelta as ToolCalls, input.tools); + const calls = event.objectDelta as ToolCalls; + const validated = filterValidToolCalls(calls, input.tools); + const kept = new Set(validated.map((call) => call.id)); + for (const call of calls) callKept.set(call.id, kept.has(call.id)); if (validated.length > 0) { + for (const call of validated) emittedCalls.set(call.id, call); emit({ type: "object-delta", port: "toolCalls", objectDelta: validated }); } return; @@ -74,5 +147,18 @@ export const OpenRouter_ToolCalling_Stream: AiProviderRunFn< .join("\n"), } ); + const items = details.items(); + const dropped = [...callKept.values()].some((isKept) => !isKept); + const payload = + items === undefined || dropped + ? undefined + : encodeReasoningTurn([...emittedCalls.values()], items); + if (payload !== undefined) { + emit({ + type: "object-delta", + port: "nativeTurn", + objectDelta: { provider: nativeKey, payload }, + }); + } emit({ type: "finish", data: { text: "", toolCalls: [] } as ToolCallingTaskOutput, usage }); }; diff --git a/providers/xai/src/ai/common/Xai_ToolCalling.ts b/providers/xai/src/ai/common/Xai_ToolCalling.ts index f7bfe3f9a5..dd65ab16ed 100644 --- a/providers/xai/src/ai/common/Xai_ToolCalling.ts +++ b/providers/xai/src/ai/common/Xai_ToolCalling.ts @@ -40,7 +40,7 @@ export const Xai_ToolCalling_Stream: AiProviderRunFn< const modelName = getModelName(model); const tools = buildOpenAITools(input.tools); - const messages = toOpenAIMessages(input); + const messages = toOpenAIMessages(input, { toolImagesInUserMessage: true }); const toolChoice = mapOpenAIToolChoice(input.toolChoice, true); const reasoningEffort = getXaiReasoningEffort(model);