From d43db68eb19d72160d2a821a9521afcfb7969833 Mon Sep 17 00:00:00 2001 From: Vic Gupta Date: Fri, 30 Oct 2026 12:21:23 -0400 Subject: [PATCH] Docs: cover verify-before-stop, token budgets and help changes since the docs split Document everything merged since #237: --verify-on-stop, --verify-command, --max-verify-reprompts and --token-limit in the CLI reference, a new docs/verify.md, the budget.warning event and TokenLimitExceeded status, token limits in --check/--init-config/eval/workflows/attempts, and README feature list and docs index entries. Add a test that every registered flag appears in docs/subcommands.md. --- README.md | 8 ++++++-- cmd/tinyagent/docs_flags_test.go | 33 ++++++++++++++++++++++++++++++++ docs/config.md | 6 +----- docs/eval-framework.md | 2 +- docs/integrations.md | 2 +- docs/json-output.md | 5 +++-- docs/long-runs.md | 6 ++++-- docs/runs.md | 6 ++++-- docs/subcommands.md | 8 ++++++-- docs/verify.md | 31 ++++++++++++++++++++++++++++++ 10 files changed, 90 insertions(+), 17 deletions(-) create mode 100644 cmd/tinyagent/docs_flags_test.go create mode 100644 docs/verify.md diff --git a/README.md b/README.md index def08d5..c357c92 100644 --- a/README.md +++ b/README.md @@ -106,7 +106,7 @@ tinyagent --read-only --cwd ~/src/project -t "Review error handling in pkg/api" cat review-task.md | tinyagent --read-only -t - -o review.json ``` -Subcommands are shortcuts for common flags: `tinyagent sandbox`, `tinyagent memory list`, `tinyagent planner --first`, `tinyagent plugins list`, `tinyagent tinygo status` (see [CLI reference](docs/subcommands.md)). +Run `tinyagent --help` (or plain `tinyagent` at a terminal) for the overview and `tinyagent --help --verbose` for every flag. Subcommands are shortcuts for common flags: `tinyagent sandbox`, `tinyagent memory list`, `tinyagent planner --first`, `tinyagent plugins list`, `tinyagent tinygo status` (see [CLI reference](docs/subcommands.md)); each one alone, such as `tinyagent sandbox`, prints its help. ## Features @@ -115,6 +115,9 @@ Subcommands are shortcuts for common flags: `tinyagent sandbox`, `tinyagent memo - Streaming text, `--json` event output, structured output, SQLite event logs and OpenTelemetry. - Project memory across runs, with approval before entries become active. - A task planner that can require a plan before any change. +- Run budgets: turn cap, cost limit and token limit (`--token-limit`), with an 80% warning to the model; delegates, workers and `--attempts` share the budget, and a run that hits it can be resumed. +- Verify before stop (`--verify-on-stop`): a run that changed files isn't accepted until a test or build command passes; otherwise it is marked unverified. +- Help everywhere: `tinyagent` or `tinyagent ` alone at a terminal prints help; `--help --verbose` lists every flag by section. - Undo for file edits, workspace rewind, resume and follow-up runs. - Sandboxed commands with a network allowlist, permission rules, hooks, untrusted-content lockdown and file-access rules. - WebAssembly plugin modules run through Extism with permission checks, a version store, rollback and prune. Go source modules are built with TinyGo, an external dependency that tinyagent downloads on first use (pinned version, checksum-verified, cached locally, `tinyagent tinygo update` to update); TinyGo is never embedded in tinyagent. @@ -128,7 +131,8 @@ Subcommands are shortcuts for common flags: `tinyagent sandbox`, `tinyagent memo | Config file, profiles, MCP and command tools | [docs/config.md](docs/config.md) | | Providers, ChatGPT subscription, reasoning and fallback | [docs/providers.md](docs/providers.md) | | `--json`, streaming, structured output, event log, OpenTelemetry | [docs/json-output.md](docs/json-output.md) | -| Reviews, worktrees, best-of-N, resume, transcripts, follow-ups, asking for input | [docs/runs.md](docs/runs.md) | +| Reviews, worktrees, best-of-N, resume, transcripts, follow-ups, asking for input, token and cost budgets | [docs/runs.md](docs/runs.md) | +| Verify before stop (`--verify-on-stop`) | [docs/verify.md](docs/verify.md) | | `undo_file` and `--rewind` | [docs/undo.md](docs/undo.md) | | Project memory | [docs/memory.md](docs/memory.md) | | Planner (`update_plan`, `--plan-first`) | [docs/planner.md](docs/planner.md) | diff --git a/cmd/tinyagent/docs_flags_test.go b/cmd/tinyagent/docs_flags_test.go new file mode 100644 index 0000000..f927044 --- /dev/null +++ b/cmd/tinyagent/docs_flags_test.go @@ -0,0 +1,33 @@ +package main + +import ( + "flag" + "os" + "strings" + "testing" +) + +// TestCLIReferenceListsEveryFlag fails when a registered flag is missing +// from the flag table in docs/subcommands.md. +func TestCLIReferenceListsEveryFlag(t *testing.T) { + b, err := os.ReadFile("../../docs/subcommands.md") + if err != nil { + t.Fatal(err) + } + doc := string(b) + newFlagSet(&options{}).VisitAll(func(f *flag.Flag) { + dash := "--" + if len(f.Name) == 1 { + dash = "-" + } + name := dash + f.Name + for _, prefix := range []string{"`", ", ", " "} { + for _, end := range []string{"`", " ", "=", "[", ","} { + if strings.Contains(doc, prefix+name+end) { + return + } + } + } + t.Errorf("flag %s is not documented in docs/subcommands.md", name) + }) +} diff --git a/docs/config.md b/docs/config.md index 130b5ae..e1f6b52 100644 --- a/docs/config.md +++ b/docs/config.md @@ -183,13 +183,9 @@ profiles: # omitted by default; named overlays selected with review: model: openai/gpt-5-mini read_only: true -``` - ### Verify before stop -With `verify_on_stop.enabled` (or `--verify-on-stop`), a run that changed files does not finish on an unverified final answer. A successful `write_file`, `edit_file` or `apply_patch` call, or a `bash` command that is not a verification command, marks changes pending. A `bash` command counts as verification when it is one of `commands` (alone or followed by arguments) or matches one of `patterns`; when both lists are empty, common test and build commands count (`go test`, `go build`, `go vet`, `cargo test`, `npm test`, `make`, `pytest` and similar, at the start of the command or after `&&`, `;` or `||`). A verification that exits 0 clears the pending changes; one that fails leaves them pending, and the model sees its output as a normal tool result. - -When the model gives a final answer (with or without the completion marker) while changes are pending, it gets a fixed user message asking it to verify, naming the configured commands, and the run continues as normal steps, so step, cost and time limits still apply. Tool calls in a submitting message are not run and get an error result saying so. After `max_reprompts` re-prompts, or straight away when `bash` cannot run (denied by the permission policy, a hook or lockdown, or unavailable as in read-only mode), the run finishes with its normal status and is marked unverified: the summary line ends with `, unverified changes`, the `--json` result has `"unverified": true`, and the trajectory has `unverified: true`. `verify_reprompts` counts the re-prompts in both. The guard runs before stop hooks, does not apply to subagent runs or `--output-schema` runs, and keeps its state across `--resume`. +`verify_on_stop` (or `--verify-on-stop`, `--verify-command`, `--max-verify-reprompts`) keeps a run that changed files from finishing until a test or build command has passed, re-prompting the model up to `max_reprompts` times and otherwise marking the run unverified. See [Verify before stop](verify.md#verify-before-stop) for the full rules. ### Profiles diff --git a/docs/eval-framework.md b/docs/eval-framework.md index 7393534..cb284ff 100644 --- a/docs/eval-framework.md +++ b/docs/eval-framework.md @@ -2,7 +2,7 @@ > **Status: experimental; v1 implemented.** `tinyagent --eval suite.yaml --eval-out dir` runs the suite sequentially (code in `internal/eval/` and `cmd/tinyagent/eval.go`). What v1 ships and how it differs from the target design below: > -> - **Shipped:** strict `version: 1` suite files (`version`, `models`, `profiles`, `trials`, and per task `id`, `repo`, `base`, `task`/`task_file`, `model`, `profile`, `checks`, `checks_on_limit`, `limits`, `tags`); a detached worktree per trial at `base`, removed afterwards (also on Ctrl-C) unless `--eval-keep`; one child tinyagent run per trial with `-o` written to `/trajectories/.json` and its output to `logs/.log`; `results.jsonl`, `report.json`, `report.md` (pass rate, pass@k and pass^k, cost, tokens, steps, wall time, outcome counts, per-task passes); suite-level `--cost-limit` / `--time-limit`; `--dry-run`; `--eval-min-pass-rate` (exit 1). Other flags on the command line (e.g. `-c`, `--no-stream`, `--ask-user`, `--step-limit`) are forwarded to every trial. +> - **Shipped:** strict `version: 1` suite files (`version`, `models`, `profiles`, `trials`, and per task `id`, `repo`, `base`, `task`/`task_file`, `model`, `profile`, `checks`, `checks_on_limit`, `limits`, `tags`); a detached worktree per trial at `base`, removed afterwards (also on Ctrl-C) unless `--eval-keep`; one child tinyagent run per trial with `-o` written to `/trajectories/.json` and its output to `logs/.log`; `results.jsonl`, `report.json`, `report.md` (pass rate, pass@k and pass^k, cost, tokens, steps, wall time, outcome counts, per-task passes); suite-level `--cost-limit` / `--time-limit`; `--dry-run`; `--eval-min-pass-rate` (exit 1); per-task `limits.token_limit`, forwarded to each trial as `--token-limit`. Other flags on the command line (e.g. `-c`, `--no-stream`, `--ask-user`, `--step-limit`) are forwarded to every trial. > - **Differences:** `trials` is required. `--time-limit` takes seconds (as in single runs), not `30m`. Check output kept in `results.jsonl` is capped to the last 4096 bytes. Checks run on the host in the worktree. A trial's `--profile` is passed only for names other than `default` (unless the config defines a profile called `default`). > - **Not yet implemented:** `--eval-jobs` (trials are sequential), `--json` eval events, `output_schema` / `expected_json` (rejected as unknown keys), `profile_digest`, the GitHub Action `eval` input. diff --git a/docs/integrations.md b/docs/integrations.md index 5e18490..1f3addf 100644 --- a/docs/integrations.md +++ b/docs/integrations.md @@ -41,7 +41,7 @@ Use the full path to the binary. Pass `-m`, `-c`, `--profile` and any policy fla - The first `session/prompt` is the session's task. Each later prompt continues the same conversation, sending the provider exactly what `--resume -t ""` would. This also works after a turn that was cancelled or hit a limit. A second prompt while one is running in the same session gets a JSON-RPC error. - Prompt content: text blocks are joined; embedded text resources are included as quoted context naming their URI; resource links become a reference line and are not fetched; images are attached like `--image` (PNG, JPEG, GIF or WebP, at most 10, each within `max_image_bytes`). Audio, binary resources and invalid images get an invalid-params error. - Streaming: assistant text arrives as `agent_message_chunk` (without the submit marker). Each tool call is a `tool_call` with the model's call id, a short title, a `kind` (`read_file` → `read`; `write_file`/`edit_file`/`apply_patch` → `edit`; `grep`/`list_files` → `search`; `bash`/`start_process` → `execute`; `web_fetch` → `fetch`; others → `other`) and absolute `locations` for file tools. It is followed by `tool_call_update`s to `in_progress` and then `completed` or `failed`, carrying the same result text the model sees, after `max_output_chars` and secret redaction. `update_plan` sends `plan` updates, and each model call sends a `usage_update` when the context window is known. -- Stop reasons: `Submitted`/`Completed` → `end_turn`, `StepLimitExceeded` → `max_turn_requests`, `OutputTruncated` → `max_tokens`, `session/cancel` → `cancelled`. Any other status (`CostLimitExceeded`, `TimeLimitExceeded`, `RepeatedFormatError`, `VerificationFailed`, `Error`) ends with a final message naming it and `end_turn`, and the session stays usable. A provider error a headless run would print a setup `hint:` for is returned as a JSON-RPC error instead. +- Stop reasons: `Submitted`/`Completed` → `end_turn`, `StepLimitExceeded` → `max_turn_requests`, `OutputTruncated` → `max_tokens`, `session/cancel` → `cancelled`. Any other status (`CostLimitExceeded`, `TokenLimitExceeded`, `TimeLimitExceeded`, `RepeatedFormatError`, `VerificationFailed`, `Error`) ends with a final message naming it and `end_turn`, and the session stays usable. A provider error a headless run would print a setup `hint:` for is returned as a JSON-RPC error instead. - `session/cancel` stops the turn the way Ctrl-C does: in-flight requests and commands are cancelled, and background processes and language servers are stopped, as at the end of every turn. Closing stdin cancels running turns, stops MCP servers and exits 0. - `--step-limit` and `--cost-limit` apply to each prompt (on top of what the session has already used), and `--time-limit` limits each prompt's wall-clock time. - `--log-db` records each prompt as a run, with an `acp.turn` event naming the session id and turn number. No trajectory files are written. diff --git a/docs/json-output.md b/docs/json-output.md index d2b9306..db5c664 100644 --- a/docs/json-output.md +++ b/docs/json-output.md @@ -26,7 +26,8 @@ Every event has a `type`. Event names and fields in schema version 1 are: | `sandbox.network_denied` | `host`, `port`, `reason`; the sandbox allowlist proxy refused a request from a command (see [Network allowlist](sandbox.md#network-allowlist)) | | `web_fetch` | `url`, `final_url`, `status`, `content_type`, `bytes`, `truncated`, `duration_ms`, and `denied_reason` (for a refused URL) or `error` (for a failed fetch); one per fetch attempt, emitted before its `tool.result` (see [Web fetch](long-runs.md#web-fetch)) | | `memory.loaded` | `path` (the store file), `loaded` (entry count), `ids` (loaded entry ids, newest first), `omitted` (active entries left out to stay within `memory.max_bytes`); once after `run.started` for a fresh run with [project memory](memory.md#project-memory) on. A resumed or follow-up run of a checkpoint that had memory on emits it too, describing the replayed checkpoint snapshot (the store is not re-read), with `from_checkpoint: true`. Never includes entry text | -| `verify.reprompt` | `reprompt`, `max_reprompts`; the model gave a final answer with unverified changes and was sent back to verify (see [Verify before stop](config.md#verify-before-stop)) | +| `budget.warning` | `step`, `usage` (what was used, as text), `tokens`, `token_limit`, `cost_usd`, `cost_limit`; once per run when a token or cost budget reaches 80% (see [Token and cost budgets](runs.md#token-and-cost-budgets)) | +| `verify.reprompt` | `reprompt`, `max_reprompts`; the model gave a final answer with unverified changes and was sent back to verify (see [Verify before stop](verify.md#verify-before-stop)) | | `verify.unverified` | `reason`, `reprompts`; the run finishes with unverified changes | | `memory.saved` | `op` (`add`, `replace` or `forget`), `id`, `status` (`active`, `pending` or `forgotten`), `redacted`, `untrusted_context`; one per successful `remember` write, emitted before its `tool.result`. Never includes entry text | | `lsp.server` | `name`, `state` (`started`, `unavailable`, `crashed` or `stopped`), `reason` (for `unavailable` and `crashed`); one per language-server state change (see [Language servers (LSP)](long-runs.md#language-servers-lsp)) | @@ -36,7 +37,7 @@ Every event has a `type`. Event names and fields in schema version 1 are: | `model.fallback` | `step`, `from`, `to`, `reason` (`http_` or `network`), `attempts`; the run switched to the next [fallback model](providers.md#model-fallback) after `from` used up its retries | | `taint` | `source` (`web_fetch`, `mcp`, `review`, `untrusted_task` or `resume`), `step`, and `tool_call_id`, `tool`, `host` (the `web_fetch` host only) and `resumed` when they apply; one per new taint source of a run with [lockdown](permissions.md#untrusted-content-lockdown) on. Never includes content or URLs | | `lockdown.denied` | `channel` (`command`, `web_fetch` or `mcp`), `tool`, `tool_call_id`, `step`, and `host` for `web_fetch`; one per call lockdown refused after taint | -| `result` | `status`, `submission` (the question for `NeedsInput`), cumulative `steps`, `structured_output` (only with `--output-schema` and status `Submitted`), `question` (`tool_call_id`, `question`, `options` when given; only for status `NeedsInput`, see [Asking for input](runs.md#asking-for-input)), `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `cost_usd`, invocation `duration_ms`, and `tainted` and `lockdown_denials` (a count; only with [lockdown](permissions.md#untrusted-content-lockdown) on), `unverified` (`true` only when the run finished with unverified changes) and `verify_reprompts` (only when > 0; see [Verify before stop](config.md#verify-before-stop)) | +| `result` | `status` (for example `Submitted`, `Completed`, `StepLimitExceeded`, `CostLimitExceeded`, `TokenLimitExceeded`, `TimeLimitExceeded`, `NeedsInput`; see [Exit status](subcommands.md#exit-status)), `submission` (the question for `NeedsInput`), cumulative `steps`, `structured_output` (only with `--output-schema` and status `Submitted`), `question` (`tool_call_id`, `question`, `options` when given; only for status `NeedsInput`, see [Asking for input](runs.md#asking-for-input)), `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`, `cost_usd`, invocation `duration_ms`, and `tainted` and `lockdown_denials` (a count; only with [lockdown](permissions.md#untrusted-content-lockdown) on), `unverified` (`true` only when the run finished with unverified changes) and `verify_reprompts` (only when > 0; see [Verify before stop](verify.md#verify-before-stop)) | When the run starts, the final event is `result`, including for model errors, interruptions, and limit stops. Exit codes are unchanged. `--resume` reports cumulative trajectory accounting; `duration_ms` covers only the current invocation. Event streams can contain prompts, code, and tool output, so handle them with the same care as trajectory files. Credentials and environment dumps are not included. diff --git a/docs/long-runs.md b/docs/long-runs.md index 91f21ed..be6e8ea 100644 --- a/docs/long-runs.md +++ b/docs/long-runs.md @@ -2,6 +2,8 @@ ## Long runs +To cap how much a long run can spend, use `--step-limit`, `--cost-limit`, `--token-limit` and `--time-limit`; see [Token and cost budgets](runs.md#token-and-cost-budgets) for the 80% warning and resuming with a higher limit. + tinyagent resends the whole history on every step, so a long run can outgrow the model's context window. Before each request it estimates the request's size (the previous request's reported input tokens plus about 4 characters per token for anything added since). Above 80% of the window, it compacts: 1. Old tool outputs are replaced with a placeholder such as `[output elided: 12,400 chars]`. The system prompt, the task, and the most recent turns (about 25% of the window, and never fewer than the last 2 assistant turns with their tool results) are kept verbatim. If that brings the estimate under 50% of the window, it stops here. @@ -101,7 +103,7 @@ A checkpoint records whether `apply_patch` was enabled (`apply_patch_enabled`), `--subagents` (or `subagents.enabled: true`) adds a `delegate` tool. A call such as `{"task": "find every caller of LoadConfig and say which ones ignore its error", "label": "callers"}` starts a child agent with a fresh context: the same system prompt and `AGENTS.md`, plus a notice that it is a subagent that must answer first and then give evidence with `file:line` references. The child runs to completion, and the parent gets back only its final report followed by a footer such as `[delegate d1: Submitted, 4 steps, 9200 in / 410 out tokens, $0.0123]`. It is off by default; with it off, requests, tools, and checkpoints are unchanged. - **Read-only.** The child has only `read_file`, `list_files`, and `grep`, whatever the parent has. It gets no `write_file`, `edit_file`, `apply_patch`, `bash`, process tools, MCP tools, `skill`, `view_image`, `update_plan`, or `delegate`, and calls to them are refused. Paths follow the parent's rules (`--cwd` and `file_access`). The tool is also offered with `--read-only`. -- **Limits.** Each child stops after `subagents.max_steps` model steps (default 30); hitting the cap, or any other unfinished status, returns a tool error such as `error: delegate d1 did not finish: StepLimit` rather than a report. Child steps don't count toward `--step-limit`; they're reported separately as `subagent_steps`. Child tokens and cost do count toward the run's totals, `--cost-limit` and `--token-limit`, and the child only gets what's left of the budget, so a delegated task can end the whole run as `CostLimit` or `TokenLimitExceeded`. `--time-limit` covers children too. Reports longer than `max_output_chars` are truncated like any tool output. Several `delegate` calls in one turn run one after another. +- **Limits.** Each child stops after `subagents.max_steps` model steps (default 30); hitting the cap, or any other unfinished status, returns a tool error such as `error: delegate d1 did not finish: StepLimit` rather than a report. Child steps don't count toward `--step-limit`; they're reported separately as `subagent_steps`. Child tokens and cost do count toward the run's totals, `--cost-limit` and `--token-limit`, and the child only gets what's left of the budget, so a delegated task can end the whole run as `CostLimitExceeded` or `TokenLimitExceeded`. `--time-limit` covers children too. Reports longer than `max_output_chars` are truncated like any tool output. Several `delegate` calls in one turn run one after another. - **Model.** `subagents.model` picks a different (typically cheaper) model; empty uses the parent's. `reasoning_effort` and `max_output_tokens` carry over only when the child uses the same provider as the parent. An unknown provider, or a missing key for a different provider, exits 2 before any request. - **Tokens.** Delegation saves the parent's context, not tokens overall: the child resends its own history each step, so a task the parent could answer with one `grep` is cheaper done directly. Use it for searches that would otherwise pull many large files into the parent's history. - **Observability.** `--json` and `--log-db` add `subagent.started` (`id`, `label`, `model`, `tool_call_id`, `parent_step`) and `subagent.finished` (`id`, `status`, `steps`, tokens, `cost_usd`) events, and the child's own events carry `"subagent": "d1"`. `-v` prints a line when each child starts and finishes; the child's model output is not streamed. Hooks run for child tool calls with `subagent` set in their input, and `pre_tool`/`post_tool` hooks also see the `delegate` call itself; stop hooks run only for the parent. With `-o`, the trajectory records `subagents_enabled`, `subagent_steps`, and a `subagents` array with each child's task, status, usage, report, and full message history. @@ -127,7 +129,7 @@ subagents: - **Result.** The worker's report, a footer such as `[worker w1: Submitted · branch tinyagent/run-…-w1 · commit abc1234 · 3 files changed, 40 insertions(+), 5 deletions(-) · 18 steps · $0.0400]` and the `git diff --stat`. - **Applying.** `apply_worker` with `{"worker": "w1"}` applies the worker's base-to-commit diff to the working tree (not the index), atomically: on a conflict nothing changes and it lists the conflicting files and the branch. It is refused when the diff touches a [`file_access`](permissions.md#restricting-file-access) `deny_write` path or a file outside `--cwd`. It counts as a file-changing tool for `--snapshots` and runs through `pre_tool`/`post_tool` hooks. - **Tools.** A worker gets the configured `tools` that the parent itself has, never `delegate`, the worker tools, `ask_user`, `update_plan`, MCP tools or `web_fetch`. Hooks run for its calls with `subagent` set to `w1`, `w2`, … and `worker: true`. -- **Budgets.** Worker usage counts toward the parent's `--cost-limit`; a worker that hits a limit stops, its partial work is still committed, and the run ends with the limit status. +- **Budgets.** Worker usage counts toward the parent's `--cost-limit` and `--token-limit` (a batch of workers shares one token budget, so all stop when it is spent); a worker that hits a limit stops, its partial work is still committed, and the run ends with the limit status. - **Concurrency.** With `max_concurrent` above 1, two or more `worker` calls in one model response run at the same time, at most `max_concurrent` at once. Permission checks, `pre_tool` hooks and worktree creation run first, one call at a time in call order, so every worker starts from the same snapshot; commits, `post_tool` hooks and results then follow in call order. The workers share the remaining `--cost-limit`: once their combined spend reaches it, or the run is interrupted or hits `--wall-time`, every running worker is cancelled, each keeps its partial work committed, and the run ends with that status. `--json` adds a `workers.batch` event (`workers`, `max_concurrent`, `stop`). Calls from different responses, and `apply_worker`, still run one at a time. - **Requirements.** `--workers` needs a Git repository with at least one commit, and can't be combined with `--read-only` or the `--review` modes. Commits use the repository's identity, or `tinyagent ` when none is configured. - **Observability.** `--json` adds `kind: "worker"`, `branch`, `commit` and `files_changed` to `subagent.started`/`subagent.finished`, and a `worker.applied` event. The trajectory records each worker's history, branch, base commit and commit. diff --git a/docs/runs.md b/docs/runs.md index 4f7c194..f9258e0 100644 --- a/docs/runs.md +++ b/docs/runs.md @@ -102,7 +102,7 @@ Compacted runs resume too. The trajectory always keeps the full history; the mod A distinct `-o`/`--output` is required, and any existing destination is refused, including symlink and hard-link aliases of the source. The input stays unchanged. The validated starting state is saved to the new destination before the first request; completed tool turns and termination then replace it using a private temporary file. Initial publication requires filesystem hard-link support to publish atomically without replacing another file. A save failure in resume mode stops further work, prints an error, and exits nonzero, preserving the last complete checkpoint. Ordinary fresh-run save failures retain their existing warning behavior. Each save writes a private temporary file, fsyncs it, renames it over the checkpoint, and fsyncs the directory (where the platform supports it), so a crash or power loss leaves either the previous or the new complete checkpoint, never a partial one. -Steps, cost, token counts, and cache totals remain cumulative. Step/cost limits are cumulative ceilings: if already exhausted, no model request is made; increase the ceiling explicitly (or set it to 0 for unlimited). Runtime limits, tool timeouts, output settings, and compaction settings use normal config/flag precedence. Wall time and the stderr elapsed duration restart for each invocation; stopped time is not charged. Normal streaming, summaries, and success/failure exit codes still apply. The next action is a new model query, which incurs ordinary context/token costs; loading a checkpoint never dispatches historical tool calls. +Steps, cost, token counts, and cache totals remain cumulative. Step, cost and token limits are cumulative ceilings: if already exhausted, no model request is made; increase the ceiling explicitly (or set it to 0 for unlimited). Runtime limits, tool timeouts, output settings, and compaction settings use normal config/flag precedence. Wall time and the stderr elapsed duration restart for each invocation; stopped time is not charged. Normal streaming, summaries, and success/failure exit codes still apply. The next action is a new model query, which incurs ordinary context/token costs; loading a checkpoint never dispatches historical tool calls. New JSON keeps the existing top-level trajectory fields and adds versioned metadata: `schema_version` (currently 2; version 1 files still load), `root`, `system_prompt`, `skip_agents_md`, `tools`, and `tool_signature`. Version 2 adds `pending_calls`, `processes`, `plan_enabled`, `plan`, and `plan_history`; runs with skills enabled also record `skills_enabled` and `skills` (name, scope, and path of each discovered skill). `resume` records the input path, previous status/submission, cumulative step, and continuation-message index, plus `follow_up: true` for a follow-up boundary; nested `previous` entries retain earlier resume boundaries. These audit paths are not read as configuration. Metadata excludes credentials and environment dumps. Checkpoints still contain prompts, code, and tool outputs: treat them as private, trusted local execution inputs. New checkpoint and temporary files use owner-only permissions (0600) where supported. @@ -182,4 +182,6 @@ The question is model-written text that can quote workspace files or fetched pag `--token-limit n` (config `token_limit`) caps the run's provider-reported input plus output tokens; prompt-cache tokens are part of input. `--cost-limit usd` (config `cost_limit`) caps its cost, computed from the same usage and the model's price (the built-in table, or `input_price_per_mtok` and `output_price_per_mtok`). Both are off (0) by default, and delegates and workers count against the parent's budget. A model with no known price can't be held to a cost limit: tinyagent warns once at start, and only the token limit is enforced. -When a token or cost budget reaches 80%, the model gets one user message saying how much is used and asking it to wrap up; the JSON stream, log database and OpenTelemetry trace get a `budget.warning` event. At 100% the run stops cleanly with `TokenLimitExceeded` or `CostLimitExceeded` (exit 1), and the printed `--resume` command doubles the limit that was hit. +When a token or cost budget reaches 80%, the model gets one user message saying how much is used and asking it to wrap up; the JSON stream, log database and OpenTelemetry trace get a `budget.warning` event. At 100% the run stops cleanly with `TokenLimitExceeded` or `CostLimitExceeded` (exit 1), and the printed `--resume` command doubles the limit that was hit. The warning fires once per run and re-arms if usage drops back under 80% (for example after resuming with a higher limit). + +`--check` shows the limits on one line (`steps …, cost …, tokens …, time …`, each labelled `(from config)` or `(from --token-limit)` and so on), and `--init-config` writes a commented `token_limit: 0`. With [`--attempts`](#best-of-n-attempts) the token budget is shared by all attempts, like cost and time. [Eval](eval-framework.md) tasks accept `limits.token_limit` and [workflows](project-context.md) accept `token_limit` in `defaults` and per stage; a top-level `--token-limit` with `--eval` caps each trial. The turn cap, `--step-limit`, is off by default. diff --git a/docs/subcommands.md b/docs/subcommands.md index bd33e01..f1dd839 100644 --- a/docs/subcommands.md +++ b/docs/subcommands.md @@ -39,7 +39,7 @@ Five subcommands are shortcuts for existing flags. They rewrite the command line | `--fallback-model ` | config `fallback_models`, else none | Model to switch to when the current one keeps failing with `429`, `5xx` or network errors after retries; repeatable, tried in order, at most 3, and replaces `fallback_models`; see [Model fallback](providers.md#model-fallback) | | `--no-fallback` | off | Don't fall back to other models (clears `fallback_models`); conflicts with `--fallback-model` | | `--openai-api` | config `openai_api`, else `auto` | Transport for the `openai` provider: `auto`, `responses`, or `chat`; see [OpenAI transport](providers.md#openai-transport) | -| `--init-config` | | Write a commented starter config to the user config path (or `-c PATH`), refuse overwrites, and exit | +| `--init-config` | | Write a commented starter config (including `token_limit` and `verify_on_stop`) to the user config path (or `-c PATH`), refuse overwrites, and exit | | `-c, --config` | user config, else built-in | YAML config file; see [Config](config.md#config) | | `--profile` | none | Apply one named profile from the config file's `profiles:` section over its base settings; see [Profiles](config.md#profiles) | | `--list-profiles` | | Print the profile names in the selected config file (`-c`) or the user config, sorted, one per line, then exit. Makes no model request and needs no credentials; see [Profiles](config.md#profiles) | @@ -65,8 +65,12 @@ Five subcommands are shortcuts for existing flags. They rewrite the command line | `--attempt-check` | | With `--attempts`: a shell command that must exit 0 in an attempt's worktree for it to pass (repeatable; replaces `attempts.checks` from the config) | | `--attempt-check-timeout` | 600 | With `--attempts`: seconds before one `--attempt-check` command is killed (1 to 3600) | | `--step-limit` | 0 (unlimited) | Max model calls | -| `--cost-limit` | 0 (unlimited) | Max USD | +| `--cost-limit` | 0 (unlimited) | Max USD; one warning at 80%. See [Token and cost budgets](runs.md#token-and-cost-budgets) | +| `--token-limit` | config `token_limit`, else 0 (unlimited) | Max provider-reported input+output tokens for the run, delegates and workers included; at 100% the run stops with `TokenLimitExceeded` (resumable), with one warning at 80%. With `--attempts`, shared by all attempts. See [Token and cost budgets](runs.md#token-and-cost-budgets) | | `--time-limit` | 0 (unlimited) | Max wall-clock seconds; overrides `wall_time_seconds` | +| `--verify-on-stop` | config `verify_on_stop.enabled`, else off | Don't accept a final answer while files changed since the last successful test/build command: send the model back to verify, up to `--max-verify-reprompts` times, then mark the run unverified. See [Verify before stop](verify.md#verify-before-stop) | +| `--verify-command ` | config `verify_on_stop.commands`, else common test/build commands | A `bash` command that counts as verification for `--verify-on-stop`; repeatable, replaces the config list | +| `--max-verify-reprompts ` | config `verify_on_stop.max_reprompts`, else 2 | With `--verify-on-stop`, how many times to send the model back before stopping unverified (1 to 10) | | `--max-format-errors` | config `max_format_errors`, else 3 | Stop with `RepeatedFormatError` after this many consecutive steps whose tool calls are all malformed; must be >= 1. See [Malformed tool calls](troubleshooting.md#malformed-tool-calls) | | `--max-output-tokens` | config `max_output_tokens`, else 0 | Per-request output-token budget sent to the provider; 0 keeps the provider default (8192 `max_tokens` on Anthropic). Not supported by `codex`. Unrelated to `max_output_chars`, which caps tool output. See [Output truncation](troubleshooting.md#output-truncation) | | `--no-compact` | off | Never compact the history; see [Long runs](long-runs.md#long-runs) | diff --git a/docs/verify.md b/docs/verify.md new file mode 100644 index 0000000..31ccc38 --- /dev/null +++ b/docs/verify.md @@ -0,0 +1,31 @@ +# Verify before stop + +An opt-in guard that keeps a run from finishing on untested changes: when files changed since the last successful test or build command, the model is sent back to verify before its final answer is accepted. Off by default. + +```bash +tinyagent --verify-on-stop -t "fix the failing test in pkg/foo" +tinyagent --verify-on-stop --verify-command "go test ./..." --max-verify-reprompts 3 -t "..." +``` + +| Flag | Config | Default | | +|---|---|---|---| +| `--verify-on-stop` | `verify_on_stop.enabled` | off | Turn the guard on (`--verify-on-stop=false` turns it off) | +| `--verify-command ` | `verify_on_stop.commands` | common test/build commands | A `bash` command that counts as verification; repeatable, replaces the config list | +| | `verify_on_stop.patterns` | none | Regular expressions; a matching `bash` command counts as verification | +| `--max-verify-reprompts ` | `verify_on_stop.max_reprompts` | 2 | How many times the model is sent back before the run is marked unverified (1 to 10) | + +## How it works + +### Verify before stop + +With `verify_on_stop.enabled` (or `--verify-on-stop`), a run that changed files does not finish on an unverified final answer. A successful `write_file`, `edit_file` or `apply_patch` call, or a `bash` command that is not a verification command, marks changes pending. A `bash` command counts as verification when it is one of `commands` (alone or followed by arguments) or matches one of `patterns`; when both lists are empty, common test and build commands count (`go test`, `go build`, `go vet`, `cargo test`, `npm test`, `make`, `pytest` and similar, at the start of the command or after `&&`, `;` or `||`). A verification that exits 0 clears the pending changes; one that fails leaves them pending, and the model sees its output as a normal tool result. + +When the model gives a final answer (with or without the completion marker) while changes are pending, it gets a fixed user message asking it to verify, naming the configured commands, and the run continues as normal steps, so step, cost and time limits still apply. Tool calls in a submitting message are not run and get an error result saying so. After `max_reprompts` re-prompts, or straight away when `bash` cannot run (denied by the permission policy, a hook or lockdown, or unavailable as in read-only mode), the run finishes with its normal status and is marked unverified: the summary line ends with `, unverified changes`, the `--json` result has `"unverified": true`, and the trajectory has `unverified: true`. `verify_reprompts` counts the re-prompts in both. The guard runs before stop hooks, does not apply to subagent runs or `--output-schema` runs, and keeps its state across `--resume`. + +## Where it shows up + +- **Summary line:** ends with `, unverified changes`. +- **`--json`:** `verify.reprompt` (`reprompt`, `max_reprompts`) each time the model is sent back, `verify.unverified` (`reason`, `reprompts`) when the run ends unverified, and the `result` event's `unverified` and `verify_reprompts` fields; see [JSON output](json-output.md#machine-readable-output). +- **Trajectory (`-o`):** `unverified: true` and `verify_reprompts`. + +The config block is shown in the [example config](config.md#config).