Skip to content

Fix #236: show token limit in --check/init-config and forward it through eval, attempts and workflows - #247

Merged
vicgupta merged 1 commit into
mainfrom
amf/issue-236
Oct 30, 2026
Merged

vicgupta merged 1 commit into
mainfrom
amf/issue-236

Conversation

@vicgupta

Copy link
Copy Markdown
Owner

Closes #236

Summary

Finishes the remaining work on #236 (the token budget itself landed in #246):

  • --check limits line now shows the token limit: steps …, cost …, tokens …, time …, with the same (from config) / (from --token-limit) labels as the other limits.
  • --init-config starter config has a commented token_limit: 0 line next to cost_limit.
  • --attempts: --token-limit is now an invocation-wide budget, like --cost-limit and --time-limit. Each attempt gets the tokens left (--token-limit <remaining>), no new attempt starts once the budget is used up, and an attempt ending TokenLimitExceeded stops the run with stop reason budget exhausted. The --check plan line says "cost, token and time limits are shared by all attempts".
  • --eval: suite tasks accept limits.token_limit, forwarded to each trial as --token-limit (validated >= 0).
  • --workflow: token_limit is accepted in defaults and on each stage, forwarded as --token-limit (validated >= 0).
  • Docs updated: docs/runs.md, docs/project-context.md, docs/eval-framework.md.

Decisions on open questions

  • Turn cap stays off by default, per Vic's comment on the issue. No default-on step limit was added.
  • In --attempts, the token limit is shared across attempts (matching how cost and time already work there) rather than applied per attempt. This is the safer reading of a budget; before this change the flag was forwarded verbatim, so each attempt got the full amount.
  • In --eval, a top-level --token-limit keeps its current behaviour (forwarded to every trial as a per-trial cap). Only the per-task limits.token_limit key is new. A suite-wide token budget, like eval's --cost-limit, was not added, to avoid changing what an existing flag means for eval.
  • All new token limits default to 0 (off), so existing configs, suites and workflows behave the same.

Testing

  • gofmt -l . clean; go build ./..., go vet ./... and go test -count=1 ./... all pass.
  • New tests (fakes only, no network or API keys): internal/attempts/token_budget_test.go, internal/harness/token_limit_test.go, internal/eval/token_limit_test.go, cmd/tinyagent/check_token_limit_test.go. Updated the existing --check limits-line expectations.

@vicgupta
vicgupta merged commit 061dbcc into main Oct 30, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Per-task turn cap and cost/token budget

1 participant