Repository navigation
Fix #236: show token limit in --check/init-config and forward it through eval, attempts and workflows - #247
Merged
Merged
Conversation
…val, attempts and workflow stages (#236)
This was referenced Oct 30, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #236
Summary
Finishes the remaining work on #236 (the token budget itself landed in #246):
--checklimits line now shows the token limit:steps …, cost …, tokens …, time …, with the same(from config)/(from --token-limit)labels as the other limits.--init-configstarter config has a commentedtoken_limit: 0line next tocost_limit.--attempts:--token-limitis now an invocation-wide budget, like--cost-limitand--time-limit. Each attempt gets the tokens left (--token-limit <remaining>), no new attempt starts once the budget is used up, and an attempt endingTokenLimitExceededstops the run with stop reasonbudget exhausted. The--checkplan line says "cost, token and time limits are shared by all attempts".--eval: suite tasks acceptlimits.token_limit, forwarded to each trial as--token-limit(validated >= 0).--workflow:token_limitis accepted indefaultsand on each stage, forwarded as--token-limit(validated >= 0).docs/runs.md,docs/project-context.md,docs/eval-framework.md.Decisions on open questions
--attempts, the token limit is shared across attempts (matching how cost and time already work there) rather than applied per attempt. This is the safer reading of a budget; before this change the flag was forwarded verbatim, so each attempt got the full amount.--eval, a top-level--token-limitkeeps its current behaviour (forwarded to every trial as a per-trial cap). Only the per-tasklimits.token_limitkey is new. A suite-wide token budget, like eval's--cost-limit, was not added, to avoid changing what an existing flag means for eval.Testing
gofmt -l .clean;go build ./...,go vet ./...andgo test -count=1 ./...all pass.internal/attempts/token_budget_test.go,internal/harness/token_limit_test.go,internal/eval/token_limit_test.go,cmd/tinyagent/check_token_limit_test.go. Updated the existing--checklimits-line expectations.