Skip to content

Fix #260: GitHub Action eval input with pass-rate gating - #261

Merged
vicgupta merged 2 commits into
mainfrom
amf/issue-260
Oct 31, 2026
Merged

vicgupta merged 2 commits into
mainfrom
amf/issue-260

Conversation

@vicgupta

Copy link
Copy Markdown
Owner

Closes #260

Summary

Adds an experimental eval input to the GitHub Action so CI can run tinyagent --eval suites and gate on pass rate.

  • New inputs: eval (suite YAML), eval-out, eval-min-pass-rate, plus cost-limit and time-limit (shared across the suite with eval).
  • eval cannot be combined with task, task-file, review-base, review-commit, attempts, model or profile (the suite defines its matrix); eval-out/eval-min-pass-rate need eval.
  • New outputs: eval-out, eval-report (report.json path) and eval-results (compact JSON: per-config pass rate and cost, below_threshold).
  • Appends the eval report.md to the job summary (respecting the 1 MiB limit); job-summary and fail-on-error behave as for tasks. Artifact upload is left to the workflow.
  • Docs: README, docs/eval-framework.md (CI example, cost limits, full-history checkout, fork-PR guidance), CHANGELOG.
  • Tests: action_eval_test.go. The existing script-safety test is narrowed to catch the eval builtin in command position only, so flags like --eval and outputs like eval-out no longer trip it (with its own positive/negative test).

Checks: gofmt, go vet ./..., go test ./..., GOOS=linux go build ./... all pass.

Generated by a Cobra run (run_mvvnyk0j901m) that was cut off before it could push; finished and opened by hand.

@vicgupta
vicgupta merged commit 2f9d12a into main Oct 31, 2026
1 check passed
@vicgupta vicgupta mentioned this pull request Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GitHub Action: add an eval input to run tinyagent --eval suites and gate CI on pass rate

2 participants