Skip to content

feat: Go eval support #305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
David Elner (delner) wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat: Go eval support #305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ Remove-Item -Recurse -Force (Join-Path $env:APPDATA "bt") -ErrorAction SilentlyC
- `bt eval "tests/**/*.eval.ts"` — glob pattern
- `bt eval a.eval.ts b.eval.ts` — one or more explicit files

Files inside `node_modules`, `.venv`, `venv`, `site-packages`, `dist-packages`, and `__pycache__` are excluded from automatic discovery. Explicit paths and globs bypass these exclusions.
Files inside `node_modules`, `.venv`, `venv`, `site-packages`, `dist-packages`, `__pycache__`, and `vendor` are excluded from automatic discovery. Explicit paths and globs bypass these exclusions.

**Runners:**

Expand All @@ -173,6 +173,25 @@ Files inside `node_modules`, `.venv`, `venv`, `site-packages`, `dist-packages`,
- If eval execution fails with ESM/top-level-await related errors, retry with:
- `bt eval --runner vite-node tutorial.eval.ts`

**Go evals:**

Go works differently from JavaScript and Python. There is no runtime loading in Go, so `bt` does not
supply the program — your compiled package _is_ the eval runner. Write it with the
[Go SDK](https://github.com/braintrustdata/braintrust-sdk-go)'s `evalrunner` package, then:

- `bt eval --language go ./cmd/evals` — point `bt` at the **package directory**. Go's compilation
unit is a directory, not a file, so all inputs must resolve to a single package.
- `bt eval ./cmd/evals` — works without `--language` when the package contains a conventionally
named `*_eval.go` file, which `bt` uses as a discovery marker.
- `bt eval --language go --runner ./bin/evals` — run a prebuilt binary instead of compiling. The
binary is spawned with no arguments; the Go runner takes all its input from the environment.
- `BT_EVAL_GO_BIN` / `BT_EVAL_GO` override which `go` toolchain is used; otherwise `bt` looks at
`GOROOT` and then `PATH`.

Two caveats. The first `go run` of a package with real dependencies pays compile time before
anything is reported. And `--watch` re-runs on changes to `*.go` files in the package directory
only — `bt` cannot follow Go imports into other packages.

**Passing arguments to the eval file:**

Use `--` to forward extra arguments to the eval file via `process.argv`:
Expand Down
Loading
Loading