feat(scorers): validate model parameters

[Cedric / ViaDézo1er (viadezo1er)]

❯ bt scorers create test-invalid-top-k-scorer \
     --slug test-invalid-top-k-scorer \
     --model gpt-4.1 \
     --top-k 10 \
     --messages '[{"role":"user","content":"Return A or B."}]' \
     --choice-scores '{"A":1,"B":0}' \
     --if-exists replace \
     --no-input \
     --json

{"error":{"message":"prompt_data.options.params.top_k: Model parameter \"top_k\" is not supported by model format \"openai\"; suggestion: remove this parameter"}}

AI Summary

Validate LLM scorer definitions through Braintrust as part of creating them.

• send the complete candidate scorer once to POST /insert-functions
• parse structured 422 validation reports and show actionable parameter errors and backend normalization suggestions
• leave model capability, parameter range, and unsupported-parameter checks authoritative on the backend
• support top_k, reasoning enablement, and reasoning budgets in scorer model configuration
• preserve structured provider authentication errors while classifying them as user errors
• reject - input sources on an interactive terminal instead of waiting for EOF

Behavior

bt scorers create uses the existing insertion request for both validation and creation. Invalid definitions fail with the user-error exit code and are not written. With --json, errors retain a stable JSON envelope and provider response details.

Related

Backend support: https://github.com/braintrustdata/braintrust/pull/18155

[github-actions]

Latest downloadable build artifacts for this PR commit 80037f75c1b7:

• Workflow run: https://github.com/braintrustdata/bt/actions/runs/32515582496
• Download all artifacts (GitHub CLI): gh run download 32515582496 --repo braintrustdata/bt
• Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

• artifacts-build-global
• artifacts-build-local-x86_64-apple-darwin
• artifacts-build-local-aarch64-pc-windows-msvc
• artifacts-build-local-x86_64-pc-windows-msvc
• artifacts-build-local-x86_64-unknown-linux-musl
• artifacts-build-local-x86_64-unknown-linux-gnu
• artifacts-build-local-aarch64-unknown-linux-gnu
• artifacts-build-local-aarch64-apple-darwin
• artifacts-plan-dist-manifest
• cargo-dist-cache