Vercel's default gateway model doesn't work with built-in LLM scorers

[joshpitzalis]

If you try to use Vercel's default gateway model with LLM scorers...

It doesn't work.

However, if you swap in any of the other providers (OpenAI, Google, etc.), it works fine.

[mattpocock]

Thanks for reporting this. All built-in LLM scorers use generateText with structured output (Output.object({ schema })), which requires the provider to support JSON schema mode. This could be why Vercel's default gateway model fails while direct providers (OpenAI, Google) work fine.

Could you share a few more details to help us dig in?

1. Evalite version (e.g. 1.0.0-beta.15)
2. AI SDK version (ai package and any provider packages like @ai-sdk/openai)
3. The exact error message from the screenshots (text form if possible)
4. Which Vercel gateway model you're using (the model ID string passed to the provider)

That will help us figure out whether this is something evalite can work around or a provider limitation.