Add BlazeRail to the AI gateway benchmarks (Anthropic, Gemini, Kimi families) - #361
Open
nazerfaze wants to merge 1 commit into
Open
Add BlazeRail to the AI gateway benchmarks (Anthropic, Gemini, Kimi families)#361nazerfaze wants to merge 1 commit into
nazerfaze wants to merge 1 commit into
Conversation
BlazeRail (https://blazerail.com) is an OpenAI-compatible AI gateway routing 220+ text models and 1,000+ media endpoints, billed at invoiced vendor rates. Entries follow the existing config conventions: - wireFormat 'openai' via /v1/chat/completions on all three families (the gateway translates Anthropic/Gemini SSE to OpenAI chunks natively; no gateway-through-gateway proxying) - anthropic/claude-haiku-4.5, google/gemini-3.6-flash and moonshotai/kimi-k3 are BlazeRail's public catalog ids for the exact target models - gemini-3.6-flash is served by exactly one upstream (Google AI Studio direct), so that route is deterministic without a provider pin; claude-haiku-4.5 and kimi-k3 are multi-upstream and routed by measured health/price, which is the product behavior being benchmarked (same posture as other multi-upstream participants) All three entries validated against the live API with this harness (BENCHSDK_NO_INGEST=1, --provider blazerail): 20/20 iterations per family, zero failures. Happy to provide an API key with standing credit for the weekly runs - contact info@deltaedgecapital.co.uk or the X account that pinged you.
Contributor License AgreementThe following contributors need CLA coverage: |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+216
to
+225
| name: 'blazerail', | ||
| requiredEnvVars: ['BLAZERAIL_API_KEY'], | ||
| wireFormat: 'openai', | ||
| model: 'moonshotai/kimi-k3', | ||
| host: 'api.blazerail.com', | ||
| path: '/v1/chat/completions', | ||
| buildHeaders: () => ({ | ||
| Authorization: `Bearer ${process.env.BLAZERAIL_API_KEY}`, | ||
| }), | ||
| }, |
Contributor
There was a problem hiding this comment.
🔴 BlazeRail Kimi requests send a rejected temperature
This entry omits extraBody, so the request keeps the default temperature: 0 from buildRequestBody (benchmarks/ai-gateway/phase-probe.ts:82). Every sibling Kimi entry strips it with temperature: undefined because kimi-k3 rejects any temperature but 1, so every BlazeRail Kimi iteration fails.
Suggested change
| name: 'blazerail', | |
| requiredEnvVars: ['BLAZERAIL_API_KEY'], | |
| wireFormat: 'openai', | |
| model: 'moonshotai/kimi-k3', | |
| host: 'api.blazerail.com', | |
| path: '/v1/chat/completions', | |
| buildHeaders: () => ({ | |
| Authorization: `Bearer ${process.env.BLAZERAIL_API_KEY}`, | |
| }), | |
| }, | |
| name: 'blazerail', | |
| requiredEnvVars: ['BLAZERAIL_API_KEY'], | |
| wireFormat: 'openai', | |
| model: 'moonshotai/kimi-k3', | |
| host: 'api.blazerail.com', | |
| path: '/v1/chat/completions', | |
| buildHeaders: () => ({ | |
| Authorization: `Bearer ${process.env.BLAZERAIL_API_KEY}`, | |
| }), | |
| extraBody: { | |
| temperature: undefined, | |
| }, | |
| reasoningCountsAsFirstToken: true, | |
| }, |
Was this helpful? React with 👍 or 👎 to provide feedback.
Comment on lines
+216
to
+225
| name: 'blazerail', | ||
| requiredEnvVars: ['BLAZERAIL_API_KEY'], | ||
| wireFormat: 'openai', | ||
| model: 'moonshotai/kimi-k3', | ||
| host: 'api.blazerail.com', | ||
| path: '/v1/chat/completions', | ||
| buildHeaders: () => ({ | ||
| Authorization: `Bearer ${process.env.BLAZERAIL_API_KEY}`, | ||
| }), | ||
| }, |
Contributor
There was a problem hiding this comment.
🔴 BlazeRail Kimi TTFT measures first visible token, not reasoning
This entry omits reasoningCountsAsFirstToken: true that every other openai Kimi entry sets, so contentRegexFor uses the non-reasoning pattern (benchmarks/ai-gateway/phase-probe.ts:64). Because kimi-k3 reasons before answering, TTFT captures the whole reasoning phase and is inflated far above the other participants.
Was this helpful? React with 👍 or 👎 to provide feedback.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Following up on the X ping: BlazeRail is an OpenAI-compatible AI gateway (220+ text models, media endpoints, invoiced vendor rates, no markup) and we'd like to join the gateway benchmarks.
What's in this PR -
blazerailprovider entries for the three families we serve today, in each file's existing config shape:providers.ts(Anthropic family):anthropic/claude-haiku-4.5via/v1/chat/completions, wireFormatopenai- the gateway translates Anthropic's native SSE to OpenAI chunks itself, no gateway-through-gateway proxyingproviders-gemini.ts:google/gemini-3.6-flash- served by exactly one upstream (Google AI Studio direct), deterministic without a provider pinproviders-kimi.ts:moonshotai/kimi-k3- multi-upstream, routed by measured health/price like the other multi-upstream participantsValidation: each entry was run against the live API with this harness (
BENCHSDK_NO_INGEST=1 ... --provider blazerail --iterations 10) - 20/20 iterations per family, zero failures, receipts carrying cf-ray ids.Credentials: we'll provide a dedicated API key with standing credit for the weekly runs - tell us where to send it (email in the commit message, or DM the X account that pinged you).
Requires
BLAZERAIL_API_KEY; per the harness's env-gating, the entries are inert until you set it.