Skip to content

fix(gemini,count_tokens): make the Gemini CLI setup usable and stop stalling on rate-limited token counts - #40

Merged
MartinForReal merged 1 commit into
mainfrom
fix/gemini-base-url-and-count-tokens-429
Aug 9, 2026
Merged

fix(gemini,count_tokens): make the Gemini CLI setup usable and stop stalling on rate-limited token counts#40
MartinForReal merged 1 commit into
mainfrom
fix/gemini-base-url-and-count-tokens-429

Conversation

@MartinForReal

Copy link
Copy Markdown
Owner

Three independent defects, all reachable from a stock --setup --gemini, plus a latency fix on the token-counting path.

1. GOOGLE_GEMINI_BASE_URL carried an API version

Setup wrote http://127.0.0.1:8314/v1beta. The Gen AI SDK appends its own version, so the request URL became /v1beta/v1beta/models/...:

const urlElement = [baseUrl];
if (httpOptions.apiVersion && httpOptions.apiVersion !== '') {
  urlElement.push(httpOptions.apiVersion);   // 'v1beta'
}

Gemini CLI passes only { baseUrl, headers }, so the SDK default survives. Google's documented example is a bare origin (https://my-proxy.com).

Measured before the fix:

path result
/v1beta/models/gemini-2.5-pro:generateContent 400 — route hit, upstream rejects the model
/v1beta/v1beta/models/gemini-2.5-pro:generateContent 404 Not found

Re-running setup rewrites a stale value.

2. The configured model was one Copilot never served

GEMINI_DEFAULT_MODEL was gemini-2.5-pro, and there were no gemini-* mappings at all — so even a correctly routed request returned model_not_supported. The catalog actually offers gemini-3.1-pro-preview, gemini-3.5-flash and gemini-3.6-flash.

A catch-all gemini- prefix now resolves to the pro tier. Flash spellings are listed separately so the existing longest-match rule keeps a flash request on a flash model — otherwise the catch-all would quietly bill the caller for a pricier tier. gemma-* is a different family and is deliberately left alone.

Config version 5 seeds these into existing files via .entry().or_insert_with(), so an id the user already pointed somewhere stays pointed there.

3. count_tokens stalled up to 14s under a rate limit

It used the shared retry helper, which treats 429 as retryable: 2s + 4s + 8s of backoff before falling through to the local estimate it would have produced immediately. Clients call this endpoint before every turn, so the stall repeated on each one.

It now issues a single request, and a 429 pauses upstream counting for 60s rather than re-asking a limiter that has already refused. The endpoint already degrades to a tiktoken estimate, so callers see no error either way.

Verification

Against the live proxy, after migration:

config_version: 5
gemini-2.5-pro    -> "OK"   modelVersion: gemini-3.1-pro-preview
gemini-3.5-flash  -> "OK"   modelVersion: gemini-3.6-flash
count_tokens (upstream) -> {"input_tokens":17}
count_tokens (fallback) -> {"input_tokens":20,"estimated":true}

A stale .env migrates correctly, preserving unrelated vars and a user-set GEMINI_API_KEY:

FOO=bar
GOOGLE_GEMINI_BASE_URL=http://127.0.0.1:8314
GEMINI_API_KEY=my-real-key

167 tests pass; cargo fmt --all --check and cargo clippy --all-targets -- -D warnings are clean.

Deliberately not changed

Auth type. Newer CLIs auto-select a new gateway type from the base URL alone, but the gemini-api-key path still honours GOOGLE_GEMINI_BASE_URL and setup always writes a placeholder key, so the existing value works on both old and new releases. Switching would have broken older installs for no gain.

Context window. Gemini CLI cannot be told a model's window — tokenLimit() is a hardcoded table whose default branch returns 1,048,576, so every model this proxy serves is assumed to be 1M. There is no server-side lever; docs/claude-code.md now documents setting model.compressionThreshold (~0.15 for a 200K model, ~0.1 for 128K) so compaction fires before the upstream rejects the request.

…talling on rate-limited token counts

Three independent defects, all reachable from a stock `--setup --gemini`.

`GOOGLE_GEMINI_BASE_URL` carried an API version. The Gen AI SDK appends
its own, so every request went to `/v1beta/v1beta/models/...` and 404ed
before reaching a handler. Google's own documented example is a bare
origin. Re-running setup rewrites a stale value.

The default model written for the CLI was `gemini-2.5-pro`, which
Copilot has never served, so even a correctly routed request came back
`model_not_supported`. There were no `gemini-*` mappings at all: the
catalog offers `gemini-3.1-pro-preview`, `gemini-3.5-flash` and
`gemini-3.6-flash`, and nothing pointed at them. A catch-all `gemini-`
prefix now resolves to the pro tier, with the flash spellings listed
separately so the longest-match rule keeps a flash request on a flash
model rather than silently upgrading it to a pricier tier. Config
version 5 seeds these into existing files, leaving any id the user
already pointed somewhere untouched.

`/v1/messages/count_tokens` inherited the shared retry helper, so a 429
cost up to 14s of backoff (2s + 4s + 8s) before falling through to the
local estimate it would have produced immediately. Clients call this
endpoint before every turn, so under a rate limit the stall repeated on
each one. It now issues a single request, and a 429 pauses upstream
counting for 60s instead of re-asking a limiter that has already
refused.

Verified against the live proxy: `gemini-2.5-pro` and `gemini-3.5-flash`
both answer and report the expected `modelVersion`, and a stale `.env`
migrates while preserving a user-set `GEMINI_API_KEY`.

The auth type was examined and deliberately left as `gemini-api-key`.
Newer CLIs auto-select the new `gateway` type from the base URL alone,
but the `gemini-api-key` path still honours `GOOGLE_GEMINI_BASE_URL`,
and a placeholder key is always written, so the existing value works on
both old and new releases.
@MartinForReal
MartinForReal merged commit 03bf226 into main Aug 9, 2026
1 check passed
@MartinForReal
MartinForReal deleted the fix/gemini-base-url-and-count-tokens-429 branch August 9, 2026 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant