Skip to content

Count proxied inference tokens in node metrics - #293

Open
webdevtodayjason wants to merge 1 commit into
mainfrom
fable/issue-250
Open

webdevtodayjason wants to merge 1 commit into
mainfrom
fable/issue-250

Conversation

@webdevtodayjason

@webdevtodayjason webdevtodayjason commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem

AINode recorded every proxied request with zero tokens because the proxy discarded upstream usage blocks. Streaming replies, Anthropic Messages replies, and embedding requests therefore left token counters and Prometheus token metrics at zero, and the Metrics view could not truthfully show token throughput.

Decision

Read completion or output token totals from non-streamed JSON replies, parse usage from SSE events without changing forwarded bytes, and count output delta events when a stream has no usage event. Record embedding prompt tokens for both fleet and local embedding paths. Restore the token throughput chart as an interval rate derived from the cumulative counter, while retaining the uptime chart.

The harness counter audit found no code change was needed: it differences counters immediately around each task and already states that concurrent node traffic is included.

Proof

  • 103 proxy, embeddings, routing, UI, and chart tests passed.
  • 168 harness, collector, retention, and Prometheus tests passed.
  • ruff check passed.
  • No GPU hardware is required because fake engines exercise JSON and fragmented SSE transports end to end.

Changelog

Fixed node token metrics and the Metrics token throughput chart for proxied completion, Messages, Responses, streaming, and embedding requests.

Closes #250


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

requests.tokens_generated is a dead counter: the proxy never passes token counts, so tokens/s is 0 on every node

1 participant