Skip to content

fix(flow-chat): report token throughput from streaming time - #3447

Merged
bobleer merged 2 commits into
GCWing:mainfrom
bobleer:openbitfun/turn-output-throughput-rate
Oct 18, 2026
Merged

bobleer merged 2 commits into
GCWing:mainfrom
bobleer:openbitfun/turn-output-throughput-rate

Conversation

@bobleer

@bobleer bobleer commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • The per-turn t/s readout divided turn-level output tokens by turn wall time, which also covers tool execution, permission waits and retries. On any tool-heavy turn the model looked an order of magnitude slower than it actually generates.
  • Throughput now divides by the streaming window. getTurnOutputDurationMs sums, per model round, streamDurationMs - firstChunkMs (falling back to firstVisibleOutputMs), so request handshake, provider queueing and prompt prefill are not charged to generation. Summing across the turn's rounds keeps the denominator aligned with the turn-level output token count that model rounds already accumulate.
  • Both fields already came from the backend (stream_duration_ms on ModelRoundCompleted) and are persisted/rehydrated; the frontend simply never used them for the rate. Rounds without stream timing (older persisted sessions) keep the previous wall-time fallback instead of reporting an invented number.
  • The turn elapsed time in the completion metadata is unchanged — a duration is not throughput.
  • modelRound.metrics.rateBasis is updated in en-US, zh-CN and zh-TW so the tooltip no longer claims the rate includes tool time.

Fixes the inaccurate X t/s value shown in each turn footer.

Type and Areas

Type: bug fix (UI metric accuracy)

Areas: web UI (flow_chat projection, model round footer, locales)

Motivation / Impact

A turn that spent 90s in tools and 10s streaming 2,000 output tokens displayed roughly 22 t/s instead of 200 t/s. The number is a user-facing quality signal for model/provider speed, so it needs to measure generation, not turn duration.

No persisted shape changes: no new fields on disk, no migration, no config surface. Older sessions without stream_duration_ms behave exactly as before.

Verification

  • pnpm --dir src/web-ui run test:run src/flow_chat/utils/turnUsageMetrics.test.ts src/flow_chat/store/modernFlowChatStore.test.ts — passed (114 tests)
  • pnpm --dir src/web-ui run test:run src/flow_chat/components/modern src/flow_chat/utils/tokenUsageDisplay.test.ts — passed (699 tests)
  • pnpm run type-check:web — passed
  • pnpm run i18n:audit — passed (0 warnings)
  • eslint on the eight changed source files — passed (no errors)

New cases cover: first-chunk subtraction, visible-output fallback, whole-window fallback when no first-output mark exists, multi-round summation, inverted timing reporting nothing instead of a negative window, and missing stream timing yielding undefined so the caller falls back.

Reviewer Notes

  • check:core-boundaries / remote-surface registry are untouched: this is a display-only projection change with no new Tauri command or persisted field.
  • Remote scenarios: no change to protocol, persisted data, or peer/dispatch surfaces. Remote control surfaces render the same footer through the same projection, so they inherit the corrected rate without extra work.
  • Observable change for reviewers: with the corrected denominator the same turn reads much higher (e.g. 8 t/s → 200 t/s), so the speedLevel tier also moves up. The 15/40/80 boundaries were calibrated against the old, deflated numbers, so a normal provider now lands in fast/veryFast more often. Worth a second opinion on whether those thresholds should be retuned in a follow-up; this PR keeps them unchanged so the only variable is the denominator.
  • The t/s value is still absent while a round is actively streaming (streamDurationMs is only known at round completion). That is pre-existing behaviour, not a regression of this PR.

Checklist

  • This PR is focused and does not include secrets, temporary prompts, generated scratch files, or unrelated artifacts.
  • Relevant verification is recorded above, or skipped checks are explained.
  • User-facing strings, docs, and locales are updated where applicable.

Generated with OpenBitFun

bobleer and others added 2 commits October 18, 2026 17:55
The per-turn t/s figure divided output tokens by turn wall time, which also
covers tool execution, permission waits and retries. On tool-heavy turns that
understated the model's actual generation speed by an order of magnitude.

Derive the rate from the streaming window instead. Per model round, subtract
the time to the first upstream item from stream_duration_ms so handshake,
queueing and prefill are not charged to generation, then sum those windows
across the turn's rounds to match the turn-level output token count.

Rounds without stream timing (older persisted sessions) keep the previous
wall-time fallback instead of reporting an invented rate. The turn elapsed
time shown as completion metadata still uses wall time, since that is a
duration, not throughput.

Co-authored-by: OpenBitFun <318544290+bitfun-ai@users.noreply.github.com>
The note described only what the rate excludes. The denominator also starts at
the first token rather than when the stream began, so say that too and keep the
tooltip reproducible from its own text.

Co-authored-by: OpenBitFun <318544290+bitfun-ai@users.noreply.github.com>
@bobleer
bobleer merged commit 1fe2d9d into GCWing:main Oct 18, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant