Repository navigation
fix(flow-chat): report token throughput from streaming time - #3447
Merged
bobleer merged 2 commits intoOct 18, 2026
Merged
Conversation
The per-turn t/s figure divided output tokens by turn wall time, which also covers tool execution, permission waits and retries. On tool-heavy turns that understated the model's actual generation speed by an order of magnitude. Derive the rate from the streaming window instead. Per model round, subtract the time to the first upstream item from stream_duration_ms so handshake, queueing and prefill are not charged to generation, then sum those windows across the turn's rounds to match the turn-level output token count. Rounds without stream timing (older persisted sessions) keep the previous wall-time fallback instead of reporting an invented rate. The turn elapsed time shown as completion metadata still uses wall time, since that is a duration, not throughput. Co-authored-by: OpenBitFun <318544290+bitfun-ai@users.noreply.github.com>
The note described only what the rate excludes. The denominator also starts at the first token rather than when the stream began, so say that too and keep the tooltip reproducible from its own text. Co-authored-by: OpenBitFun <318544290+bitfun-ai@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
t/sreadout divided turn-level output tokens by turn wall time, which also covers tool execution, permission waits and retries. On any tool-heavy turn the model looked an order of magnitude slower than it actually generates.getTurnOutputDurationMssums, per model round,streamDurationMs - firstChunkMs(falling back tofirstVisibleOutputMs), so request handshake, provider queueing and prompt prefill are not charged to generation. Summing across the turn's rounds keeps the denominator aligned with the turn-level output token count that model rounds already accumulate.stream_duration_msonModelRoundCompleted) and are persisted/rehydrated; the frontend simply never used them for the rate. Rounds without stream timing (older persisted sessions) keep the previous wall-time fallback instead of reporting an invented number.modelRound.metrics.rateBasisis updated inen-US,zh-CNandzh-TWso the tooltip no longer claims the rate includes tool time.Fixes the inaccurate
X t/svalue shown in each turn footer.Type and Areas
Type: bug fix (UI metric accuracy)
Areas: web UI (
flow_chatprojection, model round footer, locales)Motivation / Impact
A turn that spent 90s in tools and 10s streaming 2,000 output tokens displayed roughly
22 t/sinstead of200 t/s. The number is a user-facing quality signal for model/provider speed, so it needs to measure generation, not turn duration.No persisted shape changes: no new fields on disk, no migration, no config surface. Older sessions without
stream_duration_msbehave exactly as before.Verification
pnpm --dir src/web-ui run test:run src/flow_chat/utils/turnUsageMetrics.test.ts src/flow_chat/store/modernFlowChatStore.test.ts— passed (114 tests)pnpm --dir src/web-ui run test:run src/flow_chat/components/modern src/flow_chat/utils/tokenUsageDisplay.test.ts— passed (699 tests)pnpm run type-check:web— passedpnpm run i18n:audit— passed (0 warnings)eslinton the eight changed source files — passed (no errors)New cases cover: first-chunk subtraction, visible-output fallback, whole-window fallback when no first-output mark exists, multi-round summation, inverted timing reporting nothing instead of a negative window, and missing stream timing yielding
undefinedso the caller falls back.Reviewer Notes
check:core-boundaries/ remote-surface registry are untouched: this is a display-only projection change with no new Tauri command or persisted field.speedLeveltier also moves up. The 15/40/80 boundaries were calibrated against the old, deflated numbers, so a normal provider now lands infast/veryFastmore often. Worth a second opinion on whether those thresholds should be retuned in a follow-up; this PR keeps them unchanged so the only variable is the denominator.t/svalue is still absent while a round is actively streaming (streamDurationMsis only known at round completion). That is pre-existing behaviour, not a regression of this PR.Checklist
Generated with OpenBitFun