Skip to content

perf(chat): send live text in batches, not one event per token - #75

Merged
httpsphl merged 1 commit into
mainfrom
perf/chat-delta
Sep 30, 2026
Merged

httpsphl merged 1 commit into
mainfrom
perf/chat-delta

Conversation

@httpsphl

Copy link
Copy Markdown
Owner

Second slice of the performance round.

What changes

  • Before, each text_delta became its own chat.delta. For every token, each connection serialized a frame (to_value → json! → to_string), and the app re-rendered the chat. A burst could push a connection behind the event bus (Lagged), which closed it, and the app reloaded everything.
  • Now StreamReader gathers the live text and sends one chat.delta:
    • 40 ms after the first piece;
    • or once it reaches 4 KB;
    • or before any other event of the bot, so the text always comes before the reply that replaces it;
    • or when the process exits.
  • The event bus holds 4096 events (was 1024).
  • Spec 8.3 describes the batching.

Not done here (left for later): the fast parse of each line without building a Value.

Tests

  • tests/live_text.rs:
    • three pieces in one chunk arrive as one chat.delta;
    • the batched text arrives before the reply item.
  • tests/chat.rs now accepts pieces arriving joined.
  • cargo fmt, clippy -D warnings, cargo test --workspace (twice) pass locally.

Manual check (real Claude)

  • Dev daemon + app. Ask a bot for a long answer (about 5 paragraphs). The text appears smoothly in small bursts, and when the reply arrives the draft is replaced with nothing missing or doubled.

Every text_delta became its own chat.delta: at tens of tokens a second
per busy bot, each app connection serialized a frame and re-rendered the
chat for each one, and a burst could make a connection lag behind the
event bus and be dropped.

The stream reader now gathers live text and sends it as one chat.delta
40 ms after the first piece, at 4 KB, before any other event of the bot
(so the text always comes before the reply that replaces it) and when
the process exits. The event bus holds 4096 events instead of 1024.
@httpsphl
httpsphl merged commit 86dd217 into main Sep 30, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant