Repository navigation
perf(chat): send live text in batches, not one event per token - #75
Merged
Merged
Conversation
Every text_delta became its own chat.delta: at tens of tokens a second per busy bot, each app connection serialized a frame and re-rendered the chat for each one, and a burst could make a connection lag behind the event bus and be dropped. The stream reader now gathers live text and sends it as one chat.delta 40 ms after the first piece, at 4 KB, before any other event of the bot (so the text always comes before the reply that replaces it) and when the process exits. The event bus holds 4096 events instead of 1024.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second slice of the performance round.
What changes
text_deltabecame its ownchat.delta. For every token, each connection serialized a frame (to_value→json!→to_string), and the app re-rendered the chat. A burst could push a connection behind the event bus (Lagged), which closed it, and the app reloaded everything.StreamReadergathers the live text and sends onechat.delta:replythat replaces it;Not done here (left for later): the fast parse of each line without building a
Value.Tests
tests/live_text.rs:chat.delta;replyitem.tests/chat.rsnow accepts pieces arriving joined.cargo fmt,clippy -D warnings,cargo test --workspace(twice) pass locally.Manual check (real Claude)