Repository navigation
perf(store): stop waiting for the disk and scanning whole tables - #74
Merged
Merged
Conversation
- synchronous=NORMAL: in WAL mode it never corrupts the database, and a commit no longer waits for FlushFileBuffers. With FULL every chat item, read receipt and delivery paid a flush while holding the store lock. - temp_store=MEMORY, a 16 MB page cache, a WAL size limit and a bounded PRAGMA optimize on open. - Statement cache of 128; statements run on every chat item, delivery, message and session lookup go through it instead of being parsed and planned each time. - Migration 0012 indexes every foreign key that had none (deleting a bot or crew scanned the child tables once per row), the tool lookup that each tool result and permission prompt runs, the files a bot wrote, a routine's runs in rowid order and the routines that can come due. - delivery_backlog counts through the state index; the routine run queries use EXISTS so they walk only the open runs. - A test checks the query plans of the hot queries and the foreign keys.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First slice of the performance round.
What changes
synchronous=NORMAL(WAL). Before, every commit waited forFlushFileBuffers(5–30 ms on Windows) while holding the only store mutex, on every chat item,mark_readand delivery claim/finish. NORMAL never corrupts a WAL database; a power cut can lose the last commits.temp_store=MEMORY, a 16 MB cache,journal_size_limit, and a boundedPRAGMA optimizeon open.tool_item, deliveries, messages,session_id, plus everyprepare.toolUseIdlookup that each tool result and permission prompt runs;written_files;routines_dueas a partial index.delivery_backlog(polled every 15 s) counts through the state index.runs_with_dead_deliveryandstale_runsuseEXISTS.Tests
plans.rsrunsEXPLAIN QUERY PLANon the hot queries and the foreign-key lookups, and fails on a full scan or a sort.cargo fmt,clippy -D warnings,cargo test --workspacepass locally.Manual check (real install)
botloft.dbwith this build. Migration 0012 runs, bots, chats and routines load as before, and a bot's chat keeps streaming.