fix(database): batch database saves for bulk host writes - #1554
Merged
ZacharyZcR merged 2 commits intoOct 3, 2026
Merged
ZacharyZcR merged 2 commits into
ZacharyZcR merged 2 commits into
Conversation
Each SQLite save serializes and encrypts the whole database into new buffers, which V8 only releases once the event loop turns. Boot migrations and bulk host import write plugin settings per host and per key in an awaited loop, so hundreds of full copies piled up and the container ran out of memory.
On SQLite every repository write force-saves the whole database. Boot plugin data migrations, host defaults materialization and the bulk host routes write one row per host per setting, so they serialized and encrypted the full database hundreds of times in a row. DatabaseSaveTrigger.batched wraps a function so force saves made while it runs collapse into a single save when it finishes. Nested scopes fold into the outer one, and work that outlives its scope saves normally.
LukeGus
added a commit
that referenced
this pull request
Oct 4, 2026
* fix(sso): send the PKCE verifier for GitHub login (#1534) * fix(remote-desktop): clip cursor overflow without disabling zoom (#1535) * test(remote-desktop): preserve sessions beyond one hour (#1536) * fix(hosts): expose controls for overflowing editor tabs (#1537) * fix(hosts): expose controls for overflowing editor tabs * test(hosts): mock resize observation in admin panel tests * fix(status): skip TCP probes while host sessions are active (#1538) * fix(database): batch session activity persistence (#1539) * docs(api): describe cookie and API key authentication (#1540) * fix(snippets): repair missing note column on SQLite upgrades (#1541) * fix(docker): retain stored SSH credential association (#1543) * fix(file-manager): render transfer toasts without plugin hooks (#1546) * feat(hosts): keep compact row actions inline (#1547) * fix(auth): allow retrying unavailable second-factor interfaces (#1549) * fix(auth): reject unresolved legacy identity provisioning (#1550) * fix(desktop): ship the plugin SDK inside Mac App Store builds (#1551) * fix(tmux): pass full session info to the terminal's session picker (#1552) The tmux.sessions service reduced detected sessions to bare names, but the picker reads session.name, so every entry rendered blank and selecting one sent an empty name, which created a new session instead of attaching. * fix(ssh): read force keyboard-interactive as a boolean when resolving hosts (#1553) The column holds the text "true"/"false", and the resolver passed it through as-is. The string "false" is truthy, so the password provider treated every saved host as forced keyboard-interactive and left the password out. Jump hops are built straight from the resolved host, so a password hop whose server doesn't offer keyboard-interactive failed with "All configured authentication methods failed". * fix(database): batch database saves for bulk host writes (#1554) * fix(database): yield to the event loop between database saves Each SQLite save serializes and encrypts the whole database into new buffers, which V8 only releases once the event loop turns. Boot migrations and bulk host import write plugin settings per host and per key in an awaited loop, so hundreds of full copies piled up and the container ran out of memory. * fix(database): save once per bulk host write instead of once per row On SQLite every repository write force-saves the whole database. Boot plugin data migrations, host defaults materialization and the bulk host routes write one row per host per setting, so they serialized and encrypted the full database hundreds of times in a row. DatabaseSaveTrigger.batched wraps a function so force saves made while it runs collapse into a single save when it finishes. Nested scopes fold into the outer one, and work that outlives its scope saves normally. * feat(hosts): add instance-wide predefined tag suggestions (#1548) * feat(hosts): add shared predefined tag suggestions * style(hosts): format tag catalog routes * test(database): advance fake timers past the save yield (#1555) * fix(i18n): complete Simplified Chinese core and plugin translations (#1542) * fix(i18n): complete Chinese onboarding and navigation labels * fix(i18n): translate remaining Chinese core and plugin interfaces * fix(i18n): translate host editor tab overflow controls * fix(i18n): use consistent Chinese fleet terminology * fix(i18n): localize hardcoded controls and plugin views * fix(i18n): translate built-in homepage widget catalog * test(homepage): follow translated timezone placeholder * fix(i18n): translate plugin-provided homepage widgets * fix(i18n): localize feature settings section titles * fix: 2.8 oidc accounts unable to sign in after upgrading (#1381) * fix: plugins losing the saved ssh login when copying a host (#1391) * fix: fleets, proxmox and automations not getting the host sudo password * fix: host imports running a defaults pass and metrics restart per host (#1384) * fix: high cpu from status probes and full database saves on every sample (#1300) * fix: user data export freezing the server (#1393) * fix: op:// secret references rejected as ssh keys on credentials (#1394) * fix: slow file deletes from the trash lookups on every delete (#1390) * fix: add openapi docs and error handling to host tag routes * test: compare download stream buffers directly so it stops timing out * chore: drop em dash from host row comment * chore: update release notes for 2.9.1 * fix: restore space to add host tags and redesign predefined tags editor * fix: host key silently accepted when the host is missing from the database (#1397) * fix: clearer host login failed status label and fix its translations (#1396) * chore: add host key and status label fixes to release notes * fix: sso and ldap provider dialogs overflowing the screen and using mismatched toggles * chore: increment ver * fix(audit): store plugin entries with no acting user as a null user_id (#1556) Plugin audit entries written outside a request used the literal "system" as user_id. That column references users.id, so the insert was refused (Postgres logs it as an FK violation) and the entry was silently dropped. Write null instead and keep "system" / plugin:<id> in username. * chore: add sso/ldap dialog and audit log fixes to release notes * fix: tunnels and host settings missing from shared hosts on desktop * fix: remote desktop logins missing from shared hosts on desktop * fix: simplify host status to online/offline and keep it live without a refresh * chore: sync Crowdin translations for 2.9.1 --------- Co-authored-by: ZacharyZcR <zacharyzcr1984@gmail.com>
LukeGus
added a commit
that referenced
this pull request
Oct 4, 2026
* fix(sso): send the PKCE verifier for GitHub login (#1534) * fix(remote-desktop): clip cursor overflow without disabling zoom (#1535) * test(remote-desktop): preserve sessions beyond one hour (#1536) * fix(hosts): expose controls for overflowing editor tabs (#1537) * fix(hosts): expose controls for overflowing editor tabs * test(hosts): mock resize observation in admin panel tests * fix(status): skip TCP probes while host sessions are active (#1538) * fix(database): batch session activity persistence (#1539) * docs(api): describe cookie and API key authentication (#1540) * fix(snippets): repair missing note column on SQLite upgrades (#1541) * fix(docker): retain stored SSH credential association (#1543) * fix(file-manager): render transfer toasts without plugin hooks (#1546) * feat(hosts): keep compact row actions inline (#1547) * fix(auth): allow retrying unavailable second-factor interfaces (#1549) * fix(auth): reject unresolved legacy identity provisioning (#1550) * fix(desktop): ship the plugin SDK inside Mac App Store builds (#1551) * fix(tmux): pass full session info to the terminal's session picker (#1552) The tmux.sessions service reduced detected sessions to bare names, but the picker reads session.name, so every entry rendered blank and selecting one sent an empty name, which created a new session instead of attaching. * fix(ssh): read force keyboard-interactive as a boolean when resolving hosts (#1553) The column holds the text "true"/"false", and the resolver passed it through as-is. The string "false" is truthy, so the password provider treated every saved host as forced keyboard-interactive and left the password out. Jump hops are built straight from the resolved host, so a password hop whose server doesn't offer keyboard-interactive failed with "All configured authentication methods failed". * fix(database): batch database saves for bulk host writes (#1554) * fix(database): yield to the event loop between database saves Each SQLite save serializes and encrypts the whole database into new buffers, which V8 only releases once the event loop turns. Boot migrations and bulk host import write plugin settings per host and per key in an awaited loop, so hundreds of full copies piled up and the container ran out of memory. * fix(database): save once per bulk host write instead of once per row On SQLite every repository write force-saves the whole database. Boot plugin data migrations, host defaults materialization and the bulk host routes write one row per host per setting, so they serialized and encrypted the full database hundreds of times in a row. DatabaseSaveTrigger.batched wraps a function so force saves made while it runs collapse into a single save when it finishes. Nested scopes fold into the outer one, and work that outlives its scope saves normally. * feat(hosts): add instance-wide predefined tag suggestions (#1548) * feat(hosts): add shared predefined tag suggestions * style(hosts): format tag catalog routes * test(database): advance fake timers past the save yield (#1555) * fix(i18n): complete Simplified Chinese core and plugin translations (#1542) * fix(i18n): complete Chinese onboarding and navigation labels * fix(i18n): translate remaining Chinese core and plugin interfaces * fix(i18n): translate host editor tab overflow controls * fix(i18n): use consistent Chinese fleet terminology * fix(i18n): localize hardcoded controls and plugin views * fix(i18n): translate built-in homepage widget catalog * test(homepage): follow translated timezone placeholder * fix(i18n): translate plugin-provided homepage widgets * fix(i18n): localize feature settings section titles * fix: 2.8 oidc accounts unable to sign in after upgrading (#1381) * fix: plugins losing the saved ssh login when copying a host (#1391) * fix: fleets, proxmox and automations not getting the host sudo password * fix: host imports running a defaults pass and metrics restart per host (#1384) * fix: high cpu from status probes and full database saves on every sample (#1300) * fix: user data export freezing the server (#1393) * fix: op:// secret references rejected as ssh keys on credentials (#1394) * fix: slow file deletes from the trash lookups on every delete (#1390) * fix: add openapi docs and error handling to host tag routes * test: compare download stream buffers directly so it stops timing out * chore: drop em dash from host row comment * chore: update release notes for 2.9.1 * fix: restore space to add host tags and redesign predefined tags editor * fix: host key silently accepted when the host is missing from the database (#1397) * fix: clearer host login failed status label and fix its translations (#1396) * chore: add host key and status label fixes to release notes * fix: sso and ldap provider dialogs overflowing the screen and using mismatched toggles * chore: increment ver * fix(audit): store plugin entries with no acting user as a null user_id (#1556) Plugin audit entries written outside a request used the literal "system" as user_id. That column references users.id, so the insert was refused (Postgres logs it as an FK violation) and the entry was silently dropped. Write null instead and keep "system" / plugin:<id> in username. * chore: add sso/ldap dialog and audit log fixes to release notes * fix: tunnels and host settings missing from shared hosts on desktop * fix: remote desktop logins missing from shared hosts on desktop * fix: simplify host status to online/offline and keep it live without a refresh * chore: sync Crowdin translations for 2.9.1 * fix: build docker frontend/backend natively to stop arm64 hang, build sdk before openapi --------- Co-authored-by: ZacharyZcR <zacharyzcr1984@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On SQLite every repository write force-saves: the in-memory database is serialized and encrypted into fresh buffers and written to disk. 2.9.0 moved per-host options into plugin settings, so the boot plugin data migrations, host defaults materialization (run at boot and after every host write) and the bulk host routes now write one row per host per setting in awaited loops. Each loop rewrote the full database hundreds or thousands of times, and because the loops never yield, V8 could not free the copies in between. The #1380 log shows the container being OOM-killed right after
Loaded 34 plugin(s), which is exactly whererunPluginDataMigrationsruns. In a standalone repro (60 MB database, the same serialize + AES-GCM + write assaveMemoryDatabaseToFile), 100 back-to-back saves peaked at 6.4 GB RSS.Two changes:
DatabaseSaveTrigger.batched(fn)wraps a function so the force saves made while it runs collapse into one save when it finishes (anAsyncLocalStoragescope, so concurrent requests outside it still save immediately). Nested scopes fold into the outer one, and work that outlives its scope (fire-and-forget) saves normally again. Applied torunPluginDataMigrations,materializeHosts, and the/bulk-import,/ssh-config-importand/bulk-updatehandlers. A 50-host import now saves once instead of once per row.runSaveyields one event-loop turn after each save before resolving, so any remaining write loop frees each save's copies before the next one starts (369 MB peak instead of 6.4 GB in the repro above)./database/importis left as is: it already ends with an explicit save whose failure it reports to the caller. For the wrapped routes the single save runs right after the handler returns rather than after each row.Fixes Termix-SSH/Support#1384.
Fixes Termix-SSH/Support#1380.
Validation:
database-save-trigger.test.tscovers collapsing, nested scopes, empty batches, failed batches and work that outlives its scope, plus the event-loop yield; fullvitest run src/backendshows no new failures against the base (remaining failures are pre-existing local environment ones); eslint, prettier 3.9.6 andtsc -p tsconfig.node.jsonpass.