Skip to content

fix(database): batch database saves for bulk host writes - #1554

Merged
ZacharyZcR merged 2 commits into
Termix-SSH:dev-2.9.1from
ZacharyZcR:fix/support-1384-host-index-memory
Oct 3, 2026
Merged

ZacharyZcR merged 2 commits into
Termix-SSH:dev-2.9.1from
ZacharyZcR:fix/support-1384-host-index-memory

Conversation

@ZacharyZcR

@ZacharyZcR ZacharyZcR commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

On SQLite every repository write force-saves: the in-memory database is serialized and encrypted into fresh buffers and written to disk. 2.9.0 moved per-host options into plugin settings, so the boot plugin data migrations, host defaults materialization (run at boot and after every host write) and the bulk host routes now write one row per host per setting in awaited loops. Each loop rewrote the full database hundreds or thousands of times, and because the loops never yield, V8 could not free the copies in between. The #1380 log shows the container being OOM-killed right after Loaded 34 plugin(s), which is exactly where runPluginDataMigrations runs. In a standalone repro (60 MB database, the same serialize + AES-GCM + write as saveMemoryDatabaseToFile), 100 back-to-back saves peaked at 6.4 GB RSS.

Two changes:

  • DatabaseSaveTrigger.batched(fn) wraps a function so the force saves made while it runs collapse into one save when it finishes (an AsyncLocalStorage scope, so concurrent requests outside it still save immediately). Nested scopes fold into the outer one, and work that outlives its scope (fire-and-forget) saves normally again. Applied to runPluginDataMigrations, materializeHosts, and the /bulk-import, /ssh-config-import and /bulk-update handlers. A 50-host import now saves once instead of once per row.
  • runSave yields one event-loop turn after each save before resolving, so any remaining write loop frees each save's copies before the next one starts (369 MB peak instead of 6.4 GB in the repro above).

/database/import is left as is: it already ends with an explicit save whose failure it reports to the caller. For the wrapped routes the single save runs right after the handler returns rather than after each row.

Fixes Termix-SSH/Support#1384.
Fixes Termix-SSH/Support#1380.

Validation: database-save-trigger.test.ts covers collapsing, nested scopes, empty batches, failed batches and work that outlives its scope, plus the event-loop yield; full vitest run src/backend shows no new failures against the base (remaining failures are pre-existing local environment ones); eslint, prettier 3.9.6 and tsc -p tsconfig.node.json pass.

Each SQLite save serializes and encrypts the whole database into new
buffers, which V8 only releases once the event loop turns. Boot
migrations and bulk host import write plugin settings per host and per
key in an awaited loop, so hundreds of full copies piled up and the
container ran out of memory.
On SQLite every repository write force-saves the whole database. Boot
plugin data migrations, host defaults materialization and the bulk
host routes write one row per host per setting, so they serialized and
encrypted the full database hundreds of times in a row.

DatabaseSaveTrigger.batched wraps a function so force saves made while
it runs collapse into a single save when it finishes. Nested scopes fold
into the outer one, and work that outlives its scope saves normally.
@ZacharyZcR ZacharyZcR changed the title fix(database): yield to the event loop between database saves fix(database): batch database saves for bulk host writes Oct 3, 2026
@ZacharyZcR
ZacharyZcR merged commit 94a9f81 into Termix-SSH:dev-2.9.1 Oct 3, 2026
3 checks passed
LukeGus added a commit that referenced this pull request Oct 4, 2026
* fix(sso): send the PKCE verifier for GitHub login (#1534)

* fix(remote-desktop): clip cursor overflow without disabling zoom (#1535)

* test(remote-desktop): preserve sessions beyond one hour (#1536)

* fix(hosts): expose controls for overflowing editor tabs (#1537)

* fix(hosts): expose controls for overflowing editor tabs

* test(hosts): mock resize observation in admin panel tests

* fix(status): skip TCP probes while host sessions are active (#1538)

* fix(database): batch session activity persistence (#1539)

* docs(api): describe cookie and API key authentication (#1540)

* fix(snippets): repair missing note column on SQLite upgrades (#1541)

* fix(docker): retain stored SSH credential association (#1543)

* fix(file-manager): render transfer toasts without plugin hooks (#1546)

* feat(hosts): keep compact row actions inline (#1547)

* fix(auth): allow retrying unavailable second-factor interfaces (#1549)

* fix(auth): reject unresolved legacy identity provisioning (#1550)

* fix(desktop): ship the plugin SDK inside Mac App Store builds (#1551)

* fix(tmux): pass full session info to the terminal's session picker (#1552)

The tmux.sessions service reduced detected sessions to bare names, but the
picker reads session.name, so every entry rendered blank and selecting one
sent an empty name, which created a new session instead of attaching.

* fix(ssh): read force keyboard-interactive as a boolean when resolving hosts (#1553)

The column holds the text "true"/"false", and the resolver passed it
through as-is. The string "false" is truthy, so the password provider
treated every saved host as forced keyboard-interactive and left the
password out. Jump hops are built straight from the resolved host, so a
password hop whose server doesn't offer keyboard-interactive failed with
"All configured authentication methods failed".

* fix(database): batch database saves for bulk host writes (#1554)

* fix(database): yield to the event loop between database saves

Each SQLite save serializes and encrypts the whole database into new
buffers, which V8 only releases once the event loop turns. Boot
migrations and bulk host import write plugin settings per host and per
key in an awaited loop, so hundreds of full copies piled up and the
container ran out of memory.

* fix(database): save once per bulk host write instead of once per row

On SQLite every repository write force-saves the whole database. Boot
plugin data migrations, host defaults materialization and the bulk
host routes write one row per host per setting, so they serialized and
encrypted the full database hundreds of times in a row.

DatabaseSaveTrigger.batched wraps a function so force saves made while
it runs collapse into a single save when it finishes. Nested scopes fold
into the outer one, and work that outlives its scope saves normally.

* feat(hosts): add instance-wide predefined tag suggestions (#1548)

* feat(hosts): add shared predefined tag suggestions

* style(hosts): format tag catalog routes

* test(database): advance fake timers past the save yield (#1555)

* fix(i18n): complete Simplified Chinese core and plugin translations (#1542)

* fix(i18n): complete Chinese onboarding and navigation labels

* fix(i18n): translate remaining Chinese core and plugin interfaces

* fix(i18n): translate host editor tab overflow controls

* fix(i18n): use consistent Chinese fleet terminology

* fix(i18n): localize hardcoded controls and plugin views

* fix(i18n): translate built-in homepage widget catalog

* test(homepage): follow translated timezone placeholder

* fix(i18n): translate plugin-provided homepage widgets

* fix(i18n): localize feature settings section titles

* fix: 2.8 oidc accounts unable to sign in after upgrading (#1381)

* fix: plugins losing the saved ssh login when copying a host (#1391)

* fix: fleets, proxmox and automations not getting the host sudo password

* fix: host imports running a defaults pass and metrics restart per host (#1384)

* fix: high cpu from status probes and full database saves on every sample (#1300)

* fix: user data export freezing the server (#1393)

* fix: op:// secret references rejected as ssh keys on credentials (#1394)

* fix: slow file deletes from the trash lookups on every delete (#1390)

* fix: add openapi docs and error handling to host tag routes

* test: compare download stream buffers directly so it stops timing out

* chore: drop em dash from host row comment

* chore: update release notes for 2.9.1

* fix: restore space to add host tags and redesign predefined tags editor

* fix: host key silently accepted when the host is missing from the database (#1397)

* fix: clearer host login failed status label and fix its translations (#1396)

* chore: add host key and status label fixes to release notes

* fix: sso and ldap provider dialogs overflowing the screen and using mismatched toggles

* chore: increment ver

* fix(audit): store plugin entries with no acting user as a null user_id (#1556)

Plugin audit entries written outside a request used the literal "system"
as user_id. That column references users.id, so the insert was refused
(Postgres logs it as an FK violation) and the entry was silently dropped.
Write null instead and keep "system" / plugin:<id> in username.

* chore: add sso/ldap dialog and audit log fixes to release notes

* fix: tunnels and host settings missing from shared hosts on desktop

* fix: remote desktop logins missing from shared hosts on desktop

* fix: simplify host status to online/offline and keep it live without a refresh

* chore: sync Crowdin translations for 2.9.1

---------

Co-authored-by: ZacharyZcR <zacharyzcr1984@gmail.com>
LukeGus added a commit that referenced this pull request Oct 4, 2026
* fix(sso): send the PKCE verifier for GitHub login (#1534)

* fix(remote-desktop): clip cursor overflow without disabling zoom (#1535)

* test(remote-desktop): preserve sessions beyond one hour (#1536)

* fix(hosts): expose controls for overflowing editor tabs (#1537)

* fix(hosts): expose controls for overflowing editor tabs

* test(hosts): mock resize observation in admin panel tests

* fix(status): skip TCP probes while host sessions are active (#1538)

* fix(database): batch session activity persistence (#1539)

* docs(api): describe cookie and API key authentication (#1540)

* fix(snippets): repair missing note column on SQLite upgrades (#1541)

* fix(docker): retain stored SSH credential association (#1543)

* fix(file-manager): render transfer toasts without plugin hooks (#1546)

* feat(hosts): keep compact row actions inline (#1547)

* fix(auth): allow retrying unavailable second-factor interfaces (#1549)

* fix(auth): reject unresolved legacy identity provisioning (#1550)

* fix(desktop): ship the plugin SDK inside Mac App Store builds (#1551)

* fix(tmux): pass full session info to the terminal's session picker (#1552)

The tmux.sessions service reduced detected sessions to bare names, but the
picker reads session.name, so every entry rendered blank and selecting one
sent an empty name, which created a new session instead of attaching.

* fix(ssh): read force keyboard-interactive as a boolean when resolving hosts (#1553)

The column holds the text "true"/"false", and the resolver passed it
through as-is. The string "false" is truthy, so the password provider
treated every saved host as forced keyboard-interactive and left the
password out. Jump hops are built straight from the resolved host, so a
password hop whose server doesn't offer keyboard-interactive failed with
"All configured authentication methods failed".

* fix(database): batch database saves for bulk host writes (#1554)

* fix(database): yield to the event loop between database saves

Each SQLite save serializes and encrypts the whole database into new
buffers, which V8 only releases once the event loop turns. Boot
migrations and bulk host import write plugin settings per host and per
key in an awaited loop, so hundreds of full copies piled up and the
container ran out of memory.

* fix(database): save once per bulk host write instead of once per row

On SQLite every repository write force-saves the whole database. Boot
plugin data migrations, host defaults materialization and the bulk
host routes write one row per host per setting, so they serialized and
encrypted the full database hundreds of times in a row.

DatabaseSaveTrigger.batched wraps a function so force saves made while
it runs collapse into a single save when it finishes. Nested scopes fold
into the outer one, and work that outlives its scope saves normally.

* feat(hosts): add instance-wide predefined tag suggestions (#1548)

* feat(hosts): add shared predefined tag suggestions

* style(hosts): format tag catalog routes

* test(database): advance fake timers past the save yield (#1555)

* fix(i18n): complete Simplified Chinese core and plugin translations (#1542)

* fix(i18n): complete Chinese onboarding and navigation labels

* fix(i18n): translate remaining Chinese core and plugin interfaces

* fix(i18n): translate host editor tab overflow controls

* fix(i18n): use consistent Chinese fleet terminology

* fix(i18n): localize hardcoded controls and plugin views

* fix(i18n): translate built-in homepage widget catalog

* test(homepage): follow translated timezone placeholder

* fix(i18n): translate plugin-provided homepage widgets

* fix(i18n): localize feature settings section titles

* fix: 2.8 oidc accounts unable to sign in after upgrading (#1381)

* fix: plugins losing the saved ssh login when copying a host (#1391)

* fix: fleets, proxmox and automations not getting the host sudo password

* fix: host imports running a defaults pass and metrics restart per host (#1384)

* fix: high cpu from status probes and full database saves on every sample (#1300)

* fix: user data export freezing the server (#1393)

* fix: op:// secret references rejected as ssh keys on credentials (#1394)

* fix: slow file deletes from the trash lookups on every delete (#1390)

* fix: add openapi docs and error handling to host tag routes

* test: compare download stream buffers directly so it stops timing out

* chore: drop em dash from host row comment

* chore: update release notes for 2.9.1

* fix: restore space to add host tags and redesign predefined tags editor

* fix: host key silently accepted when the host is missing from the database (#1397)

* fix: clearer host login failed status label and fix its translations (#1396)

* chore: add host key and status label fixes to release notes

* fix: sso and ldap provider dialogs overflowing the screen and using mismatched toggles

* chore: increment ver

* fix(audit): store plugin entries with no acting user as a null user_id (#1556)

Plugin audit entries written outside a request used the literal "system"
as user_id. That column references users.id, so the insert was refused
(Postgres logs it as an FK violation) and the entry was silently dropped.
Write null instead and keep "system" / plugin:<id> in username.

* chore: add sso/ldap dialog and audit log fixes to release notes

* fix: tunnels and host settings missing from shared hosts on desktop

* fix: remote desktop logins missing from shared hosts on desktop

* fix: simplify host status to online/offline and keep it live without a refresh

* chore: sync Crowdin translations for 2.9.1

* fix: build docker frontend/backend natively to stop arm64 hang, build sdk before openapi

---------

Co-authored-by: ZacharyZcR <zacharyzcr1984@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant