psmux backend (performance): default warm spares and standby add about 0.75 GiB of summed working sets per idle project (3.4.0) #903

Summary

On psmux 3.4.0 each psmux server keeps three spare shells ready (warm-pool-size defaults to 3), and each registry keeps a hidden __warm__ standby server with its own shell and spare. In the ordinary fresh-registry launch path measured here, bmad-loop's command-bearing new_window and new_parked_window calls bypass psmux's spare-pane fast path. Its new_session calls pass -c, so they also bypass the ordinary standby-claim path. Held-standby claims are a separate source-level exception; manually created default-shell windows can consume the pool. Measured through bmad-loop's own launch path, the idle control session of one project costs 16 processes and about 950 MiB of summed working sets with the pool, and 4 processes and about 170 MiB with PSMUX_NO_WARM=1. Each live run session costs about 920 MiB against about 490 MiB. In the measured simulated lifecycle (one project with the TUI's control session open and one run session in flight) the sum is about 2.1 GiB, of which about 1.2 GiB are spare shells and standby. These sums can count shared pages more than once; they are not a direct measurement of unique physical RAM saved. On a memory-constrained Windows machine, this is material overhead.

Environment

psmux 3.4.0 (20aa1d7), Windows 11 10.0.26200.9457, PowerShell 7.6.6 (launching shell; SHELL removed); bmad-loop 4c00c37 (the 3.4.0 floor, #900).

Steps to reproduce

Save as repro.py in a bmad-loop checkout (needs psutil, which the dev environment already has) and run it from PowerShell 7. It drives bmad-loop's own tui.launch._ensure_ctl_session and tui.launch.start_detached (the TUI's launch path, with a --help tail so no coding CLI starts and zero tokens are spent) in a throwaway project with a fresh isolated registry, then measures that registry's psmux servers and every descendant. Memory is the sum of process working sets (psutil RSS) in MiB.

# Idle cost of one bmad-loop control session on psmux 3.4.0, with and without
# psmux's warm pool. Usage (PowerShell 7, SHELL unset, in a bmad-loop checkout):
#   uv run python repro.py default    |    uv run python repro.py nowarm
# Zero tokens: the only window runs `bmad-loop --help` and parks.
import os, shutil, subprocess, sys, tempfile, time
from pathlib import Path
import psutil

mode = sys.argv[1]
base = Path(tempfile.gettempdir()) / f"bl-warm-repro-{mode}"
shutil.rmtree(base, ignore_errors=True)
proj = base / "proj"
proj.mkdir(parents=True)
subprocess.run(["git", "init", "-q", str(proj)], check=True)
os.environ.pop("SHELL", None)
os.environ.pop("PSMUX_DATA_DIR", None)
os.environ.pop("PSMUX_NO_WARM", None)  # make "default" really psmux's default
os.environ.pop("PSMUX_WARM_POOL_SIZE", None)
os.environ["BMAD_LOOP_STATE_DIR"] = str(base / "state")
os.environ["BMAD_LOOP_MUX_BACKEND"] = "psmux"
if mode == "nowarm":
    os.environ["PSMUX_NO_WARM"] = "1"

from bmad_loop import runs
from bmad_loop.tui import launch

data = runs.export_psmux_registry_root(proj)  # the project's own registry, under base
assert data and data.startswith(str(base)), data


def tree():
    procs, servers = {}, []
    for p in psutil.process_iter(["name"]):
        try:
            if (p.info["name"] or "").lower() == "psmux.exe" and p.environ().get("PSMUX_DATA_DIR") == data:
                servers.append(" ".join(p.cmdline()[1:4]))
                for q in [p, *p.children(recursive=True)]:
                    procs[q.pid] = q
        except psutil.Error:
            pass
    mib = sum(q.memory_info().rss for q in procs.values() if q.is_running()) // 2**20
    return f"{len(procs)} processes, {mib} MiB summed working set, servers={sorted(servers)}"


launch._ensure_ctl_session(proj)
launch.start_detached(proj, ["--help"], "20261013-repro", "run")
time.sleep(15)
print("control session + one parked window:", tree())
subprocess.run(["psmux", "kill-session", "-t", "bmad-loop-ctl"], check=True)
time.sleep(10)
print("after kill-session -t bmad-loop-ctl:   ", tree())
subprocess.run(["psmux", "kill-server"], capture_output=True)  # this registry only
time.sleep(5)
print("after kill-server:                     ", tree())

Launched from PowerShell 7:

Remove-Item Env:SHELL -ErrorAction SilentlyContinue
$env:PYTHONUTF8 = '1'
uv run python repro.py default   # rc 0
uv run python repro.py nowarm    # rc 0, the control for what bmad-loop could save

Output, first of three runs per mode (the other two are within 4%):

default: control session + one parked window: 19 processes, 1197 MiB summed working set, servers=['server -s __warm__', 'server -s bmad-loop-ctl']
default: after kill-session -t bmad-loop-ctl:    6 processes, 326 MiB summed working set, servers=['server -s __warm__']
default: after kill-server:                      0 processes, 0 MiB summed working set, servers=[]
nowarm:  control session + one parked window: 7 processes, 408 MiB summed working set, servers=['server -s bmad-loop-ctl']
nowarm:  after kill-session -t bmad-loop-ctl:    0 processes, 0 MiB summed working set, servers=[]
nowarm:  after kill-server:                      0 processes, 0 MiB summed working set, servers=[]

The whole lifecycle, measured the same way through bmad-loop's own calls (three runs per mode; processes and summed working set in MiB of the project registry's psmux servers and all their descendants, 5 to 15 s after each step):

Observed alongside: no kill path left an orphaned pwsh or conhost (every process seen in the tree was either still in it or gone, in every mode, over the ten runs that carried this check), and a kill-server on the registry ended everything. No surviving tracked descendant was observed after the measured teardown paths; the cost is the steady state. PSMUX_WARM_POOL_SIZE=0 removes the spares but not the __warm__ standby; only PSMUX_NO_WARM=1 removes both.

Two projects in one state root (two registries) each keep their own standby, 6 processes and about 315 MiB each, and a project's standby stayed alive for the 60 s observed after its control session was killed (psmux ls in that registry: no server running, rc 1). No bmad-loop command ends it: none runs kill-server, and cleanup does not see it (the standby is hidden from list-sessions by design).

Expected vs actual

Source

bmad-loop, every session create passes -c (src/bmad_loop/adapters/psmux_backend.py:651-666 at 4c00c37), and it forwards this process's environment, with no warm setting, to the call that cold-starts the server (:638-646):

            proc = self._run(
                [
                    "new-session",
                    "-d",
                    "-s",
                    name,
                    "-n",
                    self._INITIAL_WINDOW_NAME,
                    "-c",
                    str(cwd),
                    *geometry,
                ],

Every explicit new_window and new_parked_window call carries a command; new_session creates its initial default-shell window without an explicit command. new_window appends self._window_launch(env, command) and new_parked_window appends self._shell_wrap(source) (src/bmad_loop/adapters/tmux_base.py:540-581; the psmux overrides at psmux_backend.py:701 and :731 delegate to them).

psmux, at v3.4.0 (20aa1d7), the two gates (source-read):

// src/pane.rs:986 (create_window; the split path repeats it at :1865)
let warm_eligible = command.is_none() && extra_env.is_empty() && !default_shell_needs_fresh_eval(&app.default_shell);
// src/main.rs:2198 (new-session's standby claim)
let claimed_warm = if held_claim || (!warm_disabled && !has_custom_config && initial_cmd.is_none() && raw_cmd_args.is_none() && start_dir.is_none() && env_vars.is_empty() && init_width.is_none() && init_height.is_none()) {

start_dir.is_none() is required in the ordinary branch, not in the held_claim branch (src/session.rs:75-112 documents the held-standby exception). The standby is spawned whenever warm_enabled holds, whatever the pool depth (src/server/mod.rs:259-260), and it ends when the data directory goes or a kill-server marker covers it (src/session.rs:1394-1409, warm_standby_verdict), which matches the PSMUX_WARM_POOL_SIZE=0 and lifetime observations above.

Impact on bmad-loop

Every bmad-loop psmux session measured here pays for the pool: the TUI's control session for as long as the operator keeps it (the docs tell them never to kill it), and each run session for the length of the run. Each project's registry also keeps a standby after the last visible session exits, until the registry is removed or a covering kill-server ends it (subject to other explicit server shutdown/configuration behavior), so an operator who has opened the TUI in N projects carries N standbys even after closing it. The per-run share matters most during unattended runs, the case where memory pressure on Windows has already been reported (#673; no causal link claimed here).

Suggested fix

Set PSMUX_NO_WARM=1 in the environment PsmuxMultiplexer.new_session already builds for its create call (env["PSMUX_ALLOW_NESTING"] = "1" sits beside it). The variable is read in the client (src/main.rs:2178) and in the server process at boot (src/types.rs:835, :2886), and the server inherits it from that call. Measured with exactly that shape (the variable present only for the duration of new_session, through bmad-loop's own calls, three runs): the control session drops to 4 processes and 162 to 168 MiB and the run session to the PSMUX_NO_WARM=1 column above. Startup latency with the proposed change has not been measured. The explicit-command windows bypass spare-pane reuse, but that alone does not establish unchanged session startup time or scheduling cost. One side effect, observed: every pane then inherits PSMUX_NO_WARM=1 from the server, so a tool running inside a bmad-loop pane that starts psmux sessions of its own would start them cold; _source_prefix could remove it in-pane if that matters (not validated).