Repository navigation
Conversation
ProcessMemory carried only lifetime peaks and the macOS footprint, so nothing could say how much of the daemon is resident right now on Linux, or how much of that is file-backed. Diagnosing the worktree footprint needs both: the store is memory-mapped, and its pages count in RSS but not in anything the Go runtime reports. Linux now reads Rss, Pss and Pss_File from /proc/self/smaps_rollup; macOS adds the current resident size from the proc_pid_rusage call that already supplies the footprint. Other platforms keep reporting 0. The rollup walks the page tables, so the doc comment now says to read it on demand.
SQLite's page caches are allocated by its C allocator, outside the Go heap, and every pooled connection may hold up to 32 MiB of them; nothing reported how much they actually hold. The pinned driver builds SQLite with SQLITE_DEFAULT_MEMSTATUS=0, so sqlite3_memory_used stays 0, and turning the statistics on would add a mutex to every SQLite allocation for the life of the process. The page cache keeps its own counter regardless: with no SQLITE_CONFIG_PAGECACHE buffer every page is an overflow allocation counted under the page-cache mutex. PageCacheBytes reads that counter and its high-water mark through sqlite3_status64.
Status and the daemon_health snapshot reported only Go runtime figures, while most of a daemon serving several worktrees can sit outside them: SQLite page caches and parser allocations in C allocators, and mapped store files as file pages. Attributing the footprint needs those beside the Go numbers, sampled at the same moment. Both surfaces now add the operating system's view (current and peak resident set; Linux proportional set and its file-backed part; macOS current and peak footprint) and SQLite's process-wide page cache. Health also reports the live checkout coordinator count. The figures are read only when status is asked for or a health subscriber's tick fires, so a daemon nobody watches pays nothing; a figure with no reader on the platform is left out rather than reported as 0.
The harness kept only 500 ms peaks and before/after brackets of process memory, ran one worktree count per daemon, and took no heap profiles, so a footprint could be neither attributed to components nor divided by the number of checkout coordinators. --gauges subscribes one extra session to daemon_health at 10 s and keeps every push in gate.gauge_series, next to an external sample of the daemon process (and the cgroup's memory.current) taken as it arrived. It also saves heap profiles at load start and at the highest footprint the series sampled during the load. The daemon runs its health ticker only while someone is subscribed, and the option is off by default, so other runs keep their workload unchanged. --worktree-steps 0,1,2,4 grows the linked worktrees on one daemon: at each count it sits idle for --duration, then serves the reading agents for --duration, with the series running throughout and per-step windows, peaks, latency and heap profiles in gate.worktree_steps. Its phase session model differs from an ordinary worktrees run, so delta will not compare the two. Both options are standalone-only; paired runs measure latency between builds and do not bind them.
The test drove its 500 ms windows with the saturation profile, so every window ended with calls in flight. Their advisory cancellations can time out on a loaded race-instrumented host, which marks the shared stream broken and fails the phase with workload errors that say nothing about the gauges. Use the agent profile, as the other fixture self-tests do, with 5.5 s windows: the agent thinks at most 5 s before its first burst, and a read phase without a single workload call is an error. Two step counts are enough to see the series follow the coordinators.
Step mode sat idle at each count before serving that count's load, so every idle window after the first followed the previous count's load: its level carried that load's garbage and the one-off growth of a first load, and the change between counts said little about a coordinator. On a 40-node corpus the first step's "idle cost" was 74 MiB, almost all of it residue. Each count is now loaded first and then idle. The idle window opens with one forced collection, taken as a heap profile with gc=1, so every idle level follows a load of its own count and holds only what the checkouts retain, and consecutive idle profiles diff into what one more coordinator keeps. Load windows still take their profiles as of the last collection; the post-warmup level before any load stays in the series ahead of the first step.
--gauges took a heap profile as the load started and another each time a series row set a new footprint high. The peak captures run inside the measured load window, where the daemon walks its memory-profile buckets, symbolises and compresses while the agents time their calls. A read run with and without --gauges, which is how the gauges' overhead is judged, therefore measured the profile captures as well as the health ticks. --gauges now records the series only. --heap-profiles, which needs --gauges, adds the load-start and peak profiles for runs that attribute memory rather than compare latency. Step mode keeps taking its profiles: it measures what each worktree costs, not latency.
The daemon_health snapshot gained the live coordinator count, the process-wide SQLite page cache and the operating system's view of the process, but the MCP reference, the feature list and the tool description still named only the Go runtime figures. They now name the new keys, say that an OS figure is left out where the platform has no reader, and that the page cache covers every connection in the process.
Status read the operating system's resident figures and SQLite's page cache only after the view census and the mutex-guarded aggregate, which can wait seconds behind a track, so the runtime block paired a Go heap from before the wait with process figures from after it. The block is now assembled right after ReadMemStats, before the mutex.
Every step run had the session model per_agent_per_worktree_step, so delta compared a run over 0,1,2,4 with one over 0,2,4 as if they had measured the same thing. The step list is not in the workload hash, and the session model is the only identity delta checks besides it, so the model now carries the counts: per_agent_per_worktree_step:0,1,2,4.
Each step's idle window lasted --duration, the 5.5 s the test needs for its load windows to carry a workload call, so two steps spent 11 s waiting on nothing. An idle window needs no calls: a package-level override, set by the test like the gauge interval, shortens it to 300 ms. The load-then-idle order and the forced collection that opens each idle window are still checked.
The gauge series sampled only the daemon process and the cgroup's total charge, so memory held by the language servers and other helpers the daemon starts could be bounded but not named, and the cgroup figure did not say how much of it was page cache. On Linux each row now carries children_pss_bytes, the proportional set of every descendant of the daemon, found through each thread's procfs children list, and in a cgroup the anon and file parts of memory.stat beside memory.current.
The series started with the measured load, after the warmup gates, so a run could not show the daemon's level in the minutes after enrichment before any agent arrived, which is where the idle gap is read. It now starts as soon as the measured daemon reports ready (in the restart scenario, after the restart) and ends with the phase. The load window only arms the heap profiles asked for with --heap-profiles, and step mode records into the same series.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Diagnosing the 4-worktree memory footprint (#999) needs two things nobody can measure today: memory outside the Go heap, sampled at the same moment as the Go figures, and the cost of each added checkout coordinator. This draft adds the gauges to the daemon, read on demand only, plus
bench/loadoptions that record them over a run and step one daemon through linked worktree counts.Draft status: this PR holds the tooling only. The diagnosis runs and the report this issue asks for are not done. The history will be squashed before review (see "Before review").
Changes
Daemon gauges (read only when asked; no sampler, no env switch)
platform.ProcessMemoryadds the current resident set. On Linux it also adds the proportional set and its file-backed part, read from/proc/self/smaps_rollup. On macOS it adds the current resident size, from the call that already supplies the footprint.sqliteconn.PageCacheBytesreads SQLite's process-wide page-cache size and peak throughsqlite3_status64.daemon status(runtime) and thedaemon_healthsnapshot carry the OS figures and the page cache. Health also carries the live checkout coordinator count.ReadMemStats, before the mutex-guarded aggregate, so the Go and OS figures describe the same moment.docs/mcp.md,docs/features.mdand the tool description list the new keys.bench/load--gaugessubscribes one extra session todaemon_healthat 10 s. Each push becomes a row ingate.gauge_series, next to an external sample of the daemon process.memory.currentand the anon and file parts ofmemory.stat.children_pss_bytes).--heap-profiles(requires--gauges) adds heap profiles at load start and at the sampled footprint peak. It is separate so that a run with and without--gaugesmeasures only the gauges' overhead.--worktree-steps 0,1,2,4grows the linked worktrees on one daemon.gate.worktree_steps.deltarefuses to compare runs over different counts.Before review
The eight
bench/loadcommits will be squashed into one, and the two daemon status commits into one, so no commit message contradicts the final code. The platform, sqliteconn and docs commits stay as they are. The squash rewrites this branch's history.Tested
On the base, before the change, the new tests do not build, so none of them can pass there.
Up to the first review round's fixes:
cmd/gortexfocused tests pass;bench/loadfocused tests pass under-race;go vetpass;A macOS smoke run of step mode (0, 1, 2 worktrees) completed with no errors. It wrote 13 series rows and 3 steps in load-then-idle order, with readable heap profiles and the expected coordinator count in each idle window.
Untested: the last five commits, which are the second review round's fixes. They cover:
Their red/green runs were queued and never ran, so CI on this PR is their first run.
The branch merges cleanly with main and with feat(bench): add strict budget collection and nightly evidence checks #1160.
Not done
--gauges);Refs #999