bench/load, daemon: record memory outside the Go heap and step through linked worktrees

bench/load/README.md

@@ -480,6 +480,44 @@ index; queryability and enrichment are separate observed upper bounds with a
 recorded polling interval. Older daemons without `loaded_roots` retain their
 legacy receipt and must still pass source-backed per-root checks.
 
+## Startup admission
+
+`--scenario startup` (never part of `all`) measures what agents see while a
+daemon starts. Every manifest repository is configured; `--worktrees` linked
+checkouts of the first one are discovered through it. Probing starts before
+the first daemon start, so the window before the socket exists is observed
+too. `--seed-store <file>` copies a store (for example one written by an
+older schema) into the private run first, which makes the initial boot an
+upgrade; the source is only read. `--restarts N` adds N warm restarts. Each
+boot lasts until every configured repository is ready and enriched, plus
+`--duration`.
+
+Every second, from the primary, the first linked worktree, and a directory
+whose `.gortex.yaml` names a workspace that a private `servers.toml` roster
+claims, the probe opens fresh sessions: `initialize` (the instructions
+variant is recorded), `search_symbols` and `resources/read
+gortex://index-health` on the legacy surface, and `workspace(operation:
+"index")` on the compact one. The roster's only server never listens, so a
+routed call falls back to the daemon itself. From the same directories it
+asks the PreToolUse hook about a native Read every second, and runs `gortex
+call search_symbols` through a pipe (from the start) and through a pty
+(`script(1)`, once the socket answers), each until its first success in that
+boot. Every answer keeps `error_code`, `retryable`, phase, percent, latency,
+exit code, and whether any refusal text advises `gortex track` or
+`gortex repos reconcile`. A watcher polls the readiness probe every 100 ms
+and, for 2 s after the primary registers, the checkout coordinator count.
+
+`startup-probe.jsonl` holds every answer. The phase gate `startup_probe`
+counts answers per surface and directory in three windows (no socket,
+loading, ready) and records each boot's time to ready and time to first
+success. Headline metrics include `startup_repo_not_tracked_before_ready`,
+`startup_nonconforming_loading_refusals`, the track-advice and hook-deny
+counts, `startup_restart_ready_to_first_ok_p95_ms`,
+`startup_restart_time_to_ready_p95_ms`, `startup_initial_time_to_ready_ms`,
+`startup_initial_first_ok_ms` and `startup_coordinators_within_2s_min`.
+Restart figures are unavailable unless every restart produced them, and
+`delta` compares two startup reports like any other.
+
 ## Measurements and artifacts
 
 One JSON file contains per-operation successful p50/p95/p99/max, successful
@@ -657,3 +695,57 @@ Inspect artifacts with `go tool pprof -top -nodecount=30 profile.cpu.pprof` and
 retain the exact run JSON alongside them. Real reference/floor baselines,
 repeatability, throughput scaling, and CPU-cause attribution require measured
 evidence; fixture tests or native diagnostics do not satisfy those gates.
+
+## Memory diagnosis
+
+`--gauges` (standalone runs only) subscribes one extra session to the daemon's
+`daemon_health` channel at a 10-second cadence for each measured phase, from
+the moment the daemon reports ready to the end of the phase, so the series also
+covers the warmup gates and the level after enrichment before any load. Every
+push becomes a row of `gate.gauge_series`: the daemon's own gauges (Go heap,
+`sys`, released heap, resident and proportional set or macOS footprint, SQLite
+page cache, live checkout coordinators) next to the harness's external sample
+of the daemon process taken as the push arrived. On Linux the sample adds
+`children_pss_bytes`, the proportional set of every descendant of the daemon
+(language servers and other helpers, found through each thread's
+`/proc/<pid>/task/<tid>/children`), and in a cgroup it adds the group's
+`memory.current`, which charges those children too, with its anonymous and
+file-backed parts from `memory.stat` (`cgroup_anon_bytes`,
+`cgroup_file_bytes`). The daemon runs its health ticker only while a session is
+subscribed, so the gauges cost nothing in runs that do not ask, and a
+`--gauges` run compared with the same run without it measures what the gauges
+cost. `--heap-profiles` (with `--gauges`) also saves two heap profiles per load
+window: one at load start and one at the highest footprint the series sampled
+during the load (or at its end, when that is higher). The peak captures run
+inside the measured window, so leave the option off when latency is being
+compared. If the OS/build has no process-footprint reader, the peak profile is
+unavailable with a reason in the phase gate or worktree step, and no peak
+artifact is advertised. Load-start and collected idle profiles remain available.
+Heap profiles are taken as of the daemon's last GC; no collection is forced
+during load.
+
+Live gauges and heap profiles are refused for `startup-probe`, which records
+its own boot observations without a gauge series, for `idle-stop`, whose
+measured interval requires every MCP session closed, and for the edit-and-probe
+`edit_sender: restart` cell, which replaces the process during its measured
+phase. Ordinary `--scenario restart` is supported: its series starts only
+after the replacement daemon is ready. `all` keeps independent per-scenario
+series; steady read/edit runs and worktree-step measurements remain supported.
+
+`--worktree-steps 0,1,2,4` (worktrees scenario, ending at `--worktrees`)
+measures what each linked worktree costs. One daemon starts with the first
+count of linked worktrees and grows through the rest. At each count it serves
+the reading agents for `--duration` from fresh sessions spread over every
+checkout, then sits idle for `--duration`; a session stays parked in each
+linked checkout throughout. Each idle window opens with one forced collection
+and a heap profile taken right after it, so every idle level follows a load of
+its own count and holds what the checkouts retain rather than the load's
+garbage. `gate.worktree_steps` records each count's load and idle windows,
+load-window process peaks, per-tool latency and heap profile paths, and the
+gauge series spans the whole run. The phase session model names the counts
+(`per_agent_per_worktree_step:0,1,2,4`), so `delta` refuses to compare a step
+run with an ordinary worktrees run or with a step run over other counts.
+Inspect a profile with
+`go tool pprof -sample_index=inuse_space -top worktrees.step3.heap-load-peak.pprof`,
+and what one more worktree retains with
+`go tool pprof -sample_index=inuse_space -top -base worktrees.step2.heap-idle.pprof worktrees.step3.heap-idle.pprof`.

bench/load/child_memory.go

@@ -0,0 +1,57 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+)
+
+// descendantPSS sums the proportional set of every process below pid (the
+// language servers and other helpers the daemon starts, and theirs) from the
+// procfs mounted at proc. Each thread lists the children it forked in
+// task/<tid>/children. A descendant that exits during the walk counts as
+// nothing; a kernel that keeps no children lists leaves the figure
+// unavailable.
+func descendantPSS(proc string, pid int) Metric {
+	total := 0.0
+	seen := map[int]bool{pid: true}
+	queue := []int{pid}
+	for len(queue) > 0 {
+		parent := queue[0]
+		queue = queue[1:]
+		lists, _ := filepath.Glob(filepath.Join(proc, strconv.Itoa(parent), "task", "*", "children"))
+		if parent == pid && len(lists) == 0 {
+			return unavailable("bytes", "procfs lists no task children")
+		}
+		for _, list := range lists {
+			data, _ := os.ReadFile(list)
+			for _, field := range strings.Fields(string(data)) {
+				child, err := strconv.Atoi(field)
+				if err != nil || seen[child] {
+					continue
+				}
+				seen[child] = true
+				queue = append(queue, child)
+				if rollup, err := os.ReadFile(filepath.Join(proc, field, "smaps_rollup")); err == nil {
+					total += parseKeyValues(string(rollup))["Pss"] * 1024
+				}
+			}
+		}
+	}
+	return measured(total, "bytes")
+}
+
+// cgroupAnonFile splits the cgroup's charge, which includes the daemon's
+// children, into anonymous memory and file-backed pages; memory.stat counts
+// both in bytes.
+func cgroupAnonFile(root string) map[string]Metric {
+	stat := make(map[string]float64)
+	if data, err := os.ReadFile(filepath.Join(root, "memory.stat")); err == nil {
+		stat = parseKeyValues(string(data))
+	}
+	return map[string]Metric{
+		"cgroup_anon_bytes": valueMetric(stat, "anon", "bytes", 1),
+		"cgroup_file_bytes": valueMetric(stat, "file", "bytes", 1),
+	}
+}

bench/load/child_memory_test.go

@@ -0,0 +1,72 @@
+package main
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+)
+
+func writeFixtureFiles(t *testing.T, root string, files map[string]string) {
+	t.Helper()
+	for name, text := range files {
+		path := filepath.Join(root, name)
+		if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil {
+			t.Fatal(err)
+		}
+		if err := os.WriteFile(path, []byte(text), 0o600); err != nil {
+			t.Fatal(err)
+		}
+	}
+}
+
+// The daemon's language servers and helpers are its descendants, forked from
+// any of its threads, and may have children of their own. Their proportional
+// sets add up once each; the daemon's own set does not count, nor does an
+// unrelated process, and a child that exits before its set is read adds
+// nothing.
+func TestDescendantPSSWalksEveryThreadsChildren(t *testing.T) {
+	proc := t.TempDir()
+	writeFixtureFiles(t, proc, map[string]string{
+		"100/smaps_rollup":      "Rss:  9000 kB\nPss:  8000 kB\n",
+		"100/task/100/children": "200 ",
+		"100/task/101/children": "300 500\n",
+		"200/smaps_rollup":      "Rss:  40 kB\nPss:  30 kB\nPss_Anon:  20 kB\nPss_File:  10 kB\n",
+		"200/task/200/children": "400",
+		"300/smaps_rollup":      "Pss:  20 kB\n",
+		"300/task/300/children": "200\n",
+		"400/smaps_rollup":      "Pss:  5 kB\n",
+		"400/task/400/children": "",
+		"900/smaps_rollup":      "Pss:  7000 kB\n",
+		"900/task/900/children": "",
+	})
+	got := descendantPSS(proc, 100)
+	if want := float64(30+20+5) * 1024; got.Value == nil || *got.Value != want || got.Unit != "bytes" {
+		t.Fatalf("descendant PSS = %+v, want %g bytes", got, want)
+	}
+	if got := descendantPSS(proc, 400); got.Value == nil || *got.Value != 0 {
+		t.Fatalf("a process without children has no descendant PSS: %+v", got)
+	}
+	if got := descendantPSS(t.TempDir(), 100); got.Value != nil {
+		t.Fatalf("no children lists fabricated a figure: %+v", got)
+	}
+}
+
+// memory.stat reports the cgroup's anonymous and file-backed charge in bytes;
+// the file_* counters break file down and must not be mistaken for it.
+func TestCgroupAnonFileReadsMemoryStat(t *testing.T) {
+	root := t.TempDir()
+	writeFixtureFiles(t, root, map[string]string{
+		"memory.stat": "anon 1048576\nfile 2097152\nkernel 65536\nfile_mapped 4096\nfile_dirty 0\nanon_thp 0\n",
+	})
+	got := cgroupAnonFile(root)
+	for key, want := range map[string]float64{"cgroup_anon_bytes": 1048576, "cgroup_file_bytes": 2097152} {
+		if m := got[key]; m.Value == nil || *m.Value != want || m.Unit != "bytes" {
+			t.Fatalf("%s = %+v, want %g", key, m, want)
+		}
+	}
+	for key, m := range cgroupAnonFile(t.TempDir()) {
+		if m.Value != nil {
+			t.Fatalf("%s fabricated without memory.stat: %+v", key, m)
+		}
+	}
+}

bench/load/client.go

@@ -89,6 +89,12 @@ func readWireLine(reader *bufio.Reader) ([]byte, error) {
 }
 
 func dialSocket(ctx context.Context, socket, cwd, mode string) (net.Conn, *bufio.Reader, string, error) {
+	return dialSocketSurface(ctx, socket, cwd, mode, "facade-v1", "hide")
+}
+
+// dialSocketSurface negotiates the session's tool surface in the handshake,
+// as the CLI does per connection; the load workload always uses the facade.
+func dialSocketSurface(ctx context.Context, socket, cwd, mode, tools, toolsMode string) (net.Conn, *bufio.Reader, string, error) {
 	conn, err := (&net.Dialer{}).DialContext(ctx, "unix", socket)
 	if err != nil {
 		return nil, nil, "", err
@@ -103,7 +109,7 @@ func dialSocket(ctx context.Context, socket, cwd, mode string) (net.Conn, *bufio
 	}
 	stop := context.AfterFunc(ctx, func() { _ = conn.Close() })
 	defer stop()
-	handshake := map[string]any{"version": 1, "mode": mode, "cwd": cwd, "client": "gortex-load", "tools": "facade-v1", "tools_mode": "hide"}
+	handshake := map[string]any{"version": 1, "mode": mode, "cwd": cwd, "client": "gortex-load", "tools": tools, "tools_mode": toolsMode}
 	if err := json.NewEncoder(conn).Encode(handshake); err != nil {
 		_ = conn.Close()
 		return nil, nil, "", err

bench/load/edit_probe_plan.go

@@ -108,6 +108,9 @@ func makeEditProbePlan(c Corpus, seed int64, count int) (editProbePlan, error) {
 }
 
 func planEditProbe(c Corpus, o runOptions) (editProbePlan, error) {
+	if c.EditSender == editProbeRestart && wantsGaugeSeries(o) {
+		return editProbePlan{}, errors.New("live memory gauges and profiles are unsupported for the in-run restart sender: its process and health session are replaced")
+	}
 	p, err := makeEditProbeTrafficPlan(c, o.Seed, o.EditCount, o.Editors)
 	if err != nil || o.Agents == 0 {
 		return p, err