diff --git a/bench/load/README.md b/bench/load/README.md index 91ffbe433..976ad529a 100644 --- a/bench/load/README.md +++ b/bench/load/README.md @@ -480,6 +480,44 @@ index; queryability and enrichment are separate observed upper bounds with a recorded polling interval. Older daemons without `loaded_roots` retain their legacy receipt and must still pass source-backed per-root checks. +## Startup admission + +`--scenario startup` (never part of `all`) measures what agents see while a +daemon starts. Every manifest repository is configured; `--worktrees` linked +checkouts of the first one are discovered through it. Probing starts before +the first daemon start, so the window before the socket exists is observed +too. `--seed-store ` copies a store (for example one written by an +older schema) into the private run first, which makes the initial boot an +upgrade; the source is only read. `--restarts N` adds N warm restarts. Each +boot lasts until every configured repository is ready and enriched, plus +`--duration`. + +Every second, from the primary, the first linked worktree, and a directory +whose `.gortex.yaml` names a workspace that a private `servers.toml` roster +claims, the probe opens fresh sessions: `initialize` (the instructions +variant is recorded), `search_symbols` and `resources/read +gortex://index-health` on the legacy surface, and `workspace(operation: +"index")` on the compact one. The roster's only server never listens, so a +routed call falls back to the daemon itself. From the same directories it +asks the PreToolUse hook about a native Read every second, and runs `gortex +call search_symbols` through a pipe (from the start) and through a pty +(`script(1)`, once the socket answers), each until its first success in that +boot. Every answer keeps `error_code`, `retryable`, phase, percent, latency, +exit code, and whether any refusal text advises `gortex track` or +`gortex repos reconcile`. A watcher polls the readiness probe every 100 ms +and, for 2 s after the primary registers, the checkout coordinator count. + +`startup-probe.jsonl` holds every answer. The phase gate `startup_probe` +counts answers per surface and directory in three windows (no socket, +loading, ready) and records each boot's time to ready and time to first +success. Headline metrics include `startup_repo_not_tracked_before_ready`, +`startup_nonconforming_loading_refusals`, the track-advice and hook-deny +counts, `startup_restart_ready_to_first_ok_p95_ms`, +`startup_restart_time_to_ready_p95_ms`, `startup_initial_time_to_ready_ms`, +`startup_initial_first_ok_ms` and `startup_coordinators_within_2s_min`. +Restart figures are unavailable unless every restart produced them, and +`delta` compares two startup reports like any other. + ## Measurements and artifacts One JSON file contains per-operation successful p50/p95/p99/max, successful @@ -657,3 +695,57 @@ Inspect artifacts with `go tool pprof -top -nodecount=30 profile.cpu.pprof` and retain the exact run JSON alongside them. Real reference/floor baselines, repeatability, throughput scaling, and CPU-cause attribution require measured evidence; fixture tests or native diagnostics do not satisfy those gates. + +## Memory diagnosis + +`--gauges` (standalone runs only) subscribes one extra session to the daemon's +`daemon_health` channel at a 10-second cadence for each measured phase, from +the moment the daemon reports ready to the end of the phase, so the series also +covers the warmup gates and the level after enrichment before any load. Every +push becomes a row of `gate.gauge_series`: the daemon's own gauges (Go heap, +`sys`, released heap, resident and proportional set or macOS footprint, SQLite +page cache, live checkout coordinators) next to the harness's external sample +of the daemon process taken as the push arrived. On Linux the sample adds +`children_pss_bytes`, the proportional set of every descendant of the daemon +(language servers and other helpers, found through each thread's +`/proc//task//children`), and in a cgroup it adds the group's +`memory.current`, which charges those children too, with its anonymous and +file-backed parts from `memory.stat` (`cgroup_anon_bytes`, +`cgroup_file_bytes`). The daemon runs its health ticker only while a session is +subscribed, so the gauges cost nothing in runs that do not ask, and a +`--gauges` run compared with the same run without it measures what the gauges +cost. `--heap-profiles` (with `--gauges`) also saves two heap profiles per load +window: one at load start and one at the highest footprint the series sampled +during the load (or at its end, when that is higher). The peak captures run +inside the measured window, so leave the option off when latency is being +compared. If the OS/build has no process-footprint reader, the peak profile is +unavailable with a reason in the phase gate or worktree step, and no peak +artifact is advertised. Load-start and collected idle profiles remain available. +Heap profiles are taken as of the daemon's last GC; no collection is forced +during load. + +Live gauges and heap profiles are refused for `startup-probe`, which records +its own boot observations without a gauge series, for `idle-stop`, whose +measured interval requires every MCP session closed, and for the edit-and-probe +`edit_sender: restart` cell, which replaces the process during its measured +phase. Ordinary `--scenario restart` is supported: its series starts only +after the replacement daemon is ready. `all` keeps independent per-scenario +series; steady read/edit runs and worktree-step measurements remain supported. + +`--worktree-steps 0,1,2,4` (worktrees scenario, ending at `--worktrees`) +measures what each linked worktree costs. One daemon starts with the first +count of linked worktrees and grows through the rest. At each count it serves +the reading agents for `--duration` from fresh sessions spread over every +checkout, then sits idle for `--duration`; a session stays parked in each +linked checkout throughout. Each idle window opens with one forced collection +and a heap profile taken right after it, so every idle level follows a load of +its own count and holds what the checkouts retain rather than the load's +garbage. `gate.worktree_steps` records each count's load and idle windows, +load-window process peaks, per-tool latency and heap profile paths, and the +gauge series spans the whole run. The phase session model names the counts +(`per_agent_per_worktree_step:0,1,2,4`), so `delta` refuses to compare a step +run with an ordinary worktrees run or with a step run over other counts. +Inspect a profile with +`go tool pprof -sample_index=inuse_space -top worktrees.step3.heap-load-peak.pprof`, +and what one more worktree retains with +`go tool pprof -sample_index=inuse_space -top -base worktrees.step2.heap-idle.pprof worktrees.step3.heap-idle.pprof`. diff --git a/bench/load/child_memory.go b/bench/load/child_memory.go new file mode 100644 index 000000000..38ab1141c --- /dev/null +++ b/bench/load/child_memory.go @@ -0,0 +1,57 @@ +package main + +import ( + "os" + "path/filepath" + "strconv" + "strings" +) + +// descendantPSS sums the proportional set of every process below pid (the +// language servers and other helpers the daemon starts, and theirs) from the +// procfs mounted at proc. Each thread lists the children it forked in +// task//children. A descendant that exits during the walk counts as +// nothing; a kernel that keeps no children lists leaves the figure +// unavailable. +func descendantPSS(proc string, pid int) Metric { + total := 0.0 + seen := map[int]bool{pid: true} + queue := []int{pid} + for len(queue) > 0 { + parent := queue[0] + queue = queue[1:] + lists, _ := filepath.Glob(filepath.Join(proc, strconv.Itoa(parent), "task", "*", "children")) + if parent == pid && len(lists) == 0 { + return unavailable("bytes", "procfs lists no task children") + } + for _, list := range lists { + data, _ := os.ReadFile(list) + for _, field := range strings.Fields(string(data)) { + child, err := strconv.Atoi(field) + if err != nil || seen[child] { + continue + } + seen[child] = true + queue = append(queue, child) + if rollup, err := os.ReadFile(filepath.Join(proc, field, "smaps_rollup")); err == nil { + total += parseKeyValues(string(rollup))["Pss"] * 1024 + } + } + } + } + return measured(total, "bytes") +} + +// cgroupAnonFile splits the cgroup's charge, which includes the daemon's +// children, into anonymous memory and file-backed pages; memory.stat counts +// both in bytes. +func cgroupAnonFile(root string) map[string]Metric { + stat := make(map[string]float64) + if data, err := os.ReadFile(filepath.Join(root, "memory.stat")); err == nil { + stat = parseKeyValues(string(data)) + } + return map[string]Metric{ + "cgroup_anon_bytes": valueMetric(stat, "anon", "bytes", 1), + "cgroup_file_bytes": valueMetric(stat, "file", "bytes", 1), + } +} diff --git a/bench/load/child_memory_test.go b/bench/load/child_memory_test.go new file mode 100644 index 000000000..9775fcd2a --- /dev/null +++ b/bench/load/child_memory_test.go @@ -0,0 +1,72 @@ +package main + +import ( + "os" + "path/filepath" + "testing" +) + +func writeFixtureFiles(t *testing.T, root string, files map[string]string) { + t.Helper() + for name, text := range files { + path := filepath.Join(root, name) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(text), 0o600); err != nil { + t.Fatal(err) + } + } +} + +// The daemon's language servers and helpers are its descendants, forked from +// any of its threads, and may have children of their own. Their proportional +// sets add up once each; the daemon's own set does not count, nor does an +// unrelated process, and a child that exits before its set is read adds +// nothing. +func TestDescendantPSSWalksEveryThreadsChildren(t *testing.T) { + proc := t.TempDir() + writeFixtureFiles(t, proc, map[string]string{ + "100/smaps_rollup": "Rss: 9000 kB\nPss: 8000 kB\n", + "100/task/100/children": "200 ", + "100/task/101/children": "300 500\n", + "200/smaps_rollup": "Rss: 40 kB\nPss: 30 kB\nPss_Anon: 20 kB\nPss_File: 10 kB\n", + "200/task/200/children": "400", + "300/smaps_rollup": "Pss: 20 kB\n", + "300/task/300/children": "200\n", + "400/smaps_rollup": "Pss: 5 kB\n", + "400/task/400/children": "", + "900/smaps_rollup": "Pss: 7000 kB\n", + "900/task/900/children": "", + }) + got := descendantPSS(proc, 100) + if want := float64(30+20+5) * 1024; got.Value == nil || *got.Value != want || got.Unit != "bytes" { + t.Fatalf("descendant PSS = %+v, want %g bytes", got, want) + } + if got := descendantPSS(proc, 400); got.Value == nil || *got.Value != 0 { + t.Fatalf("a process without children has no descendant PSS: %+v", got) + } + if got := descendantPSS(t.TempDir(), 100); got.Value != nil { + t.Fatalf("no children lists fabricated a figure: %+v", got) + } +} + +// memory.stat reports the cgroup's anonymous and file-backed charge in bytes; +// the file_* counters break file down and must not be mistaken for it. +func TestCgroupAnonFileReadsMemoryStat(t *testing.T) { + root := t.TempDir() + writeFixtureFiles(t, root, map[string]string{ + "memory.stat": "anon 1048576\nfile 2097152\nkernel 65536\nfile_mapped 4096\nfile_dirty 0\nanon_thp 0\n", + }) + got := cgroupAnonFile(root) + for key, want := range map[string]float64{"cgroup_anon_bytes": 1048576, "cgroup_file_bytes": 2097152} { + if m := got[key]; m.Value == nil || *m.Value != want || m.Unit != "bytes" { + t.Fatalf("%s = %+v, want %g", key, m, want) + } + } + for key, m := range cgroupAnonFile(t.TempDir()) { + if m.Value != nil { + t.Fatalf("%s fabricated without memory.stat: %+v", key, m) + } + } +} diff --git a/bench/load/client.go b/bench/load/client.go index 3c7536903..66f5371a5 100644 --- a/bench/load/client.go +++ b/bench/load/client.go @@ -89,6 +89,12 @@ func readWireLine(reader *bufio.Reader) ([]byte, error) { } func dialSocket(ctx context.Context, socket, cwd, mode string) (net.Conn, *bufio.Reader, string, error) { + return dialSocketSurface(ctx, socket, cwd, mode, "facade-v1", "hide") +} + +// dialSocketSurface negotiates the session's tool surface in the handshake, +// as the CLI does per connection; the load workload always uses the facade. +func dialSocketSurface(ctx context.Context, socket, cwd, mode, tools, toolsMode string) (net.Conn, *bufio.Reader, string, error) { conn, err := (&net.Dialer{}).DialContext(ctx, "unix", socket) if err != nil { return nil, nil, "", err @@ -103,7 +109,7 @@ func dialSocket(ctx context.Context, socket, cwd, mode string) (net.Conn, *bufio } stop := context.AfterFunc(ctx, func() { _ = conn.Close() }) defer stop() - handshake := map[string]any{"version": 1, "mode": mode, "cwd": cwd, "client": "gortex-load", "tools": "facade-v1", "tools_mode": "hide"} + handshake := map[string]any{"version": 1, "mode": mode, "cwd": cwd, "client": "gortex-load", "tools": tools, "tools_mode": toolsMode} if err := json.NewEncoder(conn).Encode(handshake); err != nil { _ = conn.Close() return nil, nil, "", err diff --git a/bench/load/edit_probe_plan.go b/bench/load/edit_probe_plan.go index 4e7ab041b..d10cd7249 100644 --- a/bench/load/edit_probe_plan.go +++ b/bench/load/edit_probe_plan.go @@ -108,6 +108,9 @@ func makeEditProbePlan(c Corpus, seed int64, count int) (editProbePlan, error) { } func planEditProbe(c Corpus, o runOptions) (editProbePlan, error) { + if c.EditSender == editProbeRestart && wantsGaugeSeries(o) { + return editProbePlan{}, errors.New("live memory gauges and profiles are unsupported for the in-run restart sender: its process and health session are replaced") + } p, err := makeEditProbeTrafficPlan(c, o.Seed, o.EditCount, o.Editors) if err != nil || o.Agents == 0 { return p, err diff --git a/bench/load/gauges.go b/bench/load/gauges.go new file mode 100644 index 000000000..1556ecd32 --- /dev/null +++ b/bench/load/gauges.go @@ -0,0 +1,266 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "maps" + "math" + "os" + "path/filepath" + "runtime" + "sync" + "time" +) + +// gaugeInterval is the daemon_health cadence a gauge series asks for. The +// daemon clamps it to at least a second. +var gaugeInterval = 10 * time.Second + +// gaugeRow pairs one daemon_health push — the daemon's own Go, operating +// system and SQLite gauges and its live checkout coordinators — with the +// harness's external sample of the daemon process taken as the push arrived. +type gaugeRow struct { + Received time.Time `json:"received"` + Process map[string]Metric `json:"process"` + Health json.RawMessage `json:"daemon_health"` +} + +// gaugeSeries holds one dedicated session subscribed to daemon_health. The +// daemon runs its health ticker only while a session is subscribed, so a run +// that does not ask for the series costs the daemon nothing. +type gaugeSeries struct { + client *socketClient + sampler processSampler + pid int + group string + cancel context.CancelFunc + done sync.WaitGroup + + mu sync.Mutex + rows []gaugeRow + watch *heapWindow +} + +func startGaugeSeries(ctx context.Context, p *privateRun, sampler processSampler, group string) (*gaugeSeries, error) { + client, err := openMCP(ctx, p.Socket, p.Repos[0].Path) + if err != nil { + return nil, err + } + if err := subscribeHealth(ctx, client, "subscribe"); err != nil { + _ = client.Close() + return nil, fmt.Errorf("daemon_health subscription: %w", err) + } + s := &gaugeSeries{client: client, sampler: sampler, pid: p.process.Process.Pid, group: group} + recordCtx, cancel := context.WithCancel(context.Background()) + s.cancel = cancel + s.done.Go(func() { + ticker := time.NewTicker(100 * time.Millisecond) + defer ticker.Stop() + for { + s.record() + select { + case <-recordCtx.Done(): + return + case <-ticker.C: + } + } + }) + return s, nil +} + +func subscribeHealth(ctx context.Context, client *socketClient, operation string) error { + callCtx, cancel := context.WithTimeout(ctx, 30*time.Second) + defer cancel() + data, err := client.call(callCtx, "tools/call", map[string]any{"name": "session", "arguments": map[string]any{ + "operation": operation, "channel": "daemon_health", "arguments": map[string]any{"interval_ms": gaugeInterval.Milliseconds()}, + }}) + if err == nil { + err = checkToolResult(data) + } + return err +} + +// record turns every daemon_health push received since the last call into a +// row. The dedicated session receives nothing else worth keeping, so its +// notification buffer is drained rather than left to reach its cap. +func (s *gaugeSeries) record() { + s.client.mu.Lock() + received := s.client.notifies + s.client.notifies = nil + s.client.mu.Unlock() + for _, notification := range received { + var frame struct { + Method string `json:"method"` + Params json.RawMessage `json:"params"` + } + if json.Unmarshal(notification.Payload, &frame) != nil || frame.Method != "notifications/daemon_health" { + continue + } + row := gaugeRow{Received: notification.Received, Process: s.sample(), Health: frame.Params} + s.mu.Lock() + s.rows = append(s.rows, row) + if s.watch != nil { + s.watch.observe(row.Process) + } + s.mu.Unlock() + } +} + +// sample is the external view of the daemon process. On Linux it adds the +// proportional set of the daemon's descendants; in a cgroup it adds the +// group's whole charge, which includes them, split into anonymous and file +// pages. +func (s *gaugeSeries) sample() map[string]Metric { + values := s.sampler.sample(s.pid) + out := make(map[string]Metric) + for _, key := range []string{"rss_bytes", "pss_bytes", "rss_anon_bytes", "phys_footprint_bytes"} { + out[key] = values[key] + } + if runtime.GOOS == "linux" { + out["children_pss_bytes"] = descendantPSS("/proc", s.pid) + } + if s.group != "" { + out["cgroup_memory_current_bytes"] = cgroupMetrics(s.group, os.Getpagesize())["memory_current_bytes"] + maps.Copy(out, cgroupAnonFile(s.group)) + } + return out +} + +// stop ends the subscription and returns every row recorded. It runs on +// cleanup paths too, so it does not inherit a cancelled run's context. +func (s *gaugeSeries) stop() ([]gaugeRow, error) { + err := subscribeHealth(context.Background(), s.client, "unsubscribe") + s.cancel() + s.done.Wait() + s.record() + err = errors.Join(err, s.client.Close()) + s.mu.Lock() + defer s.mu.Unlock() + return s.rows, err +} + +// heapWindow keeps two heap profiles of one load window: one taken as the +// load starts, and one at the highest daemon footprint the series sampled +// during it (or at the window's end, when that is higher). +type heapWindow struct { + ctx context.Context + status json.RawMessage + start string + peak string + peakCaptured bool + peakUnavailable string + high float64 + err error +} + +// watchLoad takes the load-start profile now and the peak profile as rows +// arrive, until endLoad. +func (s *gaugeSeries) watchLoad(ctx context.Context, status json.RawMessage, base string) *heapWindow { + w := &heapWindow{ctx: ctx, status: status, start: base + ".heap-load-start.pprof", peak: base + ".heap-load-peak.pprof", high: -1} + w.err = captureHeapProfile(ctx, status, w.start) + s.mu.Lock() + s.watch = w + s.mu.Unlock() + return w +} + +func (s *gaugeSeries) endLoad(w *heapWindow) (start, peak string, err error) { + s.record() + s.mu.Lock() + defer s.mu.Unlock() + s.watch = nil + w.observe(s.sample()) + if w.high < 0 { + w.peakUnavailable = "no external process footprint sample was available during load" + return w.start, "", w.err + } + if !w.peakCaptured { + return w.start, "", w.err + } + return w.start, w.peak, w.err +} + +// observe replaces the peak profile when this sample is the window's highest. +func (w *heapWindow) observe(process map[string]Metric) { + value, ok := footprintOf(process) + if !ok || value <= w.high { + return + } + w.high = value + partial := w.peak + ".partial" + _ = os.Remove(partial) + err := captureHeapProfile(w.ctx, w.status, partial) + if err == nil { + err = os.Rename(partial, w.peak) + } + if err != nil { + // One failure is reported; retrying on every later row would only + // repeat it. + w.err = errors.Join(w.err, fmt.Errorf("peak heap profile %s: %w", filepath.Base(w.peak), err)) + w.high = math.Inf(1) + } else { + w.peakCaptured = true + } +} + +// footprintOf picks the external memory figure peaks are judged by: PSS on +// Linux, the physical footprint on macOS, the resident set otherwise. +func footprintOf(process map[string]Metric) (float64, bool) { + for _, key := range []string{"pss_bytes", "phys_footprint_bytes", "rss_bytes"} { + if value := process[key].Value; value != nil { + return *value, true + } + } + return 0, false +} + +// wantsGaugeSeries says whether a run records the gauge series: when asked, +// and always in step mode. +func wantsGaugeSeries(o runOptions) bool { + return o.Gauges || len(o.WorktreeSteps) > 0 +} + +// watchLoadProfiles arms an ordinary load window's heap profiles when the run +// asked for them; nil otherwise. Peak profiles are captured while the load +// runs, so a series-only run is the one that measures what the gauges alone +// cost. +func (s *gaugeSeries) watchLoadProfiles(ctx context.Context, o runOptions, p *privateRun, phase *Phase) *heapWindow { + if s == nil || !o.HeapProfiles { + return nil + } + return s.watchLoad(ctx, lastStatus(phase), filepath.Join(p.Root, phase.Name)) +} + +// endLoadProfiles ends the load window and keeps its profiles as artifacts. +func (s *gaugeSeries) endLoadProfiles(w *heapWindow, phase *Phase, artifacts map[string]string) error { + if w == nil { + return nil + } + start, peak, err := s.endLoad(w) + artifacts["heap_profile_load_start"] = start + if peak != "" { + artifacts["heap_profile_load_peak"] = peak + } + if w.peakUnavailable != "" { + phase.Gate["heap_profile_load_peak_unavailable"], _ = json.Marshal(w.peakUnavailable) + } + return err +} + +// stopInto ends the series and keeps its rows in the phase. +func (s *gaugeSeries) stopInto(phase *Phase) error { + rows, err := s.stop() + phase.Gate["gauge_series"], _ = json.Marshal(rows) + return err +} + +// lastStatus is the latest daemon status receipt, which names the pprof +// listener; nil when status was never captured. +func lastStatus(phase *Phase) json.RawMessage { + if len(phase.Status) == 0 { + return nil + } + return phase.Status[len(phase.Status)-1] +} diff --git a/bench/load/gauges_ordering_test.go b/bench/load/gauges_ordering_test.go new file mode 100644 index 000000000..6be378048 --- /dev/null +++ b/bench/load/gauges_ordering_test.go @@ -0,0 +1,78 @@ +package main + +import ( + "encoding/json" + "errors" + "fmt" + "testing" + "time" +) + +// The fixture's initial replay is synchronously sent before its subscribe +// reply. Its counter also catches moving series setup after warmup, even when +// a serialized wall clock cannot distinguish the two events. +func gaugeWarmupOrderError(rows []gaugeRow, attempts []Attempt) error { + if len(rows) == 0 || rows[0].Received.IsZero() { + return errors.New("no initial gauge observation") + } + var evidence struct { + Initial bool `json:"initial_replay"` + Warmup *uint64 `json:"fixture_warmup_calls"` + } + if err := json.Unmarshal(rows[0].Health, &evidence); err != nil || !evidence.Initial || evidence.Warmup == nil || *evidence.Warmup != 0 { + return errors.New("initial gauge replay did not precede observed warmup calls") + } + first, probes := rows[0].Received, 0 + for _, attempt := range attempts { + if attempt.Role != "probe" { + continue + } + probes++ + // Equal serialized ticks preserve the causal replay proof above. + if attempt.Started.IsZero() || first.After(attempt.Started) { + return fmt.Errorf("series started after a warmup gate: first row %v, gate call %v", first, attempt.Started) + } + } + if probes == 0 { + return errors.New("no warmup gate calls recorded") + } + return nil +} + +func TestGaugeWarmupOrderPreservesCausalityAcrossClockPrecision(t *testing.T) { + base := time.Unix(1762000000, 123456000) + for _, quantum := range []time.Duration{100 * time.Nanosecond, time.Microsecond, time.Millisecond} { + t.Run(quantum.String(), func(t *testing.T) { + // These ordered events fall in one wall-clock tick. JSON round + // trips also discard Go's in-process monotonic component. + first := base.Add(10 * time.Nanosecond).Truncate(quantum) + gate := base.Add(90 * time.Nanosecond).Truncate(quantum) + rows := []gaugeRow{{Received: first, Health: json.RawMessage(`{"initial_replay":true,"fixture_warmup_calls":0}`)}} + raw, err := json.Marshal(rows) + if err != nil { + t.Fatal(err) + } + if err := json.Unmarshal(raw, &rows); err != nil { + t.Fatal(err) + } + attempts := []Attempt{{Role: "probe", Started: gate}} + rows[0].Received = gate.Add(-quantum) + if err := gaugeWarmupOrderError(rows, attempts); err != nil { + t.Errorf("earlier clock tick rejected: %v", err) + } + rows[0].Received = first + if err := gaugeWarmupOrderError(rows, attempts); err != nil { + t.Errorf("causally earlier replay rejected at equal clock precision: %v", err) + } + rows[0].Received = gate.Add(quantum) + if err := gaugeWarmupOrderError(rows, attempts); err == nil { + t.Error("a genuinely later observation was accepted") + } + rows[0].Received = gate + rows[0].Health = json.RawMessage(`{"initial_replay":true,"fixture_warmup_calls":1}`) + if err := gaugeWarmupOrderError(rows, attempts); err == nil { + t.Error("late series setup hidden by an equal clock tick was accepted") + } + }) + } +} diff --git a/bench/load/gauges_test.go b/bench/load/gauges_test.go new file mode 100644 index 000000000..eb665d289 --- /dev/null +++ b/bench/load/gauges_test.go @@ -0,0 +1,292 @@ +package main + +import ( + "context" + "encoding/json" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "slices" + "strings" + "sync/atomic" + "testing" + "time" +) + +type gaugeFixtureSampler map[string]Metric + +func (s gaugeFixtureSampler) sample(int) map[string]Metric { return s } + +func TestHeapProfilesWithoutFootprintDoNotAdvertisePeak(t *testing.T) { + for _, cell := range []struct { + name string + available bool + fail bool + keepPrior bool + }{{"unavailable", false, false, false}, {"measured", true, false, false}, {"capture_failure", true, true, false}, {"later_capture_failure", true, true, true}} { + t.Run(cell.name, func(t *testing.T) { + var requests atomic.Int32 + server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/debug/pprof/heap" { + t.Errorf("unexpected profile endpoint: %s", r.URL) + } + if count := requests.Add(1); cell.fail && (!cell.keepPrior || count > 2) { + w.WriteHeader(http.StatusServiceUnavailable) + return + } + _, _ = w.Write([]byte("retained-heap-profile")) + })) + defer server.Close() + status, _ := json.Marshal(map[string]any{"pprof_addr": strings.TrimPrefix(server.URL, "http://")}) + metrics := absentProcessMetrics("fixture has no process collector") + if cell.available { + metrics["rss_bytes"] = measured(4096, "bytes") + } + series := &gaugeSeries{client: &socketClient{}, sampler: gaugeFixtureSampler(metrics)} + window := series.watchLoad(t.Context(), status, filepath.Join(t.TempDir(), "load")) + if cell.keepPrior { + window.observe(metrics) + metrics["rss_bytes"] = measured(8192, "bytes") + } + phase := &Phase{Gate: make(map[string]json.RawMessage)} + artifacts := map[string]string{} + err := series.endLoadProfiles(window, phase, artifacts) + if cell.fail { + if err == nil || len(phase.Gate["heap_profile_load_peak_unavailable"]) != 0 { + t.Fatalf("profile capture failure became unavailable telemetry: %v %s", err, phase.Gate) + } + peak, advertised := artifacts["heap_profile_load_peak"] + if cell.keepPrior { + if data, err := os.ReadFile(peak); !advertised || err != nil || string(data) != "retained-heap-profile" { + t.Fatalf("later capture failure lost the actual earlier peak: %q %v", data, err) + } + } else if advertised { + t.Fatalf("failed peak capture advertised a nonexistent artifact: %s", peak) + } + return + } + if err != nil { + t.Fatal(err) + } + if data, err := os.ReadFile(artifacts["heap_profile_load_start"]); err != nil || string(data) != "retained-heap-profile" { + t.Fatalf("load-start profile lost: %q %v", data, err) + } + peak, advertised := artifacts["heap_profile_load_peak"] + if !cell.available { + if advertised { + t.Fatalf("unmeasured peak advertised as an artifact: %s", peak) + } + if _, err := os.Stat(window.peak); !os.IsNotExist(err) { + t.Fatalf("unavailable footprint produced a peak file: %v", err) + } + var reason string + if err := json.Unmarshal(phase.Gate["heap_profile_load_peak_unavailable"], &reason); err != nil || reason == "" { + t.Fatalf("missing peak has no explicit unavailable reason: %q %v", reason, err) + } + } else if data, err := os.ReadFile(peak); !advertised || err != nil || string(data) != "retained-heap-profile" { + t.Fatalf("measured peak profile lost: %q %v", data, err) + } + }) + } +} + +func TestWorktreeStepAndGaugeOptions(t *testing.T) { + base := []string{"--binary", "unused", "--manifest", "unused", "--scratch", t.TempDir()} + options, err := parseRun(append(base, "--scenario", "worktrees", "--worktrees", "4", "--worktree-steps", "0,1,2,4")) + if err != nil || !slices.Equal(options.WorktreeSteps, []int{0, 1, 2, 4}) { + t.Fatalf("step mode: %+v %v", options.WorktreeSteps, err) + } + if options, err := parseRun(append(base, "--gauges")); err != nil || !options.Gauges || options.HeapProfiles { + t.Fatalf("gauges: %+v %v", options, err) + } + if options, err := parseRun(append(base, "--gauges", "--heap-profiles")); err != nil || !options.Gauges || !options.HeapProfiles { + t.Fatalf("gauges with heap profiles: %+v %v", options, err) + } + for _, extra := range [][]string{ + {"--heap-profiles"}, + {"--scenario", "read", "--worktree-steps", "0,1"}, + {"--scenario", "worktrees", "--worktrees", "4", "--worktree-steps", "0,2,1,4"}, + {"--scenario", "worktrees", "--worktrees", "4", "--worktree-steps", "0,1,1,4"}, + {"--scenario", "worktrees", "--worktrees", "4", "--worktree-steps", "0,1,2"}, + {"--scenario", "worktrees", "--worktrees", "2", "--worktree-steps", "-1,2"}, + {"--scenario", "worktrees", "--worktrees", "2", "--worktree-steps", "0,x"}, + {"--scenario", "worktrees", "--worktrees", "4", "--worktree-steps", "0,4", "--cpu-profile", "10s", "--duration", "20s"}, + } { + if _, err := parseRun(append(base, extra...)); err == nil { + t.Fatalf("invalid memory options accepted: %v", extra) + } + } + // A coupled comparison measures latency between builds; memory evidence + // is taken by standalone runs. + paired := []string{"--build", "tag=/private/tag", "--build", "main=/private/main", "--manifest", "/private/manifest.json", "--scratch", t.TempDir()} + for _, extra := range [][]string{{"--gauges"}, {"--scenario", "worktrees", "--worktrees", "2", "--worktree-steps", "0,2"}} { + if _, _, err := parsePaired(append(paired, extra...)); err == nil { + t.Fatalf("paired accepted memory diagnosis options: %v", extra) + } + } +} + +// The gauge series pairs every daemon_health push with an external process +// sample and lands in the report. Heap profiles from a load window's start +// and sampled peak are taken only when asked for, so a series-only run adds +// nothing but the health ticks to the load it measures. Step mode grows the +// linked worktrees on one daemon, loaded then idle at each count, and +// profiles every load and idle window. +// +// The agent profile thinks for at most 5 s before its first burst, so a 5.5 s +// window always carries workload calls. Saturation would leave calls in +// flight at every window's end, whose advisory cancellations can time out on +// a loaded test host and invalidate the shared stream. An idle window needs +// no calls, so the step idle windows last 300 ms. +func TestMemoryGaugesRecordSeriesAndHeapProfiles(t *testing.T) { + t.Setenv("LOAD_DAEMON_FIXTURE", "serve") + interval, idle := gaugeInterval, stepIdleOverride + gaugeInterval, stepIdleOverride = 100*time.Millisecond, 300*time.Millisecond + t.Cleanup(func() { gaugeInterval, stepIdleOverride = interval, idle }) + binary, err := os.Executable() + if err != nil { + t.Fatal(err) + } + const window = 5500 * time.Millisecond + type healthRow struct { + Coordinators int `json:"coordinators"` + } + readSeries := func(t *testing.T, phase Phase) []gaugeRow { + t.Helper() + var rows []gaugeRow + if err := json.Unmarshal(phase.Gate["gauge_series"], &rows); err != nil || len(rows) < 2 { + t.Fatalf("gauge series not retained: %s %v", phase.Gate["gauge_series"], err) + } + for _, row := range rows { + if row.Received.IsZero() || len(row.Health) == 0 || row.Process["rss_bytes"].Unit != "bytes" { + t.Fatalf("incomplete gauge row: %+v", row) + } + } + return rows + } + // Load-window profiles are taken as of the daemon's last collection; an + // idle-window profile forces one first. + readProfile := func(t *testing.T, path, gc string) { + t.Helper() + if data, err := os.ReadFile(path); err != nil || string(data) != "fixture-heap-profile gc="+gc { + t.Fatalf("heap profile %q not retained: %q %v", path, data, err) + } + } + readPeak := func(t *testing.T, path, reason string, sampled, advertised bool) { + t.Helper() + if sampled { + if reason != "" || !advertised { + t.Fatalf("measured peak marked unavailable: %q %v", reason, advertised) + } + readProfile(t, path, "") + } else if reason == "" || path != "" || advertised { + t.Fatalf("unavailable peak advertised without a reason: %q %q %v", path, reason, advertised) + } + } + + t.Run("gauges", func(t *testing.T) { + for _, profiles := range []bool{false, true} { + options := runOptions{Binary: binary, Profile: "agent", Host: "native", Scenario: "read", Agents: 1, Gauges: true, HeapProfiles: profiles, Duration: window, Deadline: 2 * time.Second, WarmTimeout: 10 * time.Second, StopTimeout: 5 * time.Second} + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + phase, _, artifacts, _, _, err := runPhase(ctx, options, "read", []Corpus{gitFixture(t)}, "", t.TempDir()) + cancel() + if err != nil { + t.Fatalf("gauged read phase (heap profiles %v): %v %+v", profiles, err, phase.Errors) + } + // The series starts once the daemon is ready, ahead of the warmup + // gates, so it covers the level after enrichment before any load. + rows := readSeries(t, phase) + if err := gaugeWarmupOrderError(rows, phase.Attempts); err != nil { + t.Fatal(err) + } + if profiles { + readProfile(t, artifacts["heap_profile_load_start"], "") + sampled := false + for _, row := range rows { + if _, ok := footprintOf(row.Process); ok { + sampled = true + } + } + var reason string + if raw := phase.Gate["heap_profile_load_peak_unavailable"]; len(raw) != 0 { + if err := json.Unmarshal(raw, &reason); err != nil { + t.Fatal(err) + } + } + peak, advertised := artifacts["heap_profile_load_peak"] + readPeak(t, peak, reason, sampled, advertised) + continue + } + taken, _ := filepath.Glob(filepath.Join(artifacts["private_root"], "*.pprof*")) + if _, ok := artifacts["heap_profile_load_peak"]; ok || len(taken) > 0 { + t.Fatalf("series-only run took heap profiles: %v %v", taken, artifacts) + } + } + }) + + t.Run("worktree steps", func(t *testing.T) { + options := runOptions{Binary: binary, Profile: "agent", Host: "native", Scenario: "worktrees", Agents: 2, Worktrees: 2, WorktreeSteps: []int{0, 2}, Duration: window, Deadline: 2 * time.Second, WarmTimeout: 10 * time.Second, StopTimeout: 5 * time.Second} + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + phase, _, artifacts, _, _, err := runPhase(ctx, options, "worktrees", []Corpus{gitFixture(t)}, "", t.TempDir()) + if err != nil { + t.Fatalf("worktree steps: %v %+v", err, phase.Errors) + } + if phase.SessionModel != "per_agent_per_worktree_step:0,2" || len(phase.Tools) == 0 || phase.DurationSec <= 0 { + t.Fatalf("step phase identity/workload: %q %d %v", phase.SessionModel, len(phase.Tools), phase.DurationSec) + } + var steps []worktreeStep + if err := json.Unmarshal(phase.Gate["worktree_steps"], &steps); err != nil || len(steps) != len(options.WorktreeSteps) { + t.Fatalf("steps not retained: %s %v", phase.Gate["worktree_steps"], err) + } + for i, step := range steps { + if step.LinkedWorktrees != options.WorktreeSteps[i] || !step.LoadStarted.Before(step.LoadEnded) || step.IdleStarted.Before(step.LoadEnded) || step.IdleEnded.Sub(step.IdleStarted) < stepIdleOverride || len(step.Tools) == 0 { + t.Fatalf("step %d: %+v", i, step) + } + if step.Load["peak_rss_bytes"].Unit != "bytes" { + t.Fatalf("step %d has no load-window process peaks: %+v", i, step.Load) + } + readProfile(t, step.HeapLoadStart, "") + _, sampled := footprintOf(map[string]Metric{"rss_bytes": step.Load["peak_rss_bytes"], "pss_bytes": step.Load["peak_pss_bytes"], "phys_footprint_bytes": step.Load["peak_phys_footprint_bytes"]}) + _, advertised := artifacts[stepArtifact(i, "heap_profile_load_peak")] + readPeak(t, step.HeapLoadPeak, step.HeapLoadPeakUnavailable, sampled, advertised) + readProfile(t, step.HeapIdle, "1") + if artifacts[stepArtifact(i, "heap_profile_load_peak")] != step.HeapLoadPeak || artifacts[stepArtifact(i, "heap_profile_idle")] != step.HeapIdle { + t.Fatalf("step %d profile is not a run artifact: %v", i, artifacts) + } + } + // The series spans the steps: the daemon's coordinator count is seen + // to grow with the linked worktrees. + seen := make(map[int]bool) + for _, row := range readSeries(t, phase) { + var health healthRow + if err := json.Unmarshal(row.Health, &health); err != nil { + t.Fatal(err) + } + seen[health.Coordinators] = true + } + // The configured primary remains live while linked coordinators grow. + firstTotal := 1 + options.WorktreeSteps[0] + lastTotal := 1 + options.WorktreeSteps[len(options.WorktreeSteps)-1] + if !seen[firstTotal] || !seen[lastTotal] { + t.Fatalf("series did not follow primary+linked coordinators: want %d→%d, seen %v", firstTotal, lastTotal, seen) + } + }) +} + +// A step run is identified by the counts it walks: per-coordinator costs from +// 0,1,2,4 and from 0,2,4 are different measurements, so delta gives no verdict +// between them. +func TestDeltaRefusesStepRunsOverOtherWorktreeCounts(t *testing.T) { + a, b := sampleReport(t), sampleReport(t) + a.Phases[0].SessionModel = worktreeStepSessionModel([]int{0, 1, 2, 4}) + b.Phases[0].SessionModel = worktreeStepSessionModel([]int{0, 2, 4}) + if err := comparable(a, b); err == nil { + t.Fatalf("step runs over different counts compared: %q vs %q", a.Phases[0].SessionModel, b.Phases[0].SessionModel) + } + b.Phases[0].SessionModel = worktreeStepSessionModel([]int{0, 1, 2, 4}) + if err := comparable(a, b); err != nil { + t.Fatalf("step runs over the same counts: %v", err) + } +} diff --git a/bench/load/lifecycle.go b/bench/load/lifecycle.go index aa9d9ce8f..8a2cbd82f 100644 --- a/bench/load/lifecycle.go +++ b/bench/load/lifecycle.go @@ -280,23 +280,33 @@ func (p *privateRun) addWorktrees(ctx context.Context, count int) error { if len(p.Repos) != 1 { return errors.New("worktree scenario requires exactly one primary corpus") } - primary := p.Repos[0] - for i := range count { - path := filepath.Join(p.Root, "repos", fmt.Sprintf("linked-%d", i+1)) - if _, err := p.git(ctx, primary.Path, "worktree", "add", "--detach", path, primary.Corpus.Commit); err != nil { - return err - } - file, err := ownedFile(path, primary.Corpus.File) - if err != nil { + for range count { + if err := p.addWorktree(ctx); err != nil { return err } - corpus := primary.Corpus - corpus.Label = fmt.Sprintf("linked-%d", i+1) - p.Repos = append(p.Repos, privateRepo{Corpus: corpus, Path: path, File: file}) } return nil } +// addWorktree links the next detached worktree of the primary at its pinned +// commit; the Nth linked checkout is labelled linked-N. +func (p *privateRun) addWorktree(ctx context.Context) error { + primary := p.Repos[0] + label := fmt.Sprintf("linked-%d", len(p.Repos)) + path := filepath.Join(p.Root, "repos", label) + if _, err := p.git(ctx, primary.Path, "worktree", "add", "--detach", path, primary.Corpus.Commit); err != nil { + return err + } + file, err := ownedFile(path, primary.Corpus.File) + if err != nil { + return err + } + corpus := primary.Corpus + corpus.Label = label + p.Repos = append(p.Repos, privateRepo{Corpus: corpus, Path: path, File: file}) + return nil +} + func (p *privateRun) writeFile(repo privateRepo, content []byte) error { root, err := canonicalPath(repo.Path) if err != nil || !inside(p.Root, root) || root == p.Root { diff --git a/bench/load/memory_lifecycle_options_test.go b/bench/load/memory_lifecycle_options_test.go new file mode 100644 index 000000000..e91c4985d --- /dev/null +++ b/bench/load/memory_lifecycle_options_test.go @@ -0,0 +1,73 @@ +package main + +import ( + "strings" + "testing" +) + +func TestMemoryGaugesRefuseIdleStop(t *testing.T) { + base := []string{"--binary", "unused", "--manifest", "unused", "--scratch", t.TempDir()} + idle := append(append([]string(nil), base...), "--scenario", "idle-stop", "--agents", "0", "--edit-agents", "0", "--worktrees", "0") + if _, err := parseRun(idle); err != nil { + t.Fatalf("ordinary idle-stop refused: %v", err) + } + for _, profiles := range []bool{false, true} { + memory := append(append([]string(nil), idle...), "--gauges") + if profiles { + memory = append(memory, "--heap-profiles") + } + if _, err := parseRun(memory); err == nil || !strings.Contains(err.Error(), "unsupported") { + t.Errorf("idle-stop accepted an extra live health session: profiles=%v err=%v", profiles, err) + } + } + // Ordinary restart starts its series only after the new process is + // ready; all uses fresh per-scenario processes and excludes idle-stop. + // Step mode enables its supported gauge series implicitly. + for _, extra := range [][]string{ + {"--scenario", "read", "--gauges"}, + {"--scenario", "edit", "--gauges", "--heap-profiles"}, + {"--scenario", "restart", "--gauges"}, + {"--scenario", "restart", "--gauges", "--heap-profiles"}, + {"--scenario", "all", "--gauges"}, + {"--scenario", "all", "--gauges", "--heap-profiles"}, + {"--scenario", "worktrees", "--worktrees", "2", "--worktree-steps", "0,2"}, + } { + if _, err := parseRun(append(append([]string(nil), base...), extra...)); err != nil { + t.Errorf("supported memory measurement refused: %v: %v", extra, err) + } + } +} + +func TestEditProbeMemoryGaugesRefuseRestartSender(t *testing.T) { + corpus := editProbeCorpus() + corpus.EditSender = editProbeRestart + options := runOptions{Scenario: editProbeScenario, Seed: 42, EditCount: 1, Editors: 1} + if _, err := planEditProbe(corpus, options); err != nil { + t.Fatalf("restart cell without memory telemetry refused: %v", err) + } + for _, profiles := range []bool{false, true} { + options.Gauges, options.HeapProfiles = true, profiles + if _, err := planEditProbe(corpus, options); err == nil || !strings.Contains(err.Error(), "unsupported") { + t.Errorf("restart cell accepted stale memory telemetry: profiles=%v err=%v", profiles, err) + } + // Optional series and heap captures do not disable ordinary edit + // probes or native editor saves on one continuous daemon. + for _, supported := range []Corpus{editProbeCorpus(), editProbeEditorSaveCorpus(t, 1)} { + if _, err := planEditProbe(supported, options); err != nil { + t.Errorf("steady edit measurement refused: sender=%q profiles=%v: %v", supported.EditSender, profiles, err) + } + } + } +} + +func TestMemoryGaugesRefuseStartupProbeWithoutARecordedSeries(t *testing.T) { + if err := validateMemoryOptions(runOptions{Scenario: startupScenario}); err != nil { + t.Fatalf("ordinary startup probe refused: %v", err) + } + for _, profiles := range []bool{false, true} { + err := validateMemoryOptions(runOptions{Scenario: startupScenario, Gauges: true, HeapProfiles: profiles}) + if err == nil || !strings.Contains(err.Error(), "unsupported") { + t.Errorf("startup probe silently accepted an unrecorded series: profiles=%v err=%v", profiles, err) + } + } +} diff --git a/bench/load/plan.go b/bench/load/plan.go index 7ca8397e1..d17afa478 100644 --- a/bench/load/plan.go +++ b/bench/load/plan.go @@ -204,6 +204,7 @@ func validateAgentCounts(scenario string, readers, editors int) error { if readers < 1 { return errors.New("zero pure readers are supported only for edit/many-edit") } + case startupScenario: // probes only; reader and editor counts are unused default: return fmt.Errorf("unknown scenario %q", scenario) } diff --git a/bench/load/profile.go b/bench/load/profile.go index bceb93b9c..d877c0b8e 100644 --- a/bench/load/profile.go +++ b/bench/load/profile.go @@ -14,6 +14,23 @@ import ( ) func captureCPUProfile(ctx context.Context, status json.RawMessage, duration time.Duration, path string) error { + return capturePProf(ctx, status, "/debug/pprof/profile?seconds="+strconv.FormatInt(int64(duration/time.Second), 10), duration+15*time.Second, path) +} + +// captureHeapProfile saves the daemon's heap profile as of its last completed +// GC. It does not force a collection, which would stop the world mid-load. +func captureHeapProfile(ctx context.Context, status json.RawMessage, path string) error { + return capturePProf(ctx, status, "/debug/pprof/heap", 30*time.Second, path) +} + +// captureCollectedHeapProfile runs one garbage collection in the daemon before +// saving its heap profile, so the profile and the gauges that follow hold only +// what is still reachable. The collection stops the world: idle windows only. +func captureCollectedHeapProfile(ctx context.Context, status json.RawMessage, path string) error { + return capturePProf(ctx, status, "/debug/pprof/heap?gc=1", 30*time.Second, path) +} + +func capturePProf(ctx context.Context, status json.RawMessage, endpoint string, budget time.Duration, path string) error { var receipt struct { Addr string `json:"pprof_addr"` } @@ -28,10 +45,9 @@ func captureCPUProfile(ctx context.Context, status json.RawMessage, duration tim if ip == nil || !ip.IsLoopback() { return errors.New("refusing non-loopback pprof address") } - profileCtx, cancel := context.WithTimeout(ctx, duration+15*time.Second) + profileCtx, cancel := context.WithTimeout(ctx, budget) defer cancel() - url := "http://" + receipt.Addr + "/debug/pprof/profile?seconds=" + strconv.FormatInt(int64(duration/time.Second), 10) - req, err := http.NewRequestWithContext(profileCtx, http.MethodGet, url, nil) + req, err := http.NewRequestWithContext(profileCtx, http.MethodGet, "http://"+receipt.Addr+endpoint, nil) if err != nil { return err } @@ -51,7 +67,7 @@ func captureCPUProfile(ctx context.Context, status json.RawMessage, duration tim } written, copyErr := io.Copy(file, io.LimitReader(response.Body, 128<<20)) if written >= 128<<20 { - copyErr = errors.New("CPU profile exceeded128MiB") + copyErr = errors.New("profile exceeded 128MiB") } return errors.Join(copyErr, file.Close()) } diff --git a/bench/load/run.go b/bench/load/run.go index 917459dac..a843aeea7 100644 --- a/bench/load/run.go +++ b/bench/load/run.go @@ -32,12 +32,16 @@ type runOptions struct { WarmTimeout time.Duration StopTimeout time.Duration CPUProfile time.Duration + Gauges bool + HeapProfiles bool + WorktreeSteps []int KeepState bool SweepRepetitions int SweepBudget time.Duration EditCount int EditProbeRole string editProbeBaseline bool + Startup startupOptions pairedDriver func(context.Context, *privateRun, []*socketClient, []*socketClient, *Phase) error smokeState *privateRun // only the fixed smoke entry lends its freshly owned store smokeMarkers []string @@ -47,6 +51,7 @@ func parseRun(args []string) (runOptions, error) { var o runOptions f := flag.NewFlagSet("load run", flag.ContinueOnError) bindRunFlags(f, &o) + bindMemoryFlags(f, &o) if err := f.Parse(args); err != nil { return o, err } @@ -54,7 +59,7 @@ func parseRun(args []string) (runOptions, error) { return o, errors.New("unexpected positional run arguments") } o.editProbeBaseline = o.EditProbeRole == "baseline" - return o, validateRunOptions(o) + return o, errors.Join(validateRunOptions(o), validateMemoryOptions(o)) } func bindRunFlags(f *flag.FlagSet, o *runOptions) { @@ -79,6 +84,7 @@ func bindRunFlags(f *flag.FlagSet, o *runOptions) { f.BoolVar(&o.KeepState, "keep-state", false, "retain private corpus/store for native diagnostics only (default removes completed scenario state)") f.IntVar(&o.SweepRepetitions, "sweep-repetitions", 15, "samples per benchmark operation (smoke may explicitly use 1)") f.DurationVar(&o.SweepBudget, "sweep-duration", 10*time.Minute, "separate total budget for the repeated paced sweep") + bindStartupFlags(f, &o.Startup) f.IntVar(&o.EditCount, "edit-count", 100, "edit-and-probe: 100 logical edits, or explicit 1-edit diagnostic") f.StringVar(&o.EditProbeRole, "edit-probe-role", "", "edit-and-probe standalone role: candidate (default; phase accounting required) or baseline (absent phase timings reported unavailable)") } @@ -93,6 +99,9 @@ func validateRunOptions(o runOptions) error { if err := validateAgentCounts(o.Scenario, o.Agents, o.Editors); err != nil { return err } + if err := o.Startup.validate(o.Scenario); err != nil { + return err + } if o.Scenario == "idle-stop" && (o.Worktrees != 0 || o.CPUProfile != 0 || o.Profile != "agent") { return errors.New("idle-stop requires an agent-profile standalone run with no worktrees or CPU profiling") } @@ -136,6 +145,9 @@ func selectedScenarios(o runOptions) ([]string, error) { return []string{"idle-stop"}, nil } all := []string{"read", "edit", "many", "many-edit", "worktrees", "cold", "restart", "sweep"} + if o.Scenario == startupScenario { + return []string{startupScenario}, nil + } if o.Scenario == editProbeScenario { return []string{editProbeScenario}, nil } diff --git a/bench/load/scenario.go b/bench/load/scenario.go index f94603103..079b1cf3b 100644 --- a/bench/load/scenario.go +++ b/bench/load/scenario.go @@ -14,6 +14,9 @@ import ( ) func runPhase(ctx context.Context, o runOptions, scenario string, corpora []Corpus, group, outputRoot string) (phase Phase, sweep []SweepEntry, artifacts map[string]string, crashes map[string]int, version string, resultErr error) { + if scenario == startupScenario { + return runStartupPhase(ctx, o, corpora, outputRoot) + } phase = Phase{Name: scenario, SessionModel: "per_agent", Started: time.Now().UTC(), Gate: make(map[string]json.RawMessage), Metrics: make(map[string]Metric), EditLatency: distribution(nil, "ms"), Visibility: distribution(nil, "ms")} artifacts, crashes = make(map[string]string), make(map[string]int) var scenarioGroupStart map[string]Metric @@ -149,7 +152,7 @@ func runPhase(ctx context.Context, o runOptions, scenario string, corpora []Corp } }() if scenario == "worktrees" { - if err := p.addWorktrees(ctx, o.Worktrees); err != nil { + if err := p.addWorktrees(ctx, bootWorktrees(o)); err != nil { return phase, nil, artifacts, crashes, "", err } } @@ -196,6 +199,27 @@ func runPhase(ctx context.Context, o runOptions, scenario string, corpora []Corp if err != nil { return phase, nil, artifacts, crashes, "", err } + // The gauge series starts as soon as the measured daemon is ready, so it + // also covers the warmup gates and the level after enrichment, before any + // load. A restart replaces the process, so there it starts after the + // restart instead. + var series *gaugeSeries + startSeries := func() (err error) { + if wantsGaugeSeries(o) { + series, err = startGaugeSeries(ctx, p, newProcessSampler(), group) + } + return err + } + defer func() { + if series != nil { + resultErr = errors.Join(resultErr, series.stopInto(&phase)) + } + }() + if scenario != "restart" { + if err := startSeries(); err != nil { + return phase, nil, artifacts, crashes, "", err + } + } var readinessState readiness if err := json.Unmarshal(gate["readiness"], &readinessState); err != nil { return phase, nil, artifacts, crashes, "", err @@ -281,6 +305,9 @@ func runPhase(ctx context.Context, o runOptions, scenario string, corpora []Corp if err != nil { return phase, nil, artifacts, crashes, version, err } + if err := startSeries(); err != nil { + return phase, nil, artifacts, crashes, version, err + } client, err := openMCP(warmCtx, p.Socket, p.Repos[0].Path) if err != nil { return phase, nil, artifacts, crashes, version, err @@ -335,7 +362,7 @@ func runPhase(ctx context.Context, o runOptions, scenario string, corpora []Corp } else if o.pairedDriver != nil { err = o.pairedDriver(ctx, p, clients, sweepClients, &phase) } else { - err = measureStandalone(ctx, o, p, clients, sweepClients, sweepRequests, sweep, &phase, artifacts, sampler, group) + err = measureStandalone(ctx, o, p, clients, sweepClients, sweepRequests, sweep, &phase, artifacts, series, sampler, group) } if err != nil { resultErr = errors.Join(resultErr, err) @@ -376,7 +403,11 @@ func runPhase(ctx context.Context, o runOptions, scenario string, corpora []Corp return phase, sweep, artifacts, crashes, version, resultErr } -func measureStandalone(ctx context.Context, o runOptions, p *privateRun, clients, sweepClients []*socketClient, sweepRequests []request, sweep []SweepEntry, phase *Phase, artifacts map[string]string, sampler processSampler, group string) error { +func measureStandalone(ctx context.Context, o runOptions, p *privateRun, clients, sweepClients []*socketClient, sweepRequests []request, sweep []SweepEntry, phase *Phase, artifacts map[string]string, series *gaugeSeries, sampler processSampler, group string) error { + if len(o.WorktreeSteps) > 0 { + return measureWorktreeSteps(ctx, o, p, clients[0], phase, artifacts, series, sampler, group) + } + heap := series.watchLoadProfiles(ctx, o, p, phase) phase.Started = time.Now() monitor := startMonitor(p, sampler, group) var profileDone chan error @@ -403,6 +434,7 @@ func measureStandalone(ctx context.Context, o runOptions, p *privateRun, clients } phase.DurationSec = time.Since(phase.Started).Seconds() monitor.finish(phase) + err = errors.Join(err, series.endLoadProfiles(heap, phase, artifacts)) if profileDone != nil { err = errors.Join(err, <-profileDone) } diff --git a/bench/load/selftest_test.go b/bench/load/selftest_test.go index 2f0fd37c2..208a28bde 100644 --- a/bench/load/selftest_test.go +++ b/bench/load/selftest_test.go @@ -5,11 +5,13 @@ import ( "encoding/json" "fmt" "net" + "net/http" "os" "os/exec" "path/filepath" "strings" "sync" + "sync/atomic" "testing" "time" ) @@ -29,6 +31,17 @@ func fixtureDaemon(mode string) int { fmt.Fprintln(os.Stderr, "panic: synthetic daemon crash") return 2 } + // The startup probe runs the CLI and the hook through the same binary; + // answer them as against a ready daemon instead of starting another one. + for _, arg := range os.Args[1:] { + switch arg { + case "call": + fmt.Println(`{"results":[]}`) + return 0 + case "hook": + return 0 + } + } data, err := os.ReadFile(filepath.Join(os.Getenv("XDG_CONFIG_HOME"), "gortex", "config.yaml")) if err != nil { return 3 @@ -55,7 +68,38 @@ func fixtureDaemon(mode string) int { return 5 } defer listener.Close() + // A loopback pprof listener whose heap endpoint serves a fixed body that + // names any forced collection, so captured profiles are recognisable. + pprof, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + return 6 + } + defer pprof.Close() + go func() { + _ = http.Serve(pprof, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path == "/debug/pprof/heap" { + _, _ = w.Write([]byte("fixture-heap-profile gc=" + r.URL.Query().Get("gc"))) + return + } + http.NotFound(w, r) + })) + }() + // Linked checkouts get a view the first time a control search names them; + // daemon_health reports how many there are as its coordinator count. + var viewsMu sync.Mutex + views := make(map[string]bool) + // Configured roots already own their primary coordinators. Linked views + // join the same census when a control search first names the checkout. + for _, repo := range config.Repos { + views[filepath.Clean(repo.Path)] = true + } + coordinators := func() int { + viewsMu.Lock() + defer viewsMu.Unlock() + return len(views) + } var shutdown sync.Once + var warmupCalls atomic.Uint64 for { conn, err := listener.Accept() if err != nil { @@ -64,14 +108,22 @@ func fixtureDaemon(mode string) int { go func() { defer conn.Close() decoder := json.NewDecoder(conn) - encoder := json.NewEncoder(conn) + var writeMu sync.Mutex + encode := func(value any) error { + writeMu.Lock() + defer writeMu.Unlock() + return json.NewEncoder(conn).Encode(value) + } var handshake map[string]any if decoder.Decode(&handshake) != nil { return } - if encoder.Encode(map[string]any{"ok": true, "daemon_version": "fixture-v1"}) != nil { + if encode(map[string]any{"ok": true, "daemon_version": "fixture-v1"}) != nil { return } + healthDone := make(chan struct{}) + defer close(healthDone) + var healthOnce sync.Once for { var frame stubFrame if decoder.Decode(&frame) != nil { @@ -89,13 +141,20 @@ func fixtureDaemon(mode string) int { } result = map[string]any{"ready": true, "enriched": true, "tracked_repos": repos, "loaded_roots": roots} case "status": - result = map[string]any{"ready": true, "enriched": true, "views": map[string]any{}, "storage": map[string]any{"wal_pending_frames": 0}} + result = map[string]any{"ready": true, "enriched": true, "views": map[string]any{"coordinators": coordinators()}, "storage": map[string]any{"wal_pending_frames": 0}, "pprof_addr": pprof.Addr().String()} + case "search_symbols": + if path, _ := frame.Params["path"].(string); path != "" { + viewsMu.Lock() + views[filepath.Clean(path)] = true + viewsMu.Unlock() + } + result = map[string]any{"view": map[string]any{"kind": "worktree", "exact": true, "repo_prefix": config.Repos[0].Name}} case "shutdown": result = map[string]any{"stopping": true} default: result = map[string]any{} } - if encoder.Encode(map[string]any{"ok": true, "result": result}) != nil { + if encode(map[string]any{"ok": true, "result": result}) != nil { return } if frame.Kind == "shutdown" { @@ -116,11 +175,49 @@ func fixtureDaemon(mode string) int { } result = map[string]any{"contents": []any{map[string]any{"type": "text", "text": payload}}} case "tools/call": - if frame.Params["name"] == "analyze" { + if frame.Params["name"] != "session" { + warmupCalls.Add(1) + } + arguments, _ := frame.Params["arguments"].(map[string]any) + switch { + case frame.Params["name"] == "analyze": result = stubContent(map[string]any{"hotspots": []any{}, "total": 0}) - } else if frame.Params["name"] == "session" { + case frame.Params["name"] == "session" && arguments["channel"] == "daemon_health" && arguments["operation"] == "subscribe": + interval := time.Second + if extra, ok := arguments["arguments"].(map[string]any); ok { + if ms, ok := extra["interval_ms"].(float64); ok && ms > 0 { + interval = time.Duration(ms) * time.Millisecond + } + } + healthOnce.Do(func() { + sendHealth := func(initial bool) error { + health := map[string]any{"ts": time.Now().UTC().Format(time.RFC3339Nano), "coordinators": coordinators(), "sqlite_page_cache_bytes": 4096, "initial_replay": initial, "fixture_warmup_calls": warmupCalls.Load()} + return encode(map[string]any{"jsonrpc": "2.0", "method": "notifications/daemon_health", "params": health}) + } + // Production delivers its initial replay before the subscribe + // reply. Keep that causal boundary independent of clock ticks. + if sendHealth(true) != nil { + return + } + go func() { + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + select { + case <-healthDone: + return + case <-ticker.C: + if sendHealth(false) != nil { + return + } + } + } + }() + }) + result = stubContent(map[string]any{"subscribed": true}) + case frame.Params["name"] == "session": result = stubContent(map[string]any{}) - } else if nativeReplies != nil { + case nativeReplies != nil: args, _ := frame.Params["arguments"].(map[string]any) target, _ := args["target"].(map[string]any) if frame.Params["name"] == "read" && args["operation"] == "source" && target["symbol"] == "fixture/"+nativeCorpus.Symbol { @@ -128,11 +225,11 @@ func fixtureDaemon(mode string) int { } else { result, _ = nativeReplies(frame) } - } else { + default: result = stubContent(map[string]any{"id": "sample/sample.go::Example", "source": "func Example() {}"}) } } - if encoder.Encode(map[string]any{"jsonrpc": "2.0", "id": frame.ID, "result": result}) != nil { + if encode(map[string]any{"jsonrpc": "2.0", "id": frame.ID, "result": result}) != nil { return } } diff --git a/bench/load/startup_probe.go b/bench/load/startup_probe.go new file mode 100644 index 000000000..37bc3f4fc --- /dev/null +++ b/bench/load/startup_probe.go @@ -0,0 +1,345 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "os" + "path/filepath" + "strings" + "time" +) + +// The startup scenario observes repository admission from before the +// daemon's socket exists until every configured repository is ready and +// enriched, on an initial boot and on each optional restart. The probes keep +// sampling across the gap between boots, so a restart is seen exactly as an +// agent sees it. A seeded store makes the initial boot an upgrade: its +// migration runs before the socket listens. + +const ( + startupScenario = "startup" + startupRosterWorkspace = "probe-roster" + startupWatchInterval = 100 * time.Millisecond + startupCoordinatorWindow = 2 * time.Second +) + +type startupOptions struct { + Restarts int + SeedStore string +} + +func bindStartupFlags(f *flag.FlagSet, o *startupOptions) { + f.IntVar(&o.Restarts, "restarts", 0, "startup: warm restarts after the initial boot, each probed from before its socket exists") + f.StringVar(&o.SeedStore, "seed-store", "", "startup: store file copied into the private run before the initial boot, e.g. one written by an older schema; the source is only read") +} + +func (o startupOptions) validate(scenario string) error { + if o.Restarts < 0 { + return errors.New("--restarts must be nonnegative") + } + if scenario != startupScenario && (o.Restarts != 0 || o.SeedStore != "") { + return errors.New("--restarts and --seed-store apply to the startup scenario only") + } + return nil +} + +// bootTiming holds what the readiness watcher observed for one daemon start. +// Every instant is an observed upper bound at the watcher's poll interval. +type bootTiming struct { + Boot int `json:"boot"` + Started time.Time `json:"started"` + Stopping time.Time `json:"stopping"` + SocketAt time.Time `json:"socket_at"` + ReadyAt map[string]time.Time `json:"ready_at"` + EnrichedAt time.Time `json:"enriched_at"` + // Coordinators is the largest checkout coordinator count the status + // census read in the 2 s after the primary registered. + Coordinators *int `json:"coordinators_within_2s"` + CensusReads int `json:"census_reads"` + CensusErrors []string `json:"census_errors"` + Views json.RawMessage `json:"views"` +} + +func runStartupPhase(ctx context.Context, o runOptions, corpora []Corpus, outputRoot string) (phase Phase, _ []SweepEntry, artifacts map[string]string, crashes map[string]int, version string, resultErr error) { + phase = Phase{Name: startupScenario, SessionModel: "fresh_session_per_probe_tick", Started: time.Now().UTC(), Gate: make(map[string]json.RawMessage), Metrics: make(map[string]Metric), EditLatency: distribution(nil, "ms"), Visibility: distribution(nil, "ms")} + artifacts, crashes = make(map[string]string), make(map[string]int) + phase.Operations = startupOperations(corpora[0]) + defer func() { + phase.DurationSec = time.Since(phase.Started).Seconds() + if resultErr != nil { + phase.Errors = append(phase.Errors, resultErr.Error()) + } else if len(phase.Errors) > 0 { + resultErr = errors.New(strings.Join(phase.Errors, "; ")) + } + }() + p, err := newPrivateRun(ctx, outputRoot, o.Binary, corpora) + if err != nil { + return phase, nil, artifacts, crashes, "", err + } + artifacts["private_root"] = p.Root + defer func() { + stopCtx, cancel := context.WithTimeout(context.Background(), o.StopTimeout) + _, forced, err := p.stop(stopCtx) + cancel() + if forced { + err = errors.Join(err, errors.New("final stop was forced")) + } + logArtifacts, logCrashes, logErr := collectLogs(p.Root) + for name, path := range logArtifacts { + artifacts[name] = path + } + for name, count := range logCrashes { + crashes[name] += count + } + resultErr = errors.Join(resultErr, err, logErr, os.RemoveAll(p.SocketDir)) + if !o.KeepState { + resultErr = errors.Join(resultErr, p.removeState()) + } + }() + linked, err := addLinkedCheckouts(ctx, p, o.Worktrees) + if err != nil { + return phase, nil, artifacts, crashes, "", err + } + routerDir, err := writeRouterRoster(p) + if err != nil { + return phase, nil, artifacts, crashes, "", err + } + configured, err := p.writeBootRepos(false) + if err != nil { + return phase, nil, artifacts, crashes, "", err + } + if o.Startup.SeedStore != "" { + if err := copySeedStore(o.Startup.SeedStore, p.Store); err != nil { + return phase, nil, artifacts, crashes, "", err + } + } + cwds := []probeCWD{{Role: "primary", Dir: p.Repos[0].Path, File: p.Repos[0].File}} + cwds = append(cwds, linked[:min(1, len(linked))]...) + cwds = append(cwds, probeCWD{Role: "router", Dir: routerDir, File: p.Repos[0].File}) + logPath := filepath.Join(p.Root, "startup-probe.jsonl") + probe, err := newStartupProbe(p, cwds, corpora[0].Query, logPath) + if err != nil { + return phase, nil, artifacts, crashes, "", err + } + artifacts["startup_probe_log"] = logPath + probeCtx, stopProbe := context.WithCancel(ctx) + probe.start(probeCtx) + var boots []bootTiming + for index := 0; index <= o.Startup.Restarts; index++ { + if index > 0 { + probe.stopping() + boots[index-1].Stopping = time.Now() + stopCtx, cancel := context.WithTimeout(ctx, o.StopTimeout) + _, forced, err := p.stop(stopCtx) + cancel() + if err != nil || forced { + resultErr = fmt.Errorf("restart %d: stop failed or was forced: %v", index, err) + break + } + } + timing, err := runStartupBoot(ctx, o, p, probe, configured, index) + boots = append(boots, timing) + if err != nil { + resultErr = err + break + } + } + if resultErr == nil { + receiptCtx, cancel := context.WithTimeout(ctx, 30*time.Second) + client, err := openMCP(receiptCtx, p.Socket, p.Repos[0].Path) + if err == nil { + captureReceipt(receiptCtx, p, client, &phase) + err = client.Close() + } + cancel() + resultErr = err + } + if len(boots) > 0 && boots[len(boots)-1].Stopping.IsZero() { + boots[len(boots)-1].Stopping = time.Now() + } + stopProbe() + samples, probeErr := probe.finish() + resultErr = errors.Join(resultErr, probeErr) + summary := summarizeStartup(samples, boots, p.Repos[0].Corpus.Label, len(linked)) + summary.CLITTY = probe.ttyStatus + for name, metric := range summary.metrics() { + phase.Metrics[name] = metric + } + if data, err := json.Marshal(summary); err == nil { + phase.Gate["startup_probe"] = data + } else { + resultErr = errors.Join(resultErr, err) + } + return phase, nil, artifacts, crashes, probe.daemonVersion(), resultErr +} + +// runStartupBoot starts the daemon and follows it until every configured +// repository is ready, the daemon reports enrichment, and the measured tail +// (--duration) has passed, so the next restart begins from a warm store. +func runStartupBoot(ctx context.Context, o runOptions, p *privateRun, probe *startupProbe, configured []privateRepo, index int) (bootTiming, error) { + timing := bootTiming{Boot: index, ReadyAt: make(map[string]time.Time)} + timing.Started = time.Now() + probe.beginBoot(index) + if err := p.start(); err != nil { + return timing, err + } + primary := configured[0].Corpus.Label + for { + probeCtx, cancel := context.WithTimeout(ctx, time.Second) + data, err := control(probeCtx, p.Socket, "probe", nil) + cancel() + now := time.Now() // after the answer: an upper bound, never early + if err == nil { + if timing.SocketAt.IsZero() { + timing.SocketAt = now + probe.socketUp(index) + } + var state readiness + if err := json.Unmarshal(data, &state); err != nil { + return timing, err + } + if err := observeReadiness(&timing, state, configured, now); err != nil { + return timing, err + } + } + ready, registered := timing.ReadyAt[primary] + if registered && now.Sub(ready) <= startupCoordinatorWindow { + readCoordinators(ctx, p, &timing) + } + if !timing.EnrichedAt.IsZero() && now.Sub(timing.EnrichedAt) >= o.Duration && now.Sub(ready) > startupCoordinatorWindow { + return timing, nil + } + select { + case <-p.done: + return timing, fmt.Errorf("boot %d: daemon exited: %v", index, p.exitErr) + default: + } + if now.Sub(timing.Started) > o.WarmTimeout { + return timing, fmt.Errorf("boot %d: configured repositories not ready and enriched within %s", index, o.WarmTimeout) + } + if err := waitContext(ctx, startupWatchInterval); err != nil { + return timing, err + } + } +} + +// observeReadiness stamps each configured repository the first time the probe +// lists it as ready. Daemons that predate per-repository state list a +// registered repository with its prefix and no state. +func observeReadiness(timing *bootTiming, state readiness, configured []privateRepo, now time.Time) error { + for _, repo := range configured { + if _, seen := timing.ReadyAt[repo.Corpus.Label]; seen { + continue + } + for _, tracked := range state.TrackedRepos { + if filepath.Clean(tracked.Path) == filepath.Clean(repo.Path) && tracked.Prefix != "" && (tracked.State == "ready" || tracked.State == "") { + timing.ReadyAt[repo.Corpus.Label] = now + } + } + } + covered, err := readyCoverage(state, configured) + if err == nil && covered && state.Ready && state.Enriched && timing.EnrichedAt.IsZero() && len(timing.ReadyAt) == len(configured) { + timing.EnrichedAt = now + } + return err +} + +func readCoordinators(ctx context.Context, p *privateRun, timing *bootTiming) { + statusCtx, cancel := context.WithTimeout(ctx, time.Second) + data, err := control(statusCtx, p.Socket, "status", nil) + cancel() + timing.CensusReads++ + var status struct { + Views json.RawMessage `json:"views"` + } + var views struct { + Coordinators *int `json:"coordinators"` + } + if err == nil { + err = json.Unmarshal(data, &status) + } + if err == nil && len(status.Views) > 0 { + err = json.Unmarshal(status.Views, &views) + } + if err == nil && views.Coordinators == nil { + err = errors.New("status has no checkout coordinator census") + } + if err != nil { + if len(timing.CensusErrors) < 5 { + timing.CensusErrors = append(timing.CensusErrors, err.Error()) + } + return + } + timing.Views = status.Views + if timing.Coordinators == nil || *views.Coordinators > *timing.Coordinators { + timing.Coordinators = views.Coordinators + } +} + +// addLinkedCheckouts adds linked worktrees of the first manifest repository. +// They are not configured: the daemon discovers them through the primary. +func addLinkedCheckouts(ctx context.Context, p *privateRun, count int) ([]probeCWD, error) { + primary := p.Repos[0] + var linked []probeCWD + for i := range count { + path := filepath.Join(p.Root, "repos", fmt.Sprintf("linked-%d", i+1)) + if _, err := p.git(ctx, primary.Path, "worktree", "add", "--detach", path, primary.Corpus.Commit); err != nil { + return nil, err + } + file, err := ownedFile(path, primary.Corpus.File) + if err != nil { + return nil, err + } + linked = append(linked, probeCWD{Role: "worktree", Dir: path, File: file}) + } + return linked, nil +} + +// writeRouterRoster gives one cwd inside the primary a workspace that a +// servers.toml roster claims. The roster's only server is a socket that never +// exists, so a routed call fails to dial and the daemon answers locally; the +// primary and worktree cwds keep their ordinary path. +func writeRouterRoster(p *privateRun) (string, error) { + dir := filepath.Join(p.Repos[0].Path, ".gortex-probe-router") + if err := os.MkdirAll(dir, 0o700); err != nil { + return "", err + } + if err := os.WriteFile(filepath.Join(dir, ".gortex.yaml"), []byte("workspace: "+startupRosterWorkspace+"\n"), 0o600); err != nil { + return "", err + } + configDir := filepath.Join(p.Root, "config", "gortex") + if err := os.MkdirAll(configDir, 0o700); err != nil { + return "", err + } + remote := "unix://" + filepath.ToSlash(filepath.Join(p.Root, "absent-remote.sock")) + roster := fmt.Sprintf("[[server]]\nslug = \"probe-remote\"\nurl = %q\nworkspaces = [%q]\n", remote, startupRosterWorkspace) + return dir, os.WriteFile(filepath.Join(configDir, "servers.toml"), []byte(roster), 0o600) +} + +// copySeedStore copies a quiescent store, and its write-ahead log when there +// is one, into the private run. The source is opened read-only. +func copySeedStore(source, dest string) error { + for _, suffix := range []string{"", "-wal"} { + in, err := os.Open(source + suffix) + if suffix != "" && errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + return fmt.Errorf("seed store: %w", err) + } + out, err := os.OpenFile(dest+suffix, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600) + if err != nil { + _ = in.Close() + return fmt.Errorf("seed store: %w", err) + } + _, err = io.Copy(out, in) + if err = errors.Join(err, in.Close(), out.Close()); err != nil { + return fmt.Errorf("seed store: %w", err) + } + } + return nil +} diff --git a/bench/load/startup_probe_sample.go b/bench/load/startup_probe_sample.go new file mode 100644 index 000000000..0b6b01eb0 --- /dev/null +++ b/bench/load/startup_probe_sample.go @@ -0,0 +1,610 @@ +package main + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "errors" + "fmt" + "net" + "os" + "os/exec" + "regexp" + "runtime" + "strconv" + "strings" + "sync" + "time" +) + +const ( + startupProbeInterval = time.Second + startupCallTimeout = 10 * time.Second + // An interactive CLI call waits up to 30 s for a loading repository. + startupCLITimeout = 45 * time.Second + startupTextLimit = 240 +) + +// probeCWD is one directory an agent could start in, and an indexed file the +// native-Read hook probe reads from there. +type probeCWD struct { + Role string // primary, worktree or router + Dir string + File string +} + +// probeSample is one answer as the per-second log records it. Outcome fields +// come from the public wire only: JSON-RPC error data, tool results, CLI exit +// codes and output, and the hook's decision. +type probeSample struct { + CWD string `json:"cwd"` + Surface string `json:"surface"` + Started time.Time `json:"started"` + LatencyMS float64 `json:"latency_ms"` + OK bool `json:"ok"` + Code string `json:"error_code,omitempty"` + Retryable *bool `json:"retryable,omitempty"` + Phase string `json:"phase,omitempty"` + Percent *int `json:"percent,omitempty"` + Status string `json:"status,omitempty"` + Variant string `json:"variant,omitempty"` + ExitCode *int `json:"exit_code,omitempty"` + TrackAdvice bool `json:"track_advice,omitempty"` + Text string `json:"text,omitempty"` +} + +func (s probeSample) completed() time.Time { + return s.Started.Add(time.Duration(s.LatencyMS * float64(time.Millisecond))) +} + +// startupOperations is the offered MCP probe plan. The same values build the +// frames, so the plan hash describes exactly what was sent. +func startupOperations(c Corpus) []request { + return []request{ + {Tool: "initialize"}, + {Tool: "search_symbols", Arguments: map[string]any{"query": c.Query, "limit": 1}}, + {Tool: "resources/read", Operation: "index-health", Arguments: map[string]any{"uri": "gortex://index-health"}}, + {Tool: "workspace", Operation: "index", Arguments: map[string]any{"operation": "index"}}, + } +} + +type startupProbe struct { + socket, binary, query string + env []string + cwds []probeCWD + plan []request + ttyStatus string + workers sync.WaitGroup + + mu sync.Mutex + samples []probeSample + log *os.File + encoder *json.Encoder + logErr error + boot int + socketOn bool + version string +} + +func newStartupProbe(p *privateRun, cwds []probeCWD, query, logPath string) (*startupProbe, error) { + log, err := os.OpenFile(logPath, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600) + if err != nil { + return nil, err + } + s := &startupProbe{socket: p.Socket, binary: p.Binary, query: query, env: p.Env, cwds: cwds, plan: startupOperations(Corpus{Query: query}), log: log, encoder: json.NewEncoder(log), boot: -1} + s.ttyStatus = "script(1) pty" + if _, err := ttyCommand(context.Background(), "true", nil); err != nil { + s.ttyStatus = "unavailable: " + err.Error() + } else if _, err := exec.LookPath("script"); err != nil { + s.ttyStatus = "unavailable: " + err.Error() + } + return s, nil +} + +// start launches every sampler. They run until ctx ends, across boots. +func (s *startupProbe) start(ctx context.Context) { + for _, cwd := range s.cwds { + s.every(ctx, func(ctx context.Context) { s.mcpTick(ctx, cwd) }) + s.every(ctx, func(ctx context.Context) { s.recordLive(ctx, s.hookRead(ctx, cwd)) }) + s.cliLoop(ctx, cwd, false) + if strings.HasPrefix(s.ttyStatus, "script") { + s.cliLoop(ctx, cwd, true) + } + } +} + +func (s *startupProbe) every(ctx context.Context, tick func(context.Context)) { + s.workers.Go(func() { + ticker := time.NewTicker(startupProbeInterval) + defer ticker.Stop() + for ctx.Err() == nil { + tick(ctx) + select { + case <-ctx.Done(): + case <-ticker.C: + } + } + }) +} + +// cliLoop runs one CLI variant every second until it first succeeds in the +// current boot. The pipe variant also runs before the socket exists; the pty +// variant starts once it does, because an interactive call waits through the +// load and its time to exit is the measurement. +func (s *startupProbe) cliLoop(ctx context.Context, cwd probeCWD, tty bool) { + done := -1 + s.every(ctx, func(ctx context.Context) { + boot, socketOn := s.bootState() + if boot < 0 || boot == done || (tty && !socketOn) { + return + } + sample := s.cliCall(ctx, cwd, tty) + if s.recordLive(ctx, sample) && sample.OK { + done = boot + } + }) +} + +func (s *startupProbe) beginBoot(index int) { + s.mu.Lock() + s.boot, s.socketOn = index, false + s.mu.Unlock() +} + +func (s *startupProbe) socketUp(index int) { + s.mu.Lock() + if s.boot == index { + s.socketOn = true + } + s.mu.Unlock() +} + +func (s *startupProbe) stopping() { + s.mu.Lock() + s.socketOn = false + s.mu.Unlock() +} + +func (s *startupProbe) bootState() (int, bool) { + s.mu.Lock() + defer s.mu.Unlock() + return s.boot, s.socketOn +} + +func (s *startupProbe) daemonVersion() string { + s.mu.Lock() + defer s.mu.Unlock() + return s.version +} + +// recordLive keeps a sample unless the probe was stopping when it ended: a +// call cut short by the end of the run is not an answer. +func (s *startupProbe) recordLive(ctx context.Context, sample probeSample) bool { + if ctx.Err() != nil { + return false + } + s.mu.Lock() + defer s.mu.Unlock() + s.samples = append(s.samples, sample) + if s.logErr == nil { + s.logErr = s.encoder.Encode(sample) + } + return true +} + +func (s *startupProbe) finish() ([]probeSample, error) { + s.workers.Wait() + s.mu.Lock() + defer s.mu.Unlock() + return s.samples, errors.Join(s.logErr, s.log.Close()) +} + +// mcpTick opens fresh sessions from cwd, as a newly started agent would: the +// legacy surface for search_symbols and the index-health resource, and the +// compact surface for workspace(operation:"index"). Each session records the +// instructions variant its initialize returned. +func (s *startupProbe) mcpTick(ctx context.Context, cwd probeCWD) { + sessions := []struct { + tools, mode, initialize string + steps []request + }{ + {"core", "defer", "initialize", s.plan[1:3]}, + {"facade-v1", "hide", "initialize_facade", s.plan[3:]}, + } + for _, session := range sessions { + if !s.mcpSession(ctx, cwd, session.tools, session.mode, session.initialize, session.steps) { + return + } + } +} + +func (s *startupProbe) mcpSession(ctx context.Context, cwd probeCWD, tools, mode, initialize string, steps []request) bool { + started := time.Now() + dialCtx, cancel := context.WithTimeout(ctx, startupCallTimeout) + conn, reader, version, err := dialSocketSurface(dialCtx, s.socket, cwd.Dir, "mcp", tools, mode) + cancel() + if err != nil { + code := "handshake_error" + var opErr *net.OpError + if errors.As(err, &opErr) && opErr.Op == "dial" { + code = "no_socket" + } + s.recordLive(ctx, probeSample{CWD: cwd.Role, Surface: "connect", Started: started, LatencyMS: elapsedMS(started), Code: code, Text: clip(err.Error())}) + return false + } + defer conn.Close() + s.mu.Lock() + s.version = version + s.mu.Unlock() + c := &probeConn{conn: conn, reader: reader} + initParams := map[string]any{"protocolVersion": "2024-11-05", "capabilities": map[string]any{}, "clientInfo": map[string]string{"name": "gortex-load", "version": "1"}} + calls := []probeCall{{initialize, "initialize", initParams}} + for _, step := range steps { + switch { + case step.Tool == "resources/read": + calls = append(calls, probeCall{"index_health_resource", step.Tool, step.Arguments}) + case step.Operation != "": + calls = append(calls, probeCall{step.Tool + "_" + step.Operation, "tools/call", map[string]any{"name": step.Tool, "arguments": step.Arguments}}) + default: + calls = append(calls, probeCall{step.Tool, "tools/call", map[string]any{"name": step.Tool, "arguments": step.Arguments}}) + } + } + for i, call := range calls { + start := time.Now() + callCtx, cancel := context.WithTimeout(ctx, startupCallTimeout) + frame, err := c.roundTrip(callCtx, call.method, call.params) + timedOut := callCtx.Err() == context.DeadlineExceeded + cancel() + var sample probeSample + switch { + case err == nil: + sample = decodeProbeFrame(frame) + case timedOut: + sample = probeSample{Code: "timeout", Text: clip(err.Error())} + default: + sample = probeSample{Code: "transport_error", Text: clip(err.Error())} + } + sample.CWD, sample.Surface, sample.Started, sample.LatencyMS = cwd.Role, call.surface, start, elapsedMS(start) + if !s.recordLive(ctx, sample) || err != nil { + return ctx.Err() == nil + } + if i == 0 && c.notify("notifications/initialized") != nil { + return true + } + } + return true +} + +type probeCall struct { + surface, method string + params any +} + +type probeConn struct { + conn net.Conn + reader *bufio.Reader + next int +} + +// roundTrip sends one request and returns its response frame, skipping +// notifications and server-initiated requests. +func (c *probeConn) roundTrip(ctx context.Context, method string, params any) ([]byte, error) { + c.next++ + id := strconv.Itoa(c.next) + data, err := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": c.next, "method": method, "params": params}) + if err != nil { + return nil, err + } + if deadline, ok := ctx.Deadline(); ok { + if err := c.conn.SetDeadline(deadline); err != nil { + return nil, err + } + } + stop := context.AfterFunc(ctx, func() { _ = c.conn.SetDeadline(time.Now()) }) + defer stop() + if _, err := c.conn.Write(append(data, '\n')); err != nil { + return nil, err + } + for { + line, err := readWireLine(c.reader) + if err != nil { + return nil, err + } + var peek struct { + ID json.RawMessage `json:"id"` + Method string `json:"method"` + } + if err := json.Unmarshal(line, &peek); err != nil { + return nil, fmt.Errorf("invalid daemon JSON: %w", err) + } + if peek.Method == "" && string(peek.ID) == id { + return line, nil + } + } +} + +func (c *probeConn) notify(method string) error { + data, err := json.Marshal(map[string]any{"jsonrpc": "2.0", "method": method}) + if err == nil { + _, err = c.conn.Write(append(data, '\n')) + } + return err +} + +// admissionFields are the machine-readable fields of a refusal's JSON-RPC +// error data, or of a tool's JSON payload. +type admissionFields struct { + ErrorCode string `json:"error_code"` + Retryable *bool `json:"retryable"` + Phase string `json:"phase"` + Percent *float64 `json:"percent"` + Status string `json:"status"` +} + +func (f admissionFields) apply(s *probeSample) { + if f.ErrorCode != "" { + s.Code = f.ErrorCode + } + s.Retryable, s.Phase, s.Status = f.Retryable, f.Phase, f.Status + if f.Percent != nil { + percent := int(*f.Percent) + s.Percent = &percent + } +} + +// decodeProbeFrame classifies one JSON-RPC response frame. +func decodeProbeFrame(frame []byte) probeSample { + var reply struct { + Error *struct { + Code int `json:"code"` + Message string `json:"message"` + Data json.RawMessage `json:"data"` + } `json:"error"` + Result *struct { + Instructions *string `json:"instructions"` + IsError bool `json:"isError"` + Content []struct { + Text string `json:"text"` + } `json:"content"` + Contents []struct { + Text string `json:"text"` + } `json:"contents"` + } `json:"result"` + } + if err := json.Unmarshal(frame, &reply); err != nil || (reply.Error == nil && reply.Result == nil) { + return probeSample{Code: "invalid_response", Text: clip(string(frame))} + } + // Successful graph answers can quote indexed text that mentions the + // commands, so only refusals and instructions are checked for advice. + var sample probeSample + if reply.Error != nil { + sample.TrackAdvice = adviceIn(string(frame)) + sample.Code = fmt.Sprintf("rpc_%d", reply.Error.Code) + var fields admissionFields + _ = json.Unmarshal(reply.Error.Data, &fields) + fields.apply(&sample) + sample.Text = clip(reply.Error.Message) + return sample + } + result := reply.Result + if result.Instructions != nil { + sample.OK, sample.Variant, sample.TrackAdvice = true, instructionsVariant(*result.Instructions), adviceIn(*result.Instructions) + if sample.Variant != "active" { + sample.Text = clip(*result.Instructions) + } + return sample + } + text := "" + for _, block := range append(result.Content, result.Contents...) { + if text = block.Text; text != "" { + break + } + } + var fields admissionFields + _ = json.Unmarshal([]byte(text), &fields) + fields.apply(&sample) + sample.OK = !result.IsError && fields.ErrorCode == "" + if !sample.OK { + if sample.Code == "" { + sample.Code = "tool_error" + } + sample.Text, sample.TrackAdvice = clip(text), adviceIn(string(frame)) + } + return sample +} + +// instructionsVariant names the initialize instructions an agent received. +// Anything without one of the inactive or loading preambles is active. +func instructionsVariant(text string) string { + for _, variant := range []struct{ prefix, name string }{ + {"Gortex is LOADING graph admission", "admission_pending"}, + {"Gortex is LOADING ", "loading"}, + {"Gortex is connected but could not load", "unavailable"}, + {"Gortex is connected but INACTIVE", "untracked"}, + {"Gortex is connected; checkout discovery is pending", "checkout_pending"}, + {"Gortex checkout discovery is unavailable", "checkout_admission_error"}, + } { + if strings.HasPrefix(text, variant.prefix) { + return variant.name + } + } + if strings.TrimSpace(text) == "" { + return "empty" + } + return "active" +} + +// trackAdvicePattern finds advice to run `gortex track` or `gortex repos +// reconcile`. A negated mention ("Do not run `gortex track`") is guidance +// against it and is captured in the first group so it can be ignored. +var trackAdvicePattern = regexp.MustCompile("(?i)(\\bnot\\s+(?:run\\s+)?|n't\\s+(?:run\\s+)?|\\bnever\\s+(?:run\\s+)?)?`?gortex\\s+(?:track|repos\\s+reconcile)\\b") + +func adviceIn(text string) bool { + for _, match := range trackAdvicePattern.FindAllStringSubmatch(text, -1) { + if match[1] == "" { + return true + } + } + return false +} + +// cliCall runs `gortex call search_symbols` from cwd. Without a pty the CLI +// must fail fast with exit code 75 while the repository loads; with one it +// waits for the load, so its time to exit includes that wait. +func (s *startupProbe) cliCall(ctx context.Context, cwd probeCWD, tty bool) probeSample { + surface := "cli_pipe" + if tty { + surface = "cli_tty" + } + args, _ := json.Marshal(map[string]any{"query": s.query, "limit": 1}) + callCtx, cancel := context.WithTimeout(ctx, startupCLITimeout) + defer cancel() + argv := []string{"call", "search_symbols", "--json", string(args)} + var cmd *exec.Cmd + var err error + if tty { + cmd, err = ttyCommand(callCtx, s.binary, argv) + } else { + cmd = exec.CommandContext(callCtx, s.binary, argv...) + } + started := time.Now() + if err != nil { + return probeSample{CWD: cwd.Role, Surface: surface, Started: started, Code: "unavailable", Text: clip(err.Error())} + } + var output bytes.Buffer + cmd.Stdout, cmd.Stderr = &output, &output + exitCode, err := runOwned(cmd, cwd.Dir, s.env) + sample := probeSample{CWD: cwd.Role, Surface: surface, Started: started, LatencyMS: elapsedMS(started), ExitCode: &exitCode} + text := output.String() + sample.OK, sample.Code = classifyCLI(exitCode, text) + if callCtx.Err() == context.DeadlineExceeded { + sample.OK, sample.Code = false, "timeout" + } else if err != nil && exitCode < 0 { + sample.Code = "exec_error" + text += err.Error() + } + if !sample.OK { + sample.Text, sample.TrackAdvice = clip(text), adviceIn(text) + } + return sample +} + +func classifyCLI(exitCode int, text string) (bool, string) { + switch { + case exitCode == 0: + return true, "" + case exitCode == 75: + return false, "repo_loading" + case strings.Contains(text, "does not track"): + return false, "repo_not_tracked" + case strings.Contains(text, "no gortex daemon is running"): + return false, "no_daemon" + default: + return false, fmt.Sprintf("exit_%d", exitCode) + } +} + +// ttyCommand runs argv under script(1) so the CLI's stdout is a terminal. +func ttyCommand(ctx context.Context, binary string, argv []string) (*exec.Cmd, error) { + switch runtime.GOOS { + case "darwin", "freebsd": + return exec.CommandContext(ctx, "script", append([]string{"-q", "/dev/null", binary}, argv...)...), nil + case "linux": + quoted := make([]string, 0, len(argv)+1) + for _, arg := range append([]string{binary}, argv...) { + quoted = append(quoted, "'"+strings.ReplaceAll(arg, "'", `'\''`)+"'") + } + return exec.CommandContext(ctx, "script", "-q", "-e", "-c", strings.Join(quoted, " "), "/dev/null"), nil + default: + return nil, fmt.Errorf("no pty runner on %s", runtime.GOOS) + } +} + +// hookRead asks the PreToolUse hook about a native Read of an indexed file. +func (s *startupProbe) hookRead(ctx context.Context, cwd probeCWD) probeSample { + payload, _ := json.Marshal(map[string]any{"hook_event_name": "PreToolUse", "session_id": "gortex-load-startup", "tool_name": "Read", "tool_input": map[string]any{"file_path": cwd.File}, "cwd": cwd.Dir}) + callCtx, cancel := context.WithTimeout(ctx, startupCallTimeout) + defer cancel() + cmd := exec.CommandContext(callCtx, s.binary, "hook") + cmd.Stdin = bytes.NewReader(payload) + var stdout, stderr bytes.Buffer + cmd.Stdout, cmd.Stderr = &stdout, &stderr + started := time.Now() + exitCode, err := runOwned(cmd, cwd.Dir, s.env) + sample := probeSample{CWD: cwd.Role, Surface: "hook_read", Started: started, LatencyMS: elapsedMS(started), ExitCode: &exitCode} + decision, text := decodeHookOutput(stdout.Bytes()) + sample.Variant, sample.TrackAdvice, sample.Text = decision, adviceIn(text), clip(text) + sample.OK = err == nil + if err != nil { + sample.Variant, sample.Code = "", "hook_error" + if callCtx.Err() == context.DeadlineExceeded { + sample.Code = "timeout" + } + sample.Text = clip(err.Error() + " " + stderr.String()) + } + return sample +} + +// decodeHookOutput reads the default hook protocol's PreToolUse reply. No +// output, or context without a decision, lets the native tool run. +func decodeHookOutput(output []byte) (decision, text string) { + output = bytes.TrimSpace(output) + if len(output) == 0 { + return "allow", "" + } + var reply struct { + Decision string `json:"decision"` + Reason string `json:"reason"` + Specific *struct { + Decision string `json:"permissionDecision"` + Reason string `json:"permissionDecisionReason"` + Context string `json:"additionalContext"` + } `json:"hookSpecificOutput"` + } + if json.Unmarshal(output, &reply) != nil { + return "unparsed", string(output) + } + decision, text = "allow", reply.Reason + if reply.Specific != nil { + if reply.Specific.Decision != "" { + decision = reply.Specific.Decision + } + text = strings.TrimSpace(text + " " + reply.Specific.Reason + " " + reply.Specific.Context) + } + if reply.Decision == "block" { + decision = "deny" + } + return decision, text +} + +// runOwned runs a probe command in its own process group, so a call cut +// short by its deadline cannot leave children behind. +func runOwned(cmd *exec.Cmd, dir string, env []string) (int, error) { + configureOwnedProcess(cmd) + cmd.Dir, cmd.Env, cmd.WaitDelay = dir, env, 5*time.Second + cmd.Cancel = func() error { + if cmd.Process == nil { + return os.ErrProcessDone + } + return killOwnedProcess(cmd) + } + err := cmd.Run() + if err != nil && cmd.Process != nil { + _ = cleanupExitedOwnedProcess(cmd) + } + if cmd.ProcessState == nil { + return -1, err + } + return cmd.ProcessState.ExitCode(), err +} + +func elapsedMS(start time.Time) float64 { return durationMS(time.Since(start)) } + +func clip(text string) string { + text = strings.TrimSpace(text) + if len(text) <= startupTextLimit { + return text + } + return text[:startupTextLimit] + "..." +} diff --git a/bench/load/startup_probe_summary.go b/bench/load/startup_probe_summary.go new file mode 100644 index 000000000..495af080d --- /dev/null +++ b/bench/load/startup_probe_summary.go @@ -0,0 +1,247 @@ +package main + +import ( + "fmt" + "strings" + "time" +) + +// graphSurfaces answer from the repository graph. Their first success after +// the primary registers is how long an agent waits for a working tool. +var graphSurfaces = []string{"search_symbols", "index_health_resource", "cli_pipe", "cli_tty"} + +var probeRoles = []string{"primary", "worktree", "router"} + +type firstOKTiming struct { + FromStartMS Metric `json:"from_start_ms"` + FromReadyMS Metric `json:"from_ready_ms"` +} + +type bootSummary struct { + bootTiming + TimeToReadyMS Metric `json:"time_to_ready_ms"` + FirstOK map[string]firstOKTiming `json:"first_ok"` +} + +// startupSummary is the startup probe's gate receipt. Answers are keyed +// "surface/cwd" and split into windows by what the readiness watcher saw: +// no_socket before the socket answered, loading until the primary was ready, +// and ready after. +type startupSummary struct { + PrimaryRepo string `json:"primary_repo"` + LinkedWorktrees int `json:"linked_worktrees"` + WatchIntervalMS float64 `json:"watch_interval_ms"` + CLITTY string `json:"cli_tty"` + Boots []bootSummary `json:"boots"` + // Outcomes counts each answer as ok, ok:, an error code (with + // repo_loading(incomplete) for a loading refusal missing retryable=true, + // phase or percent), an initialize instructions variant, or a hook + // decision. + Outcomes map[string]map[string]map[string]int `json:"outcomes"` + TrackAdvice map[string]map[string]int `json:"track_advice"` + RestartReadyToFirstOK map[string]Distribution `json:"restart_ready_to_first_ok_ms"` + BetweenBoots int `json:"samples_between_boots"` +} + +// firstOK is when the earliest successful sample for surface and cwd that +// started in [from, until) completed; a zero until is open. Measured from a +// boot's start it is time to first OK; measured from readiness it is ready +// to first OK, for any warm or cold time-to-ready comparison. +func firstOK(samples []probeSample, surface, cwd string, from, until time.Time) (time.Time, bool) { + var first time.Time + for _, s := range samples { + if !s.OK || s.Surface != surface || s.CWD != cwd || s.Started.Before(from) || (!until.IsZero() && !s.Started.Before(until)) { + continue + } + if done := s.completed(); first.IsZero() || done.Before(first) { + first = done + } + } + return first, !first.IsZero() +} + +// bootIndex places a sample in the boot it observed. The probe starts just +// before the first daemon does, so earlier samples belong to the first boot; +// samples between a stop request and the next start belong to none. +func bootIndex(boots []bootTiming, at time.Time) int { + for i, boot := range boots { + if (i == 0 || !at.Before(boot.Started)) && (boot.Stopping.IsZero() || at.Before(boot.Stopping)) { + return i + } + } + return -1 +} + +func startupWindow(boot bootTiming, primary string, at time.Time) string { + if boot.SocketAt.IsZero() || at.Before(boot.SocketAt) { + return "no_socket" + } + if ready, ok := boot.ReadyAt[primary]; !ok || at.Before(ready) { + return "loading" + } + return "ready" +} + +func outcomeKey(s probeSample) string { + switch { + case s.Variant != "": + return s.Variant + case s.OK && s.Status != "": + return "ok:" + s.Status + case s.OK: + return "ok" + case s.Code == "repo_loading" && !strings.HasPrefix(s.Surface, "cli_") && (s.Retryable == nil || !*s.Retryable || s.Phase == "" || s.Percent == nil): + return "repo_loading(incomplete)" + default: + return s.Code + } +} + +func summarizeStartup(samples []probeSample, boots []bootTiming, primary string, linked int) startupSummary { + sum := startupSummary{PrimaryRepo: primary, LinkedWorktrees: linked, WatchIntervalMS: durationMS(startupWatchInterval), Outcomes: make(map[string]map[string]map[string]int), TrackAdvice: make(map[string]map[string]int), RestartReadyToFirstOK: make(map[string]Distribution)} + for _, s := range samples { + index := bootIndex(boots, s.Started) + if index < 0 { + sum.BetweenBoots++ + continue + } + key, window := s.Surface+"/"+s.CWD, startupWindow(boots[index], primary, s.Started) + if sum.Outcomes[key] == nil { + sum.Outcomes[key] = make(map[string]map[string]int) + } + if sum.Outcomes[key][window] == nil { + sum.Outcomes[key][window] = make(map[string]int) + } + sum.Outcomes[key][window][outcomeKey(s)]++ + if s.TrackAdvice { + if sum.TrackAdvice[key] == nil { + sum.TrackAdvice[key] = make(map[string]int) + } + sum.TrackAdvice[key][window]++ + } + } + restarts := make(map[string][]float64) + for i, boot := range boots { + row := bootSummary{bootTiming: boot, TimeToReadyMS: unavailable("ms", "primary repository never ready"), FirstOK: make(map[string]firstOKTiming)} + ready, registered := boot.ReadyAt[primary] + if registered { + row.TimeToReadyMS = measured(durationMS(ready.Sub(boot.Started)), "ms") + } + for _, surface := range graphSurfaces { + for _, cwd := range probeRoles { + first, ok := firstOK(samples, surface, cwd, boot.Started, boot.Stopping) + if !ok { + continue + } + key := surface + "/" + cwd + timing := firstOKTiming{FromStartMS: measured(durationMS(first.Sub(boot.Started)), "ms"), FromReadyMS: unavailable("ms", "primary repository never ready")} + if registered { + // The watcher stamps readiness up to one poll late, so a + // success can precede it; that wait is zero. + fromReady := max(0, durationMS(first.Sub(ready))) + timing.FromReadyMS = measured(fromReady, "ms") + if i > 0 { + restarts[key] = append(restarts[key], fromReady) + } + } + row.FirstOK[key] = timing + } + } + sum.Boots = append(sum.Boots, row) + } + for key, values := range restarts { + sum.RestartReadyToFirstOK[key] = distribution(values, "ms") + } + return sum +} + +// metrics are the headline counters. Counts cover every boot; restart +// figures cover boots after the first and are unavailable unless every +// restart produced the observation. +func (sum startupSummary) metrics() map[string]Metric { + count := func(match func(surface, window, outcome string) bool) Metric { + total := 0 + for key, windows := range sum.Outcomes { + surface, _, _ := strings.Cut(key, "/") + for window, outcomes := range windows { + for outcome, n := range outcomes { + if match(surface, window, outcome) { + total += n + } + } + } + } + return measured(float64(total), "count") + } + advice := func(window string) Metric { + total := 0 + for _, windows := range sum.TrackAdvice { + total += windows[window] + } + return measured(float64(total), "count") + } + m := map[string]Metric{ + "startup_boots": measured(float64(len(sum.Boots)), "count"), + "startup_linked_worktrees": measured(float64(sum.LinkedWorktrees), "count"), + // The untracked initialize variant is the same answer at handshake time. + "startup_repo_not_tracked_before_ready": count(func(_, window, outcome string) bool { + return window != "ready" && (outcome == "repo_not_tracked" || outcome == "untracked") + }), + "startup_nonconforming_loading_refusals": count(func(surface, window, outcome string) bool { + return window == "loading" && surface != "hook_read" && !strings.HasPrefix(surface, "initialize") && outcome != "repo_loading" && !strings.HasPrefix(outcome, "ok") + }), + "startup_hook_denies_while_loading": count(func(surface, window, outcome string) bool { + return surface == "hook_read" && window != "ready" && outcome == "deny" + }), + "startup_hook_denies_after_ready": count(func(surface, window, outcome string) bool { + return surface == "hook_read" && window == "ready" && outcome == "deny" + }), + "startup_track_advice_while_loading": advice("loading"), + "startup_track_advice_without_socket": advice("no_socket"), + "startup_initial_time_to_ready_ms": unavailable("ms", "no boot"), + "startup_initial_first_ok_ms": unavailable("ms", "no successful search_symbols from the primary"), + } + if len(sum.Boots) > 0 { + m["startup_initial_time_to_ready_ms"] = sum.Boots[0].TimeToReadyMS + if first, ok := sum.Boots[0].FirstOK["search_symbols/primary"]; ok { + m["startup_initial_first_ok_ms"] = first.FromStartMS + } + } + restarts := max(0, len(sum.Boots)-1) + var toReady []float64 + for _, boot := range sum.Boots[min(1, len(sum.Boots)):] { + if boot.TimeToReadyMS.Value != nil { + toReady = append(toReady, *boot.TimeToReadyMS.Value) + } + } + m["startup_restart_time_to_ready_p95_ms"] = restartP95(distribution(toReady, "ms"), restarts, "reached ready") + m["startup_restart_ready_to_first_ok_p95_ms"] = restartP95(sum.RestartReadyToFirstOK["search_symbols/primary"], restarts, "had a successful search_symbols from the primary") + // The census is per boot; with no restart the initial boot is all there is. + census := sum.Boots[min(1, len(sum.Boots)):] + if len(sum.Boots) == 1 { + census = sum.Boots + } + m["startup_coordinators_within_2s_min"] = unavailable("count", "no boot") + for i, boot := range census { + if boot.Coordinators == nil { + m["startup_coordinators_within_2s_min"] = unavailable("count", fmt.Sprintf("boot %d has no coordinator census", boot.Boot)) + break + } + if value := float64(*boot.Coordinators); i == 0 || value < *m["startup_coordinators_within_2s_min"].Value { + m["startup_coordinators_within_2s_min"] = measured(value, "count") + } + } + return m +} + +func restartP95(d Distribution, restarts int, observation string) Metric { + if restarts == 0 { + return unavailable("ms", "no restarts") + } + if d.Count < restarts { + return unavailable("ms", fmt.Sprintf("%d of %d restarts %s", d.Count, restarts, observation)) + } + return d.P95 +} + +func durationMS(d time.Duration) float64 { return float64(d) / float64(time.Millisecond) } diff --git a/bench/load/startup_probe_test.go b/bench/load/startup_probe_test.go new file mode 100644 index 000000000..dd55103ce --- /dev/null +++ b/bench/load/startup_probe_test.go @@ -0,0 +1,329 @@ +package main + +import ( + "bufio" + "context" + "encoding/json" + "net" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestDecodeProbeFrameReadsAdmissionAnswers(t *testing.T) { + active := "When Gortex can serve the step: MUST use Gortex MCP.\n\nWorktree and branch routing:\n\n- A linked worktree is an automatic overlay; do not `track` it. Graph calls return `view_building` until discovery completes; retry them, no reconnect is needed. Do not run `gortex track`." + for _, test := range []struct { + name, frame, outcome string + ok, advice bool + check func(t *testing.T, s probeSample) + }{ + {name: "loading refusal", outcome: "repo_loading", + frame: `{"jsonrpc":"2.0","id":2,"error":{"code":-32000,"message":"repository loading: app (resolving, 40%)","data":{"error_code":"repo_loading","retryable":true,"phase":"resolving","percent":40,"suggestion":"Gortex is still loading app after a restart; tools start working in place, no reconnect needed. Retry shortly. Do not change tracking configuration."}}}`, + check: func(t *testing.T, s probeSample) { + if s.Retryable == nil || !*s.Retryable || s.Phase != "resolving" || s.Percent == nil || *s.Percent != 40 { + t.Fatalf("loading fields lost: %+v", s) + } + }}, + {name: "loading refusal without progress", outcome: "repo_loading(incomplete)", + frame: `{"jsonrpc":"2.0","id":2,"error":{"code":-32000,"message":"repository loading","data":{"error_code":"repo_loading","retryable":true}}}`}, + {name: "untracked refusal advises tracking", outcome: "repo_not_tracked", advice: true, + frame: "{\"jsonrpc\":\"2.0\",\"id\":2,\"error\":{\"code\":-32000,\"message\":\"repository not tracked: /w\",\"data\":{\"error_code\":\"repo_not_tracked\",\"path\":\"/w\",\"suggestion\":\"Run `gortex track /w` to include this repo in the shared graph.\"}}}"}, + {name: "unavailable is final", outcome: "repo_unavailable", advice: true, + frame: "{\"jsonrpc\":\"2.0\",\"id\":2,\"error\":{\"code\":-32000,\"message\":\"repository unavailable\",\"data\":{\"error_code\":\"repo_unavailable\",\"retryable\":false,\"suggestion\":\"Gortex could not load /w after the restart. Run `gortex track /w` to index it again.\"}}}", + check: func(t *testing.T, s probeSample) { + if s.Retryable == nil || *s.Retryable { + t.Fatalf("retryable=false must be kept: %+v", s) + } + }}, + {name: "busy without data", outcome: "rpc_-32002", + frame: `{"jsonrpc":"2.0","id":2,"error":{"code":-32002,"message":"busy"}}`}, + {name: "loading index health answers", outcome: "ok:loading", ok: true, + frame: `{"jsonrpc":"2.0","id":3,"result":{"content":[{"type":"text","text":"{\"status\":\"loading\",\"phase\":\"starting\",\"percent\":0,\"repos\":[]}"}]}}`, + check: func(t *testing.T, s probeSample) { + if s.Phase != "starting" || s.Percent == nil || *s.Percent != 0 { + t.Fatalf("a zero percent is an observation, not a missing one: %+v", s) + } + }}, + {name: "resource read", outcome: "ok:ok", ok: true, + frame: `{"jsonrpc":"2.0","id":4,"result":{"contents":[{"uri":"gortex://index-health","text":"{\"status\":\"ok\"}"}]}}`}, + {name: "search results may quote the commands", outcome: "ok", ok: true, + frame: "{\"jsonrpc\":\"2.0\",\"id\":2,\"result\":{\"content\":[{\"type\":\"text\",\"text\":\"{\\\"results\\\":[{\\\"doc\\\":\\\"run `gortex track .` first\\\"}]}\"}]}}"}, + {name: "tool error payload", outcome: "view_building", + frame: `{"jsonrpc":"2.0","id":2,"result":{"isError":true,"content":[{"type":"text","text":"{\"error_code\":\"view_building\",\"message\":\"retry\"}"}]}}`}, + {name: "plain tool error", outcome: "tool_error", + frame: `{"jsonrpc":"2.0","id":2,"result":{"isError":true,"content":[{"type":"text","text":"tool not found"}]}}`}, + {name: "loading instructions", outcome: "loading", ok: true, + frame: mustInitialize(t, "Gortex is LOADING app after a restart (starting, 0%). Graph tools return repo_loading (retryable) until it is ready; Do not re-track it or change tracking configuration: the repository is already configured.\n\n"+active)}, + {name: "admission pending instructions", outcome: "admission_pending", ok: true, + frame: mustInitialize(t, "Gortex is LOADING graph admission after a restart (identity_resolution, 0%). Graph admission for this path is not yet established.")}, + {name: "unavailable instructions", outcome: "unavailable", ok: true, advice: true, + frame: mustInitialize(t, "Gortex is connected but could not load /w after the restart: boom.\n\nGraph tools return repo_unavailable for it. To repair, run `gortex track /w` to index it again.")}, + {name: "untracked instructions", outcome: "untracked", ok: true, advice: true, + frame: mustInitialize(t, "Gortex is connected but INACTIVE for this directory: \"/w\" is not covered by any tracked repository.\n\nTo activate: run `gortex track /w`, then retry graph calls; no reconnect is needed.")}, + {name: "active instructions forbid tracking", outcome: "active", ok: true, + frame: mustInitialize(t, active)}, + {name: "not JSON", outcome: "invalid_response", frame: `daemon: serving`}, + } { + t.Run(test.name, func(t *testing.T) { + s := decodeProbeFrame([]byte(test.frame)) + if got := outcomeKey(s); got != test.outcome || s.OK != test.ok || s.TrackAdvice != test.advice { + t.Fatalf("outcome %q ok=%v advice=%v, want %q ok=%v advice=%v: %+v", got, s.OK, s.TrackAdvice, test.outcome, test.ok, test.advice, s) + } + if test.check != nil { + test.check(t, s) + } + }) + } +} + +func mustInitialize(t *testing.T, instructions string) string { + t.Helper() + data, err := json.Marshal(map[string]any{"jsonrpc": "2.0", "id": 1, "result": map[string]any{"protocolVersion": "2024-11-05", "instructions": instructions}}) + if err != nil { + t.Fatal(err) + } + return string(data) +} + +func TestTrackAdviceIgnoresGuidanceAgainstIt(t *testing.T) { + for text, want := range map[string]bool{ + "Run `gortex track /w` to include this repo": true, + "the gortex daemon does not track /w — add it with `gortex track /w`": true, + "run `gortex repos reconcile /w` and retry": true, + "start it, then track this repo with gortex track /w": true, + "Do not run `gortex track`.": false, + "Don't run gortex track here": false, + "never gortex repos reconcile a loading family": false, + "remove it with `gortex untrack /w`": false, + "gortex tracks hundreds of repositories": false, + "the repository is already tracked; retry shortly": false, + "Do not run `gortex track`. To activate: run `gortex track /w`.": true, + } { + if got := adviceIn(text); got != want { + t.Errorf("adviceIn(%q) = %v, want %v", text, got, want) + } + } +} + +func TestCLIAndHookAnswers(t *testing.T) { + for _, test := range []struct { + exit int + text string + ok bool + code string + }{ + {0, `{"results":[]}`, true, ""}, + {75, "the gortex daemon is still loading app after a restart (starting, 0%)", false, "repo_loading"}, + {1, "the gortex daemon does not track /w — add it with `gortex track /w`", false, "repo_not_tracked"}, + {1, "no gortex daemon is running — start it with `gortex daemon start --detach`", false, "no_daemon"}, + {2, "unknown flag", false, "exit_2"}, + } { + if ok, code := classifyCLI(test.exit, test.text); ok != test.ok || code != test.code { + t.Errorf("classifyCLI(%d, %q) = %v %q, want %v %q", test.exit, test.text, ok, code, test.ok, test.code) + } + } + for output, want := range map[string]string{ + "": "allow", + `{"hookSpecificOutput":{"hookEventName":"PreToolUse","permissionDecision":"deny","permissionDecisionReason":"use read(operation:\"file\")"}}`: "deny", + `{"hookSpecificOutput":{"hookEventName":"PreToolUse","additionalContext":"app is still loading (repo_loading); the native Read is allowed"}}`: "allow", + `{"decision":"block","reason":"indexed"}`: "deny", + `not json`: "unparsed", + } { + if got, _ := decodeHookOutput([]byte(output)); got != want { + t.Errorf("decodeHookOutput(%q) = %q, want %q", output, got, want) + } + } +} + +// roundTrip must skip pushes the daemon interleaves before the response. +func TestProbeRoundTripSkipsNotifications(t *testing.T) { + client, server := net.Pipe() + defer client.Close() + go func() { + defer server.Close() + request, err := bufio.NewReader(server).ReadString('\n') + if err != nil || !strings.Contains(request, `"method":"tools/call"`) { + return + } + _, _ = server.Write([]byte(`{"jsonrpc":"2.0","method":"notifications/message","params":{"data":"loading"}}` + "\n" + + `{"jsonrpc":"2.0","id":7,"method":"roots/list"}` + "\n" + + `{"jsonrpc":"2.0","id":1,"result":{"content":[{"type":"text","text":"{}"}]}}` + "\n")) + }() + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + c := &probeConn{conn: client, reader: bufio.NewReader(client)} + frame, err := c.roundTrip(ctx, "tools/call", map[string]any{"name": "search_symbols"}) + if err != nil || !strings.Contains(string(frame), `"id":1`) { + t.Fatalf("roundTrip = %s, %v", frame, err) + } +} + +func TestFirstOKUsesEarliestCompletionInsideTheBoot(t *testing.T) { + t0 := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + samples := []probeSample{ + {CWD: "primary", Surface: "search_symbols", Started: t0, OK: true}, // previous boot + {CWD: "primary", Surface: "search_symbols", Started: t0.Add(11 * time.Second), LatencyMS: 3000, OK: true}, // slow + {CWD: "primary", Surface: "search_symbols", Started: t0.Add(12 * time.Second), LatencyMS: 100, OK: true}, // first to finish + {CWD: "worktree", Surface: "search_symbols", Started: t0.Add(11 * time.Second), LatencyMS: 10, OK: true}, // other cwd + {CWD: "primary", Surface: "search_symbols", Started: t0.Add(10500 * time.Millisecond), LatencyMS: 5, Code: "repo_loading"}, + {CWD: "primary", Surface: "search_symbols", Started: t0.Add(30 * time.Second), LatencyMS: 1, OK: true}, // next boot + } + first, ok := firstOK(samples, "search_symbols", "primary", t0.Add(10*time.Second), t0.Add(20*time.Second)) + if want := t0.Add(12100 * time.Millisecond); !ok || !first.Equal(want) { + t.Fatalf("firstOK = %v %v, want %v", first, ok, want) + } + if _, ok := firstOK(samples, "cli_pipe", "primary", t0, time.Time{}); ok { + t.Fatal("no sample of the surface is not a first OK") + } +} + +func TestSummarizeStartupCountsPerWindowAndRestart(t *testing.T) { + t0 := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC) + at := func(ms int) time.Time { return t0.Add(time.Duration(ms) * time.Millisecond) } + coordinators := func(n int) *int { return &n } + boots := []bootTiming{ + {Boot: 0, Started: at(0), SocketAt: at(2000), ReadyAt: map[string]time.Time{"app": at(5000)}, Stopping: at(20000), Coordinators: coordinators(3)}, + {Boot: 1, Started: at(21000), SocketAt: at(21500), ReadyAt: map[string]time.Time{"app": at(22000)}, Stopping: at(30000), Coordinators: coordinators(4)}, + {Boot: 2, Started: at(31000), SocketAt: at(31200), ReadyAt: map[string]time.Time{"app": at(31600)}, Coordinators: coordinators(2)}, + } + yes, percent := true, 10 + samples := []probeSample{ + {CWD: "primary", Surface: "connect", Started: at(-500), Code: "no_socket"}, // before the first start + {CWD: "primary", Surface: "cli_pipe", Started: at(1000), Code: "no_daemon", TrackAdvice: true}, + {CWD: "primary", Surface: "initialize", Started: at(3000), OK: true, Variant: "loading"}, + {CWD: "router", Surface: "initialize", Started: at(3000), OK: true, Variant: "untracked", TrackAdvice: true}, + {CWD: "primary", Surface: "search_symbols", Started: at(3000), Code: "repo_loading", Retryable: &yes, Phase: "starting", Percent: &percent}, + {CWD: "primary", Surface: "search_symbols", Started: at(4000), Code: "repo_loading", Retryable: &yes}, + {CWD: "worktree", Surface: "search_symbols", Started: at(3000), Code: "repo_not_tracked", TrackAdvice: true}, + {CWD: "primary", Surface: "workspace_index", Started: at(3000), OK: true, Status: "loading"}, + {CWD: "primary", Surface: "hook_read", Started: at(3000), OK: true, Variant: "allow"}, + {CWD: "primary", Surface: "hook_read", Started: at(6000), OK: true, Variant: "deny"}, + {CWD: "primary", Surface: "search_symbols", Started: at(5500), LatencyMS: 100, OK: true}, + {CWD: "primary", Surface: "search_symbols", Started: at(20500), Code: "no_socket"}, // between boots + {CWD: "primary", Surface: "search_symbols", Started: at(23000), LatencyMS: 50, OK: true}, + {CWD: "primary", Surface: "search_symbols", Started: at(31500), LatencyMS: 50, OK: true}, // before the stamp + } + sum := summarizeStartup(samples, boots, "app", 24) + if got := sum.Outcomes["search_symbols/primary"]["loading"]; got["repo_loading"] != 1 || got["repo_loading(incomplete)"] != 1 || got["ok"] != 1 { + t.Fatalf("loading outcomes = %v", got) + } + if got := sum.Outcomes["search_symbols/primary"]["ready"]["ok"]; got != 2 { + t.Fatalf("ready successes = %d", got) + } + if sum.Outcomes["connect/primary"]["no_socket"]["no_socket"] != 1 || sum.BetweenBoots != 1 || sum.TrackAdvice["cli_pipe/primary"]["no_socket"] != 1 || sum.TrackAdvice["search_symbols/worktree"]["loading"] != 1 { + t.Fatalf("between=%d advice=%v", sum.BetweenBoots, sum.TrackAdvice) + } + if first := sum.Boots[0].FirstOK["search_symbols/primary"]; *first.FromStartMS.Value != 5600 || *first.FromReadyMS.Value != 600 { + t.Fatalf("initial first OK = %+v", first) + } + if d := sum.RestartReadyToFirstOK["search_symbols/primary"]; d.Count != 2 || *d.P95.Value != 1050 || *d.P50.Value != 0 { + t.Fatalf("restart ready to first OK = %+v", d) + } + metrics := sum.metrics() + for name, want := range map[string]float64{ + "startup_boots": 3, + "startup_linked_worktrees": 24, + "startup_repo_not_tracked_before_ready": 2, + "startup_nonconforming_loading_refusals": 2, + "startup_hook_denies_while_loading": 0, + "startup_hook_denies_after_ready": 1, + "startup_track_advice_while_loading": 2, + "startup_track_advice_without_socket": 1, + "startup_initial_time_to_ready_ms": 5000, + "startup_initial_first_ok_ms": 5600, + "startup_restart_time_to_ready_p95_ms": 1000, + "startup_restart_ready_to_first_ok_p95_ms": 1050, + "startup_coordinators_within_2s_min": 2, + } { + if m := metrics[name]; m.Value == nil || *m.Value != want { + t.Errorf("%s = %+v, want %v", name, m, want) + } + } + + // A restart that never answered must not vanish from the p95. + sum = summarizeStartup(samples[:len(samples)-1], boots, "app", 24) + if m := sum.metrics()["startup_restart_ready_to_first_ok_p95_ms"]; m.Value != nil || !strings.Contains(m.Reason, "1 of 2 restarts") { + t.Fatalf("missing restart observation = %+v", m) + } + boots[2].Coordinators = nil + if m := summarizeStartup(samples, boots, "app", 24).metrics()["startup_coordinators_within_2s_min"]; m.Value != nil { + t.Fatalf("a boot without a census cannot report a minimum: %+v", m) + } +} + +func TestStartupOptionsStayScoped(t *testing.T) { + base := []string{"--binary", "/b", "--manifest", "/m", "--scratch", "/s"} + o, err := parseRun(append(base, "--scenario", "startup", "--restarts", "10", "--seed-store", "/v21.db")) + if err != nil || o.Startup.Restarts != 10 || o.Startup.SeedStore != "/v21.db" { + t.Fatalf("startup options = %+v, %v", o.Startup, err) + } + if scenarios, _ := selectedScenarios(o); len(scenarios) != 1 || scenarios[0] != "startup" { + t.Fatalf("startup scenarios = %v", scenarios) + } + o, err = parseRun(append(base, "--scenario", "all")) + if scenarios, _ := selectedScenarios(o); err != nil || strings.Contains(strings.Join(scenarios, ","), "startup") { + t.Fatalf("all must not restart daemons on its own: %v %v", scenarios, err) + } + for _, args := range [][]string{{"--scenario", "read", "--restarts", "1"}, {"--scenario", "cold", "--seed-store", "/v21.db"}, {"--scenario", "startup", "--restarts", "-1"}} { + if _, err := parseRun(append(base, args...)); err == nil { + t.Errorf("parseRun(%v) accepted startup options outside their scope", args) + } + } +} + +// The whole scenario against the fixture daemon: probing starts before the +// first socket, keeps sampling through a restart, and the gate, metrics and +// per-second log come from that one run. +func TestStartupScenarioProbesAcrossARestart(t *testing.T) { + t.Setenv("LOAD_DAEMON_FIXTURE", "serve") + corpus := gitFixture(t) + binary, err := os.Executable() + if err != nil { + t.Fatal(err) + } + options := runOptions{Binary: binary, Profile: "agent", Host: "native", Agents: 1, Worktrees: 1, Duration: time.Second, Deadline: 2 * time.Second, WarmTimeout: 20 * time.Second, StopTimeout: 5 * time.Second, KeepState: true, Startup: startupOptions{Restarts: 1}} + ctx, cancel := context.WithTimeout(context.Background(), time.Minute) + defer cancel() + phase, _, artifacts, crashes, version, err := runPhase(ctx, options, startupScenario, []Corpus{corpus}, "", t.TempDir()) + if err != nil { + t.Fatalf("startup scenario: %v %+v", err, phase.Errors) + } + var summary startupSummary + if err := json.Unmarshal(phase.Gate["startup_probe"], &summary); err != nil { + t.Fatal(err) + } + if version != "fixture-v1" || len(summary.Boots) != 2 || len(phase.Status) != 1 || len(phase.Stats) != 1 || crashes["panic_lines"] != 0 { + t.Fatalf("version=%q boots=%d status=%d stats=%d crashes=%v", version, len(summary.Boots), len(phase.Status), len(phase.Stats), crashes) + } + for _, key := range []string{"initialize/primary", "initialize_facade/worktree", "search_symbols/primary", "search_symbols/worktree", "search_symbols/router", "index_health_resource/router", "workspace_index/primary", "hook_read/worktree", "cli_pipe/primary"} { + answered := 0 + for _, outcomes := range summary.Outcomes[key] { + answered += outcomes["ok"] + outcomes["allow"] + } + if answered == 0 { + t.Errorf("%s has no successful answer: %v", key, summary.Outcomes[key]) + } + } + for _, name := range []string{"startup_initial_first_ok_ms", "startup_restart_ready_to_first_ok_p95_ms", "startup_restart_time_to_ready_p95_ms"} { + if phase.Metrics[name].Value == nil { + t.Errorf("%s unavailable: %s", name, phase.Metrics[name].Reason) + } + } + if m := phase.Metrics["startup_coordinators_within_2s_min"]; m.Value == nil || *m.Value != 1 || *phase.Metrics["startup_linked_worktrees"].Value != 1 { + t.Errorf("coordinator census = %+v", m) + } + log, err := os.ReadFile(artifacts["startup_probe_log"]) + if err != nil || strings.Count(string(log), "\n") < 10 { + t.Fatalf("per-second log: %d bytes, %v", len(log), err) + } + roster, err := os.ReadFile(filepath.Join(artifacts["private_root"], "config", "gortex", "servers.toml")) + if err != nil || !strings.Contains(string(roster), startupRosterWorkspace) { + t.Fatalf("router roster: %s %v", roster, err) + } + claim, err := os.ReadFile(filepath.Join(artifacts["private_root"], "repos", corpus.Label, ".gortex-probe-router", ".gortex.yaml")) + if err != nil || string(claim) != "workspace: "+startupRosterWorkspace+"\n" { + t.Fatalf("router cwd claim: %q %v", claim, err) + } +} diff --git a/bench/load/validate.go b/bench/load/validate.go index b875be35c..85fd3863a 100644 --- a/bench/load/validate.go +++ b/bench/load/validate.go @@ -130,7 +130,7 @@ func validateReport(data []byte, r Report) error { if len(phase.Operations) == 0 || len(phase.Gate) == 0 || len(phase.Status) == 0 || len(phase.Stats) == 0 { return errors.New("complete phase is missing plan/gate/status/stats observations") } - if phase.Name != "cold" && phase.Name != "restart" && len(phase.Tools) == 0 { + if phase.Name != "cold" && phase.Name != "restart" && phase.Name != startupScenario && len(phase.Tools) == 0 { return errors.New("complete workload phase has no tool observations") } } diff --git a/bench/load/worktree_steps.go b/bench/load/worktree_steps.go new file mode 100644 index 000000000..a34a4db96 --- /dev/null +++ b/bench/load/worktree_steps.go @@ -0,0 +1,228 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "flag" + "fmt" + "path/filepath" + "strconv" + "strings" + "time" +) + +// Worktree step mode measures what each linked worktree costs a daemon. +// +// One daemon starts with the primary and the first step's linked worktrees, +// then grows through each later count on the same process. At every count it +// serves the reading agents for --duration from fresh sessions spread over +// every checkout, then sits idle for --duration. Each idle window follows a +// load window of its own count and opens with one garbage collection, so +// every idle level holds what the checkouts retain after use rather than the +// load's garbage or a first load's one-off growth. The gauge series runs +// throughout; each load window keeps a heap profile from its start and from +// its highest sampled footprint, and each idle window one taken right after +// its collection. The change in load peak and in idle level between counts, +// divided by the worktrees added, is the cost of one checkout coordinator. + +// stepIdleOverride, when set, replaces --duration as the length of each +// step's idle window. Tests set it: an idle window needs no workload calls, +// so it need not last as long as a load window. +var stepIdleOverride time.Duration + +type worktreeStep struct { + LinkedWorktrees int `json:"linked_worktrees"` + LoadStarted time.Time `json:"load_started"` + LoadEnded time.Time `json:"load_ended"` + IdleStarted time.Time `json:"idle_started"` + IdleEnded time.Time `json:"idle_ended"` + Load map[string]Metric `json:"load_metrics"` + Tools map[string]ToolMetrics `json:"tools"` + HeapLoadStart string `json:"heap_profile_load_start"` + HeapLoadPeak string `json:"heap_profile_load_peak"` + HeapLoadPeakUnavailable string `json:"heap_profile_load_peak_unavailable,omitempty"` + HeapIdle string `json:"heap_profile_idle"` +} + +// bindMemoryFlags registers the standalone memory-diagnosis options. Paired +// runs do not bind them: a coupled comparison measures latency between builds. +func bindMemoryFlags(f *flag.FlagSet, o *runOptions) { + f.BoolVar(&o.Gauges, "gauges", false, "record a 10s series of daemon_health gauges with external process memory (always on with --worktree-steps)") + f.BoolVar(&o.HeapProfiles, "heap-profiles", false, "with --gauges, also save daemon heap profiles at load start and at the highest sampled footprint; peak captures run inside the measured load window (always on with --worktree-steps)") + f.Func("worktree-steps", "worktrees scenario: ascending linked-worktree counts ending at --worktrees, e.g. 0,1,2,4; one daemon is loaded then idle for --duration at each", func(value string) error { + o.WorktreeSteps = nil + for field := range strings.SplitSeq(value, ",") { + count, err := strconv.Atoi(strings.TrimSpace(field)) + if err != nil || count < 0 || len(o.WorktreeSteps) > 0 && count <= o.WorktreeSteps[len(o.WorktreeSteps)-1] { + return fmt.Errorf("worktree steps must be ascending nonnegative counts: %q", value) + } + o.WorktreeSteps = append(o.WorktreeSteps, count) + } + return nil + }) +} + +func validateMemoryOptions(o runOptions) error { + if o.Scenario == startupScenario && wantsGaugeSeries(o) { + return errors.New("--gauges and --heap-profiles are unsupported for startup-probe") + } + if o.Scenario == "idle-stop" && wantsGaugeSeries(o) { + return errors.New("live memory gauges and profiles are unsupported for idle-stop: the extra health session prevents an idle interval") + } + if len(o.WorktreeSteps) == 0 { + if o.HeapProfiles && !o.Gauges { + return errors.New("--heap-profiles needs --gauges: the peak profile is chosen from the gauge series") + } + return nil + } + if o.Scenario != "worktrees" || o.WorktreeSteps[len(o.WorktreeSteps)-1] != o.Worktrees { + return errors.New("--worktree-steps applies to the worktrees scenario and must end at --worktrees") + } + if o.CPUProfile != 0 { + return errors.New("worktree step mode takes heap profiles per step, not --cpu-profile") + } + return nil +} + +// bootWorktrees is how many linked worktrees exist before the daemon starts. +func bootWorktrees(o runOptions) int { + if len(o.WorktreeSteps) > 0 { + return o.WorktreeSteps[0] + } + return o.Worktrees +} + +func stepArtifact(step int, name string) string { + return fmt.Sprintf("step%d_%s", step, name) +} + +// worktreeStepSessionModel names a step run's sessions by the counts it walks, +// so delta compares step runs only when they stepped through the same counts. +func worktreeStepSessionModel(steps []int) string { + counts := make([]string, len(steps)) + for i, count := range steps { + counts[i] = strconv.Itoa(count) + } + return "per_agent_per_worktree_step:" + strings.Join(counts, ",") +} + +func measureWorktreeSteps(ctx context.Context, o runOptions, p *privateRun, receipts *socketClient, phase *Phase, artifacts map[string]string, series *gaugeSeries, sampler processSampler, group string) error { + phase.SessionModel = worktreeStepSessionModel(o.WorktreeSteps) + phase.Started = time.Now() + monitor := startMonitor(p, sampler, group) + var parked []*socketClient + var steps []worktreeStep + loaded := time.Duration(0) + defer func() { + closeClients(parked) + phase.DurationSec = loaded.Seconds() + monitor.finish(phase) + phase.Gate["worktree_steps"], _ = json.Marshal(steps) + }() + for k, linked := range o.WorktreeSteps { + for len(p.Repos)-1 < linked { + client, err := attachWorktree(ctx, o, p, phase) + if client != nil { + parked = append(parked, client) + } + if err != nil { + return err + } + } + captureReceipt(ctx, p, receipts, phase) + step := worktreeStep{LinkedWorktrees: linked} + agents, err := openAgents(ctx, p, o.Agents) + if err != nil { + return err + } + base := filepath.Join(p.Root, fmt.Sprintf("%s.step%d", phase.Name, k)) + window := series.watchLoad(ctx, lastStatus(phase), base) + stepMonitor := startMonitor(p, sampler, group) + first := len(phase.Attempts) + step.LoadStarted = time.Now() + loadErr := runAgents(ctx, o, p, agents, false, phase) + step.LoadEnded = time.Now() + closeClients(agents) + loaded += step.LoadEnded.Sub(step.LoadStarted) + load := Phase{Metrics: make(map[string]Metric), Attempts: phase.Attempts[first:]} + stepMonitor.finish(&load) + step.Load = load.Metrics + step.Tools = summarize(load.Attempts, step.LoadEnded.Sub(step.LoadStarted).Seconds(), "workload") + var heapErr error + step.HeapLoadStart, step.HeapLoadPeak, heapErr = series.endLoad(window) + step.HeapLoadPeakUnavailable = window.peakUnavailable + artifacts[stepArtifact(k, "heap_profile_load_start")] = step.HeapLoadStart + if step.HeapLoadPeak != "" { + artifacts[stepArtifact(k, "heap_profile_load_peak")] = step.HeapLoadPeak + } + stepErr := errors.Join(loadErr, heapErr) + if stepErr == nil { + step.HeapIdle = base + ".heap-idle.pprof" + artifacts[stepArtifact(k, "heap_profile_idle")] = step.HeapIdle + stepErr = captureCollectedHeapProfile(ctx, lastStatus(phase), step.HeapIdle) + step.IdleStarted = time.Now() + if stepErr == nil { + idle := o.Duration + if stepIdleOverride > 0 { + idle = stepIdleOverride + } + stepErr = waitContext(ctx, idle) + } + step.IdleEnded = time.Now() + } + steps = append(steps, step) + if stepErr != nil { + return stepErr + } + } + return nil +} + +// attachWorktree links one more worktree, opens a session in it and waits as +// startup does for a linked checkout: an exact worktree view, a source-backed +// read and a current analysis publication. The session stays open for the +// rest of the run, standing for an agent parked in that checkout. +func attachWorktree(ctx context.Context, o runOptions, p *privateRun, phase *Phase) (*socketClient, error) { + warmCtx, cancel := context.WithTimeout(ctx, o.WarmTimeout) + defer cancel() + if err := p.addWorktree(warmCtx); err != nil { + return nil, err + } + repo := &p.Repos[len(p.Repos)-1] + repo.Prefix = p.Repos[0].Prefix + client, err := openMCP(warmCtx, p.Socket, repo.Path) + if err != nil { + return nil, err + } + if err := preflightRoot(warmCtx, p, client, *repo, true, o.Deadline, phase); err != nil { + return client, err + } + gate, rows, err := warmAnalysis(warmCtx, client, o.Deadline) + phase.Attempts = append(phase.Attempts, rows...) + for name, value := range gate { + phase.Gate[repo.Corpus.Label+"."+name] = value + } + return client, err +} + +// openAgents opens one fresh session per reading agent, agent i in checkout +// i modulo the checkouts, which is the checkout runAgents addresses it to. +func openAgents(ctx context.Context, p *privateRun, count int) ([]*socketClient, error) { + clients := make([]*socketClient, 0, count) + for i := range count { + client, err := openMCP(ctx, p.Socket, p.Repos[i%len(p.Repos)].Path) + if err != nil { + closeClients(clients) + return nil, err + } + clients = append(clients, client) + } + return clients, nil +} + +func closeClients(clients []*socketClient) { + for _, client := range clients { + _ = client.Close() + } +} diff --git a/cmd/gortex/cli_daemon.go b/cmd/gortex/cli_daemon.go index 018d601b7..9e6ee8047 100644 --- a/cmd/gortex/cli_daemon.go +++ b/cmd/gortex/cli_daemon.go @@ -140,7 +140,11 @@ func extractToolResult(resp []byte) (json.RawMessage, error) { if repo == "" { repo = data.RepoPath } - return nil, &repoLoadingError{repo: repo, phase: data.Phase, percent: data.Percent} + admissionReason := "" + if data.Reason == "identity_pending" || data.Reason == "scope_binding_pending" { + admissionReason = data.Reason + } + return nil, &repoLoadingError{repo: repo, phase: data.Phase, percent: data.Percent, admissionReason: admissionReason} case string(gortexmcp.ErrCodeRepoUnavailable): return nil, fmt.Errorf("the gortex daemon could not load %s after its restart (%s). %s", data.RepoPath, data.Reason, data.Suggestion) diff --git a/cmd/gortex/cli_loading_socket_test.go b/cmd/gortex/cli_loading_socket_test.go new file mode 100644 index 000000000..50d01764f --- /dev/null +++ b/cmd/gortex/cli_loading_socket_test.go @@ -0,0 +1,73 @@ +package main + +import ( + "fmt" + "os" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + + "github.com/zzet/gortex/internal/daemon" + "github.com/zzet/gortex/internal/testenv" +) + +// Exercise the real CLI routing/probe/MCP/error path with piped stdout. Process +// startup is deliberately outside this bound; the runtime probes also measure +// the complete installed CLI invocation on the reference host. +func TestNonTTYCLI_LoadingSocketReturnsTempFailWithin100ms(t *testing.T) { + dispatcher, mi, _, loading := loadingDispatcherSetup(t) + controller := probeController(t, fmt.Sprintf("repos:\n - path: %q\n", loading)) + controller.multiIndexer = mi + controller.graph = mi.Graph() + dir := testenv.ShortTempDir(t) + socket := filepath.Join(dir, "s") + t.Setenv("GORTEX_DAEMON_SOCKET", socket) + t.Setenv("GORTEX_DAEMON_PIDFILE", filepath.Join(dir, "p")) + t.Setenv("GORTEX_DAEMON_STATEFILE", filepath.Join(dir, "state.json")) + t.Setenv("GORTEX_AUTOSTART", "0") + srv := daemon.New(socket, "test", zap.NewNop()) + srv.Controller = controller + srv.MCPDispatcher = dispatcher + require.NoError(t, srv.Listen()) + done := make(chan error, 1) + go func() { done <- srv.Serve() }() + t.Cleanup(func() { + require.NoError(t, srv.Shutdown()) + select { + case err := <-done: + require.NoError(t, err) + case <-time.After(5 * time.Second): + t.Error("owned CLI loading server did not stop") + } + }) + + reader, writer, err := os.Pipe() + require.NoError(t, err) + stdout := os.Stdout + os.Stdout = writer + flag := rootCmd.PersistentFlags().Lookup("loading-wait") + changed := flag.Changed + flag.Changed = false + t.Cleanup(func() { + os.Stdout = stdout + flag.Changed = changed + _ = writer.Close() + _ = reader.Close() + }) + require.Zero(t, effectiveLoadingWait(), "piped CLI stdout must not use the interactive wait") + + started := time.Now() + _, err = requireDaemonTool(loading, "search_symbols", map[string]any{"query": "LoadProbe"}) + elapsed := time.Since(started) + var coded interface{ ExitCode() int } + require.ErrorAs(t, err, &coded) + assert.Equal(t, exitTempFail, coded.ExitCode(), "execute must propagate the loading exit contract") + assert.LessOrEqual(t, elapsed, 100*time.Millisecond) + assert.Contains(t, err.Error(), "still loading") + assert.NotContains(t, err.Error(), "gortex track") + assert.NotContains(t, err.Error(), "repos reconcile") +} diff --git a/cmd/gortex/cli_repo_loading.go b/cmd/gortex/cli_repo_loading.go index d2b24a958..50a82b0fa 100644 --- a/cmd/gortex/cli_repo_loading.go +++ b/cmd/gortex/cli_repo_loading.go @@ -38,9 +38,10 @@ func init() { // It carries exit code 75, so a script or hook that runs the CLI can retry // later without parsing the message. type repoLoadingError struct { - repo string - phase string - percent int + admissionReason string + repo string + phase string + percent int // waited is how long the CLI already waited before giving up for now. waited time.Duration } @@ -50,6 +51,13 @@ func (e *repoLoadingError) Error() string { if e.waited > 0 { msg += fmt.Sprintf("; still loading after waiting %s", e.waited.Round(time.Second)) } + if e.admissionReason != "" { + msg = fmt.Sprintf("the gortex daemon is still resolving graph admission (repo_loading, %s)", e.admissionReason) + if e.waited > 0 { + msg += fmt.Sprintf("; still loading after waiting %s", e.waited.Round(time.Second)) + } + return msg + " — graph admission for this path is not yet established; retry shortly (or pass --loading-wait to wait for it)" + } return msg + " — the repository is already tracked; retry shortly (or pass --loading-wait to wait for it)" } diff --git a/cmd/gortex/daemon.go b/cmd/gortex/daemon.go index 40b3b725f..ebeaf9cb9 100644 --- a/cmd/gortex/daemon.go +++ b/cmd/gortex/daemon.go @@ -656,7 +656,7 @@ func runDaemonStart(cmd *cobra.Command, _ []string) error { state.lifecycle.EnableDeferredSeedRetirements() } - if err := srv.Listen(); err != nil { + if err := listenConfiguredDaemon(srv, state); err != nil { startupReporter.Fail(err) return err } diff --git a/cmd/gortex/daemon_controller.go b/cmd/gortex/daemon_controller.go index 44fbe2671..df8ec5892 100644 --- a/cmd/gortex/daemon_controller.go +++ b/cmd/gortex/daemon_controller.go @@ -166,9 +166,6 @@ type daemonWatcherSlot struct { // Track indexes a new repository and persists it to the global config. // Path is resolved to an absolute form before the MultiIndexer sees it. func (c *realController) Track(ctx context.Context, p daemon.TrackParams) (json.RawMessage, error) { - c.mu.Lock() - defer c.mu.Unlock() - if c.multiIndexer == nil { return nil, fmt.Errorf("multi-repo indexer not initialized") } @@ -180,6 +177,11 @@ func (c *realController) Track(ctx context.Context, p daemon.TrackParams) (json. return nil, fmt.Errorf("checkout lifecycle not initialized") } entry := config.RepoEntry{Path: absPath, Name: p.Name, Ref: p.Ref, AsWorktree: p.AsWorktree} + if err := c.lifecycle.PreflightTrack(ctx, entry); err != nil { + return nil, err + } + c.mu.Lock() + defer c.mu.Unlock() // Project association from TrackParams.Project isn't wired yet — the // config package doesn't expose an AddRepoToProject helper. Callers @@ -935,6 +937,10 @@ func (c *realController) Probe(ctx context.Context) (daemon.ProbeResponse, error if rec, ok := c.multiIndexer.RepoLoadFor(path); ok { repo.State = publicRepoState(rec.State) repo.StateReason = rec.Reason + if rec.ErrorCode == daemon.ErrPrefixCollision { + repo.State = daemon.RepoStatePrefixCollision + repo.ErrorCode, repo.OwnerPath, repo.OwnerState = rec.ErrorCode, rec.OwnerPath, rec.OwnerState + } } resp.TrackedRepos = append(resp.TrackedRepos, repo) } @@ -1083,6 +1089,20 @@ func (c *realController) status(ctx context.Context, waitForAggregate bool) (dae // every queued control request wait out the pause as well. var mem runtime.MemStats runtime.ReadMemStats(&mem) + // The operating system's and SQLite's figures are read in the same breath, + // so the runtime block describes one moment. Read after the view census + // and the aggregate, which can wait seconds for the mutex, they would + // describe a later process than the Go heap beside them. + runtimeStats := withProcessGauges(daemon.RuntimeStats{ + Alloc: mem.Alloc, + Sys: mem.Sys, + HeapInuse: mem.HeapInuse, + HeapIdle: mem.HeapIdle, + HeapReleased: mem.HeapReleased, + StackInuse: mem.StackInuse, + NumGC: mem.NumGC, + NumGoroutine: runtime.NumGoroutine(), + }) // Stat every configured repo path before taking c.mu. A stat is cheap // on a live path and on a deleted one, but an unresponsive network @@ -1132,26 +1152,23 @@ func (c *realController) status(ctx context.Context, waitForAggregate bool) (dae // subsequent busy pass. tracked := append([]daemon.TrackedRepoStatus(nil), agg.tracked...) tracked = append(tracked, reconcileUnloadedRepos(configRepos, repoMissing, agg.tracked, ready)...) + for i := range tracked { + if rec, ok := c.multiIndexer.RepoLoadFor(tracked[i].Path); ok && rec.ErrorCode == daemon.ErrPrefixCollision { + tracked[i].Unloaded, tracked[i].Loading = true, false + tracked[i].ErrorCode, tracked[i].OwnerPath, tracked[i].OwnerState = rec.ErrorCode, rec.OwnerPath, rec.OwnerState + } + } - // mem was sampled before the mutex was taken — see the note at the top - // of status. + // mem and the runtime block were sampled before the mutex was taken — + // see the note at the top of status. resp := daemon.StatusResponse{ - TrackedRepos: tracked, - MemoryBytes: mem.Alloc, - SearchBackend: agg.searchBackend, - TrigramCache: trigramCacheForResponse(), - GraphIntegrity: daemon.GraphIntegrityStatusFor(g), - Runtime: daemon.RuntimeStats{ - Alloc: mem.Alloc, - Sys: mem.Sys, - HeapInuse: mem.HeapInuse, - HeapIdle: mem.HeapIdle, - HeapReleased: mem.HeapReleased, - StackInuse: mem.StackInuse, - NumGC: mem.NumGC, - NumGoroutine: runtime.NumGoroutine(), - }, + TrackedRepos: tracked, + MemoryBytes: mem.Alloc, + SearchBackend: agg.searchBackend, + TrigramCache: trigramCacheForResponse(), + GraphIntegrity: daemon.GraphIntegrityStatusFor(g), + Runtime: runtimeStats, PProfAddr: daemonPProfAddr(), Ready: ready, WarmupSeconds: c.warmupSeconds.Load(), @@ -1599,9 +1616,8 @@ func lookupRepoMissing(missing map[string]bool, root string) bool { // reconcileUnloadedRepos returns a synthetic row for every configured // repo the daemon holds no index for. Matching is by path first (what -// track resolves and persists) and by resolved prefix second, so a repo -// registered under a derived worktree-instance prefix still matches its -// config entry. +// track resolves and persists). Prefix-only matching is reserved for entries +// without a path: another root under the same prefix is not a loaded entry. func reconcileUnloadedRepos(entries []config.RepoEntry, missing map[string]bool, loaded []daemon.TrackedRepoStatus, ready bool) []daemon.TrackedRepoStatus { if len(entries) == 0 { return nil @@ -1611,7 +1627,7 @@ func reconcileUnloadedRepos(entries []config.RepoEntry, missing map[string]bool, prefix := config.ResolvePrefix(e) isLoaded := false for _, r := range loaded { - if (e.Path != "" && pathkey.EqualPaths(r.Path, e.Path)) || (prefix != "" && r.Prefix == prefix) { + if e.Path != "" && pathkey.EqualPaths(r.Path, e.Path) || e.Path == "" && prefix != "" && r.Prefix == prefix { isLoaded = true break } diff --git a/cmd/gortex/daemon_controller_collision_test.go b/cmd/gortex/daemon_controller_collision_test.go new file mode 100644 index 000000000..b7e467057 --- /dev/null +++ b/cmd/gortex/daemon_controller_collision_test.go @@ -0,0 +1,37 @@ +package main + +import ( + "context" + "os" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/daemon" + "github.com/zzet/gortex/internal/indexer" +) + +func TestBusyControllerCollisionWithin1s(t *testing.T) { + c, _, _, dir := buildCatalogController(t) + _, owner := trackEnrichRepo(t, c, dir, "src") + loser := filepath.Join(t.TempDir(), "src") + require.NoError(t, os.MkdirAll(loser, 0o755)) + c.mu.Lock() // another mutation owns the controller for this entire call + result := make(chan error, 1) + go func() { + _, err := c.Track(context.Background(), daemon.TrackParams{Path: loser}) + result <- err + }() + select { + case err := <-result: + c.mu.Unlock() + var collision *indexer.PrefixCollisionError + require.ErrorAs(t, err, &collision) + require.Equal(t, owner, collision.ExistingPath) + case <-time.After(time.Second): + c.mu.Unlock() + <-result // join the admitted call before fixture teardown + t.Fatal("collision queued behind the controller mutex") + } +} diff --git a/cmd/gortex/daemon_controller_view.go b/cmd/gortex/daemon_controller_view.go index 5d38ba073..3b55bdd75 100644 --- a/cmd/gortex/daemon_controller_view.go +++ b/cmd/gortex/daemon_controller_view.go @@ -310,6 +310,10 @@ func fallbackProbeView(kind, checkoutID, repoPrefix, reason string) *daemon.Prob // and one whose view is not built yet is answered as uncovered so the hook // allows the native tool through. func (c *realController) FileCoverage(ctx context.Context, p daemon.FileCoverageParams) (daemon.FileCoverageResult, error) { + return c.fileCoverageWithProbe(ctx, p, c.resolveProbeView) +} + +func (c *realController) fileCoverageWithProbe(ctx context.Context, p daemon.FileCoverageParams, resolve func(context.Context, string) probeView) (daemon.FileCoverageResult, error) { if c == nil || p.Path == "" { return daemon.FileCoverageResult{}, nil } @@ -319,7 +323,7 @@ func (c *realController) FileCoverage(ctx context.Context, p daemon.FileCoverage } mutationProbe := c.lifecycle.CaptureCheckoutMutationProbe() - view := c.resolveProbeView(ctx, abs) + view := resolve(ctx, abs) defer view.release() out := daemon.FileCoverageResult{View: view.answer, Tracked: c.pathTracked(abs, view)} @@ -329,8 +333,17 @@ func (c *realController) FileCoverage(ctx context.Context, p daemon.FileCoverage // rows and reports the file covered while every graph call refuses with // repo_loading. The state rides on the reply so the hook can let the // native tool through instead of denying the only read path left. - if repo, loading := c.loadingRepoForPath(abs); loading { - out.RepoState, out.Repo = daemon.RepoStateLoading, repo + if rec, classified := c.pendingRepoLoadForPath(abs); classified { + if rec.ErrorCode == daemon.ErrPrefixCollision { + out.RepoState, out.Repo, out.Tracked = daemon.RepoStatePrefixCollision, rec.Label(), false + return out, nil + } + if rec.State.Loading() && rec.State != indexer.RepoLoadIdentityPending { + out.RepoState, out.Repo = daemon.RepoStateLoading, rec.Label() + } else if !out.Tracked && (rec.State == indexer.RepoLoadIdentityPending || rec.State.Ready()) { + out.RepoState = daemon.RepoStateIdentityPending + return out, nil // unknown or newly ready needs a fresh authoritative probe + } } if !view.servable { return out, nil @@ -414,6 +427,10 @@ const dirCoverageWalkBudget = 500 * time.Millisecond // there. It is the scope verdict a PreToolUse hook turns into a Grep/Glob // deny; both halves are exact rather than sampled. func (c *realController) DirCoverage(ctx context.Context, p daemon.DirCoverageParams) (daemon.DirCoverageResult, error) { + return c.dirCoverageWithProbe(ctx, p, c.resolveProbeView) +} + +func (c *realController) dirCoverageWithProbe(ctx context.Context, p daemon.DirCoverageParams, resolve func(context.Context, string) probeView) (daemon.DirCoverageResult, error) { if c == nil || p.Path == "" { return daemon.DirCoverageResult{}, nil } @@ -422,13 +439,22 @@ func (c *realController) DirCoverage(ctx context.Context, p daemon.DirCoveragePa abs = resolved } - view := c.resolveProbeView(ctx, abs) + view := resolve(ctx, abs) defer view.release() out := daemon.DirCoverageResult{View: view.answer, Tracked: c.pathTracked(abs, view)} // See FileCoverage: a loading repository's scope must not deny Grep. - if repo, loading := c.loadingRepoForPath(abs); loading { - out.RepoState, out.Repo = daemon.RepoStateLoading, repo + if rec, classified := c.pendingRepoLoadForPath(abs); classified { + if rec.ErrorCode == daemon.ErrPrefixCollision { + out.RepoState, out.Repo, out.Tracked = daemon.RepoStatePrefixCollision, rec.Label(), false + return out, nil + } + if rec.State.Loading() && rec.State != indexer.RepoLoadIdentityPending { + out.RepoState, out.Repo = daemon.RepoStateLoading, rec.Label() + } else if !out.Tracked && (rec.State == indexer.RepoLoadIdentityPending || rec.State.Ready()) { + out.RepoState = daemon.RepoStateIdentityPending + return out, nil // unknown or newly ready needs a fresh authoritative probe + } } if !view.servable { return out, nil diff --git a/cmd/gortex/daemon_controller_worktree_test.go b/cmd/gortex/daemon_controller_worktree_test.go index dfeaeb105..a39318f61 100644 --- a/cmd/gortex/daemon_controller_worktree_test.go +++ b/cmd/gortex/daemon_controller_worktree_test.go @@ -75,8 +75,12 @@ func buildWorktreeController(t *testing.T) (*realController, *indexer.MultiIndex reg := parser.NewRegistry() languages.RegisterAll(reg) mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + // Exercise the same registration path as a cold production startup. This + // fixture asserts worktree identity/reload effects, not batch resolution. + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } // An in-memory graph has no checkout catalog, so the lifecycle runs in // its degraded shape here: no identities, and the same index / watcher / diff --git a/cmd/gortex/daemon_health_snapshot.go b/cmd/gortex/daemon_health_snapshot.go index e0ef95fb2..cc40d2e2c 100644 --- a/cmd/gortex/daemon_health_snapshot.go +++ b/cmd/gortex/daemon_health_snapshot.go @@ -6,7 +6,9 @@ import ( "github.com/zzet/gortex/internal/daemon" "github.com/zzet/gortex/internal/graph" + "github.com/zzet/gortex/internal/platform" "github.com/zzet/gortex/internal/runtimeactivity" + "github.com/zzet/gortex/internal/sqliteconn" ) var daemonHealthMetricNames = [...]string{ @@ -39,7 +41,7 @@ func daemonHealthRuntimeStats() daemon.RuntimeStats { heapUnused := values["/memory/classes/heap/unused:bytes"] heapFree := values["/memory/classes/heap/free:bytes"] heapReleased := values["/memory/classes/heap/released:bytes"] - return daemon.RuntimeStats{ + return withProcessGauges(daemon.RuntimeStats{ Alloc: heapObjects, Sys: values["/memory/classes/total:bytes"], HeapInuse: heapObjects + heapUnused, @@ -48,7 +50,20 @@ func daemonHealthRuntimeStats() daemon.RuntimeStats { StackInuse: values["/memory/classes/heap/stacks:bytes"], NumGC: uint32(values["/gc/cycles/total:gc-cycles"]), NumGoroutine: int(values["/sched/goroutines:goroutines"]), - } + }) +} + +// withProcessGauges adds what the Go runtime cannot see to one of its samples: +// the operating system's resident accounting and SQLite's page caches. Status +// and health read them only when someone asks, so a daemon nobody watches +// pays nothing for them. +func withProcessGauges(stats daemon.RuntimeStats) daemon.RuntimeStats { + mem := platform.ReadProcessMemory() + stats.RSS, stats.RSSPeak, stats.PSS, stats.PSSFile = mem.RSSBytes, mem.PeakRSSBytes, mem.PSSBytes, mem.PSSFileBytes + stats.Footprint, stats.FootprintPeak = mem.FootprintBytes, mem.PeakFootprintBytes + pageCache, pageCachePeak := sqliteconn.PageCacheBytes() + stats.SQLitePageCache, stats.SQLitePageCachePeak = uint64(max(pageCache, 0)), uint64(max(pageCachePeak, 0)) + return stats } // buildDaemonHealthSnapshot is intentionally query-free. Health subscribers @@ -79,11 +94,28 @@ func buildDaemonHealthSnapshot( "stack_inuse_bytes": runtimeStats.StackInuse, "num_goroutine": runtimeStats.NumGoroutine, "num_gc": runtimeStats.NumGC, + + "sqlite_page_cache_bytes": runtimeStats.SQLitePageCache, + "sqlite_page_cache_peak_bytes": runtimeStats.SQLitePageCachePeak, + } + // A figure the platform has no reader for is left out, not reported as 0. + for key, value := range map[string]uint64{ + "rss_bytes": runtimeStats.RSS, + "rss_peak_bytes": runtimeStats.RSSPeak, + "pss_bytes": runtimeStats.PSS, + "pss_file_bytes": runtimeStats.PSSFile, + "footprint_bytes": runtimeStats.Footprint, + "footprint_peak_bytes": runtimeStats.FootprintPeak, + } { + if value > 0 { + out[key] = value + } } if controller != nil { out["ready"] = controller.IsReady() out["enriched"] = controller.IsEnriched() out["warmup_seconds"] = controller.warmupSeconds.Load() + out["coordinators"] = controller.lifecycle.LiveCoordinators("") if lspStatus := controller.collectLSPRouterStatus(); lspStatus != nil { out["lsp_alive"] = len(lspStatus.ActiveProviders) out["lsp_specs_registered"] = len(lspStatus.EnabledSpecs) diff --git a/cmd/gortex/daemon_janitor_ownership_test.go b/cmd/gortex/daemon_janitor_ownership_test.go index 271fd0ea7..033a4d1fd 100644 --- a/cmd/gortex/daemon_janitor_ownership_test.go +++ b/cmd/gortex/daemon_janitor_ownership_test.go @@ -1,6 +1,7 @@ package main import ( + "context" "runtime" "strings" "sync" @@ -46,7 +47,7 @@ func TestJanitorCanceledCallerRetainsAdmittedWorkerUntilRealStackClose(t *testin state, err := buildDaemonState(zap.New(core)) // Actual shared cleanup registration. require.NoError(t, err) t.Cleanup(func() { _ = state.shared.Close() }) - _, err = state.multiIndexer.IndexAll() + _, err = state.multiIndexer.TrackRepoCtx(context.Background(), gc.Repos[0]) require.NoError(t, err) nodes := state.graph.GetFileNodes("tracked/main.go") require.NotEmpty(t, nodes) diff --git a/cmd/gortex/daemon_mcp.go b/cmd/gortex/daemon_mcp.go index 707cb23cc..44912b21e 100644 --- a/cmd/gortex/daemon_mcp.go +++ b/cmd/gortex/daemon_mcp.go @@ -133,7 +133,7 @@ func (d *mcpDispatcher) Dispatch(ctx context.Context, sess *daemon.Session, fram admissionErr = nil } var terminalFamilyLoad []indexer.RepoLoad - if pending != nil && !pending.load.State.Loading() && agents.PendingAutomaticCheckout(sess.CWD, d.familyRoots()) { + if pending != nil && pending.load.State.Unavailable() && agents.PendingAutomaticCheckout(sess.CWD, d.familyRoots()) { terminalFamilyLoad = []indexer.RepoLoad{pending.load} pending = nil // still unadmitted; only the guidance/refusal classification changes } @@ -145,7 +145,7 @@ func (d *mcpDispatcher) Dispatch(ctx context.Context, sess *daemon.Session, fram return d.familyDiscoveryRefusal(sess.CWD, frame, "family_member"), nil } if pending != nil { - if refused := d.pendingRepoRefusal(sess, frame, pending.load); refused != nil { + if refused := d.pendingRepoRefusal(sess, frame, pending); refused != nil { return refused, nil } } @@ -245,6 +245,9 @@ func (d *mcpDispatcher) Dispatch(ctx context.Context, sess *daemon.Session, fram if admissionErr != nil { out = rewriteCheckoutAdmissionResponse(peekFrameMethod(frame), out, sess.CWD, admissionErr) } + if pending != nil && pending.familyInstructions != "" { + out = rewriteMCPAdmissionInstructions(peekFrameMethod(frame), out, pending.familyInstructions) + } if pending != nil { d.notePendingRepoSession(sess, peekFrameMethod(frame), pending.load) } @@ -437,6 +440,10 @@ func checkoutAdmissionError(cwd string, inbound []byte, err error) []byte { } func rewriteCheckoutAdmissionResponse(method string, out []byte, cwd string, err error) []byte { + return rewriteMCPAdmissionInstructions(method, out, gortexmcp.CheckoutAdmissionInstructions(cwd, err)) +} + +func rewriteMCPAdmissionInstructions(method string, out []byte, instructions string) []byte { if method != "initialize" && method != "server/discover" { return out } @@ -448,7 +455,7 @@ func rewriteCheckoutAdmissionResponse(method string, out []byte, cwd string, err if !ok { return out } - result["instructions"] = gortexmcp.CheckoutAdmissionInstructions(cwd, err) + result["instructions"] = instructions updated, marshalErr := json.Marshal(reply) if marshalErr != nil { return out @@ -460,7 +467,7 @@ func (d *mcpDispatcher) cwdReachableChecked(ctx context.Context, cwd string) (bo if cwd == "" { return true, nil } - if d.isCWDTracked(cwd) { + if d.registeredCWDReachable(cwd) { return true, nil } // Workspace-root fallback: a cwd that CONTAINS tracked repos — an @@ -478,12 +485,6 @@ func (d *mcpDispatcher) cwdReachableChecked(ctx context.Context, cwd string) (bo // left the session with no working call and a remedy (`gortex track // `) that would have indexed every child a second time. if d.multiIndexer != nil { - if _, _, _, ok := d.multiIndexer.ScopeForCWD(cwd); ok { - return true, nil - } - if _, _, ok := d.multiIndexer.ContainedReposScope(cwd); ok { - return true, nil - } // Configured but not registered yet: warmup has not reached the // repository, or its load ended without registering it. Decided // before the checkout and router arms, so a worktree of a loading @@ -509,18 +510,61 @@ func (d *mcpDispatcher) cwdReachableChecked(ctx context.Context, cwd string) (bo return err == nil && reachable, err } } - rtr := d.router.Load() - if rtr == nil { - return false, nil + if rtr := d.router.Load(); rtr != nil { + lookup := rtr.LookupForCwd(cwd, "") + switch lookup.Source { + case "scope-override", "config-yaml", "roster": + return true, nil + } } - lookup := rtr.LookupForCwd(cwd, "") - switch lookup.Source { - case "scope-override", "config-yaml", "roster": - return true, nil + if err := ctx.Err(); err != nil { + return false, err + } + if d.multiIndexer != nil { + if rec, classified := d.multiIndexer.RepoLoadAdmissionForPath(cwd); classified { + if rec.State.Ready() { + // Registration can publish during the probes above. The load table is + // identity evidence, not graph authority: recheck ordinary scope once. + if d.registeredCWDReachable(cwd) { + return true, nil + } + if probe != nil { + reachable, err := probe(ctx, cwd) + if err != nil || reachable { + return err == nil && reachable, err + } + } + if instructions, known := gortexmcp.PendingAutomaticCheckoutInstructions(cwd, d.familyRoots()...); known { + return false, &repoPendingAdmission{load: rec, familyInstructions: instructions} + } + rec = indexer.RepoLoad{State: indexer.RepoLoadIdentityPending, Reason: "scope_binding_pending"} + } + if err := ctx.Err(); err != nil { + return false, err + } + return false, &repoPendingAdmission{load: rec} + } } return false, nil } +// registeredCWDReachable is the existing registry/scope positive authority, +// also rechecked when registration races a negative checkout probe. +func (d *mcpDispatcher) registeredCWDReachable(cwd string) bool { + if d.isCWDTracked(cwd) { + return true + } + if d.multiIndexer != nil { + if _, _, _, ok := d.multiIndexer.ScopeForCWD(cwd); ok { + return true + } + if _, _, ok := d.multiIndexer.ContainedReposScope(cwd); ok { + return true + } + } + return false +} + // isCWDTracked reports whether the proxy's cwd lies inside any tracked // repo. Equal paths or any subdirectory of a tracked root qualify — // e.g. a proxy in ~/projects/myapp/internal counts as tracked when diff --git a/cmd/gortex/daemon_mcp_repo_loading.go b/cmd/gortex/daemon_mcp_repo_loading.go index 8bd3cc143..04bd8e341 100644 --- a/cmd/gortex/daemon_mcp_repo_loading.go +++ b/cmd/gortex/daemon_mcp_repo_loading.go @@ -17,7 +17,8 @@ import ( // failure of the check: Dispatch unwraps it and answers the session with the // repository's load state instead of treating the cwd as untracked. type repoPendingAdmission struct { - load indexer.RepoLoad + load indexer.RepoLoad + familyInstructions string } func (a *repoPendingAdmission) Error() string { @@ -34,8 +35,15 @@ func (a *repoPendingAdmission) Error() string { // the session gets the untracked rules — capabilities and track still answer, // because tracking again is the repair — with repo_unavailable and the reason // instead of repo_not_tracked. -func (d *mcpDispatcher) pendingRepoRefusal(sess *daemon.Session, frame []byte, rec indexer.RepoLoad) []byte { +func (d *mcpDispatcher) pendingRepoRefusal(sess *daemon.Session, frame []byte, pending *repoPendingAdmission) []byte { + rec := pending.load method := peekFrameMethod(frame) + if pending.familyInstructions != "" { + if method == "tools/call" && !untrackedBootstrapCall(frame) { + return familyDiscoveryError(sess.CWD, frame, "family_pending") + } + return nil + } if !rec.State.Loading() { if method == "tools/call" && !untrackedBootstrapCall(frame) { return repoUnavailableError(sess, frame, rec) @@ -53,6 +61,9 @@ func (d *mcpDispatcher) pendingRepoRefusal(sess *daemon.Session, frame []byte, r case legacy == "index_health": return d.loadingIndexHealth(frame, rec) case loadingAdmittedTool(legacy): + if rec.State == indexer.RepoLoadIdentityPending && (legacy == "track_repository" || legacy == "index_repository") { + return d.repoLoadingError(sess, frame, rec) + } return nil default: return d.repoLoadingError(sess, frame, rec) @@ -104,6 +115,17 @@ func peekFrameToolCall(frame []byte) (string, map[string]any) { // repoLoadingError is the refusal for a graph-backed frame from a loading // session. Retryable, with progress, and it never suggests tracking. func (d *mcpDispatcher) repoLoadingError(sess *daemon.Session, inbound []byte, rec indexer.RepoLoad) []byte { + if rec.State == indexer.RepoLoadIdentityPending { + reason, phase, suggestion := "identity_pending", "identity_resolution", gortexmcp.ServerInstructionsIdentityPending() + if rec.Reason == "scope_binding_pending" { + reason, phase, suggestion = rec.Reason, "scope_binding", gortexmcp.ServerInstructionsScopeBindingPending() + } + return jsonRPCErrorFrame(inbound, "graph admission is pending", map[string]any{ + "error_code": string(gortexmcp.ErrCodeRepoLoading), "retryable": true, "path": sess.CWD, + "state": reason, "reason": reason, "phase": phase, "percent": 0, + "suggestion": suggestion, + }) + } phase, percent := "starting", 0 if d.srv != nil { phase, percent = d.srv.WarmupProgress() @@ -125,6 +147,14 @@ func (d *mcpDispatcher) repoLoadingError(sess *daemon.Session, inbound []byte, r // repoUnavailableError is the refusal for a session whose configured // repository's load ended without registering it. func repoUnavailableError(sess *daemon.Session, inbound []byte, rec indexer.RepoLoad) []byte { + if rec.ErrorCode == daemon.ErrPrefixCollision { + return jsonRPCErrorFrame(inbound, "repository prefix collision: "+rec.Reason, map[string]any{ + "error_code": daemon.ErrPrefixCollision, "retryable": false, + "path": sess.CWD, "repo_path": rec.Root, "prefix": rec.Prefix, + "owner_path": rec.OwnerPath, "owner_state": rec.OwnerState, + "suggestion": gortexmcp.PrefixCollisionSuggestion(rec), + }) + } return jsonRPCErrorFrame(inbound, fmt.Sprintf("repository unavailable: %s (%s)", rec.Label(), rec.Reason), map[string]any{ "error_code": string(gortexmcp.ErrCodeRepoUnavailable), "retryable": false, @@ -146,17 +176,20 @@ func (d *mcpDispatcher) loadingIndexHealth(inbound []byte, rec indexer.RepoLoad) if d.srv != nil { phase, percent = d.srv.WarmupProgress() } + repos := []map[string]any{} + message := gortexmcp.ServerInstructionsIdentityPending() + reason := "" + if rec.State == indexer.RepoLoadIdentityPending { + phase, percent, reason = "identity_resolution", 0, "identity_pending" + if rec.Reason == "scope_binding_pending" { + phase, reason, message = "scope_binding", rec.Reason, gortexmcp.ServerInstructionsScopeBindingPending() + } + } else { + repos = append(repos, map[string]any{"repo": rec.Prefix, "path": rec.Root, "state": string(rec.State), "since": rec.Since}) + message = gortexmcp.RepoLoadingSuggestion(rec.Label()) + } payload, _ := json.Marshal(map[string]any{ - "status": "loading", - "phase": phase, - "percent": percent, - "repos": []map[string]any{{ - "repo": rec.Prefix, - "path": rec.Root, - "state": string(rec.State), - "since": rec.Since, - }}, - "message": gortexmcp.RepoLoadingSuggestion(rec.Label()), + "status": "loading", "phase": phase, "percent": percent, "repos": repos, "message": message, "reason": reason, }) var req struct { ID json.RawMessage `json:"id"` diff --git a/cmd/gortex/daemon_mcp_repo_loading_test.go b/cmd/gortex/daemon_mcp_repo_loading_test.go index 18934ba35..184b82d20 100644 --- a/cmd/gortex/daemon_mcp_repo_loading_test.go +++ b/cmd/gortex/daemon_mcp_repo_loading_test.go @@ -273,3 +273,45 @@ func TestDispatcher_LoadingRepo_ToolsWorkInPlaceAfterRegistration(t *testing.T) assert.Empty(t, errorCode(parsed), "the same session must be served after registration: %v", parsed) assert.Contains(t, toolResultText(t, parsed), "LoadProbe", "the session's scope must bind to the repository it is in") } + +func TestDispatcher_ColdTrackCollisionBecomesTypedPromptly(t *testing.T) { + d, mi, registered, loading := loadingDispatcherSetup(t) + mi.StartRepoLoad(loading) + // Use the real cold-track path and a genuine ownership failure, not a + // synthetic FinishRepoLoad error. The existing owner must stay registered. + _, err := mi.TrackRepoCtx(context.Background(), config.RepoEntry{Path: loading, Name: filepath.Base(registered)}) + var collision *indexer.PrefixCollisionError + require.ErrorAs(t, err, &collision) + failed := time.Now() + mi.FinishRepoLoad(loading, err) + sess := &daemon.Session{ID: "cold_failure", CWD: loading} + data := errorData(t, dispatchJSON(t, d, sess, + `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"LoadProbe"}}}`)) + assert.Equal(t, "prefix_collision", data["error_code"]) + assert.Equal(t, false, data["retryable"]) + assert.Equal(t, registered, data["owner_path"]) + assert.Equal(t, filepath.Base(registered), data["prefix"]) + assert.Equal(t, "present", data["owner_state"]) + assert.Less(t, time.Since(failed), time.Second, "a completed cold-track failure must not remain repo_loading") + assert.NotNil(t, mi.GetMetadata(filepath.Base(registered)), "failure must not displace the prefix owner") +} + +func TestDispatcher_ColdTrackFailureBecomesUnavailablePromptly(t *testing.T) { + d, mi, registered, loading := loadingDispatcherSetup(t) + mi.StartRepoLoad(loading) + // A real missing-root failure remains unavailable; it must not acquire + // collision semantics merely because another repository is registered. + require.NoError(t, os.RemoveAll(loading)) + _, err := mi.TrackRepoCtx(context.Background(), config.RepoEntry{Path: loading}) + require.Error(t, err) + failed := time.Now() + mi.FinishRepoLoad(loading, err) + sess := &daemon.Session{ID: "cold_failure", CWD: loading} + data := errorData(t, dispatchJSON(t, d, sess, + `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"LoadProbe"}}}`)) + assert.Equal(t, "repo_unavailable", data["error_code"]) + assert.Equal(t, false, data["retryable"]) + assert.Contains(t, data["reason"], "path does not exist") + assert.Less(t, time.Since(failed), time.Second) + assert.NotNil(t, mi.GetMetadata(filepath.Base(registered))) +} diff --git a/cmd/gortex/daemon_prefix_collision_test.go b/cmd/gortex/daemon_prefix_collision_test.go index f8fa7c393..5b5f1a243 100644 --- a/cmd/gortex/daemon_prefix_collision_test.go +++ b/cmd/gortex/daemon_prefix_collision_test.go @@ -86,7 +86,9 @@ func TestWarmupPrefixCollisionPrefersPublishedCatalogOwner(t *testing.T) { load, ok := mi.RepoLoadFor(rejected) require.True(t, ok, "a skipped collision must stay in the load table, not read as untracked") assert.Equal(t, indexer.RepoLoadFailed, load.State) - assert.Contains(t, load.Reason, "prefix_taken") + assert.Contains(t, load.Reason, "prefix_collision") + assert.Equal(t, "prefix_collision", load.ErrorCode) + assert.Equal(t, owner, load.OwnerPath) } assert.NotEmpty(t, g.GetRepoNodes("src"), "skipping a collision must not make its owner an orphan") after, found, err := catalog.GetDedicatedGraph(t.Context(), binding.GraphID) diff --git a/cmd/gortex/daemon_process_gauges_test.go b/cmd/gortex/daemon_process_gauges_test.go new file mode 100644 index 000000000..581521efd --- /dev/null +++ b/cmd/gortex/daemon_process_gauges_test.go @@ -0,0 +1,77 @@ +package main + +import ( + "context" + "runtime" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// The daemon's footprint is mostly memory the Go runtime cannot see: SQLite's +// page caches and parser allocations live outside the Go heap, and mapped +// store files are file pages. Both on-demand surfaces — the daemon_health +// snapshot and `daemon status` — must carry the operating system's view and +// SQLite's page-cache figure beside the Go runtime's, and health must say how +// many checkout build loops are running. +func TestDaemonReportsMemoryOutsideTheGoHeap(t *testing.T) { + c := probeController(t, "repos:\n - path: /work/alpha\n") + health := buildDaemonHealthSnapshot(time.Now(), c, nil, nil) + st, err := c.Status(context.Background()) + require.NoError(t, err) + + pageCache, ok := health["sqlite_page_cache_bytes"].(uint64) + require.True(t, ok, "health has no SQLite page-cache gauge: %#v", health["sqlite_page_cache_bytes"]) + peak, ok := health["sqlite_page_cache_peak_bytes"].(uint64) + require.True(t, ok, "health has no SQLite page-cache peak: %#v", health["sqlite_page_cache_peak_bytes"]) + assert.GreaterOrEqual(t, peak, pageCache) + assert.GreaterOrEqual(t, st.Runtime.SQLitePageCachePeak, st.Runtime.SQLitePageCache) + coordinators, ok := health["coordinators"].(int) + require.True(t, ok, "health has no coordinator count: %#v", health["coordinators"]) + assert.Zero(t, coordinators, "a controller without a checkout lifecycle runs no build loops") + + switch runtime.GOOS { + case "linux": + for _, key := range []string{"rss_bytes", "rss_peak_bytes", "pss_bytes"} { + value, _ := health[key].(uint64) + assert.Positive(t, value, "health %s", key) + } + assert.Positive(t, st.Runtime.RSS) + assert.Positive(t, st.Runtime.PSS) + case "darwin": + for _, key := range []string{"rss_bytes", "rss_peak_bytes", "footprint_bytes", "footprint_peak_bytes"} { + value, _ := health[key].(uint64) + assert.Positive(t, value, "health %s", key) + } + assert.Positive(t, st.Runtime.RSS) + assert.Positive(t, st.Runtime.Footprint) + assert.GreaterOrEqual(t, st.Runtime.FootprintPeak, st.Runtime.Footprint) + } +} + +// The runtime block of a status answer is one moment of the process. Go's +// figures come from ReadMemStats before the mutex; the operating system's and +// SQLite's are read beside them rather than after the view census and the +// mutex-guarded aggregate, which can wait seconds behind a track. Goroutines +// started while the aggregate is assembled must therefore not be counted. +func TestStatusSamplesTheRuntimeBlockBeforeTheAggregate(t *testing.T) { + c := probeController(t, "repos:\n - path: /work/alpha\n") + release := make(chan struct{}) + defer close(release) + const late = 1000 + c.toolSurface = func() (string, string, int) { // runs inside the aggregate, under c.mu + for range late { + go func() { <-release }() + } + return "core", "defer", 0 + } + + before := runtime.NumGoroutine() + st, err := c.Status(context.Background()) + require.NoError(t, err) + require.Equal(t, "core", st.ToolPreset, "the aggregate was not assembled") + assert.Less(t, st.Runtime.NumGoroutine, before+late/2, + "the runtime block was sampled after the aggregate, not beside ReadMemStats") +} diff --git a/cmd/gortex/daemon_repo_collision_test.go b/cmd/gortex/daemon_repo_collision_test.go new file mode 100644 index 000000000..6908b913f --- /dev/null +++ b/cmd/gortex/daemon_repo_collision_test.go @@ -0,0 +1,59 @@ +package main + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/daemon" +) + +func TestConfiguredCollisionProbeAndDispatcherPrecedence(t *testing.T) { + owner, loser := filepath.Join(t.TempDir(), "src"), filepath.Join(t.TempDir(), "src") + for _, root := range []string{owner, loser} { + require.NoError(t, os.MkdirAll(root, 0o755)) + } + d, mi := trackedPathMCPSetup(t, owner) + t.Cleanup(func() { require.NoError(t, mi.Close(context.Background())) }) + entries := []config.RepoEntry{{Path: owner}, {Path: loser}} + mi.ExpectRepos(entries) + cm, err := config.NewConfigManager(filepath.Join(t.TempDir(), "config.yaml")) + require.NoError(t, err) + cm.Global().Repos = entries // reproduce a legacy conflicting configuration + c := &realController{multiIndexer: mi, configManager: cm} + probe, err := c.Probe(t.Context()) + require.NoError(t, err) + var found bool + for _, rec := range probe.TrackedRepos { + if rec.Path == loser { + found = true + require.Equal(t, daemon.RepoStatePrefixCollision, rec.State) + require.Equal(t, daemon.ErrPrefixCollision, rec.ErrorCode) + require.Equal(t, owner, rec.OwnerPath) + } + } + require.True(t, found) + sess := &daemon.Session{ID: "prefix-collision", CWD: loser} + d.SessionStarted(sess, func([]byte) error { return nil }) + t.Cleanup(func() { d.SessionEnded(sess) }) + init := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"collision","version":"1"}}}`) + body, err := json.Marshal(init) + require.NoError(t, err) + require.Contains(t, string(body), "prefix_collision") + require.NotContains(t, string(body), "already configured") + reply := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"OwnerOnly"}}}`) + data := errorData(t, reply) + require.Equal(t, daemon.ErrPrefixCollision, data["error_code"]) + require.Equal(t, owner, data["owner_path"]) + require.Equal(t, "present", data["owner_state"]) + require.NotContains(t, data["suggestion"], "untrack") + file, err := c.FileCoverage(t.Context(), daemon.FileCoverageParams{Path: filepath.Join(loser, "file.go")}) + require.NoError(t, err) + require.Equal(t, daemon.RepoStatePrefixCollision, file.RepoState) + require.False(t, file.Tracked) + require.False(t, file.Answered) +} diff --git a/cmd/gortex/daemon_repo_identity_pending_test.go b/cmd/gortex/daemon_repo_identity_pending_test.go new file mode 100644 index 000000000..ad6913037 --- /dev/null +++ b/cmd/gortex/daemon_repo_identity_pending_test.go @@ -0,0 +1,176 @@ +package main + +import ( + "context" + "encoding/json" + "os" + "path/filepath" + "runtime" + "testing" + "time" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/daemon" + "go.uber.org/zap" +) + +func TestDispatcher_ColdIdentityPendingHasNoInventedScope(t *testing.T) { + d, mi, registered, _ := loadingDispatcherSetup(t) + physical := filepath.Join(t.TempDir(), "physical-root") + require.NoError(t, os.MkdirAll(physical, 0755)) + alias := filepath.Join(t.TempDir(), "different-configured-name") + require.NoError(t, os.Symlink(physical, alias)) + entries := []config.RepoEntry{{Path: registered}, {Path: alias, Name: "private-config-name"}} + mi.DeclareRepos(entries) + for _, cwd := range []string{physical, t.TempDir()} { + sess := &daemon.Session{ID: "identity-" + filepath.Base(cwd), CWD: cwd} + d.SessionStarted(sess, func([]byte) error { return nil }) + t.Cleanup(func() { d.SessionEnded(sess) }) + init := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"identity-test","version":"1"}}}`) + body, err := json.Marshal(init) + require.NoError(t, err) + require.Contains(t, string(body), "identity_pending") + require.NotContains(t, string(body), "INACTIVE") + require.NotContains(t, string(body), "already configured") + require.NotContains(t, string(body), "private-config-name") + require.NotContains(t, string(body), "gortex track") + discover := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":8,"method":"server/discover","params":{"_meta":{"io.modelcontextprotocol/protocolVersion":"2026-07-28","io.modelcontextprotocol/clientInfo":{"name":"claude-code","version":"2.1.312"},"io.modelcontextprotocol/clientCapabilities":{}}}}`) + discovery, marshalErr := json.Marshal(discover) + require.NoError(t, marshalErr) + require.Contains(t, string(discovery), "identity_pending") + require.NotContains(t, string(discovery), "INACTIVE") + require.NotContains(t, string(discovery), "private-config-name") + start := time.Now() + reply := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search","arguments":{"query":"RegisteredOnly"}}}`) + require.Less(t, time.Since(start), 100*time.Millisecond, "negative identity admission never waits for resolution") + data := errorData(t, reply) + require.Equal(t, "repo_loading", data["error_code"]) + require.Equal(t, true, data["retryable"]) + require.Equal(t, "identity_resolution", data["phase"]) + require.EqualValues(t, 0, data["percent"]) + require.Equal(t, "identity_pending", data["reason"]) + require.NotContains(t, data, "repo") + require.NotContains(t, data, "repo_path") + require.NotContains(t, reply, "result") + for _, tool := range []string{ + `"name":"index_health","arguments":{}`, + `"name":"workspace","arguments":{"operation":"index"}`, + } { + health := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{`+tool+`}}`) + text := toolResultText(t, health) + var payload map[string]any + require.NoError(t, json.Unmarshal([]byte(text), &payload)) + require.Equal(t, "identity_pending", payload["reason"]) + require.Empty(t, payload["repos"], "an unbound diagnostic must not disclose named repositories") + } + track := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"track_repository","arguments":{"path":"."}}}`) + require.Equal(t, "identity_pending", errorData(t, track)["reason"], "unknown ownership must not duplicate a configured alias through bootstrap tracking") + } + mi.ExpectRepos(entries) + physicalSess := &daemon.Session{ID: "resolved-alias", CWD: physical} + data := errorData(t, dispatchJSON(t, d, physicalSess, `{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search","arguments":{"query":"RegisteredOnly"}}}`)) + require.Equal(t, "repo_loading", data["error_code"]) + require.Equal(t, alias, data["repo_path"], "completed owner resolution can now prove configured ownership") + unknown, err := d.cwdReachableChecked(t.Context(), t.TempDir()) + require.NoError(t, err) + require.False(t, unknown, "a completed identity set permits a definitive negative") +} + +func TestDispatcher_IdentityPendingPreservesAuthoritativePositives(t *testing.T) { + d, mi, registered, _ := loadingDispatcherSetup(t) + mi.DeclareRepos([]config.RepoEntry{{Path: registered}, {Path: filepath.Join(t.TempDir(), "unresolved")}}) + require.True(t, mi.RepoIdentityPending()) + ok, err := d.cwdReachableChecked(t.Context(), registered) + require.NoError(t, err) + require.True(t, ok) + healthySession := &daemon.Session{ID: "registered-while-identity-pending", CWD: registered} + d.SessionStarted(healthySession, func([]byte) error { return nil }) + t.Cleanup(func() { d.SessionEnded(healthySession) }) + dispatchJSON(t, d, healthySession, `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"positive-control","version":"1"}}}`) + healthyReply := dispatchJSON(t, d, healthySession, `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search","arguments":{"query":"RegisteredOnly"}}}`) + require.Empty(t, errorCode(healthyReply)) + require.Contains(t, toolResultText(t, healthyReply), "RegisteredOnly") + checkout := t.TempDir() + d.checkoutCWDProbe = func(_ context.Context, cwd string) (bool, error) { return cwd == checkout, nil } + ok, err = d.cwdReachableChecked(t.Context(), checkout) + require.NoError(t, err) + require.True(t, ok, "an authoritative existing checkout still serves while another root is unresolved") + remote := t.TempDir() + require.NoError(t, writeFile(filepath.Join(remote, ".gortex.yaml"), "workspace: remote-ws\n")) + cfg := &daemon.ServersConfig{Server: []daemon.ServerEntry{ + {Slug: "self", URL: "unix:///tmp/never-dialed.sock", Default: true}, + {Slug: "remote", URL: "https://remote.example.com", Workspaces: []string{"remote-ws"}}, + }} + require.NoError(t, cfg.Validate()) + d.SetRouter(daemon.NewRouter(daemon.RouterConfig{Servers: cfg, Rosters: daemon.NewWorkspaceRosterCache(0), LocalSlug: "self", Logger: zap.NewNop()})) + ok, err = d.cwdReachableChecked(t.Context(), remote) + require.NoError(t, err) + require.True(t, ok, "router authority takes precedence over an unresolved negative") + ctx, cancel := context.WithCancel(t.Context()) + cancel() + _, err = d.cwdReachableChecked(ctx, t.TempDir()) + require.ErrorIs(t, err, context.Canceled) +} + +func TestCoverage_IdentityPendingIsNeitherTrackedNorGraphAnswer(t *testing.T) { + _, mi, registered, _ := loadingDispatcherSetup(t) + mi.DeclareRepos([]config.RepoEntry{{Path: registered}, {Path: filepath.Join(t.TempDir(), "unresolved")}}) + c := &realController{multiIndexer: mi} + path := filepath.Join(t.TempDir(), "file.go") + file, err := c.FileCoverage(t.Context(), daemon.FileCoverageParams{Path: path}) + require.NoError(t, err) + require.Equal(t, daemon.RepoStateIdentityPending, file.RepoState) + require.Empty(t, file.Repo) + require.False(t, file.Tracked) + require.False(t, file.Answered) + require.False(t, file.Covered) + dir, err := c.DirCoverage(t.Context(), daemon.DirCoverageParams{Path: filepath.Dir(path)}) + require.NoError(t, err) + require.Equal(t, daemon.RepoStateIdentityPending, dir.RepoState) + require.Empty(t, dir.Repo) + require.False(t, dir.Tracked) + require.False(t, dir.Answered) + require.False(t, dir.HasSource) + healthy, err := c.FileCoverage(t.Context(), daemon.FileCoverageParams{Path: filepath.Join(registered, "a.go")}) + require.NoError(t, err) + require.Empty(t, healthy.RepoState) +} + +func TestExtractToolResult_IdentityPendingDoesNotClaimTracking(t *testing.T) { + _, err := extractToolResult([]byte(`{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"identity pending","data":{"error_code":"repo_loading","retryable":true,"reason":"identity_pending","phase":"identity_resolution","percent":0}}}`)) + var loading *repoLoadingError + require.ErrorAs(t, err, &loading) + require.Equal(t, exitTempFail, loading.ExitCode()) + require.Contains(t, err.Error(), "identity_pending") + require.NotContains(t, err.Error(), "already tracked") + require.NotContains(t, err.Error(), "gortex track") +} + +func TestDispatcher_ColdDarwinTmpAliasUsesIdentityPending(t *testing.T) { + if runtime.GOOS != "darwin" { + t.Skip("requires Darwin's /tmp -> /private/tmp alias") + } + canonical, err := filepath.EvalSymlinks("/tmp") + require.NoError(t, err) + if canonical != "/private/tmp" { + t.Skip("host does not expose the /tmp -> /private/tmp alias") + } + root, err := os.MkdirTemp("/tmp", "g1023-wire-alias-") + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, os.RemoveAll(root)) }) + d, mi := trackedPathMCPSetup(t, t.TempDir()) + mi.DeclareRepos([]config.RepoEntry{{Path: root}}) + sess := &daemon.Session{ID: "actual-tmp-alias", CWD: filepath.Join(canonical, filepath.Base(root))} + d.SessionStarted(sess, func([]byte) error { return nil }) + t.Cleanup(func() { d.SessionEnded(sess) }) + initialized := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"alias-test","version":"1"}}}`) + raw, err := json.Marshal(initialized) + require.NoError(t, err) + require.Contains(t, string(raw), "identity_pending") + require.NotContains(t, string(raw), "INACTIVE") + data := errorData(t, dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"Anything"}}}`)) + require.Equal(t, "repo_loading", data["error_code"]) + require.Equal(t, "identity_pending", data["reason"]) + require.NotContains(t, data, "repo_path") +} diff --git a/cmd/gortex/daemon_repo_loading.go b/cmd/gortex/daemon_repo_loading.go index 0159b3d41..4e3074229 100644 --- a/cmd/gortex/daemon_repo_loading.go +++ b/cmd/gortex/daemon_repo_loading.go @@ -5,20 +5,27 @@ import ( "github.com/zzet/gortex/internal/indexer" ) -// announceConfiguredRepos makes every configured repository read as loading. -// -// The warmup goroutine calls it first, ahead of checkout seeding rather than -// when warmup reaches its job loop: seeding takes seconds on a large worktree -// farm, and for that whole window the admission gate otherwise told sessions -// their configured repository was untracked. It is not called before Listen: -// it stats every configured root, and a hung mount must not keep the socket -// from coming up. +// listenConfiguredDaemon publishes lexical configured roots before opening +// either listener. It performs no filesystem validation of those roots; warmup +// does that later so a hung mount cannot delay serving other configured repos. +func listenConfiguredDaemon(srv *daemon.Server, state *daemonState) error { + if state != nil && state.multiIndexer != nil && state.configManager != nil { + if gc := state.configManager.Global(); gc != nil { + state.multiIndexer.DeclareRepos(gc.ReposSnapshot()) + } + } + return srv.Listen() +} + +// announceConfiguredRepos fills the declarations' filesystem identity before +// checkout seeding. Serving has already started, so filesystem work stays here +// rather than in the lexical declaration that precedes Listen. func announceConfiguredRepos(state *daemonState) { if state == nil || state.multiIndexer == nil || state.configManager == nil { return } if gc := state.configManager.Global(); gc != nil { - state.multiIndexer.ExpectRepos(gc.Repos) + state.multiIndexer.ExpectRepos(gc.ReposSnapshot()) } } @@ -60,16 +67,12 @@ func publicRepoState(state indexer.RepoLoadState) string { return "" } -// loadingRepoForPath names the configured repository abs belongs to when that -// repository is still loading, for the coverage replies the hooks turn into a -// deny. Lock-free; reads no store. -func (c *realController) loadingRepoForPath(abs string) (repo string, loading bool) { +// pendingRepoLoadForPath reads one snapshot: ownership and unresolved identity +// must never be separated by a concurrent warmup publication. Ready records +// still require ordinary coverage authority, not permission from this table. +func (c *realController) pendingRepoLoadForPath(abs string) (indexer.RepoLoad, bool) { if c == nil || c.multiIndexer == nil { - return "", false - } - rec, pending := c.multiIndexer.PendingRepoForPath(abs) - if !pending || !rec.State.Loading() { - return "", false + return indexer.RepoLoad{}, false } - return rec.Label(), true + return c.multiIndexer.RepoLoadAdmissionForPath(abs) } diff --git a/cmd/gortex/daemon_repo_publication_test.go b/cmd/gortex/daemon_repo_publication_test.go new file mode 100644 index 000000000..e34f14ff7 --- /dev/null +++ b/cmd/gortex/daemon_repo_publication_test.go @@ -0,0 +1,146 @@ +package main + +import ( + "context" + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/daemon" + "github.com/zzet/gortex/internal/indexer" +) + +func TestDispatcher_IdentityPublicationDuringCheckoutProbe(t *testing.T) { + for _, register := range []bool{false, true} { + name := "identity" + if register { + name = "registration" + } + t.Run(name, func(t *testing.T) { + d, mi, registered, _ := loadingDispatcherSetup(t) + physical := t.TempDir() + require.NoError(t, writeFile(filepath.Join(physical, "source.go"), "package source\nfunc Published() {}\n")) + alias := filepath.Join(t.TempDir(), "configured-alias") + require.NoError(t, os.Symlink(physical, alias)) + entries := []config.RepoEntry{{Path: registered}, {Path: alias, Name: "published"}} + mi.DeclareRepos(entries) + calls := 0 + d.checkoutCWDProbe = func(ctx context.Context, path string) (bool, error) { + calls++ + require.Equal(t, physical, path) + mi.ExpectRepos(entries) + if register { + _, err := mi.TrackRepoCtx(ctx, config.RepoEntry{Path: physical, Name: "published"}) + require.NoError(t, err) + } + return false, nil // The earlier negative probe is now stale. + } + reachable, err := d.cwdReachableChecked(t.Context(), physical) + require.Equal(t, 1, calls) + if register { + require.NoError(t, err) + require.True(t, reachable, "normal registered scope must be rechecked after publication") + } else { + var pending *repoPendingAdmission + require.ErrorAs(t, err, &pending) + require.False(t, reachable) + require.Equal(t, alias, pending.load.Root) + require.Equal(t, indexer.RepoLoadQueued, pending.load.State) + } + }) + } +} + +func TestCoverage_IdentityPublicationDuringPublicViewProbe(t *testing.T) { + for _, door := range []string{"file", "directory"} { + t.Run(door, func(t *testing.T) { + _, mi, registered, _ := loadingDispatcherSetup(t) + physical := t.TempDir() + alias := filepath.Join(t.TempDir(), "configured-alias") + require.NoError(t, os.Symlink(physical, alias)) + entries := []config.RepoEntry{{Path: registered}, {Path: alias, Name: "published"}} + mi.DeclareRepos(entries) + c := &realController{multiIndexer: mi} + probe := func(ctx context.Context, path string) probeView { + view := c.resolveProbeView(ctx, path) + mi.ExpectRepos(entries) + return view + } + if door == "file" { + result, err := c.fileCoverageWithProbe(t.Context(), daemon.FileCoverageParams{Path: filepath.Join(physical, "source.go")}, probe) + require.NoError(t, err) + require.Equal(t, daemon.RepoStateLoading, result.RepoState) + require.Equal(t, "published", result.Repo) + require.False(t, result.Covered) + } else { + result, err := c.dirCoverageWithProbe(t.Context(), daemon.DirCoverageParams{Path: physical}, probe) + require.NoError(t, err) + require.Equal(t, daemon.RepoStateLoading, result.RepoState) + require.Equal(t, "published", result.Repo) + require.False(t, result.HasSource) + } + }) + } +} + +func TestDispatcher_ReadyFamilyIdentityDoesNotGrantUnprovenCheckoutScope(t *testing.T) { + d, mi, registered, primary := loadingDispatcherSetup(t) + linked := t.TempDir() + admin := filepath.Join(primary, ".git", "worktrees", "linked") + require.NoError(t, os.MkdirAll(admin, 0755)) + require.NoError(t, os.WriteFile(filepath.Join(admin, "commondir"), []byte("../.."), 0644)) + require.NoError(t, os.WriteFile(filepath.Join(linked, ".git"), []byte("gitdir: "+admin), 0644)) + entries := []config.RepoEntry{{Path: registered}, {Path: primary, Name: "loading"}} + mi.DeclareRepos(entries) + calls := 0 + d.checkoutCWDProbe = func(ctx context.Context, path string) (bool, error) { + calls++ + require.Equal(t, linked, path) + if calls == 1 { + mi.ExpectRepos(entries) + _, err := mi.TrackRepoCtx(ctx, config.RepoEntry{Path: primary, Name: "loading"}) + require.NoError(t, err) + } + return false, nil // No authoritative checkout binding is available yet. + } + sess := &daemon.Session{ID: "registered-family-not-route", CWD: linked} + reply := dispatchJSON(t, d, sess, `{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"LoadProbe"}}}`) + require.Equal(t, 2, calls, "a matched-ready identity permits only one ordinary authority recheck") + data := errorData(t, reply) + require.Equal(t, "view_building", data["error_code"]) + require.Equal(t, "family_pending", data["reason"]) + require.Equal(t, "discovery_pending", data["state"]) + require.NotContains(t, data, "repo_path") + require.NotContains(t, reply, "result", "family identity alone grants no graph scope") + // Retain the actual typed result before a later filesystem change. Metadata + // rendering and graph refusal must not re-probe and become untracked. + d.checkoutCWDProbe = nil + _, err := d.cwdReachableChecked(t.Context(), linked) + var classified *repoPendingAdmission + require.ErrorAs(t, err, &classified) + require.NotEmpty(t, classified.familyInstructions) + require.NoError(t, os.Remove(filepath.Join(linked, ".git"))) + frame := []byte(`{"jsonrpc":"2.0","id":7,"method":"tools/call","params":{"name":"read","arguments":{}}}`) + require.Contains(t, string(d.pendingRepoRefusal(sess, frame, classified)), `"error_code":"view_building"`) + initialized := rewriteMCPAdmissionInstructions("initialize", []byte(`{"jsonrpc":"2.0","id":8,"result":{"instructions":"INACTIVE"}}`), classified.familyInstructions) + require.Contains(t, string(initialized), "view_building") + require.NotContains(t, string(initialized), "INACTIVE") + unknown := t.TempDir() + require.NoError(t, os.MkdirAll(filepath.Join(unknown, ".git"), 0755)) + require.False(t, mi.RepoIdentityPending()) + outside := &daemon.Session{ID: "foreign-family", CWD: unknown} + negative := dispatchJSON(t, d, outside, `{"jsonrpc":"2.0","id":9,"method":"tools/call","params":{"name":"read","arguments":{}}}`) + require.Equal(t, "repo_not_tracked", errorCode(negative), "a genuine foreign family stays definitively untracked after identity completion") +} + +func TestExtractToolResult_ScopeBindingPendingDoesNotClaimTracking(t *testing.T) { + _, err := extractToolResult([]byte(`{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"scope binding pending","data":{"error_code":"repo_loading","retryable":true,"reason":"scope_binding_pending","phase":"scope_binding","percent":0}}}`)) + var loading *repoLoadingError + require.ErrorAs(t, err, &loading) + require.Equal(t, exitTempFail, loading.ExitCode()) + require.Contains(t, err.Error(), "scope_binding_pending") + require.NotContains(t, err.Error(), "already tracked") + require.NotContains(t, err.Error(), "gortex track") +} diff --git a/cmd/gortex/daemon_startup_loading_test.go b/cmd/gortex/daemon_startup_loading_test.go new file mode 100644 index 000000000..64c12ee46 --- /dev/null +++ b/cmd/gortex/daemon_startup_loading_test.go @@ -0,0 +1,120 @@ +package main + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + + "github.com/zzet/gortex/internal/daemon" + gortexmcp "github.com/zzet/gortex/internal/mcp" +) + +// Exercise the real pre-listen helper and public MCP socket before ANY warmup +// announcement, seeding or repository indexing. Starting registration by hand +// here would conceal the startup window this test is intended to cover. +func TestConfiguredLoadingPublishedBeforeServing(t *testing.T) { + registered := t.TempDir() + d, mi := trackedPathMCPSetup(t, registered) + root := filepath.Join(t.TempDir(), "configured") + admin := filepath.Join(root, ".git", "worktrees", "linked") + linked := filepath.Join(t.TempDir(), "linked") + require.NoError(t, os.MkdirAll(admin, 0o755)) + require.NoError(t, os.MkdirAll(filepath.Join(root, "pkg"), 0o755)) + require.NoError(t, os.MkdirAll(filepath.Join(linked, "pkg"), 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(admin, "commondir"), []byte("../..\n"), 0o600)) + require.NoError(t, os.WriteFile(filepath.Join(linked, ".git"), []byte("gitdir: "+admin+"\n"), 0o600)) + c := probeController(t, fmt.Sprintf("repos:\n - path: %q\n name: startup\n", root)) + state := &daemonState{multiIndexer: mi, configManager: c.configManager} + + file, err := os.CreateTemp(os.TempDir(), "gx-startup-*.sock") + require.NoError(t, err) + socket := file.Name() + require.NoError(t, file.Close()) + require.NoError(t, os.Remove(socket)) + t.Setenv("GORTEX_DAEMON_PIDFILE", filepath.Join(t.TempDir(), "daemon.pid")) + t.Setenv("GORTEX_DAEMON_STATEFILE", filepath.Join(t.TempDir(), "daemon.state.json")) + srv := daemon.New(socket, "test", zap.NewNop()) + srv.MCPDispatcher = d + require.NoError(t, listenConfiguredDaemon(srv, state)) + served := make(chan error, 1) + go func() { served <- srv.Serve() }() + t.Cleanup(func() { + require.NoError(t, srv.Shutdown()) + select { + case err := <-served: + require.NoError(t, err) + case <-time.After(5 * time.Second): + t.Error("owned startup test server did not stop") + } + }) + + for name, cwd := range map[string]string{ + "primary": root, "subdirectory": filepath.Join(root, "pkg"), + "linked": linked, "linked subdirectory": filepath.Join(linked, "pkg"), + } { + t.Run(name, func(t *testing.T) { + client, err := daemon.DialTo(socket, daemon.Handshake{Mode: daemon.ModeMCP, CWD: cwd}) + require.NoError(t, err) + defer client.Close() + require.NoError(t, client.Conn.SetDeadline(time.Now().Add(5*time.Second))) + require.NoError(t, client.WriteMCPFrame([]byte(`{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"startup-probe","version":"1"}}}`))) + initialized, err := client.ReadMCPFrame() + require.NoError(t, err) + var initReply struct { + Result struct { + Instructions string `json:"instructions"` + } `json:"result"` + } + require.NoError(t, json.Unmarshal(initialized, &initReply)) + expectedInstructions := gortexmcp.ServerInstructionsLoading("startup", "starting", 0) + if strings.HasPrefix(name, "linked") { + expectedInstructions = gortexmcp.ServerInstructionsIdentityPending() + } + assert.Contains(t, initReply.Result.Instructions, expectedInstructions, + "the first real initialize must carry an honest loading variant before identity warmup") + require.NoError(t, client.WriteMCPFrame([]byte(`{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"Startup"}}}`))) + frame, err := client.ReadMCPFrame() + require.NoError(t, err) + var reply map[string]any + require.NoError(t, json.Unmarshal(frame, &reply)) + data := errorData(t, reply) + assert.Equal(t, "repo_loading", data["error_code"]) + assert.Equal(t, true, data["retryable"]) + if strings.HasPrefix(name, "linked") { + assert.Equal(t, "identity_pending", data["reason"]) + assert.NotContains(t, data, "repo_path", "an unresolved lexical declaration cannot name a linked request's family") + } else { + assert.Equal(t, root, data["repo_path"]) + } + assert.Contains(t, data, "phase") + assert.Contains(t, data, "percent") + assert.NotContains(t, string(frame), "gortex track") + assert.NotContains(t, string(frame), "repos reconcile") + }) + } + assert.Empty(t, mi.GetMetadata("startup"), "pending declarations must never register or admit graph access") + untracked := &daemon.Session{ID: "unconfigured", CWD: t.TempDir()} + before := errorData(t, dispatchJSON(t, d, untracked, + `{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"Startup"}}}`)) + assert.Equal(t, "repo_loading", before["error_code"]) + assert.Equal(t, "identity_pending", before["reason"]) + assert.NotContains(t, before, "repo_path") + // The real existing warmup owner now resolves identity. A linked path can + // then name its configured family and an unrelated path is truly negative. + announceConfiguredRepos(state) + linkedSession := &daemon.Session{ID: "resolved-linked", CWD: linked} + after := errorData(t, dispatchJSON(t, d, linkedSession, + `{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"Startup"}}}`)) + assert.Equal(t, "repo_loading", after["error_code"]) + assert.Equal(t, root, after["repo_path"]) + assert.Equal(t, "repo_not_tracked", errorCode(dispatchJSON(t, d, untracked, + `{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"Startup"}}}`))) +} diff --git a/cmd/gortex/daemon_state.go b/cmd/gortex/daemon_state.go index 9c252ddec..2d7d81663 100644 --- a/cmd/gortex/daemon_state.go +++ b/cmd/gortex/daemon_state.go @@ -736,6 +736,10 @@ func warmupDaemonStateContext(ctx context.Context, state *daemonState, logger *z var changedPrefixes sync.Map var scopeUnknown atomic.Bool deltaFrontier := newWarmupDeltaFrontier() + accounting := warmupRepoAccounting{ + changed: &changedRepos, files: &filesReindexed, unknown: &scopeUnknown, + prefixes: &changedPrefixes, frontier: deltaFrontier, + } if err := state.multiIndexer.RunRepositoryTopologyBatch(ctx, func(batchCtx context.Context) error { coordinatedBulk, _ := state.graph.(graph.CoordinatedBulkLoader) coordinatedBulkActive := coordinatedBulk != nil && coordinatedBulk.BeginCoordinatedBulkLoad() @@ -809,82 +813,8 @@ func warmupDaemonStateContext(ctx context.Context, state *daemonState, logger *z } priorMtimes = nil } - pathFn := "track" - // indexed names the repo prefix whose generation-0 - // index this dispatch just brought up to date, and is - // empty when it did not. It is the completion signal - // the committed-base publication waits on: a base is - // published from a Git tree, not from generation 0, - // but its identity names the checkout's committed tree - // and its builder runs against this repository's own - // admissions, so both must be settled first. - indexed := "" - if priorMtimes != nil { - pathFn = "reconcile" - res, err := state.multiIndexer.ReconcileRepoCtx(batchCtx, entry, priorMtimes) - switch { - case err != nil: - jobErr = err - logger.Warn("daemon: startup reconcile failed", - zap.String("path", entry.Path), zap.Error(err)) - // Treat a failed reconcile as "changed" so the global - // passes still run — degrade toward correctness, not - // toward the fast path, when we can't trust the delta. - changedRepos.Add(1) - scopeUnknown.Store(true) - deltaFrontier.invalidate() - case res != nil && (res.StaleFileCount > 0 || res.DeletedFileCount > 0 || len(res.FailedFiles) > 0 || res.FullRetrack): - changedRepos.Add(1) - filesReindexed.Add(int64(reconcileFileCount(res))) - deltaFrontier.record(res) - indexed = warmupRepoPrefix(state, entry, res.RepoPrefix) - if res.RepoPrefix != "" { - changedPrefixes.Store(res.RepoPrefix, struct{}{}) - } else { - scopeUnknown.Store(true) - } - default: - // Warm no-op path: the repo re-indexed nothing, so its - // graph is served straight from the persisted store and - // no global pass has to re-run for it. - prefix := "" - if res != nil { - prefix = res.RepoPrefix - } - indexed = warmupRepoPrefix(state, entry, prefix) - } - } else { - // No prior mtimes → full cold (re)index of this repo, - // which is "changed" by definition. - deltaFrontier.invalidate() - changedRepos.Add(1) - if res, err := state.multiIndexer.TrackRepoCtx(batchCtx, entry); err != nil { - jobErr = err - logger.Warn("daemon: startup track failed", - zap.String("path", entry.Path), zap.Error(err)) - scopeUnknown.Store(true) - } else if res != nil && res.RepoPrefix != "" { - // A cold TrackRepoCtx is itself a full retrack — its - // FileCount is the whole repo's file-level work. - filesReindexed.Add(int64(res.FileCount)) - changedPrefixes.Store(res.RepoPrefix, struct{}{}) - indexed = warmupRepoPrefix(state, entry, res.RepoPrefix) - } else { - scopeUnknown.Store(true) - } - } - // Schedule, do not publish. A committed base is a full - // index of a committed tree; running it here would put - // a second whole-repository index in front of the - // readiness flip. Schedule only appends to an unbounded - // pending list — it cannot park this worker, whatever - // the repository count — and nothing is drained until - // BeginDraining below, which runs after markReady. The - // runtime cancels the publisher the moment publisher - // admission closes, so shutdown stays bounded. - if indexed != "" { - state.basePublisher.Schedule(indexed) - } + pathFn, loadErr := accounting.load(batchCtx, state, entry, priorMtimes, logger) + jobErr = loadErr elapsed := time.Since(repoStart) if elapsed > 2*time.Second { logger.Info("daemon: warmup repo elapsed", diff --git a/cmd/gortex/daemon_status_repo_counts_test.go b/cmd/gortex/daemon_status_repo_counts_test.go index 9cb431a04..e558d44c8 100644 --- a/cmd/gortex/daemon_status_repo_counts_test.go +++ b/cmd/gortex/daemon_status_repo_counts_test.go @@ -54,7 +54,7 @@ func TestStatus_TrackedRepoRowFollowsTheLiveGraph(t *testing.T) { reg := parser.NewRegistry() languages.RegisterAll(reg) mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() + _, err = mi.TrackRepoCtx(context.Background(), config.RepoEntry{Path: dir}) require.NoError(t, err) metas := mi.AllMetadata() @@ -130,7 +130,7 @@ func TestStatus_TwoReposEachReportTheirOwnBucket(t *testing.T) { // A is indexed while it is the only configured repo. g := graph.New() mi1 := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi1.IndexAll() + _, err = mi1.TrackRepoCtx(context.Background(), config.RepoEntry{Path: dirA}) require.NoError(t, err) var priorA map[string]int64 for _, m := range mi1.AllMetadata() { diff --git a/cmd/gortex/daemon_warmup_collision_skip_test.go b/cmd/gortex/daemon_warmup_collision_skip_test.go new file mode 100644 index 000000000..7e3d5e934 --- /dev/null +++ b/cmd/gortex/daemon_warmup_collision_skip_test.go @@ -0,0 +1,126 @@ +package main + +import ( + "context" + "os" + "path/filepath" + "sync" + "sync/atomic" + "testing" + + "github.com/stretchr/testify/require" + "go.uber.org/zap" + + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/indexer" +) + +func TestWarmupCollisionTypedSkip(t *testing.T) { + for _, mode := range []string{"warm", "cold"} { + t.Run(mode, func(t *testing.T) { + base := startupPublicationEnv(t) + owner := startupPublicationRepo(t, base, "a/src") + rejected := startupPublicationRepo(t, base, "b/src") + require.NoError(t, os.WriteFile(filepath.Join(owner, ".gortex.yaml"), + []byte("workspace: owner\nexclude:\n - owner-only\n"), 0o600)) + require.NoError(t, os.WriteFile(filepath.Join(rejected, ".gortex.yaml"), + []byte("workspace: rejected\nexclude:\n - rejected-only\n"), 0o600)) + state, err := buildDaemonState(zap.NewNop()) + require.NoError(t, err) + t.Cleanup(func() { + state.basePublisher.Close() + require.NoError(t, state.shared.Close()) + }) + require.NotNil(t, state.basePublisher, "a real publication queue is required to prove no scheduling") + registered, err := state.lifecycle.Register(t.Context(), config.RepoEntry{Path: owner, Name: "src"}, indexer.TrackSourceCLI) + require.NoError(t, err) + require.NoError(t, registered.CatalogErr) + ownerConfig, revision := state.configManager.GetRepoConfig("src"), state.configManager.Revision() + nodes, edges := state.graph.GetRepoNodes("src"), state.graph.AllEdges() + entry := config.RepoEntry{Path: rejected, Name: "src"} + state.multiIndexer.ExpectRepos([]config.RepoEntry{entry}) + state.multiIndexer.StartRepoLoad(rejected) + var prior map[string]int64 + if mode == "warm" { + prior = map[string]int64{"main.go": 1} + } + var changed, files atomic.Int64 + var unknown atomic.Bool + var prefixes sync.Map + frontier := newWarmupDeltaFrontier() + accounting := warmupRepoAccounting{changed: &changed, files: &files, unknown: &unknown, prefixes: &prefixes, frontier: frontier} + var pathFn string + var jobErr error + require.NoError(t, state.multiIndexer.RunRepositoryTopologyBatch(t.Context(), func(ctx context.Context) error { + pathFn, jobErr = accounting.load(ctx, state, entry, prior, zap.NewNop()) + return nil + })) + var skip *indexer.RepositoryCollisionSkip + require.ErrorAs(t, jobErr, &skip) + require.Equal(t, "src", skip.Collision.Prefix) + require.Equal(t, owner, skip.Collision.ExistingPath) + if mode == "warm" { + require.Equal(t, "reconcile", pathFn) + } else { + require.Equal(t, "track", pathFn) + } + require.Zero(t, changed.Load()) + require.Zero(t, files.Load()) + require.False(t, unknown.Load()) + prefixes.Range(func(key, value any) bool { + t.Errorf("collision recorded changed prefix %v", key) + return true + }) + frontier.mu.Lock() + exact, plans, crossFiles := frontier.exact, len(frontier.plans), len(frontier.crossRepoFiles) + frontier.mu.Unlock() + require.True(t, exact, "a rejected root must not invalidate the existing delta frontier") + require.Zero(t, plans) + require.Zero(t, crossFiles) + require.Zero(t, state.basePublisher.Pending()) + require.Empty(t, state.basePublisher.Outcomes()) + require.Equal(t, nodes, state.graph.GetRepoNodes("src")) + require.Equal(t, edges, state.graph.AllEdges()) + require.Equal(t, ownerConfig, state.configManager.GetRepoConfig("src")) + require.Equal(t, revision, state.configManager.Revision()) + state.multiIndexer.FinishRepoLoad(rejected, jobErr) + load, found := state.multiIndexer.RepoLoadFor(rejected) + require.True(t, found) + require.Equal(t, indexer.RepoLoadFailed, load.State) + require.Equal(t, "prefix_collision", load.ErrorCode) + require.Equal(t, owner, load.OwnerPath) + require.Equal(t, "present", load.OwnerState) + // Positive control uses the same accounting and real queue. A + // registered owner's warm no-op still schedules its own base. + _, err = accounting.load(t.Context(), state, config.RepoEntry{Path: owner, Name: "src"}, map[string]int64{"main.go": 1}, zap.NewNop()) + require.NoError(t, err) + require.Equal(t, 1, state.basePublisher.Pending()) + require.Zero(t, changed.Load()) + }) + } +} + +func TestWarmupNonCollisionFailureStillInvalidates(t *testing.T) { + base := startupPublicationEnv(t) + state, err := buildDaemonState(zap.NewNop()) + require.NoError(t, err) + t.Cleanup(func() { + state.basePublisher.Close() + require.NoError(t, state.shared.Close()) + }) + var changed, files atomic.Int64 + var unknown atomic.Bool + var prefixes sync.Map + frontier := newWarmupDeltaFrontier() + accounting := warmupRepoAccounting{changed: &changed, files: &files, unknown: &unknown, prefixes: &prefixes, frontier: frontier} + _, err = accounting.load(t.Context(), state, config.RepoEntry{Path: filepath.Join(base, "missing"), Name: "missing"}, map[string]int64{"old.go": 1}, zap.NewNop()) + require.Error(t, err) + var skip *indexer.RepositoryCollisionSkip + require.NotErrorAs(t, err, &skip) + require.Equal(t, int64(1), changed.Load()) + require.True(t, unknown.Load()) + frontier.mu.Lock() + exact := frontier.exact + frontier.mu.Unlock() + require.False(t, exact, "unrelated failures must retain conservative invalidation") +} diff --git a/cmd/gortex/daemon_warmup_repo.go b/cmd/gortex/daemon_warmup_repo.go new file mode 100644 index 000000000..8eceabc7d --- /dev/null +++ b/cmd/gortex/daemon_warmup_repo.go @@ -0,0 +1,97 @@ +package main + +import ( + "context" + "errors" + "sync" + "sync/atomic" + + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/indexer" + "go.uber.org/zap" +) + +// warmupRepoAccounting points at the existing worker-loop counters. Keeping +// dispatch and its effects together makes a typed collision exit before any +// change/frontier/publication accounting in either the warm or cold branch. +type warmupRepoAccounting struct { + changed *atomic.Int64 + files *atomic.Int64 + unknown *atomic.Bool + prefixes *sync.Map + frontier *warmupDeltaFrontier +} + +func (a warmupRepoAccounting) load(ctx context.Context, state *daemonState, entry config.RepoEntry, priorMtimes map[string]int64, logger *zap.Logger) (string, error) { + pathFn := "track" + var res *indexer.IndexResult + var err error + if priorMtimes != nil { + pathFn = "reconcile" + res, err = state.multiIndexer.ReconcileRepoCtx(ctx, entry, priorMtimes) + } else { + res, err = state.multiIndexer.TrackRepoCtx(ctx, entry) + } + var collision *indexer.PrefixCollisionError + if errors.As(err, &collision) { + if pathFn == "track" { + logger.Warn("daemon: startup track rejected prefix collision", + zap.String("prefix", collision.Prefix), zap.String("owner", collision.ExistingPath), + zap.String("path", entry.Path), zap.String("action", "skipped"), zap.Error(err)) + } + return pathFn, &indexer.RepositoryCollisionSkip{Collision: collision} + } + + indexed := "" + if priorMtimes != nil { + switch { + case err != nil: + logger.Warn("daemon: startup reconcile failed", zap.String("path", entry.Path), zap.Error(err)) + // Other failures may have partially mutated the graph: preserve + // the existing conservative whole-workspace invalidation. + a.changed.Add(1) + a.unknown.Store(true) + a.frontier.invalidate() + case res != nil && (res.StaleFileCount > 0 || res.DeletedFileCount > 0 || len(res.FailedFiles) > 0 || res.FullRetrack): + a.changed.Add(1) + a.files.Add(int64(reconcileFileCount(res))) + a.frontier.record(res) + indexed = warmupRepoPrefix(state, entry, res.RepoPrefix) + if res.RepoPrefix != "" { + a.prefixes.Store(res.RepoPrefix, struct{}{}) + } else { + a.unknown.Store(true) + } + default: + // A genuine warm no-op registered this root. A rejected root + // already returned through the typed skip above. + prefix := "" + if res != nil { + prefix = res.RepoPrefix + } + indexed = warmupRepoPrefix(state, entry, prefix) + } + } else { + // A non-collision cold result retains the previous accounting: it + // performed a full index or failed with potentially partial work. + a.frontier.invalidate() + a.changed.Add(1) + switch { + case err != nil: + logger.Warn("daemon: startup track failed", zap.String("path", entry.Path), zap.Error(err)) + a.unknown.Store(true) + case res != nil && res.RepoPrefix != "": + a.files.Add(int64(res.FileCount)) + a.prefixes.Store(res.RepoPrefix, struct{}{}) + indexed = warmupRepoPrefix(state, entry, res.RepoPrefix) + default: + a.unknown.Store(true) + } + } + // Scheduling does no indexing here; the existing publisher drains only + // after readiness. A skipped root must never enqueue the owner's prefix. + if indexed != "" { + state.basePublisher.Schedule(indexed) + } + return pathFn, err +} diff --git a/cmd/gortex/ensure_daemon.go b/cmd/gortex/ensure_daemon.go index eb5af8f47..a62dc011e 100644 --- a/cmd/gortex/ensure_daemon.go +++ b/cmd/gortex/ensure_daemon.go @@ -24,16 +24,19 @@ const ( daemonUnavailable // autostart off, or spawn failed/timed out ) +// spawnLockInactivityTimeout is deliberately an inactivity budget, not a +// total startup budget. The lock winner holds the spawn lock until the +// detached child's socket is live; a schema migration can legitimately keep +// that socket closed for longer than a minute. Fresh PID-bound startup state +// extends this deadline in the same way spawnDetachedDaemon extends its own +// wait, while a missing/stale heartbeat still releases callers promptly. +// Tests shorten it only in an isolated helper process; there is no user +// configuration or environment override for this production budget. +var spawnLockInactivityTimeout = 60 * time.Second + const ( - // spawnLockInactivityTimeout is deliberately an inactivity budget, not a - // total startup budget. The lock winner holds the spawn lock until the - // detached child's socket is live; a schema migration can legitimately keep - // that socket closed for longer than a minute. Fresh PID-bound startup state - // extends this deadline in the same way spawnDetachedDaemon extends its own - // wait, while a missing/stale heartbeat still releases callers promptly. - spawnLockInactivityTimeout = 60 * time.Second - spawnLockPollInterval = 100 * time.Millisecond - spawnProgressFreshness = 10 * time.Second + spawnLockPollInterval = 100 * time.Millisecond + spawnProgressFreshness = 10 * time.Second // spawnFailCooldown bounds serial retries of a broken spawn: within // the window at most one spawn attempt runs across contending callers. spawnFailCooldown = 5 * time.Second diff --git a/cmd/gortex/migration_window_proxy_test.go b/cmd/gortex/migration_window_proxy_test.go new file mode 100644 index 000000000..3277f50a1 --- /dev/null +++ b/cmd/gortex/migration_window_proxy_test.go @@ -0,0 +1,348 @@ +package main + +import ( + "bytes" + "context" + "database/sql" + "encoding/json" + "fmt" + "os" + "os/exec" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "github.com/gofrs/flock" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "go.uber.org/zap" + + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/daemon" + "github.com/zzet/gortex/internal/graph/store_sqlite" + gortexmcp "github.com/zzet/gortex/internal/mcp" + "github.com/zzet/gortex/internal/testenv" +) + +const migrationProxyTestInactivity = 50 * time.Millisecond + +type migrationProxyWaitWitness struct { + At time.Time `json:"at"` + Checks int `json:"checks"` + LockContended bool `json:"lock_contended"` + State daemon.RuntimeState `json:"state"` +} + +type migrationProxyHelperInput struct { + Root string `json:"root"` + ParentPID int `json:"parent_pid"` + SpawnLock string `json:"spawn_lock"` + GlobalPath string `json:"global_config"` +} + +// The helper's TestMain installs a new sandbox before this test runs. Restore +// only these paths derived from its parent's fresh short fixture, not an +// inherited environment dump or any of the user's per-user state. +func migrationProxyOwnedEnv(root string) map[string]string { + home := filepath.Join(root, "home") + return map[string]string{ + "HOME": home, "USERPROFILE": home, + "AppData": filepath.Join(home, "AppData", "Roaming"), + "LocalAppData": filepath.Join(home, "AppData", "Local"), + "XDG_CONFIG_HOME": filepath.Join(root, "config"), + "XDG_DATA_HOME": filepath.Join(root, "data"), + "XDG_CACHE_HOME": filepath.Join(root, "cache"), + "XDG_STATE_HOME": filepath.Join(root, "state"), + "XDG_RUNTIME_DIR": filepath.Join(root, "runtime"), + "GORTEX_DAEMON_SOCKET": filepath.Join(root, "s"), + "GORTEX_DAEMON_PIDFILE": filepath.Join(root, "p"), + "GORTEX_DAEMON_STATEFILE": filepath.Join(root, "state.json"), + "GORTEX_DAEMON_LOGFILE": filepath.Join(root, "daemon.log"), + "GORTEX_DAEMON_SNAPSHOT": filepath.Join(root, "daemon.gob.gz"), + } +} + +// This helper runs the real MCP selection and proxy in a fresh process. Only +// that process shortens the private inactivity budget. Its liveness seam still +// returns the actual socket result; the extra lock/heartbeat reads only record +// that it reached the contended wait before the parent begins its hold. +func TestMigrationWindowMCPProxyHelper(t *testing.T) { + if os.Getenv("GORTEX_TEST_MIGRATION_PROXY") != "1" { + t.Skip("invoked only by TestMCPWaitsThroughMigrationWindow") + } + spawnLockInactivityTimeout = migrationProxyTestInactivity + inputPath := os.Getenv("GORTEX_TEST_MIGRATION_INPUT") + require.True(t, filepath.IsAbs(inputPath)) + body, err := os.ReadFile(inputPath) + require.NoError(t, err) + var input migrationProxyHelperInput + require.NoError(t, json.Unmarshal(body, &input)) + require.Equal(t, os.Getppid(), input.ParentPID) + require.True(t, filepath.IsAbs(input.Root)) + require.True(t, strings.HasPrefix(filepath.Base(input.Root), "gx")) + require.Equal(t, filepath.Join(input.Root, "migration-helper-input.json"), inputPath) + for name, value := range migrationProxyOwnedEnv(input.Root) { + t.Setenv(name, value) + } + require.Equal(t, input.SpawnLock, daemon.SpawnLockPath()) + require.Equal(t, input.GlobalPath, config.DefaultGlobalConfigPath()) + witnessDir := input.Root + checks := 0 + isDaemonRunning = func() bool { + running := daemon.IsRunning() + checks++ + if !running && checks >= 2 { + probe := flock.New(daemon.SpawnLockPath()) + locked, err := probe.TryLock() + require.NoError(t, err) + require.NoError(t, probe.Close()) + if locked { + return running // the parent may just have released startup + } + state, ok := daemon.ReadRuntimeState() + if !ok || state.StartupPhase != daemon.StartupMigrating { + return running // migration may have completed before Listen + } + require.True(t, state.StartupProgressFresh(time.Now(), spawnProgressFreshness)) + w := migrationProxyWaitWitness{At: time.Now(), Checks: checks, LockContended: true, State: state} + body, err := json.Marshal(w) + require.NoError(t, err) + path := filepath.Join(witnessDir, "latest.json") + if checks == 2 { + path = filepath.Join(witnessDir, "first.json") + } + require.NoError(t, os.WriteFile(path+".tmp", body, 0o600)) + require.NoError(t, os.Rename(path+".tmp", path)) + } + return running + } + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + mcpCmd.SetContext(ctx) + require.NoError(t, runMCP(mcpCmd, nil)) +} + +// This exercises an actual pending SQLite migration, heartbeat file, spawn +// lock, runMCP selection, stdio relay and public loading reply. The short hold +// proves renewal from a fresh PID-bound startup record, not the reporter's +// two-second periodic heartbeat. Historical schema-v21 workload acceptance is +// separate; this fixture deliberately replays the idempotent v40 step. +func TestMCPWaitsThroughMigrationWindow(t *testing.T) { + base := testenv.ShortTempDir(t) + // Keep this migration regression's config and actual child CWD under the + // same physical spelling. Alias admission is a separate unresolved case; + // /tmp versus /private/tmp must not conceal the migration wait contract. + base, err := filepath.EvalSymlinks(base) + require.NoError(t, err) + testenv.SandboxAt(t, filepath.Join(base, "home")) + for name, value := range migrationProxyOwnedEnv(base) { + t.Setenv(name, value) + } + t.Setenv("GORTEX_AUTOSTART", "1") + t.Setenv("GORTEX_TELEMETRY", "0") + t.Setenv("GORTEX_TOOLS", "") + root := filepath.Join(base, "repo") + require.NoError(t, os.MkdirAll(root, 0o755)) + require.NoError(t, os.WriteFile(filepath.Join(root, "main.go"), []byte("package p\nfunc MigrationProbe() {}\n"), 0o600)) + globalPath := config.DefaultGlobalConfigPath() + require.NoError(t, os.MkdirAll(filepath.Dir(globalPath), 0o700)) + // Opting into the fallback makes a premature embedded selection visible; + // it must not replace the configured daemon during an active migration. + require.NoError(t, os.WriteFile(globalPath, []byte(fmt.Sprintf("repos:\n - path: %q\n name: migrating\nmcp:\n allow_embedded: true\n", root)), 0o600)) + oldConfig, oldPath, oldBackend := cfgFile, daemonBackendPath, daemonBackend + t.Cleanup(func() { cfgFile, daemonBackendPath, daemonBackend = oldConfig, oldPath, oldBackend }) + cfgFile, daemonBackendPath, daemonBackend = filepath.Join(base, "repo.yaml"), filepath.Join(base, "store.sqlite"), "sqlite" + require.NoError(t, os.WriteFile(cfgFile, []byte("semantic:\n enabled: false\n"), 0o600)) + seed, err := store_sqlite.Open(daemonBackendPath) + require.NoError(t, err) + closeSeed := sync.OnceValue(seed.Close) + t.Cleanup(func() { _ = closeSeed() }) + require.NoError(t, closeSeed()) + db, err := sql.Open("sqlite", daemonBackendPath) + require.NoError(t, err) + t.Cleanup(func() { _ = db.Close() }) + _, err = db.Exec("PRAGMA user_version=39") + require.NoError(t, err) + require.NoError(t, db.Close()) + + lock := flock.New(daemon.SpawnLockPath()) + require.NoError(t, os.MkdirAll(filepath.Dir(daemon.SpawnLockPath()), 0o700)) + require.NoError(t, lock.Lock()) + t.Cleanup(func() { require.NoError(t, lock.Close()) }) + reporter := newDaemonStartupReporter(zap.NewNop()) + t.Cleanup(reporter.Stop) + entered, release := make(chan struct{}), make(chan struct{}) + var releaseOnce sync.Once + type buildResult struct { + state *daemonState + err error + } + built := make(chan buildResult, 1) + go func() { + state, buildErr := buildDaemonState(zap.NewNop(), func(p store_sqlite.MigrationProgress) { + reporter.ObserveMigration(p) + if p.Version == 40 && p.Phase == store_sqlite.MigrationStarted { + close(entered) + <-release + } + }) + built <- buildResult{state, buildErr} + }() + var result buildResult + buildJoined := false + t.Cleanup(func() { + releaseOnce.Do(func() { close(release) }) + if !buildJoined { + select { + case result = <-built: + case <-time.After(10 * time.Second): + t.Error("migration build did not join") + return + } + } + if result.state != nil { + require.NoError(t, result.state.shared.Close()) + } + }) + select { + case <-entered: + case <-time.After(10 * time.Second): + t.Fatal("the actual v40 migration did not start") + } + state, ok := daemon.ReadRuntimeState() + require.True(t, ok) + require.Equal(t, daemon.StartupMigrating, state.StartupPhase) + require.Equal(t, os.Getpid(), state.PID) + require.False(t, daemon.IsRunning(), "the real socket must be absent during migration") + + executable, err := os.Executable() + require.NoError(t, err) + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() + child := exec.CommandContext(ctx, executable, "-test.run=^TestMigrationWindowMCPProxyHelper$") + child.Dir = root + inputPath := filepath.Join(base, "migration-helper-input.json") + input, err := json.Marshal(migrationProxyHelperInput{ + Root: base, ParentPID: os.Getpid(), SpawnLock: daemon.SpawnLockPath(), GlobalPath: globalPath, + }) + require.NoError(t, err) + require.NoError(t, os.WriteFile(inputPath, input, 0o600)) + child.Env = append(os.Environ(), "GORTEX_TEST_MIGRATION_PROXY=1", "GORTEX_TEST_MIGRATION_INPUT="+inputPath) + stdin, err := child.StdinPipe() + require.NoError(t, err) + stdout, err := child.StdoutPipe() + require.NoError(t, err) + var stderr bytes.Buffer + child.Stderr = &stderr + require.NoError(t, child.Start()) + exited := make(chan error, 1) + go func() { exited <- child.Wait() }() + reaped := false + t.Cleanup(func() { + _ = stdin.Close() + cancel() + if !reaped { + select { + case <-exited: + case <-time.After(2 * time.Second): + t.Error("owned MCP helper did not reap") + } + } + }) + frames := streamProxyFrames(ctx, stdout) + _, err = fmt.Fprintln(stdin, `{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"migration-probe","version":"1"}}}`) + require.NoError(t, err) + var first migrationProxyWaitWitness + require.Eventually(t, func() bool { + body, readErr := os.ReadFile(filepath.Join(base, "first.json")) + return readErr == nil && json.Unmarshal(body, &first) == nil + }, 5*time.Second, 5*time.Millisecond) + require.True(t, first.LockContended) + require.Equal(t, os.Getpid(), first.State.PID) + require.Equal(t, daemon.StartupMigrating, first.State.StartupPhase) + // Time the hold only after an independent subprocess has witnessed both + // the actual failed flock and the actual active migration heartbeat. + select { + case frame := <-frames: + t.Fatalf("premature fallback/embedded output during migration: %s (%v)", frame.frame, frame.err) + case childErr := <-exited: + reaped = true + t.Fatalf("MCP exited during migration: %v", childErr) + case <-time.After(3 * migrationProxyTestInactivity): + } + var latest migrationProxyWaitWitness + body, err := os.ReadFile(filepath.Join(base, "latest.json")) + require.NoError(t, err) + require.NoError(t, json.Unmarshal(body, &latest)) + require.Greater(t, latest.Checks, first.Checks, "the real contended wait must continue after its initial inactivity budget") + require.True(t, latest.LockContended) + require.Equal(t, first.State.PID, latest.State.PID) + select { + case frame := <-frames: + t.Fatalf("MCP produced output before migration release: %s (%v)", frame.frame, frame.err) + default: + } + releaseOnce.Do(func() { close(release) }) + select { + case result = <-built: + buildJoined = true + case <-ctx.Done(): + t.Fatal("migration did not complete after release") + } + require.NoError(t, result.err) + srv := daemon.New(os.Getenv("GORTEX_DAEMON_SOCKET"), "test", zap.NewNop()) + srv.MCPDispatcher = newMCPDispatcher(result.state.mcpServer, result.state.multiIndexer, zap.NewNop()) + require.NoError(t, listenConfiguredDaemon(srv, result.state)) + reporter.Serving(daemonBackendPath) + done := make(chan error, 1) + go func() { done <- srv.Serve() }() + t.Cleanup(func() { + require.NoError(t, srv.Shutdown()) + select { + case serveErr := <-done: + require.NoError(t, serveErr) + case <-time.After(5 * time.Second): + t.Error("owned migration-window server did not join") + } + }) + require.NoError(t, lock.Unlock()) + readReply := func() []byte { + t.Helper() + select { + case frame := <-frames: + require.NoError(t, frame.err) + return frame.frame + case <-ctx.Done(): + t.Fatal("MCP reply deadline") + return nil + } + } + var initialized struct { + Result struct{ Instructions string } `json:"result"` + } + require.NoError(t, json.Unmarshal(readReply(), &initialized)) + assert.Contains(t, initialized.Result.Instructions, gortexmcp.ServerInstructionsLoading("migrating", "starting", 0)) + _, err = fmt.Fprintln(stdin, `{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"search_symbols","arguments":{"query":"MigrationProbe"}}}`) + require.NoError(t, err) + reply := readReply() + var parsed map[string]any + require.NoError(t, json.Unmarshal(reply, &parsed)) + data := errorData(t, parsed) + assert.Equal(t, "repo_loading", data["error_code"]) + assert.Equal(t, true, data["retryable"]) + assert.Contains(t, data, "phase") + assert.Contains(t, data, "percent") + assert.NotContains(t, string(reply), "gortex track") + assert.NotContains(t, string(reply), "repos reconcile") + require.NoError(t, stdin.Close()) + select { + case childErr := <-exited: + reaped = true + require.NoError(t, childErr, "%s", stderr.String()) + case <-ctx.Done(): + t.Fatal("MCP helper did not exit after closing stdio") + } + assert.NotContains(t, stderr.String(), "embedded") +} diff --git a/cmd/gortex/repo_inventory_loading_test.go b/cmd/gortex/repo_inventory_loading_test.go index 258448261..c1ba7e51e 100644 --- a/cmd/gortex/repo_inventory_loading_test.go +++ b/cmd/gortex/repo_inventory_loading_test.go @@ -18,11 +18,12 @@ func TestReconcileUnloadedRepos_Warmup(t *testing.T) { path := t.TempDir() entry := config.RepoEntry{Path: path, Name: "repo"} for _, tt := range []struct { - name string - ready bool - missing bool - loaded []daemon.TrackedRepoStatus - want []daemon.TrackedRepoStatus + name string + ready bool + missing bool + pathless bool + loaded []daemon.TrackedRepoStatus + want []daemon.TrackedRepoStatus }{ { name: "warming with existing path", @@ -44,17 +45,37 @@ func TestReconcileUnloadedRepos_Warmup(t *testing.T) { loaded: []daemon.TrackedRepoStatus{{Prefix: "worktree-instance", Path: path}}, }, { - name: "already loaded by prefix", + name: "another root with the same prefix remains visible", loaded: []daemon.TrackedRepoStatus{{Prefix: "repo", Path: filepath.Join(path, "worktree")}}, + want: []daemon.TrackedRepoStatus{{Prefix: "repo", Path: path, Name: "repo", + Workspace: "repo", WorkspaceProject: "repo", Unloaded: true, Loading: true}}, + }, + { + name: "legacy pathless configuration can match by prefix", + pathless: true, + loaded: []daemon.TrackedRepoStatus{{Prefix: "repo"}}, }, } { t.Run(tt.name, func(t *testing.T) { - got := reconcileUnloadedRepos([]config.RepoEntry{entry}, map[string]bool{path: tt.missing}, tt.loaded, tt.ready) + configured := entry + if tt.pathless { + configured.Path = "" + } + got := reconcileUnloadedRepos([]config.RepoEntry{configured}, map[string]bool{path: tt.missing}, tt.loaded, tt.ready) assert.Equal(t, tt.want, got) }) } } +func TestReconcileUnloadedReposKeepsACollidingRootVisible(t *testing.T) { + owner, loser := filepath.Join(t.TempDir(), "src"), filepath.Join(t.TempDir(), "src") + rows := reconcileUnloadedRepos([]config.RepoEntry{{Path: loser}}, map[string]bool{loser: false}, + []daemon.TrackedRepoStatus{{Path: owner, Prefix: "src"}}, true) + require.Len(t, rows, 1) + require.Equal(t, loser, rows[0].Path) + require.True(t, rows[0].Unloaded) +} + func TestControllerStatus_LoadingFollowsReadiness(t *testing.T) { path := t.TempDir() c := probeController(t, "repos:\n - path: "+path+"\n name: repo\n") diff --git a/cmd/gortex/repo_inventory_missing_test.go b/cmd/gortex/repo_inventory_missing_test.go index 53ff7d2cc..3fe0053c0 100644 --- a/cmd/gortex/repo_inventory_missing_test.go +++ b/cmd/gortex/repo_inventory_missing_test.go @@ -117,7 +117,14 @@ func newInventoryController(t *testing.T, repos []config.RepoEntry) *realControl reg := parser.NewRegistry() languages.RegisterAll(reg) mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, _ = mi.IndexAll() + mi.ExpectRepos(repos) + for _, entry := range repos { + mi.StartRepoLoad(entry.Path) + _, trackErr := mi.TrackRepoCtx(context.Background(), entry) + // Missing entries deliberately remain configured but unavailable; + // inventory/status assertions below must still report those entries. + mi.FinishRepoLoad(entry.Path, trackErr) + } return &realController{graph: g, multiIndexer: mi, configManager: cm, logger: zap.NewNop()} } diff --git a/cmd/gortex/track.go b/cmd/gortex/track.go index fa3d92629..4d2a1b42a 100644 --- a/cmd/gortex/track.go +++ b/cmd/gortex/track.go @@ -292,7 +292,8 @@ func runTrack(cmd *cobra.Command, args []string) error { return nil } var collision *trackPrefixCollisionError - if errors.As(notifyErr, &collision) { + var loading *repoLoadingError + if errors.As(notifyErr, &collision) || errors.As(notifyErr, &loading) { if !already { if err := removeAddedRepoEntry(absPath); err != nil { return fmt.Errorf("%w; removing rejected entry from config: %v", notifyErr, err) @@ -512,6 +513,9 @@ func notifyDaemonTrack(absPath, name string, deadline time.Time) error { return err } if !resp.OK { + if resp.ErrorCode == daemon.ErrRepoLoading { + return &repoLoadingError{admissionReason: "identity_pending", phase: "identity_resolution"} + } if resp.ErrorCode == daemon.ErrPrefixCollision { return &trackPrefixCollisionError{message: resp.ErrorMsg} } diff --git a/cmd/gortex/track_prefix_collision_test.go b/cmd/gortex/track_prefix_collision_test.go index 376baa8eb..0435035a0 100644 --- a/cmd/gortex/track_prefix_collision_test.go +++ b/cmd/gortex/track_prefix_collision_test.go @@ -65,6 +65,48 @@ func TestRunTrackPrefixCollision(t *testing.T) { } } +func TestRunTrackIdentityPendingRemovesOnlyItsNewEntry(t *testing.T) { + for _, wait := range []bool{false, true} { + for _, preexisting := range []bool{false, true} { + t.Run(fmt.Sprintf("wait=%v/preexisting=%v", wait, preexisting), func(t *testing.T) { + t.Setenv("XDG_CONFIG_HOME", t.TempDir()) + t.Setenv("GORTEX_DAEMON_SOCKET", filepath.Join(t.TempDir(), "absent.sock")) + root := pathkey.CanonicalExistingRoot(t.TempDir()) + gc, err := config.LoadGlobal() + require.NoError(t, err) + if preexisting { + require.NoError(t, gc.AddRepo(config.RepoEntry{Path: root, Name: "chosen"})) + } + require.NoError(t, gc.Save()) + oldWait, oldName, oldAsWorktree := trackWait, trackName, trackAsWorktree + oldEnsure, oldNotify := trackEnsureDaemonReadyFn, trackNotifyDaemonTrackFn + t.Cleanup(func() { + trackWait, trackName, trackAsWorktree = oldWait, oldName, oldAsWorktree + trackEnsureDaemonReadyFn, trackNotifyDaemonTrackFn = oldEnsure, oldNotify + }) + trackWait, trackName, trackAsWorktree = wait, "chosen", false + trackEnsureDaemonReadyFn = func(bool) daemonDecision { return daemonReady } + trackNotifyDaemonTrackFn = func(string, string, time.Time) error { + return &repoLoadingError{admissionReason: "identity_pending", phase: "identity_resolution"} + } + cmd := &cobra.Command{} + cmd.SetErr(&bytes.Buffer{}) + err = runTrack(cmd, []string{root}) + var pending *repoLoadingError + require.ErrorAs(t, err, &pending) + require.Equal(t, exitTempFail, pending.ExitCode()) + gc, err = config.LoadGlobal() + require.NoError(t, err) + if preexisting { + require.Len(t, gc.Repos, 1) + } else { + require.Empty(t, gc.Repos, "uncertainty cannot persist an unproved new physical-root entry") + } + }) + } + } +} + // The rollback removes only the entry this run added, never another // spelling of the same directory that was already in config. func TestRunTrackPrefixCollisionKeepsAliasEntry(t *testing.T) { diff --git a/docs/cli.md b/docs/cli.md index 8c67d4941..c0620031f 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -140,6 +140,17 @@ without changing config. Do not untrack the live owner just to resolve a duplicate name. If the owner's path no longer exists, inspect the untrack preview before confirming removal of its corpus. +Live collisions return `prefix_collision` and a non-zero exit with or without +`--wait`, including while another track holds the controller lock. The error +names the owner. A warmup loser does not change the owner's corpus, workspace +configuration, watcher or pending base publication. Probes report the collision +instead of presenting that root as loading or already tracked. + +An unresolved configured alias is not proof of a new root. If startup has not +finished resolving those identities, retry after that resolution completes; +Gortex leaves the tracking configuration unchanged rather than adding a second +entry for the same physical directory. + ## Per-repo setup ```bash diff --git a/docs/features.md b/docs/features.md index 0b5f62ea1..0b3cb6227 100644 --- a/docs/features.md +++ b/docs/features.md @@ -100,7 +100,7 @@ Five proactive push channels — per-session opt-in, delta-filtered, initial rep - **`notifications/diagnostics`** — LSP `publishDiagnostics` fan-out. Filter by `min_severity` / `path_prefix`. - **`notifications/workspace_readiness`** — daemon warmup phase transitions (snapshot_loaded → parallel_parse → deferred_passes_all → global_resolve → end_batch → watcher_started → ready). Late subscribers get the last-known phase replayed. -- **`notifications/daemon_health`** — periodic ticker (default 15 s, clampable 1 s..5 min). Snapshots uptime, alloc/sys/heap, num_goroutine, num_gc, tracked_repos, sessions, lsp_alive, graph nodes/edges. Only runs while ≥1 subscriber is attached. +- **`notifications/daemon_health`** — periodic ticker (default 15 s, clampable 1 s..5 min). Snapshots uptime, alloc/sys/heap, num_goroutine, num_gc, tracked_repos, sessions, lsp_alive, graph nodes/edges, live checkout coordinators, the process-wide SQLite page cache, and the OS view of the process (resident set; Linux proportional set and its file-backed part; macOS physical footprint; each omitted where the platform has no reader). Only runs while ≥1 subscriber is attached. - **`notifications/stale_refs`** — per-session intersect of watcher symbol-change events against the session's viewed/modified working set. Fires only when a change actually touches what *this* session has consumed. - **`notifications/graph_invalidated`** — coarse "the graph was rebuilt, drop cached results" signal. `{node_count, edge_count, reason, ts}`. Unfiltered. diff --git a/docs/mcp.md b/docs/mcp.md index a56f42aac..df13918d7 100644 --- a/docs/mcp.md +++ b/docs/mcp.md @@ -241,7 +241,7 @@ Four additional push channels modeled on `subscribe_diagnostics` — per-session |------|-------------| | `subscribe_workspace_readiness` | `notifications/workspace_readiness` — daemon warmup phase transitions (snapshot_loaded → parallel_parse → deferred_passes_all → global_resolve → end_batch → watcher_started → ready), plus one `{event:"repo_state", repo, repo_path, state:"ready"}` event each time a configured repository registers. Last-known phase replayed to late subscribers. A graph tool *called during warmup* does not need this subscription to cope: for a session whose repository is already registered it returns an in-band `warming` block plus best-effort partial results; for a session whose repository warmup has not registered yet it returns `repo_loading` (see [Startup states](#startup-states)) | | `unsubscribe_workspace_readiness` | Opt back out — idempotent | -| `subscribe_daemon_health` | `notifications/daemon_health` — periodic ticker (default 15 s, `interval_ms` clamped to 1 s..5 min) snapshots uptime, alloc/sys/heap, num_goroutine, num_gc, tracked_repos, sessions, lsp_alive, graph nodes/edges. Ticker only runs while ≥1 subscriber is attached | +| `subscribe_daemon_health` | `notifications/daemon_health` — periodic ticker (default 15 s, `interval_ms` clamped to 1 s..5 min) snapshots uptime, alloc/sys/heap, num_goroutine, num_gc, tracked_repos, sessions, lsp_alive, graph nodes/edges, live checkout `coordinators`, the process-wide SQLite page cache (`sqlite_page_cache_bytes`, `sqlite_page_cache_peak_bytes`) and the OS view of the process: resident set (`rss_bytes`, `rss_peak_bytes`), Linux proportional set (`pss_bytes`, `pss_file_bytes`), macOS physical footprint (`footprint_bytes`, `footprint_peak_bytes`); an OS figure is omitted where the platform has no reader. Ticker only runs while ≥1 subscriber is attached | | `unsubscribe_daemon_health` | Opt back out — idempotent | | `subscribe_stale_refs` | `notifications/stale_refs` — per-session intersect of watcher symbol-change events against the session's viewed/modified working set. Fires only when a change actually touches what *this* session has consumed | | `unsubscribe_stale_refs` | Opt back out — idempotent | @@ -255,13 +255,29 @@ After a daemon start, a repository is served once warmup registers it, which on | State | When | `tools/call` | `initialize` instructions | |---|---|---|---| | loading | configured, warmup job queued or running | `repo_loading`, `retryable:true`, with `phase`, `percent`, `repo`, `repo_path` | a LOADING preamble ahead of the ordinary instructions | +| prefix collision | the configured root's prefix belongs to another root | `prefix_collision`, `retryable:false`, with `prefix`, `owner_path`, `owner_state` | owner-specific naming guidance | | failed | the warmup job ended without registering it | `repo_unavailable`, `retryable:false`, with `reason` | how to repair it | | missing | the configured path does not exist | `repo_unavailable`, `retryable:false` | how to repair it | | ready | registered | served normally | ordinary | While loading, `index_health` (also `workspace(operation:"index")`), `capabilities`, `tools_search` and the `subscribe_*` / `unsubscribe_*` tools answer; `index_health` lists only the caller's repository. `track_repository`, `index_repository` and `workspace_admin(operation:"track")` for the repository answer `{status:"already_configured", state:"loading"}`; once its load has failed they run normally, because tracking it again is the repair. Every other tool call, `resources/read` and `prompts/get` returns `repo_loading`, and mutations stay refused. -Tools start working **in place** when the repository registers: no reconnect is needed. The session's cached scope is dropped at registration, and a session whose `initialize` reported the repository as loading receives one `notifications/message` saying it is ready. A directory outside every configured repository still gets `repo_not_tracked`. The control-socket probe that hooks and the CLI pre-flight read reports the same per-repository `state` (`loading`, `ready`, `failed`, `missing`), the hook lets native Read/Grep/Edit through for a loading repository instead of denying them, and the CLI maps `repo_loading` to "the daemon is still loading ": on a terminal it waits up to `--loading-wait` (default 30 s), otherwise it exits with code 75 (`EX_TEMPFAIL`). +Tools start working **in place** when the repository registers: no reconnect is needed. The session's cached scope is dropped at registration, and a session whose `initialize` reported the repository as loading receives one `notifications/message` saying it is ready. A directory outside every configured repository still gets `repo_not_tracked`. The control-socket probe that hooks and the CLI pre-flight read reports the same per-repository `state` (`loading`, `ready`, `failed`, `missing`, `prefix_collision`), the hook lets native Read/Grep/Edit through for a loading repository instead of denying them, and the CLI maps `repo_loading` to "the daemon is still loading ": on a terminal it waits up to `--loading-wait` (default 30 s), otherwise it exits with code 75 (`EX_TEMPFAIL`). + +A known prefix collision takes precedence over loading guidance: re-tracking +that configured root returns `prefix_collision`, never `already_configured`. +Use `name` for a new root, or edit the configured root's `name` and reload. +`owner_state` is `present`, `missing` or `unknown`; only `missing` suggests +inspecting an untrack of the owner. + +`track_repository` may answer `accepted` while its detached initial index +continues. Its final outcome is retained for ten minutes under the canonical +root and the request's `name`, `as_worktree` and `force` options. A retry reads +that outcome without starting another index. Collision outcomes are errors with +structured `error_code: prefix_collision`, `prefix`, `owner_path` and +`owner_state` fields. Changing the options, configured ownership or live owner +invalidates reuse, so choosing a different name can repair the request. Native +file reads and edits remain available while a collision prevents graph access. ## Request lifetime (why a call can never hang) diff --git a/docs/multi-repo.md b/docs/multi-repo.md index 259f5d57d..ed8722ddf 100644 --- a/docs/multi-repo.md +++ b/docs/multi-repo.md @@ -59,6 +59,16 @@ would disappear on restart. Set `name:` in the global config and run `gortex daemon reload`. Tracking a configured but unloaded root without a name keeps its configured name. +Collisions use the `prefix_collision` code across control, MCP and probe +responses, with the prefix, owner path and captured owner state. A missing +owner is different from an owner whose availability could not be established: +only a confirmed missing owner produces advice to inspect its untrack preview. +The colliding path is never a remedy for removing the owner's corpus. + +Same-directory spellings and same-folder-name symlink aliases remain no-ops. +Selected configured roots retain their pinned name and workspace; admission +does not scan unrelated configured filesystems. + Alternatively, `gortex untrack ` can remove an exact configured root that the daemon proves is not loaded, then reload the config. It does not untrack a containing live root or start retirement of an implicit live root. diff --git a/internal/config/manager.go b/internal/config/manager.go index a8d6162b3..7482297fc 100644 --- a/internal/config/manager.go +++ b/internal/config/manager.go @@ -44,6 +44,9 @@ type ConfigManager struct { lookupGlobal *GlobalConfig lookupRevision uint64 pathSnapshot *RepoPathSnapshot + trackGlobal *GlobalConfig + trackRevision uint64 + trackSnapshot *RepoTrackSnapshot } // NewConfigManager creates a ConfigManager by loading the GlobalConfig diff --git a/internal/config/track_lookup.go b/internal/config/track_lookup.go new file mode 100644 index 000000000..bfc1ad5f5 --- /dev/null +++ b/internal/config/track_lookup.go @@ -0,0 +1,81 @@ +package config + +import ( + "path/filepath" + "strings" + + "github.com/zzet/gortex/internal/pathkey" +) + +// RepoTrackSnapshot indexes admission candidates without canonicalizing every +// configured root. Identity remains the caller's responsibility: these lexical +// buckets are a filter, never evidence that two directories are the same. +type RepoTrackSnapshot struct { + byPrefix map[string][]RepoEntry + byBase map[string][]RepoEntry +} + +func (s *RepoTrackSnapshot) PrefixCandidates(prefix string) []RepoEntry { + return copyTrackEntries(s.byPrefix[prefix]) +} + +func (s *RepoTrackSnapshot) RootCandidates(root string) []RepoEntry { + return copyTrackEntries(s.byBase[pathkey.IdentityKey(filepath.Base(root))]) +} + +func copyTrackEntries(entries []RepoEntry) []RepoEntry { + out := append([]RepoEntry(nil), entries...) + for i := range out { + out[i].Exclude = append([]string(nil), out[i].Exclude...) + } + return out +} + +// RepoTrackSnapshot is revision-cached separately from the startup path +// snapshot. A track of one new root must not stat all configured roots first. +func (cm *ConfigManager) RepoTrackSnapshot() *RepoTrackSnapshot { + gc := cm.Global() + cm.lookupMu.Lock() + defer cm.lookupMu.Unlock() + globalConfigMu.Lock() + defer globalConfigMu.Unlock() + if cm.trackGlobal == gc && cm.trackRevision == gc.repoRevision && cm.trackSnapshot != nil { + return cm.trackSnapshot + } + s := &RepoTrackSnapshot{byPrefix: make(map[string][]RepoEntry), byBase: make(map[string][]RepoEntry)} + // Prefix ownership follows the existing top-level admission contract. + // Project-only entries still supply overrides through the root lookup. + for _, entry := range gc.Repos { + entry.Exclude = append([]string(nil), entry.Exclude...) + prefix := ResolvePrefix(entry) + s.byPrefix[prefix] = append(s.byPrefix[prefix], entry) + } + for _, entry := range gc.repoEntriesLocked() { + entry.Exclude = append([]string(nil), entry.Exclude...) + base := pathkey.IdentityKey(filepath.Base(entry.Path)) + s.byBase[base] = append(s.byBase[base], entry) + } + cm.trackGlobal, cm.trackRevision, cm.trackSnapshot = gc, gc.repoRevision, s + return s +} + +func (s *RepoTrackSnapshot) InstanceCandidates(prefix string) []RepoEntry { + out := s.PrefixCandidates(prefix) + // Basenames themselves may contain @. Each separator is a possible + // instancing boundary; look up only those finite lexical keys. + for offset := 0; offset < len(prefix); { + index := strings.IndexByte(prefix[offset:], '@') + if index < 0 { + break + } + index += offset + base := prefix[:index] + for _, entry := range copyTrackEntries(s.byPrefix[base]) { + if entry.Name == "" { + out = append(out, entry) + } + } + offset = index + 1 + } + return out +} diff --git a/internal/config/track_lookup_test.go b/internal/config/track_lookup_test.go new file mode 100644 index 000000000..ae6bc925e --- /dev/null +++ b/internal/config/track_lookup_test.go @@ -0,0 +1,29 @@ +package config + +import ( + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +func TestTrackCandidatesAreLexicalImmutableAndRevisionScoped(t *testing.T) { + cm, err := NewConfigManager(filepath.Join(t.TempDir(), "config.yaml")) + require.NoError(t, err) + root := filepath.Join(t.TempDir(), "offline", "src") // deliberately absent + require.NoError(t, cm.Global().AddRepo(RepoEntry{Path: root, Name: "owner", Exclude: []string{"kept"}})) + first := cm.RepoTrackSnapshot() + require.Same(t, first, cm.RepoTrackSnapshot()) + entry := first.RootCandidates(filepath.Join(t.TempDir(), "src")) + require.Len(t, entry, 1) + require.Equal(t, root, entry[0].Path) + entry[0].Name, entry[0].Exclude[0] = "changed", "changed" + require.Equal(t, "owner", first.PrefixCandidates("owner")[0].Name) + require.Equal(t, []string{"kept"}, first.RootCandidates(root)[0].Exclude) + require.Nil(t, cm.pathSnapshot, "admission must not build the eager canonical-root snapshot") + require.NoError(t, cm.Global().AddRepo(RepoEntry{Path: filepath.Join(t.TempDir(), "next")})) + require.NotSame(t, first, cm.RepoTrackSnapshot()) + require.Empty(t, first.PrefixCandidates("next"), "old snapshots stay immutable") + require.NoError(t, cm.Global().AddRepo(RepoEntry{Path: filepath.Join(t.TempDir(), "src@component")})) + require.Len(t, cm.RepoTrackSnapshot().InstanceCandidates("src@component@review"), 1) +} diff --git a/internal/daemon/control_identity_pending_test.go b/internal/daemon/control_identity_pending_test.go new file mode 100644 index 000000000..7e217b63d --- /dev/null +++ b/internal/daemon/control_identity_pending_test.go @@ -0,0 +1,26 @@ +package daemon + +import ( + "context" + "encoding/json" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/indexer" +) + +type identityPendingController struct{ fakeController } + +func (*identityPendingController) Track(context.Context, TrackParams) (json.RawMessage, error) { + return nil, indexer.ErrRepoIdentityPending +} + +func TestControlTrackIdentityPendingIsRetryable(t *testing.T) { + s := &Server{Controller: &identityPendingController{}} + params, err := json.Marshal(TrackParams{Path: "unresolved-root"}) + require.NoError(t, err) + response := s.handleControl(t.Context(), nil, ControlRequest{Kind: ControlTrack, Params: params}) + require.False(t, response.OK) + require.Equal(t, ErrRepoLoading, response.ErrorCode) + require.True(t, response.Retryable) +} diff --git a/internal/daemon/proto.go b/internal/daemon/proto.go index 6bbd9f604..1e02423bc 100644 --- a/internal/daemon/proto.go +++ b/internal/daemon/proto.go @@ -96,6 +96,7 @@ const ( ErrUnsupportedMode = "unsupported_mode" ErrRepoNotTracked = "repo_not_tracked" ErrPrefixCollision = "prefix_collision" + ErrRepoLoading = "repo_loading" ErrInternal = "internal" // ErrTimeout means the daemon accepted the control request but did not // finish it inside the kind's budget — almost always because a @@ -118,6 +119,7 @@ type ControlResponse struct { OK bool `json:"ok"` ErrorCode string `json:"error_code,omitempty"` ErrorMsg string `json:"error_msg,omitempty"` + Retryable bool `json:"retryable,omitempty"` Result json.RawMessage `json:"result,omitempty"` // Warnings report cleanup errors after shutdown was accepted. OK does not // certify store-close success or process exit; callers still wait for exit. @@ -316,6 +318,9 @@ type ProbeRepo struct { State string `json:"state,omitempty"` // StateReason explains a failed or missing repo. StateReason string `json:"state_reason,omitempty"` + ErrorCode string `json:"error_code,omitempty"` + OwnerPath string `json:"owner_path,omitempty"` + OwnerState string `json:"owner_state,omitempty"` } // Per-repo states a probe or coverage reply reports. @@ -323,12 +328,18 @@ const ( // RepoStateLoading: configured, and the daemon has not registered it // since it started. Graph calls answer repo_loading until it is ready. RepoStateLoading = "loading" + // RepoStateIdentityPending means configured identities are unresolved; it + // does not claim that the requested path belongs to a repository. + RepoStateIdentityPending = "identity_pending" // RepoStateReady: registered; graph calls answer. RepoStateReady = "ready" // RepoStateFailed: its load ended without registering it. RepoStateFailed = "failed" // RepoStateMissing: the configured path does not exist. RepoStateMissing = "missing" + // RepoStatePrefixCollision is a configured root whose prefix belongs to + // another root. It cannot gain admission by waiting for warmup. + RepoStatePrefixCollision = "prefix_collision" ) // StatusResponse is the payload returned under Result on a successful @@ -728,6 +739,19 @@ type RuntimeStats struct { StackInuse uint64 `json:"stack_inuse"` // goroutine stacks NumGC uint32 `json:"num_gc"` // completed GC cycles NumGoroutine int `json:"num_goroutine"` // live goroutines + + // What the Go runtime cannot see. Most of a daemon's footprint can be + // outside the Go heap: SQLite's page caches and parser allocations come + // from C allocators, and memory-mapped store files are file pages. The + // operating-system figures are omitted where the platform has no reader. + RSS uint64 `json:"rss,omitempty"` // resident set now + RSSPeak uint64 `json:"rss_peak,omitempty"` // lifetime maximum resident set + PSS uint64 `json:"pss,omitempty"` // Linux proportional set: shared pages divided among sharers + PSSFile uint64 `json:"pss_file,omitempty"` // file-backed part of PSS: mapped store files, executable + Footprint uint64 `json:"footprint,omitempty"` // macOS physical footprint now + FootprintPeak uint64 `json:"footprint_peak,omitempty"` // macOS lifetime maximum physical footprint + SQLitePageCache uint64 `json:"sqlite_page_cache,omitempty"` // every SQLite connection's page cache, process-wide + SQLitePageCachePeak uint64 `json:"sqlite_page_cache_peak,omitempty"` // lifetime maximum of the above } // TrigramCacheStats reports the process-wide trigram searcher cache: how @@ -1034,6 +1058,8 @@ type FileCoverageResult struct { // Graph calls for such a repo answer repo_loading, so a coverage verdict // read from its persisted rows must not turn into a deny of the native // tool: the agent would have no read path at all. Repo names it. + // RepoStateIdentityPending instead reports unresolved configured identities + // without asserting ownership of Path; Repo is empty and no graph is read. RepoState string `json:"repo_state,omitempty"` Repo string `json:"repo,omitempty"` // MutationState is read_only while the current checkout's exact route has @@ -1259,7 +1285,10 @@ type TrackedRepoStatus struct { // Loading is true when the repo is configured and its path exists, but // the daemon has not finished warmup, so its index is not loaded yet. // Unloaded remains true so older clients keep their existing behaviour. - Loading bool `json:"loading,omitempty"` + Loading bool `json:"loading,omitempty"` + ErrorCode string `json:"error_code,omitempty"` + OwnerPath string `json:"owner_path,omitempty"` + OwnerState string `json:"owner_state,omitempty"` // File indexing health includes unresolved failures and ledger read errors. IndexHealthError string `json:"index_health_error,omitempty"` FailedFiles int `json:"failed_files,omitempty"` diff --git a/internal/daemon/server.go b/internal/daemon/server.go index e45815cc1..0d5a95680 100644 --- a/internal/daemon/server.go +++ b/internal/daemon/server.go @@ -784,6 +784,9 @@ func (s *Server) handleControl(ctx context.Context, _ *Session, req ControlReque } result, err := s.Controller.Track(ctx, p) if err != nil { + if errors.Is(err, indexer.ErrRepoIdentityPending) { + return ControlResponse{ErrorCode: ErrRepoLoading, ErrorMsg: err.Error(), Retryable: true} + } var collision *indexer.PrefixCollisionError if errors.As(err, &collision) { return controlErr(ErrPrefixCollision, err.Error()) diff --git a/internal/graph/graph.go b/internal/graph/graph.go index d39ebbc10..3f930448b 100644 --- a/internal/graph/graph.go +++ b/internal/graph/graph.go @@ -465,10 +465,8 @@ func removeEdgeFromBucket(bucket map[string][]*Edge, keys map[string][]edgeHash, // constructed against this Graph. Per-shard locks protect individual // node/edge writes, but resolution phases (ResolveAll, ResolveFile) // iterate every shard and mutate edge targets in place — two -// resolvers racing on the same edge produced the data race that -// MultiIndexer.indexMultiRepo triggered when its per-repo goroutines -// each created their own Resolver. Routing every resolver through -// this single mutex serialises those phases without blocking +// per-repository resolvers can race on the same shared edge. Routing every +// resolver through this single mutex serialises those phases without blocking // ordinary shard-scoped reads and writes. type Graph struct { // shards holds the lock-sharded node/edge maps. Its length is diff --git a/internal/hooks/external_agent.go b/internal/hooks/external_agent.go index 14499af1d..ded8d7a63 100644 --- a/internal/hooks/external_agent.go +++ b/internal/hooks/external_agent.go @@ -99,6 +99,15 @@ func buildStaleIndexHint(cwd string) string { case err != nil: return "" } + if cwd != "" { + abs, err := filepath.Abs(cwd) + if err != nil { + abs = cwd + } + if collision := prefixCollisionCoverage(abs, status.TrackedRepos); collision != "" { + return "[Gortex] " + collision + } + } if !status.Ready { return fmt.Sprintf("[Gortex] daemon is warming up (%s elapsed); graph answers may be partial until it is ready.", formatDuration(status.WarmupSeconds)) } diff --git a/internal/hooks/pretooluse.go b/internal/hooks/pretooluse.go index 665c2dee5..987760724 100644 --- a/internal/hooks/pretooluse.go +++ b/internal/hooks/pretooluse.go @@ -1205,7 +1205,7 @@ func readOnlyNativeAllowance(writing bool) enrichResult { // hook lets it through because the path's repository is still loading. func repoLoadingNativeAllowance(repo string) string { if repo == "" { - repo = "this repository" + return "[Gortex] Graph admission for this path is still resolving and is not yet established (repo_loading, identity_pending). Native tools are allowed while this temporary state resolves. Retry shortly without changing tracking configuration.\n" } return fmt.Sprintf("[Gortex] %s is still loading after a restart; graph tools return repo_loading until it is ready. "+ "Native reads are allowed until then.\n", repo) @@ -1314,7 +1314,7 @@ func fileIndexStatusFromCoverage(result daemon.FileCoverageResult) fileIndexStat st.ProbeOK, st.ReadOnly, st.MutationKnown, st.MutationReason = true, true, true, result.MutationReason } st.MutationKnown = st.MutationKnown || result.MutationState == "ready" - if result.RepoState == daemon.RepoStateLoading { + if result.RepoState == daemon.RepoStateLoading || result.RepoState == daemon.RepoStateIdentityPending { // Coverage read from a loading repository's persisted rows is not a // reason to deny: report it as loading, never as indexed. st.ProbeOK, st.Loading, st.LoadingRepo = true, true, result.Repo @@ -1640,7 +1640,7 @@ var dirCoverageFn = dirCoverageViaDaemon // Only a completed walk that claimed nothing proves a scope holds no source. // From the graph alone a scope mid-walk looks exactly like an excluded one. func scopeTrackedFromCoverage(result daemon.DirCoverageResult) (hasSource, probeOK bool) { - if result.RepoState == daemon.RepoStateLoading { + if result.RepoState == daemon.RepoStateLoading || result.RepoState == daemon.RepoStateIdentityPending { // A loading repository's scope holds nothing the graph will answer // for yet, whatever its persisted rows say (see fileIndexStatus). return false, true diff --git a/internal/hooks/repo_loading_test.go b/internal/hooks/repo_loading_test.go index a3554a72d..21a6725e3 100644 --- a/internal/hooks/repo_loading_test.go +++ b/internal/hooks/repo_loading_test.go @@ -55,3 +55,31 @@ func TestLoadingRepo_SearchScopeAllowed(t *testing.T) { t.Fatalf("a loading repository's scope must read as holding nothing the graph answers for: hasSource=%v probeOK=%v", hasSource, probeOK) } } + +func TestIdentityPending_NativeDoorsDoNotClaimOwnership(t *testing.T) { + st := fileIndexStatusFromCoverage(daemon.FileCoverageResult{RepoState: daemon.RepoStateIdentityPending}) + if !st.Loading || st.Indexed || st.Tracked || st.LoadingRepo != "" { + t.Fatalf("uncertain identity is not an indexed or tracked verdict: %#v", st) + } + stubFileIndexScope(t, st) + for name, res := range map[string]enrichResult{ + "Read": enrichRead(map[string]any{"file_path": "internal/graph/graph.go"}, "/repo"), + "Edit": enrichEdit(map[string]any{"file_path": "internal/graph/graph.go"}, "/repo"), + "Bash cat": enrichBash(map[string]any{"command": "cat internal/graph/graph.go"}, "/repo"), + } { + if res.deny { + t.Fatalf("%s denied while graph admission is unresolved: %#v", name, res) + } + if strings.Contains(res.context, "this repository is still loading") || strings.Contains(res.context, "already configured") { + t.Fatalf("%s invented repository ownership: %s", name, res.context) + } + } + read := enrichRead(map[string]any{"file_path": "internal/graph/graph.go"}, "/repo") + if !strings.Contains(read.context, "identity_pending") { + t.Fatalf("missing honest reason: %s", read.context) + } + hasSource, ok := scopeTrackedFromCoverage(daemon.DirCoverageResult{RepoState: daemon.RepoStateIdentityPending}) + if hasSource || !ok { + t.Fatalf("pending directory must not redirect to graph: source=%v known=%v", hasSource, ok) + } +} diff --git a/internal/hooks/sessionstart.go b/internal/hooks/sessionstart.go index 65d81188e..8ff412b28 100644 --- a/internal/hooks/sessionstart.go +++ b/internal/hooks/sessionstart.go @@ -178,14 +178,17 @@ func sessionStartStatusFromProbe(probe daemon.ProbeResponse) *daemon.StatusRespo } for _, r := range probe.TrackedRepos { status.TrackedRepos = append(status.TrackedRepos, daemon.TrackedRepoStatus{ - Path: r.Path, - Prefix: r.Prefix, - Name: r.Name, - Workspace: r.Workspace, - Project: r.Project, - Loading: r.State == daemon.RepoStateLoading, - Unloaded: r.State == daemon.RepoStateLoading || r.State == daemon.RepoStateFailed || r.State == daemon.RepoStateMissing, - Missing: r.State == daemon.RepoStateMissing, + Path: r.Path, + Prefix: r.Prefix, + Name: r.Name, + Workspace: r.Workspace, + Project: r.Project, + Loading: r.State == daemon.RepoStateLoading, + Unloaded: r.State == daemon.RepoStateLoading || r.State == daemon.RepoStateFailed || r.State == daemon.RepoStateMissing || r.State == daemon.RepoStatePrefixCollision, + Missing: r.State == daemon.RepoStateMissing, + ErrorCode: r.ErrorCode, + OwnerPath: r.OwnerPath, + OwnerState: r.OwnerState, }) } return status @@ -328,6 +331,9 @@ func renderLeanReadiness(cwd string, s *daemon.StatusResponse) string { } } if abs != "" { + if collision := prefixCollisionCoverage(abs, s.TrackedRepos); collision != "" { + return line + "\n\n" + collision + } if exact, _ := classifyCwd(abs, s.TrackedRepos); exact != nil { return fmt.Sprintf("%s cwd tracked as `%s` — enforcement active.\n", line, trackedRepoName(*exact)) } @@ -381,6 +387,9 @@ func renderCwdCoverage(cwd string, s *daemon.StatusResponse) string { if err != nil { abs = cwd } + if collision := prefixCollisionCoverage(abs, s.TrackedRepos); collision != "" { + return collision + } exact, contained := classifyCwd(abs, s.TrackedRepos) switch { @@ -440,6 +449,25 @@ func renderCwdCoverage(cwd string, s *daemon.StatusResponse) string { } } +func prefixCollisionCoverage(cwd string, repos []daemon.TrackedRepoStatus) string { + var collision *daemon.TrackedRepoStatus + for i := range repos { + rec := &repos[i] + if rec.ErrorCode == daemon.ErrPrefixCollision && (pathkey.EqualPaths(cwd, rec.Path) || hasPathPrefix(cwd, rec.Path)) && + (collision == nil || len(rec.Path) > len(collision.Path)) { + collision = rec + } + } + if collision == nil { + return "" + } + remedy := "Set a unique name in the global config and run `gortex daemon reload`." + if collision.OwnerState == "missing" { + remedy = fmt.Sprintf("The owner no longer exists; inspect `gortex untrack %s` for that owner before retrying.", collision.OwnerPath) + } + return fmt.Sprintf("Gortex cannot admit `%s`: prefix_collision for `%s`, owned by `%s`. %s Native file reads and edits remain available.\n", collision.Path, collision.Prefix, collision.OwnerPath, remedy) +} + // trackedRepoName labels a tracked repo for display. The daemon leaves Name // empty unless the config entry names the repo, so fall back to its prefix, // then to the checkout directory's base name (#933). diff --git a/internal/hooks/worktree_guidance_test.go b/internal/hooks/worktree_guidance_test.go index c9c526c7d..625dd8e09 100644 --- a/internal/hooks/worktree_guidance_test.go +++ b/internal/hooks/worktree_guidance_test.go @@ -57,10 +57,10 @@ func TestDirectUnloadedFamilyPrimaryHooksKeepDiscoveryGuidance(t *testing.T) { } func TestSessionStartProbePreservesLoadingAndTerminalRegistrationState(t *testing.T) { - for _, state := range []string{daemon.RepoStateLoading, daemon.RepoStateFailed, daemon.RepoStateMissing, daemon.RepoStateReady, ""} { + for _, state := range []string{daemon.RepoStateLoading, daemon.RepoStateFailed, daemon.RepoStateMissing, daemon.RepoStatePrefixCollision, daemon.RepoStateReady, ""} { status := sessionStartStatusFromProbe(daemon.ProbeResponse{TrackedRepos: []daemon.ProbeRepo{{Path: "/fixture", State: state}}}) row := status.TrackedRepos[0] - if row.Loading != (state == daemon.RepoStateLoading) || row.Unloaded != (state == daemon.RepoStateLoading || state == daemon.RepoStateFailed || state == daemon.RepoStateMissing) || row.Missing != (state == daemon.RepoStateMissing) { + if row.Loading != (state == daemon.RepoStateLoading) || row.Unloaded != (state == daemon.RepoStateLoading || state == daemon.RepoStateFailed || state == daemon.RepoStateMissing || state == daemon.RepoStatePrefixCollision) || row.Missing != (state == daemon.RepoStateMissing) { t.Errorf("probe state %q lost terminal/loading distinction: %+v", state, row) } covered, _ := classifyCwd("/fixture", status.TrackedRepos) @@ -69,3 +69,24 @@ func TestSessionStartProbePreservesLoadingAndTerminalRegistrationState(t *testin } } } + +func TestPrefixCollisionHooksPreserveOwnerGuidance(t *testing.T) { + root, owner := filepath.Join(t.TempDir(), "src"), filepath.Join(t.TempDir(), "src") + status := sessionStartStatusFromProbe(daemon.ProbeResponse{TrackedRepos: []daemon.ProbeRepo{{ + Path: root, Prefix: "src", State: daemon.RepoStatePrefixCollision, + ErrorCode: daemon.ErrPrefixCollision, OwnerPath: owner, OwnerState: "present", + }}}) + oldStatus := sessionStartStatusFn + t.Cleanup(func() { sessionStartStatusFn = oldStatus }) + sessionStartStatusFn = func() (*daemon.StatusResponse, error) { return status, nil } + for name, text := range map[string]string{ + "session-start": renderCwdCoverage(root, status), + "lean-session-start": renderLeanReadiness(root, status), + "external-agent": buildStaleIndexHint(root), + } { + if !strings.Contains(text, "prefix_collision") || !strings.Contains(text, owner) || + strings.Contains(text, "untrack") || strings.Contains(text, "discovery is pending") || strings.Contains(text, "enforcement active") { + t.Errorf("%s lost collision owner/remedy: %s", name, text) + } + } +} diff --git a/internal/indexer/checkout_lifecycle.go b/internal/indexer/checkout_lifecycle.go index d9631964e..dc33102f2 100644 --- a/internal/indexer/checkout_lifecycle.go +++ b/internal/indexer/checkout_lifecycle.go @@ -386,6 +386,13 @@ type CheckoutLifecycle struct { refViewRetention RefViewRetention // indexBarrier is the promotion's test seam; nil in production. indexBarrier func() + // afterRegisterTrack is a test-only scheduling barrier, nil in production: + // registry ownership can disappear after a nil Track result and before + // checkout recording. It does not replace the real track implementation. + afterRegisterTrack func() + // afterPromotionIndex schedules an ownership change after the real index + // and before publication. Nil in production; it does not replace indexing. + afterPromotionIndex func() // routeBarrier stands in for the route withdrawal a promotion runs after // the mode flip, which is the one write no fixture can make the catalog // refuse. A test seam; nil in production. @@ -739,55 +746,40 @@ func (l *CheckoutLifecycle) Register( ctx context.Context, entry config.RepoEntry, source store_sqlite.IntentSourceKind, -) (RegisterResult, error) { +) (outcome RegisterResult, err error) { if l == nil || l.mi == nil { return RegisterResult{}, errors.New("indexer: checkout lifecycle is not wired") } - absPath, err := filepath.Abs(entry.Path) - if err != nil { - return RegisterResult{}, fmt.Errorf("resolve path %s: %w", entry.Path, err) - } - // Existing config is the restart authority. Do not accept a rename that - // AddRepo would discard, and keep a pinned name when warmup has not loaded it. - if l.cfgMgr != nil { - if configured := l.cfgMgr.RepoPathLookup()(absPath); configured != nil { - if entry.Name != "" && entry.Name != config.ResolvePrefix(*configured) { - return RegisterResult{}, fmt.Errorf("%s is already configured; set name: %s in the global config and run `gortex daemon reload`", absPath, entry.Name) - } - if entry.Name == "" { - entry.Name = configured.Name - } + defer func() { + var collision *PrefixCollisionError + if errors.As(err, &collision) { + l.mi.logCollision(collision) } + }() + entry, err = l.prepareRegistrationEntry(ctx, entry, time.Now().Add(prefixOwnershipTimeout)) + if err != nil { + return RegisterResult{}, err } - // One fan-out for the whole registration: the family reconciliation at the - // end drives the same cleanup hooks a sweep does, and each of them tells - // the sessions the tracked set moved. + absPath := entry.Path + // Coalesce notifications only after side-effect-free admission preparation. defer l.beginBatch()() - // A linked worktree of a family that already has a corpus is named by the - // rule at the naming seam rather than by its basename or its branch. The - // name is pinned onto the entry, so the indexer persists it and every later - // pass reads the decision back instead of taking it again. - if entry.Name == "" { - if prefix := l.dedicatedPrefixFor(ctx, absPath); prefix != "" { - entry.Name = prefix - } - } - // The source tells a config reload's re-registration from a user's track. result, err := l.mi.TrackRepoCtx(WithIndexOrigin(ctx, "checkout_register_"+string(source)), entry) if err != nil { return RegisterResult{}, err } + if l.afterRegisterTrack != nil { + l.afterRegisterTrack() + } out := RegisterResult{Index: result, AlreadyTracked: result == nil} - switch { - case result != nil && result.RepoPrefix != "": - out.Prefix = result.RepoPrefix - default: - out.Prefix = l.ResolvePrefix(absPath) + requested := config.ResolvePrefix(entry) + if result != nil && result.RepoPrefix != "" { + requested = result.RepoPrefix } - if out.Prefix == "" { - out.Prefix = config.ResolvePrefix(entry) + out.Prefix, err = l.mi.registeredPrefixForExactRoot(ctx, requested, absPath) + if err != nil { + return RegisterResult{}, err } l.implicitRoots.Delete(out.Prefix) @@ -821,14 +813,28 @@ func (l *CheckoutLifecycle) Register( // the next boot's seeding would read back as explicit configuration and mint a // manual_config intent for, turning an intent-less observation into intent one // restart later. -func (l *CheckoutLifecycle) RecordImplicit(ctx context.Context, root string) error { +func (l *CheckoutLifecycle) RecordImplicit(ctx context.Context, root string) (err error) { if l == nil || l.mi == nil { return nil } - prefix := l.ResolvePrefix(root) - if prefix == "" { - return fmt.Errorf("%w: %s", ErrCheckoutNotTracked, root) + defer func() { + var collision *PrefixCollisionError + if errors.As(err, &collision) { + l.mi.logCollision(collision) + } + }() + if root == "" { + return fmt.Errorf("%w: empty implicit repository root", ErrCheckoutNotTracked) + } + abs, err := filepath.Abs(root) + if err != nil { + return fmt.Errorf("resolve implicit root %s: %w", root, err) + } + prefix, err := l.mi.registeredPrefixForExactRoot(ctx, config.ResolvePrefix(config.RepoEntry{Path: abs}), abs) + if err != nil { + return err } + root = abs defer l.beginBatch()() identity, err := l.recordCheckout(ctx, prefix, root, TrackSourceImplicit, false) diff --git a/internal/indexer/checkout_prefix_registration_test.go b/internal/indexer/checkout_prefix_registration_test.go new file mode 100644 index 000000000..b1c297707 --- /dev/null +++ b/internal/indexer/checkout_prefix_registration_test.go @@ -0,0 +1,96 @@ +package indexer + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestRegisterCollisionNoCatalogNoWatcherNoSave(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + root := f.gitRepo("src") + indexed, err := f.mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: root, Name: "src"}) + require.NoError(t, err) + require.NotNil(t, indexed) + before, err := os.ReadFile(f.cfgPath) + require.NoError(t, err) + beforeConfig, beforeRevision := f.cm.GetRepoConfig("src"), f.cm.Revision() + beforePrefixes := f.cm.WorkspacePrefixes() + beforeInvalidated, beforeAnalysed := f.notify.counts() + // Exercise a real nil track result, then withdraw its registry witness at + // the recording boundary. A previous successful no-op is not authority to + // recreate catalog/watcher/config state after ownership has disappeared. + f.lc.afterRegisterTrack = func() { + f.mi.mu.Lock() + delete(f.mi.repos, "src") + f.mi.mu.Unlock() + } + result, err := f.lc.Register(t.Context(), config.RepoEntry{Path: root, Name: "src"}, TrackSourceCLI) + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision) + require.Equal(t, "src", collision.Prefix) + require.Equal(t, "unknown", collision.OwnerState) + require.Empty(t, collision.ExistingPath, "unknown ownership must not invent another root") + require.Empty(t, result.Prefix) + families, err := f.catalog.ListRepositoryFamilies(t.Context()) + require.NoError(t, err) + require.Empty(t, families, "registration must stop before catalog allocation") + after, err := os.ReadFile(f.cfgPath) + require.NoError(t, err) + require.Equal(t, before, after) + require.Equal(t, beforeConfig, f.cm.GetRepoConfig("src")) + require.Equal(t, beforeRevision, f.cm.Revision(), "nil track must not publish a prepared candidate config") + require.Equal(t, beforePrefixes, f.cm.WorkspacePrefixes()) + f.watcher.mu.Lock() + calls := append([]string(nil), f.watcher.calls...) + f.watcher.mu.Unlock() + require.Empty(t, calls) + invalidated, analysed := f.notify.counts() + require.Equal(t, beforeInvalidated, invalidated) + require.Equal(t, beforeAnalysed, analysed) +} + +func TestRecordImplicitRequiresExactRegisteredRoot(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + require.ErrorIs(t, f.lc.RecordImplicit(t.Context(), ""), ErrCheckoutNotTracked, + "an empty implicit root must not silently resolve to the process cwd") + owner := f.gitRepo("owner") + _, err := f.lc.Register(t.Context(), config.RepoEntry{Path: owner}, TrackSourceCLI) + require.NoError(t, err) + child := filepath.Join(owner, "nested") + require.NoError(t, os.MkdirAll(child, 0o700)) + require.Equal(t, "owner", f.lc.ResolvePrefix(child), "containment remains useful for reads") + before, err := os.ReadFile(f.cfgPath) + require.NoError(t, err) + graphs, err := f.catalog.ListDedicatedGraphs(t.Context(), f.familyOf("owner").FamilyID) + require.NoError(t, err) + nodes, edges := f.store.NodeCount(), f.store.EdgeCount() + f.watcher.mu.Lock() + watcherCalls := append([]string(nil), f.watcher.calls...) + f.watcher.mu.Unlock() + invalidated, analysed := f.notify.counts() + err = f.lc.RecordImplicit(t.Context(), child) + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision, "an ancestor prefix is not exact registration authority") + require.Equal(t, "unknown", collision.OwnerState) + require.Equal(t, nodes, f.store.NodeCount()) + require.Equal(t, edges, f.store.EdgeCount()) + afterGraphs, err := f.catalog.ListDedicatedGraphs(t.Context(), f.familyOf("owner").FamilyID) + require.NoError(t, err) + require.Equal(t, graphs, afterGraphs) + after, err := os.ReadFile(f.cfgPath) + require.NoError(t, err) + require.Equal(t, before, after) + f.watcher.mu.Lock() + afterWatcherCalls := append([]string(nil), f.watcher.calls...) + f.watcher.mu.Unlock() + require.Equal(t, watcherCalls, afterWatcherCalls) + afterInvalidated, afterAnalysed := f.notify.counts() + require.Equal(t, invalidated, afterInvalidated) + require.Equal(t, analysed, afterAnalysed) +} diff --git a/internal/indexer/checkout_promote.go b/internal/indexer/checkout_promote.go index bdfe4d337..4d01e0e80 100644 --- a/internal/indexer/checkout_promote.go +++ b/internal/indexer/checkout_promote.go @@ -57,11 +57,10 @@ type PromoteResult struct { // the checkout exists does anything the query surface reads change — the mode // flips to dedicated and the automatic route and coordinator are retired. // -// Every failure before that flip leaves the automatic view serving. The corpus -// half-built for the new prefix is evicted, the graph row that named it is -// dropped, and the journalled intent stays pending, which is what Retryable -// means: nothing about the checkout changed, so asking again is the whole of -// the recovery. +// Every failure before that flip leaves the automatic view serving. Cleanup +// evicts only the witnessed registration and its own unpublished graph row; +// replaced or unknown ownership is retained rather than purged by prefix. The +// journalled intent stays pending, which is what Retryable means. func (l *CheckoutLifecycle) PromoteCheckout( ctx context.Context, checkoutID string, source store_sqlite.IntentSourceKind, ) (PromoteResult, error) { @@ -202,12 +201,12 @@ func (l *CheckoutLifecycle) promoteCheckoutTransition( return out, l.promotionFailed(ctx, &out, transition, fmt.Errorf("indexer: no dedicated prefix can be derived for %s", checkout.RootPath)) } - index, resampled, err := l.indexPromotedCorpus(ctx, checkout, out.Prefix) + index, resampled, owned, err := l.indexPromotedCorpus(ctx, checkout, out.Prefix) out.Index, out.Resampled = index, resampled if err != nil { var collision *PrefixCollisionError if !errors.As(err, &collision) { - l.rollbackPromotionOrLog(ctx, out.Prefix, "") + err = errors.Join(err, l.rollbackPromotionOrLog(ctx, out.Prefix, "", checkout, owned)) } return out, l.promotionFailed(ctx, &out, transition, err) } @@ -221,12 +220,11 @@ func (l *CheckoutLifecycle) promoteCheckoutTransition( IsPrimaryBase: false, State: reconcile.GraphStateReady, } - if err := l.catalog.UpsertDedicatedGraph(ctx, row); err != nil { - l.rollbackPromotionOrLog(ctx, out.Prefix, "") - return out, l.promotionFailed(ctx, &out, transition, err) + if l.afterPromotionIndex != nil { + l.afterPromotionIndex() } - if err := l.serveFromOwnCorpus(ctx, checkout); err != nil { - l.rollbackPromotionOrLog(ctx, out.Prefix, out.GraphID) + if err := l.publishPromotedCorpus(ctx, checkout, row, owned); err != nil { + err = errors.Join(err, l.rollbackPromotionOrLog(ctx, out.Prefix, out.GraphID, checkout, owned)) return out, l.promotionFailed(ctx, &out, transition, err) } @@ -240,6 +238,47 @@ func (l *CheckoutLifecycle) promoteCheckoutTransition( return out, nil } +// publishPromotedCorpus retains the existing repository mutation lane across +// the owner recheck and publication. Untrack/retrack cannot substitute a new +// corpus between that check and the mode flip. No filesystem work or catalog +// call runs under mi.mu. +func (l *CheckoutLifecycle) publishPromotedCorpus(ctx context.Context, checkout store_sqlite.Checkout, row store_sqlite.DedicatedGraph, owned prefixOwnerWitness) error { + l.mi.mu.RLock() + err := l.mi.validatePrefixCleanupOwnerLocked(row.RepoPrefix, &owned) + idx := l.mi.indexers[row.RepoPrefix] + l.mi.mu.RUnlock() + if err != nil { + return err + } + lane, current := l.mi.repositoryMutationCoordinatorForSnapshot(row.RepoPrefix, owned.meta, idx) + if !current || lane == nil { + return errPrefixOwnerChanged + } + return lane.runExclusive(ctx, func() error { + l.mi.mu.RLock() + err := l.mi.validatePrefixCleanupOwnerLocked(row.RepoPrefix, &owned) + l.mi.mu.RUnlock() + if err != nil { + return err + } + return idx.withOutputGeneration(ctx, OutputEntryPromotionPublication, func() error { + // Legacy unpublished catalog rows allow rebinding. Promotion cannot use + // that compatibility behavior to overwrite a different checkout owner. + binding, found, err := l.catalog.GetDedicatedGraph(ctx, row.GraphID) + if err != nil { + return err + } + if found && (binding.OwnerCheckoutID != checkout.CheckoutID || binding.FamilyID != checkout.FamilyID) { + return fmt.Errorf("%w: promotion graph %s belongs to another checkout", store_sqlite.ErrCatalogStaleGuard, row.GraphID) + } + if err := l.catalog.UpsertDedicatedGraph(ctx, row); err != nil { + return err + } + return l.serveFromOwnCorpus(ctx, checkout) + }) + }) +} + // checkoutSample is the state a promotion has to describe: what the checkout // is committed at, and what its working tree holds on top. type checkoutSample struct { @@ -276,37 +315,48 @@ func sampleCheckout(ctx context.Context, root string) (checkoutSample, error) { // automatic view is still there to fall back on. func (l *CheckoutLifecycle) indexPromotedCorpus( ctx context.Context, checkout store_sqlite.Checkout, prefix string, -) (*IndexResult, int, error) { +) (*IndexResult, int, prefixOwnerWitness, error) { resampled := 0 + var owned prefixOwnerWitness for attempt := 0; attempt < 2; attempt++ { before, err := sampleCheckout(ctx, checkout.RootPath) if err != nil { - return nil, resampled, err + return nil, resampled, owned, err } if l.indexBarrier != nil { l.indexBarrier() } result, err := l.mi.TrackRepoCtx(WithIndexOrigin(ctx, "checkout_promotion"), config.RepoEntry{Path: checkout.RootPath, Name: prefix}) if err != nil { - return nil, resampled, err + // Another registration may have claimed this exact root under a + // different name after promotion selected its prefix. Configured + // name validation then refuses Track before its normal ownership + // check. Preserve the typed collision for this established owner; + // neither the requested corpus nor its rollback is authorized. + observation, ownerErr := l.mi.observePrefixOwnership(ctx, prefix, checkout.RootPath) + if ownerErr == nil && observation.registeredPrefix != "" && observation.registeredPrefix != prefix { + return nil, resampled, owned, newObservedPrefixCollision(prefix, checkout.RootPath, checkout.RootPath, observation.candidate) + } + return nil, resampled, owned, err } - if result == nil && l.mi.GetMetadata(prefix) == nil { - return nil, resampled, fmt.Errorf( - "indexer: %s could not be indexed under prefix %s", checkout.RootPath, prefix) + owned, err = l.mi.prefixOwnerForRoot(ctx, prefix, checkout.RootPath) + if err != nil { + return nil, resampled, owned, err } after, err := sampleCheckout(ctx, checkout.RootPath) if err != nil { - return nil, resampled, err + return nil, resampled, owned, err } if after == before { - return result, resampled, nil + return result, resampled, owned, nil } resampled++ - if _, _, err := l.mi.UntrackRepoChecked(ctx, prefix); err != nil { - return nil, resampled, err + if _, _, err := l.mi.purgeRepoOwnedChecked(ctx, owned, nil); err != nil { + return nil, resampled, owned, err } + owned = prefixOwnerWitness{} } - return nil, resampled, fmt.Errorf("%w: %s moved under two full indexes", + return nil, resampled, owned, fmt.Errorf("%w: %s moved under two full indexes", ErrCheckoutMoved, checkout.RootPath) } @@ -373,31 +423,62 @@ func (l *CheckoutLifecycle) withdrawAutomaticRoute(ctx context.Context, checkout // rollbackPromotion undoes what a failed promotion built. The automatic view // was never touched, so putting the corpus and the graph row back the way they // were is the whole of it. -func (l *CheckoutLifecycle) rollbackPromotion(ctx context.Context, prefix, graphID string) error { - if prefix != "" { - l.detachWatcher(prefix) - if _, _, err := l.mi.purgeRepoChecked(ctx, prefix, nil); err != nil { - return fmt.Errorf("indexer: purge rolled-back repository %q: %w", prefix, err) +func (l *CheckoutLifecycle) rollbackPromotion(ctx context.Context, prefix, graphID string, checkout store_sqlite.Checkout, owned prefixOwnerWitness) error { + if prefix == "" || owned.meta == nil || owned.prefix != prefix { + return errPrefixIdentityUnknown + } + // Even a failure before our Upsert must not purge a corpus now bound to + // another checkout/family. Read the actual row; never infer ownership from + // its graph id or the class of the primary failure. + binding, found, err := l.catalog.GetDedicatedGraph(ctx, GraphIDFor(prefix)) + if err != nil { + return err + } + if found { + if binding.OwnerCheckoutID != checkout.CheckoutID || binding.FamilyID != checkout.FamilyID { + owner, ownerFound, ownerErr := l.catalog.GetCheckout(ctx, binding.OwnerCheckoutID) + if ownerErr != nil { + return ownerErr + } + root := "" + if ownerFound { + root = owner.RootPath + } + return newObservedPrefixCollision(prefix, root, checkout.RootPath, prefixStatResult{err: errPrefixOwnerChanged}) + } + _, published, err := l.catalog.DedicatedBasePublication(ctx, binding.GraphID) + if err != nil { + return err + } + if published { + return fmt.Errorf("%w: promotion rollback refuses published graph %s", store_sqlite.ErrCatalogStaleGuard, binding.GraphID) } } - if graphID != "" { - if err := l.catalog.DeleteDedicatedGraph(ctx, graphID); err != nil && - !errors.Is(err, store_sqlite.ErrCatalogNotFound) { - return fmt.Errorf("indexer: drop rolled-back graph binding %q: %w", graphID, err) + if graphID != "" && graphID != GraphIDFor(prefix) { + return fmt.Errorf("%w: promotion rollback graph/prefix mismatch", store_sqlite.ErrCatalogStaleGuard) + } + if _, _, err := l.mi.purgeRepoOwnedChecked(ctx, owned, func() { l.detachWatcher(prefix) }); err != nil { + return fmt.Errorf("indexer: purge owned rolled-back repository %q: %w", prefix, err) + } + if graphID != "" && found { + // This guarded delete compares the complete row and refuses publication + // acquired after the observation. It never deletes a replacement binding. + if err := l.catalog.DeleteUnpublishedDedicatedGraph(ctx, binding); err != nil { + return fmt.Errorf("indexer: drop owned rolled-back graph binding %q: %w", graphID, err) } } return nil } -// rollbackPromotionOrLog performs the best-effort rollback of a failed -// promotion and only logs any cleanup error: the caller is already returning -// the promotion failure that triggered the rollback, so a cleanup error is -// secondary and has nothing to act on beyond the record. -func (l *CheckoutLifecycle) rollbackPromotionOrLog(ctx context.Context, prefix, graphID string) { - if err := l.rollbackPromotion(ctx, prefix, graphID); err != nil { +// rollbackPromotionOrLog retains cleanup errors alongside the primary failure. +// Unknown/replaced ownership is a refusal to purge, never a successful cleanup. +func (l *CheckoutLifecycle) rollbackPromotionOrLog(ctx context.Context, prefix, graphID string, checkout store_sqlite.Checkout, owned prefixOwnerWitness) error { + err := l.rollbackPromotion(ctx, prefix, graphID, checkout, owned) + if err != nil { l.logger.Warn("checkout lifecycle: could not roll back a failed promotion", - zap.String("prefix", prefix), zap.String("graph", graphID), zap.Error(err)) + zap.String("prefix", prefix), zap.String("root", checkout.RootPath), zap.String("graph", graphID), zap.Error(err)) } + return err } // beginModeChange journals a mode change, adopting the entry an interrupted diff --git a/internal/indexer/checkout_promotion_ownership_test.go b/internal/indexer/checkout_promotion_ownership_test.go new file mode 100644 index 000000000..37f685c8c --- /dev/null +++ b/internal/indexer/checkout_promotion_ownership_test.go @@ -0,0 +1,182 @@ +package indexer + +import ( + "os" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/graph/store_sqlite" + "github.com/zzet/gortex/internal/reconcile" +) + +func TestPromotionCollisionKeepsOtherCorpus(t *testing.T) { + for _, replacement := range []string{"registry", "catalog-authority"} { + t.Run(replacement, func(t *testing.T) { + f := newFamilyFixture(t, "late-collision") + t.Cleanup(f.close) + prefix := f.mainPrefix + "@" + f.automatic.AdminName + other := f.gitRepo("other-owner") + var binding store_sqlite.DedicatedGraph + var configBytes []byte + var watcherCalls []string + var nodes, edges int + f.lc.afterPromotionIndex = func() { + // Use real teardown and registration, not a forged registry entry. + // The catalog-only case leaves the indexed registration in place + // but gives the graph real authority owned by another checkout. + name := "other" + if replacement == "registry" { + _, _, err := f.mi.UntrackRepoChecked(t.Context(), prefix) + require.NoError(t, err) + require.NoError(t, f.cm.Global().RemoveRepo(f.automatic.RootPath)) + name = prefix + } + registered, err := f.lc.Register(t.Context(), config.RepoEntry{Path: other, Name: name}, TrackSourceCLI) + require.NoError(t, err) + require.NoError(t, registered.CatalogErr) + var found bool + binding, found, err = f.catalog.GetDedicatedGraph(t.Context(), registered.GraphID) + require.NoError(t, err) + require.True(t, found) + if replacement == "catalog-authority" { + require.NoError(t, f.catalog.DeleteDedicatedGraph(t.Context(), binding.GraphID)) + binding.GraphID, binding.RepoPrefix = GraphIDFor(prefix), prefix + require.NoError(t, f.catalog.UpsertDedicatedGraph(t.Context(), binding)) + guardConfiguredOwner(t, f.lifecycleFixture, prefix) + } + configBytes, err = os.ReadFile(f.cfgPath) + require.NoError(t, err) + f.watcher.mu.Lock() + watcherCalls = append([]string(nil), f.watcher.calls...) + f.watcher.mu.Unlock() + nodes, edges = f.store.NodeCount(), f.store.EdgeCount() + } + transition, err := f.lc.beginModeChange(t.Context(), f.automatic, + store_sqlite.CheckoutModeDedicated, promotionTransitionCause, nil) + require.NoError(t, err) + // Run the real durable transition on the test goroutine so a failed + // fixture assertion inside the scheduling hook cannot strand a waiter. + outcome := f.lc.executeModeTransition(t.Context(), transition) + out, err := outcome.promotion, outcome.err + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision) + require.True(t, out.Retryable) + require.Equal(t, prefix, collision.Prefix) + if replacement == "catalog-authority" { + require.ErrorIs(t, err, store_sqlite.ErrCatalogStaleGuard) + _, published, err := f.catalog.DedicatedBasePublication(t.Context(), binding.GraphID) + require.NoError(t, err) + require.True(t, published, "rollback must retain the replacement authority") + } else { + require.Equal(t, other, f.mi.GetMetadata(prefix).RootPath) + } + afterBinding, found, err := f.catalog.GetDedicatedGraph(t.Context(), binding.GraphID) + require.NoError(t, err) + require.True(t, found) + require.Equal(t, binding, afterBinding) + require.Equal(t, nodes, f.store.NodeCount()) + require.Equal(t, edges, f.store.EdgeCount()) + require.NotEmpty(t, f.store.GetRepoNodes(prefix)) + afterConfig, err := os.ReadFile(f.cfgPath) + require.NoError(t, err) + require.Equal(t, configBytes, afterConfig) + f.watcher.mu.Lock() + afterCalls := append([]string(nil), f.watcher.calls...) + f.watcher.mu.Unlock() + require.Equal(t, watcherCalls, afterCalls, "failed promotion must not detach the replacement watcher") + checkout, found, err := f.catalog.GetCheckout(t.Context(), f.automatic.CheckoutID) + require.NoError(t, err) + require.True(t, found) + require.Equal(t, store_sqlite.CheckoutModeAutomatic, checkout.EffectiveMode) + }) + } +} + +func TestPromotionPurgeRejectsAReplacedRegistration(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + first, other := f.gitRepo("first"), f.gitRepo("other") + _, err := f.lc.Register(t.Context(), config.RepoEntry{Path: first, Name: "src"}, TrackSourceCLI) + require.NoError(t, err) + owned, err := f.mi.prefixOwnerForRoot(t.Context(), "src", first) + require.NoError(t, err) + // Retire through the lifecycle so the exact old reader/publisher admission + // is finalized before installing a replacement. Deleting only the registry + // and catalog row must not bypass that owner's admission tombstone. + retired, err := f.lc.Untrack(t.Context(), first) + require.NoError(t, err) + if retired.Pending { + finishRepositoryCleanup(t, f.lc, "src") + } + _, found, err := f.catalog.GetDedicatedGraph(t.Context(), GraphIDFor("src")) + require.NoError(t, err) + require.False(t, found) + registered, err := f.lc.Register(t.Context(), config.RepoEntry{Path: other, Name: "src"}, TrackSourceCLI) + require.NoError(t, err) + require.NoError(t, registered.CatalogErr) + nodes, edges := f.store.NodeCount(), f.store.EdgeCount() + stopped := false + _, _, err = f.mi.purgeRepoOwnedChecked(t.Context(), owned, func() { stopped = true }) + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision) + require.False(t, stopped) + require.Equal(t, other, f.mi.GetMetadata("src").RootPath) + require.True(t, f.watcher.isAttached("src")) + require.Equal(t, nodes, f.store.NodeCount()) + require.Equal(t, edges, f.store.EdgeCount()) + require.NotEmpty(t, f.store.GetRepoNodes("src")) +} + +func TestPromotionRollbackRemovesOnlyItsUnpublishedCorpus(t *testing.T) { + f := newFamilyFixture(t, "owned-rollback") + t.Cleanup(f.close) + prefix := f.mainPrefix + "@" + f.automatic.AdminName + _, err := f.mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: f.automatic.RootPath, Name: prefix}) + require.NoError(t, err) + owned, err := f.mi.prefixOwnerForRoot(t.Context(), prefix, f.automatic.RootPath) + require.NoError(t, err) + binding := store_sqlite.DedicatedGraph{ + GraphID: GraphIDFor(prefix), RepoPrefix: prefix, OwnerCheckoutID: f.automatic.CheckoutID, + FamilyID: f.automatic.FamilyID, State: reconcile.GraphStateReady, + } + require.NoError(t, f.catalog.UpsertDedicatedGraph(t.Context(), binding)) + primary := contentIdentities(f.store, f.mainPrefix) + require.NotEmpty(t, primary) + require.NotEmpty(t, f.store.GetRepoNodes(prefix)) + require.NoError(t, f.lc.rollbackPromotion(t.Context(), prefix, binding.GraphID, f.automatic, owned)) + require.Nil(t, f.mi.GetMetadata(prefix)) + require.Empty(t, f.store.GetRepoNodes(prefix)) + _, found, err := f.catalog.GetDedicatedGraph(t.Context(), binding.GraphID) + require.NoError(t, err) + require.False(t, found) + require.Equal(t, primary, contentIdentities(f.store, f.mainPrefix)) + require.NotNil(t, f.mi.GetMetadata(f.mainPrefix)) +} + +func TestPromotionDoesNotBorrowAnExactRootUnderAnotherName(t *testing.T) { + f := newFamilyFixture(t, "named-root") + t.Cleanup(f.close) + var registrationErr error + var before []string + f.lc.indexBarrier = func() { + // Let promotion decide its requested prefix first. A pre-existing + // registration is intentionally adopted by dedicatedPrefixFor; the + // race under test is a different name appearing after that decision. + _, registrationErr = f.mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: f.automatic.RootPath, Name: "existing-name"}) + before = contentIdentities(f.store, "existing-name") + } + out, err := f.lc.PromoteCheckout(t.Context(), f.automatic.CheckoutID, TrackSourceCLI) + require.NoError(t, registrationErr) + require.NotEmpty(t, before) + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision, "a nil Track result under another name does not authorize the requested corpus") + require.True(t, out.Retryable) + require.Nil(t, f.mi.GetMetadata(out.Prefix)) + require.NotNil(t, f.mi.GetMetadata("existing-name")) + require.Equal(t, before, contentIdentities(f.store, "existing-name")) + _, found, err := f.catalog.GetDedicatedGraph(t.Context(), GraphIDFor(out.Prefix)) + require.NoError(t, err) + require.False(t, found) +} diff --git a/internal/indexer/cold_manifest_census_test.go b/internal/indexer/cold_manifest_census_test.go index 4ba7f20c7..deb84a138 100644 --- a/internal/indexer/cold_manifest_census_test.go +++ b/internal/indexer/cold_manifest_census_test.go @@ -777,7 +777,7 @@ func TestColdScopedDeferredTailPreservesUnselectedAttempt(t *testing.T) { // This is the actual cold orchestration entry whose sorted lane set only // contains "selected". The previously registered "overlap" repository's // lane is not part of this batch, despite its pending deferred contracts. - results, err := mi.indexMultiRepo([]config.RepoEntry{{Path: selectedRoot, Name: "selected"}}) + results, err := mi.indexFixtureRepos([]config.RepoEntry{{Path: selectedRoot, Name: "selected"}}) require.NoError(t, err) require.Contains(t, results, "selected") assert.Same(t, registry, unselected.pendingContractReg, diff --git a/internal/indexer/configured_repos.go b/internal/indexer/configured_repos.go index f99cbd4bf..a26721f45 100644 --- a/internal/indexer/configured_repos.go +++ b/internal/indexer/configured_repos.go @@ -3,6 +3,7 @@ package indexer import ( "context" "strings" + "time" "go.uber.org/zap" @@ -121,31 +122,89 @@ func (mi *MultiIndexer) ResolveConfiguredRepos(ctx context.Context, entries []co // loaded yet. Filter by base name before resolving workspaces: the ordinary // one-root case needs no workspace reads or catalog ownership pass. func (mi *MultiIndexer) configuredPrefixCollision(prefix, root string) error { + return mi.configuredPrefixCollisionBefore(context.Background(), prefix, root, time.Now().Add(prefixOwnershipTimeout)) +} + +func (mi *MultiIndexer) configuredPrefixCollisionBefore(ctx context.Context, prefix, root string, before time.Time) error { if mi.configMgr == nil { return nil } - if mi.configMgr.RepoPathSnapshot().IsOnlyRepoForPrefix(prefix, root) { + candidates := mi.configMgr.RepoTrackSnapshot().InstanceCandidates(prefix) + if len(candidates) == 1 && !strings.Contains(prefix, "@") && candidates[0].Path == root && config.ResolvePrefix(candidates[0]) == prefix { return nil } - var candidates []config.RepoEntry - for _, entry := range mi.configMgr.Global().ReposSnapshot() { - base := config.ResolvePrefix(entry) - if prefix == base || entry.Name == "" && strings.HasPrefix(prefix, base+"@") { - candidates = append(candidates, entry) + var entries []config.RepoEntry + for _, entry := range candidates { + effective, err := mi.effectiveTrackPrefix(ctx, entry, before) + if err != nil { + return err + } + if effective == prefix { + entries = append(entries, entry) } } - if len(candidates) == 0 { + if len(entries) == 0 { return nil } - if len(candidates) == 1 && (pathkey.SamePathIdentity(candidates[0].Path, root) || sameDirectory(candidates[0].Path, root)) { - return nil + budget, cancel := context.WithDeadline(ctx, before) + defer cancel() + observed := make(map[string]prefixStatResult) + stat := func(path string) prefixStatResult { + if result, ok := observed[path]; ok { + return result + } + result := mi.prefixStats.observe(budget, path) + observed[path] = result + return result + } + same := func(a, b string) bool { + return a == b || sameObservedDirectory(stat(a), stat(b)) } - for _, entry := range mi.ResolveConfiguredRepos(context.Background(), candidates) { - if entry.Name == prefix && !pathkey.SamePathIdentity(entry.Path, root) && !sameDirectory(entry.Path, root) { - return NewPrefixCollisionError(prefix, entry.Path, root) + owner := entries[0].Path + // Match startup's priority: a live owner, then a published catalog owner + // still configured under this prefix, then configuration order. + if live := mi.GetMetadata(prefix); live != nil { + configured := false + for _, entry := range entries { + if same(entry.Path, live.RootPath) { + owner, configured = entry.Path, true + break + } + } + if !configured { + return nil // live ownership is checked separately at admission } + } else if provider, ok := mi.graph.(interface{ Catalog() *store_sqlite.Catalog }); ok && provider.Catalog() != nil { + catalog := provider.Catalog() + binding, found, err := catalog.GetDedicatedGraph(budget, GraphIDFor(prefix)) + if err != nil { + mi.logger.Warn("reading configured repository owner; using config order", zap.Error(err)) + } + if err == nil && found { + _, published, err := catalog.DedicatedBasePublication(budget, binding.GraphID) + if err != nil { + mi.logger.Warn("reading configured repository publication; using config order", zap.Error(err)) + } + if err == nil && published { + checkout, found, err := catalog.GetCheckout(budget, binding.OwnerCheckoutID) + if err != nil { + mi.logger.Warn("reading configured repository owner; using config order", zap.Error(err)) + } + if err == nil && found { + for _, entry := range entries { + if same(entry.Path, checkout.RootPath) { + owner = entry.Path + break + } + } + } + } + } + } + if same(owner, root) { + return nil } - return nil + return newObservedPrefixCollision(prefix, owner, root, stat(owner)) } // Bucket only; SamePathIdentity still distinguishes case-sensitive directories. @@ -161,16 +220,20 @@ func configuredRootKey(root string) string { // snapshot invalidates the result before the coordinated recheck. Live ownership // is checked separately under the registry lock at every admission boundary. func (mi *MultiIndexer) configuredPrefixCheck() func(string, string) error { - var snapshot *config.RepoPathSnapshot + return mi.configuredPrefixCheckContext(context.Background(), time.Now().Add(prefixOwnershipTimeout)) +} + +func (mi *MultiIndexer) configuredPrefixCheckContext(ctx context.Context, before time.Time) func(string, string) error { + var snapshot *config.RepoTrackSnapshot var checkedPrefix, checkedRoot string var collision error return func(prefix, root string) error { if mi.configMgr == nil { return nil } - current := mi.configMgr.RepoPathSnapshot() + current := mi.configMgr.RepoTrackSnapshot() if current != snapshot || prefix != checkedPrefix || root != checkedRoot { - collision = mi.configuredPrefixCollision(prefix, root) + collision = mi.configuredPrefixCollisionBefore(ctx, prefix, root, before) snapshot, checkedPrefix, checkedRoot = current, prefix, root } return collision diff --git a/internal/indexer/configured_repos_regression_test.go b/internal/indexer/configured_repos_regression_test.go index 4a88daa82..673351db1 100644 --- a/internal/indexer/configured_repos_regression_test.go +++ b/internal/indexer/configured_repos_regression_test.go @@ -401,7 +401,7 @@ func TestConfiguredPrefixCheckReusesResolutionUntilConfigChanges(t *testing.T) { f.cm.Global().Repos = []config.RepoEntry{{Path: first, Name: "src"}, {Path: second, Name: "src"}} require.NoError(t, f.cm.Global().Save()) calls := 0 - f.mi.canonicalRoot = func(root string) string { calls++; return pathkey.CanonicalPath(root) } + f.mi.prefixStats.stat = func(root string) (os.FileInfo, error) { calls++; return os.Stat(root) } check := f.mi.configuredPrefixCheck() require.NoError(t, check("src", second)) firstCalls := calls diff --git a/internal/indexer/edit_delta_capability_test.go b/internal/indexer/edit_delta_capability_test.go index e3406b473..22bff4e4e 100644 --- a/internal/indexer/edit_delta_capability_test.go +++ b/internal/indexer/edit_delta_capability_test.go @@ -105,7 +105,6 @@ var editDeltaAbsentCapabilities = map[string]string{ "graph.ContractOwnerScalarInvalidator": "optional companion of ContractOwnerReplacer; the fallback re-derives the scalars through ordinary node writes", "graph.ContractOwnerScalarLiveness": "optional companion of ContractOwnerReplacer; absent, the fallback does not assume liveness and checks it through reads", "graph.ContractStateStore": "per-repository ledger read (index state, file metadata, mtimes, failures, receipts): absent, the engine treats nothing as recorded and takes its conservative path; a delta forces its explicit paths", - "graph.CoordinatedBulkLoader": "whole-repository or administrative operation (full index, repository reset, untrack or cleanup, warm start, vector corpus, bulk load windows, WAL maintenance); not on a file delta's per-save path", "graph.CoverageEnrichmentWriter": "node-keyed enrichment side tables (blame, churn, coverage, releases): their rows for an evicted node are the layer below's, hidden with the node; a delta writes none", "graph.CrossRepoFlagMarker": "cross-repository edge flagging; single-repository delta", "graph.CurrentGenerationRepoEvicter": "whole-repository or administrative operation (full index, repository reset, untrack or cleanup, warm start, vector corpus, bulk load windows, WAL maintenance); not on a file delta's per-save path", diff --git a/internal/indexer/multi.go b/internal/indexer/multi.go index 75ef935f1..37cd1b9dc 100644 --- a/internal/indexer/multi.go +++ b/internal/indexer/multi.go @@ -2,6 +2,7 @@ package indexer import ( "context" + "errors" "fmt" "maps" "os" @@ -101,6 +102,9 @@ type MultiIndexer struct { logger *zap.Logger loggedCollisions sync.Map canonicalRoot func(string) string + // prefixStats bounds only the filesystem observations used to establish + // prefix ownership. A timed-out native Stat is not a store producer. + prefixStats prefixStatAllowance // newIndexer is instance-local so lifecycle tests can observe a constructor // failure without publishing the candidate Indexer. Production instances set // it to New; every per-repository construction flows through this factory. @@ -268,12 +272,12 @@ type MultiIndexer struct { // every per-repo Indexer. When nil (the default), per-repo // deferred passes skip semantic enrichment — this is the root // cause of "enrich:0" in daemon mode. Set by the daemon via - // SetSemanticManager before IndexAll / TrackRepo. + // SetSemanticManager before TrackRepo. semanticMgr *semantic.Manager } // SetEmbedder installs the embedding provider every per-repo indexer -// should use. Must be called before IndexAll / TrackRepo for vectors +// should use. Must be called before TrackRepo for vectors // to land in the graph — without this the fresh Indexer created per // repo has embedder=nil and vector-plan preparation skips the embedding pass. // Safe to call zero or one times; subsequent calls silently replace. @@ -428,7 +432,7 @@ func (mi *MultiIndexer) newPerRepoIndexerGuardedWithMode( // SetEmbeddingChunkOptions installs the AST sub-chunking configuration // every per-repo Indexer this MultiIndexer constructs should use, and // re-applies it to every per-repo Indexer already built. Call before -// IndexAll / TrackRepo so the warmup indexers pick it up. +// TrackRepo so the warmup indexers pick it up. func (mi *MultiIndexer) SetEmbeddingChunkOptions(opts embedding.ChunkOptions) { mi.mu.Lock() mi.embedChunkOpts = opts @@ -626,16 +630,6 @@ func (mi *MultiIndexer) RunDeferredPassesAllResult(ctx context.Context) Deferred return mi.BeginDeferredPasses(ctx, nil).FinishTailResult() } -// finishColdDeferredPasses completes the cold-index deferred tail without -// repeating the whole-graph cross-repository resolver. The receipt-guided tail -// already performs either exact cross-repository catch-up or a fail-closed full -// pass; only contract-bridge reconciliation remains afterward. -func (mi *MultiIndexer) finishColdDeferredPasses(ctx context.Context, completedIndexers []*Indexer) DeferredPassesResult { - result := mi.beginDeferredPasses(ctx, nil, completedIndexers, true).FinishTailResult() - mi.ReconcileContractEdges() - return result -} - // current requires this Indexer's repository mutation lane to be held. func (a *deferredPassAttempt) current() bool { return a.indexer.deferredAttempt == a && a.indexer.pendingContractReg == a.registry @@ -2074,79 +2068,6 @@ func NewMultiIndexer( } } -// IndexAll indexes all active repos concurrently. Each repo gets its own -// Indexer instance with repo-specific config. Returns per-repo IndexResults. -func (mi *MultiIndexer) IndexAll() (map[string]*IndexResult, error) { - return mi.IndexScoped("", "") -} - -// IndexScoped is IndexAll restricted to repos whose workspace and -// project slugs match. Empty filters disable that axis (so empty/empty -// is equivalent to IndexAll). Resolution honours the same precedence -// as resolveWorkspaceID/resolveProjectID — RepoEntry override → -// `.gortex.yaml::workspace` → repo prefix — so a `gortex server -// --workspace foo` invocation matches both repos that declare -// `workspace: foo` in their own `.gortex.yaml` and repos pinned to -// `foo` from the user's global config. -// -// Returns an error when the filters exclude every active repo, so a -// `--workspace typo` surfaces as a startup failure rather than a -// silently empty graph. -func (mi *MultiIndexer) IndexScoped(workspaceSlug, projectSlug string) (map[string]*IndexResult, error) { - repos := mi.configMgr.ActiveRepos() - if len(repos) == 0 { - return nil, nil - } - if workspaceSlug != "" || projectSlug != "" { - filtered := mi.filterReposByScope(repos, workspaceSlug, projectSlug) - if len(filtered) == 0 { - return nil, fmt.Errorf("scope filter matched zero of %d active repos (workspace=%q project=%q)", len(repos), workspaceSlug, projectSlug) - } - repos = filtered - } - - // indexMultiRepo owns the complete coordinated pipeline, including the - // one contract reconciliation required before graph-wide derivation. Do - // not repeat it here: reconciliation evicts and rebuilds a generation and - // a second identical pass is pure global work. - // - // A lone repo takes the same path as any other — it is simply the first - // tracked repo, and its nodes carry its prefix like everyone else's. - return mi.indexMultiRepo(repos) -} - -// filterReposByScope returns the subset of repos whose resolved -// workspace and project slugs match the supplied filters. Empty -// filters disable that axis. Loads each repo's `.gortex.yaml` first so -// resolution sees the workspace/project declared there — matching only -// against `RepoEntry.Workspace` would miss repos that declare their -// slug in their own config file (the typical case for first-party -// repos). -func (mi *MultiIndexer) filterReposByScope(repos []config.RepoEntry, workspaceSlug, projectSlug string) []config.RepoEntry { - if workspaceSlug == "" && projectSlug == "" { - return repos - } - out := make([]config.RepoEntry, 0, len(repos)) - for _, e := range repos { - absPath, err := filepath.Abs(e.Path) - if err != nil { - continue - } - prefix := config.ResolvePrefix(e) - mi.configMgr.LoadWorkspaceConfig(prefix, absPath) - cfg := mi.configMgr.GetRepoConfig(prefix) - entryCopy := e - if workspaceSlug != "" && resolveWorkspaceID(&entryCopy, cfg, prefix) != workspaceSlug { - continue - } - if projectSlug != "" && resolveProjectID(&entryCopy, cfg, prefix) != projectSlug { - continue - } - out = append(out, e) - } - return out -} - func (idx *Indexer) installGoResolverContext() { mi := idx.repositoryMutationOwner if mi == nil { @@ -2292,370 +2213,6 @@ func readGoModModule(repoPath string) string { return modfile.ModulePath(data) } -// rebuildColdRefFacts seeds the durable resolved-reference sidecar once for -// the complete successful repository set. The backend implementation is a -// single set-oriented transaction; keeping the capability probe here avoids a -// per-repository persistence loop and leaves in-memory stores as a no-op. -func (mi *MultiIndexer) rebuildColdRefFacts(ctx context.Context, repoPrefixes []string) error { - if err := ctx.Err(); err != nil { - return err - } - rebuilder, ok := mi.graph.(graph.RefFactsRebuilder) - if !ok { - return nil - } - if err := rebuilder.RebuildRefFactsForRepos(repoPrefixes); err != nil { - return err - } - // The current backend contract is atomic but not interruptible in flight. - // Observing cancellation here prevents completion/global consumers from - // publishing a generation whose ref-fact boundary the caller abandoned. - return ctx.Err() -} - -// indexMultiRepo indexes multiple repos concurrently with repo prefixes. -func (mi *MultiIndexer) indexMultiRepo(repos []config.RepoEntry) (map[string]*IndexResult, error) { - type repoResult struct { - prefix string - result *IndexResult - idx *Indexer - meta *RepoMetadata - err error - } - - // Resolve each repo's prefix serially first: git-worktree instancing, - // the derived-name persistence, and the cross-batch collision guard - // must be deterministic, not raced across the parallel index - // goroutines below. The same loop builds the tracked-module map used - // for cross-repo dependency detection. - type resolvedRepo struct { - entry config.RepoEntry - prefix string - cfg *config.Config - absPath string - identity *RepoIdentity - } - var configLookup config.RepoPathLookup - if mi.configMgr != nil { - configLookup = mi.configMgr.Global().SnapshotRepoPaths() - } - moduleRoots := make(map[string]string) - resolved := make([]resolvedRepo, 0, len(repos)) - seenPrefix := make(map[string]string, len(repos)) // prefix → absPath - var resolveErrors []string - for _, entry := range repos { - absPath, err := filepath.Abs(entry.Path) - if err != nil { - resolveErrors = append(resolveErrors, fmt.Sprintf("resolving path %s: %v", entry.Path, err)) - continue - } - info, err := os.Stat(absPath) - if err != nil { - resolveErrors = append(resolveErrors, fmt.Sprintf("opening repository %s: %v", entry.Path, err)) - continue - } - if !info.IsDir() { - resolveErrors = append(resolveErrors, fmt.Sprintf("opening repository %s: not a directory", entry.Path)) - continue - } - identity, _ := DetectIdentity(absPath) - e := entry - prefix, cfg := mi.resolveTrackPrefix(&e, absPath, identity, configLookup, mi.configuredPrefixCheck()) - // Two distinct checkouts can still land on the same derived prefix - // (e.g. two worktrees declaring the same workspace). resolveTrackPrefix - // only sees already-tracked repos, so guard against collisions within - // this batch too. - if prev, ok := seenPrefix[prefix]; ok { - if pathkey.SamePathIdentity(prev, absPath) { - // Duplicate config entries for one repository must not launch - // concurrent raw workers behind a single held repository lane. - continue - } - prefix += "-" + shortPathHash(absPath) - e.Name = prefix - // This prefix is minted for THIS batch and is not the one the - // lifecycle registered an owner under, so the receipt opened for - // it below finds no owner and the repository is left unwitnessed - // (openSourceWitness). That is the honest direction — the nodes - // this pass writes carry the disambiguated prefix too, so no pin - // taken on the colliding prefix is reading them — but a request - // against the disambiguated prefix gets ErrBaseCorpusUnwitnessed - // rather than an exactness label until an owner is registered for - // the name this batch chose. - } - seenPrefix[prefix] = absPath - resolved = append(resolved, resolvedRepo{entry: e, prefix: prefix, cfg: cfg, absPath: absPath, identity: identity}) - moduleRoots[prefix] = absPath - } - trackedModules := readTrackedGoModules(moduleRoots) - for prefix, mod := range trackedModules { - mi.logger.Debug("tracked repo module", zap.String("repo", prefix), zap.String("module", mod)) - } - - if len(resolved) == 0 { - if len(resolveErrors) == 0 { - return map[string]*IndexResult{}, nil - } - sort.Strings(resolveErrors) - return nil, fmt.Errorf("all repos failed to index: %s", strings.Join(resolveErrors, "; ")) - } - - prefixes := make([]string, 0, len(resolved)) - for _, repo := range resolved { - prefixes = append(prefixes, repo.prefix) - } - // Freeze one batch-mode generation before acquiring any stable repository - // lane. This global gate -> sorted lanes order prevents a queued transition - // writer from owning the gate while waiting for a lane held by this batch. - mi.batchMutationGate.RLock() - defer mi.batchMutationGate.RUnlock() - - // One receipt per OUTPUT OWNER this batch writes, opened after every lane - // is held and before any payload moves. The whole batch is one decision: if - // a newer mutation takes over any of these owners while it runs, the batch - // cannot fulfil that owner's generation. - // - // Owners are deduplicated, and not by prefix. Two tracked entries can name - // one repository root (two config entries with different explicit Names and - // one Path: resolveTrackPrefix honours an explicit Name verbatim, so the - // prefix collision guard above never fires). legacyOutputTargetFor keys the - // owner on the root, so both entries name ONE generation-zero output. - // Issuing two receipts for it would make the second supersede the first and - // refuse a cold index that fully succeeded. - batchTargets := make([]OutputMutationTarget, 0, len(resolved)) - seenOutputOwner := make(map[string]bool, len(resolved)) - for _, repo := range resolved { - target := legacyOutputTargetFor(outputStoreIdentity(mi.graph), repo.prefix, repo.absPath, "prefix:"+repo.prefix) - if seenOutputOwner[target.OwnerKey] { - continue - } - seenOutputOwner[target.OwnerKey] = true - batchTargets = append(batchTargets, target) - } - - var finalResults map[string]*IndexResult - laneErr := mi.withRepositoryMutationLanes(context.Background(), prefixes, func() error { - receipts, receiptErr := mi.outputGenerationAuthority().BeginAll( - context.Background(), OutputEntryIndexMultiRepo, batchTargets) - if receiptErr != nil { - return receiptErr - } - settled := false - defer func() { - if !settled { - receipts.Abandon() - } - }() - // Preventive half: refuse before the cold batch writes anything if any - // owner was already taken over while admission blocked on its gates. - if err := receipts.Current(); err != nil { - return err - } - batchErr := func() error { - // The one topology writer covers parse, publication, deferred/cross-repo - // tails, ref-facts, and global derivation. - batchMode := mi.currentBatchMode() - finishTopologyMutation := reach.BeginTopologyMutation(mi.graph) - defer finishTopologyMutation(true) - - pipelineResults, pipelineErr := func() (map[string]*IndexResult, error) { - resultCh := make(chan repoResult, len(resolved)) - var wg sync.WaitGroup - coordinatedBulk, _ := mi.graph.(graph.CoordinatedBulkLoader) - coordinatedBulkActive := coordinatedBulk != nil && coordinatedBulk.BeginCoordinatedBulkLoad() - defer func() { - if coordinatedBulkActive { - if err := coordinatedBulk.EndCoordinatedBulkLoad(); err != nil { - mi.logger.Error("multi-repo bulk-load cleanup failed", zap.Error(err)) - } - } - }() - - for _, rr := range resolved { - wg.Add(1) - go func(r resolvedRepo) { - defer wg.Done() - - idx := mi.newPerRepoIndexerGuardedWithMode(r.cfg.Index, batchMode) - idx.SetRepoPrefix(r.prefix) - entryCopy := r.entry - idx.SetWorkspaceID(resolveWorkspaceID(&entryCopy, r.cfg, r.prefix)) - idx.SetProjectID(resolveProjectID(&entryCopy, r.cfg, r.prefix)) - idx.SetTrackedRepoModules(trackedModules) - // Defer the per-repo cross-cutting passes (ResolveAll, - // semantic enrich, contract extract+commit) so they don't - // race against each other across goroutines on the shared - // graph. They run serially below via RunDeferredPasses after - // wg.Wait(). The graph-wide derivation passes run once after - // the loop via the shared global-pass pipeline. - idx.SetDeferResolve(true) - - result, err := idx.indexCtxRaw(WithIndexOrigin(context.Background(), "index_all"), r.absPath) - if err != nil { - idx.Close() - resultCh <- repoResult{prefix: r.prefix, err: fmt.Errorf("indexing %s: %w", r.absPath, err)} - return - } - if result == nil { - idx.Close() - resultCh <- repoResult{prefix: r.prefix, err: fmt.Errorf("indexing %s returned a nil result", r.absPath)} - return - } - result.RepoPrefix = r.prefix - - meta := &RepoMetadata{ - RepoPrefix: r.prefix, - RootPath: r.absPath, - Identity: r.identity, - LastIndexTime: time.Now(), - FileCount: result.FileCount, - NodeCount: result.NodeCount, - EdgeCount: result.EdgeCount, - ParseErrors: result.Errors, - FileMtimes: idx.publishFileMtimes(), - - IsWorktree: ResolveWorktree(r.absPath).IsWorktree, - } - - resultCh <- repoResult{prefix: r.prefix, result: result, idx: idx, meta: meta} - }(rr) - } - - go func() { - wg.Wait() - close(resultCh) - }() - - results := make(map[string]*IndexResult) - indexErrors := resolveErrors - completed := make([]repoResult, 0, len(resolved)) - - // Drain workers without holding the registry lock. Constructors and future - // worker tails may need read access to MultiIndexer state; pinning mi.mu - // across the channel range turns any such read into a producer/consumer - // deadlock and blocks unrelated registry readers for the whole warmup. - for rr := range resultCh { - if rr.err != nil { - mi.logger.Error("failed to index repo", zap.String("prefix", rr.prefix), zap.Error(rr.err)) - indexErrors = append(indexErrors, rr.err.Error()) - continue - } - completed = append(completed, rr) - results[rr.prefix] = rr.result - } - oldIndexers := make([]*Indexer, 0, len(completed)) - mi.mu.Lock() - for _, rr := range completed { - if old := mi.indexers[rr.prefix]; old != nil && old != rr.idx { - oldIndexers = append(oldIndexers, old) - } - mi.repos[rr.prefix] = rr.meta - mi.indexers[rr.prefix] = rr.idx - } - mi.mu.Unlock() - for _, old := range oldIndexers { - old.Close() - } - if coordinatedBulkActive { - if err := coordinatedBulk.EndCoordinatedBulkLoad(); err != nil { - return nil, fmt.Errorf("multi-repo bulk-load finalize: %w", err) - } - coordinatedBulkActive = false - } - for _, rr := range completed { - if err := mi.postPublishGoModules(context.Background(), rr.idx, true); err != nil { - mi.logger.Warn("post-publish Go module follow-up", zap.Error(err)) - } - } - - // Do not publish a completed pipeline when no repository reached the - // deferred stages. Besides making the failure deterministic, this prevents - // global passes from deriving edges from a partially-drained failed batch. - if len(indexErrors) > 0 && len(results) == 0 { - sort.Strings(indexErrors) - return nil, fmt.Errorf("all repos failed to index: %s", strings.Join(indexErrors, "; ")) - } - - // Complete cold multi-repo indexing through the same coordinated pipeline - // used by daemon warmup: - // 1. materialise go.mod contracts once, then run one shared base resolve; - // 2. enrich repositories in bounded language-aware batches (large Go - // repositories remain exclusive), committing contracts only after each - // batch drains; - // 3. use the mutation receipt to perform only the exact catch-up needed for - // semantic/contract mutations. - // - // The old loop called idx.RunDeferredPasses with - // skipResolveInDeferred=false, so every repository performed ResolveAll over - // the entire shared graph. At R repositories and E edges that was O(R*E) and - // was the dominant cold-index regression. runDeferredGoMod is generation- - // idempotent, so RunDeferredPassesAll does not repeat the pre-resolve work. - deferCtx := context.Background() - if err := mi.RunPreEnrichResolve(deferCtx, nil, nil); err != nil { - return results, fmt.Errorf("multi-repo pre-enrichment resolve: %w", err) - } - completedIndexers := make([]*Indexer, 0, len(completed)) - for _, rr := range completed { - completedIndexers = append(completedIndexers, rr.idx) - } - deferredResult := mi.finishColdDeferredPasses(deferCtx, completedIndexers) - mi.logger.Info("multi-repo coordinated deferred passes complete", - zap.Int("repos_indexed", len(results)), - zap.Int("repos_failed", len(indexErrors)), - zap.Int("enrich_scheduled", deferredResult.EnrichScheduled), - zap.Bool("exact_cross_repo_complete", deferredResult.ExactCrossRepoComplete)) - - // ResolveAll normally seeds ref_facts after a full resolve. The coordinated - // cold path intentionally bypasses per-repository ResolveAll, so seed the - // successful repository set once after every base, semantic catch-up, and - // cross-repository mutation has settled. Sorting makes the boundary stable - // for tracing/tests; SQLite consumes the whole slice in one transaction. - successfulPrefixes := make([]string, 0, len(results)) - for prefix := range results { - successfulPrefixes = append(successfulPrefixes, prefix) - } - sort.Strings(successfulPrefixes) - if err := mi.rebuildColdRefFacts(deferCtx, successfulPrefixes); err != nil { - return results, fmt.Errorf("multi-repo reference-fact rebuild: %w", err) - } - - // Graph-wide derivation passes run exactly once after every repo - // has been parsed, every per-repo and cross-repo resolver has lifted - // placeholder edges, and contract bridges are in place. RunDeferredPasses - // intentionally skips these so we don't pay an O(global) walk per - // repo (was the dominant cost at R≈100+). - mi.runGlobalGraphPassesTopologyHeld(context.Background(), nil, false) - - return results, nil - }() - finalResults = pipelineResults - return pipelineErr - }() - if batchErr != nil { - return batchErr - } - settled = true - // Fulfilment is refused when a newer mutation took over any of these - // owners while the batch ran; the batch then reports that refusal - // instead of announcing a completed cold index. - if err := receipts.Complete(); err != nil { - return err - } - // Every completed repository's journal is accepted, failed files - // included: the cold pass staged a failure envelope for each of them - // (indexCtxRaw), so the accepted namespace excludes exactly that set. - for prefix, result := range finalResults { - if result != nil { - if err := mi.acceptInstalledContractCoreInputs(context.Background(), prefix); err != nil { - return err - } - } - } - return nil - }) - return finalResults, laneErr -} - // IndexRepo re-indexes a single repo by prefix. Evicts existing data first. func (mi *MultiIndexer) IndexRepo(repoPrefix string) (*IndexResult, error) { mi.mu.RLock() @@ -2732,8 +2289,8 @@ func (mi *MultiIndexer) indexRepoRaw(repoPrefix string) (*IndexResult, error) { } }() // Always stamp the repo prefix, even when this is the only tracked repo. - // The multi-repo cold path (indexMultiRepo) already prefixes - // unconditionally; gating the single-repo re-index on repo count left the + // First registration also prefixes unconditionally; gating re-indexing + // on repo count left the // two paths producing different id / path shapes for the same repo, so // adding a second repo turned the first from bare into prefixed — a lossy // whole-graph rewrite. Uniform prefixing removes the single/multi data-shape @@ -3099,15 +2656,22 @@ type PrefixCollisionError struct { ExistingPath string Path string OwnerMissing bool + // OwnerState is immutable evidence from the admission observation: + // present, missing (ENOENT only), or unknown (timeout/permission/absence + // of an owner witness). Formatting never re-probes the filesystem. + OwnerState string } // NewPrefixCollisionError samples owner availability once, outside formatting. func NewPrefixCollisionError(prefix, owner, root string) *PrefixCollisionError { - _, err := os.Stat(owner) - return &PrefixCollisionError{Prefix: prefix, ExistingPath: owner, Path: root, OwnerMissing: os.IsNotExist(err)} + info, err := os.Stat(owner) + return newObservedPrefixCollision(prefix, owner, root, prefixStatResult{info: info, err: err}) } func (e *PrefixCollisionError) Error() string { + if e.ExistingPath == "" { + return fmt.Sprintf("repository root %s has no confirmed registered owner for prefix %q; no checkout or watcher was recorded", e.Path, e.Prefix) + } if e.OwnerMissing { return fmt.Sprintf("repository prefix %q is already used by %s, which no longer exists; inspect `gortex untrack %s` before tracking %s again", e.Prefix, e.ExistingPath, e.ExistingPath, e.Path) @@ -3115,28 +2679,10 @@ func (e *PrefixCollisionError) Error() string { return fmt.Sprintf("repository prefix %q is already used by %s; use `gortex track --name %s` for a new entry, or set a unique name in the global config and reload", e.Prefix, e.ExistingPath, e.Path) } -// trackedLocked reports whether path is already tracked under any prefix, -// or returns a PrefixCollisionError when prefix belongs to another -// repository. The same-path scan runs first, so a repository tracked under -// another name (for example after `track --name`) stays a no-op. -// Callers hold mi.mu. -func (mi *MultiIndexer) trackedLocked(prefix, path string) (bool, error) { - for _, meta := range mi.repos { - if meta != nil && pathkey.SamePathIdentity(meta.RootPath, path) { - return true, nil - } - } - existing, ok := mi.repos[prefix] - if ok && existing != nil && !sameDirectory(existing.RootPath, path) { - return true, NewPrefixCollisionError(prefix, existing.RootPath, path) - } - return ok, nil -} - // logCollision warns about a configured repository that is not loaded // because another repository owns its prefix. func (mi *MultiIndexer) logCollision(err error) { - if err == nil { + if err == nil || mi.logger == nil { return } if _, logged := mi.loggedCollisions.LoadOrStore(err.Error(), struct{}{}); !logged { @@ -3162,7 +2708,21 @@ func (mi *MultiIndexer) TrackRepo(entry config.RepoEntry) (*IndexResult, error) // pins existing and family admin-name prefixes before reaching this fallback. // The prepared config is published only after live and configured ownership // checks succeed. A derived instance name is written back to entry for config. +type trackPrefixResolution struct { + prefix string + cfg *config.Config + ownership prefixOwnershipObservation +} + func (mi *MultiIndexer) resolveTrackPrefix(entry *config.RepoEntry, absPath string, identity *RepoIdentity, lookup config.RepoPathLookup, checkConfigured func(string, string) error) (string, *config.Config) { + resolved, err := mi.resolveTrackPrefixContext(context.Background(), entry, absPath, identity, lookup, checkConfigured, time.Now().Add(prefixOwnershipTimeout), nil) + if err != nil && mi.configMgr != nil { + return resolved.prefix, mi.configMgr.GetRepoConfig(resolved.prefix) + } + return resolved.prefix, resolved.cfg +} + +func (mi *MultiIndexer) resolveTrackPrefixContext(ctx context.Context, entry *config.RepoEntry, absPath string, identity *RepoIdentity, lookup config.RepoPathLookup, checkConfigured func(string, string) error, before time.Time, candidate *prefixStatResult) (trackPrefixResolution, error) { base := config.ResolvePrefix(*entry) if base == "" || base == "." { if identity != nil { @@ -3170,11 +2730,20 @@ func (mi *MultiIndexer) resolveTrackPrefix(entry *config.RepoEntry, absPath stri } } if mi.configMgr == nil { - return base, config.Default() + observation, err := mi.observePrefixOwnershipBefore(ctx, base, absPath, before, candidate) + return trackPrefixResolution{prefix: base, cfg: config.Default(), ownership: observation}, err } + configuration := mi.configMgr.RepoTrackSnapshot() // Read the workspace once, before acquiring the registry lock. The same // snapshot supplies both membership and the eventual config publication. + if lookup == nil { + configured, err := mi.configuredTrackEntry(ctx, absPath, before, base) + if err != nil { + return trackPrefixResolution{prefix: base}, err + } + lookup = func(string) *config.RepoEntry { return configured } + } cfg, publish := mi.configMgr.PrepareWorkspaceConfig(base, absPath, lookup) declaredWS := entry.Workspace if declaredWS == "" { @@ -3185,26 +2754,47 @@ func (mi *MultiIndexer) resolveTrackPrefix(entry *config.RepoEntry, absPath stri if entry.Name == "" { if name, separate := WorktreeInstanceName(absPath, base, declaredWS, entry.AsWorktree); separate { prefix = name - mi.mu.RLock() - existing, ok := mi.repos[prefix] - mi.mu.RUnlock() - if ok && existing != nil && !pathkey.SamePathIdentity(existing.RootPath, absPath) { + observation, err := mi.observePrefixOwnershipBefore(ctx, prefix, absPath, before, candidate) + if err != nil { + var collision *PrefixCollisionError + if !errors.As(err, &collision) || collision.OwnerState == "unknown" { + return trackPrefixResolution{prefix: prefix, cfg: cfg}, err + } prefix = WorktreeInstanceCollisionName(name, absPath) + } else if observation.registeredPrefix != "" { + return trackPrefixResolution{prefix: prefix, cfg: cfg, ownership: observation}, nil } entry.Name = prefix // persist the decision so warmup reproduces it } } + observation, err := mi.observePrefixOwnershipBefore(ctx, prefix, absPath, before, candidate) + resolution := trackPrefixResolution{prefix: prefix, cfg: cfg, ownership: observation} + if err != nil { + return resolution, err + } // Configured owners can be absent from the live registry during warmup. - if checkConfigured(prefix, absPath) != nil { - return prefix, mi.configMgr.GetRepoConfig(prefix) + // An exact live registration under another name remains a no-op even if + // this request's default basename belongs to a different configured root. + if observation.registeredPrefix == "" { + if err := checkConfigured(prefix, absPath); err != nil { + return resolution, err + } } - // Re-check ownership before publishing the prepared snapshot. + // Prepared config publication is memory-only. Revalidate its exact owner + // witness under mu; no filesystem callback or Stat runs under this lock. mi.mu.Lock() - tracked, collision := mi.trackedLocked(prefix, absPath) - if collision != nil || tracked && !mi.ownsPrefixLocked(prefix, absPath) { + if mi.configMgr.RepoTrackSnapshot() != configuration { + mi.mu.Unlock() + return resolution, errPrefixConfigChanged + } + if !observation.snapshot.matchesLocked(mi) { mi.mu.Unlock() - return prefix, mi.configMgr.GetRepoConfig(prefix) + return resolution, errPrefixOwnerChanged + } + if observation.registeredPrefix != "" && observation.registeredPrefix != prefix { + mi.mu.Unlock() + return resolution, nil } publish(prefix) mi.mu.Unlock() @@ -3216,14 +2806,7 @@ func (mi *MultiIndexer) resolveTrackPrefix(entry *config.RepoEntry, absPath stri if effective := mi.configMgr.EffectiveExclude(prefix); slices.Equal(effective, cfg.Exclude) { cfg.Exclude, cfg.Index.Exclude, cfg.Watch.Exclude = effective, effective, effective } - return prefix, cfg -} - -// ownsPrefixLocked reports whether path is the repository registered under -// prefix. Callers hold mi.mu. -func (mi *MultiIndexer) ownsPrefixLocked(prefix, path string) bool { - existing := mi.repos[prefix] - return existing != nil && (pathkey.SamePathIdentity(existing.RootPath, path) || sameDirectory(existing.RootPath, path)) + return resolution, nil } // EffectiveRepoPrefix returns the prefix a repo entry is tracked under, @@ -3272,8 +2855,14 @@ func (mi *MultiIndexer) TrackRepoCtx(ctx context.Context, entry config.RepoEntry // the source and must keep it open until this method returns. func (mi *MultiIndexer) trackRepoSourceCtx( ctx context.Context, entry config.RepoEntry, content source.ContentSource, -) (*IndexResult, error) { - prepared, err := mi.prepareTrackRepo(entry) +) (result *IndexResult, err error) { + defer func() { + var collision *PrefixCollisionError + if errors.As(err, &collision) { + mi.logCollision(collision) + } + }() + prepared, err := mi.prepareTrackRepoContext(ctx, entry) if err != nil || prepared == nil { return nil, err } @@ -3295,6 +2884,11 @@ type preparedTrackRepo struct { // hook. It does not itself construct or register an Indexer; config loading and // the hook remain side effects, so this is not a pure preflight operation. func (mi *MultiIndexer) prepareTrackRepo(entry config.RepoEntry) (*preparedTrackRepo, error) { + return mi.prepareTrackRepoContext(context.Background(), entry) +} + +func (mi *MultiIndexer) prepareTrackRepoContext(ctx context.Context, entry config.RepoEntry) (*preparedTrackRepo, error) { + before := time.Now().Add(prefixOwnershipTimeout) absPath, err := filepath.Abs(entry.Path) if err != nil { return nil, fmt.Errorf("resolving path %s: %w", entry.Path, err) @@ -3305,18 +2899,36 @@ func (mi *MultiIndexer) prepareTrackRepo(entry config.RepoEntry) (*preparedTrack // no-op on POSIX. absPath = pathkey.NormalizeVolume(absPath) - // Validate path exists and is a directory. - info, err := os.Stat(absPath) - if err != nil { - return nil, fmt.Errorf("path does not exist: %s", absPath) + // Candidate validation and all owner stats/rechecks share one budget. + statCtx, cancelStat := context.WithDeadline(ctx, before) + candidate := mi.prefixStats.observe(statCtx, absPath) + cancelStat() + if candidate.err != nil { + if errors.Is(candidate.err, os.ErrNotExist) { + return nil, fmt.Errorf("path does not exist: %s: %w", absPath, candidate.err) + } + return nil, fmt.Errorf("cannot inspect repository path %s: %w", absPath, candidate.err) } - if !info.IsDir() { + if candidate.info == nil || !candidate.info.IsDir() { return nil, fmt.Errorf("path is not a directory: %s", absPath) } if reason, blocked := unsafeRootBlocked(absPath, entry.Force); blocked { return nil, fmt.Errorf("%s; pass force to track it anyway", reason) } + entry, err = mi.resolveConfiguredTrackAnchor(ctx, entry, absPath, before) + if err != nil { + return nil, err + } + if entry.Path != absPath { + absPath = entry.Path + statCtx, cancel := context.WithDeadline(ctx, before) + candidate = mi.prefixStats.observe(statCtx, absPath) + cancel() + if candidate.err != nil || candidate.info == nil || !candidate.info.IsDir() { + return nil, fmt.Errorf("%w: configured root changed during alias admission", errPrefixIdentityUnknown) + } + } identity, err := DetectIdentity(absPath) if err != nil { @@ -3333,25 +2945,18 @@ func (mi *MultiIndexer) prepareTrackRepo(entry config.RepoEntry) (*preparedTrack // slugs declared inside the repo (without it every repo would fall // back to `workspace = repoPrefix`, making shared-workspace cross-repo // matching impossible to express). - checkConfigured := mi.configuredPrefixCheck() - prefix, cfg := mi.resolveTrackPrefix(&entry, absPath, identity, nil, checkConfigured) - - // Check if already tracked. The prefix-keyed check catches the common - // case; the path scan additionally catches a case-only or Unicode - // spelling variant of an already-tracked directory on a - // case-insensitive filesystem (which resolves to a different derived - // prefix and would otherwise be minted as a bogus second instance — - // the very thing that flips the daemon into multi-repo mode, #270). - mi.mu.RLock() - if tracked, err := mi.trackedLocked(prefix, absPath); tracked { - mi.mu.RUnlock() - return nil, err // nil when this directory is already tracked + checkConfigured := mi.configuredPrefixCheckContext(ctx, before) + resolved, err := mi.resolveTrackPrefixContext(ctx, &entry, absPath, identity, nil, checkConfigured, before, &candidate) + if err != nil { + return nil, err } + prefix, cfg := resolved.prefix, resolved.cfg + if resolved.ownership.registeredPrefix != "" { + return nil, nil + } + mi.mu.RLock() hook := mi.onRepoTracked mi.mu.RUnlock() - if err := checkConfigured(prefix, absPath); err != nil { - return nil, err - } if hook != nil { hook(prefix, absPath) @@ -3414,12 +3019,13 @@ func (mi *MultiIndexer) trackPreparedRepoSourceCtx( // A concurrent track for the same resolved prefix — or for the same // directory through a case/Unicode spelling that resolved to another // prefix lane — may have completed while this caller waited. - mi.mu.RLock() - exists, collisionErr := mi.trackedLocked(prefix, absPath) - mi.mu.RUnlock() - if exists { + ownership, collisionErr := mi.observePrefixOwnership(ctx, prefix, absPath) + if collisionErr != nil { return collisionErr } + if ownership.registeredPrefix != "" { + return nil + } if err := checkConfigured(prefix, absPath); err != nil { return err } @@ -3501,6 +3107,7 @@ func (mi *MultiIndexer) trackPreparedRepoSourceCtx( // prior mtimes are available — in that case there's nothing to // reconcile against and a full index is the correct path. func (mi *MultiIndexer) ReconcileRepoCtx(ctx context.Context, entry config.RepoEntry, priorMtimes map[string]int64) (*IndexResult, error) { + before := time.Now().Add(prefixOwnershipTimeout) start := time.Now() setupTiming := mi.startReconcilePhase(mi.logger, entry.Name, "repository_setup", zap.Int("prior_files", len(priorMtimes))) @@ -3513,13 +3120,31 @@ func (mi *MultiIndexer) ReconcileRepoCtx(ctx context.Context, entry config.RepoE // Normalise the volume (upper-case a Windows drive letter) to converge // with os.Getwd's convention. Cosmetic; never touches the basename. absPath = pathkey.NormalizeVolume(absPath) - info, err := os.Stat(absPath) - if err != nil { - return nil, fmt.Errorf("path does not exist: %s", absPath) + statCtx, cancelStat := context.WithDeadline(ctx, before) + candidate := mi.prefixStats.observe(statCtx, absPath) + cancelStat() + if candidate.err != nil { + if errors.Is(candidate.err, os.ErrNotExist) { + return nil, fmt.Errorf("path does not exist: %s: %w", absPath, candidate.err) + } + return nil, fmt.Errorf("cannot inspect repository path %s: %w", absPath, candidate.err) } - if !info.IsDir() { + if candidate.info == nil || !candidate.info.IsDir() { return nil, fmt.Errorf("path is not a directory: %s", absPath) } + entry, err = mi.resolveConfiguredTrackAnchor(ctx, entry, absPath, before) + if err != nil { + return nil, err + } + if entry.Path != absPath { + absPath = entry.Path + statCtx, cancel := context.WithDeadline(ctx, before) + candidate = mi.prefixStats.observe(statCtx, absPath) + cancel() + if candidate.err != nil || candidate.info == nil || !candidate.info.IsDir() { + return nil, fmt.Errorf("%w: configured root changed during alias admission", errPrefixIdentityUnknown) + } + } identity, err := DetectIdentity(absPath) if err != nil { return nil, fmt.Errorf("detecting identity for %s: %w", absPath, err) @@ -3530,24 +3155,19 @@ func (mi *MultiIndexer) ReconcileRepoCtx(ctx context.Context, entry config.RepoE // TrackRepoCtx so a worktree instance keeps its derived prefix on the // reconcile path too; config can change between sessions and warmup- // time reconcile runs after a daemon restart. - checkConfigured := mi.configuredPrefixCheck() - prefix, cfg := mi.resolveTrackPrefix(&entry, absPath, identity, nil, checkConfigured) - setupTiming.complete(nil, zap.String("prefix", prefix)) - - // Already tracked — nothing to do. - mi.mu.RLock() - exists, collisionErr := mi.trackedLocked(prefix, absPath) - mi.mu.RUnlock() - if exists { - // Startup keeps the base outcome for a config entry whose prefix - // another repository owns: it is not indexed. Say why instead of - // failing warmup on every restart. - mi.logCollision(collisionErr) - return nil, nil + checkConfigured := mi.configuredPrefixCheckContext(ctx, before) + resolved, ownershipErr := mi.resolveTrackPrefixContext(ctx, &entry, absPath, identity, nil, checkConfigured, before, &candidate) + prefix, cfg := resolved.prefix, resolved.cfg + setupTiming.complete(ownershipErr, zap.String("prefix", prefix)) + if ownershipErr != nil { + var collision *PrefixCollisionError + if errors.As(ownershipErr, &collision) { + mi.logCollision(collision) + return nil, repositoryCollisionSkip(ownershipErr) + } + return nil, ownershipErr } - - if collision := checkConfigured(prefix, absPath); collision != nil { - mi.logCollision(collision) + if resolved.ownership.registeredPrefix != "" { return nil, nil } @@ -3558,7 +3178,8 @@ func (mi *MultiIndexer) ReconcileRepoCtx(ctx context.Context, entry config.RepoE // already pinned by resolveTrackPrefix, so the fallback reproduces // the same prefix. if len(priorMtimes) == 0 { - return mi.TrackRepoCtx(ctx, entry) + result, err := mi.TrackRepoCtx(ctx, entry) + return result, repositoryCollisionSkip(err) } // Prefix unconditionally, as TrackRepoCtx does — see the note there. @@ -3591,16 +3212,21 @@ func (mi *MultiIndexer) ReconcileRepoCtx(ctx context.Context, entry config.RepoE // Snapshot restoration may race another admission for this prefix or // the same directory through a case/Unicode spelling on another lane. // Recheck both identities only after owning the stable lane. - mi.mu.RLock() - exists, collisionErr := mi.trackedLocked(prefix, absPath) - mi.mu.RUnlock() - if exists { - mi.logCollision(collisionErr) + ownership, ownershipErr := mi.observePrefixOwnership(ctx, prefix, absPath) + if ownershipErr != nil { + var collision *PrefixCollisionError + if errors.As(ownershipErr, &collision) { + mi.logCollision(collision) + return repositoryCollisionSkip(ownershipErr) + } + return ownershipErr + } + if ownership.registeredPrefix != "" { return nil } if collision := checkConfigured(prefix, absPath); collision != nil { mi.logCollision(collision) - return nil + return repositoryCollisionSkip(collision) } topologyChanged = true diff --git a/internal/indexer/multi_cold_orchestration_test.go b/internal/indexer/multi_cold_orchestration_test.go index 28bc6ba3a..71f0fa479 100644 --- a/internal/indexer/multi_cold_orchestration_test.go +++ b/internal/indexer/multi_cold_orchestration_test.go @@ -98,7 +98,7 @@ func TestFinishColdDeferredPassesDoesNotRepeatFullCrossRepoResolve(t *testing.T) store := &coldOrchestrationProbeStore{Graph: graph.New()} mi := newColdOrchestrationMulti(store, zap.NewNop()) - result := mi.finishColdDeferredPasses(t.Context(), nil) + result := mi.finishFixtureDeferredPasses(t.Context(), nil) assert.True(t, result.ExactCrossRepoComplete) assert.Zero(t, store.unresolvedScans.Load(), @@ -112,7 +112,7 @@ func TestIndexMultiRepoUsesOneCoordinatedBaseResolve(t *testing.T) { mi := newColdOrchestrationMulti(graph.New(), zap.New(core)) repos := coldOrchestrationRepos(t, 8) - results, err := mi.indexMultiRepo(repos) + results, err := mi.indexFixtureRepos(repos) require.NoError(t, err) require.Len(t, results, len(repos)) @@ -139,7 +139,7 @@ func TestIndexMultiRepoAllFailuresDoNotRunCompletionPipeline(t *testing.T) { }) } - results, err := mi.indexMultiRepo(repos) + results, err := mi.indexFixtureRepos(repos) require.Error(t, err) assert.Nil(t, results) assert.Empty(t, logs.FilterMessage("DEFERRED-TIMING master.ResolveAll").All()) @@ -332,7 +332,7 @@ func snapshotColdGraph(store graph.Store, repos []config.RepoEntry) coldGraphSha func TestCoordinatedColdPipelineMatchesSerialReferenceGraph(t *testing.T) { repos := coldOrchestrationRepos(t, 5) coordinated := newColdOrchestrationMulti(graph.New(), zap.NewNop()) - _, err := coordinated.indexMultiRepo(repos) + _, err := coordinated.indexFixtureRepos(repos) require.NoError(t, err) serial := serialColdReference(t, repos) @@ -407,7 +407,7 @@ func TestIndexMultiRepoRefFactsMatchesSingleFullAndUsesOneSetRebuild(t *testing. counting := &coldRefFactsCountingStore{Store: store} multi := newColdOrchestrationMulti(counting, zap.NewNop()) - results, err := multi.indexMultiRepo(requested) + results, err := multi.indexFixtureRepos(requested) require.NoError(t, err) require.Len(t, results, len(repos)) require.Equal(t, 1, counting.rebuildCalls, @@ -449,11 +449,11 @@ func TestColdRefFactRebuildPreservesFailureAndCancellation(t *testing.T) { ctx, cancel := context.WithCancel(t.Context()) cancel() - err := mi.rebuildColdRefFacts(ctx, []string{"repo-a", "repo-b"}) + err := mi.rebuildFixtureRefFacts(ctx, []string{"repo-a", "repo-b"}) require.ErrorIs(t, err, context.Canceled) require.Zero(t, probe.rebuildCalls, "cancellation before the boundary must skip the rebuild") - err = mi.rebuildColdRefFacts(t.Context(), []string{"repo-a", "repo-b"}) + err = mi.rebuildFixtureRefFacts(t.Context(), []string{"repo-a", "repo-b"}) require.ErrorIs(t, err, sentinel) require.Equal(t, 1, probe.rebuildCalls) require.Equal(t, []string{"repo-a", "repo-b"}, probe.rebuildPrefixes) @@ -465,7 +465,7 @@ func TestIndexMultiRepoRefFactFailureStopsGlobalConsumers(t *testing.T) { probe := &coldRefFactsProbeStore{Store: graph.New(), rebuildErr: sentinel} mi := newColdOrchestrationMulti(probe, zap.New(core)) - results, err := mi.indexMultiRepo(coldRefFactRepos(t, 2)) + results, err := mi.indexFixtureRepos(coldRefFactRepos(t, 2)) require.ErrorIs(t, err, sentinel) require.Len(t, results, 2, "successful parse results remain available with the completion error") require.Equal(t, 1, probe.rebuildCalls) diff --git a/internal/indexer/multi_fixture_test.go b/internal/indexer/multi_fixture_test.go new file mode 100644 index 000000000..2b21c49cf --- /dev/null +++ b/internal/indexer/multi_fixture_test.go @@ -0,0 +1,464 @@ +package indexer + +import ( + "context" + "fmt" + "os" + "path/filepath" + "sort" + "strings" + "sync" + "time" + + "go.uber.org/zap" + + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/graph" + "github.com/zzet/gortex/internal/pathkey" + "github.com/zzet/gortex/internal/reach" +) + +// These fixtures retain the atomic cold-cohort shape used by lane, topology, +// output-generation and resolver tests. They are deliberately test-only: +// production startup registers roots through TrackRepoCtx and coordinates its +// deferred stages in the daemon. External fixtures use that production path. +// The cohort fixture names the existing topology output entry, not a separate +// production mutation door. +// finishFixtureDeferredPasses completes the cold-index deferred tail without +// repeating the whole-graph cross-repository resolver. The receipt-guided tail +// already performs either exact cross-repository catch-up or a fail-closed full +// pass; only contract-bridge reconciliation remains afterward. +func (mi *MultiIndexer) finishFixtureDeferredPasses(ctx context.Context, completedIndexers []*Indexer) DeferredPassesResult { + result := mi.beginDeferredPasses(ctx, nil, completedIndexers, true).FinishTailResult() + mi.ReconcileContractEdges() + return result +} + +// IndexAll indexes all active repos concurrently. Each repo gets its own +// Indexer instance with repo-specific config. Returns per-repo IndexResults. +func (mi *MultiIndexer) IndexAll() (map[string]*IndexResult, error) { + return mi.indexFixtureScope("", "") +} + +// indexFixtureScope selects fixture roots with the same workspace/project +// precedence as registered repositories: entry override, workspace config, +// then prefix. Empty filters retain every active fixture root; a typo refuses +// an empty test graph. This is not a production startup selection API. +func (mi *MultiIndexer) indexFixtureScope(workspaceSlug, projectSlug string) (map[string]*IndexResult, error) { + repos := mi.configMgr.ActiveRepos() + if len(repos) == 0 { + return nil, nil + } + if workspaceSlug != "" || projectSlug != "" { + filtered := mi.filterFixtureReposByScope(repos, workspaceSlug, projectSlug) + if len(filtered) == 0 { + return nil, fmt.Errorf("scope filter matched zero of %d active repos (workspace=%q project=%q)", len(repos), workspaceSlug, projectSlug) + } + repos = filtered + } + + // indexFixtureRepos owns the complete coordinated pipeline, including the + // one contract reconciliation required before graph-wide derivation. Do + // not repeat it here: reconciliation evicts and rebuilds a generation and + // a second identical pass is pure global work. + // + // A lone repo takes the same path as any other — it is simply the first + // tracked repo, and its nodes carry its prefix like everyone else's. + return mi.indexFixtureRepos(repos) +} + +// filterFixtureReposByScope returns the subset of repos whose resolved +// workspace and project slugs match the supplied filters. Empty +// filters disable that axis. Loads each repo's `.gortex.yaml` first so +// resolution sees the workspace/project declared there — matching only +// against `RepoEntry.Workspace` would miss repos that declare their +// slug in their own config file (the typical case for first-party +// repos). +func (mi *MultiIndexer) filterFixtureReposByScope(repos []config.RepoEntry, workspaceSlug, projectSlug string) []config.RepoEntry { + if workspaceSlug == "" && projectSlug == "" { + return repos + } + out := make([]config.RepoEntry, 0, len(repos)) + for _, e := range repos { + absPath, err := filepath.Abs(e.Path) + if err != nil { + continue + } + prefix := config.ResolvePrefix(e) + mi.configMgr.LoadWorkspaceConfig(prefix, absPath) + cfg := mi.configMgr.GetRepoConfig(prefix) + entryCopy := e + if workspaceSlug != "" && resolveWorkspaceID(&entryCopy, cfg, prefix) != workspaceSlug { + continue + } + if projectSlug != "" && resolveProjectID(&entryCopy, cfg, prefix) != projectSlug { + continue + } + out = append(out, e) + } + return out +} + +// rebuildFixtureRefFacts seeds the durable resolved-reference sidecar once for +// the complete successful repository set. The backend implementation is a +// single set-oriented transaction; keeping the capability probe here avoids a +// per-repository persistence loop and leaves in-memory stores as a no-op. +func (mi *MultiIndexer) rebuildFixtureRefFacts(ctx context.Context, repoPrefixes []string) error { + if err := ctx.Err(); err != nil { + return err + } + rebuilder, ok := mi.graph.(graph.RefFactsRebuilder) + if !ok { + return nil + } + if err := rebuilder.RebuildRefFactsForRepos(repoPrefixes); err != nil { + return err + } + // The current backend contract is atomic but not interruptible in flight. + // Observing cancellation here prevents completion/global consumers from + // publishing a generation whose ref-fact boundary the caller abandoned. + return ctx.Err() +} + +// indexFixtureRepos indexes multiple repos concurrently with repo prefixes. +func (mi *MultiIndexer) indexFixtureRepos(repos []config.RepoEntry) (map[string]*IndexResult, error) { + type repoResult struct { + prefix string + result *IndexResult + idx *Indexer + meta *RepoMetadata + err error + } + + // Resolve each repo's prefix serially first: git-worktree instancing, + // the derived-name persistence, and the cross-batch collision guard + // must be deterministic, not raced across the parallel index + // goroutines below. The same loop builds the tracked-module map used + // for cross-repo dependency detection. + type resolvedRepo struct { + entry config.RepoEntry + prefix string + cfg *config.Config + absPath string + identity *RepoIdentity + } + var configLookup config.RepoPathLookup + if mi.configMgr != nil { + configLookup = mi.configMgr.Global().SnapshotRepoPaths() + } + moduleRoots := make(map[string]string) + resolved := make([]resolvedRepo, 0, len(repos)) + seenPrefix := make(map[string]string, len(repos)) // prefix → absPath + var resolveErrors []string + for _, entry := range repos { + absPath, err := filepath.Abs(entry.Path) + if err != nil { + resolveErrors = append(resolveErrors, fmt.Sprintf("resolving path %s: %v", entry.Path, err)) + continue + } + info, err := os.Stat(absPath) + if err != nil { + resolveErrors = append(resolveErrors, fmt.Sprintf("opening repository %s: %v", entry.Path, err)) + continue + } + if !info.IsDir() { + resolveErrors = append(resolveErrors, fmt.Sprintf("opening repository %s: not a directory", entry.Path)) + continue + } + identity, _ := DetectIdentity(absPath) + e := entry + prefix, cfg := mi.resolveTrackPrefix(&e, absPath, identity, configLookup, mi.configuredPrefixCheck()) + // Two distinct checkouts can still land on the same derived prefix + // (e.g. two worktrees declaring the same workspace). resolveTrackPrefix + // only sees already-tracked repos, so guard against collisions within + // this batch too. + if prev, ok := seenPrefix[prefix]; ok { + if pathkey.SamePathIdentity(prev, absPath) { + // Duplicate config entries for one repository must not launch + // concurrent raw workers behind a single held repository lane. + continue + } + prefix += "-" + shortPathHash(absPath) + e.Name = prefix + // This prefix is minted for THIS batch and is not the one the + // lifecycle registered an owner under, so the receipt opened for + // it below finds no owner and the repository is left unwitnessed + // (openSourceWitness). That is the honest direction — the nodes + // this pass writes carry the disambiguated prefix too, so no pin + // taken on the colliding prefix is reading them — but a request + // against the disambiguated prefix gets ErrBaseCorpusUnwitnessed + // rather than an exactness label until an owner is registered for + // the name this batch chose. + } + seenPrefix[prefix] = absPath + resolved = append(resolved, resolvedRepo{entry: e, prefix: prefix, cfg: cfg, absPath: absPath, identity: identity}) + moduleRoots[prefix] = absPath + } + trackedModules := readTrackedGoModules(moduleRoots) + for prefix, mod := range trackedModules { + mi.logger.Debug("tracked repo module", zap.String("repo", prefix), zap.String("module", mod)) + } + + if len(resolved) == 0 { + if len(resolveErrors) == 0 { + return map[string]*IndexResult{}, nil + } + sort.Strings(resolveErrors) + return nil, fmt.Errorf("all repos failed to index: %s", strings.Join(resolveErrors, "; ")) + } + + prefixes := make([]string, 0, len(resolved)) + for _, repo := range resolved { + prefixes = append(prefixes, repo.prefix) + } + // Freeze one batch-mode generation before acquiring any stable repository + // lane. This global gate -> sorted lanes order prevents a queued transition + // writer from owning the gate while waiting for a lane held by this batch. + mi.batchMutationGate.RLock() + defer mi.batchMutationGate.RUnlock() + + // One receipt per OUTPUT OWNER this batch writes, opened after every lane + // is held and before any payload moves. The whole batch is one decision: if + // a newer mutation takes over any of these owners while it runs, the batch + // cannot fulfil that owner's generation. + // + // Owners are deduplicated, and not by prefix. Two tracked entries can name + // one repository root (two config entries with different explicit Names and + // one Path: resolveTrackPrefix honours an explicit Name verbatim, so the + // prefix collision guard above never fires). legacyOutputTargetFor keys the + // owner on the root, so both entries name ONE generation-zero output. + // Issuing two receipts for it would make the second supersede the first and + // refuse a cold index that fully succeeded. + batchTargets := make([]OutputMutationTarget, 0, len(resolved)) + seenOutputOwner := make(map[string]bool, len(resolved)) + for _, repo := range resolved { + target := legacyOutputTargetFor(outputStoreIdentity(mi.graph), repo.prefix, repo.absPath, "prefix:"+repo.prefix) + if seenOutputOwner[target.OwnerKey] { + continue + } + seenOutputOwner[target.OwnerKey] = true + batchTargets = append(batchTargets, target) + } + + var finalResults map[string]*IndexResult + laneErr := mi.withRepositoryMutationLanes(context.Background(), prefixes, func() error { + receipts, receiptErr := mi.outputGenerationAuthority().BeginAll( + context.Background(), OutputEntryRepositoryTopology, batchTargets) + if receiptErr != nil { + return receiptErr + } + settled := false + defer func() { + if !settled { + receipts.Abandon() + } + }() + // Preventive half: refuse before the cold batch writes anything if any + // owner was already taken over while admission blocked on its gates. + if err := receipts.Current(); err != nil { + return err + } + batchErr := func() error { + // The one topology writer covers parse, publication, deferred/cross-repo + // tails, ref-facts, and global derivation. + batchMode := mi.currentBatchMode() + finishTopologyMutation := reach.BeginTopologyMutation(mi.graph) + defer finishTopologyMutation(true) + + pipelineResults, pipelineErr := func() (map[string]*IndexResult, error) { + resultCh := make(chan repoResult, len(resolved)) + var wg sync.WaitGroup + coordinatedBulk, _ := mi.graph.(graph.CoordinatedBulkLoader) + coordinatedBulkActive := coordinatedBulk != nil && coordinatedBulk.BeginCoordinatedBulkLoad() + defer func() { + if coordinatedBulkActive { + if err := coordinatedBulk.EndCoordinatedBulkLoad(); err != nil { + mi.logger.Error("multi-repo bulk-load cleanup failed", zap.Error(err)) + } + } + }() + + for _, rr := range resolved { + wg.Add(1) + go func(r resolvedRepo) { + defer wg.Done() + + idx := mi.newPerRepoIndexerGuardedWithMode(r.cfg.Index, batchMode) + idx.SetRepoPrefix(r.prefix) + entryCopy := r.entry + idx.SetWorkspaceID(resolveWorkspaceID(&entryCopy, r.cfg, r.prefix)) + idx.SetProjectID(resolveProjectID(&entryCopy, r.cfg, r.prefix)) + idx.SetTrackedRepoModules(trackedModules) + // Defer the per-repo cross-cutting passes (ResolveAll, + // semantic enrich, contract extract+commit) so they don't + // race against each other across goroutines on the shared + // graph. They run serially below via RunDeferredPasses after + // wg.Wait(). The graph-wide derivation passes run once after + // the loop via the shared global-pass pipeline. + idx.SetDeferResolve(true) + + result, err := idx.indexCtxRaw(WithIndexOrigin(context.Background(), "index_all"), r.absPath) + if err != nil { + idx.Close() + resultCh <- repoResult{prefix: r.prefix, err: fmt.Errorf("indexing %s: %w", r.absPath, err)} + return + } + if result == nil { + idx.Close() + resultCh <- repoResult{prefix: r.prefix, err: fmt.Errorf("indexing %s returned a nil result", r.absPath)} + return + } + result.RepoPrefix = r.prefix + + meta := &RepoMetadata{ + RepoPrefix: r.prefix, + RootPath: r.absPath, + Identity: r.identity, + LastIndexTime: time.Now(), + FileCount: result.FileCount, + NodeCount: result.NodeCount, + EdgeCount: result.EdgeCount, + ParseErrors: result.Errors, + FileMtimes: idx.publishFileMtimes(), + + IsWorktree: ResolveWorktree(r.absPath).IsWorktree, + } + + resultCh <- repoResult{prefix: r.prefix, result: result, idx: idx, meta: meta} + }(rr) + } + + go func() { + wg.Wait() + close(resultCh) + }() + + results := make(map[string]*IndexResult) + indexErrors := resolveErrors + completed := make([]repoResult, 0, len(resolved)) + + // Drain workers without holding the registry lock. Constructors and future + // worker tails may need read access to MultiIndexer state; pinning mi.mu + // across the channel range turns any such read into a producer/consumer + // deadlock and blocks unrelated registry readers for the whole warmup. + for rr := range resultCh { + if rr.err != nil { + mi.logger.Error("failed to index repo", zap.String("prefix", rr.prefix), zap.Error(rr.err)) + indexErrors = append(indexErrors, rr.err.Error()) + continue + } + completed = append(completed, rr) + results[rr.prefix] = rr.result + } + oldIndexers := make([]*Indexer, 0, len(completed)) + mi.mu.Lock() + for _, rr := range completed { + if old := mi.indexers[rr.prefix]; old != nil && old != rr.idx { + oldIndexers = append(oldIndexers, old) + } + mi.repos[rr.prefix] = rr.meta + mi.indexers[rr.prefix] = rr.idx + } + mi.mu.Unlock() + for _, old := range oldIndexers { + old.Close() + } + if coordinatedBulkActive { + if err := coordinatedBulk.EndCoordinatedBulkLoad(); err != nil { + return nil, fmt.Errorf("multi-repo bulk-load finalize: %w", err) + } + coordinatedBulkActive = false + } + for _, rr := range completed { + if err := mi.postPublishGoModules(context.Background(), rr.idx, true); err != nil { + mi.logger.Warn("post-publish Go module follow-up", zap.Error(err)) + } + } + + // Do not publish a completed pipeline when no repository reached the + // deferred stages. Besides making the failure deterministic, this prevents + // global passes from deriving edges from a partially-drained failed batch. + if len(indexErrors) > 0 && len(results) == 0 { + sort.Strings(indexErrors) + return nil, fmt.Errorf("all repos failed to index: %s", strings.Join(indexErrors, "; ")) + } + + // Complete cold multi-repo indexing through the same coordinated pipeline + // used by daemon warmup: + // 1. materialise go.mod contracts once, then run one shared base resolve; + // 2. enrich repositories in bounded language-aware batches (large Go + // repositories remain exclusive), committing contracts only after each + // batch drains; + // 3. use the mutation receipt to perform only the exact catch-up needed for + // semantic/contract mutations. + // + // The old loop called idx.RunDeferredPasses with + // skipResolveInDeferred=false, so every repository performed ResolveAll over + // the entire shared graph. At R repositories and E edges that was O(R*E) and + // was the dominant cold-index regression. runDeferredGoMod is generation- + // idempotent, so RunDeferredPassesAll does not repeat the pre-resolve work. + deferCtx := context.Background() + if err := mi.RunPreEnrichResolve(deferCtx, nil, nil); err != nil { + return results, fmt.Errorf("multi-repo pre-enrichment resolve: %w", err) + } + completedIndexers := make([]*Indexer, 0, len(completed)) + for _, rr := range completed { + completedIndexers = append(completedIndexers, rr.idx) + } + deferredResult := mi.finishFixtureDeferredPasses(deferCtx, completedIndexers) + mi.logger.Info("multi-repo coordinated deferred passes complete", + zap.Int("repos_indexed", len(results)), + zap.Int("repos_failed", len(indexErrors)), + zap.Int("enrich_scheduled", deferredResult.EnrichScheduled), + zap.Bool("exact_cross_repo_complete", deferredResult.ExactCrossRepoComplete)) + + // ResolveAll normally seeds ref_facts after a full resolve. The coordinated + // cold path intentionally bypasses per-repository ResolveAll, so seed the + // successful repository set once after every base, semantic catch-up, and + // cross-repository mutation has settled. Sorting makes the boundary stable + // for tracing/tests; SQLite consumes the whole slice in one transaction. + successfulPrefixes := make([]string, 0, len(results)) + for prefix := range results { + successfulPrefixes = append(successfulPrefixes, prefix) + } + sort.Strings(successfulPrefixes) + if err := mi.rebuildFixtureRefFacts(deferCtx, successfulPrefixes); err != nil { + return results, fmt.Errorf("multi-repo reference-fact rebuild: %w", err) + } + + // Graph-wide derivation passes run exactly once after every repo + // has been parsed, every per-repo and cross-repo resolver has lifted + // placeholder edges, and contract bridges are in place. RunDeferredPasses + // intentionally skips these so we don't pay an O(global) walk per + // repo (was the dominant cost at R≈100+). + mi.runGlobalGraphPassesTopologyHeld(context.Background(), nil, false) + + return results, nil + }() + finalResults = pipelineResults + return pipelineErr + }() + if batchErr != nil { + return batchErr + } + settled = true + // Fulfilment is refused when a newer mutation took over any of these + // owners while the batch ran; the batch then reports that refusal + // instead of announcing a completed cold index. + if err := receipts.Complete(); err != nil { + return err + } + // Every completed repository's journal is accepted, failed files + // included: the cold pass staged a failure envelope for each of them + // (indexCtxRaw), so the accepted namespace excludes exactly that set. + for prefix, result := range finalResults { + if result != nil { + if err := mi.acceptInstalledContractCoreInputs(context.Background(), prefix); err != nil { + return err + } + } + } + return nil + }) + return finalResults, laneErr +} diff --git a/internal/indexer/multi_prefix_collision_test.go b/internal/indexer/multi_prefix_collision_test.go index 79dd8101c..bd9c09e71 100644 --- a/internal/indexer/multi_prefix_collision_test.go +++ b/internal/indexer/multi_prefix_collision_test.go @@ -41,17 +41,16 @@ func TestRepositoryPrefixCollision(t *testing.T) { result, err = mi.TrackRepoCtx(t.Context(), entry) } if reconcile { - // Startup keeps the base outcome: not loaded, warning only. - require.NoError(t, err) - } else { - var collision *PrefixCollisionError - require.ErrorAs(t, err, &collision) - assert.Equal(t, "src", collision.Prefix) - assert.Equal(t, first, collision.ExistingPath) - assert.Equal(t, second, collision.Path) - assert.Contains(t, err.Error(), first) - assert.Contains(t, err.Error(), "gortex track --name ") + var skipped *RepositoryCollisionSkip + require.ErrorAs(t, err, &skipped, "reconcile must not certify a rejected root as a successful no-op") } + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision) + assert.Equal(t, "src", collision.Prefix) + assert.Equal(t, first, collision.ExistingPath) + assert.Equal(t, second, collision.Path) + assert.Contains(t, err.Error(), first) + assert.Contains(t, err.Error(), "gortex track --name ") assert.Nil(t, result) assert.Len(t, mi.AllMetadata(), 1) assert.Equal(t, ownerMeta, *mi.GetMetadata("src")) @@ -189,12 +188,16 @@ func TestRepositoryPrefixCollisionAfterCoordination(t *testing.T) { result, err = mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: second}) } require.True(t, admitted) - if samePath || reconcile { + if samePath { require.NoError(t, err) } else { var collision *PrefixCollisionError require.ErrorAs(t, err, &collision) assert.Equal(t, first, collision.ExistingPath) + if reconcile { + var skipped *RepositoryCollisionSkip + require.ErrorAs(t, err, &skipped, "the coordinated loser must remain a typed unchanged skip") + } } assert.Nil(t, result) assert.Len(t, mi.AllMetadata(), 1) diff --git a/internal/indexer/multi_scoped_test.go b/internal/indexer/multi_scoped_test.go index fc0281449..e019b7cc4 100644 --- a/internal/indexer/multi_scoped_test.go +++ b/internal/indexer/multi_scoped_test.go @@ -14,11 +14,11 @@ import ( "github.com/zzet/gortex/internal/search" ) -// TestMultiIndexer_IndexScoped_WorkspaceFromGortexYAML verifies that -// IndexScoped picks up the workspace slug declared in a repo's own +// TestMultiIndexer_FixtureScope_WorkspaceFromGortexYAML verifies that +// indexFixtureScope picks up the workspace slug declared in a repo's own // `.gortex.yaml` — the typical first-party setup. Repos with a // different workspace slug must be excluded. -func TestMultiIndexer_IndexScoped_WorkspaceFromGortexYAML(t *testing.T) { +func TestMultiIndexer_FixtureScope_WorkspaceFromGortexYAML(t *testing.T) { repoA := setupRepoDir(t, "repo-a") repoB := setupRepoDir(t, "repo-b") require.NoError(t, os.WriteFile(filepath.Join(repoA, ".gortex.yaml"), @@ -42,7 +42,7 @@ func TestMultiIndexer_IndexScoped_WorkspaceFromGortexYAML(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - results, err := mi.IndexScoped("alpha", "") + results, err := mi.indexFixtureScope("alpha", "") require.NoError(t, err) require.Len(t, results, 1) assert.Contains(t, results, "repo-a") @@ -50,11 +50,11 @@ func TestMultiIndexer_IndexScoped_WorkspaceFromGortexYAML(t *testing.T) { assert.Nil(t, mi.GetMetadata("repo-b")) } -// TestMultiIndexer_IndexScoped_RepoEntryOverride verifies that a +// TestMultiIndexer_FixtureScope_RepoEntryOverride verifies that a // global-config RepoEntry.Workspace override beats the repo's own // `.gortex.yaml::workspace` slug — the user pinning OSS / read-only // repos into a workspace without leaving an artifact in them. -func TestMultiIndexer_IndexScoped_RepoEntryOverride(t *testing.T) { +func TestMultiIndexer_FixtureScope_RepoEntryOverride(t *testing.T) { repoA := setupRepoDir(t, "repo-a") require.NoError(t, os.WriteFile(filepath.Join(repoA, ".gortex.yaml"), []byte("workspace: upstream\n"), 0o644)) @@ -75,19 +75,19 @@ func TestMultiIndexer_IndexScoped_RepoEntryOverride(t *testing.T) { mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) // Override wins over .gortex.yaml: "upstream" matches nothing. - _, err = mi.IndexScoped("upstream", "") + _, err = mi.indexFixtureScope("upstream", "") require.Error(t, err) // "local-pin" matches the override. - results, err := mi.IndexScoped("local-pin", "") + results, err := mi.indexFixtureScope("local-pin", "") require.NoError(t, err) require.Len(t, results, 1) } -// TestMultiIndexer_IndexScoped_FallsBackToRepoPrefix verifies the +// TestMultiIndexer_FixtureScope_FallsBackToRepoPrefix verifies the // default: repos with no `.gortex.yaml::workspace` and no // RepoEntry.Workspace use the repo prefix as their workspace slug. -func TestMultiIndexer_IndexScoped_FallsBackToRepoPrefix(t *testing.T) { +func TestMultiIndexer_FixtureScope_FallsBackToRepoPrefix(t *testing.T) { repoA := setupRepoDir(t, "repo-a") repoB := setupRepoDir(t, "repo-b") @@ -107,17 +107,17 @@ func TestMultiIndexer_IndexScoped_FallsBackToRepoPrefix(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - results, err := mi.IndexScoped("repo-b", "") + results, err := mi.indexFixtureScope("repo-b", "") require.NoError(t, err) require.Len(t, results, 1) assert.Contains(t, results, "repo-b") } -// TestMultiIndexer_IndexScoped_ProjectNarrowsInsideWorkspace verifies +// TestMultiIndexer_FixtureScope_ProjectNarrowsInsideWorkspace verifies // that workspace + project filters are AND-combined. Two repos in the // same workspace but different projects → only the matching project // gets indexed. -func TestMultiIndexer_IndexScoped_ProjectNarrowsInsideWorkspace(t *testing.T) { +func TestMultiIndexer_FixtureScope_ProjectNarrowsInsideWorkspace(t *testing.T) { repoA := setupRepoDir(t, "repo-a") repoB := setupRepoDir(t, "repo-b") require.NoError(t, os.WriteFile(filepath.Join(repoA, ".gortex.yaml"), @@ -141,16 +141,16 @@ func TestMultiIndexer_IndexScoped_ProjectNarrowsInsideWorkspace(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - results, err := mi.IndexScoped("shared", "api") + results, err := mi.indexFixtureScope("shared", "api") require.NoError(t, err) require.Len(t, results, 1) assert.Contains(t, results, "repo-a") } -// TestMultiIndexer_IndexScoped_NoMatchErrors verifies that a typo'd +// TestMultiIndexer_FixtureScope_NoMatchErrors verifies that a typo'd // filter surfaces as an error rather than producing a silent empty // graph. -func TestMultiIndexer_IndexScoped_NoMatchErrors(t *testing.T) { +func TestMultiIndexer_FixtureScope_NoMatchErrors(t *testing.T) { repoA := setupRepoDir(t, "repo-a") tmpCfg := filepath.Join(t.TempDir(), "config.yaml") @@ -166,16 +166,16 @@ func TestMultiIndexer_IndexScoped_NoMatchErrors(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexScoped("nonexistent", "") + _, err = mi.indexFixtureScope("nonexistent", "") require.Error(t, err) assert.Contains(t, err.Error(), "scope filter matched zero") assert.Equal(t, 0, g.NodeCount(), "no nodes should be added on a zero-match scope") } -// TestMultiIndexer_IndexScoped_EmptyFiltersEqualsIndexAll verifies +// TestMultiIndexer_FixtureScope_EmptyFiltersEqualsIndexAll verifies // that empty/empty filters degrade to IndexAll behaviour: every // active repo is indexed. -func TestMultiIndexer_IndexScoped_EmptyFiltersEqualsIndexAll(t *testing.T) { +func TestMultiIndexer_FixtureScope_EmptyFiltersEqualsIndexAll(t *testing.T) { repoA := setupRepoDir(t, "repo-a") repoB := setupRepoDir(t, "repo-b") require.NoError(t, os.WriteFile(filepath.Join(repoA, ".gortex.yaml"), @@ -197,7 +197,7 @@ func TestMultiIndexer_IndexScoped_EmptyFiltersEqualsIndexAll(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - results, err := mi.IndexScoped("", "") + results, err := mi.indexFixtureScope("", "") require.NoError(t, err) assert.Len(t, results, 2) assert.Contains(t, results, "repo-a") diff --git a/internal/indexer/output_generation_authority_test.go b/internal/indexer/output_generation_authority_test.go index 1e6a0e747..66765567a 100644 --- a/internal/indexer/output_generation_authority_test.go +++ b/internal/indexer/output_generation_authority_test.go @@ -298,7 +298,6 @@ func TestEveryProductionMutationEntryPointIsRegistered(t *testing.T) { // same function still names an entry. required := []mutationDoor{ {file: "contract_baseline_reconcile.go", fn: "reconcilePrimaryContractBaseline", door: "withRepositoryMutationLanes", entries: []string{"OutputEntryContractBaseline"}}, - {file: "multi.go", fn: "indexMultiRepo", door: "withRepositoryMutationLanes", entries: []string{"OutputEntryIndexMultiRepo"}}, {file: "multi.go", fn: "IndexRepo", door: "runExclusive", entries: []string{"OutputEntryIndexRepo"}}, {file: "multi.go", fn: "incrementalDiscoverRepo", door: "runExclusive", entries: []string{"OutputEntryIncrementalDiscoverRepo"}}, {file: "repository_topology_batch.go", fn: "coordinateRepositoryTopologyMutation", door: "runExclusiveLaneOnly", entries: []string{"OutputEntryRepositoryTopology"}}, @@ -407,7 +406,6 @@ func outputMutationEntryValue(identifier string) (string, bool) { "OutputEntryReresolveFileScoped": OutputEntryReresolveFileScoped, "OutputEntryDeferredPasses": OutputEntryDeferredPasses, "OutputEntryRepositoryTopology": OutputEntryRepositoryTopology, - "OutputEntryIndexMultiRepo": OutputEntryIndexMultiRepo, "OutputEntryIndexRepo": OutputEntryIndexRepo, "OutputEntryIncrementalDiscoverRepo": OutputEntryIncrementalDiscoverRepo, "OutputEntryGitWatcherFinalize": OutputEntryGitWatcherFinalize, @@ -416,6 +414,7 @@ func outputMutationEntryValue(identifier string) (string, bool) { "OutputEntryWatcherPatchGraph": OutputEntryWatcherPatchGraph, "OutputEntryWatcherEnqueueReresolve": OutputEntryWatcherEnqueueReresolve, "OutputEntryCheckoutSourceMutation": OutputEntryCheckoutSourceMutation, + "OutputEntryPromotionPublication": OutputEntryPromotionPublication, "OutputEntryRepositoryReconcileLane": OutputEntryRepositoryReconcileLane, "OutputEntryEnrichmentCorpus": OutputEntryEnrichmentCorpus, "OutputEntryContractBaseline": OutputEntryContractBaseline, @@ -619,7 +618,7 @@ func TestSupersededWorkIsRefusedBeforeItWrites(t *testing.T) { } // The batch form refuses the same way, before the batch writes anything. - batchOlder, err := authority.Begin(ctx, OutputEntryIndexMultiRepo, target) + batchOlder, err := authority.Begin(ctx, OutputEntryRepositoryTopology, target) if err != nil { t.Fatalf("batch receipt: %v", err) } @@ -627,7 +626,7 @@ func TestSupersededWorkIsRefusedBeforeItWrites(t *testing.T) { if err := batch.Current(); err != nil { t.Fatalf("an uncontested batch must be current: %v", err) } - contender, err := authority.Begin(ctx, OutputEntryIndexMultiRepo, target) + contender, err := authority.Begin(ctx, OutputEntryRepositoryTopology, target) if err != nil { t.Fatalf("batch contender: %v", err) } @@ -1429,9 +1428,9 @@ func TestColdBatchOpensOneReceiptPerRepositoryOwner(t *testing.T) { t.Fatalf("IndexAll: %v", err) } stats := authority.Stats() - if got := stats.Entries[OutputEntryIndexMultiRepo]; got != 2 { + if got := stats.Entries[OutputEntryRepositoryTopology]; got != 2 { t.Fatalf("the cold batch opened %d receipts under %s, want one per repository owner (2)", - got, OutputEntryIndexMultiRepo) + got, OutputEntryRepositoryTopology) } if stats.Superseded != 0 || stats.Settled != stats.Issued { t.Fatalf("every batch receipt must settle cleanly: %+v", stats) diff --git a/internal/indexer/prefix_config_witness_test.go b/internal/indexer/prefix_config_witness_test.go new file mode 100644 index 000000000..9d71a834c --- /dev/null +++ b/internal/indexer/prefix_config_witness_test.go @@ -0,0 +1,36 @@ +package indexer + +import ( + "context" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestPreparedPrefixConfigDoesNotPublishAcrossOwnerReplacement(t *testing.T) { + owner, candidate := setupRepoDir(t, "src"), setupRepoDir(t, "src") + writeFile(t, filepath.Join(owner, ".gortex.yaml"), "workspace: owner\nexclude:\n - owner-only\n") + writeFile(t, filepath.Join(candidate, ".gortex.yaml"), "workspace: candidate\nexclude:\n - rejected-only\n") + _, mi, cm := newWorktreeTestIndexer(t) + t.Cleanup(func() { require.NoError(t, mi.Close(context.Background())) }) + cm.LoadWorkspaceConfig("src", owner) + before, revision := cm.GetRepoConfig("src"), cm.Revision() + entry := config.RepoEntry{Path: candidate, Name: "src"} + // The configured-owner check is already an explicit preparation seam. + // Replace the previously absent live owner between observation and final + // in-memory publication: no hook runs under the admission lock. + check := func(string, string) error { + mi.mu.Lock() + mi.repos["src"] = &RepoMetadata{RepoPrefix: "src", RootPath: owner} + mi.mu.Unlock() + return nil + } + _, err := mi.resolveTrackPrefixContext(t.Context(), &entry, candidate, nil, nil, check, + time.Now().Add(prefixOwnershipTimeout), nil) + require.ErrorIs(t, err, errPrefixOwnerChanged) + require.Equal(t, before, cm.GetRepoConfig("src")) + require.Equal(t, revision, cm.Revision(), "a stale preparation must not overwrite the owner's workspace or gitignore root") +} diff --git a/internal/indexer/prefix_ownership.go b/internal/indexer/prefix_ownership.go new file mode 100644 index 000000000..a8da85371 --- /dev/null +++ b/internal/indexer/prefix_ownership.go @@ -0,0 +1,297 @@ +package indexer + +import ( + "context" + "errors" + "fmt" + "os" + "sort" + "sync" + "time" + + "github.com/zzet/gortex/internal/pathkey" +) + +const prefixOwnershipTimeout = 2 * time.Second + +var ( + errPrefixStatUnavailable = errors.New("repository identity stat allowance is occupied") + errPrefixOwnerChanged = errors.New("repository prefix owner changed during identity observation; retry the operation") + errPrefixConfigChanged = errors.New("repository configuration changed during prefix preparation; retry the operation") + errPrefixIdentityUnknown = errors.New("repository root identity could not be established") +) + +// ErrRepoIdentityPending is retryable uncertainty about a configured physical +// root. No track/config/catalog publication is authorized by this outcome. +var ErrRepoIdentityPending = errPrefixIdentityUnknown + +// prefixStatAllowance is deliberately not a filesystem worker service. Native +// filesystem calls have no portable cancellation API. At most two observations per indexer +// may remain physically stuck; further callers abstain without starting work. +// Each worker captures only immutable path/function/channel inputs, never an +// indexer, store or publication callback. Late results fit in a private buffer +// and are discarded. Close therefore does NOT join a hung OS Stat, and makes no +// physical OS cancellation guarantee; these workers cannot access the store. +type prefixStatAllowance struct { + once sync.Once + slots chan struct{} + stat func(string) (os.FileInfo, error) // nil in production; instance-local test seam +} + +type prefixStatResult struct { + info os.FileInfo + err error +} + +func (a *prefixStatAllowance) observe(ctx context.Context, path string) prefixStatResult { + if err := ctx.Err(); err != nil { + return prefixStatResult{err: err} + } + a.once.Do(func() { a.slots = make(chan struct{}, 2) }) + select { + case a.slots <- struct{}{}: + default: + return prefixStatResult{err: errPrefixStatUnavailable} + } + stat := a.stat + if stat == nil { + stat = os.Stat + } + slots := a.slots + done := make(chan prefixStatResult, 1) + go func() { + defer func() { <-slots }() + info, err := stat(path) + done <- prefixStatResult{info: info, err: err} + }() + select { + case <-ctx.Done(): + return prefixStatResult{err: ctx.Err()} + case result := <-done: + // Do not accept a late observation when deadline/result race. + if err := ctx.Err(); err != nil { + return prefixStatResult{err: err} + } + return result + } +} + +// A witness copies identity strings under mu. Its metadata pointer is used +// only for equality outside the lock, never dereferenced there. Replacing an +// entry at the same root (ABA by spelling) must invalidate the observation. +type prefixOwnerWitness struct { + prefix string + root string + meta *RepoMetadata +} + +type prefixOwnerSnapshot struct { + requested string + root string + owner prefixOwnerWitness + aliases []prefixOwnerWitness +} + +// prefixOwnerSnapshotLocked performs no filesystem operations. Only the +// requested owner and cheap lexical same-root candidates can require stats; +// it never stats every registered root on a track. +func (mi *MultiIndexer) prefixOwnerSnapshotLocked(prefix, root string) prefixOwnerSnapshot { + out := prefixOwnerSnapshot{requested: prefix, root: root} + if owner := mi.repos[prefix]; owner != nil { + out.owner = prefixOwnerWitness{prefix: prefix, root: owner.RootPath, meta: owner} + } + for name, meta := range mi.repos { + if meta != nil && pathkey.EqualPaths(meta.RootPath, root) { + out.aliases = append(out.aliases, prefixOwnerWitness{prefix: name, root: meta.RootPath, meta: meta}) + } + } + sort.Slice(out.aliases, func(i, j int) bool { return out.aliases[i].prefix < out.aliases[j].prefix }) + return out +} + +func (s prefixOwnerSnapshot) matchesLocked(mi *MultiIndexer) bool { + next := mi.prefixOwnerSnapshotLocked(s.requested, s.root) + if s.owner != next.owner || len(s.aliases) != len(next.aliases) { + return false + } + for i := range s.aliases { + if s.aliases[i] != next.aliases[i] { + return false + } + } + return true +} + +type prefixOwnershipObservation struct { + snapshot prefixOwnerSnapshot + registeredPrefix string + candidate prefixStatResult + owner prefixStatResult + collision *PrefixCollisionError +} + +// observePrefixOwnership returns a witnessed decision, not a registration +// reservation. A caller that publishes under mu must still check matchesLocked +// there. The stable repository mutation lane retains its own admission recheck. +// Both stats and one owner-replacement retry share this ONE deadline. +func (mi *MultiIndexer) observePrefixOwnership(ctx context.Context, prefix, root string) (prefixOwnershipObservation, error) { + return mi.observePrefixOwnershipBefore(ctx, prefix, root, time.Now().Add(prefixOwnershipTimeout), nil) +} + +// before is shared with candidate validation during one preparation. A later +// coordinated re-admission receives its own budget, not an expired deadline +// from time spent waiting for the repository's stable mutation lane. +func (mi *MultiIndexer) observePrefixOwnershipBefore(ctx context.Context, prefix, root string, before time.Time, candidate *prefixStatResult) (prefixOwnershipObservation, error) { + if err := ctx.Err(); err != nil { + return prefixOwnershipObservation{}, err + } + budget, cancel := context.WithDeadline(ctx, before) + defer cancel() + for attempt := 0; attempt < 2; attempt++ { + mi.mu.RLock() + snapshot := mi.prefixOwnerSnapshotLocked(prefix, root) + mi.mu.RUnlock() + observation, err := mi.samplePrefixOwnership(budget, snapshot, candidate) + if ctx.Err() != nil { + return prefixOwnershipObservation{}, ctx.Err() + } + mi.mu.RLock() + current := snapshot.matchesLocked(mi) + mi.mu.RUnlock() + if current { + return observation, err + } + // Do not reuse FileInfo across a replacement, even when the new owner + // has the same spelling. Resample under the original remaining budget. + candidate = nil + } + return prefixOwnershipObservation{}, errPrefixOwnerChanged +} + +func (mi *MultiIndexer) samplePrefixOwnership(ctx context.Context, snapshot prefixOwnerSnapshot, candidate *prefixStatResult) (prefixOwnershipObservation, error) { + out := prefixOwnershipObservation{snapshot: snapshot} + for _, alias := range snapshot.aliases { + if alias.root == snapshot.root { + out.registeredPrefix = alias.prefix + return out, nil // byte-identical registered identity needs no Stat + } + } + if snapshot.owner.meta == nil && len(snapshot.aliases) == 0 { + return out, nil + } + if candidate != nil { + out.candidate = *candidate + } else { + out.candidate = mi.prefixStats.observe(ctx, snapshot.root) + } + owners := make(map[string]prefixStatResult, len(snapshot.aliases)+1) + ownerStat := func(root string) prefixStatResult { + if result, ok := owners[root]; ok { + return result + } + result := mi.prefixStats.observe(ctx, root) + owners[root] = result + return result + } + for _, alias := range snapshot.aliases { + owner := ownerStat(alias.root) + if sameObservedDirectory(out.candidate, owner) { + out.registeredPrefix = alias.prefix + return out, nil + } + } + if snapshot.owner.meta == nil { + if out.candidate.err != nil { + return out, fmt.Errorf("%w: %w", errPrefixIdentityUnknown, out.candidate.err) + } + if out.candidate.info == nil { + return out, errPrefixIdentityUnknown + } + // A folded candidate with unavailable filesystem identity cannot be + // certified distinct merely because the requested prefix is unused. + for _, alias := range snapshot.aliases { + owner := ownerStat(alias.root) + if owner.err != nil { + return out, fmt.Errorf("%w: %w", errPrefixIdentityUnknown, owner.err) + } + if owner.info == nil { + return out, errPrefixIdentityUnknown + } + } + return out, nil + } + out.owner = ownerStat(snapshot.owner.root) + if sameObservedDirectory(out.candidate, out.owner) { + out.registeredPrefix = snapshot.owner.prefix + return out, nil + } + out.collision = newObservedPrefixCollision(snapshot.requested, snapshot.owner.root, snapshot.root, out.owner) + return out, out.collision +} + +func newObservedPrefixCollision(prefix, owner, root string, observed prefixStatResult) *PrefixCollisionError { + state := "unknown" + if errors.Is(observed.err, os.ErrNotExist) { + state = "missing" + } else if observed.err == nil && observed.info != nil { + state = "present" + } + return &PrefixCollisionError{Prefix: prefix, ExistingPath: owner, Path: root, OwnerMissing: state == "missing", OwnerState: state} +} + +// registeredPrefixForExactRoot deliberately does not use ResolvePrefix: a +// registered ancestor is useful for reads, but cannot authorize recording a +// new checkout or graph binding for a descendant that was never indexed. +func (mi *MultiIndexer) registeredPrefixForExactRoot(ctx context.Context, requested, root string) (string, error) { + observation, err := mi.observePrefixOwnership(ctx, requested, root) + if err != nil { + return "", err + } + if observation.registeredPrefix != "" { + return observation.registeredPrefix, nil + } + return "", newObservedPrefixCollision(requested, observation.snapshot.owner.root, root, + prefixStatResult{err: errPrefixIdentityUnknown}) +} + +// prefixOwnerForRoot is stricter than a same-root registration under ANY name: +// promotion and rollback need this specific prefix's corpus to be the root's. +func (mi *MultiIndexer) prefixOwnerForRoot(ctx context.Context, prefix, root string) (prefixOwnerWitness, error) { + observation, err := mi.observePrefixOwnership(ctx, prefix, root) + if err != nil { + return prefixOwnerWitness{}, err + } + owner := observation.snapshot.owner + if owner.meta != nil && (owner.root == root || observation.registeredPrefix == prefix) { + return owner, nil + } + return prefixOwnerWitness{}, newObservedPrefixCollision(prefix, owner.root, root, observation.owner) +} + +// Callers hold mi.mu. The comparison is memory-only, including the original +// immutable root string; no availability probe runs while closing admission. +func (mi *MultiIndexer) validatePrefixCleanupOwnerLocked(prefix string, expected *prefixOwnerWitness) error { + if expected == nil { + return nil // ordinary authoritative untrack keeps its existing contract + } + current := mi.repos[prefix] + if current != nil && current != expected.meta { + if current.RootPath != expected.root { + return newObservedPrefixCollision(prefix, current.RootPath, expected.root, prefixStatResult{err: errPrefixOwnerChanged}) + } + return errPrefixOwnerChanged + } + owned := current + if state := mi.pendingRepositoryUntracks[prefix]; state != nil { + owned = state.metadata // retained cleanup owns this immutable snapshot + } + if prefix != expected.prefix || owned == nil || owned != expected.meta || owned.RootPath != expected.root { + return errPrefixOwnerChanged + } + return nil +} + +func sameObservedDirectory(a, b prefixStatResult) bool { + return a.err == nil && b.err == nil && a.info != nil && b.info != nil && + a.info.IsDir() && b.info.IsDir() && os.SameFile(a.info, b.info) +} diff --git a/internal/indexer/prefix_ownership_test.go b/internal/indexer/prefix_ownership_test.go new file mode 100644 index 000000000..548f2d04b --- /dev/null +++ b/internal/indexer/prefix_ownership_test.go @@ -0,0 +1,180 @@ +package indexer + +import ( + "context" + "errors" + "os" + "path/filepath" + "sort" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestHungOwnerStatBounded(t *testing.T) { + owner, candidate := t.TempDir(), t.TempDir() + metadata := &RepoMetadata{RepoPrefix: "src", RootPath: owner} + mi := &MultiIndexer{repos: map[string]*RepoMetadata{"src": metadata}} + entered, release, physicallyReturned := make(chan struct{}), make(chan struct{}), make(chan struct{}) + var releaseOnce sync.Once + unblock := func() { releaseOnce.Do(func() { close(release) }) } + t.Cleanup(func() { + unblock() + require.Eventually(t, func() bool { return len(mi.prefixStats.slots) == 0 }, time.Second, time.Millisecond) + }) + mi.prefixStats.stat = func(path string) (os.FileInfo, error) { + if path == owner { + close(entered) + <-release // Models an OS Stat that cannot be canceled. + defer close(physicallyReturned) + } + return os.Stat(path) + } + result := make(chan error, 1) + started := time.Now() + go func() { + _, err := mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: candidate, Name: "src"}) + result <- err + }() + select { + case <-entered: + case <-time.After(5 * time.Second): + t.Fatal("owner Stat did not enter") + } + latencies := make([]time.Duration, 200) + for i := range latencies { + start := time.Now() + got := mi.GetMetadata("src") + latencies[i] = time.Since(start) + require.Same(t, metadata, got) + } + sort.Slice(latencies, func(i, j int) bool { return latencies[i] < latencies[j] }) + require.Less(t, latencies[197], 10*time.Millisecond, "GetMetadata must not queue behind the blocked filesystem call") + select { + case err := <-result: + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision) + require.False(t, collision.OwnerMissing, "deadline is unknown, never missing-owner evidence") + require.Less(t, time.Since(started), 2500*time.Millisecond) + case <-time.After(3 * time.Second): + t.Fatal("ownership observation exceeded its shared two-second budget") + } + select { + case <-physicallyReturned: + t.Fatal("the caller deadline must not be described as physical OS completion") + default: + } + require.Same(t, metadata, mi.GetMetadata("src"), "a retired stat has no registry/store publication") +} + +func TestPrefixStatAllowanceDiscardsLateResults(t *testing.T) { + var allowance prefixStatAllowance + entered := make(chan struct{}, 2) + release := make(chan struct{}) + var releases sync.Once + unblock := func() { releases.Do(func() { close(release) }) } + var calls atomic.Int32 + allowance.stat = func(string) (os.FileInfo, error) { + calls.Add(1) + entered <- struct{}{} + <-release + return nil, os.ErrNotExist + } + t.Cleanup(func() { + unblock() + require.Eventually(t, func() bool { return len(allowance.slots) == 0 }, time.Second, time.Millisecond) + }) + ctx, cancel := context.WithCancel(t.Context()) + defer cancel() + results := make(chan prefixStatResult, 2) + for i := 0; i < 2; i++ { + go func() { results <- allowance.observe(ctx, "immutable-root") }() + } + for i := 0; i < 2; i++ { + select { + case <-entered: + case <-time.After(5 * time.Second): + t.Fatal("bounded stat allowance did not start its admitted worker") + } + } + third := allowance.observe(t.Context(), "another-root") + require.ErrorIs(t, third.err, errPrefixStatUnavailable) + require.Equal(t, int32(2), calls.Load(), "saturation must not launch another physically hung call") + cancel() + for i := 0; i < 2; i++ { + select { + case result := <-results: + require.ErrorIs(t, result.err, context.Canceled) + case <-time.After(time.Second): + t.Fatal("retired probe waited for the physical stat") + } + } + unblock() + require.Eventually(t, func() bool { return len(allowance.slots) == 0 }, time.Second, time.Millisecond, + "late result delivery must fit its buffer and release its allowance") +} + +func TestPrefixOwnerWitnessReplacementIsResampledOnce(t *testing.T) { + for _, unstable := range []bool{false, true} { + t.Run(map[bool]string{false: "one replacement", true: "unstable twice"}[unstable], func(t *testing.T) { + first, next, candidate := t.TempDir(), t.TempDir(), t.TempDir() + mi := &MultiIndexer{repos: map[string]*RepoMetadata{"src": {RepoPrefix: "src", RootPath: first}}} + var ownerCalls, candidateCalls atomic.Int32 + mi.prefixStats.stat = func(path string) (os.FileInfo, error) { + if path == candidate { + candidateCalls.Add(1) + } else { + n := ownerCalls.Add(1) + if n == 1 || unstable { + mi.mu.Lock() + // Replacement at the same prefix has a distinct witness, + // including when the spelling returns to its first root. + root := next + if n == 2 { + root = first + } + mi.repos["src"] = &RepoMetadata{RepoPrefix: "src", RootPath: root} + mi.mu.Unlock() + } + } + return os.Stat(path) + } + _, err := mi.observePrefixOwnership(t.Context(), "src", candidate) + require.Equal(t, int32(2), ownerCalls.Load()) + require.Equal(t, int32(2), candidateCalls.Load(), "replacement must not reuse the first candidate FileInfo") + if unstable { + require.ErrorIs(t, err, errPrefixOwnerChanged) + } else { + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision) + require.Equal(t, next, collision.ExistingPath) + } + }) + } +} + +func TestPrefixOwnershipPreservesExactAndPhysicalAliases(t *testing.T) { + root := t.TempDir() + mi := &MultiIndexer{repos: map[string]*RepoMetadata{ + "named-owner": {RepoPrefix: "named-owner", RootPath: root}, + "src": {RepoPrefix: "src", RootPath: t.TempDir()}, + }} + mi.prefixStats.stat = func(string) (os.FileInfo, error) { + return nil, errors.New("byte-identical registered root must not stat") + } + observation, err := mi.observePrefixOwnership(t.Context(), "src", root) + require.NoError(t, err) + require.Equal(t, "named-owner", observation.registeredPrefix, "same root under another name wins over a requested-prefix collision") + alias := filepath.Join(t.TempDir(), "src") + if err := os.Symlink(root, alias); err != nil { + t.Skipf("physical symlink control unavailable: %v", err) + } + mi = &MultiIndexer{repos: map[string]*RepoMetadata{"src": {RepoPrefix: "src", RootPath: root}}} + observation, err = mi.observePrefixOwnership(t.Context(), "src", alias) + require.NoError(t, err) + require.Equal(t, "src", observation.registeredPrefix) +} diff --git a/internal/indexer/repo_load_collision_test.go b/internal/indexer/repo_load_collision_test.go new file mode 100644 index 000000000..7ee51d459 --- /dev/null +++ b/internal/indexer/repo_load_collision_test.go @@ -0,0 +1,55 @@ +package indexer + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestRepoLoadCollisionPrecedesQueuedStateFromOneSnapshot(t *testing.T) { + owner, loser := filepath.Join(t.TempDir(), "src"), filepath.Join(t.TempDir(), "src") + for _, root := range []string{owner, loser} { + require.NoError(t, os.MkdirAll(root, 0o755)) + } + mi := newRepoLoadTestMulti(t) + mi.ExpectRepos([]config.RepoEntry{{Path: owner}, {Path: loser}}) + before := mi.repoLoads.load() + mi.markRepoRegistered("src", owner) + rec, ok := mi.RepoLoadFor(loser) + require.True(t, ok) + require.Equal(t, "prefix_collision", rec.ErrorCode) + require.Equal(t, RepoLoadFailed, rec.State) + require.Equal(t, owner, rec.OwnerPath) + require.Equal(t, "present", rec.OwnerState) + old, ok := before.admissionForPath(loser) + require.True(t, ok) + require.Empty(t, old.ErrorCode, "old readers cannot combine a later owner with their old identity set") + raw := mi.repoLoads.load() + i, ok := raw.find(loser) + require.True(t, ok) + require.Equal(t, RepoLoadQueued, raw.records[i].State, "the producer retains its actual queued state") +} + +func TestRepoLoadCollisionDoesNotGuessAliasOrLinkedNaming(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + owner := f.gitRepo("src") + linked := filepath.Join(t.TempDir(), "src") + runGit(t, owner, "worktree", "add", "-q", "-b", "review", linked) + f.mi.ExpectRepos([]config.RepoEntry{{Path: owner}, {Path: linked}}) + f.mi.markRepoRegistered("src", owner) + rec, ok := f.mi.RepoLoadFor(linked) + require.True(t, ok) + require.Empty(t, rec.ErrorCode, "linked worktree basename cannot override lifecycle admin naming") + alias := filepath.Join(t.TempDir(), "src") + if err := os.Symlink(owner, alias); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + f.mi.ExpectRepos([]config.RepoEntry{{Path: owner}, {Path: linked}, {Path: alias}}) + rec, ok = f.mi.RepoLoadFor(alias) + require.True(t, ok) + require.Empty(t, rec.ErrorCode, "a physical alias never collides with its own root") +} diff --git a/internal/indexer/repo_load_identity_test.go b/internal/indexer/repo_load_identity_test.go new file mode 100644 index 000000000..921b3b374 --- /dev/null +++ b/internal/indexer/repo_load_identity_test.go @@ -0,0 +1,224 @@ +package indexer + +import ( + "os" + "path/filepath" + "runtime" + "syscall" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/pathkey" +) + +func TestRepoIdentityColdAliasPublication(t *testing.T) { + base := t.TempDir() + physical := filepath.Join(base, "physical-repository") + alias := filepath.Join(base, "different-configured-name") + require.NoError(t, os.MkdirAll(filepath.Join(physical, ".git"), 0755)) + require.NoError(t, os.Symlink(physical, alias)) + mi := newRepoLoadTestMulti(t) + entries := []config.RepoEntry{{Path: alias}} + mi.DeclareRepos(entries) + require.True(t, mi.RepoIdentityPending()) + _, known := mi.PendingRepoForPath(physical) + require.False(t, known, "an unresolved declaration cannot invent ownership") + mi.ExpectRepos(entries) + require.False(t, mi.RepoIdentityPending()) + rec, known := mi.PendingRepoForPath(physical) + require.True(t, known) + require.Equal(t, alias, rec.Root) +} + +func TestRepoIdentityDarwinTmpAliasPublication(t *testing.T) { + if runtime.GOOS != "darwin" { + t.Skip("requires Darwin's /tmp -> /private/tmp alias") + } + canonical, err := filepath.EvalSymlinks("/tmp") + require.NoError(t, err) + if canonical != "/private/tmp" { + t.Skip("host does not expose the /tmp -> /private/tmp alias") + } + root, err := os.MkdirTemp("/tmp", "repo-load-alias-") + require.NoError(t, err) + t.Cleanup(func() { require.NoError(t, os.RemoveAll(root)) }) + mi := newRepoLoadTestMulti(t) + entries := []config.RepoEntry{{Path: root}} + mi.DeclareRepos(entries) + physical := filepath.Join(canonical, filepath.Base(root)) + require.True(t, mi.RepoIdentityPending()) + _, known := mi.PendingRepoForPath(physical) + require.False(t, known) + mi.ExpectRepos(entries) + rec, known := mi.PendingRepoForPath(physical) + require.True(t, known) + require.Equal(t, root, rec.Root) +} + +func TestRepoIdentityPublicationDoesNotOverrideRegistrationOrRedeclaration(t *testing.T) { + for _, transition := range []string{"registration", "redeclaration", "running"} { + t.Run(transition, func(t *testing.T) { + root := t.TempDir() + mi := newRepoLoadTestMulti(t) + entries := []config.RepoEntry{{Path: root}} + mi.DeclareRepos(entries) + before := mi.RepoLoads()[0] + resolved, err := resolveRepoLoadIdentity(before) + require.NoError(t, err) + switch transition { + case "registration": + mi.markRepoRegistered("ready", root) + case "redeclaration": + mi.DeclareRepos(nil) + mi.DeclareRepos(entries) + case "running": + mi.StartRepoLoad(root) + } + mi.publishRepoLoadIdentity(before, resolved) + after, ok := mi.RepoLoadFor(root) + require.True(t, ok) + switch transition { + case "registration": + require.Equal(t, RepoLoadReady, after.State) + require.False(t, mi.RepoIdentityPending()) + case "redeclaration": + require.True(t, mi.RepoIdentityPending(), "stale owner result must not resolve a replacement declaration") + case "running": + require.Equal(t, RepoLoadRunning, after.State) + require.False(t, mi.RepoIdentityPending()) + } + }) + } +} + +func TestRepoIdentityUncertainFilesystemRetainsNegativeFence(t *testing.T) { + root := t.TempDir() + // A directory where commondir should be a readable file produces a real + // non-ENOENT I/O failure regardless of the test user's permission privileges. + admin := filepath.Join(t.TempDir(), "admin") + require.NoError(t, os.MkdirAll(filepath.Join(admin, "commondir"), 0755)) + require.NoError(t, os.WriteFile(filepath.Join(root, ".git"), []byte("gitdir: "+admin), 0644)) + mi := newRepoLoadTestMulti(t) + entries := []config.RepoEntry{{Path: root}} + mi.ExpectRepos(entries) + require.True(t, mi.RepoIdentityPending()) + rec, _ := mi.RepoLoadFor(root) + require.Equal(t, RepoLoadQueued, rec.State) + _, err := resolveRepoLoadIdentity(rec) + require.Error(t, err) + require.False(t, os.IsNotExist(err)) + require.NoError(t, os.Remove(filepath.Join(root, ".git"))) + mi.ExpectRepos(entries) + require.False(t, mi.RepoIdentityPending()) +} + +func TestRepoIdentitySeparateGitDirAndLinkedBoundary(t *testing.T) { + primary, linked := t.TempDir(), t.TempDir() + common := filepath.Join(t.TempDir(), "external-git") + admin := filepath.Join(common, "worktrees", "linked") + require.NoError(t, os.MkdirAll(admin, 0755)) + require.NoError(t, os.WriteFile(filepath.Join(primary, ".git"), []byte("gitdir: "+common), 0644)) + require.NoError(t, os.WriteFile(filepath.Join(linked, ".git"), []byte("gitdir: "+admin), 0644)) + require.NoError(t, os.WriteFile(filepath.Join(admin, "commondir"), []byte("../.."), 0644)) + info := ResolveWorktree(primary) + require.False(t, info.IsWorktree) + require.Equal(t, primary, info.MainRepoPath) + require.True(t, pathkey.EqualPaths(common, info.GitCommonDir)) + mi := newRepoLoadTestMulti(t) + mi.ExpectRepos([]config.RepoEntry{{Path: primary}}) + rec, known := mi.PendingRepoForPath(linked) + require.True(t, known) + require.Equal(t, primary, rec.Root) + nested := filepath.Join(linked, "nested") + require.NoError(t, os.MkdirAll(filepath.Join(nested, ".git"), 0755)) + _, known = mi.PendingRepoForPath(nested) + require.False(t, known, "the nearest unrelated git identity is a boundary") +} + +func TestRepoIdentityRegistrationPublishesCanonicalBinding(t *testing.T) { + // RepoLoadFor reads snapshot keys only; the temporary directory may itself + // be an alias, such as Darwin's /var -> /private/var. + physical, err := filepath.EvalSymlinks(t.TempDir()) + require.NoError(t, err) + alias := filepath.Join(t.TempDir(), "different-name") + require.NoError(t, os.Symlink(physical, alias)) + mi := newRepoLoadTestMulti(t) + mi.DeclareRepos([]config.RepoEntry{{Path: alias}}) + before := mi.RepoLoads()[0] + resolved, err := resolveRepoLoadIdentity(before) + require.NoError(t, err) + mi.markRepoRegistered("registered", alias) + mi.publishRepoLoadIdentity(before, resolved) + require.False(t, mi.RepoIdentityPending()) + rec, found := mi.RepoLoadFor(physical) + require.True(t, found, "registration must publish its own proven canonical key even when the warmup publisher loses the race") + require.Equal(t, RepoLoadReady, rec.State) + require.Equal(t, "registered", rec.Prefix) +} + +func TestRepoIdentityRejectsInvalidSeparateGitDir(t *testing.T) { + for _, kind := range []string{"missing", "file"} { + t.Run(kind, func(t *testing.T) { + root := t.TempDir() + target := filepath.Join(t.TempDir(), "not-a-git-directory") + if kind == "file" { + require.NoError(t, os.WriteFile(target, []byte("not a directory"), 0644)) + } + require.NoError(t, os.WriteFile(filepath.Join(root, ".git"), []byte("gitdir: "+target), 0644)) + info := ResolveWorktree(root) + require.Empty(t, info.GitCommonDir) + require.False(t, info.IsWorktree) + require.Equal(t, root, info.MainRepoPath) + mi := newRepoLoadTestMulti(t) + mi.ExpectRepos([]config.RepoEntry{{Path: root}}) + require.True(t, mi.RepoIdentityPending()) + require.Empty(t, enclosingWorktreeCommonDir(root)) + }) + } +} + +func TestRepoIdentityPermissionAndDeviceErrorsAreNotNegativeEvidence(t *testing.T) { + for _, cause := range []error{syscall.EACCES, syscall.EIO} { + t.Run(cause.Error(), func(t *testing.T) { + mi := newRepoLoadTestMulti(t) + root := t.TempDir() + mi.DeclareRepos([]config.RepoEntry{{Path: root}}) + before := mi.RepoLoads()[0] + failure := &os.PathError{Op: "lstat", Path: root, Err: cause} + after, err := resolveRepoLoadIdentityWith(before, func(path string) (string, error) { + require.Equal(t, root, path) + return root, failure // even an absolute fallback is not proof on error + }) + require.ErrorIs(t, err, cause) + require.True(t, after.identityPending) + require.Equal(t, RepoLoadQueued, after.State) + require.Empty(t, after.canonicalKey) + require.True(t, mi.RepoIdentityPending()) + }) + } +} + +func TestRepoIdentityOldPublisherCannotClearNewRegistrationUncertainty(t *testing.T) { + root := t.TempDir() + mi := newRepoLoadTestMulti(t) + entries := []config.RepoEntry{{Path: root}} + mi.DeclareRepos(entries) + before := mi.RepoLoads()[0] + resolved, err := resolveRepoLoadIdentity(before) + require.NoError(t, err) + require.NoError(t, os.WriteFile(filepath.Join(root, ".git"), []byte("unreadable identity format"), 0644)) + mi.markRepoRegistered("registered", root) + require.True(t, mi.RepoIdentityPending()) + mi.publishRepoLoadIdentity(before, resolved) + require.True(t, mi.RepoIdentityPending(), "an older successful observation must not replace a newer uncertain registration identity") + rec, found := mi.RepoLoadFor(root) + require.True(t, found) + require.Equal(t, RepoLoadReady, rec.State, "uncertain alias identity does not revoke authoritative graph registration") + require.Contains(t, rec.Reason, "identity resolution pending") + require.NoError(t, os.Remove(filepath.Join(root, ".git"))) + mi.ExpectRepos(entries) + require.False(t, mi.RepoIdentityPending()) + rec, _ = mi.RepoLoadFor(root) + require.Empty(t, rec.Reason) +} diff --git a/internal/indexer/repo_load_publication_test.go b/internal/indexer/repo_load_publication_test.go new file mode 100644 index 000000000..78a51a974 --- /dev/null +++ b/internal/indexer/repo_load_publication_test.go @@ -0,0 +1,66 @@ +package indexer + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestRepoLoadAdmissionUsesOneSnapshotAcrossPublication(t *testing.T) { + physical := t.TempDir() + alias := filepath.Join(t.TempDir(), "configured-alias") + require.NoError(t, os.Symlink(physical, alias)) + mi := newRepoLoadTestMulti(t) + entries := []config.RepoEntry{{Path: alias}} + mi.DeclareRepos(entries) + before := mi.repoLoads.load() + mi.ExpectRepos(entries) + // Publication after an ownership miss must not combine the old miss with + // the new all-resolved flag and produce a definitive untracked answer. + pending, classified := before.admissionForPath(physical) + require.True(t, classified) + require.Equal(t, RepoLoadIdentityPending, pending.State) + require.Empty(t, pending.Root) + loading, classified := mi.RepoLoadAdmissionForPath(physical) + require.True(t, classified) + require.Equal(t, RepoLoadQueued, loading.State) + require.Equal(t, alias, loading.Root) + mi.markRepoRegistered("ready", alias) + ready, classified := mi.RepoLoadAdmissionForPath(physical) + require.True(t, classified) + require.Equal(t, RepoLoadReady, ready.State, "final admission must expose a registration that raced positive checks") + _, pendingOwnership := mi.PendingRepoForPath(physical) + require.False(t, pendingOwnership, "existing ownership-only API must not reinterpret ready as loading") +} + +func TestRepoLoadAdmissionRetainsReadyLinkedFamilyIdentity(t *testing.T) { + root, linked := linkedWorktreeLayout(t) + mi := newRepoLoadTestMulti(t) + mi.ExpectRepos([]config.RepoEntry{{Path: root}}) + mi.markRepoRegistered("ready", root) + rec, classified := mi.RepoLoadAdmissionForPath(linked) + require.True(t, classified) + require.Equal(t, RepoLoadReady, rec.State) + require.Equal(t, root, rec.Root) + _, pending := mi.PendingRepoForPath(linked) + require.False(t, pending, "a ready family is identity evidence, not a pending ownership verdict") + nested := filepath.Join(linked, "nested") + require.NoError(t, os.MkdirAll(filepath.Join(nested, ".git"), 0755)) + _, classified = mi.RepoLoadAdmissionForPath(nested) + require.False(t, classified, "a different nearest git identity cannot inherit ready family evidence") + // A specific pending family match keeps its existing priority; adding ready + // identity evidence must not displace it merely because ready was recorded first. + loading := t.TempDir() + admin := filepath.Join(root, ".git", "worktrees", "another") + require.NoError(t, os.MkdirAll(admin, 0755)) + require.NoError(t, os.WriteFile(filepath.Join(admin, "commondir"), []byte("../.."), 0644)) + require.NoError(t, os.WriteFile(filepath.Join(loading, ".git"), []byte("gitdir: "+admin), 0644)) + mi.ExpectRepos([]config.RepoEntry{{Path: root}, {Path: loading, Name: "pending-family"}}) + rec, classified = mi.RepoLoadAdmissionForPath(linked) + require.True(t, classified) + require.Equal(t, RepoLoadQueued, rec.State) + require.Equal(t, loading, rec.Root) +} diff --git a/internal/indexer/repo_load_state.go b/internal/indexer/repo_load_state.go index 080b705d4..0a9fc0182 100644 --- a/internal/indexer/repo_load_state.go +++ b/internal/indexer/repo_load_state.go @@ -1,6 +1,7 @@ package indexer import ( + "errors" "os" "path/filepath" "slices" @@ -27,6 +28,9 @@ import ( type RepoLoadState string const ( + // RepoLoadIdentityPending is an unmatched request while configured root + // identities are unresolved. It names no repository and grants no scope. + RepoLoadIdentityPending RepoLoadState = "identity_pending" // RepoLoadQueued is a configured repository warmup has not started yet. RepoLoadQueued RepoLoadState = "queued" // RepoLoadRunning is a configured repository warmup is indexing or @@ -43,7 +47,7 @@ const ( // Loading reports a state that has not reached a terminal answer yet. func (s RepoLoadState) Loading() bool { - return s == RepoLoadQueued || s == RepoLoadRunning + return s == RepoLoadQueued || s == RepoLoadRunning || s == RepoLoadIdentityPending } // Ready reports a registered repository. @@ -65,6 +69,11 @@ type RepoLoad struct { State RepoLoadState `json:"state"` // Reason explains a failed or missing repository. Reason string `json:"reason,omitempty"` + // Collision identity is captured by the loading operation, not restatted + // on a probe/read path. Unknown is distinct from a missing owner. + ErrorCode string `json:"error_code,omitempty"` + OwnerPath string `json:"owner_path,omitempty"` + OwnerState string `json:"owner_state,omitempty"` // Since is when the repository entered State. Since time.Time `json:"since"` @@ -73,10 +82,20 @@ type RepoLoad struct { // never touches the filesystem. key string canonicalKey string - // commonDir is Root's shared git directory, read once when the record is - // created, so a linked worktree of a loading primary is recognised without - // spawning git on the admission path. + // commonDir is the shared git directory resolved off the serving path. + // A declaration uses only root/.git as a loading-only candidate until + // warmup resolves identity; a request must independently resolve its own + // linked-worktree marker before this can produce a pending refusal. commonDir string + // identityPending distinguishes a lexical boot declaration from the + // filesystem identity warmup fills in. It grants no graph admission. + identityPending bool + identityToken *byte + // Prefix inference is safe before registration only for an explicit name + // or a resolved ordinary root. Linked worktrees retain lifecycle naming. + prefixPinned bool + plainRoot bool + forceInstance bool } // Label names the repository in user-facing text: its prefix, or its root @@ -104,19 +123,27 @@ type repoLoadSnapshot struct { index map[string]int // loading counts records in a non-terminal state, so the worktree arm // can skip its filesystem walk once warmup has settled. - loading int + loading int + identityPending bool + readyByPrefix map[string]int } func newRepoLoadSnapshot(records []RepoLoad) *repoLoadSnapshot { - snap := &repoLoadSnapshot{records: records, index: make(map[string]int, 2*len(records))} + snap := &repoLoadSnapshot{records: records, index: make(map[string]int, 2*len(records)), readyByPrefix: make(map[string]int)} for i, rec := range records { snap.index[rec.key] = i + if rec.identityPending { + snap.identityPending = true + } if rec.canonicalKey != "" { snap.index[rec.canonicalKey] = i } if rec.State.Loading() { snap.loading++ } + if rec.State.Ready() && rec.Prefix != "" { + snap.readyByPrefix[rec.Prefix] = i + } } return snap } @@ -156,18 +183,17 @@ func (t *repoLoadTable) update(mutate func(records []RepoLoad) ([]RepoLoad, bool // canonical spelling and git common directory. Called only off the hot path: // when a start announces its repositories and when one registers. func newRepoLoad(root, prefix string, state RepoLoadState, now time.Time) RepoLoad { - rec := RepoLoad{Root: root, Prefix: prefix, State: state, Since: now, key: pathkey.IdentityKey(root)} - if canonical := pathkey.CanonicalExistingRoot(root); canonical != "" { - if key := pathkey.IdentityKey(canonical); key != rec.key { - rec.canonicalKey = key - } - } - if state.Loading() { - if commonDir := ResolveWorktree(root).GitCommonDir; commonDir != "" { - rec.commonDir = pathkey.CanonicalExistingRoot(commonDir) - } + rec := RepoLoad{Root: root, Prefix: prefix, State: state, Since: now, + key: pathkey.IdentityKey(root), identityPending: true, identityToken: new(byte), prefixPinned: state.Ready()} + resolved, err := resolveRepoLoadIdentity(rec) + if err != nil { + rec.Reason = "identity resolution pending: " + err.Error() + return rec } - return rec + // Registration is authoritative about ready state even if its directory + // disappeared since indexing. Identity absence is a separate observation. + resolved.State = state + return resolved } // absRepoRoot spells a configured path the way registration does. @@ -183,14 +209,27 @@ func absRepoRoot(path string) string { return pathkey.NormalizeVolume(abs) } +// DeclareRepos publishes configured roots before the daemon listens without +// touching those roots, resolving symlinks or reading git metadata. A hung +// configured mount must not delay the listener or hide the other declarations. +// ExpectRepos fills in filesystem identity and missing-path state during warmup. +func (mi *MultiIndexer) DeclareRepos(entries []config.RepoEntry) { + mi.expectRepos(entries, false) +} + // ExpectRepos records the configured repositories a start is about to load. // -// Called once before checkout seeding, so a configured repository reads as -// loading from the moment the socket serves, and again when warmup takes its -// own snapshot of the config. A repository already recorded keeps its record; +// Called off the serving path before checkout seeding and again when warmup +// takes its own snapshot of the config. DeclareRepos publishes lexical roots +// before the socket serves; this enriches those declarations without resetting +// progress or a registration that raced with warmup. A repository already recorded keeps its record; // one that is no longer configured and not registered is dropped, so a stale // entry can never read as loading forever. func (mi *MultiIndexer) ExpectRepos(entries []config.RepoEntry) { + mi.expectRepos(entries, true) +} + +func (mi *MultiIndexer) expectRepos(entries []config.RepoEntry, inspect bool) { if mi == nil { return } @@ -202,32 +241,128 @@ func (mi *MultiIndexer) ExpectRepos(entries []config.RepoEntry) { var added []RepoLoad for _, entry := range entries { root := absRepoRoot(entry.Path) - if root == "" || addedKeys[pathkey.IdentityKey(root)] { + key := pathkey.IdentityKey(root) + if root == "" || addedKeys[key] { continue } if i, ok := current.find(root); ok { wanted[i] = true continue } - state := RepoLoadQueued - reason := "" - if info, err := os.Stat(root); err != nil || !info.IsDir() { - state, reason = RepoLoadMissing, "the configured path does not exist" - } - rec := newRepoLoad(root, config.ResolvePrefix(entry), state, now) - rec.Reason = reason - added = append(added, rec) - addedKeys[rec.key] = true + added = append(added, RepoLoad{Root: root, Prefix: config.ResolvePrefix(entry), State: RepoLoadQueued, + Since: now, key: key, commonDir: filepath.Join(root, ".git"), identityPending: true, identityToken: new(byte), + prefixPinned: entry.Name != "", forceInstance: entry.AsWorktree}) + addedKeys[key] = true } next := make([]RepoLoad, 0, len(records)+len(added)) for i, rec := range records { - if wanted[i] || rec.State == RepoLoadReady { + if wanted[i] || rec.State.Ready() { next = append(next, rec) } } next = append(next, added...) return next, len(added) > 0 || len(next) != len(records) }) + if !inspect { + return + } + // Only the existing warmup owner resolves identities. No filesystem access + // holds the publication lock; each completed root becomes visible before the + // next root is inspected, even if that next mount never answers. + for _, rec := range mi.repoLoads.load().records { + if !rec.identityPending { + continue + } + resolved, err := resolveRepoLoadIdentity(rec) + if err != nil { + // Retain the owner's diagnostic without turning uncertainty into a + // terminal absent-alias result. No request performs this operation again. + mi.repoLoads.update(func(records []RepoLoad) ([]RepoLoad, bool) { + i, ok := newRepoLoadSnapshot(records).find(rec.Root) + if !ok || !records[i].identityPending || records[i].identityToken != rec.identityToken { + return records, false + } + reason := "identity resolution pending: " + err.Error() + if records[i].Reason == reason { + return records, false + } + records[i].Reason = reason + return records, true + }) + continue + } + mi.publishRepoLoadIdentity(rec, resolved) + } +} + +func resolveRepoLoadIdentity(rec RepoLoad) (RepoLoad, error) { + return resolveRepoLoadIdentityWith(rec, filepath.EvalSymlinks) +} + +// The resolver argument lets source-level I/O failures be exercised without +// depending on host privileges or an unavailable device. Production uses only +// the existing warmup owner's synchronous filesystem resolver. +func resolveRepoLoadIdentityWith(rec RepoLoad, canonicalize func(string) (string, error)) (RepoLoad, error) { + canonical, err := canonicalize(rec.Root) + if err != nil { + if !errors.Is(err, os.ErrNotExist) { + return rec, err + } + rec.State, rec.Reason = RepoLoadMissing, "the configured path does not exist" + } else { + st, statErr := os.Stat(canonical) + if statErr != nil { + return rec, statErr + } + if !st.IsDir() { + rec.State, rec.Reason = RepoLoadMissing, "the configured path is not a directory" + } else { + rec.canonicalKey = pathkey.IdentityKey(pathkey.NormalizeVolume(canonical)) + info, resolveErr := resolveWorktreeChecked(canonical) + if resolveErr != nil { + return rec, resolveErr + } + rec.plainRoot = !info.IsWorktree + rec.commonDir = "" + if info.GitCommonDir != "" { + common, commonErr := filepath.EvalSymlinks(info.GitCommonDir) + if commonErr != nil { + return rec, commonErr + } + rec.commonDir = pathkey.NormalizeVolume(common) + } + } + } + rec.identityPending = false + return rec, nil +} + +func (mi *MultiIndexer) publishRepoLoadIdentity(before, resolved RepoLoad) { + mi.repoLoads.update(func(records []RepoLoad) ([]RepoLoad, bool) { + i, ok := newRepoLoadSnapshot(records).find(before.Root) + if !ok || !records[i].identityPending || records[i].identityToken != before.identityToken { + return records, false + } + rec := &records[i] + rec.canonicalKey, rec.commonDir, rec.identityPending = resolved.canonicalKey, resolved.commonDir, false + rec.plainRoot = resolved.plainRoot + if strings.HasPrefix(rec.Reason, "identity resolution pending: ") { + rec.Reason = "" + } + if rec.State == RepoLoadQueued && resolved.State == RepoLoadMissing { + rec.State, rec.Reason, rec.Since = resolved.State, resolved.Reason, time.Now() + } + return records, true + }) +} + +// RepoIdentityPending is a cheap negative-admission fence, never proof that a +// particular path belongs to a configured repository. Authoritative positive +// scope/checkout/router answers must take precedence. It remains true after an +// uncertain filesystem outcome until warmup resolves identity or registration +// itself proves ownership. +func (mi *MultiIndexer) RepoIdentityPending() bool { + return mi != nil && mi.repoLoads.load().identityPending } // StartRepoLoad marks a queued repository as being worked on by warmup. @@ -256,6 +391,19 @@ func (mi *MultiIndexer) FinishRepoLoad(path string, jobErr error) { return } rec := snap.records[i] + var collision *PrefixCollisionError + if errors.As(jobErr, &collision) { + mi.setRepoLoad(root, func(rec *RepoLoad) bool { + if !rec.State.Loading() { + return false + } + rec.State, rec.Reason, rec.Since = RepoLoadFailed, "prefix_collision: "+collision.Error(), time.Now() + rec.Prefix, rec.ErrorCode = collision.Prefix, "prefix_collision" + rec.OwnerPath, rec.OwnerState = collision.ExistingPath, collision.OwnerState + return true + }) + return + } // The job may have found the repository registered under another spelling // of the same directory, which is a success the registration hook could // not attribute to this record. @@ -348,20 +496,31 @@ func (mi *MultiIndexer) markRepoRegistered(prefix, root string) { return } now := time.Now() + registered := newRepoLoad(root, prefix, RepoLoadReady, now) var announced RepoLoad var listeners []func(RepoLoad) mi.repoLoads.update(func(records []RepoLoad) ([]RepoLoad, bool) { listeners = slices.Clone(mi.repoLoads.listeners) if i, ok := newRepoLoadSnapshot(records).find(root); ok { records[i].State, records[i].Reason, records[i].Since = RepoLoadReady, "", now - records[i].commonDir = "" + records[i].ErrorCode, records[i].OwnerPath, records[i].OwnerState = "", "", "" + if registered.canonicalKey != "" { + records[i].canonicalKey = registered.canonicalKey + } + records[i].commonDir = registered.commonDir + records[i].identityPending = registered.identityPending + records[i].identityToken = registered.identityToken + records[i].prefixPinned, records[i].plainRoot = true, registered.plainRoot + if registered.identityPending { + records[i].Reason = registered.Reason + } if prefix != "" { records[i].Prefix = prefix } announced = records[i] return records, true } - announced = newRepoLoad(root, prefix, RepoLoadReady, now) + announced = registered return append(records, announced), true }) if len(listeners) == 0 { @@ -395,7 +554,12 @@ func (mi *MultiIndexer) RepoLoads() []RepoLoad { if mi == nil { return nil } - return slices.Clone(mi.repoLoads.load().records) + snap := mi.repoLoads.load() + out := slices.Clone(snap.records) + for i := range out { + out[i] = snap.withPrefixCollision(out[i]) + } + return out } // ConfiguredRepoRoots includes declarations whose loading finished without @@ -419,7 +583,7 @@ func (mi *MultiIndexer) RepoLoadFor(path string) (RepoLoad, bool) { if !ok { return RepoLoad{}, false } - return snap.records[i], true + return snap.withPrefixCollision(snap.records[i]), true } // PendingRepoForPath returns the configured, not-yet-registered repository @@ -436,34 +600,102 @@ func (mi *MultiIndexer) PendingRepoForPath(path string) (RepoLoad, bool) { return RepoLoad{}, false } snap := mi.repoLoads.load() + rec, matched := snap.matchForPath(path, false) + if matched { + rec = snap.withPrefixCollision(rec) + } + return rec, matched +} + +// RepoLoadAdmissionForPath classifies a final negative from one immutable +// snapshot. A matched Ready record is identity proof only: the caller must +// recheck ordinary graph/scope authority or retry, never grant access from it. +// An anonymous identity-pending record names no configured repository. +func (mi *MultiIndexer) RepoLoadAdmissionForPath(path string) (RepoLoad, bool) { + if mi == nil || path == "" { + return RepoLoad{}, false + } + return mi.repoLoads.load().admissionForPath(path) +} + +func (s *repoLoadSnapshot) admissionForPath(path string) (RepoLoad, bool) { + if rec, matched := s.matchForPath(path, true); matched { + return s.withPrefixCollision(rec), true + } + if s.identityPending { + return RepoLoad{State: RepoLoadIdentityPending}, true + } + return RepoLoad{}, false +} + +// withPrefixCollision uses completed identity facts from this same snapshot. +// It performs no I/O, takes no registry lock, and does not change the producer's +// queued/running state. Unknown identities and linked naming never guess. +func (s *repoLoadSnapshot) withPrefixCollision(rec RepoLoad) RepoLoad { + if rec.State.Ready() || rec.ErrorCode != "" || rec.identityPending || rec.canonicalKey == "" || + !rec.prefixPinned && (!rec.plainRoot || rec.forceInstance) { + return rec + } + index, ok := s.readyByPrefix[rec.Prefix] + if !ok { + return rec + } + owner := s.records[index] + if owner.identityPending || owner.canonicalKey == "" || owner.canonicalKey == rec.canonicalKey { + return rec + } + collision := &PrefixCollisionError{Prefix: rec.Prefix, ExistingPath: owner.Root, Path: rec.Root, OwnerState: "present"} + rec.State, rec.ErrorCode = RepoLoadFailed, "prefix_collision" + rec.OwnerPath, rec.OwnerState = owner.Root, "present" + rec.Reason = "prefix_collision: " + collision.Error() + return rec +} + +func (snap *repoLoadSnapshot) matchForPath(path string, includeReady bool) (RepoLoad, bool) { if len(snap.records) == 0 { return RepoLoad{}, false } abs := absRepoRoot(path) if rec, ok := snap.enclosing(abs); ok { - return rec, !rec.State.Ready() + return rec, includeReady || !rec.State.Ready() + } + // Until the owner resolves aliases, an unmatched request cannot prove + // absence. Do not walk its filesystem merely to manufacture a negative. + if snap.identityPending { + if rec, ok := snap.contained(abs, includeReady); ok { + return rec, true + } + return RepoLoad{}, false } if canonical := pathkey.CanonicalPath(abs); canonical != abs { if rec, ok := snap.enclosing(canonical); ok { - return rec, !rec.State.Ready() + return rec, includeReady || !rec.State.Ready() } } - if rec, ok := snap.contained(abs); ok { + if rec, ok := snap.contained(abs, includeReady); ok { return rec, true } - if snap.loading == 0 { + if snap.loading == 0 && !includeReady { return RepoLoad{}, false } commonDir := enclosingWorktreeCommonDir(abs) if commonDir == "" { return RepoLoad{}, false } + var ready RepoLoad + hasReady := false for _, rec := range snap.records { - if rec.State.Loading() && rec.commonDir != "" && pathkey.EqualPaths(rec.commonDir, commonDir) { + if rec.commonDir == "" || !pathkey.EqualPaths(rec.commonDir, commonDir) { + continue + } + if rec.State.Loading() { return rec, true } + if includeReady && rec.State.Ready() && !hasReady { + ready, hasReady = rec, true + } } - return RepoLoad{}, false + return ready, hasReady } // enclosing finds the nearest recorded root at or above path. @@ -480,21 +712,22 @@ func (s *repoLoadSnapshot) enclosing(path string) (RepoLoad, bool) { } } -// contained finds a not-ready recorded root below path — the shape of an +// contained finds a recorded root below path — the shape of an // agent opened at the directory above its repositories. A loading root wins -// over an unavailable one, because retrying is the useful answer while any -// repository under path can still become ready. -func (s *repoLoadSnapshot) contained(path string) (RepoLoad, bool) { +// over an unavailable or ready one, because retrying is useful while any +// repository under path can still become ready. The ownership-only public +// lookup excludes ready records; final admission includes them for rechecking. +func (s *repoLoadSnapshot) contained(path string, includeReady bool) (RepoLoad, bool) { var found RepoLoad ok := false for _, rec := range s.records { - if rec.State.Ready() || pathkey.EqualPaths(rec.Root, path) || !pathkey.HasPathPrefix(rec.Root, path) { + if (!includeReady && rec.State.Ready()) || pathkey.EqualPaths(rec.Root, path) || !pathkey.HasPathPrefix(rec.Root, path) { continue } if rec.State.Loading() { return rec, true } - if !ok { + if !ok || (found.State.Ready() && !rec.State.Ready()) { found, ok = rec, true } } @@ -506,9 +739,16 @@ func (s *repoLoadSnapshot) contained(path string) (RepoLoad, bool) { // path decides: a main checkout or a submodule is not a linked worktree. func enclosingWorktreeCommonDir(path string) string { for dir := filepath.Clean(path); ; { - if _, err := os.Lstat(filepath.Join(dir, ".git")); err == nil { - info := ResolveWorktree(dir) - if !info.IsWorktree || info.GitCommonDir == "" { + _, err := os.Lstat(filepath.Join(dir, ".git")) + if err != nil && !errors.Is(err, os.ErrNotExist) { + return "" + } + if err == nil { + info, resolveErr := resolveWorktreeChecked(dir) + if resolveErr != nil { + return "" + } + if info.GitCommonDir == "" { return "" } // Git writes a worktree's gitdir with symlinks resolved while a diff --git a/internal/indexer/repo_load_state_test.go b/internal/indexer/repo_load_state_test.go index cdcc8ee84..e6c146aca 100644 --- a/internal/indexer/repo_load_state_test.go +++ b/internal/indexer/repo_load_state_test.go @@ -3,6 +3,7 @@ package indexer import ( "context" "errors" + "fmt" "os" "path/filepath" "sync/atomic" @@ -250,3 +251,153 @@ func TestPrimaryRegisteredStartsTheCoordinatorsThatWaitedForIt(t *testing.T) { require.Eventually(t, func() bool { return f.lc.hasCoordinator(checkout.CheckoutID) }, 5*time.Second, 20*time.Millisecond, "the primary's registration must start the coordinator that waited for it") } + +// A selected restart cohort must all be retried by registration, rather than +// only the first checkout noticed before the primary's index was registered. +// These are real Git worktrees and durable catalog identities; this test is not +// the separate reference-host warm-restart workload or its latency evidence. +func TestPrimaryRegisteredStartsAll24WaitingCoordinators(t *testing.T) { + f := newLifecycleFixture(t) + defer f.close() + ctx := context.Background() + main := f.gitRepo("await-cohort-main") + const wanted = 24 + for i := range wanted { + f.worktreeOf(main, fmt.Sprintf("await-cohort-%02d", i)) + } + tracked, err := f.lc.Register(ctx, config.RepoEntry{Path: main, Name: "await-cohort-main"}, TrackSourceCLI) + require.NoError(t, err) + require.NoError(t, tracked.CatalogErr) + + f.restart() + require.NoError(t, f.lc.Seed(ctx)) + require.Nil(t, f.mi.GetMetadata(tracked.Prefix)) + checkouts, err := f.catalog.ListCheckouts(ctx, tracked.FamilyID) + require.NoError(t, err) + require.Len(t, checkouts, wanted+1, "the primary and every linked checkout must survive restart") + for _, checkout := range checkouts { + if checkout.CheckoutID == tracked.CheckoutID { + continue + } + // The same startup edge as an activation reaching ensureCoordinator: + // the catalog identity exists, but its primary is not registered yet. + f.lc.ensureCoordinator(ctx, tracked.GraphID, checkout) + require.False(t, f.lc.hasCoordinator(checkout.CheckoutID)) + } + f.lc.awaitingPrimaryMu.Lock() + waiting := len(f.lc.awaitingPrimary[tracked.GraphID]) + f.lc.awaitingPrimaryMu.Unlock() + require.Equal(t, wanted, waiting) + + f.lc.PrimaryRegistered("unrelated-primary") + require.Zero(t, f.lc.LiveCoordinators(tracked.FamilyID), + "another repository's event must not wake this cohort") + registered := make(chan time.Time, 1) + f.mi.OnRepoRegistered(func(rec RepoLoad) { + if rec.Prefix == tracked.Prefix { + registered <- time.Now() + } + f.lc.PrimaryRegistered(rec.Prefix) + }) + _, err = f.mi.TrackRepoCtx(ctx, config.RepoEntry{Path: main, Name: tracked.Prefix}) + require.NoError(t, err) + var registeredAt time.Time + select { + case registeredAt = <-registered: + case <-time.After(2 * time.Second): + t.Fatal("the real registration event did not drive the retry") + } + remaining := time.Until(registeredAt.Add(2 * time.Second)) + require.Positive(t, remaining, "the retry must start within the registration budget") + require.Eventually(t, func() bool { + return f.lc.LiveCoordinators(tracked.FamilyID) == wanted + }, remaining, 5*time.Millisecond, + "all waiting build loops must be live without a janitor sweep") + t.Logf("%d/%d coordinators live %s after primary registration", wanted, wanted, time.Since(registeredAt)) + assert.Empty(t, f.lc.CoordinatorStartFailures()) + f.lc.awaitingPrimaryMu.Lock() + waiting = len(f.lc.awaitingPrimary[tracked.GraphID]) + f.lc.awaitingPrimaryMu.Unlock() + assert.Zero(t, waiting, "successful retries must drain the waiting cohort") +} + +func TestDeclareRepos_DefersFilesystemIdentityUntilWarmup(t *testing.T) { + mi := newRepoLoadTestMulti(t) + missing := filepath.Join(t.TempDir(), "not-mounted") + entries := []config.RepoEntry{{Path: missing, Name: "pending"}, {Path: missing}} + mi.DeclareRepos(entries) + require.Len(t, mi.RepoLoads(), 1) + rec, ok := mi.RepoLoadFor(missing) + require.True(t, ok) + assert.Equal(t, RepoLoadQueued, rec.State, "declaration must not stat a configured root before Listen") + assert.True(t, rec.identityPending) + assert.Empty(t, rec.Reason) + rec, pending := mi.PendingRepoForPath(filepath.Join(missing, "pkg")) + require.True(t, pending) + assert.Equal(t, "pending", rec.Prefix) + + mi.ExpectRepos(entries) + rec, ok = mi.RepoLoadFor(missing) + require.True(t, ok) + assert.Equal(t, RepoLoadMissing, rec.State, "warmup still diagnoses missing paths") + assert.False(t, rec.identityPending) +} + +func TestDeclareRepos_WarmupEnrichmentPreservesRunningState(t *testing.T) { + mi := newRepoLoadTestMulti(t) + root, worktree := linkedWorktreeLayout(t) + entries := []config.RepoEntry{{Path: root}} + mi.DeclareRepos(entries) + _, pending := mi.PendingRepoForPath(filepath.Join(worktree, "pkg")) + require.False(t, pending, "an unresolved declaration cannot name the worktree's family without request filesystem work") + require.True(t, mi.RepoIdentityPending(), "negative startup admission must remain generically retryable until the owner resolves identity") + + mi.StartRepoLoad(root) + before, _ := mi.RepoLoadFor(root) + mi.ExpectRepos(entries) + after, ok := mi.RepoLoadFor(root) + require.True(t, ok) + assert.Equal(t, RepoLoadRunning, after.State) + assert.Equal(t, before.Since, after.Since) + assert.False(t, after.identityPending) + rec, pending := mi.PendingRepoForPath(worktree) + assert.True(t, pending) + assert.Equal(t, root, rec.Root, "the owner now proves the linked worktree's configured family") + + mi.DeclareRepos(nil) + assert.Empty(t, mi.RepoLoads(), "removed declarations must not remain loading forever") +} + +// The daemon invalidates cached scopes and publishes workspace_readiness from +// OnRepoRegistered. A real file-save mutation must not trigger that boot event. +func TestRepoRegistration_FileSaveDoesNotAnnounceAgain(t *testing.T) { + mi := newRepoLoadTestMulti(t) + root := filepath.Join(t.TempDir(), "saved") + writeRepoLoadSource(t, root) + mi.ExpectRepos([]config.RepoEntry{{Path: root}}) + announced := make(chan RepoLoad, 4) + mi.OnRepoRegistered(func(rec RepoLoad) { announced <- rec }) + res, err := mi.TrackRepoCtx(context.Background(), config.RepoEntry{Path: root}) + require.NoError(t, err) + select { + case <-announced: + case <-time.After(time.Second): + t.Fatal("initial cold registration was not announced") + } + before, ok := mi.RepoLoadFor(root) + require.True(t, ok) + require.Equal(t, RepoLoadReady, before.State) + idx := mi.GetIndexer(res.RepoPrefix) + require.NotNil(t, idx) + require.NoError(t, os.WriteFile(filepath.Join(root, "main.go"), []byte("package main\n\nfunc SavedAfterRegistration() {}\n"), 0o644)) + require.NoError(t, idx.IndexFile("main.go")) + after, ok := mi.RepoLoadFor(root) + require.True(t, ok) + assert.Equal(t, before.Since, after.Since, "a save must not create another registration epoch") + assert.Equal(t, before.State, after.State) + select { + case rec := <-announced: + t.Fatalf("file save emitted a boot readiness/scope-invalidation event: %+v", rec) + case <-time.After(100 * time.Millisecond): + } +} diff --git a/internal/indexer/repository_mutation_coordinator.go b/internal/indexer/repository_mutation_coordinator.go index d387879f1..4159ca0e8 100644 --- a/internal/indexer/repository_mutation_coordinator.go +++ b/internal/indexer/repository_mutation_coordinator.go @@ -1098,7 +1098,6 @@ const ( // lane worker, which names OutputEntryRepositoryReconcileLane below. OutputEntryDeferredPasses OutputMutationEntry = "indexer.MultiIndexer.withDeferredRepositoryMutation" OutputEntryRepositoryTopology OutputMutationEntry = "indexer.MultiIndexer.coordinateRepositoryTopologyMutation" - OutputEntryIndexMultiRepo OutputMutationEntry = "indexer.MultiIndexer.indexMultiRepo" OutputEntryIndexRepo OutputMutationEntry = "indexer.MultiIndexer.IndexRepo" OutputEntryIncrementalDiscoverRepo OutputMutationEntry = "indexer.MultiIndexer.incrementalDiscoverRepo" OutputEntryGitWatcherFinalize OutputMutationEntry = "indexer.GitWatcher.finalizeReconcile" @@ -1107,6 +1106,7 @@ const ( OutputEntryWatcherPatchGraph OutputMutationEntry = "indexer.Watcher.patchGraphWithReceiptState" OutputEntryWatcherEnqueueReresolve OutputMutationEntry = "indexer.Watcher.enqueueReresolve" OutputEntryCheckoutSourceMutation OutputMutationEntry = "indexer.CheckoutLifecycle.BeginCheckoutMutation" + OutputEntryPromotionPublication OutputMutationEntry = "indexer.CheckoutLifecycle.publishPromotedCorpus" // OutputEntryRepositoryReconcileLane is the coalescing lane worker itself. // Reconcile requests merge, so the execution — not the queuing caller — is // the entry point that names an output generation. @@ -1149,7 +1149,6 @@ var outputMutationEntryKinds = map[OutputMutationEntry]OutputGenerationKind{ OutputEntryReresolveFileScoped: OutputGenerationLegacy, OutputEntryDeferredPasses: OutputGenerationLegacy, OutputEntryRepositoryTopology: OutputGenerationLegacy, - OutputEntryIndexMultiRepo: OutputGenerationLegacy, OutputEntryIndexRepo: OutputGenerationLegacy, OutputEntryIncrementalDiscoverRepo: OutputGenerationLegacy, OutputEntryGitWatcherFinalize: OutputGenerationLegacy, @@ -1158,6 +1157,7 @@ var outputMutationEntryKinds = map[OutputMutationEntry]OutputGenerationKind{ OutputEntryWatcherPatchGraph: OutputGenerationLegacy, OutputEntryWatcherEnqueueReresolve: OutputGenerationLegacy, OutputEntryCheckoutSourceMutation: OutputGenerationCheckout, + OutputEntryPromotionPublication: OutputGenerationLegacy, OutputEntryRepositoryReconcileLane: OutputGenerationLegacy, OutputEntryEnrichmentCorpus: OutputGenerationLegacy, OutputEntryContractBaseline: OutputGenerationLegacy, diff --git a/internal/indexer/repository_skip.go b/internal/indexer/repository_skip.go new file mode 100644 index 000000000..e1ba1ae62 --- /dev/null +++ b/internal/indexer/repository_skip.go @@ -0,0 +1,33 @@ +package indexer + +import "errors" + +// RepositoryCollisionSkip is an unchanged repository load, not a successful +// registration or an uncertain failed mutation. Warmup must record the failed +// job without invalidating the delta frontier or publishing the owner's base. +// Unwrap retains the structured collision for other callers. +type RepositoryCollisionSkip struct { + Collision *PrefixCollisionError +} + +func (s *RepositoryCollisionSkip) Error() string { + if s.Collision == nil { + return "repository load skipped because prefix ownership is unknown" + } + return s.Collision.Error() +} + +func (s *RepositoryCollisionSkip) Unwrap() error { + if s.Collision == nil { + return nil + } + return s.Collision +} + +func repositoryCollisionSkip(err error) error { + var collision *PrefixCollisionError + if errors.As(err, &collision) { + return &RepositoryCollisionSkip{Collision: collision} + } + return err +} diff --git a/internal/indexer/repository_untrack.go b/internal/indexer/repository_untrack.go index 3915be809..1e107e50c 100644 --- a/internal/indexer/repository_untrack.go +++ b/internal/indexer/repository_untrack.go @@ -49,6 +49,29 @@ func (mi *MultiIndexer) untrackRepoCheckedRetainingAdmission( force bool, finalize func(*RepoMetadata) error, retainAdmission bool, +) (nodesRemoved, edgesRemoved int, err error) { + return mi.untrackRepoWithOwner(ctx, repoPrefix, force, finalize, retainAdmission, nil, nil) +} + +// purgeRepoOwnedChecked is promotion rollback, not authoritative untrack. It +// may remove only the registration that attempt actually observed. The witness +// is checked again after the existing stable lane drains and before payload or +// registry teardown; a replacement owner is never selected by prefix alone. +func (mi *MultiIndexer) purgeRepoOwnedChecked(ctx context.Context, expected prefixOwnerWitness, stopWatcher func()) (int, int, error) { + if expected.meta == nil || expected.prefix == "" { + return 0, 0, errPrefixIdentityUnknown + } + return mi.untrackRepoWithOwner(ctx, expected.prefix, true, nil, false, &expected, stopWatcher) +} + +func (mi *MultiIndexer) untrackRepoWithOwner( + ctx context.Context, + repoPrefix string, + force bool, + finalize func(*RepoMetadata) error, + retainAdmission bool, + expected *prefixOwnerWitness, + stopOwnedWatcher func(), ) (nodesRemoved, edgesRemoved int, err error) { if repoPrefix == "" { return 0, 0, fmt.Errorf("indexer: repository teardown refuses an empty prefix") @@ -67,7 +90,11 @@ func (mi *MultiIndexer) untrackRepoCheckedRetainingAdmission( state := mi.pendingRepositoryUntracks[repoPrefix] metaSnapshot, tracked := mi.repos[repoPrefix] idxSnapshot := mi.indexers[repoPrefix] + ownerErr := mi.validatePrefixCleanupOwnerLocked(repoPrefix, expected) mi.mu.RUnlock() + if ownerErr != nil { + return 0, 0, ownerErr + } var coordinator *repositoryMutationCoordinator if state != nil { @@ -101,6 +128,12 @@ func (mi *MultiIndexer) untrackRepoCheckedRetainingAdmission( if err := coordinator.closeAndWait(ctx); err != nil { return 0, 0, fmt.Errorf("indexer: drain repository %s before teardown: %w", repoPrefix, err) } + mi.mu.RLock() + ownerErr = mi.validatePrefixCleanupOwnerLocked(repoPrefix, expected) + mi.mu.RUnlock() + if ownerErr != nil { + return 0, 0, ownerErr + } mi.batchMutationGate.RLock() defer mi.batchMutationGate.RUnlock() @@ -115,6 +148,10 @@ func (mi *MultiIndexer) untrackRepoCheckedRetainingAdmission( hadRules := mi.configMgr != nil && len(mi.configMgr.GetRepoConfig(repoPrefix).CrossWorkspaceDeps) > 0 goModuleAffected := hadGoModule || hadRules || len(graph.ReadRepoFilePaths(mi.graph, repoPrefix, "", []string{"go"}, nil)) > 0 mi.mu.Lock() + if err := mi.validatePrefixCleanupOwnerLocked(repoPrefix, expected); err != nil { + mi.mu.Unlock() + return 0, 0, err + } state = mi.pendingRepositoryUntracks[repoPrefix] if state == nil { meta := mi.repos[repoPrefix] @@ -150,6 +187,12 @@ func (mi *MultiIndexer) untrackRepoCheckedRetainingAdmission( if state.metadata != nil { mi.forgetRepoLoad(state.metadata.RootPath) } + if expected != nil && stopOwnedWatcher != nil { + // The witness was accepted inside the drained stable lane and the + // registry is hidden. Stop only that owner's watcher, outside mi.mu + // and the cleanup continuation lock, before any payload is removed. + stopOwnedWatcher() + } state.mu.Lock() defer state.mu.Unlock() diff --git a/internal/indexer/scope_for_cwd_test.go b/internal/indexer/scope_for_cwd_test.go index f77cfa75f..c5e32f5f3 100644 --- a/internal/indexer/scope_for_cwd_test.go +++ b/internal/indexer/scope_for_cwd_test.go @@ -49,7 +49,7 @@ func TestScopeForCWD_And_ReposInWorkspace(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") // empty scope → index every configured repo + _, err = mi.indexFixtureScope("", "") // empty scope → index every configured repo require.NoError(t, err) // cwd inside repo-a → workspace "alpha", home repo "repo-a". @@ -236,7 +236,7 @@ func TestScopeForCWD_WorkspaceRoot(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") + _, err = mi.indexFixtureScope("", "") require.NoError(t, err) // Workspace root above exactly one tracked repo: binds with the @@ -321,7 +321,7 @@ func TestContainedReposScope(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") + _, err = mi.indexFixtureScope("", "") require.NoError(t, err) // ScopeForCWD still fails closed on the mixed parent — this change @@ -403,7 +403,7 @@ func TestContainedReposScope_RefusesOverbroadRoots(t *testing.T) { g := graph.New() mi := NewMultiIndexer(g, newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") + _, err = mi.indexFixtureScope("", "") require.NoError(t, err) // Precondition: a genuine parent inside home still binds, so the @@ -455,7 +455,7 @@ func TestScopeForCWD_RefusesOverbroadRootSharedWorkspace(t *testing.T) { require.NoError(t, err) mi := NewMultiIndexer(graph.New(), newTestRegistry(), search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") + _, err = mi.indexFixtureScope("", "") require.NoError(t, err) // Precondition: the shared slug makes the reverse-containment arm diff --git a/internal/indexer/track_collision_log_test.go b/internal/indexer/track_collision_log_test.go new file mode 100644 index 000000000..de9ac2b2e --- /dev/null +++ b/internal/indexer/track_collision_log_test.go @@ -0,0 +1,39 @@ +package indexer + +import ( + "os" + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + "go.uber.org/zap" + "go.uber.org/zap/zaptest/observer" + + "github.com/zzet/gortex/internal/config" +) + +func TestPrefixCollisionWarnsOnceAcrossAdmissionSurfaces(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + owner := f.gitRepo("src") + _, err := f.lc.Register(t.Context(), config.RepoEntry{Path: owner}, TrackSourceCLI) + require.NoError(t, err) + loser := filepath.Join(t.TempDir(), "src") + require.NoError(t, os.MkdirAll(loser, 0o755)) + core, logs := observer.New(zap.WarnLevel) + f.mi.logger = zap.New(core) + entry := config.RepoEntry{Path: loser} + require.Error(t, f.lc.PreflightTrack(t.Context(), entry)) + _, err = f.lc.Register(t.Context(), entry, TrackSourceCLI) + require.Error(t, err) + _, err = f.mi.TrackRepoCtx(t.Context(), entry) + require.Error(t, err) + require.Error(t, f.lc.RecordImplicit(t.Context(), loser)) + warnings := logs.All() + require.Len(t, warnings, 1) + text := warnings[0].ContextMap()["error"] + require.Contains(t, text, owner) + require.Contains(t, text, loser) + require.Contains(t, text, "use `gortex track --name") + require.NotContains(t, text, "watcher already exists") +} diff --git a/internal/indexer/track_naming.go b/internal/indexer/track_naming.go new file mode 100644 index 000000000..3505de97f --- /dev/null +++ b/internal/indexer/track_naming.go @@ -0,0 +1,181 @@ +package indexer + +import ( + "context" + "errors" + "fmt" + "os" + "path/filepath" + "time" + + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/gitstate" +) + +// All filesystem inputs to advisory naming are captured without retaining a +// lifecycle, config manager, store or indexer. The same two-slot allowance as +// root Stat bounds physically stuck discovery. Late results cannot publish. +type trackNamingFacts struct { + canonical string + workspace *config.Config + parseErr error + inventory *gitstate.FamilyInventory + record *gitstate.WorktreeRecord + err error +} + +func trackNamingFailure(ctx context.Context, err error) error { + if ctx.Err() == nil && (errors.Is(err, context.DeadlineExceeded) || errors.Is(err, errPrefixStatUnavailable)) { + return fmt.Errorf("%w: %w", errPrefixIdentityUnknown, err) + } + return err +} + +func readTrackCanonical(_ context.Context, root string) trackNamingFacts { + canonical, err := filepath.EvalSymlinks(root) + if err != nil { + return trackNamingFacts{err: err} + } + info, err := os.Stat(canonical) + if err != nil { + return trackNamingFacts{err: err} + } + if !info.IsDir() { + return trackNamingFacts{err: fmt.Errorf("invalid path: %s (not a directory)", root)} + } + return trackNamingFacts{canonical: canonical} +} + +func readTrackNaming(ctx context.Context, root string) trackNamingFacts { + facts := readTrackCanonical(ctx, root) + if facts.err != nil { + return facts + } + facts.workspace, facts.parseErr = config.ParseWorkspaceFile(filepath.Join(root, ".gortex.yaml")) + // Non-git directories retain basename naming, as ordinary track does. + facts.inventory, _ = gitstate.Inventory(ctx, root) + if facts.inventory != nil { + facts.record = recordForRoot(facts.inventory, root) + } + return facts +} + +func (a *prefixStatAllowance) naming(ctx context.Context, root string) (trackNamingFacts, error) { + return a.observeNaming(ctx, root, readTrackNaming) +} + +func (a *prefixStatAllowance) canonical(ctx context.Context, root string) (trackNamingFacts, error) { + return a.observeNaming(ctx, root, readTrackCanonical) +} + +// The two fixed discovery operations share the existing identity allowance. +// Canonical membership lookup must not parse workspace configuration or run git. +func (a *prefixStatAllowance) observeNaming(ctx context.Context, root string, read func(context.Context, string) trackNamingFacts) (trackNamingFacts, error) { + if err := ctx.Err(); err != nil { + return trackNamingFacts{}, err + } + a.once.Do(func() { a.slots = make(chan struct{}, 2) }) + select { + case a.slots <- struct{}{}: + default: + return trackNamingFacts{}, errPrefixStatUnavailable + } + slots := a.slots + done := make(chan trackNamingFacts, 1) + deadline, _ := ctx.Deadline() + workerCtx, cancel := context.WithDeadline(context.Background(), deadline) + defer cancel() + go func() { + defer func() { <-slots }() + done <- read(workerCtx, root) + }() + select { + case <-ctx.Done(): + return trackNamingFacts{}, ctx.Err() + case facts := <-done: + if err := ctx.Err(); err != nil { + return trackNamingFacts{}, err + } + return facts, facts.err + } +} + +func (mi *MultiIndexer) effectiveTrackPrefix(ctx context.Context, entry config.RepoEntry, before time.Time) (string, error) { + base := config.ResolvePrefix(entry) + if entry.Name != "" { + return base, nil + } + budget, cancel := context.WithDeadline(ctx, before) + defer cancel() + facts, err := mi.prefixStats.naming(budget, entry.Path) + if err != nil { + if os.IsNotExist(err) { + return base, nil // absent configured owners retain their basename + } + return "", trackNamingFailure(ctx, err) + } + return mi.trackPrefixFromFacts(entry, facts), nil +} + +func (mi *MultiIndexer) trackPrefixFromFacts(entry config.RepoEntry, facts trackNamingFacts) string { + base := config.ResolvePrefix(entry) + if entry.Name != "" { + return base + } + workspace := entry.Workspace + if workspace == "" { + cfg := facts.workspace + if facts.parseErr != nil && !os.IsNotExist(facts.parseErr) && mi.configMgr != nil { + cfg = mi.configMgr.LastGoodWorkspace(base, entry.Path) + } + if cfg != nil { + workspace = cfg.Workspace + } + } + linked := facts.record != nil && facts.record.AdminName != gitstate.MainAdminName + if !entry.AsWorktree && (!linked || workspace == "" || workspace == base) { + return base + } + tag := "" + if workspace != "" && workspace != base { + tag = workspace + } else if facts.record != nil { + tag = facts.record.Branch + } + if tag = sanitizeInstanceTag(tag); tag != "" { + return base + "@" + tag + } + return base + "@" + shortPathHash(entry.Path) +} + +// Catalog naming consumes completed discovery; no unjoinable worker can retain +// this lifecycle. Reads respect the remaining admission budget. +func (l *CheckoutLifecycle) dedicatedPrefixFromFacts(ctx context.Context, root string, facts trackNamingFacts, before time.Time) (string, error) { + if l.catalog == nil || facts.inventory == nil || facts.record == nil || facts.record.AdminName == gitstate.MainAdminName { + return "", nil + } + familyID := FamilyIDFor(facts.inventory.CommonDir) + if prefix := l.boundPrefixForAdminName(ctx, familyID, facts.record.AdminName); prefix != "" { + return prefix, nil + } + base := l.familyBasePrefix(ctx, familyID) + if base == "" { + return "", nil + } + prefix := DedicatedWorktreePrefix(base, facts.record.AdminName, root, nil) + holder, taken := l.prefixHolder(ctx)(prefix) + if !taken || holder == root || holder == facts.canonical { + return prefix, nil + } + budget, cancel := context.WithDeadline(ctx, before) + defer cancel() + owner := l.mi.prefixStats.observe(budget, holder) + candidate := l.mi.prefixStats.observe(budget, root) + if sameObservedDirectory(owner, candidate) { + return prefix, nil + } + if owner.err != nil || candidate.err != nil { + return "", errPrefixIdentityUnknown + } + return WorktreeInstanceCollisionName(prefix, root), nil +} diff --git a/internal/indexer/track_outcome_witness.go b/internal/indexer/track_outcome_witness.go new file mode 100644 index 000000000..1bb00c31f --- /dev/null +++ b/internal/indexer/track_outcome_witness.go @@ -0,0 +1,48 @@ +package indexer + +import "github.com/zzet/gortex/internal/config" + +// TrackOutcomeWitness carries only memory identity for a completed outcome. +// A cache cannot reuse it across owner replacement or configuration revision. +type TrackOutcomeWitness struct { + prefix string + owner *RepoMetadata + root string + config *config.RepoTrackSnapshot + loads *repoLoadSnapshot + valid bool +} + +func (mi *MultiIndexer) CaptureTrackOutcome(prefix, collisionOwner string) TrackOutcomeWitness { + mi.mu.RLock() + owner := mi.repos[prefix] + witness := TrackOutcomeWitness{prefix: prefix, owner: owner, valid: true} + witness.loads = mi.repoLoads.snap.Load() + if owner != nil { + witness.root = owner.RootPath + } + mi.mu.RUnlock() + if collisionOwner != "" && owner != nil && witness.root != collisionOwner { + witness.valid = false + } + if mi.configMgr != nil { + witness.config = mi.configMgr.RepoTrackSnapshot() + } + return witness +} + +func (mi *MultiIndexer) TrackOutcomeCurrent(witness TrackOutcomeWitness) bool { + if !witness.valid { + return false + } + if mi.repoLoads.snap.Load() != witness.loads { + return false // completed alias discovery makes a pending retry actionable + } + if mi.configMgr != nil && mi.configMgr.RepoTrackSnapshot() != witness.config { + return false + } + mi.mu.RLock() + defer mi.mu.RUnlock() + owner := mi.repos[witness.prefix] + return owner == witness.owner && (owner == nil || owner.RootPath == witness.root) +} diff --git a/internal/indexer/track_outcome_witness_test.go b/internal/indexer/track_outcome_witness_test.go new file mode 100644 index 000000000..7e2410f83 --- /dev/null +++ b/internal/indexer/track_outcome_witness_test.go @@ -0,0 +1,32 @@ +package indexer + +import ( + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestTrackOutcomeWitnessRejectsOwnerAndConfigReplacement(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + owner := f.gitRepo("owner") + _, err := f.lc.Register(t.Context(), config.RepoEntry{Path: owner, Name: "src"}, TrackSourceCLI) + require.NoError(t, err) + witness := f.mi.CaptureTrackOutcome("src", owner) + require.True(t, f.mi.TrackOutcomeCurrent(witness)) + f.mi.mu.Lock() + original := f.mi.repos["src"] + f.mi.repos["src"] = &RepoMetadata{RootPath: owner, RepoPrefix: "src"} + f.mi.mu.Unlock() + require.False(t, f.mi.TrackOutcomeCurrent(witness), "same-root owner replacement still invalidates the outcome") + f.mi.mu.Lock() + f.mi.repos["src"] = original + f.mi.mu.Unlock() + require.True(t, f.mi.TrackOutcomeCurrent(witness)) + f.mi.DeclareRepos(append(f.cm.Global().ReposSnapshot(), config.RepoEntry{Path: t.TempDir()})) + require.False(t, f.mi.TrackOutcomeCurrent(witness), "new alias identity publication invalidates a pending outcome") + witness = f.mi.CaptureTrackOutcome("src", owner) + require.NoError(t, f.cm.Global().AddRepo(config.RepoEntry{Path: t.TempDir(), Name: "next"})) + require.False(t, f.mi.TrackOutcomeCurrent(witness), "a changed configuration requires fresh admission") +} diff --git a/internal/indexer/track_preflight.go b/internal/indexer/track_preflight.go new file mode 100644 index 000000000..196da5ba6 --- /dev/null +++ b/internal/indexer/track_preflight.go @@ -0,0 +1,226 @@ +package indexer + +import ( + "context" + "errors" + "fmt" + "path/filepath" + "time" + + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/pathkey" +) + +// configuredTrackEntry inspects selected lexical/known canonical candidates. +// It does not require identity discovery for unrelated configured roots. +func (mi *MultiIndexer) configuredTrackEntry(ctx context.Context, root string, before time.Time, requested ...string) (*config.RepoEntry, error) { + entry, err := mi.configuredTrackEntryWithCanonical(ctx, root, "", before, requested...) + if err != nil || entry != nil || mi.configMgr == nil { + return entry, err + } + budget, cancel := context.WithDeadline(ctx, before) + defer cancel() + facts, err := mi.prefixStats.canonical(budget, root) + if err != nil { + return nil, trackNamingFailure(ctx, err) + } + return mi.configuredTrackEntryWithCanonical(ctx, root, facts.canonical, before, requested...) +} + +func (mi *MultiIndexer) configuredTrackEntryWithCanonical(ctx context.Context, root, canonical string, before time.Time, requested ...string) (*config.RepoEntry, error) { + if mi.configMgr == nil { + return nil, nil + } + snapshot := mi.configMgr.RepoTrackSnapshot() + entries := snapshot.RootCandidates(root) + seen := make(map[string]bool, len(entries)) + for _, entry := range entries { + seen[entry.Path] = true + } + for _, prefix := range requested { + for _, entry := range snapshot.InstanceCandidates(prefix) { + if !seen[entry.Path] { + entries = append(entries, entry) + seen[entry.Path] = true + } + } + } + appendCandidates := func(path string) { + for _, entry := range snapshot.RootCandidates(path) { + if !seen[entry.Path] { + entries = append(entries, entry) + seen[entry.Path] = true + } + } + } + if canonical != "" { + appendCandidates(canonical) + // Warmup may already know an alias configured under another basename. + // Reading that completed identity never starts all-root discovery. + if known, ok := mi.RepoLoadFor(canonical); ok { + appendCandidates(known.Root) + } + } + for _, entry := range entries { + if entry.Path == root { + return &entry, nil + } + } + budget, cancel := context.WithDeadline(ctx, before) + defer cancel() + var candidate *prefixStatResult + for _, entry := range entries { + // Distinct roots in a basename bucket need no I/O unless one might + // actually be an alias. Same-basename symlinks are intentionally kept. + if candidate == nil { + observed := mi.prefixStats.observe(budget, root) + candidate = &observed + } + owner := mi.prefixStats.observe(budget, entry.Path) + if sameObservedDirectory(*candidate, owner) { + return &entry, nil + } + if pathkey.EqualPaths(entry.Path, root) && (candidate.err != nil || owner.err != nil) { + return nil, errPrefixIdentityUnknown + } + } + return nil, nil +} + +// prepareRegistrationEntry is shared by the advisory preflight and real +// registration. It reads names but never publishes workspace/config/catalog +// state, attaches a watcher, or admits an indexing producer. +func (l *CheckoutLifecycle) prepareRegistrationEntry(ctx context.Context, entry config.RepoEntry, before time.Time) (config.RepoEntry, error) { + prepared, _, err := l.prepareRegistrationNaming(ctx, entry, before) + return prepared, err +} + +func applyConfiguredTrackEntry(entry config.RepoEntry, configured *config.RepoEntry) (config.RepoEntry, error) { + if configured == nil { + return entry, nil + } + if entry.Name != "" && entry.Name != config.ResolvePrefix(*configured) { + return entry, fmt.Errorf("%s is already configured; set name: %s in the global config and run `gortex daemon reload`", entry.Path, entry.Name) + } + return configuredTrackAnchor(entry, configured), nil +} + +func configuredTrackAnchor(entry config.RepoEntry, configured *config.RepoEntry) config.RepoEntry { + if configured == nil { + return entry + } + if entry.Name == "" { + entry.Name = configured.Name + } + if entry.Workspace == "" { + entry.Workspace = configured.Workspace + } + // Keep the configured spelling as the persistence anchor. AddRepo's + // lexical dedup must never append another entry for this physical alias. + entry.Path = configured.Path + return entry +} + +func (mi *MultiIndexer) resolveConfiguredTrackAnchor(ctx context.Context, entry config.RepoEntry, root string, before time.Time) (config.RepoEntry, error) { + configured, err := mi.configuredTrackEntry(ctx, root, before, config.ResolvePrefix(entry)) + if err != nil { + return entry, err + } + entry.Path = root + if configured != nil && configured.Name != "" { + return applyConfiguredTrackEntry(entry, configured) + } + // Internal warmup may pin a derived instance name for a legacy entry + // whose persisted Name is empty. Retain that existing naming contract. + return configuredTrackAnchor(entry, configured), nil +} + +func (l *CheckoutLifecycle) prepareRegistrationNaming(ctx context.Context, entry config.RepoEntry, before time.Time) (config.RepoEntry, trackNamingFacts, error) { + var facts trackNamingFacts + abs, err := filepath.Abs(entry.Path) + if err != nil { + return entry, facts, fmt.Errorf("resolve path %s: %w", entry.Path, err) + } + entry.Path = pathkey.NormalizeVolume(abs) + budget, cancel := context.WithDeadline(ctx, before) + defer cancel() + if entry.Name != "" { + // A pinned prefix cannot depend on workspace or worktree naming. + // Keep cheap preflight independent of the size of git's worktree list; + // actual registration still records its lifecycle inventory normally. + facts, err = l.mi.prefixStats.canonical(budget, entry.Path) + } else { + facts, err = l.mi.prefixStats.naming(budget, entry.Path) + } + if err != nil { + return entry, facts, trackNamingFailure(ctx, err) + } + configured, err := l.mi.configuredTrackEntryWithCanonical(budget, entry.Path, facts.canonical, before, config.ResolvePrefix(entry)) + if err != nil { + return entry, facts, err + } + entry, err = applyConfiguredTrackEntry(entry, configured) + if err != nil { + return entry, facts, err + } + observed, ownershipErr := l.mi.observePrefixOwnershipBefore(budget, config.ResolvePrefix(entry), entry.Path, before, nil) + if observed.registeredPrefix != "" { + entry.Name = observed.registeredPrefix + // A byte-exact registered root is already a stable cache key; aliases + // still get bounded canonical discovery below. + } + if entry.Name != "" && ownershipErr != nil { + return entry, facts, ownershipErr + } + if entry.Name == "" { + prefix, err := l.dedicatedPrefixFromFacts(budget, entry.Path, facts, before) + if err != nil { + return entry, facts, err + } + entry.Name = prefix + } + return entry, facts, nil +} + +// PreflightTrack rejects a known collision before a controller lock or MCP +// goroutine. It is advisory: registration still rechecks ownership in its +// mutation lane. A nil result is never permission to publish a checkout. +func (l *CheckoutLifecycle) PreflightTrack(ctx context.Context, entry config.RepoEntry) error { + _, err := l.PreflightTrackRoot(ctx, entry) + return err +} + +// PreflightTrackRoot also returns a canonical root from bounded discovery, so +// deferred track outcomes never call EvalSymlinks on the serving goroutine. +func (l *CheckoutLifecycle) PreflightTrackRoot(ctx context.Context, entry config.RepoEntry) (root string, err error) { + if l == nil || l.mi == nil { + return "", nil // the owning surface reports its wiring error + } + defer func() { + var collision *PrefixCollisionError + if errors.As(err, &collision) { + l.mi.logCollision(collision) + } + }() + before := time.Now().Add(prefixOwnershipTimeout) + entry, facts, err := l.prepareRegistrationNaming(ctx, entry, before) + if err != nil { + return facts.canonical, err + } + prefix := l.mi.trackPrefixFromFacts(entry, facts) + observation, err := l.mi.observePrefixOwnershipBefore(ctx, prefix, entry.Path, before, nil) + if entry.Name == "" && prefix != config.ResolvePrefix(entry) { + var collision *PrefixCollisionError + if errors.As(err, &collision) && collision.OwnerState != "unknown" { + prefix = WorktreeInstanceCollisionName(prefix, entry.Path) + observation, err = l.mi.observePrefixOwnershipBefore(ctx, prefix, entry.Path, before, nil) + } + } + if err != nil { + return facts.canonical, err + } + if observation.registeredPrefix != "" { + return facts.canonical, nil + } + return facts.canonical, l.mi.configuredPrefixCollisionBefore(ctx, prefix, entry.Path, before) +} diff --git a/internal/indexer/track_preflight_test.go b/internal/indexer/track_preflight_test.go new file mode 100644 index 000000000..287b8497f --- /dev/null +++ b/internal/indexer/track_preflight_test.go @@ -0,0 +1,117 @@ +package indexer + +import ( + "context" + "fmt" + "os" + "path/filepath" + "testing" + "time" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestTrackPreflightCollisionDoesNotPublishWorkspace(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + owner, loser := f.gitRepo("owner"), f.gitRepo("loser") + _, err := f.lc.Register(t.Context(), config.RepoEntry{Path: owner, Name: "src"}, TrackSourceCLI) + require.NoError(t, err) + before, revision := f.cm.GetRepoConfig("src"), f.cm.Revision() + writeFile(t, filepath.Join(loser, ".gortex.yaml"), "workspace: rejected\nexclude:\n - rejected-only\n") + var collision *PrefixCollisionError + require.ErrorAs(t, f.lc.PreflightTrack(t.Context(), config.RepoEntry{Path: loser, Name: "src"}), &collision) + require.Equal(t, owner, collision.ExistingPath) + require.Equal(t, before, f.cm.GetRepoConfig("src")) + require.Equal(t, revision, f.cm.Revision()) + require.NoError(t, f.lc.PreflightTrack(t.Context(), config.RepoEntry{Path: owner}), "a root registered under another name remains a no-op") +} + +func TestTrackPreflightRetainsDedicatedAdminName(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + main := f.gitRepo("src") + _, err := f.lc.Register(t.Context(), config.RepoEntry{Path: main}, TrackSourceCLI) + require.NoError(t, err) + linked := filepath.Join(t.TempDir(), "src") + runGit(t, main, "worktree", "add", "-q", "-b", "review", linked) + require.NoError(t, f.lc.PreflightTrack(t.Context(), config.RepoEntry{Path: linked})) + prepared, err := f.lc.prepareRegistrationEntry(t.Context(), config.RepoEntry{Path: linked}, time.Now().Add(prefixOwnershipTimeout)) + require.NoError(t, err) + require.Equal(t, f.lc.dedicatedPrefixFor(t.Context(), linked), prepared.Name) + require.NotEmpty(t, prepared.Name) + require.NotEqual(t, "src", prepared.Name) +} + +func TestPreparedPrefixRejectsConfigurationReplacement(t *testing.T) { + _, mi, cm := newWorktreeTestIndexer(t) + t.Cleanup(func() { require.NoError(t, mi.Close(context.Background())) }) + root := t.TempDir() + before := cm.Revision() + check := func(string, string) error { + return cm.Global().AddRepo(config.RepoEntry{Path: t.TempDir(), Name: "new-owner"}) + } + entry := config.RepoEntry{Path: root, Name: "src"} + _, err := mi.resolveTrackPrefixContext(t.Context(), &entry, root, nil, nil, check, time.Now().Add(prefixOwnershipTimeout), nil) + require.ErrorIs(t, err, errPrefixConfigChanged) + require.Equal(t, before, cm.Revision(), "rejected prepared workspace cannot publish a config revision") + require.Nil(t, mi.GetMetadata("src")) +} + +func TestConfiguredTrackAliasUsesBoundedIdentity(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + owner := f.gitRepo("src") + alias := filepath.Join(t.TempDir(), "src") + if err := os.Symlink(owner, alias); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + require.NoError(t, f.cm.Global().AddRepo(config.RepoEntry{Path: owner, Name: "pinned"})) + entry, err := f.lc.prepareRegistrationEntry(t.Context(), config.RepoEntry{Path: alias}, time.Now().Add(prefixOwnershipTimeout)) + require.NoError(t, err) + require.Equal(t, "pinned", entry.Name) +} + +func TestTrackPreflightDoesNotInspectUnrelatedConfiguredRoots(t *testing.T) { + f := newLifecycleFixture(t) + t.Cleanup(f.close) + owner := f.gitRepo("owner") + _, err := f.lc.Register(t.Context(), config.RepoEntry{Path: owner, Name: "src"}, TrackSourceCLI) + require.NoError(t, err) + entries := f.cm.Global().ReposSnapshot() + offline := t.TempDir() + for i := range 1000 { + entries = append(entries, config.RepoEntry{Path: filepath.Join(offline, fmt.Sprintf("offline-%04d", i))}) + } + f.cm.Global().Repos = entries + require.NoError(t, f.cm.Global().Save()) + f.mi.ExpectRepos(entries) // startup resolves the absent roots before admission + loser := t.TempDir() + var foreign string + f.mi.prefixStats.stat = func(root string) (os.FileInfo, error) { + if root != owner && root != loser { + foreign = root + } + return os.Stat(root) + } + var collision *PrefixCollisionError + require.ErrorAs(t, f.lc.PreflightTrack(t.Context(), config.RepoEntry{Path: loser, Name: "src"}), &collision) + require.Empty(t, foreign, "track admission must not stat the other 1000 configured roots") +} + +func TestColdKnownConfiguredOwnerStillReturnsTypedCollision(t *testing.T) { + _, mi, cm := newWorktreeTestIndexer(t) + t.Cleanup(func() { require.NoError(t, mi.Close(context.Background())) }) + owner, other := filepath.Join(t.TempDir(), "owner"), filepath.Join(t.TempDir(), "other") + for _, root := range []string{owner, other} { + require.NoError(t, os.MkdirAll(root, 0o755)) + } + require.NoError(t, cm.Global().AddRepo(config.RepoEntry{Path: owner, Name: "src"})) + _, err := mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: other, Name: "src"}) + var collision *PrefixCollisionError + require.ErrorAs(t, err, &collision, "freshly proved distinct configured ownership is not downgraded to uncertainty") + require.Equal(t, owner, collision.ExistingPath) + require.Equal(t, "present", collision.OwnerState) + require.Empty(t, mi.AllMetadata()) +} diff --git a/internal/indexer/vector_multirepo_test.go b/internal/indexer/vector_multirepo_test.go index c40a81da6..c1484030f 100644 --- a/internal/indexer/vector_multirepo_test.go +++ b/internal/indexer/vector_multirepo_test.go @@ -39,7 +39,7 @@ func TestIndexMultiRepoPublishesCombinedDurableVectorCorpus(t *testing.T) { mi := NewMultiIndexer(store, coldOrchestrationRegistry(), sw, nil, zap.NewNop()) mi.SetEmbedder(&poolEmbedder{}) - results, err := mi.indexMultiRepo(repos) + results, err := mi.indexFixtureRepos(repos) require.NoError(t, err) require.Len(t, results, len(repos)) stats, err := store.VectorCorpusStats(context.Background(), 3) @@ -64,7 +64,7 @@ func TestUntrackRepoRemovesVectorsAndRepublishesAggregateStats(t *testing.T) { sw := search.NewSwappable(initialSearchBackend(store)) mi := NewMultiIndexer(store, coldOrchestrationRegistry(), sw, cm, zap.NewNop()) mi.SetEmbedder(&poolEmbedder{}) - _, err = mi.indexMultiRepo(repos) + _, err = mi.indexFixtureRepos(repos) require.NoError(t, err) prefixA := config.ResolvePrefix(repos[0]) diff --git a/internal/indexer/vector_warm_route_test.go b/internal/indexer/vector_warm_route_test.go index 8fd12b506..fe9a7fb80 100644 --- a/internal/indexer/vector_warm_route_test.go +++ b/internal/indexer/vector_warm_route_test.go @@ -36,7 +36,7 @@ func TestWarmScopedReconcileRestoresDurableVectorCorpus(t *testing.T) { seed := NewMultiIndexer(graph.Store(store), newTestRegistry(), seedSw, cm, zap.NewNop()) seedEmb := &poolEmbedder{} seed.SetEmbedder(seedEmb) - results, err := seed.indexMultiRepo([]config.RepoEntry{entry}) + results, err := seed.indexFixtureRepos([]config.RepoEntry{entry}) require.NoError(t, err) require.Len(t, results, 1) require.Greater(t, seedEmb.calls, int32(0)) diff --git a/internal/indexer/workspace_prefix_snapshot_test.go b/internal/indexer/workspace_prefix_snapshot_test.go index 3dd37e0b2..c625d776c 100644 --- a/internal/indexer/workspace_prefix_snapshot_test.go +++ b/internal/indexer/workspace_prefix_snapshot_test.go @@ -16,11 +16,11 @@ import ( func TestIndexMultiRepoWithoutConfigManager(t *testing.T) { mi := newColdOrchestrationMulti(graph.New(), zap.NewNop()) t.Cleanup(func() { require.NoError(t, mi.Close(context.Background())) }) - results, err := mi.indexMultiRepo(nil) + results, err := mi.indexFixtureRepos(nil) require.NoError(t, err) assert.Empty(t, results) repos := coldOrchestrationRepos(t, 1) - results, err = mi.indexMultiRepo(repos) + results, err = mi.indexFixtureRepos(repos) require.NoError(t, err) require.Len(t, results, 1) assert.NotNil(t, mi.GetMetadata(config.ResolvePrefix(repos[0]))) diff --git a/internal/indexer/worktree.go b/internal/indexer/worktree.go index cef1f9641..596f204d7 100644 --- a/internal/indexer/worktree.go +++ b/internal/indexer/worktree.go @@ -2,6 +2,7 @@ package indexer import ( "context" + "errors" "fmt" "hash/fnv" "os" @@ -43,6 +44,14 @@ type WorktreeInfo struct { // repository have multiple checkouts, each assigned to its own // workspace and indexed from its own branch / working tree. func ResolveWorktree(path string) WorktreeInfo { + info, _ := resolveWorktreeChecked(path) + return info +} + +// resolveWorktreeChecked keeps I/O uncertainty distinct from a non-git root. +// Startup identity publication must not treat a best-effort fallback as proof +// that no configured alias can own a request. +func resolveWorktreeChecked(path string) (WorktreeInfo, error) { abs, err := filepath.Abs(path) if err != nil { abs = path @@ -52,35 +61,53 @@ func ResolveWorktree(path string) WorktreeInfo { gitPath := filepath.Join(abs, ".git") st, err := os.Stat(gitPath) if err != nil { - return info // not a git repository + if errors.Is(err, os.ErrNotExist) { + return info, nil + } + return info, err } if st.IsDir() { info.GitCommonDir = gitPath - return info // the main checkout + return info, nil // the main checkout } // `.git` is a file: a linked worktree or a submodule. Read the // gitdir indirection. content, err := os.ReadFile(gitPath) if err != nil { - return info + return info, err } line := strings.TrimSpace(string(content)) wtGitDir := strings.TrimSpace(strings.TrimPrefix(line, "gitdir:")) if wtGitDir == "" || wtGitDir == line { - return info // malformed .git file + return info, fmt.Errorf("malformed gitdir in %s", gitPath) } if !filepath.IsAbs(wtGitDir) { wtGitDir = filepath.Join(abs, wtGitDir) } + gitInfo, err := os.Stat(wtGitDir) + if err != nil { + return info, err + } + if !gitInfo.IsDir() { + return info, fmt.Errorf("gitdir %s is not a directory", wtGitDir) + } + // Only a worktree's gitdir carries a `commondir` file; a // submodule's does not. Its absence means "not a worktree". commonRaw, err := os.ReadFile(filepath.Join(wtGitDir, "commondir")) if err != nil { - return info + if errors.Is(err, os.ErrNotExist) { + info.GitCommonDir = filepath.Clean(wtGitDir) + return info, nil // separate gitdir or submodule: still its own primary + } + return info, err } common := strings.TrimSpace(string(commonRaw)) + if common == "" { + return info, fmt.Errorf("empty commondir in %s", wtGitDir) + } if !filepath.IsAbs(common) { common = filepath.Join(wtGitDir, common) } @@ -92,7 +119,7 @@ func ResolveWorktree(path string) WorktreeInfo { if filepath.Base(common) == ".git" { info.MainRepoPath = filepath.Dir(common) } - return info + return info, nil } // WorktreeInstanceName decides the repo prefix that absPath should be diff --git a/internal/mcp/analyze_scope_test.go b/internal/mcp/analyze_scope_test.go index 79484cc8d..b44060715 100644 --- a/internal/mcp/analyze_scope_test.go +++ b/internal/mcp/analyze_scope_test.go @@ -82,8 +82,7 @@ func newAnalyzeServer(t *testing.T, flagOn bool, repos ...analyzeRepoSpec) (*Ser reg := testRegistry() bm := search.NewNull() mi := indexer.NewMultiIndexer(g, reg, bm, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) eng.SetSearch(bm) diff --git a/internal/mcp/auto_index.go b/internal/mcp/auto_index.go index a53a36f51..0404f5ab0 100644 --- a/internal/mcp/auto_index.go +++ b/internal/mcp/auto_index.go @@ -2,6 +2,7 @@ package mcp import ( "context" + "errors" "os" "strings" "time" @@ -63,6 +64,10 @@ func (s *Server) autoIndexCWDBackground() { } ctx := context.Background() if _, err := s.multiIndexer.TrackRepoCtx(indexer.WithIndexOrigin(ctx, "mcp_auto_index"), config.RepoEntry{Path: root}); err != nil { + var collision *indexer.PrefixCollisionError + if errors.As(err, &collision) { + return // shared admission already logged this collision once + } if s.logger != nil { s.logger.Warn("auto-index: track failed", zap.String("root", root), zap.Error(err)) } @@ -74,6 +79,10 @@ func (s *Server) autoIndexCWDBackground() { // its own initiative, and an intent would claim otherwise. if s.lifecycle != nil { if err := s.lifecycle.RecordImplicit(ctx, root); err != nil && s.logger != nil { + var collision *indexer.PrefixCollisionError + if errors.As(err, &collision) { + return // shared exact-root admission owns the warning + } s.logger.Warn("auto-index: recording the checkout failed", zap.String("root", root), zap.Error(err)) } diff --git a/internal/mcp/auto_index_collision_test.go b/internal/mcp/auto_index_collision_test.go new file mode 100644 index 000000000..5adb7bda0 --- /dev/null +++ b/internal/mcp/auto_index_collision_test.go @@ -0,0 +1,39 @@ +package mcp + +import ( + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/indexer" +) + +func TestAutoIndexCollisionNoRecordImplicit(t *testing.T) { + s, mi, catalog, _ := newCatalogMCPServer(t) + owner, loser := filepath.Join(t.TempDir(), "src"), filepath.Join(t.TempDir(), "src") + gitInitWorktreeRepo(t, owner) + gitInitWorktreeRepo(t, loser) + registered, err := s.lifecycle.Register(t.Context(), config.RepoEntry{Path: owner}, indexer.TrackSourceCLI) + require.NoError(t, err) + before, found, err := catalog.GetDedicatedGraph(t.Context(), registered.GraphID) + require.NoError(t, err) + require.True(t, found) + configBefore := s.configManager.Global().ReposSnapshot() + loserFamily := indexer.FamilyIDFor(filepath.Join(loser, ".git")) + checkouts, err := catalog.ListCheckouts(t.Context(), loserFamily) + require.NoError(t, err) + require.Empty(t, checkouts) + t.Chdir(loser) + s.autoIndexCWDBackground() // synchronous entry into the real background body + checkouts, err = catalog.ListCheckouts(t.Context(), loserFamily) + require.NoError(t, err) + require.Empty(t, checkouts, "a collision must not record an implicit checkout") + after, found, err := catalog.GetDedicatedGraph(t.Context(), registered.GraphID) + require.NoError(t, err) + require.True(t, found) + require.Equal(t, before, after) + require.Equal(t, configBefore, s.configManager.Global().ReposSnapshot()) + require.Equal(t, owner, mi.GetMetadata("src").RootPath) + require.Empty(t, mi.RepoForFile(loser)) +} diff --git a/internal/mcp/batch_read_repair_admission_test.go b/internal/mcp/batch_read_repair_admission_test.go index 7cc0e9cc8..37c588f35 100644 --- a/internal/mcp/batch_read_repair_admission_test.go +++ b/internal/mcp/batch_read_repair_admission_test.go @@ -54,8 +54,7 @@ func testBatchReaderAcrossAdmission(t *testing.T, finishBeforeAdmission bool) { g := graph.New() mi := indexer.NewMultiIndexer(g, testRegistry(), search.NewNull(), cm, logger) t.Cleanup(func() { require.NoError(t, mi.Close(context.Background())) }) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) idx := mi.GetIndexer("fixture") require.NotNil(t, idx) watcher := &canonicalPublicationWatcher{hold: true} diff --git a/internal/mcp/callers_chain_scoped_golden_test.go b/internal/mcp/callers_chain_scoped_golden_test.go index 61cee207e..96ed5be85 100644 --- a/internal/mcp/callers_chain_scoped_golden_test.go +++ b/internal/mcp/callers_chain_scoped_golden_test.go @@ -69,6 +69,7 @@ func TestGetCallersAndCallChainScopedGoldenMultiRepo(t *testing.T) { production := payloads(server(store, true)) require.Equal(t, payloads(server(struct{ graph.Store }{store}, true)), production, "optional capabilities changed a scoped payload") unwrapped := payloads(server(store, false)) + require.Equal(t, payloads(server(struct{ graph.Store }{store}, false)), unwrapped, "optional capabilities changed a scoped payload containing cross-repository matches") var callers struct { Nodes []struct { @@ -91,4 +92,9 @@ func TestGetCallersAndCallChainScopedGoldenMultiRepo(t *testing.T) { } require.Contains(t, unwrapped[0], `"kind":"matches"`, "the unwrapped caller walk crosses the match") require.Contains(t, unwrapped[3], `"kind":"matches"`, "the unwrapped call chain crosses the match") + var unwrappedCallers struct { + Boundaries []json.RawMessage `json:"boundaries"` + } + require.NoError(t, json.Unmarshal([]byte(unwrapped[0]), &unwrappedCallers)) + require.NotEmpty(t, unwrappedCallers.Boundaries, "the unwrapped oracle comparison includes the interface dispatch boundary") } diff --git a/internal/mcp/change_contract_edit_paths_test.go b/internal/mcp/change_contract_edit_paths_test.go index 00a76746b..9605aeda4 100644 --- a/internal/mcp/change_contract_edit_paths_test.go +++ b/internal/mcp/change_contract_edit_paths_test.go @@ -107,8 +107,7 @@ func setupContractMultiRepoServer(t *testing.T, names ...string) (*Server, *grap g := graph.New() multi := indexer.NewMultiIndexer(g, testRegistry(), search.NewNull(), manager, zap.NewNop()) - _, err = multi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, multi, manager) srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil) srv.multiIndexer = multi diff --git a/internal/mcp/checkout_binding.go b/internal/mcp/checkout_binding.go index 1e2391137..baa0fcfd9 100644 --- a/internal/mcp/checkout_binding.go +++ b/internal/mcp/checkout_binding.go @@ -771,7 +771,10 @@ func (s *Server) checkoutForRequestPath(ctx context.Context, path string) (store return checkout, true, nil } } - if s.lifecycle == nil { + if s.lifecycle == nil || (s.multiIndexer != nil && s.multiIndexer.RepoIdentityPending()) { + // Registered catalog positives above remain usable. An unresolved configured + // identity is not a reason to allocate a checkout-observation job on a miss; + // startup admission will report identity-pending without guessing ownership. return store_sqlite.Checkout{}, false, parentErr } // CWD observation establishes a session's scope. A different explicit diff --git a/internal/mcp/diff_repo_scope_test.go b/internal/mcp/diff_repo_scope_test.go index 213da63c0..4036d9199 100644 --- a/internal/mcp/diff_repo_scope_test.go +++ b/internal/mcp/diff_repo_scope_test.go @@ -42,8 +42,10 @@ func TestDiffRepoScope(t *testing.T) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ ConfigManager: cm, diff --git a/internal/mcp/ensure_fresh_self_heal_test.go b/internal/mcp/ensure_fresh_self_heal_test.go index 193cf393a..8df3c63f5 100644 --- a/internal/mcp/ensure_fresh_self_heal_test.go +++ b/internal/mcp/ensure_fresh_self_heal_test.go @@ -1,6 +1,7 @@ package mcp import ( + "context" "testing" "github.com/stretchr/testify/assert" @@ -44,8 +45,12 @@ func TestEnsureFresh_MultiRepoSelfHealsStaleFile(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + // These fixtures test path ownership/freshness, not cross-repo batch + // resolution. Register their explicit roots through the production path. + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } require.True(t, mi.IsMultiRepo()) eng := query.NewEngine(g) diff --git a/internal/mcp/errors.go b/internal/mcp/errors.go index 2a644890c..b1d82e30c 100644 --- a/internal/mcp/errors.go +++ b/internal/mcp/errors.go @@ -55,6 +55,8 @@ const ( // index, a taken prefix) or whose path is gone. Not retryable as is; // the error's reason names what to repair. ErrCodeRepoUnavailable ErrorCode = "repo_unavailable" + // ErrCodePrefixCollision names an already-owned repository namespace. + ErrCodePrefixCollision ErrorCode = "prefix_collision" // ErrCodeSymbolNotFound — the requested symbol id is not in the // index. Recoverable: the agent can search for it instead. diff --git a/internal/mcp/facade_shorthand_scope_test.go b/internal/mcp/facade_shorthand_scope_test.go index 251c74a86..2fdf4ca42 100644 --- a/internal/mcp/facade_shorthand_scope_test.go +++ b/internal/mcp/facade_shorthand_scope_test.go @@ -93,8 +93,7 @@ func newShorthandScopeFixture(t *testing.T) shorthandScopeFixture { g := graph.New() mi := indexer.NewMultiIndexer(g, testRegistry(), search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) for i := range 40 { repo := fmt.Sprintf("far-%02d", i) for _, name := range []string{"Needle", "Twin", "Shared"} { diff --git a/internal/mcp/freshness_rider_extractor_lang_test.go b/internal/mcp/freshness_rider_extractor_lang_test.go index d5bd0bb66..ab7c8c735 100644 --- a/internal/mcp/freshness_rider_extractor_lang_test.go +++ b/internal/mcp/freshness_rider_extractor_lang_test.go @@ -1,6 +1,7 @@ package mcp import ( + "context" "encoding/json" "os" "path/filepath" @@ -64,8 +65,10 @@ func TestFreshnessRider_ExtractorStalenessIsScopedToTheFilesLanguage(t *testing. g := graph.New() store := &riderIndexStateStore{Store: g} mi := indexer.NewMultiIndexer(store, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } // Use the actual policy snapshot persisted by this index, so unrelated // extractor or global policy bumps cannot accidentally stale the Go file. diff --git a/internal/mcp/freshness_rider_test.go b/internal/mcp/freshness_rider_test.go index dca6a327f..213148f8c 100644 --- a/internal/mcp/freshness_rider_test.go +++ b/internal/mcp/freshness_rider_test.go @@ -1,6 +1,7 @@ package mcp import ( + "context" "encoding/json" "os" "path/filepath" @@ -106,8 +107,12 @@ func TestMultiRepoRiderAndMissingFileFlag(t *testing.T) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + // These fixtures test path ownership/freshness, not cross-repo batch + // resolution. Register their explicit roots through the production path. + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ ConfigManager: cm, diff --git a/internal/mcp/graph_stats_per_repo_test.go b/internal/mcp/graph_stats_per_repo_test.go index 6401705ef..8690161bd 100644 --- a/internal/mcp/graph_stats_per_repo_test.go +++ b/internal/mcp/graph_stats_per_repo_test.go @@ -41,8 +41,10 @@ func setupMultiRepoStatsServer(t *testing.T) *Server { g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } require.True(t, mi.IsMultiRepo()) eng := query.NewEngine(g) diff --git a/internal/mcp/health.go b/internal/mcp/health.go index 96e19f84e..b5e523481 100644 --- a/internal/mcp/health.go +++ b/internal/mcp/health.go @@ -319,7 +319,7 @@ func (s *Server) StopHealthBroadcaster() { func (s *Server) registerHealthTools() { s.addTool( mcp.NewTool("subscribe_daemon_health", - mcp.WithDescription("Opt the current MCP session into `notifications/daemon_health` push events. A periodic ticker (default 15 s) snapshots `{uptime_seconds, alloc_bytes, sys_bytes, num_goroutine, num_gc, tracked_repos, sessions, lsp_alive, …}` and pushes it to subscribed sessions. Repeated-identical snapshots are suppressed by a payload-hash delta filter. The current snapshot is replayed immediately as `initial_replay: true`. Optional `interval_ms` overrides the cadence (clamped to [1000, 300000]). Pair with `unsubscribe_daemon_health` to opt back out."), + mcp.WithDescription("Opt the current MCP session into `notifications/daemon_health` push events. A periodic ticker (default 15 s) snapshots `{uptime_seconds, alloc_bytes, sys_bytes, num_goroutine, num_gc, tracked_repos, sessions, lsp_alive, coordinators, sqlite_page_cache_bytes, rss_bytes, pss_bytes, footprint_bytes, …}` and pushes it to subscribed sessions; an OS memory figure is omitted where the platform has no reader, and the SQLite page cache is process-wide. Repeated-identical snapshots are suppressed by a payload-hash delta filter. The current snapshot is replayed immediately as `initial_replay: true`. Optional `interval_ms` overrides the cadence (clamped to [1000, 300000]). Pair with `unsubscribe_daemon_health` to opt back out."), mcp.WithNumber("interval_ms", mcp.Description("Publish cadence in milliseconds. Default 15000; clamped to [1000, 300000].")), ), s.handleSubscribeHealth, diff --git a/internal/mcp/index_health_empty_repo_test.go b/internal/mcp/index_health_empty_repo_test.go index 323b5355b..090ace518 100644 --- a/internal/mcp/index_health_empty_repo_test.go +++ b/internal/mcp/index_health_empty_repo_test.go @@ -1,6 +1,7 @@ package mcp import ( + "context" "os" "path/filepath" "testing" @@ -43,8 +44,10 @@ func healthServerWithIndexedRepos(t *testing.T, named map[string]string) *Server reg := parser.NewRegistry() reg.Register(languages.NewGoExtractor()) mi := indexer.NewMultiIndexer(graph.New(), reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } srv.configManager = cm srv.multiIndexer = mi diff --git a/internal/mcp/instructions_test.go b/internal/mcp/instructions_test.go index 825054787..3447eea77 100644 --- a/internal/mcp/instructions_test.go +++ b/internal/mcp/instructions_test.go @@ -179,8 +179,10 @@ func TestStateAwareInstructionsVariants(t *testing.T) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ ConfigManager: cm, diff --git a/internal/mcp/overlay_view_request_test.go b/internal/mcp/overlay_view_request_test.go index e29753dbd..e50d5e490 100644 --- a/internal/mcp/overlay_view_request_test.go +++ b/internal/mcp/overlay_view_request_test.go @@ -86,8 +86,7 @@ func newOverlayRequestFixture(t *testing.T) overlayRequestFixture { g := graph.New() bm := search.NewNull() mi := indexer.NewMultiIndexer(g, testRegistry(), bm, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) eng.SetSearch(bm) diff --git a/internal/mcp/overlay_workspace_stamp_test.go b/internal/mcp/overlay_workspace_stamp_test.go index 084c86495..083ba59fe 100644 --- a/internal/mcp/overlay_workspace_stamp_test.go +++ b/internal/mcp/overlay_workspace_stamp_test.go @@ -64,8 +64,7 @@ func newOverlayStampFixture(t *testing.T) stampFixture { g := graph.New() bm := search.NewNull() mi := indexer.NewMultiIndexer(g, testRegistry(), bm, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) eng.SetSearch(bm) diff --git a/internal/mcp/path_indexability_test.go b/internal/mcp/path_indexability_test.go index c77a59582..2ab93da4c 100644 --- a/internal/mcp/path_indexability_test.go +++ b/internal/mcp/path_indexability_test.go @@ -1,6 +1,7 @@ package mcp import ( + "context" "os" "path/filepath" "testing" @@ -151,8 +152,12 @@ func newPrefixFixture(t *testing.T) (*Server, string, string) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + // These fixtures test path ownership/freshness, not cross-repo batch + // resolution. Register their explicit roots through the production path. + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } return &Server{multiIndexer: mi}, repoA, repoB } diff --git a/internal/mcp/read_repair_observation_test.go b/internal/mcp/read_repair_observation_test.go index e4636dbc7..68eb0ca75 100644 --- a/internal/mcp/read_repair_observation_test.go +++ b/internal/mcp/read_repair_observation_test.go @@ -82,8 +82,7 @@ func testReadRepairBeforeWatcherAdmission(t *testing.T, inFlight, earlyExecution g := graph.New() mi := indexer.NewMultiIndexer(g, testRegistry(), search.NewNull(), cm, zap.NewNop()) t.Cleanup(func() { require.NoError(t, mi.Close(context.Background())) }) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) idx := mi.GetIndexer("fixture") require.NotNil(t, idx) s := newMutationServer() diff --git a/internal/mcp/read_repair_straddling_commit_test.go b/internal/mcp/read_repair_straddling_commit_test.go index a41e43bfa..a7d765cee 100644 --- a/internal/mcp/read_repair_straddling_commit_test.go +++ b/internal/mcp/read_repair_straddling_commit_test.go @@ -45,8 +45,7 @@ func TestReadRepairActualExecutorStraddlesCommittedBookkeeping(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, testRegistry(), search.NewNull(), cm, logger) t.Cleanup(func() { require.NoError(t, mi.Close(context.Background())) }) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) idx := mi.GetIndexer("fixture") require.NotNil(t, idx) s := newMutationServer() diff --git a/internal/mcp/rename_recovery_test.go b/internal/mcp/rename_recovery_test.go index 98783f09f..36df9eac5 100644 --- a/internal/mcp/rename_recovery_test.go +++ b/internal/mcp/rename_recovery_test.go @@ -235,8 +235,7 @@ func setupMultiRepoRenameRecoveryServer( registry := testRegistry() store := graph.New() multi := indexer.NewMultiIndexer(store, registry, search.NewNull(), manager, zap.NewNop()) - _, err = multi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, multi, manager) var base *indexer.Indexer if baseCfg != nil { diff --git a/internal/mcp/repo_identity_pending_test.go b/internal/mcp/repo_identity_pending_test.go new file mode 100644 index 000000000..a054cded4 --- /dev/null +++ b/internal/mcp/repo_identity_pending_test.go @@ -0,0 +1,122 @@ +package mcp + +import ( + "context" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestStateAwareInstructions_ColdIdentityPendingAndRecovery(t *testing.T) { + srv, mi, configured := newLoadingServer(t) + physical := wsRepo(t, "physical", "alias-workspace", "AliasSymbol") + alias := filepath.Join(t.TempDir(), "different-configured-name") + require.NoError(t, os.Symlink(physical, alias)) + entries := []config.RepoEntry{{Path: configured}, {Path: alias, Name: "alias-repo"}} + mi.DeclareRepos(entries) + for _, cwd := range []string{physical, t.TempDir()} { + for _, client := range []string{"claude-code", "codex", ""} { + instructions := srv.stateAwareInstructionsForClient(cwd, client) + require.Contains(t, instructions, "identity_pending") + require.NotContains(t, instructions, "INACTIVE") + require.NotContains(t, instructions, "already configured") + require.NotContains(t, instructions, "alias-repo") + require.NotContains(t, instructions, "gortex track") + } + } + ctx := sessionCtx("unresolved-alias", physical) + ws, _, bound := srv.sessionScope(ctx) + require.True(t, bound) + require.True(t, strings.HasPrefix(ws, unresolvedWorkspacePrefix), "uncertainty grants no graph scope") + mi.ExpectRepos(entries) + require.Contains(t, srv.stateAwareInstructions(physical), "LOADING alias-repo") + _, err := mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: physical, Name: "alias-repo"}) + require.NoError(t, err) + ws, _, bound = srv.sessionScope(ctx) + require.True(t, bound) + require.Equal(t, "alias-workspace", ws, "the same session re-resolves after identity and registration complete") + require.NotContains(t, srv.stateAwareInstructions(physical), "identity_pending") + require.Contains(t, srv.stateAwareInstructions(t.TempDir()), "INACTIVE", "only complete negative identity evidence permits inactive advice") +} + +func TestIdentityPendingPreservesRegisteredAutomaticCheckout(t *testing.T) { + v := newViewStack(t) + v.srv.multiIndexer.DeclareRepos([]config.RepoEntry{{Path: v.repoRoot}, {Path: v.otherRoot}, {Path: filepath.Join(t.TempDir(), "unknown")}}) + require.True(t, v.srv.multiIndexer.RepoIdentityPending()) + alias := filepath.Join(t.TempDir(), "registered-checkout-alias") + require.NoError(t, os.Symlink(v.worktreeRoot, alias)) + for _, path := range []string{v.worktreeRoot, alias} { + served, err := v.srv.CheckoutServesCWDChecked(t.Context(), path) + require.NoError(t, err) + require.True(t, served) + require.NotContains(t, v.srv.stateAwareInstructions(path), "identity_pending") + ctx := sessionCtx("pending-other-root-"+filepath.Base(path), path) + ws, _, bound := v.srv.sessionScope(ctx) + require.True(t, bound) + require.Equal(t, "main-ws", ws) + } +} + +func TestStateAwareInstructions_PublicationDuringCheckoutLookup(t *testing.T) { + for _, register := range []bool{false, true} { + name := "identity" + if register { + name = "registration" + } + t.Run(name, func(t *testing.T) { + srv, mi, configured := newLoadingServer(t) + physical := wsRepo(t, "physical", "published-workspace", "Published") + alias := filepath.Join(t.TempDir(), "configured-alias") + require.NoError(t, os.Symlink(physical, alias)) + entries := []config.RepoEntry{{Path: configured}, {Path: alias, Name: "published"}} + mi.DeclareRepos(entries) + calls := 0 + instructions := srv.stateAwareInstructionsWithCheckout(t.Context(), physical, serverInstructions, func(ctx context.Context, path string) (bool, error) { + calls++ + require.Equal(t, physical, path) + mi.ExpectRepos(entries) + if register { + _, err := mi.TrackRepoCtx(ctx, config.RepoEntry{Path: physical, Name: "published"}) + require.NoError(t, err) + } + return false, nil + }) + require.Equal(t, 1, calls) + require.NotContains(t, instructions, "INACTIVE") + require.NotContains(t, instructions, "gortex track") + if register { + require.NotContains(t, instructions, "LOADING") + } else { + require.Contains(t, instructions, "LOADING published") + } + }) + } +} + +func TestStateAwareInstructions_ReadyFamilyWithoutScopeStillRetries(t *testing.T) { + srv, mi, primary := newLoadingServer(t) + linked := t.TempDir() + admin := filepath.Join(primary, ".git", "worktrees", "linked") + require.NoError(t, os.MkdirAll(admin, 0755)) + require.NoError(t, os.WriteFile(filepath.Join(admin, "commondir"), []byte("../.."), 0644)) + require.NoError(t, os.WriteFile(filepath.Join(linked, ".git"), []byte("gitdir: "+admin), 0644)) + calls := 0 + instructions := srv.stateAwareInstructionsWithCheckout(t.Context(), linked, serverInstructions, func(ctx context.Context, path string) (bool, error) { + calls++ + if calls == 1 { + _, err := mi.TrackRepoCtx(ctx, config.RepoEntry{Path: primary, Name: "repo-l"}) + require.NoError(t, err) + } + return false, nil + }) + require.Equal(t, 2, calls) + require.Contains(t, instructions, "view_building") + require.Contains(t, instructions, "checkout discovery is pending") + require.NotContains(t, instructions, "scope_binding_pending", "verified family pending keeps the canonical existing producer") + require.NotContains(t, instructions, "INACTIVE") + require.NotContains(t, instructions, "gortex track "+linked) +} diff --git a/internal/mcp/repo_loading.go b/internal/mcp/repo_loading.go index 7a85efd25..fc80f9c7a 100644 --- a/internal/mcp/repo_loading.go +++ b/internal/mcp/repo_loading.go @@ -36,6 +36,25 @@ func ServerInstructionsLoading(repo, phase string, percent int) string { "the repository is already configured.", repo, loadingPhaseLabel(phase), percent) } +// ServerInstructionsIdentityPending describes uncertainty, not ownership of +// the caller's path. It is shared by initialization and modern discovery. +func ServerInstructionsIdentityPending() string { + return serverInstructionsAdmissionPending("identity_resolution", "identity_pending") +} + +// ServerInstructionsScopeBindingPending applies after registration when the +// ordinary checkout/scope authority has not established the path's route yet. +func ServerInstructionsScopeBindingPending() string { + return serverInstructionsAdmissionPending("scope_binding", "scope_binding_pending") +} + +func serverInstructionsAdmissionPending(phase, reason string) string { + return fmt.Sprintf("Gortex is LOADING graph admission after a restart (%s, 0%%). "+ + "Graph admission for this path is not yet established; calls return repo_loading (retryable, %s). "+ + "Retry shortly or continue work natively while admission completes. No graph scope has been granted; "+ + "do not change tracking configuration or reconnect to resolve this temporary state.", phase, reason) +} + // ServerInstructionsUnavailable is the inactive initialize variant for a // configured repository whose load ended without registering it. func ServerInstructionsUnavailable(repo, reason string) string { @@ -58,12 +77,29 @@ func RepoLoadingSuggestion(repo string) string { // RepoUnavailableSuggestion is the remedy text a repo_unavailable refusal // carries. func RepoUnavailableSuggestion(rec indexer.RepoLoad) string { + if rec.ErrorCode == string(ErrCodePrefixCollision) { + return PrefixCollisionSuggestion(rec) + } if rec.State == indexer.RepoLoadMissing { return fmt.Sprintf("The configured path %s does not exist. Restore it, or remove it with `gortex untrack %s`.", rec.Root, rec.Root) } return fmt.Sprintf("Gortex could not load %s after the restart. Run `gortex track %s` to index it again, or `gortex doctor` to diagnose.", rec.Root, rec.Root) } +// PrefixCollisionSuggestion uses only the captured owner evidence. Missing +// means an observed ENOENT, never a timeout or permission failure. +func PrefixCollisionSuggestion(rec indexer.RepoLoad) string { + if rec.OwnerState == "missing" { + return fmt.Sprintf("The owner %s no longer exists. Inspect `gortex untrack %s` for that owner before retrying %s.", rec.OwnerPath, rec.OwnerPath, rec.Root) + } + return fmt.Sprintf("Prefix %q belongs to %s. Use a different name parameter for a new root; for the configured root %s set a unique name in the global config and run `gortex daemon reload`.", rec.Prefix, rec.OwnerPath, rec.Root) +} + +func ServerInstructionsPrefixCollision(rec indexer.RepoLoad) string { + return "Gortex cannot admit this configured root (prefix_collision). " + PrefixCollisionSuggestion(rec) + + " Graph tools cannot answer for this root until its prefix is unique. Native file reads and edits remain available." +} + // loadingPhaseLabel names the warmup phase; before the first phase is // published the daemon is still seeding its checkout catalog. func loadingPhaseLabel(phase string) string { diff --git a/internal/mcp/repo_registration_fixture_test.go b/internal/mcp/repo_registration_fixture_test.go new file mode 100644 index 000000000..1d5ea61c5 --- /dev/null +++ b/internal/mcp/repo_registration_fixture_test.go @@ -0,0 +1,44 @@ +package mcp + +import ( + "context" + "sort" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/graph" + "github.com/zzet/gortex/internal/indexer" +) + +// registerMCPFixtureRepos uses the daemon's registration and shared deferred +// stages. Cross-repository queries need all roots published before resolution +// and contract matching; independent TrackRepo calls do not establish that +// boundary. These fixtures start empty and have no concurrent lifecycle writer. +func registerMCPFixtureRepos(t *testing.T, mi *indexer.MultiIndexer, cm *config.ConfigManager) { + t.Helper() + ctx := context.Background() + mi.BeginParallelBatch() + defer mi.ResetBatch() + entries := cm.ActiveRepos() + mi.ExpectRepos(entries) + var prefixes []string + for _, entry := range mi.ResolveConfiguredRepos(ctx, entries) { + mi.StartRepoLoad(entry.Path) + result, err := mi.TrackRepoCtx(ctx, entry) + mi.FinishRepoLoad(entry.Path, err) + require.NoError(t, err) + prefixes = append(prefixes, result.RepoPrefix) + } + require.NoError(t, mi.RunPreEnrichResolve(ctx, nil, nil)) + mi.RunDeferredPassesAllResult(ctx) + mi.ReconcileContractEdges() + // SQLite query fixtures also need their durable resolved-reference sidecar + // after the shared resolution, rather than after each individual root. + if rebuilder, ok := mi.Graph().(graph.RefFactsRebuilder); ok { + sort.Strings(prefixes) + require.NoError(t, rebuilder.RebuildRefFactsForRepos(prefixes)) + } + mi.EndBatch() +} diff --git a/internal/mcp/savings_retrieval_test.go b/internal/mcp/savings_retrieval_test.go index 7f022784f..3fb58778a 100644 --- a/internal/mcp/savings_retrieval_test.go +++ b/internal/mcp/savings_retrieval_test.go @@ -68,8 +68,7 @@ func Widget%d%d(key string, depth int) (string, bool) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ ConfigManager: cm, diff --git a/internal/mcp/scope_resolve_test.go b/internal/mcp/scope_resolve_test.go index 76bde705e..ce8f13c2a 100644 --- a/internal/mcp/scope_resolve_test.go +++ b/internal/mcp/scope_resolve_test.go @@ -79,8 +79,7 @@ func newSharedWorkspaceServer(t *testing.T, flagOn bool) sharedWSOptions { reg := testRegistry() bm := search.NewNull() mi := indexer.NewMultiIndexer(g, reg, bm, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) eng.SetSearch(bm) @@ -117,8 +116,7 @@ func newSplitProjectWorkspaceServer(t *testing.T, flagOn bool) sharedWSOptions { reg := testRegistry() bm := search.NewNull() mi := indexer.NewMultiIndexer(g, reg, bm, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) eng.SetSearch(bm) @@ -696,8 +694,7 @@ func newLoneRepoServer(t *testing.T, flagOn bool) (*Server, string) { reg := testRegistry() bm := search.NewNull() mi := indexer.NewMultiIndexer(g, reg, bm, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) eng.SetSearch(bm) @@ -838,8 +835,7 @@ func newTwoRepoServer(t *testing.T) (*Server, string) { backend := search.NewSymbolSearcherBackend(store) mi := indexer.NewMultiIndexer(store, testRegistry(), backend, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) requireSymbolFTS(t, store) eng := query.NewEngine(store) diff --git a/internal/mcp/search_corpus_scope_test.go b/internal/mcp/search_corpus_scope_test.go index 5e51acdd0..786d7a26b 100644 --- a/internal/mcp/search_corpus_scope_test.go +++ b/internal/mcp/search_corpus_scope_test.go @@ -58,8 +58,10 @@ func newWorkspaceRootBoundServer(t *testing.T) (s *Server, root string) { reg := parser.NewRegistry() reg.Register(languages.NewGoExtractor()) mi := indexer.NewMultiIndexer(store, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } // A content node owned by z, with its body in the content index. store.AddNode(&graph.Node{ diff --git a/internal/mcp/server.go b/internal/mcp/server.go index 36b1efd0b..d0e52bdc7 100644 --- a/internal/mcp/server.go +++ b/internal/mcp/server.go @@ -164,7 +164,9 @@ type Server struct { // resetTestIndexProbe clears it on every analysis pass rather than // pinning the first (possibly mid-warmup) answer for the daemon's life. testIndexProbe sync.Map - // trackInFlight guards first-index runs by absolute repo path. + // trackInFlight guards first-index runs by canonical root and request + // options, retaining final outcomes for ten minutes while owner/config + // witnesses remain current. Lexical aliases share the same job pointer. // track answers before the index finishes (#326), so without this a // second call for the same path would start a duplicate full index: // TrackRepoCtx's already-tracked check only sees repos that finished. @@ -1582,6 +1584,16 @@ func ServerInstructionsPendingAutomaticCheckout(cwd string, roots ...string) str return appendTrackedRootsDiagnostic(msg+"\n\n"+toolref.TransportClause()+" "+toolref.DegradedModeLine(), roots) } +// PendingAutomaticCheckoutInstructions captures the verified family verdict +// with its canonical rendered guidance. A later reader must not re-probe the +// filesystem and accidentally turn this pending result into untracked advice. +func PendingAutomaticCheckoutInstructions(cwd string, roots ...string) (string, bool) { + if !agents.PendingAutomaticCheckout(cwd, roots) { + return "", false + } + return ServerInstructionsPendingAutomaticCheckout(cwd, roots...), true +} + // ServerInstructionsPendingFamilyLoad retains the terminal load failure as // context, without suggesting a second track operation for a declared family. func ServerInstructionsPendingFamilyLoad(cwd string, load indexer.RepoLoad, roots ...string) string { @@ -1677,6 +1689,12 @@ func (s *Server) stateAwareInstructionsWithBase(cwd, base string) string { } func (s *Server) stateAwareInstructionsContext(ctx context.Context, cwd, base string) string { + return s.stateAwareInstructionsWithCheckout(ctx, cwd, base, s.CheckoutServesCWDChecked) +} + +// Keep the existing checkout lookup explicit so publication during that +// lookup can be exercised without a global hook or changing the transport. +func (s *Server) stateAwareInstructionsWithCheckout(ctx context.Context, cwd, base string, lookup func(context.Context, string) (bool, error)) string { // The reachability test must match the dispatcher's gate exactly. // Telling a session it is INACTIVE while every call succeeds is the // same divergence as the reverse, just quieter. @@ -1688,24 +1706,33 @@ func (s *Server) stateAwareInstructionsContext(ctx context.Context, cwd, base st // untracked: telling it to track itself and reconnect is the // advice that made agents re-track or stop on every restart. if rec, pending := s.multiIndexer.PendingRepoForPath(cwd); pending { - if !rec.State.Loading() { - roots := agents.FamilyRepoRoots(s.trackedRepoRoots(), s.multiIndexer.ConfiguredRepoRoots()) - if agents.PendingAutomaticCheckout(cwd, roots) { - return ServerInstructionsPendingFamilyLoad(cwd, rec, roots...) - } - return ServerInstructionsUnavailable(rec.Root, rec.Reason) - } - // The ordinary instructions follow the loading preamble: the - // client reads them once, and they must still be right after - // the repository is ready. - phase, percent := s.WarmupProgress() - return ServerInstructionsLoading(rec.Label(), phase, percent) + "\n\n" + base + return s.instructionsForRepoLoad(cwd, base, rec) } - if serves, err := s.CheckoutServesCWDChecked(ctx, cwd); err != nil { + if serves, err := lookup(ctx, cwd); err != nil { return CheckoutAdmissionInstructions(cwd, err) } else if serves { return s.appendLiveInstructions(cwd, base) } + if rec, classified := s.multiIndexer.RepoLoadAdmissionForPath(cwd); classified { + if rec.State.Ready() { + _, _, _, inside = s.multiIndexer.ScopeForCWD(cwd) + _, _, contains = s.multiIndexer.ContainedReposScope(cwd) + if inside || contains { + return s.appendLiveInstructions(cwd, base) + } + if serves, err := lookup(ctx, cwd); err != nil { + return CheckoutAdmissionInstructions(cwd, err) + } else if serves { + return s.appendLiveInstructions(cwd, base) + } + roots := agents.FamilyRepoRoots(s.trackedRepoRoots(), s.multiIndexer.ConfiguredRepoRoots()) + if instructions, known := PendingAutomaticCheckoutInstructions(cwd, roots...); known { + return instructions + } + return ServerInstructionsScopeBindingPending() + "\n\n" + base + } + return s.instructionsForRepoLoad(cwd, base, rec) + } roots := agents.FamilyRepoRoots(s.trackedRepoRoots(), s.multiIndexer.ConfiguredRepoRoots()) if agents.PendingAutomaticCheckout(cwd, roots) { return ServerInstructionsPendingAutomaticCheckout(cwd, roots...) @@ -1716,6 +1743,24 @@ func (s *Server) stateAwareInstructionsContext(ctx context.Context, cwd, base st return s.appendLiveInstructions(cwd, base) } +func (s *Server) instructionsForRepoLoad(cwd, base string, rec indexer.RepoLoad) string { + if rec.ErrorCode == string(ErrCodePrefixCollision) { + return ServerInstructionsPrefixCollision(rec) + } + if rec.State == indexer.RepoLoadIdentityPending { + return ServerInstructionsIdentityPending() + "\n\n" + base + } + if !rec.State.Loading() { + roots := agents.FamilyRepoRoots(s.trackedRepoRoots(), s.multiIndexer.ConfiguredRepoRoots()) + if agents.PendingAutomaticCheckout(cwd, roots) { + return ServerInstructionsPendingFamilyLoad(cwd, rec, roots...) + } + return ServerInstructionsUnavailable(rec.Root, rec.Reason) + } + phase, percent := s.WarmupProgress() + return ServerInstructionsLoading(rec.Label(), phase, percent) + "\n\n" + base +} + func (s *Server) appendLiveInstructions(cwd, base string) string { if facts := s.liveInstructionFacts(cwd); facts != "" { base = base + "\n\n" + facts @@ -2439,7 +2484,7 @@ func (s *Server) sessionScope(ctx context.Context) (workspaceID, projectID strin // too, but only for now. Caching the sentinel would leave the session // blind after the repository registers, so it is not cached: the next // call resolves again. - if rec, pending := s.multiIndexer.PendingRepoForPath(cwd); pending && rec.State.Loading() { + if _, classified := s.multiIndexer.RepoLoadAdmissionForPath(cwd); classified { ss.scopeResolved = false } ss.scopeWorkspaceID = unresolvedWorkspacePrefix + cwd diff --git a/internal/mcp/tools_contract_bridge_test.go b/internal/mcp/tools_contract_bridge_test.go index 7323dd1c7..fc37168fc 100644 --- a/internal/mcp/tools_contract_bridge_test.go +++ b/internal/mcp/tools_contract_bridge_test.go @@ -127,8 +127,7 @@ func fetchUsers() { g := graph.New() mi := indexer.NewMultiIndexer(g, preg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) return NewServer(eng, g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ diff --git a/internal/mcp/tools_contracts_filter_test.go b/internal/mcp/tools_contracts_filter_test.go index a61dd188e..b4e1836db 100644 --- a/internal/mcp/tools_contracts_filter_test.go +++ b/internal/mcp/tools_contracts_filter_test.go @@ -40,8 +40,8 @@ func TestHandleContracts_FiltersByProjectAndRef(t *testing.T) { orphanRepo := setupRepoWithHTTPConsumer(t, root, "orphan-svc") tmpCfg := filepath.Join(t.TempDir(), "config.yaml") - // Top-level Repos drives `mi.IndexAll()` so every repo actually - // gets walked; projects + refs are metadata used for filter + // Top-level Repos selects the startup registration fixtures; projects + // and refs are metadata used for filter // resolution only. Listing a repo in both places is valid — the // prefix resolves identically (it's derived from Name), so the // allow-set never double-counts. @@ -70,8 +70,7 @@ func TestHandleContracts_FiltersByProjectAndRef(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, preg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) srv := NewServer(eng, g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ diff --git a/internal/mcp/tools_contracts_test.go b/internal/mcp/tools_contracts_test.go index cffa49b20..fb1692a3b 100644 --- a/internal/mcp/tools_contracts_test.go +++ b/internal/mcp/tools_contracts_test.go @@ -92,8 +92,7 @@ func TestHandleContracts_ReflectsRuntimeTrackedRepos(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, preg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) singleton := indexer.New(g, preg, config.IndexConfig{}, zap.NewNop()) @@ -161,8 +160,7 @@ func TestHandleContracts_MatchesGraphContractCount(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, preg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) srv := NewServer(eng, g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ diff --git a/internal/mcp/tools_core_index_test.go b/internal/mcp/tools_core_index_test.go index b996f5d6e..4086fed6b 100644 --- a/internal/mcp/tools_core_index_test.go +++ b/internal/mcp/tools_core_index_test.go @@ -60,8 +60,7 @@ func TestHandleIndexRepository_MultiRepoRoutesThroughMultiIndexer(t *testing.T) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) require.True(t, mi.IsMultiRepo()) // Invariant before: every node has a non-empty RepoPrefix, and both repos @@ -196,8 +195,7 @@ func TestHandleIndexRepository_MultiRepoRejectsUntrackedPath(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) srv := NewServer(eng, g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ diff --git a/internal/mcp/tools_core_reindex_test.go b/internal/mcp/tools_core_reindex_test.go index e8ee12caf..168d5cc6a 100644 --- a/internal/mcp/tools_core_reindex_test.go +++ b/internal/mcp/tools_core_reindex_test.go @@ -276,8 +276,7 @@ func TestHandleReindexRepository_MultiRepoRoutesByPrefix(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) require.True(t, mi.IsMultiRepo()) eng := query.NewEngine(g) @@ -334,8 +333,7 @@ func TestHandleReindexRepository_MultiRepoPathScoped(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) srv := NewServer(eng, g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ @@ -388,8 +386,7 @@ func TestHandleReindexRepository_MultiRepoRejectsUntrackedPath(t *testing.T) { g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) srv := NewServer(eng, g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ diff --git a/internal/mcp/tools_fileops_singlerepo_test.go b/internal/mcp/tools_fileops_singlerepo_test.go index 5125bccac..2f80f3da6 100644 --- a/internal/mcp/tools_fileops_singlerepo_test.go +++ b/internal/mcp/tools_fileops_singlerepo_test.go @@ -39,8 +39,12 @@ func newSingleRepoServer(t *testing.T) (*Server, *graph.Graph, string) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + // These fixtures test path ownership/freshness, not cross-repo batch + // resolution. Register their explicit roots through the production path. + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ ConfigManager: cm, @@ -117,8 +121,12 @@ func TestResolveFilePath_MultiRepoBareRelativeStillAmbiguous(t *testing.T) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + // These fixtures test path ownership/freshness, not cross-repo batch + // resolution. Register their explicit roots through the production path. + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ ConfigManager: cm, diff --git a/internal/mcp/tools_fileops_worktree_test.go b/internal/mcp/tools_fileops_worktree_test.go index 35f09d94b..28e94ff96 100644 --- a/internal/mcp/tools_fileops_worktree_test.go +++ b/internal/mcp/tools_fileops_worktree_test.go @@ -87,8 +87,7 @@ func setupWorktreePair(t *testing.T) (mainRepo, worktree string, srv *Server) { g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) require.True(t, mi.IsMultiRepo()) eng := query.NewEngine(g) @@ -420,8 +419,7 @@ func TestEditTools_UnprefixedPathAmbiguousAcrossTwoWorktrees(t *testing.T) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ ConfigManager: cm, MultiIndexer: mi, diff --git a/internal/mcp/tools_find_declaration_test.go b/internal/mcp/tools_find_declaration_test.go index 9cecd6569..86736e4be 100644 --- a/internal/mcp/tools_find_declaration_test.go +++ b/internal/mcp/tools_find_declaration_test.go @@ -251,8 +251,7 @@ func consumer() int { return uniqueDecl() + uniqueDecl() } reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) require.True(t, mi.IsMultiRepo()) eng := query.NewEngine(g) diff --git a/internal/mcp/tools_find_files_scoped_test.go b/internal/mcp/tools_find_files_scoped_test.go index 103ae148b..f7d1edcbb 100644 --- a/internal/mcp/tools_find_files_scoped_test.go +++ b/internal/mcp/tools_find_files_scoped_test.go @@ -18,6 +18,7 @@ import ( type findFilesProjectionStore struct { *store_sqlite.Store globalReads int + edgeReads int projectedRepos []string fullReads int summaryReads int @@ -42,6 +43,11 @@ func (s *findFilesProjectionStore) AllNodesLight() []*graph.Node { return s.Store.AllNodesLight() } +func (s *findFilesProjectionStore) AllEdges() []*graph.Edge { + s.edgeReads++ + return s.Store.AllEdges() +} + func (s *findFilesProjectionStore) NodesInScopeSeq(repos, files []string, kinds ...graph.NodeKind) iter.Seq[*graph.Node] { s.fullReads++ s.projectedRepos = append(s.projectedRepos, repos...) @@ -98,6 +104,7 @@ func TestFindFilesScopedNeverScansAndRowsStayFlat(t *testing.T) { require.NoError(t, err) require.False(t, res.IsError) require.Zero(t, tracked.globalReads, "a scoped find_files read the whole store") + require.Zero(t, tracked.edgeReads, "a scoped find_files read every edge") require.Zero(t, tracked.fullReads) require.Zero(t, tracked.summaryReads, "no project narrows the scope, so file locations suffice") require.Equal(t, 1, tracked.locationReads) diff --git a/internal/mcp/tools_multi.go b/internal/mcp/tools_multi.go index 6855640aa..841c5730e 100644 --- a/internal/mcp/tools_multi.go +++ b/internal/mcp/tools_multi.go @@ -4,7 +4,6 @@ import ( "context" "errors" "fmt" - "os" "path/filepath" "sort" "strings" @@ -15,6 +14,7 @@ import ( "github.com/zzet/gortex/internal/config" "github.com/zzet/gortex/internal/indexer" + "github.com/zzet/gortex/internal/pathkey" "github.com/zzet/gortex/internal/viewmetrics" ) @@ -90,21 +90,9 @@ func (s *Server) handleTrackRepository(ctx context.Context, req mcp.CallToolRequ return mcp.NewToolResultError("path is required"), nil } - // Validate path exists and is a directory. - info, statErr := os.Stat(path) - if statErr != nil { - return mcp.NewToolResultError(fmt.Sprintf("invalid path: %s", path)), nil - } - if !info.IsDir() { - return mcp.NewToolResultError(fmt.Sprintf("invalid path: %s (not a directory)", path)), nil - } - if s.multiIndexer == nil { return mcp.NewToolResultError("multi-repo indexing is not enabled"), nil } - if answer, loading := s.configuredRepoStillLoading(ctx, req, path); loading { - return answer, nil - } entry := config.RepoEntry{Path: path} if name, ok := req.GetArguments()["name"].(string); ok && name != "" { @@ -136,27 +124,44 @@ func (s *Server) handleTrackRepository(ctx context.Context, req mcp.CallToolRequ // cycle (#326). Run the index on a context detached from the request and // answer `accepted` before the deadline, so the work continues // server-side the way boot indexing does. - absPath, absErr := filepath.Abs(path) - if absErr != nil { - absPath = path + absPath, err := filepath.Abs(path) + if err != nil { + return mcp.NewToolResultError(fmt.Sprintf("invalid path: %s", path)), nil } - if _, busy := s.trackInFlight.LoadOrStore(absPath, struct{}{}); busy { - return s.respondJSONOrTOON(ctx, req, map[string]any{ - "status": "accepted", - "path": path, - "detail": "an initial index for this path is already running; call track again later to read its result", - }) + absPath = pathkey.NormalizeVolume(absPath) // lexical only; no serving-path EvalSymlinks + key := trackRequestKey(absPath, entry) + if result, err, found := s.storedTrackOutcome(ctx, req, path, entry, key); found { + return result, err } - if s.lifecycle == nil { return mcp.NewToolResultError("checkout lifecycle is not wired"), nil } - - type trackOutcome struct { - result indexer.RegisterResult - err error + // Cancellation stops advisory preflight, but does not cancel the detached + // registration contract (#326). That producer performs authoritative + // admission even when this request already spent its deadline. + canonical, preflightErr := s.lifecycle.PreflightTrackRoot(ctx, entry) + if canonical != "" { + key = trackRequestKey(pathkey.NormalizeVolume(canonical), entry) + if result, err, found := s.storedTrackOutcome(ctx, req, path, entry, key); found { + return result, err + } + } + canceledPreflight := ctx.Err() != nil && errors.Is(preflightErr, ctx.Err()) + if preflightErr != nil && !canceledPreflight { + return s.trackRepositoryOutcome(ctx, req, path, entry, repositoryTrackOutcome{err: preflightErr}, false) } - done := make(chan trackOutcome, 1) + if answer, loading := s.configuredRepoStillLoading(ctx, req, path); loading { + return answer, nil + } + s.expireTrackOutcomes(time.Now()) + job := &repositoryTrackJob{done: make(chan struct{})} + if _, busy := s.trackInFlight.LoadOrStore(key, job); busy { + return s.trackRepositoryAccepted(ctx, req, path) + } + // Remember the lexical spelling too: the next retry can read its result + // before touching the filesystem. Canonical aliases share producer admission. + s.trackInFlight.Store(trackRequestKey(absPath, entry), job) + // The index below outlives this request by design, so it must not outlive // the payload the request was reading: join the view's lease before the // goroutine starts and release it when the goroutine actually exits, @@ -165,7 +170,6 @@ func (s *Server) handleTrackRepository(ctx context.Context, req mcp.CallToolRequ pin := handoffRequestView(ctx, viewmetrics.HandoffRepositoryIndex) go func() { defer pin.release() - defer s.trackInFlight.Delete(absPath) // WithoutCancel keeps the request's values (progress token, session) // while dropping its cancellation, so the daemon's request lifetime // can no longer kill a half-written first index. @@ -182,7 +186,16 @@ func (s *Server) handleTrackRepository(ctx context.Context, req mcp.CallToolRequ s.logger.Warn("track: recording the checkout identity failed", zap.String("path", path), zap.Error(res.CatalogErr)) } - done <- trackOutcome{result: res, err: trackErr} + job.outcome = snapshotTrackOutcome(res, trackErr) + prefix, collisionOwner := res.Prefix, "" + var collision *indexer.PrefixCollisionError + if errors.As(trackErr, &collision) { + prefix, collisionOwner = collision.Prefix, collision.ExistingPath + } + job.witness = s.multiIndexer.CaptureTrackOutcome(prefix, collisionOwner) + job.completed = time.Now() + pin.release() // done publishes a physically joined handoff tail + close(job.done) }() // A nil channel blocks forever, so a request with no deadline (CLI, tests) @@ -196,38 +209,8 @@ func (s *Server) handleTrackRepository(ctx context.Context, req mcp.CallToolRequ } select { - case out := <-done: - if out.err != nil { - var collision *indexer.PrefixCollisionError - if errors.As(out.err, &collision) { - if collision.OwnerMissing { - return mcp.NewToolResultError(fmt.Sprintf( - "repository prefix %q is already used by %s, which no longer exists; inspect untrack_repository for that path before tracking %s again", - collision.Prefix, collision.ExistingPath, collision.Path)), nil - } - return mcp.NewToolResultError(fmt.Sprintf( - "repository prefix %q is already used by %s; for a new path set the name parameter to a different prefix for %s; for an already configured path set name: in the global config and run `gortex daemon reload`", - collision.Prefix, collision.ExistingPath, collision.Path)), nil - } - return mcp.NewToolResultError(out.err.Error()), nil - } - // Already tracked — the corpus held the repo, so only its identity - // and side effects were brought up to date. - if out.result.AlreadyTracked { - return mcp.NewToolResultText("repository already tracked"), nil - } - prefix := out.result.Prefix - if prefix == "" { - prefix = config.ResolvePrefix(entry) - } - return s.respondJSONOrTOON(ctx, req, map[string]any{ - "status": "tracked", - "path": path, - "prefix": prefix, - "file_count": out.result.Index.FileCount, - "node_count": out.result.Index.NodeCount, - "edge_count": out.result.Index.EdgeCount, - }) + case <-job.done: + return s.trackRepositoryOutcome(ctx, req, path, entry, job.outcome, false) case <-answerBy: return s.respondJSONOrTOON(ctx, req, map[string]any{ "status": "accepted", diff --git a/internal/mcp/tools_multi_track_deadline_test.go b/internal/mcp/tools_multi_track_deadline_test.go index 0553c6586..aff1e2402 100644 --- a/internal/mcp/tools_multi_track_deadline_test.go +++ b/internal/mcp/tools_multi_track_deadline_test.go @@ -12,6 +12,7 @@ import ( mcplib "github.com/mark3labs/mcp-go/mcp" "github.com/stretchr/testify/require" "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/pathkey" ) // TestTrackRepositoryDetachesFirstIndexFromRequestLifetime pins #326. @@ -137,18 +138,18 @@ func TestTrackRepositoryPrefixCollisionNamesMCPParameter(t *testing.T) { require.NotNil(t, res) text := extractTextFromContent(t, res.Content) require.True(t, res.IsError, "expected prefix collision, got: %s", text) - require.Contains(t, text, "name parameter") - require.NotContains(t, text, "gortex track --name") - require.Contains(t, text, first) - // The buffered result can arrive before the producer's deferred cleanup. - // Wait for this exact failed track worker to retire before changing the - // owner fixture and exercising a fresh missing-owner collision request. + var payload map[string]any + require.NoError(t, json.Unmarshal([]byte(text), &payload)) + require.Contains(t, payload["message"], "name parameter") + require.NotContains(t, payload["message"], "gortex track --name") + require.Equal(t, "prefix_collision", payload["error_code"]) + require.Equal(t, first, payload["owner_path"], "compare decoded paths, including Windows JSON escapes") + // Preflight rejects the collision before a track job exists, so there is + // no worker to retire before the missing-owner retry below. absSecond, err := filepath.Abs(second) require.NoError(t, err) - require.Eventually(t, func() bool { - _, busy := srv.trackInFlight.Load(absSecond) - return !busy - }, 5*time.Second, time.Millisecond, "first collision worker must retire before retry") + _, stored := srv.trackInFlight.Load(trackRequestKey(pathkey.NormalizeVolume(absSecond), config.RepoEntry{Path: second})) + require.False(t, stored, "a preflight collision must not store a track job") require.NoError(t, os.RemoveAll(first)) res, err = srv.handleTrackRepository(t.Context(), req) require.NoError(t, err) diff --git a/internal/mcp/tools_multi_worktree_test.go b/internal/mcp/tools_multi_worktree_test.go index 147b36fa6..0002af3df 100644 --- a/internal/mcp/tools_multi_worktree_test.go +++ b/internal/mcp/tools_multi_worktree_test.go @@ -35,8 +35,7 @@ func newWorktreeMCPServer(t *testing.T, repos ...config.RepoEntry) (*Server, *in preg := testRegistry() g := graph.New() mi := indexer.NewMultiIndexer(g, preg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) singleton := indexer.New(g, preg, config.IndexConfig{}, zap.NewNop()) diff --git a/internal/mcp/tools_review_rulepack_test.go b/internal/mcp/tools_review_rulepack_test.go index 2c7668063..d938323ca 100644 --- a/internal/mcp/tools_review_rulepack_test.go +++ b/internal/mcp/tools_review_rulepack_test.go @@ -51,8 +51,10 @@ func prefixedServerOver(t *testing.T, dir, name string) (*Server, string) { g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } srv := NewServer(query.NewEngine(g), g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ ConfigManager: cm, diff --git a/internal/mcp/tools_search_text_pathsep_test.go b/internal/mcp/tools_search_text_pathsep_test.go index abf0a31cb..7d9e5989b 100644 --- a/internal/mcp/tools_search_text_pathsep_test.go +++ b/internal/mcp/tools_search_text_pathsep_test.go @@ -62,8 +62,10 @@ func nestedRepoServer(t *testing.T, entries []config.RepoEntry) (*Server, *graph reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + for _, entry := range cm.Global().ReposSnapshot() { + _, err := mi.TrackRepoCtx(context.Background(), entry) + require.NoError(t, err) + } singleton := indexer.New(g, reg, config.IndexConfig{}, zap.NewNop()) srv := NewServer(query.NewEngine(g), g, singleton, nil, zap.NewNop(), nil, MultiRepoOptions{ diff --git a/internal/mcp/tools_search_text_test.go b/internal/mcp/tools_search_text_test.go index 4960e11f6..3cb886404 100644 --- a/internal/mcp/tools_search_text_test.go +++ b/internal/mcp/tools_search_text_test.go @@ -183,8 +183,7 @@ func TestSearchText_MultiRepoFanout(t *testing.T) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) require.True(t, mi.IsMultiRepo()) eng := query.NewEngine(g) @@ -239,8 +238,7 @@ func TestSearchText_MultiRepoScopedGrepDoesNotLetOutOfScopeRepoConsumeLimit(t *t reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) flagOn := true @@ -297,8 +295,7 @@ func TestFilterTextMatchesByResolvedScope_FailsClosed(t *testing.T) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) srv := NewServer(eng, g, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ @@ -367,8 +364,7 @@ func TestGetFileSummary_MultiRepoRelativePath(t *testing.T) { reg.Register(languages.NewGoExtractor()) g := graph.New() mi := indexer.NewMultiIndexer(g, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) require.True(t, mi.IsMultiRepo()) eng := query.NewEngine(g) @@ -514,8 +510,7 @@ func TestSearchTextScopedNeverScansAndRowsStayFlat(t *testing.T) { reg.Register(languages.NewGoExtractor()) counter := newScopedReadCounter(openContractCoreCapStore(t)) mi := indexer.NewMultiIndexer(counter, reg, search.NewNull(), cm, zap.NewNop()) - _, err = mi.IndexAll() - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) srv := NewServer(query.NewEngine(counter), counter, nil, nil, zap.NewNop(), nil, MultiRepoOptions{ConfigManager: cm, MultiIndexer: mi}) installContractCoreKindTestRuntime(t, srv) return srv, counter diff --git a/internal/mcp/track_outcome.go b/internal/mcp/track_outcome.go new file mode 100644 index 000000000..a3eab74d8 --- /dev/null +++ b/internal/mcp/track_outcome.go @@ -0,0 +1,144 @@ +package mcp + +import ( + "context" + "errors" + "fmt" + "time" + + "github.com/mark3labs/mcp-go/mcp" + "github.com/zzet/gortex/internal/config" + "github.com/zzet/gortex/internal/indexer" +) + +const trackOutcomeRetention = 10 * time.Minute + +type repositoryTrackOutcome struct { + result indexer.RegisterResult + err error +} + +func snapshotTrackOutcome(result indexer.RegisterResult, err error) repositoryTrackOutcome { + // Keep only wire-visible scalars. The original index result can retain + // derivation capabilities that have no role in a ten-minute retry cache. + out := indexer.RegisterResult{Prefix: result.Prefix, AlreadyTracked: result.AlreadyTracked} + if result.Index != nil { + out.Index = &indexer.IndexResult{FileCount: result.Index.FileCount, NodeCount: result.Index.NodeCount, EdgeCount: result.Index.EdgeCount} + } + return repositoryTrackOutcome{result: out, err: err} +} + +// Closing done publishes the immutable final outcome and completion time. +// Running jobs never expire. There is no timer or background cache producer. +type repositoryTrackJob struct { + done chan struct{} + outcome repositoryTrackOutcome + completed time.Time + witness indexer.TrackOutcomeWitness +} + +type repositoryTrackKey struct { + root string + name string + asWorktree bool + force bool +} + +func trackRequestKey(root string, entry config.RepoEntry) repositoryTrackKey { + return repositoryTrackKey{root: root, name: entry.Name, asWorktree: entry.AsWorktree, force: entry.Force} +} + +func (s *Server) storedTrackOutcome(ctx context.Context, req mcp.CallToolRequest, path string, entry config.RepoEntry, key repositoryTrackKey) (*mcp.CallToolResult, error, bool) { + cached, ok := s.trackInFlight.Load(key) + if !ok { + return nil, nil, false + } + job := cached.(*repositoryTrackJob) + select { + case <-job.done: + if time.Since(job.completed) < trackOutcomeRetention && s.multiIndexer.TrackOutcomeCurrent(job.witness) { + result, err := s.trackRepositoryOutcome(ctx, req, path, entry, job.outcome, true) + return result, err, true + } + s.trackInFlight.CompareAndDelete(key, job) + return nil, nil, false + default: + result, err := s.trackRepositoryAccepted(ctx, req, path) + return result, err, true + } +} + +func (s *Server) expireTrackOutcomes(now time.Time) { + s.trackInFlight.Range(func(key, value any) bool { + job := value.(*repositoryTrackJob) + select { + case <-job.done: + if now.Sub(job.completed) >= trackOutcomeRetention { + s.trackInFlight.CompareAndDelete(key, job) + } + default: + } + return true + }) +} + +func (s *Server) trackRepositoryAccepted(ctx context.Context, req mcp.CallToolRequest, path string) (*mcp.CallToolResult, error) { + return s.respondJSONOrTOON(ctx, req, map[string]any{ + "status": "accepted", "path": path, + "detail": "an initial index for this path is already running; call track again later to read its result", + }) +} + +func (s *Server) trackRepositoryOutcome(ctx context.Context, req mcp.CallToolRequest, path string, entry config.RepoEntry, out repositoryTrackOutcome, cached bool) (*mcp.CallToolResult, error) { + if out.err != nil { + if errors.Is(out.err, indexer.ErrRepoIdentityPending) { + payload := map[string]any{ + "error_code": string(ErrCodeRepoLoading), "retryable": true, + "reason": "identity_pending", "phase": "identity_resolution", "percent": 0, + "path": path, "suggestion": ServerInstructionsIdentityPending(), + } + result, err := s.respondJSONOrTOON(ctx, req, payload) + if result != nil { + result.IsError, result.StructuredContent = true, payload + } + return result, err + } + var collision *indexer.PrefixCollisionError + if !errors.As(out.err, &collision) { + return mcp.NewToolResultError(out.err.Error()), nil + } + message := fmt.Sprintf("repository prefix %q is already used by %s; for a new path set the name parameter to a different prefix for %s; for an already configured path set name: in the global config and run `gortex daemon reload`", + collision.Prefix, collision.ExistingPath, collision.Path) + if collision.OwnerMissing { + message = fmt.Sprintf("repository prefix %q is already used by %s, which no longer exists; inspect untrack_repository for that owner path before tracking %s again", collision.Prefix, collision.ExistingPath, collision.Path) + } + payload := map[string]any{ + "error_code": "prefix_collision", "message": message, + "prefix": collision.Prefix, "owner_path": collision.ExistingPath, + "owner_state": collision.OwnerState, + } + result, err := mcp.NewToolResultJSON(payload) + if err != nil { + return nil, err + } + result.IsError = true + result.StructuredContent = payload + return result, nil + } + if out.result.AlreadyTracked { + return mcp.NewToolResultText("repository already tracked"), nil + } + prefix := out.result.Prefix + if prefix == "" { + prefix = config.ResolvePrefix(entry) + } + payload := map[string]any{ + "status": "tracked", "path": path, "prefix": prefix, + "file_count": out.result.Index.FileCount, "node_count": out.result.Index.NodeCount, + "edge_count": out.result.Index.EdgeCount, + } + if cached { + payload["detail"] = "repository already tracked; returning the stored final track outcome" + } + return s.respondJSONOrTOON(ctx, req, payload) +} diff --git a/internal/mcp/track_outcome_test.go b/internal/mcp/track_outcome_test.go new file mode 100644 index 000000000..2f03cea73 --- /dev/null +++ b/internal/mcp/track_outcome_test.go @@ -0,0 +1,127 @@ +package mcp + +import ( + "context" + "os" + "path/filepath" + "sync/atomic" + "testing" + "time" + + mcplib "github.com/mark3labs/mcp-go/mcp" + "github.com/stretchr/testify/require" + "github.com/zzet/gortex/internal/config" +) + +func TestMCPDeferredCollisionOutcome(t *testing.T) { + s, mi := newWorktreeMCPServer(t) + owner, loser := filepath.Join(t.TempDir(), "src"), filepath.Join(t.TempDir(), "src") + for _, root := range []string{owner, loser} { + require.NoError(t, os.MkdirAll(root, 0o755)) + } + entered, release := make(chan struct{}), make(chan struct{}) + var calls atomic.Int32 + mi.SetOnRepoTracked(func(_, root string) { + if root == loser && calls.Add(1) == 1 { + close(entered) + <-release + } + }) + // Always release the real producer before fixture cleanup, including when + // an assertion fails before the owner is installed. + var released atomic.Bool + unblock := func() { + if released.CompareAndSwap(false, true) { + close(release) + } + } + t.Cleanup(func() { + unblock() + s.trackInFlight.Range(func(_, value any) bool { + select { + case <-value.(*repositoryTrackJob).done: + case <-time.After(10 * time.Second): + t.Error("detached track did not finish after its barrier was released") + } + return true + }) + }) + req := mcplib.CallToolRequest{} + req.Params.Arguments = map[string]any{"path": loser} + ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(trackAcceptedHeadroom+time.Millisecond)) + defer cancel() + first, err := s.handleTrackRepository(ctx, req) + require.NoError(t, err) + require.False(t, first.IsError) + require.Contains(t, extractTextFromContent(t, first.Content), "accepted") + select { + case <-entered: + case <-time.After(10 * time.Second): + t.Fatal("detached track did not reach its real pre-registration barrier") + } + _, err = mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: owner}) + require.NoError(t, err) + unblock() + key := trackRequestKey(loser, config.RepoEntry{Path: loser}) + value, ok := s.trackInFlight.Load(key) + require.True(t, ok) + job := value.(*repositoryTrackJob) + select { + case <-job.done: + case <-time.After(10 * time.Second): + t.Fatal("detached track did not physically finish") + } + second, err := s.handleTrackRepository(t.Context(), req) + require.NoError(t, err) + require.True(t, second.IsError) + payload := second.StructuredContent.(map[string]any) + require.Equal(t, "prefix_collision", payload["error_code"]) + require.Equal(t, "src", payload["prefix"]) + require.Equal(t, owner, payload["owner_path"]) + require.Equal(t, "present", payload["owner_state"]) + require.Equal(t, int32(1), calls.Load(), "reading a stored collision starts no new track") + // A remediation request has a different identity and must not replay the + // previous collision for the same physical root. + req.Params.Arguments = map[string]any{"path": loser, "name": "other-src"} + third, err := s.handleTrackRepository(t.Context(), req) + require.NoError(t, err) + require.False(t, third.IsError, "%+v", third.Content) + require.NotNil(t, mi.GetMetadata("other-src")) + require.Equal(t, int32(2), calls.Load()) +} + +func TestTrackOutcomeExpiryNeverEvictsARunningProducer(t *testing.T) { + s, mi := newWorktreeMCPServer(t) + entry := config.RepoEntry{Path: t.TempDir()} + key := trackRequestKey(entry.Path, entry) + running := &repositoryTrackJob{done: make(chan struct{})} + s.trackInFlight.Store(key, running) + s.expireTrackOutcomes(time.Now().Add(time.Hour)) + value, found := s.trackInFlight.Load(key) + require.True(t, found) + require.Same(t, running, value) + running.completed = time.Now().Add(-trackOutcomeRetention) + running.witness = mi.CaptureTrackOutcome("", "") + close(running.done) + s.expireTrackOutcomes(time.Now()) + _, found = s.trackInFlight.Load(key) + require.False(t, found) +} + +func TestMCPConfiguredCollisionPrecedesAlreadyConfigured(t *testing.T) { + s, mi := newWorktreeMCPServer(t) + owner, loser := filepath.Join(t.TempDir(), "src"), filepath.Join(t.TempDir(), "src") + for _, root := range []string{owner, loser} { + require.NoError(t, os.MkdirAll(root, 0o755)) + } + _, err := mi.TrackRepoCtx(t.Context(), config.RepoEntry{Path: owner}) + require.NoError(t, err) + mi.ExpectRepos([]config.RepoEntry{{Path: owner}, {Path: loser}}) + req := mcplib.CallToolRequest{} + req.Params.Arguments = map[string]any{"path": loser} + result, err := s.handleTrackRepository(t.Context(), req) + require.NoError(t, err) + require.True(t, result.IsError) + require.Equal(t, "prefix_collision", result.StructuredContent.(map[string]any)["error_code"]) + require.NotContains(t, extractTextFromContent(t, result.Content), "already_configured") +} diff --git a/internal/mcp/view_ref_test.go b/internal/mcp/view_ref_test.go index 12736d83c..027c699af 100644 --- a/internal/mcp/view_ref_test.go +++ b/internal/mcp/view_ref_test.go @@ -168,9 +168,7 @@ func newUnadmittedRefStack(t *testing.T) *refStack { languages.RegisterAll(reg) bm := search.NewNull() mi := indexer.NewMultiIndexer(store, reg, bm, cm, zap.NewNop()) - if _, err := mi.IndexScoped("", ""); err != nil { - t.Fatalf("index the fixture repository: %v", err) - } + registerMCPFixtureRepos(t, mi, cm) graphID := indexer.GraphIDFor(refTestPrefix) seedRefCatalog(t, store, graphID, repo, headTree) diff --git a/internal/mcp/view_request_test.go b/internal/mcp/view_request_test.go index e82421293..5b674ab22 100644 --- a/internal/mcp/view_request_test.go +++ b/internal/mcp/view_request_test.go @@ -154,9 +154,7 @@ func newViewStackWithRepos(t *testing.T, includeOther bool) *viewStack { languages.RegisterAll(reg) bm := search.NewNull() mi := indexer.NewMultiIndexer(store, reg, bm, cm, zap.NewNop()) - if _, err := mi.IndexScoped("", ""); err != nil { - t.Fatalf("index fixture repos: %v", err) - } + registerMCPFixtureRepos(t, mi, cm) graphID := indexer.GraphIDFor("repo") stack := &viewStack{ diff --git a/internal/mcp/workspace_isolation_test.go b/internal/mcp/workspace_isolation_test.go index 6801f1000..13d272a10 100644 --- a/internal/mcp/workspace_isolation_test.go +++ b/internal/mcp/workspace_isolation_test.go @@ -59,8 +59,7 @@ func newIsolationServer(t *testing.T) (srv *Server, repoA, repoB string) { reg := testRegistry() bm := search.NewNull() mi := indexer.NewMultiIndexer(g, reg, bm, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") // index every configured repo - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) // index every configured repo eng := query.NewEngine(g) eng.SetSearch(bm) diff --git a/internal/mcp/workspace_root_scope_test.go b/internal/mcp/workspace_root_scope_test.go index 386e9df04..4c478f239 100644 --- a/internal/mcp/workspace_root_scope_test.go +++ b/internal/mcp/workspace_root_scope_test.go @@ -67,8 +67,7 @@ func newContainedFixture(t *testing.T) containedFixture { g := graph.New() bm := search.NewNull() mi := indexer.NewMultiIndexer(g, testRegistry(), bm, cm, zap.NewNop()) - _, err = mi.IndexScoped("", "") - require.NoError(t, err) + registerMCPFixtureRepos(t, mi, cm) eng := query.NewEngine(g) eng.SetSearch(bm) diff --git a/internal/platform/process_mem.go b/internal/platform/process_mem.go index 996fa6481..d04fe0af2 100644 --- a/internal/platform/process_mem.go +++ b/internal/platform/process_mem.go @@ -1,5 +1,11 @@ package platform +import ( + "bufio" + "bytes" + "strconv" +) + // ProcessMemory is the operating system's view of this process's memory, // beyond what the Go runtime can see: resident pages, and on macOS the // physical footprint (resident plus compressed and swapped anonymous pages, @@ -13,13 +19,49 @@ type ProcessMemory struct { // maximum physical footprint (macOS proc_pid_rusage); 0 elsewhere. FootprintBytes uint64 PeakFootprintBytes uint64 + // RSSBytes is the current resident set (macOS proc_pid_rusage, Linux + // /proc/self/smaps_rollup); 0 elsewhere. + RSSBytes uint64 + // PSSBytes is the Linux proportional set: resident pages, each page shared + // with other processes divided among them. PSSFileBytes is its file-backed + // part — memory-mapped store files and the executable — and is 0 on + // kernels that predate the split. Both are 0 off Linux. + PSSBytes uint64 + PSSFileBytes uint64 } // ReadProcessMemory samples the process's memory as the operating system -// accounts it. It is cheap (two system calls) and never fails: a figure the -// platform cannot provide is 0. +// accounts it. It never fails: a figure the platform cannot provide is 0. It +// is a few system calls, but on Linux the kernel walks the page tables to sum +// the rollup — milliseconds for a process of several GiB — so read it on +// demand, not on a hot path. func ReadProcessMemory() ProcessMemory { out := ProcessMemory{PeakRSSBytes: peakRSSBytes()} - out.FootprintBytes, out.PeakFootprintBytes = physFootprintBytes() + readCurrentMemory(&out) return out } + +// parseSmapsRollup fills the resident and proportional set sizes from the +// contents of /proc//smaps_rollup, whose counters are in kB. +func parseSmapsRollup(data []byte, out *ProcessMemory) { + scanner := bufio.NewScanner(bytes.NewReader(data)) + for scanner.Scan() { + key, rest, ok := bytes.Cut(scanner.Bytes(), []byte(":")) + fields := bytes.Fields(rest) + if !ok || len(fields) == 0 { + continue + } + kb, err := strconv.ParseUint(string(fields[0]), 10, 64) + if err != nil { + continue + } + switch string(key) { + case "Rss": + out.RSSBytes = kb << 10 + case "Pss": + out.PSSBytes = kb << 10 + case "Pss_File": + out.PSSFileBytes = kb << 10 + } + } +} diff --git a/internal/platform/process_mem_darwin.go b/internal/platform/process_mem_darwin.go index 85c58e259..836aefa5e 100644 --- a/internal/platform/process_mem_darwin.go +++ b/internal/platform/process_mem_darwin.go @@ -6,24 +6,25 @@ package platform #include #include -static int gortex_phys_footprint(unsigned long long *current, unsigned long long *peak) { +static int gortex_process_memory(unsigned long long *footprint, unsigned long long *peak, unsigned long long *resident) { struct rusage_info_v4 info; if (proc_pid_rusage(getpid(), RUSAGE_INFO_V4, (rusage_info_t *)&info) != 0) { return -1; } - *current = info.ri_phys_footprint; + *footprint = info.ri_phys_footprint; *peak = info.ri_lifetime_max_phys_footprint; + *resident = info.ri_resident_size; return 0; } */ import "C" -// physFootprintBytes is the process's current and lifetime maximum physical -// footprint (proc_pid_rusage RUSAGE_INFO_V4). -func physFootprintBytes() (current, peak uint64) { - var cur, max C.ulonglong - if C.gortex_phys_footprint(&cur, &max) != 0 { - return 0, 0 +// readCurrentMemory reads the process's current and lifetime maximum physical +// footprint and its current resident size (proc_pid_rusage RUSAGE_INFO_V4). +func readCurrentMemory(out *ProcessMemory) { + var footprint, peak, resident C.ulonglong + if C.gortex_process_memory(&footprint, &peak, &resident) != 0 { + return } - return uint64(cur), uint64(max) + out.FootprintBytes, out.PeakFootprintBytes, out.RSSBytes = uint64(footprint), uint64(peak), uint64(resident) } diff --git a/internal/platform/process_mem_linux.go b/internal/platform/process_mem_linux.go new file mode 100644 index 000000000..c2a0f1446 --- /dev/null +++ b/internal/platform/process_mem_linux.go @@ -0,0 +1,13 @@ +//go:build linux + +package platform + +import "os" + +// readCurrentMemory reads the resident and proportional set sizes from the +// kernel's per-process rollup of /proc/self/smaps. +func readCurrentMemory(out *ProcessMemory) { + if data, err := os.ReadFile("/proc/self/smaps_rollup"); err == nil { + parseSmapsRollup(data, out) + } +} diff --git a/internal/platform/process_mem_nofootprint.go b/internal/platform/process_mem_nofootprint.go deleted file mode 100644 index 9d85f5d17..000000000 --- a/internal/platform/process_mem_nofootprint.go +++ /dev/null @@ -1,5 +0,0 @@ -//go:build !(darwin && cgo) - -package platform - -func physFootprintBytes() (current, peak uint64) { return 0, 0 } diff --git a/internal/platform/process_mem_other.go b/internal/platform/process_mem_other.go new file mode 100644 index 000000000..39ba31354 --- /dev/null +++ b/internal/platform/process_mem_other.go @@ -0,0 +1,5 @@ +//go:build !linux && !(darwin && cgo) + +package platform + +func readCurrentMemory(*ProcessMemory) {} diff --git a/internal/platform/process_mem_test.go b/internal/platform/process_mem_test.go index b69e0ad2c..1a055790b 100644 --- a/internal/platform/process_mem_test.go +++ b/internal/platform/process_mem_test.go @@ -8,7 +8,9 @@ import ( // TestReadProcessMemoryReportsThisProcess: the peak resident set is known on // every Unix system and at least what the Go runtime holds resident now; on // macOS with cgo the physical footprint is too, and its lifetime peak is at -// least the current figure. Allocating and touching 64 MiB raises the peaks. +// least the current figure. Allocating and touching 64 MiB raises the peaks +// and puts at least that much in the current resident (and, on Linux, +// proportional) set. func TestReadProcessMemoryReportsThisProcess(t *testing.T) { before := ReadProcessMemory() buf := make([]byte, 64<<20) @@ -38,4 +40,32 @@ func TestReadProcessMemoryReportsThisProcess(t *testing.T) { t.Fatalf("peak footprint went down: %d -> %d", before.PeakFootprintBytes, after.PeakFootprintBytes) } } + if (runtime.GOOS == "linux" || runtime.GOOS == "darwin" && cgoEnabled) && after.RSSBytes < 64<<20 { + t.Fatalf("resident set %d is below the 64 MiB this test touched", after.RSSBytes) + } + if runtime.GOOS == "linux" && (after.PSSBytes < 64<<20 || after.PSSBytes > after.RSSBytes) { + t.Fatalf("proportional set %d, resident set %d", after.PSSBytes, after.RSSBytes) + } +} + +func TestParseSmapsRollup(t *testing.T) { + var got ProcessMemory + parseSmapsRollup([]byte(`55d4c0a00000-7ffd5e5fe000 ---p 00000000 00:00 0 [rollup] +Rss: 2097152 kB +Pss: 1048576 kB +Pss_Dirty: 524288 kB +Pss_Anon: 786432 kB +Pss_File: 262144 kB +Pss_Shmem: 0 kB +Anonymous: 786432 kB +`), &got) + if want := (ProcessMemory{RSSBytes: 2 << 30, PSSBytes: 1 << 30, PSSFileBytes: 256 << 20}); got != want { + t.Fatalf("parsed %+v, want %+v", got, want) + } + // A kernel that predates the Pss_File split still reports the totals. + got = ProcessMemory{} + parseSmapsRollup([]byte("Rss: 4 kB\nPss: 2 kB\nnot a counter\n"), &got) + if want := (ProcessMemory{RSSBytes: 4096, PSSBytes: 2048}); got != want { + t.Fatalf("parsed %+v from an older kernel's rollup, want %+v", got, want) + } } diff --git a/internal/sqliteconn/page_cache.go b/internal/sqliteconn/page_cache.go new file mode 100644 index 000000000..1e6d524ab --- /dev/null +++ b/internal/sqliteconn/page_cache.go @@ -0,0 +1,31 @@ +package sqliteconn + +import ( + "unsafe" + + "modernc.org/libc" + sqlite3 "modernc.org/sqlite/lib" +) + +// PageCacheBytes reports how many bytes the page caches of every SQLite +// connection in the process hold now, and the most they have held since the +// process started. The memory comes from SQLite's C allocator, outside the Go +// heap and its accounting. +// +// The pinned driver builds SQLite with SQLITE_DEFAULT_MEMSTATUS=0, so +// sqlite3_memory_used stays 0; turning the allocator statistics on would put +// a mutex on every SQLite allocation for the life of the process. The page +// cache keeps its own counter regardless: with no SQLITE_CONFIG_PAGECACHE +// buffer every page is an "overflow" allocation, counted under the page-cache +// mutex. Only each cache's small initial bulk block (20 pages) escapes it. +func PageCacheBytes() (current, peak int64) { + tls := libc.NewTLS() + defer tls.Close() + out := tls.Alloc(16) + defer tls.Free(16) + high := out + 8 + if sqlite3.Xsqlite3_status64(tls, sqlite3.SQLITE_STATUS_PAGECACHE_OVERFLOW, out, high, 0) != sqlite3.SQLITE_OK { + return 0, 0 + } + return **(**int64)(unsafe.Pointer(&out)), **(**int64)(unsafe.Pointer(&high)) +} diff --git a/internal/sqliteconn/page_cache_test.go b/internal/sqliteconn/page_cache_test.go new file mode 100644 index 000000000..37840c445 --- /dev/null +++ b/internal/sqliteconn/page_cache_test.go @@ -0,0 +1,36 @@ +package sqliteconn + +import ( + "path/filepath" + "testing" + + "github.com/stretchr/testify/require" +) + +// TestPageCacheBytesFollowsConnections: the pinned driver builds SQLite with +// its allocator statistics off, so the page-cache counter is the one memory +// figure SQLite still keeps. Writing 8 MiB through a connection whose cache +// can hold it must raise the process-wide figure by most of that, and closing +// the connection must give it back. +func TestPageCacheBytesFollowsConnections(t *testing.T) { + const written = 8 << 20 + before, _ := PageCacheBytes() + + db, err := Open(filepath.Join(t.TempDir(), "cache.db")) + require.NoError(t, err) + db.SetMaxOpenConns(1) + ctx := t.Context() + _, err = db.ExecContext(ctx, `PRAGMA cache_size=-16384; CREATE TABLE t(v BLOB)`) + require.NoError(t, err) + _, err = db.ExecContext(ctx, `WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i+1 FROM n WHERE i < 2048) + INSERT INTO t SELECT randomblob(4096) FROM n`) + require.NoError(t, err) + + cached, peak := PageCacheBytes() + require.GreaterOrEqual(t, cached-before, int64(written/2), "page cache before %d, after writing %d bytes %d", before, written, cached) + require.GreaterOrEqual(t, peak, cached) + + require.NoError(t, db.Close()) + closed, _ := PageCacheBytes() + require.LessOrEqual(t, closed, cached-written/2, "closing the only connection kept its page cache: %d -> %d", cached, closed) +}