Skip to content

indexer: bound checkout discovery per family, not daemon-wide - #1235

Draft
zzet wants to merge 4 commits into
mainfrom
fix/scope-checkout-discovery-per-family
Draft

zzet wants to merge 4 commits into
mainfrom
fix/scope-checkout-discovery-per-family

Conversation

@zzet

@zzet zzet commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Every new-path checkout discovery shared one daemon-wide limit of 32 jobs, and a job was allocated before its family was known. One hung mount, a burst of worktree adds in one family, or sessions opened in untracked Git directories could fill that limit. From then on, every new worktree in every family was refused with "capacity reached". This draft bounds discovery per family, so a pending discovery is refused only inside its own worktree.

Draft status: the first three commits are done and tested. The last commit is a labelled wip: commit holding review fixes that have not been compiled or run yet, so CI on this draft may be red.

Changes

  • gitstate.MarkerCommonDir. It reads a checkout's family common dir from its .git marker, with the reader the dirty sampler already uses, and resolves symlinks the way git rev-parse --path-format=absolute does. No Git process is spawned. A test asserts it matches Git for a main checkout, a linked worktree, a subdirectory and a relative path.
  • Indexer: per-family bound.
    • The family is resolved from the marker before a job is allocated. A path outside Git, or in an unknown family, returns not-found with no job and no Git spawn.
    • A discovery already running for a path still answers it, so a binding that changes under a pending proof is still reported stale.
    • Each family runs at most 8 discoveries at once, and only that family's new paths are refused while it is full. Finished jobs that only keep a result do not count.
    • The daemon-wide bound of 32 now counts only workers still running past their work deadline, and refuses only families that own one.
    • The proof inside the job is unchanged.
  • Typed refusal. A refusal is a CheckoutDiscoveryPendingError naming the checkout root, the family and a reason: in_progress, draining, family_capacity or stuck_capacity. It still unwraps to ErrCheckoutMutationBusy, so every caller keeps treating it as retryable.
  • Daemon admission refusal. It carries data.path (the checkout root), data.family, data.reason and state: discovery_pending, and stays retryable: true.

Tested (first three commits, at 213c320)

  • On the base, the new indexer and daemon tests fail: one family fills all 32 discovery slots, and another family's new worktree is refused with "capacity reached".
  • At 213c320, -race runs pass, 108 top-level tests in all:
    • gitstate TestResolveFamilyDirs*;
    • indexer TestObserveCheckoutPath*, including both new stuck-family tests and every existing binding-change subtest;
    • the internal/mcp checkout and worktree discovery, admission, source, mutation, freshness and ref-view tests;
    • the cmd/gortex admission, discovery and reload tests.
  • cmd/gortex coverage includes:
    • the new daemon-level TestDispatcher_PendingDiscoveryStaysInItsWorktree: one family's discovery is hung; another family's worktree, the primary checkout and an unrelated tracked repository keep answering, and only paths inside the hung worktree are refused;
    • TestCheckoutAdmissionErrorPendingDiscoveryGolden.
  • golangci-lint 2.14.0 over gitstate, indexer, mcp and cmd/gortex reports 0 issues.

Untested wip commit (926c2bb)

This commit holds the fixes for the last review round. None of them has been compiled or run.

  • A request that joins a running discovery is refused with that job's own root and family, not with what its own marker read finds now.
  • A proven job that no caller has authorized runs no Git work, so it no longer counts against its family's bound.
  • data.path falls back to the session path when a refusal names no root.
  • Two tests are tightened:
    • the indexer test now fails on any refusal of another family's worktree other than its own in_progress;
    • the daemon test first waits for a stuck session to be refused as draining.
  • New tests: TestObserveCheckoutPathJoinedRefusalNamesItsOwnCheckout and TestObserveCheckoutPathDeniedDiscoveriesLeaveFamilyCapacity.

Once these fixes pass, they will be squashed into the indexer and daemon commits.

Not done

Refs #1033

zzet added 4 commits October 31, 2026 17:22
Checkout discovery needs to know whether a request path belongs to a known
Git family before it spends a discovery slot on it. Asking Git costs a
process spawn per request, which is exactly the work a path in an unknown
family should never cause.

MarkerCommonDir walks up to the nearest .git marker and follows it with the
reader the dirty sampler already uses (a .git directory or a gitdir file,
then commondir), then resolves symlinks the way `git rev-parse
--path-format=absolute` reports the common dir, so both key the same family.
The family-dirs tests now assert the marker read matches Git for a main
checkout, a linked worktree, a subdirectory and a relative path, and that it
finds nothing outside Git.
Every new-path discovery shared one daemon-wide limit of 32 jobs, and a job
was allocated before its family was known. One hung mount, a burst of
worktree adds in one family, or sessions opened in untracked Git
directories could fill it, and from then on every new worktree in every
family was refused with "capacity reached".

Resolve the family from the .git marker before allocating: a path outside
Git or in an unknown family returns not-found without a job or a Git spawn.
A discovery already running for a path still answers it, so a binding that
changes under its pending proof is still reported stale.
A family runs at most 8 discoveries at once, and only its own new paths are
refused while it is full; finished jobs that only retain a result do not
count. The daemon-wide bound of 32 now counts only workers still running
past their work deadline and refuses only families that own one, so work
that ignores cancellation stays capped. The proof inside the job and its
stale checks are unchanged.

Refusals are a CheckoutDiscoveryPendingError naming the checkout root, the
family and a reason (in_progress, draining, family_capacity,
stuck_capacity). It still unwraps to ErrCheckoutMutationBusy, so every
caller keeps treating it as retryable.
A discovery-pending admission refusal reported only the session cwd, so an
agent or hook could not tell one pending worktree from an unavailable
daemon. It now carries the checkout root as data.path, plus data.family,
data.reason and state discovery_pending, and stays retryable.

A daemon-level test pins the containment: with one family's discovery hung
past its deadline, sessions in its primary checkout, in a directory beside
an unregistered nested worktree and in an unrelated tracked repository keep
answering, another family's new worktree gets its identity, and only paths
inside the pending worktrees are refused. checkoutForRequestPath now states
that guard.
Unfinished review fixes: build the join and draining refusals from the pending job's own root and family, leave idle authorization waits out of the family bound, and tighten two tests. The focused tests have not run on this commit; squash into the indexer and daemon commits once they pass.
@zzet zzet added this to the v0.64.12 milestone Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant