Skip to content

feat(bench): probe startup admission from before the socket exists - #1236

Closed
zzet wants to merge 35 commits into
mainfrom
feat/startup-admission-probe
Closed

zzet wants to merge 35 commits into
mainfrom
feat/startup-admission-probe

Conversation

@zzet

@zzet zzet commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Adds a startup scenario to the bench/load harness. It records what agents actually receive while a daemon starts: from before its socket exists (including a first boot that runs a schema upgrade), through loading, to ready, on an initial boot and on warm restarts. It produces the runtime evidence #1023 asks for. Its time-to-ready and time-to-first-success figures can be compared between builds with delta.

Includes PR #1162. This branch was cut from #1162's refreshed head (252b9d2), so it contains all of #1162's commits. Only the last commit (ac0447a, feat(bench): probe startup admission from before the socket exists) is new here. #1162 must land first; after that, this PR's diff is that one commit. The commit touches bench/load only, and bench/load on main is the same as on #1162's head, so it also applies to main on its own.

Draft status: the probe is written, but none of its checks have run yet (see "Testing").

What it does

  • --scenario startup is opt-in and never part of all. It configures every manifest repository and adds --worktrees linked checkouts of the first one.
  • --seed-store copies an existing store into the private run first, read-only from its source, so the initial boot runs its migration before the socket listens. --restarts N adds N warm restarts.
  • Session probes. Every second, it probes from three directories: the primary, a linked worktree, and a directory whose .gortex.yaml names a workspace claimed by a private servers.toml roster. From each, it opens fresh sessions and records:
    • initialize (which instructions variant was served), search_symbols and resources/read gortex://index-health on the legacy surface;
    • workspace(operation:"index") on the compact surface.
  • Hook and CLI probes. From the same directories it records the PreToolUse hook's decision on a native Read every second. It also runs gortex call search_symbols through a pipe and through a pty, each until its first success in that boot.
  • What each answer keeps: error_code, retryable, phase, percent, latency, exit code, and whether a refusal advises gortex track or gortex repos reconcile. Negated guidance is ignored.
  • Watcher. It polls the readiness probe every 100 ms. For 2 s after the primary registers, it also reads the checkout coordinator count from daemon status.
  • Output:
    • a per-second startup-probe.jsonl;
    • a startup_probe gate that counts answers per surface, directory and window (no socket, loading, ready), plus per-boot timings;
    • headline metrics for delta, such as startup_repo_not_tracked_before_ready, startup_nonconforming_loading_refusals, startup_restart_time_to_ready_p95_ms, startup_initial_time_to_ready_ms and startup_coordinators_within_2s_min. Restart figures are reported unavailable unless every restart produced them.

The scenario lives in new startup_probe*.go files. The shared harness files gain only its registration, a few lines each. The fixture daemon now answers the CLI and hook probes, so the whole scenario runs end to end in tests.

Testing

The tests are written but have not been run: the local check run (gofmt, go vet on three platforms, focused race tests, an end-to-end test run three times, pinned lint) was interrupted before it started. CI on this PR is their first run. The written tests cover:

  • unit tests for the decoder, advice detection, CLI and hook classification, response framing, first-success timing and the per-window counters;
  • an end-to-end test that runs the whole scenario against the harness's fixture daemon across one restart.

Not done

Refs #1023

zzet added 30 commits October 28, 2026 02:15
zzet added 5 commits October 28, 2026 10:14
The janitor ownership test arrived from main still seeding its graph
with MultiIndexer.IndexAll, which this branch keeps only as an indexer
test fixture, so the cmd/gortex test build no longer compiled after the
merge. Register the configured repository with TrackRepoCtx instead,
the same production path the other daemon fixtures already use. The
test asserts janitor and shared-stack lifetime, not batch resolution,
so per-repository registration covers what it needs.
Startup loading is covered by unit and integration tests only. Nothing
records what an agent actually receives while a daemon starts: before its
socket exists (including a first boot that runs a schema migration), while
configured repositories load, and once they are ready, on a cold start and
on warm restarts.

The opt-in startup scenario starts probing before the first daemon start.
Every second, from the primary, a linked worktree and a directory a
servers.toml roster claims, it opens fresh sessions and records the
initialize instructions variant, search_symbols, the index-health resource
and workspace(operation:"index"); asks the PreToolUse hook about a native
Read; and runs `gortex call search_symbols` through a pipe and a pty until
each first succeeds in that boot. Every answer keeps error_code, retryable,
phase, percent, latency, exit code and whether a refusal advises tracking
or reconciling. A watcher stamps socket, readiness and enrichment, and
reads the checkout coordinator count for 2 s after the primary registers.

--seed-store copies an existing store into the private run so the initial
boot is an upgrade; --restarts adds warm restarts. The gate counts answers
per surface, directory and window and reports time to ready and time to
first success per boot, so `delta` can compare startup between builds.

The scenario lives in new files; the shared harness files gain only its
registration, and the fixture daemon answers the CLI and hook probes so
the whole scenario runs end to end in tests.
@zzet

zzet commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #1162, which now contains the exact probe commit ac0447a in its published ancestry. The combined integration passed focused race and prescribed pinned checks before publication; fresh exact-head CI runs on #1162. The original probe branch and its evidence remain preserved. Startup runtime and performance acceptance remain open in #1023.

@zzet zzet closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant