diff --git a/.changeset/port-claim-settle-why.md b/.changeset/port-claim-settle-why.md new file mode 100644 index 0000000..fe2c99e --- /dev/null +++ b/.changeset/port-claim-settle-why.md @@ -0,0 +1,5 @@ +--- +"minor" +--- + +Lock resume waits until claimed ports are free before re-ensuring, so a pause no longer races a dirty bind. Composite servers declare a port span or named subports so sibling worktrees rebind a whole claim block (relative offsets move, absolute ports stay singleton), and `devctl why` keeps the refusal lines from the last run across ensure retries instead of going blank on exit 0. diff --git a/.changeset/quiet-upgrade-restore.md b/.changeset/quiet-upgrade-restore.md new file mode 100644 index 0000000..e82e2ab --- /dev/null +++ b/.changeset/quiet-upgrade-restore.md @@ -0,0 +1,5 @@ +--- +"devctl": patch +--- + +Installing or restarting the daemon no longer floods the menu bar with false "server crashed" alerts for the expected bounce, and boot restore finishes before clients connect so upgrade re-ensure does not race and leave servers on a bad port. `devctl why` also diagnoses file-backed projects correctly, and an unexpected server exit tears down leftover child processes so the next ensure is not blocked by stale locks. diff --git a/.changeset/worktree-coexistence.md b/.changeset/worktree-coexistence.md new file mode 100644 index 0000000..70f245a --- /dev/null +++ b/.changeset/worktree-coexistence.md @@ -0,0 +1,5 @@ +--- +"devctl": minor +--- + +Two git worktrees of one repo can both run under supervision without editing `devservers.json`. When a sibling checkout already holds the committed port, `ensure` auto-rebinds to a free port, keeps the main origin on the declared host, and gives the worktree an ephemeral `worktree-*..localhost` URL; unrelated projects and unmanaged listeners still get a loud `port-held` naming the holder. Session context and ensure/status JSON warn about latent and rebound conflicts so agents use the live URL, and `lock --no-pause` lets a harness take the mutex without stopping the server it reuses. Discarding a worktree stops its servers and frees the ephemeral host without a manual doctor pass. Bad-state servers in the session-start block still lead with a `devctl why` recommendation and a stderr line count, never the server's own output. diff --git a/BACKLOG.md b/BACKLOG.md index 05984f4..6bf896c 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -3,11 +3,16 @@ Open work only; entries are removed by the change that resolves them. - Orphan re-adoption without the bounce: after a daemon crash, re-adopt live orphan servers (pid + start-time match, resume spool tailing) instead of group-kill + restart. Blocked on: exit codes are unknowable for non-children; needs a design for degraded forensics. -- Reverse proxy on :80/:443 routing by host signature, making ports disappear from `*.localhost` URLs (Valet/Herd territory). +- Reverse proxy on :80/:443 routing by host signature, making ports disappear from `*.localhost` URLs (Valet/Herd territory). Ephemeral `worktree-*..localhost` hosts are the unprivileged addressing half; the proxy would drop ports from URLs entirely. - MenuBarExtraAccess (orchetect) if `.window` presentation quirks bite in practice. - Populate Apple Developer ID + App Store Connect API secrets for `.github/workflows/release-dmg.yml` (and a Homebrew tap) so the Release→DMG dispatch can notarize on Actions; until then, mint locally (`SIGN_IDENTITY=… make dmg` + `scripts/notarize.sh` + `gh release upload`). `v1.3.0` already has a stapled DMG from the local path. - App Intents / Shortcuts wrappers over the existing `DeepLink` verbs (`open`, `ensure`, `stop`, `why`) for Siri / Gemini-Siri and Control Center. The `devctl://` URL table and `DeepLinkRunner` are the shared surface; intents should call the runner, not reimplement dispatch. Also the next plausible path for Spotlight ranking (IndexedEntity) once Core Spotlight levers are exhausted. - swift-subprocess 0.5 occasionally fatals in its kqueue AsyncIO cleanup at process exit ("Failed to close kqueue fds: Bad file descriptor"), seen once under parallel test load; harmless to the long-lived daemon but track against upstream releases (pinned revision in Package.swift). -- Field-level config editing in the dashboard to preserve `devservers.json` formatting instead of normalizing writes. +- Lock-release false `crashed` (2026-07-25 pnpm monorepo, healthy then exit 0 in ~230ms): unreproducible on fixture rapid acquire/release (N=20), grandchild fixture, and live pirl `lock d1` cycles (2026-07-28). Resume now settles until the PortClaim is free or refuses to spawn dirty. Reopen with a failing repro that names exiting pid vs resumed pid before changing the exit classifier. - Spotlight thumbnails: confirm config icons render in the real Spotlight UI. Within-app ranking levers are maxed (lastUsed preserved across sync, incremental index updates, live/pinned rankingHint, alternateNames). Outranking filesystem / Cursor Top Hits remains an Apple ceiling; do not chase without a new system API. -- Automatic port deconflicting / pre-spawn allocation: today is refuse (`port-held`) + observe framework bumps (`observedPort`). True auto-pick needs env injection, URL/head rewrite, and ephemeral-vs-committed policy. +- Machine-wide resource lock opt-in: locks are already path-scoped (`canonicalPath::resource`) and pause only that path's declarers. A rare shared system resource (one Docker Postgres, a fixed system daemon) may still want an explicit machine-wide scope so two projects serialize. Not the Cloudflare worktree pause case (that was misread against path-scoped keys); defer until a concrete cross-project shared-resource incident. +- `healthcheck.type: "http"` with no `url` silently degrades to a TCP probe on the declared port, or to `.none` (`EffectiveHealthcheck.resolve`), with no config-check warning. A typo in the `url` key yields a green server that was never HTTP-probed. The validator should warn (or error) on `type: http` with no resolvable url. +- `shell: true` joins argv with spaces (`ServerSupervisor.effectiveArgv`), so any argv word containing a space or glob is re-split by zsh. The two spawn modes are not equivalent; a shell-quoting pass (or documenting the limit) would close the gap. +- `IntegrationTests` is a single `placeholder()` while `docs/design.md` promises a real end-to-end suite there (port-conflict from a second project, concurrent double-ensure). Those now live in `scripts/smoke.sh` and unit suites; either build the integration target out or retire the promise in the design doc. +- The `` algorithm exists twice, byte-identical: inline in `Paths.serverLogDir` and `ProjectConfigLoader.defaultSlug`. Consolidate to one home. +- `CLAUDE.md`'s codebase map lists `Switch`, `Lock`, `Link` as files under `Sources/devctl` and `PortGuard` under `Health/`; the first three are structs inside the ~1500-line `CLI.swift` and `PortGuard` sits in `HealthProber.swift`. Either fix the map or split those command structs into their own files (which also brings `CLI.swift` back under the size where splitting is worth asking about). diff --git a/CLAUDE.md b/CLAUDE.md index 7599d08..681cc15 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -9,10 +9,10 @@ Identity and stack - Three products, one daemon: devctld owns all server processes; devctl (CLI) and devctl.app (SwiftUI MenuBarExtra) are thin clients over a unix socket (default ~/Library/Application Support/devctl/daemon.sock; DEVCTL_SOCKET overrides; /tmp fallback near the sun_path limit), NDJSON protocol. devctl daemon install/uninstall/start/stop/restart manage the LaunchAgent (dev.quantizor.devctl); tests and the smoke gate run devctld --foreground. Codebase map -- Sources/DevCtlKit: shared core, the unit-test target of record. Models.swift (specs, phases, ServerStatus), Wire.swift (JSONCoding, typed request/response/event frames, NDJSON framing, stable error codes), Client/DaemonClient.swift (blocking-POSIX socket actor used unchanged by CLI and app), Paths/Paths.swift (path constants, canonical project path, atomic write + defensive load with per-call unique temp names so two writers in one process cannot rename each other's temp away, portable SHA-256; agent.path holds login-shell PATH for the daemon), Setup/SetupPlanner.swift (first-run / upgrade decisions, harness offers, stage-and-rename binary install), Launchd/ (LaunchdAdmin: dual install path; SMAppService via app deep link when /Applications/devctl.app exists, else legacy home LaunchAgent + Application Support bin/devctld; --legacy forces the home path; DaemonRecoveryPolicy decides whether an unreachable daemon is auto-restarted; AgentRebindPolicy + agent.rebind settle the ad-hoc CDHash window on DMG replace), DeepLink/ (parse/serialize + DeepLinkRunner + notification action map), Log/DevCtlLog.swift (OSLog facade with a recording backend for tests). +- Sources/DevCtlKit: shared core, the unit-test target of record. Models.swift (specs, phases, ServerStatus), Wire.swift (JSONCoding, typed request/response/event frames, NDJSON framing, stable error codes), Client/DaemonClient.swift (blocking-POSIX socket actor used unchanged by CLI and app), Paths/Paths.swift (path constants, canonical project path, atomic write + defensive load with per-call unique temp names so two writers in one process cannot rename each other's temp away, portable SHA-256; agent.path holds login-shell PATH for the daemon), Setup/SetupPlanner.swift (first-run / upgrade decisions, harness offers, stage-and-rename binary install), Agent/ (AgentContext: the pure session-context renderer the hook injects, bad-state servers first with a devctl why recommendation and devctl's own stderr count, never raw child output; DiscoveryStanza), Net/LoopbackProbe.swift (dual-stack loopback listen probe shared by the daemon port pre-check and CLI doctor), Net/PortClaim.swift + PortMaterializer.swift (effectivePort claim: portSpan and named ports; env injection and URL rewrite), Launchd/ (LaunchdAdmin: dual install path; SMAppService via app deep link when /Applications/devctl.app exists, else legacy home LaunchAgent + Application Support bin/devctld; --legacy forces the home path; DaemonRecoveryPolicy decides whether an unreachable daemon is auto-restarted; AgentRebindPolicy + agent.rebind settle the ad-hoc CDHash window on DMG replace), DeepLink/ (parse/serialize + DeepLinkRunner + notification action map), Log/DevCtlLog.swift (OSLog facade with a recording backend for tests). - Sources/DevCtlDaemonCore: daemon logic as a library. Supervisor/ (ServerSupervisor actor per server: spawn, spool capture, health-gated phase machine, ensure/wait, group + descendant teardown with ProcessIdentity start-time revalidation; the ProcessLauncher seam; ProcessTree QA1123 sysctl sweep), Health/ (EffectiveHealthcheck resolution, the HealthProber seam with ephemeral URLSession HTTP probes + BSD TCP, PortGuard lsof diagnostics), Registry/ (owner of registry.json and state.json), Control/ (Router method dispatch + port pre-check + persisted resource locks with daemon-owned pause/resume and dead-holder auto-release + NWListener ControlServer with stateUpdateHandlers). - Sources/devctld: thin main; identical behavior under launchd and --foreground (tests and the smoke gate use foreground). Applies agent.path into process env before spawn. -- Sources/devctl: CLI (swift-argument-parser); HookSupport (AgentContext + HarnessAdapter registry; adding a harness: CONTRIBUTING.md), Switch (branch switching + lifecycle playbooks), Lock (run-under-resource-lock), Link / x-url (deep links). +- Sources/devctl: CLI (swift-argument-parser); HookSupport (HookContext, the thin socket fetch over DevCtlKit's AgentContext renderer, + HarnessAdapter registry; adding a harness: CONTRIBUTING.md), Switch (branch switching + lifecycle playbooks), Lock (run-under-resource-lock), Link / x-url (deep links). - Sources/DevCtlApp: menu bar app (DaemonModel 2s-polling model + crash notifications with Open/Why actions; AgentService wraps SMAppService.agent for Login Items registration, escalating to unregister + register when the agent reads `enabled` but the socket stays silent (a bootout or replaced bundle leaves the registration intact with nothing loaded) and reporting `requiresApproval` as its own failure instead of retrying; unregister records the stop intent so the recovery poll does not undo it; an unreachable daemon self-recovers via AgentService under DaemonRecoveryPolicy, falling back to legacy LaunchdAdmin only when the bundle carries no agent plist; DaemonDownRow offers Start, or Open Login Items while approval is pending, since a deliberate `daemon stop` stands down auto recovery; PresenceLabel is AppKit-drawn colored tally dots only with renderingMode(.original); popover autogrows to a cap; nested head rows with UserDefaults-persisted pins; DashboardView logs/timeline/config tabs; SpotlightIndexer named Core Spotlight index; AppDeepLink handles `devctl://` including `daemon/ensure` and `daemon/unregister`; SetupPanel first-run / upgrade installer from bundle Resources). Pure DaemonClient consumer. - Sources/fixture-server: test double dev server (heartbeat printer; TCP-listen, timed-exit, grandchild, ignore-sigterm, binary, flood modes; see its header comment). @@ -28,7 +28,7 @@ Commands Hard rules - Output capture is spool-file fds, never pipes: children must survive daemon death without SIGPIPE. Do not introduce pipe-based capture anywhere. -- Teardown signals the process group AND every live descendant found by a sysctl sweep, snapshotted before the first signal (children that setpgid/setsid escape the group; orphans reparent to launchd and fall out of the parent-pid chain). Escalation unions the snapshot with a fresh sweep. Keep both halves when touching stop(). +- Teardown signals the process group AND every live descendant found by a sysctl sweep, snapshotted while the root still parents them (children that setpgid/setsid escape the group; orphans reparent to launchd and fall out of the parent-pid chain). Deliberate stop escalates after grace (snapshot union fresh sweep). Unexpected root exit applies the same two halves with SIGTERM using the run's last descendant snapshot. Keep both halves when touching stop() or the crash exit path. - All JSON goes through JSONCoding: sorted keys, ISO-8601 UTC with milliseconds, no interior newlines. Never construct a raw JSONEncoder or JSONDecoder; the golden tests and NDJSON line framing depend on this determinism. - Wire methods are typed end to end: the daemon sniffs the {id, method} head, then re-decodes the full typed frame. A new method extends WireMethod plus Codable params/result types in Wire.swift; no untyped dictionaries on the wire. - Every CLI command supports --json with a stable schema generated from the shared Codable types; failures emit {ok:false, error:{code,message,hint}} on stdout, hint being the literal remediation command. Error codes grow append-only. Golden tests in Tests/DevCtlKitTests assert exact schema strings; a changed field is an API change: update docs/cli-contract.md in the same commit, then the golden. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7a26a55..4e59e2a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -26,7 +26,7 @@ Local path: `SIGN_IDENTITY="Developer ID Application: …" make dmg` then `scrip The session-context payload is harness-agnostic: `devctl context` prints a fenced plain-text block describing the current project's servers, and `devctl statusline` prints a one-line presence summary from statusline stdin JSON. Wiring those into a harness is the only per-harness work. 1. Conform to `HarnessAdapter` in `Sources/devctl/HookSupport.swift`: a `name` (the `--harness` value) and an idempotent `install(devctlPath:)` that merges a session-start hook into the harness's settings without clobbering what is already there. -2. If the harness wants a structured payload (as Claude Code does with `hookSpecificOutput.additionalContext`, or Cursor with `{additional_context}`), add a hidden subcommand like `HookClaudeSessionStart` / `HookCursorSessionStart` that adapts `AgentContext.render` output to that shape. Keep the guarantees: exit 0 always, fast, silent when there is nothing to say, never auto-starting the daemon, and never emitting raw log lines or command strings (child output and committed configs are attacker-influenceable). Resolve the session directory via `HookSessionCwd` (Cursor: `workspace_roots` / `CURSOR_PROJECT_DIR`; Claude: `cwd`). +2. If the harness wants a structured payload (as Claude Code does with `hookSpecificOutput.additionalContext`, or Cursor with `{additional_context}`), add a hidden subcommand like `HookClaudeSessionStart` / `HookCursorSessionStart` that adapts the `HookContext.render` output (a thin socket fetch over the pure `AgentContext.render` renderer in DevCtlKit) to that shape. Keep the guarantees: exit 0 always, fast, silent when there is nothing to say, never auto-starting the daemon, and never emitting raw log lines or command strings (child output and committed configs are attacker-influenceable). Resolve the session directory via `HookSessionCwd` (Cursor: `workspace_roots` / `CURSOR_PROJECT_DIR`; Claude: `cwd`). 3. Register the adapter in `harnessAdapters` and document the harness in `docs/cli-contract.md` under `devctl hook install`. A harness that only supports plain-text injection needs no adapter code at all: point its hook at `devctl context`. Shipped adapters today: `claude`, `cursor`. diff --git a/Sources/DevCtlApp/DaemonModel.swift b/Sources/DevCtlApp/DaemonModel.swift index 710770f..f6903d3 100644 --- a/Sources/DevCtlApp/DaemonModel.swift +++ b/Sources/DevCtlApp/DaemonModel.swift @@ -381,7 +381,8 @@ final class DaemonModel { else { return } for event in feed.events where event.at > lastEventCheck { lastEventCheck = max(lastEventCheck, event.at) - guard event.kind == .crashed || event.kind == .failed else { continue } + guard CrashNotificationPolicy.shouldNotify(kind: event.kind, detail: event.detail) + else { continue } let content = UNMutableNotificationContent() content.title = "\(event.server) \(event.kind.rawValue)" content.body = event.detail.map { "\($0) · \((event.project as NSString).lastPathComponent)" } diff --git a/Sources/DevCtlDaemonCore/Control/ControlServer.swift b/Sources/DevCtlDaemonCore/Control/ControlServer.swift index 176b50c..8075f6e 100644 --- a/Sources/DevCtlDaemonCore/Control/ControlServer.swift +++ b/Sources/DevCtlDaemonCore/Control/ControlServer.swift @@ -23,7 +23,26 @@ public actor Router { self.paths = paths self.registry = registry self.resourceLocks = - AtomicFile.loadDefensively(LocksFile.self, from: paths.locksFile)?.locks ?? [:] + Self.normalizedLocks( + AtomicFile.loadDefensively(LocksFile.self, from: paths.locksFile)?.locks ?? [:]) + } + + private static func lockKey(project: String, resource: String) -> String { + "\(canonicalProjectPath(project))::\(resource)" + } + + private static func normalizedLocks(_ locks: [String: LockHolder]) -> [String: LockHolder] { + var out: [String: LockHolder] = [:] + for (key, holder) in locks { + guard let separator = key.range(of: "::") else { + out[key] = holder + continue + } + let project = String(key[key.startIndex...self, from: line) - try await registry.register(project: request.params.project, spec: request.params.spec) - let supervisor = await supervisor(project: request.params.project, spec: request.params.spec) + let project = canonicalProjectPath(request.params.project) + try await registry.register(project: project, spec: request.params.spec) + let supervisor = await supervisor(project: project, spec: request.params.spec) await events.post( - kind: .registered, project: request.params.project, server: request.params.spec.name) + kind: .registered, project: project, server: request.params.spec.name) return try respond(id: head.id, result: ServerResult(server: await supervisor.status())) case .serverEnsure: let request = try decoder.decode(WireRequest.self, from: line) - let target = ServerTargetParams(name: request.params.name, project: request.params.project) - let merged = try await mergedSpecs(project: target.project) + let project = canonicalProjectPath(request.params.project) + let target = ServerTargetParams( + name: request.params.name, port: request.params.port, project: project) + let merged = try await mergedSpecs(project: project) let supervisor = try await resolvedSupervisor(target) - await recordTrustIfNeeded(project: target.project, name: target.name, fileNames: merged.fileNames) + await recordTrustIfNeeded(project: project, name: target.name, fileNames: merged.fileNames) if let spec = merged.specs.first(where: { $0.name == target.name }) { - try await lockGate(project: target.project, spec: spec) + try await lockGate(project: project, spec: spec) } - try await portPreCheck(target: target, supervisor: supervisor) + try await prepareSpawn(target: target, supervisor: supervisor, portOverride: request.params.port) let result = await supervisor.ensure(timeoutSeconds: request.params.timeoutSeconds) DevCtlLog.daemon.info( - "ensure \(target.name)@\(target.project) -> \(result.server.phase.rawValue)") + "ensure \(target.name)@\(project) -> \(result.server.phase.rawValue)") return try respond(id: head.id, result: result) case .serverStart: let request = try decoder.decode(WireRequest.self, from: line) - let merged = try await mergedSpecs(project: request.params.project) - let supervisor = try await resolvedSupervisor(request.params) + let project = canonicalProjectPath(request.params.project) + let target = ServerTargetParams( + name: request.params.name, port: request.params.port, project: project) + let merged = try await mergedSpecs(project: project) + let supervisor = try await resolvedSupervisor(target) await recordTrustIfNeeded( - project: request.params.project, name: request.params.name, fileNames: merged.fileNames) - if let spec = merged.specs.first(where: { $0.name == request.params.name }) { - try await lockGate(project: request.params.project, spec: spec) + project: project, name: target.name, fileNames: merged.fileNames) + if let spec = merged.specs.first(where: { $0.name == target.name }) { + try await lockGate(project: project, spec: spec) } - try await portPreCheck(target: request.params, supervisor: supervisor) + try await prepareSpawn(target: target, supervisor: supervisor, portOverride: request.params.port) return try respond(id: head.id, result: ServerResult(server: await supervisor.start())) case .serverStatus: let request = try decoder.decode(WireRequest.self, from: line) - return try respond(id: head.id, result: try await statusList(request.params)) + /** Empty project means machine-wide; do not canonicalize it or it + becomes the daemon cwd and the sweep never runs. */ + let project = request.params.project.isEmpty + ? "" + : canonicalProjectPath(request.params.project) + let params = ProjectParams(name: request.params.name, project: project) + return try respond(id: head.id, result: try await statusList(params)) case .projectTrust: let request = try decoder.decode(WireRequest.self, from: line) - try await registry.setTrusted(project: request.params.project) + try await registry.setTrusted( + project: canonicalProjectPath(request.params.project)) return try respond(id: head.id, result: WireEmpty()) case .projectCheck: let request = try decoder.decode(WireRequest.self, from: line) - let url = ProjectConfigLoader.configURL(project: request.params.project) + let project = canonicalProjectPath(request.params.project) + let url = ProjectConfigLoader.configURL(project: project) guard FileManager.default.fileExists(atPath: url.path) else { return try respond( id: head.id, result: CheckResult(errors: ["no devservers.json at \(url.path)"])) } do { - guard let view = try ProjectConfigLoader.load(project: request.params.project) else { + guard let view = try ProjectConfigLoader.load(project: project) else { return try respond( id: head.id, result: CheckResult(errors: ["cannot read \(url.path)"])) } @@ -152,19 +185,28 @@ public actor Router { host: view.host, servers: view.specs.map(\.name), warnings: view.warnings)) case .groupUp: let request = try decoder.decode(WireRequest.self, from: line) - return try respond(id: head.id, result: try await groupUp(request.params)) + var params = request.params + params.project = canonicalProjectPath(params.project) + return try respond(id: head.id, result: try await groupUp(params)) case .groupDown: let request = try decoder.decode(WireRequest.self, from: line) - return try respond(id: head.id, result: try await groupDown(request.params)) + var params = request.params + params.project = canonicalProjectPath(params.project) + return try respond(id: head.id, result: try await groupDown(params)) case .serverStop: let request = try decoder.decode(WireRequest.self, from: line) - let supervisor = try await resolvedSupervisor(request.params) + let target = ServerTargetParams( + name: request.params.name, port: request.params.port, + project: canonicalProjectPath(request.params.project)) + let supervisor = try await resolvedSupervisor(target) let stopped = await supervisor.stop() - DevCtlLog.daemon.info("stop \(request.params.name)@\(request.params.project)") + DevCtlLog.daemon.info("stop \(target.name)@\(target.project)") return try respond(id: head.id, result: ServerResult(server: stopped)) case .serverWait: let request = try decoder.decode(WireRequest.self, from: line) - let target = ServerTargetParams(name: request.params.name, project: request.params.project) + let target = ServerTargetParams( + name: request.params.name, + project: canonicalProjectPath(request.params.project)) let supervisor = try await resolvedSupervisor(target) let reason = await supervisor.wait( for: request.params.condition, timeoutSeconds: request.params.timeoutSeconds) @@ -172,11 +214,15 @@ public actor Router { id: head.id, result: EnsureResult(reason: reason, server: await supervisor.status())) case .lockAcquire: let request = try decoder.decode(WireRequest.self, from: line) - let result = try await acquireLock(request.params) + var params = request.params + params.project = canonicalProjectPath(params.project) + let result = try await acquireLock(params) return try respond(id: head.id, result: result) case .lockRelease: let request = try decoder.decode(WireRequest.self, from: line) - let result = try await releaseLock(request.params) + var params = request.params + params.project = canonicalProjectPath(params.project) + let result = try await releaseLock(params) return try respond(id: head.id, result: result) case .logsQuery: let request = try decoder.decode(WireRequest.self, from: line) @@ -192,6 +238,12 @@ public actor Router { } since = markDate } + if let pattern = request.params.grep, let why = LogQuery.grepRejection(pattern) { + throw WireError( + code: .usage, + hint: "fix the pattern, or drop --grep to see every line", + message: "--grep is not a valid regular expression: \(why)") + } let options = LogQueryOptions( grep: request.params.grep, since: since, @@ -239,9 +291,10 @@ public actor Router { case .serverWhy: let request = try decoder.decode(WireRequest.self, from: line) _ = try await resolvedSupervisor(request.params) + let merged = try await mergedSpecs(project: request.params.project) var statuses: [String: ServerStatus] = [:] var specsByName: [String: ServerSpec] = [:] - for spec in await registry.specs(project: request.params.project) { + for spec in merged.specs { let supervisor = await supervisor(project: request.params.project, spec: spec) statuses[spec.name] = await supervisor.status() specsByName[spec.name] = spec @@ -252,11 +305,17 @@ public actor Router { target: request.params.name, statuses: statuses, specs: specsByName, - errTail: { server in - LogQuery.run( + evidenceLines: { server in + let since = + statuses[server]?.lastExit?.at + ?? statuses[server]?.uptimeSec.map { + Date().addingTimeInterval(TimeInterval(-$0)) + } + return LogQuery.run( current: paths.structuredLogFile(project: project, server: server), - options: LogQueryOptions(streams: [.err], tail: 5) - ).map(\.text) + options: LogQueryOptions( + since: since, streams: [.err, .out, .sys], tail: 40) + ).map(\.contextLine) }) return try respond(id: head.id, result: result) case .serverUnregister: @@ -289,15 +348,29 @@ public actor Router { } } - /** Startup recovery: reconcile persisted locks first, then restore servers - with boot intent. A recorded pid that is gone becomes crashed(daemon-restart); - a live orphan (its spool fd kept it healthy while the daemon was away) is - group-killed, since exit forensics are unknowable for non-children. Never - adopted silently. What comes back: any server whose start intent survives - (resumeOnBoot), which a machine shutdown's drain leaves set, plus the - classic daemon-crash case of a phase left running/starting. A deliberate - stop clears the flag, so only those stay down. Servers still paused under - a live resource lock are left alone: starting them would fight the harness. + /** Forget registered projects whose checkout path is gone: stop children, + bounce orphan pids, drop registry/state/locks/supervisors. Opportunistic + (boot + machine-wide status), not a watcher. */ + /** Forget registered projects whose checkout path is gone: stop children, + bounce orphan pids, drop registry/state/locks/supervisors. Opportunistic + (boot + machine-wide status), not a watcher. */ + public func pruneMissingProjects() async { + for project in await registry.allProjects() { + if FileManager.default.fileExists(atPath: project) { continue } + await forgetMissingProject(project) + } + } + + /** Startup recovery: prune vanished checkouts first, reconcile persisted + locks, then restore servers with boot intent. A recorded pid that is gone + becomes crashed(daemon-restart); a live orphan (its spool fd kept it + healthy while the daemon was away) is group-killed, since exit forensics + are unknowable for non-children. Never adopted silently. What comes back: + any server whose start intent survives (resumeOnBoot), which a machine + shutdown's drain leaves set, plus the classic daemon-crash case of a + phase left running/starting. A deliberate stop clears the flag, so only + those stay down. Servers still paused under a live resource lock are left + alone: starting them would fight the harness. Specs resolve through the merged view (devservers.json + ad-hoc registry), the same path ensure/status use. Config-defined servers are never written @@ -305,6 +378,7 @@ public actor Router { committed server on boot. A rename/delete with no matching spec drops the orphaned state row instead of retrying forever. */ public func recoverAtStartup() async { + await pruneMissingProjects() await reconcileLocksAtStartup() var toStart: [(project: String, spec: ServerSpec)] = [] for (id, persisted) in await registry.allPersistedState() { @@ -330,27 +404,10 @@ public actor Router { "recover defer \(name)@\(project): config unreadable; keeping resume intent") continue case .found(let spec): - if let pid = persisted.pid.map(pid_t.init), kill(pid, 0) == 0 { - let sweep = ProcessTree.descendants(of: pid) - if case .failed(let code) = sweep { - DevCtlLog.daemon.error( - "orphan bounce descendant sweep failed (errno \(code)); group-only") - } - let descendants = sweep.identities - ProcessTree.signalTree(rootPid: pid, descendants: descendants, signal: SIGTERM) - /** Poll instead of a fixed 2s sleep: most orphans die on - SIGTERM in well under a second. */ - let graceDeadline = ContinuousClock.now.advanced(by: .seconds(2)) - while ContinuousClock.now < graceDeadline, kill(pid, 0) == 0 { - try? await Task.sleep(for: .milliseconds(50)) - } - if kill(pid, 0) == 0 { - ProcessTree.signalTree( - rootPid: pid, descendants: descendants, signal: SIGKILL, revalidate: true) - } - await events.post( - kind: .crashed, project: project, server: name, - detail: "daemon-restart: orphan pid \(pid) bounced") + if let pid = persisted.pid.map(pid_t.init), + let identity = ProcessTree.identity(of: pid) + { + await bounceOrphan(identity, project: project, name: name) } else if leftActive { await events.post( kind: .crashed, project: project, server: name, detail: "daemon-restart") @@ -365,19 +422,23 @@ public actor Router { } } } - /** Start restores in parallel: a serial loop made a multi-server boot - look like a slow daemon even after the socket was already up. */ - if !toStart.isEmpty { - await withTaskGroup(of: Void.self) { group in - for item in toStart { - group.addTask { - DevCtlLog.daemon.info("recover start \(item.spec.name)@\(item.project)") - let supervisor = await self.supervisor( - project: item.project, spec: item.spec) - _ = await supervisor.start() - } - } + for item in toStart { + let supervisor = await self.supervisor(project: item.project, spec: item.spec) + do { + try await self.prepareSpawn( + target: ServerTargetParams(name: item.spec.name, project: item.project), + supervisor: supervisor) + } catch let error as WireError { + DevCtlLog.daemon.error( + "recover skip \(item.spec.name)@\(item.project): \(error.message)") + continue + } catch { + DevCtlLog.daemon.error( + "recover skip \(item.spec.name)@\(item.project): \(error.localizedDescription)") + continue } + DevCtlLog.daemon.info("recover start \(item.spec.name)@\(item.project)") + _ = await supervisor.start() } /** Second pass: drop leftover rows for renamed/deleted servers even when they carry no resume intent (e.g. a deliberate stop under the old @@ -394,6 +455,119 @@ public actor Router { } } + /** Group-kill a live non-child left over from a prior daemon (or a prune that + could not stop through the supervisor). Signals only while the snapshotted + start time still matches: a recycled pid must not be killed. */ + private func bounceOrphan( + _ root: ProcessIdentity, project: String, name: String + ) async { + guard ProcessTree.shouldSignal( + snapshotted: root, live: ProcessTree.identity(of: root.pid)) + else { + DevCtlLog.daemon.info( + "orphan bounce skip \(name)@\(project): pid \(root.pid) gone or reused") + return + } + let pid = root.pid + let sweep = ProcessTree.descendants(of: pid) + if case .failed(let code) = sweep { + DevCtlLog.daemon.error( + "orphan bounce descendant sweep failed (errno \(code)); group-only") + } + let descendants = sweep.identities + ProcessTree.signalTree( + descendants: descendants, revalidate: true, rootIdentity: root, rootPid: pid, + signal: SIGTERM) + /** Poll with identity, not kill(pid,0): a recycled number must end the + wait as "gone" rather than escalate into the new process. */ + let graceDeadline = ContinuousClock.now.advanced(by: .seconds(2)) + while ContinuousClock.now < graceDeadline, + ProcessTree.shouldSignal( + snapshotted: root, live: ProcessTree.identity(of: pid)) + { + try? await Task.sleep(for: .milliseconds(50)) + } + if ProcessTree.shouldSignal( + snapshotted: root, live: ProcessTree.identity(of: pid)) + { + ProcessTree.signalTree( + descendants: descendants, revalidate: true, rootIdentity: root, rootPid: pid, + signal: SIGKILL) + } + await events.post( + kind: .crashed, project: project, server: name, + detail: "daemon-restart: orphan pid \(pid) bounced") + } + + /** Stop and forget one vanished checkout. Config is unreadable once the path + is gone, so this walks supervisors + registry + state directly instead of + groupDown / mergedSpecs. `project` must be the registry key as stored + (already canonical at registration): re-canonicalizing a deleted path can + change `/private/var` ↔ `/var` spelling and miss every lookup. */ + private func forgetMissingProject(_ project: String) async { + let prefix = "\(project)::" + /** Snapshot identities before teardown: a composite tree can outlive a + no-op stop (phase already crashed/failed after the checkout vanished), + so we keep the recorded ProcessIdentity and bounce only while that + start time still matches. */ + var liveRoots: [(identity: ProcessIdentity, name: String)] = [] + var names = Set(await registry.specs(project: project).map(\.name)) + for (id, supervisor) in supervisors where id.hasPrefix(prefix) { + guard let separator = id.range(of: "::") else { continue } + let name = String(id[separator.upperBound...]) + names.insert(name) + let status = await supervisor.status() + if let pid = status.pid.map(pid_t.init), + let identity = ProcessTree.identity(of: pid) + { + liveRoots.append((identity: identity, name: name)) + } + } + for (id, persisted) in await registry.allPersistedState() where id.hasPrefix(prefix) { + guard let separator = id.range(of: "::") else { continue } + let name = String(id[separator.upperBound...]) + names.insert(name) + if let pid = persisted.pid.map(pid_t.init), + let identity = ProcessTree.identity(of: pid), + !liveRoots.contains(where: { $0.identity.pid == pid }) + { + liveRoots.append((identity: identity, name: name)) + } + } + let sortedNames = names.sorted() + DevCtlLog.daemon.info( + "prune missing project \(project) (\(sortedNames.joined(separator: ",")))") + for name in sortedNames { + let id = serverID(project: project, name: name) + if let supervisor = supervisors[id] { + _ = await supervisor.stop() + await events.post( + kind: .stopped, project: project, server: name, detail: "project path gone") + } + supervisors[id] = nil + } + for entry in liveRoots { + guard ProcessTree.shouldSignal( + snapshotted: entry.identity, live: ProcessTree.identity(of: entry.identity.pid)) + else { continue } + await bounceOrphan(entry.identity, project: project, name: entry.name) + } + configCache[project] = nil + let lockKeys = resourceLocks.keys.filter { $0.hasPrefix(prefix) } + if !lockKeys.isEmpty { + for key in lockKeys { + resourceLocks[key] = nil + } + persistLocks() + } + try? await registry.removeState(forProject: project) + try? await registry.removeProject(project) + for name in sortedNames { + await events.post( + kind: .unregistered, project: project, server: name, detail: "project path gone") + } + } + private enum RecoverSpec { case found(ServerSpec) /** Config loaded cleanly and the name is absent: rename/delete. */ @@ -436,12 +610,12 @@ public actor Router { exit(0) } - /** Fails a start-shaped request when the declared port is already held: by a - managed server elsewhere (the two-worktrees case, named precisely) or by - an unmanaged squatter (pid + command when lsof can see it). Runs only when - the target itself is not already up, so a server never conflicts with its - own listener. */ - private func portPreCheck(target: ServerTargetParams, supervisor: ServerSupervisor) async throws { + /** Resolve effective port, apply overlay/worktree host/materialization, and + either auto-rebind a sibling conflict or refuse with port-held. Every + start-shaped path routes through here. */ + private func prepareSpawn( + target: ServerTargetParams, supervisor: ServerSupervisor, portOverride: Int? = nil + ) async throws { let current = await supervisor.status() switch current.phase { case .running, .starting, .stopping, .unhealthy: @@ -449,45 +623,236 @@ public actor Router { case .crashed, .failed, .stopped: break } - guard let port = current.declaredPort else { return } - let targetID = serverID(project: target.project, name: target.name) - for (id, other) in supervisors where id != targetID { - let status = await other.status() - let active = status.phase == .running || status.phase == .starting || status.phase == .unhealthy - if active, status.declaredPort == port || status.observedPort == port { - DevCtlLog.daemon.error( - "port-held \(port) by \(status.server)@\(status.project) for \(target.name)") - throw WireError( - code: .portHeld, - hint: "run: devctl stop \(status.server) --project \(status.project)", - message: "port \(port) is held by managed server '\(status.server)' in \(status.project)" - ) + let merged = try await mergedSpecs(project: target.project) + guard var spec = merged.specs.first(where: { $0.name == target.name }) else { + throw WireError( + code: .notFound, + hint: "run: devctl status --json", + message: "no server named '\(target.name)' in \(target.project)") + } + let overlay = LocalOverlay.load(project: target.project) + let overlayServer = overlay?.servers?[target.name] + spec = LocalOverlay.apply(spec: spec, overlay: overlayServer, project: target.project) + let committedHost = merged.host + let preferred = CheckoutIdentity.preferredSubdomain( + project: target.project, committedHost: committedHost) + let taken = await takenHosts() + let defaultSlugHost = + "\(ProjectConfigLoader.defaultSlug(project: target.project)).localhost" + /** Host printed in committed urls/heads before any worktree swap; materialize + matches against this so preferred-host URLs rewrite to the ephemeral + label even after `spec.host` already moved. */ + let matchHost = spec.host ?? committedHost ?? defaultSlugHost + if let worktreeHost = CheckoutIdentity.worktreeHost( + project: target.project, preferred: preferred, takenHosts: taken), + overlayServer?.host == nil, + spec.host == nil || spec.host == committedHost || spec.host == defaultSlugHost + { + spec.host = worktreeHost + if let port = spec.port { + let urlHost = URL(string: spec.url ?? "")?.host + if spec.url == nil || urlHost == committedHost || urlHost == defaultSlugHost { + spec.url = "http://\(worktreeHost):\(port)/" + } } } - if PortGuard.isListening(port: port) { - if let squatter = PortGuard.listenerInfo(port: port) { + let declaredPort = spec.port + let id = serverID(project: target.project, name: target.name) + let persistedBound = await registry.persistedState(serverID: id)?.boundPort + var effective = + portOverride ?? overlayServer?.port ?? persistedBound ?? declaredPort + var conflict: PortConflict? + if let port = effective { + let targetID = id + let draft = PortClaim.resolve(spec: spec, effectivePort: port) + guard let draftClaim = draft.claim, draft.error == nil else { throw WireError( - code: .portHeld, - hint: "run: kill \(squatter.pid) (verify first: ps -p \(squatter.pid))", - message: "port \(port) is held by unmanaged pid \(squatter.pid) (\(squatter.command))" - ) + code: .configInvalid, hint: "run: devctl config check", + message: draft.error ?? "invalid port claim") + } + if let busy = await firstBusyPort(in: draftClaim, excluding: targetID) { + let holder = await managedHolder(port: busy.port, excluding: targetID) + if let holder, CheckoutIdentity.shareCommonDir(target.project, holder.project), + draftClaim.relative.contains(busy.port) + { + let rebound = await allocateSiblingPort( + declared: declaredPort ?? port, excluding: targetID, project: target.project, + spec: spec) + conflict = PortConflict( + declaredPort: declaredPort ?? port, + effectivePort: rebound, + holder: "\(holder.server)@\(holder.project)", + message: + "port \(busy.port) held by sibling '\(holder.server)' in \(holder.project); rebound to \(rebound)", + state: .rebound) + effective = rebound + try? await registry.updateState(serverID: id) { $0.boundPort = rebound } + } else if let holder { + DevCtlLog.daemon.error( + "port-held \(busy.port) by \(holder.server)@\(holder.project) for \(target.name)") + throw WireError( + code: .portHeld, + hint: "run: devctl stop \(holder.server) --project \(holder.project)", + message: + "port \(busy.port) is held by managed server '\(holder.server)' in \(holder.project)" + ) + } else if let squatter = PortGuard.listenerInfo(port: busy.port) { + throw WireError( + code: .portHeld, + hint: "run: kill \(squatter.pid) (verify first: ps -p \(squatter.pid))", + message: + "port \(busy.port) is held by unmanaged pid \(squatter.pid) (\(squatter.command))" + ) + } else { + throw WireError( + code: .portHeld, + message: "port \(busy.port) already has a listener that devctl does not manage" + ) + } } + } + let resolved = PortClaim.resolve(spec: spec, effectivePort: effective) + guard let claim = resolved.claim, resolved.error == nil else { + throw WireError( + code: .configInvalid, hint: "run: devctl config check", + message: resolved.error ?? "invalid port claim") + } + if let busy = await firstBusyPort(in: claim, excluding: id) { throw WireError( code: .portHeld, - message: "port \(port) already has a listener that devctl does not manage" - ) + message: "port \(busy.port) is still busy after rebind resolution") + } + let materialized = PortMaterializer.materialize( + spec: spec, effectivePort: effective, effectiveHost: spec.host, matchHost: matchHost) + await supervisor.updateSpec(materialized) + await supervisor.setPortMeta( + claim: claim, declaredPort: declaredPort, effectivePort: effective, + portConflict: conflict) + } + + /** First claimed port that is held (managed or unmanaged). Absolutes and + relatives are treated the same for freeness; only sibling rebind cares + which set a conflict came from. */ + private func firstBusyPort(in claim: PortClaim, excluding targetID: String) async -> ( + port: Int, managed: Bool + )? { + for port in claim.allPorts { + if await managedHolder(port: port, excluding: targetID) != nil { + return (port: port, managed: true) + } + if PortGuard.isListening(port: port) { + return (port: port, managed: false) + } + } + return nil + } + + private func allocateSiblingPort( + declared: Int, excluding: String, project: String, spec: ServerSpec + ) async -> Int { + var candidate = CheckoutIdentity.siblingPortCandidate(declared: declared, project: project) + for _ in 0..<200 { + let resolved = PortClaim.resolve(spec: spec, effectivePort: candidate) + if let claim = resolved.claim, resolved.error == nil, + await firstBusyPort(in: claim, excluding: excluding) == nil + { + return candidate + } + candidate += 1 + if candidate > 65_000 { candidate = 10_000 } + } + return candidate + } + + /** Wait until every claimed port is free, or return the first still-busy port. */ + private func waitForClaimFree(claim: PortClaim, budgetSeconds: Double = 2) async -> Int? { + let deadline = ContinuousClock.now.advanced(by: .seconds(budgetSeconds)) + while true { + var busy: Int? + for port in claim.allPorts where PortGuard.isListening(port: port) { + busy = port + break + } + if busy == nil { return nil } + if ContinuousClock.now >= deadline { return busy } + try? await Task.sleep(for: .milliseconds(50)) + } + } + + private func takenHosts() async -> Set { + var hosts: Set = [] + for project in await registry.allProjects() { + if let view = try? loadConfig(project: project) { + hosts.insert(view.host) + for spec in view.specs { + if let host = spec.host { hosts.insert(host) } + if let url = spec.url, let host = URL(string: url)?.host { + hosts.insert(host) + } + } + } } + return hosts + } + + /** Which managed server holds `port`, if any. The resident supervisor pool + answers for servers this daemon has been asked about; state.json answers + for the rest, since the pool is built lazily and a server started before + this daemon's first request for it has no entry there at all. A persisted + row counts only while its recorded pid is still alive. */ + private func managedHolder(port: Int, excluding targetID: String) async -> ( + project: String, server: String + )? { + for (id, other) in supervisors where id != targetID { + let status = await other.status() + let active = + status.phase == .running || status.phase == .starting || status.phase == .unhealthy + if active, + status.effectivePort == port || status.declaredPort == port + || status.observedPort == port + || (status.ports?.values.contains(port) ?? false) + { + return (project: status.project, server: status.server) + } + } + for (id, persisted) in await registry.allPersistedState() where id != targetID { + guard supervisors[id] == nil else { continue } + let active = + persisted.phase == .running || persisted.phase == .starting + || persisted.phase == .unhealthy + guard active, let pid = persisted.pid.map(pid_t.init), kill(pid, 0) == 0 else { + continue + } + guard let separator = id.range(of: "::") else { continue } + let project = String(id[id.startIndex.. (specs: [ServerSpec], fileNames: Set) { + private func mergedSpecs(project: String) async throws -> ( + host: String?, specs: [ServerSpec], fileNames: Set + ) { + let project = canonicalProjectPath(project) var specs: [String: ServerSpec] = [:] for spec in await registry.specs(project: project) { specs[spec.name] = spec } var fileNames: Set = [] + var host: String? if let view = try loadConfig(project: project) { guard view.errors.isEmpty else { throw WireError( @@ -495,12 +860,15 @@ public actor Router { hint: "run: devctl config check", message: "devservers.json is invalid: \(view.errors.joined(separator: "; "))") } + host = view.host for spec in view.specs { specs[spec.name] = spec fileNames.insert(spec.name) } } - return (specs: specs.values.sorted { $0.name < $1.name }, fileNames: fileNames) + return ( + host: host, specs: specs.values.sorted { $0.name < $1.name }, fileNames: fileNames + ) } private func loadConfig(project: String) throws -> ProjectConfigView? { @@ -530,7 +898,7 @@ public actor Router { /** Acquire: refuse if another live holder owns it, pause active declarers without retiring boot intent, persist the hold, return who was paused. */ private func acquireLock(_ params: LockParams) async throws -> LockResult { - let key = "\(params.project)::\(params.resource)" + let key = Self.lockKey(project: params.project, resource: params.resource) await releaseOrphanedLock(key: key) if let holder = resourceLocks[key], holder.pid != params.holderPid { throw WireError( @@ -546,23 +914,26 @@ public actor Router { return LockResult(paused: existing.paused) } var paused: [String] = [] + let shouldPause = params.pause ?? true let merged = try? await mergedSpecs(project: params.project) - for spec in merged?.specs ?? [] where (spec.locks ?? []).contains(params.resource) { - let supervisor = await supervisor(project: params.project, spec: spec) - let status = await supervisor.status() - switch status.phase { - case .running, .starting, .unhealthy, .stopping: - /** Non-retiring stop: boot intent survives so a daemon crash - mid-hold can still bring the server back if the holder is gone. */ - _ = await supervisor.stop(deliberate: false) - paused.append(spec.name) - DevCtlLog.daemon.info( - "lock \(params.resource) paused \(spec.name)@\(params.project)") - case .stopped, .crashed, .failed: - break + if shouldPause { + for spec in merged?.specs ?? [] where (spec.locks ?? []).contains(params.resource) { + let supervisor = await supervisor(project: params.project, spec: spec) + let status = await supervisor.status() + switch status.phase { + case .running, .starting, .unhealthy, .stopping: + /** Non-retiring stop: boot intent survives so a daemon crash + mid-hold can still bring the server back if the holder is gone. */ + _ = await supervisor.stop(deliberate: false) + paused.append(spec.name) + DevCtlLog.daemon.info( + "lock \(params.resource) paused \(spec.name)@\(params.project)") + case .stopped, .crashed, .failed: + break + } } + paused.sort() } - paused.sort() resourceLocks[key] = LockHolder( paused: paused, pid: params.holderPid, resumeTimeoutSeconds: params.resumeTimeoutSeconds, since: Date()) @@ -573,7 +944,7 @@ public actor Router { /** Release: only the matching holder clears the lock; then ensure everyone that was paused. */ private func releaseLock(_ params: LockParams) async throws -> LockResult { - let key = "\(params.project)::\(params.resource)" + let key = Self.lockKey(project: params.project, resource: params.resource) guard let holder = resourceLocks[key], holder.pid == params.holderPid else { return LockResult(paused: []) } @@ -618,8 +989,21 @@ public actor Router { let merged = try await mergedSpecs(project: project) guard let spec = merged.specs.first(where: { $0.name == name }) else { continue } let supervisor = await supervisor(project: project, spec: spec) + let id = serverID(project: project, name: name) + let bound = await registry.persistedState(serverID: id)?.boundPort ?? spec.port + let resolved = PortClaim.resolve(spec: spec, effectivePort: bound) + if let claim = resolved.claim, let busy = await waitForClaimFree(claim: claim) { + DevCtlLog.daemon.error( + "lock \(resource) resume refused \(name)@\(project): port \(busy) still busy") + continue + } + try await prepareSpawn( + target: ServerTargetParams(name: name, project: project), supervisor: supervisor) _ = await supervisor.ensure(timeoutSeconds: timeoutSeconds) DevCtlLog.daemon.info("lock \(resource) resumed \(name)@\(project)") + } catch let error as WireError { + DevCtlLog.daemon.error( + "lock \(resource) could not resume \(name)@\(project): \(error.message)") } catch { DevCtlLog.daemon.error( "lock \(resource) could not resume \(name)@\(project): \(error.localizedDescription)" @@ -642,8 +1026,9 @@ public actor Router { } private func isPausedUnderLiveLock(project: String, name: String) -> Bool { + let prefix = "\(canonicalProjectPath(project))::" for (key, holder) in resourceLocks { - guard key.hasPrefix("\(project)::") else { continue } + guard key.hasPrefix(prefix) else { continue } guard kill(pid_t(holder.pid), 0) == 0 else { continue } if holder.paused.contains(name) { return true } } @@ -661,7 +1046,7 @@ public actor Router { the lock exists to prevent. */ private func lockGate(project: String, spec: ServerSpec) async throws { for resource in spec.locks ?? [] { - let key = "\(project)::\(resource)" + let key = Self.lockKey(project: project, resource: resource) await releaseOrphanedLock(key: key) if let holder = resourceLocks[key] { throw WireError( @@ -690,6 +1075,7 @@ public actor Router { /** An empty project means machine-wide (daemon restart, doctor, the app); machine-wide reads skip config errors rather than failing the sweep. */ if params.project.isEmpty { + await pruneMissingProjects() var statuses: [ServerStatus] = [] for project in await registry.allProjects() { var specs = (try? await mergedSpecs(project: project))?.specs @@ -710,12 +1096,48 @@ public actor Router { for spec in merged.specs { if let name = params.name, name != spec.name { continue } let supervisor = await supervisor(project: params.project, spec: spec) - statuses.append(await supervisor.status()) + var status = await supervisor.status() + status = await annotateLatentPortConflict(status, excluding: serverID(project: params.project, name: spec.name)) + statuses.append(status) } return ServerListResult( servers: statuses, trusted: await registry.isTrusted(project: params.project)) } + /** When a server is not up but its declared port is held, surface a latent + conflict so session context warns before the agent runs ensure. */ + private func annotateLatentPortConflict(_ status: ServerStatus, excluding: String) async -> ServerStatus { + guard status.portConflict == nil else { return status } + switch status.phase { + case .stopped, .crashed, .failed: + break + case .running, .starting, .stopping, .unhealthy: + return status + } + guard let port = status.declaredPort ?? status.effectivePort else { return status } + var annotated = status + if let holder = await managedHolder(port: port, excluding: excluding) { + let sibling = CheckoutIdentity.shareCommonDir(status.project, holder.project) + annotated.portConflict = PortConflict( + declaredPort: port, + holder: "\(holder.server)@\(holder.project)", + message: sibling + ? "port \(port) held by sibling '\(holder.server)' in \(holder.project); ensure will auto-rebind" + : "port \(port) held by '\(holder.server)' in \(holder.project); run: devctl stop \(holder.server) --project \(holder.project)", + state: .held) + } else if PortGuard.isListening(port: port) { + let detail = PortGuard.listenerInfo(port: port).map { + "unmanaged pid \($0.pid) (\($0.command))" + } ?? "an unmanaged listener" + annotated.portConflict = PortConflict( + declaredPort: port, + holder: detail, + message: "port \(port) held by \(detail)", + state: .held) + } + return annotated + } + /** Wave-parallel group start honoring the dependency graph: a wave holds servers whose dependencies all settled in earlier waves. waitFor .started launches without blocking on health; the default blocks until healthy. */ @@ -741,6 +1163,18 @@ public actor Router { await recordTrustIfNeeded( project: params.project, name: spec.name, fileNames: merged.fileNames) } + /** Port ownership is checked for the whole set before anything spawns, so + a held port refuses the rollout instead of leaving half a project up + next to a server that lost a race it never knew it entered. Servers + already up skip the check against their own listeners. */ + for spec in wanted { + let target = ServerTargetParams( + name: spec.name, port: params.port, project: params.project) + try await prepareSpawn( + target: target, + supervisor: await supervisor(project: params.project, spec: spec), + portOverride: params.port) + } guard case .success(let waves) = DependencyGraph.waves(specs: wanted) else { throw WireError( code: .configInvalid, @@ -817,9 +1251,19 @@ public actor Router { } private func supervisor(project: String, spec: ServerSpec) async -> ServerSupervisor { + let project = canonicalProjectPath(project) let id = serverID(project: project, name: spec.name) if let existing = supervisors[id] { - await existing.updateSpec(spec) + /** A live run holds a materialized spawn spec (effective port, worktree + host, rewritten url). Re-resolving committed config for status must + not clobber that, or agents see the declared origin and a false + "config changed since start". */ + switch await existing.status().phase { + case .running, .starting, .unhealthy, .stopping: + break + case .stopped, .crashed, .failed: + await existing.updateSpec(spec) + } return existing } let created = ServerSupervisor( diff --git a/Sources/DevCtlDaemonCore/Control/Why.swift b/Sources/DevCtlDaemonCore/Control/Why.swift index 1d5f974..40ad8db 100644 --- a/Sources/DevCtlDaemonCore/Control/Why.swift +++ b/Sources/DevCtlDaemonCore/Control/Why.swift @@ -2,7 +2,7 @@ import DevCtlKit import Foundation /** The rule-based diagnosis behind `devctl why`: turn the data the daemon - already holds (phase, forensics, health, err-stream tails, the dependency + already holds (phase, forensics, health, spool evidence, the dependency graph) into a finding chain with a root-cause statement. */ enum WhyEngine { /** Diagnoses `target` and walks dependsOn to find the deepest broken @@ -11,7 +11,9 @@ enum WhyEngine { target: String, statuses: [String: ServerStatus], specs: [String: ServerSpec], - errTail: (String) -> [String] + /** Fallback evidence when status has no recentLogTail / terminalEvidence + (structured-log window since last exit or start). */ + evidenceLines: (String) -> [String] ) -> WhyResult { var findings: [WhyFinding] = [] var visited = Set() @@ -19,7 +21,7 @@ enum WhyEngine { var frontier = [target] while let name = frontier.popLast() { guard visited.insert(name).inserted, let status = statuses[name] else { continue } - let finding = describe(status: status, errTail: errTail) + let finding = describe(status: status, evidenceLines: evidenceLines) findings.append(finding) let broken = status.phase != .running && status.phase != .stopped if broken || status.phase == .stopped { @@ -38,7 +40,9 @@ enum WhyEngine { return WhyResult(findings: findings, rootCause: rootCause) } - private static func describe(status: ServerStatus, errTail: (String) -> [String]) -> WhyFinding { + private static func describe( + status: ServerStatus, evidenceLines: (String) -> [String] + ) -> WhyFinding { var evidence: [String] = [] var summary: String switch status.phase { @@ -50,11 +54,13 @@ enum WhyEngine { ?? status.lastExit?.signal.map { "signal \($0)" } ?? "unknown cause" let when = status.lastExit.map { JSONCoding.formatISO8601($0.at) } ?? "unknown time" summary = "crashed (\(cause)) at \(when)" - evidence.append(contentsOf: errTail(status.server).map { "err: \($0)" }) + if status.lastExit?.code == 0 { + summary += + "; exit 0 is often a controlled refusal (framework lock, bind failure), not a crash" + } case .unhealthy: let since = status.lastHealthAt.map { JSONCoding.formatISO8601($0) } ?? "startup" summary = "healthcheck failing; last healthy \(since)" - evidence.append(contentsOf: errTail(status.server).map { "err: \($0)" }) case .starting: summary = "still starting; healthcheck (\(status.healthcheck.rawValue)) has not passed yet" case .stopped: @@ -64,10 +70,38 @@ enum WhyEngine { case .running: summary = "running and healthy" } - if let declared = status.declaredPort, let observed = status.observedPort, declared != observed { + switch status.phase { + case .crashed, .failed, .unhealthy, .starting: + if let tail = status.recentLogTail, !tail.isEmpty { + evidence.append(contentsOf: tail) + } else if let persisted = status.terminalEvidence, !persisted.isEmpty { + evidence.append(contentsOf: persisted) + } else { + evidence.append(contentsOf: evidenceLines(status.server)) + } + case .running, .stopped, .stopping: + break + } + if let effective = status.effectivePort, let observed = status.observedPort, + effective != observed + { + summary += "; NOTE: listening on \(observed), not the effective \(effective)" + evidence.append("effective port \(effective), observed \(observed)") + } else if let declared = status.declaredPort, let observed = status.observedPort, + declared != observed + { summary += "; NOTE: listening on \(observed), not the declared \(declared)" evidence.append("declared port \(declared), observed \(observed)") } + if let conflict = status.portConflict { + evidence.append("port conflict (\(conflict.state.rawValue)): \(conflict.message)") + } + if let ports = status.ports, !ports.isEmpty { + let rendered = ports.sorted { $0.key < $1.key } + .map { "\($0.key)=\($0.value)" } + .joined(separator: ", ") + evidence.append("named ports: \(rendered)") + } if status.specStale == true { evidence.append("config changed since this process started (restart to apply)") } diff --git a/Sources/DevCtlDaemonCore/Health/HealthProber.swift b/Sources/DevCtlDaemonCore/Health/HealthProber.swift index b5d3573..e668514 100644 --- a/Sources/DevCtlDaemonCore/Health/HealthProber.swift +++ b/Sources/DevCtlDaemonCore/Health/HealthProber.swift @@ -127,64 +127,17 @@ public struct NetworkHealthProber: HealthProber { } } - /** Non-blocking connect to a loopback port with a poll deadline. Dev - servers bind either stack: Vite 5+ listens on `::1` only, older - tooling on `127.0.0.1` only, so both loopback families are tried and - either answering counts as healthy. */ static func tcpConnects(port: Int, timeoutMs: Int) -> Bool { - tcpConnects(port: port, timeoutMs: timeoutMs, family: AF_INET) - || tcpConnects(port: port, timeoutMs: timeoutMs, family: AF_INET6) - } - - private static func tcpConnects(port: Int, timeoutMs: Int, family: Int32) -> Bool { - let sock = socket(family, SOCK_STREAM, 0) - guard sock >= 0 else { return false } - defer { close(sock) } - let flags = fcntl(sock, F_GETFL) - _ = fcntl(sock, F_SETFL, flags | O_NONBLOCK) - var storage = sockaddr_storage() - let sockLen: socklen_t - if family == AF_INET6 { - var addr = sockaddr_in6() - addr.sin6_family = sa_family_t(AF_INET6) - addr.sin6_port = UInt16(port).bigEndian - addr.sin6_addr = in6addr_loopback - sockLen = socklen_t(MemoryLayout.size) - withUnsafeMutablePointer(to: &storage) { ptr in - ptr.withMemoryRebound(to: sockaddr_in6.self, capacity: 1) { $0.pointee = addr } - } - } else { - var addr = sockaddr_in() - addr.sin_family = sa_family_t(AF_INET) - addr.sin_port = UInt16(port).bigEndian - addr.sin_addr = in_addr(s_addr: inet_addr("127.0.0.1")) - sockLen = socklen_t(MemoryLayout.size) - withUnsafeMutablePointer(to: &storage) { ptr in - ptr.withMemoryRebound(to: sockaddr_in.self, capacity: 1) { $0.pointee = addr } - } - } - let result = withUnsafePointer(to: &storage) { ptr in - ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in - connect(sock, sa, sockLen) - } - } - if result == 0 { return true } - guard errno == EINPROGRESS else { return false } - var pollTarget = pollfd(fd: sock, events: Int16(POLLOUT), revents: 0) - guard poll(&pollTarget, 1, Int32(timeoutMs)) == 1 else { return false } - var soError: Int32 = 0 - var soLen = socklen_t(MemoryLayout.size) - getsockopt(sock, SOL_SOCKET, SO_ERROR, &soError, &soLen) - return soError == 0 + LoopbackProbe.isListening(port: port, timeoutMs: timeoutMs) } } /** Port ownership diagnostics: best-effort lsof shell-outs. These inform errors and status; supervision never depends on them. */ public enum PortGuard { - /** True when something accepts on 127.0.0.1:port right now. */ + /** True when something accepts on either loopback family for that port. */ public static func isListening(port: Int) -> Bool { - NetworkHealthProber.tcpConnects(port: port, timeoutMs: 300) + LoopbackProbe.isListening(port: port) } /** The pid + command listening on a port, when lsof can name it. */ diff --git a/Sources/DevCtlDaemonCore/Registry/Registry.swift b/Sources/DevCtlDaemonCore/Registry/Registry.swift index 0decc0b..3120f54 100644 --- a/Sources/DevCtlDaemonCore/Registry/Registry.swift +++ b/Sources/DevCtlDaemonCore/Registry/Registry.swift @@ -32,6 +32,12 @@ public struct StateFile: Codable, Sendable { } public struct PersistedServerState: Codable, Sendable { + /** Sibling-rebind assignment; optional so older state files keep parsing. */ + public var boundPort: Int? + /** Error-stream tally from the last run, so a daemon restart does not erase + the forensics an agent needs to understand why a server is down. + Optional so state files written before this field existed keep parsing. */ + public var errorSummary: ErrorSummary? public var lastExit: LastExit? public var phase: ServerPhase public var pid: Int? @@ -43,21 +49,29 @@ public struct PersistedServerState: Codable, Sendable { public var resumeOnBoot: Bool? public var spawnError: SpawnError? public var startedAt: Date? + /** Last terminal spool lines for `why` after ensure truncate / rehydrate. */ + public var terminalEvidence: [String]? public init( + boundPort: Int? = nil, + errorSummary: ErrorSummary? = nil, lastExit: LastExit? = nil, phase: ServerPhase = .stopped, pid: Int? = nil, resumeOnBoot: Bool? = nil, spawnError: SpawnError? = nil, - startedAt: Date? = nil + startedAt: Date? = nil, + terminalEvidence: [String]? = nil ) { + self.boundPort = boundPort + self.errorSummary = errorSummary self.lastExit = lastExit self.phase = phase self.pid = pid self.resumeOnBoot = resumeOnBoot self.spawnError = spawnError self.startedAt = startedAt + self.terminalEvidence = terminalEvidence } } @@ -79,10 +93,11 @@ public actor Registry { } public func project(_ path: String) -> RegisteredProject? { - registry.projects[path] + registry.projects[Self.normalize(path)] } public func register(project: String, spec: ServerSpec) throws { + let project = Self.normalize(project) var entry = registry.projects[project] ?? RegisteredProject() entry.servers[spec.name] = spec registry.projects[project] = entry @@ -90,18 +105,21 @@ public actor Registry { } public func spec(project: String, name: String) -> ServerSpec? { - registry.projects[project]?.servers[name] + registry.projects[Self.normalize(project)]?.servers[name] } public func specs(project: String) -> [ServerSpec] { - (registry.projects[project]?.servers ?? [:]).values.sorted { $0.name < $1.name } + (registry.projects[Self.normalize(project)]?.servers ?? [:]).values.sorted { + $0.name < $1.name + } } public func isTrusted(project: String) -> Bool { - registry.projects[project]?.trusted ?? false + registry.projects[Self.normalize(project)]?.trusted ?? false } public func setTrusted(project: String) throws { + let project = Self.normalize(project) var entry = registry.projects[project] ?? RegisteredProject() entry.trusted = true registry.projects[project] = entry @@ -109,6 +127,7 @@ public actor Registry { } public func unregister(project: String, name: String) throws { + let project = Self.normalize(project) registry.projects[project]?.servers[name] = nil if let entry = registry.projects[project], entry.servers.isEmpty { registry.projects[project] = nil @@ -116,8 +135,17 @@ public actor Registry { try persistRegistry() } + /** Drop a project entirely (trust + ad-hoc servers). Used when the checkout + path is gone so registry/Spotlight stop claiming it. */ + public func removeProject(_ path: String) throws { + let project = Self.normalize(path) + guard registry.projects[project] != nil else { return } + registry.projects[project] = nil + try persistRegistry() + } + public func persistedState(serverID: String) -> PersistedServerState? { - state.servers[serverID] + state.servers[Self.normalizeServerID(serverID)] } public func allPersistedState() -> [String: PersistedServerState] { @@ -125,6 +153,7 @@ public actor Registry { } public func updateState(serverID: String, _ mutate: (inout PersistedServerState) -> Void) throws { + let serverID = Self.normalizeServerID(serverID) var entry = state.servers[serverID] ?? PersistedServerState() mutate(&entry) state.servers[serverID] = entry @@ -134,11 +163,34 @@ public actor Registry { /** Drop a state row whose server no longer exists in config or the registry (rename / delete). Keeps recoverAtStartup from re-visiting ghosts. */ public func removeState(serverID: String) throws { + let serverID = Self.normalizeServerID(serverID) guard state.servers[serverID] != nil else { return } state.servers[serverID] = nil try persistState() } + /** Drop every state row whose project prefix matches (discarded checkout). */ + public func removeState(forProject path: String) throws { + let prefix = "\(Self.normalize(path))::" + let keys = state.servers.keys.filter { $0.hasPrefix(prefix) } + guard !keys.isEmpty else { return } + for key in keys { + state.servers[key] = nil + } + try persistState() + } + + private static func normalize(_ project: String) -> String { + canonicalProjectPath(project) + } + + private static func normalizeServerID(_ id: String) -> String { + guard let separator = id.range(of: "::") else { return id } + let project = String(id[id.startIndex.. ProcessIdentity? { + /** Live identity for `pid`, or nil if gone / not readable. */ + public static func identity(of pid: pid_t) -> ProcessIdentity? { var mib: [Int32] = [CTL_KERN, KERN_PROC, KERN_PROC_PID, pid] var length = MemoryLayout.stride var info = kinfo_proc() diff --git a/Sources/DevCtlDaemonCore/Supervisor/ServerSupervisor.swift b/Sources/DevCtlDaemonCore/Supervisor/ServerSupervisor.swift index 9d7b32e..7a9ca30 100644 --- a/Sources/DevCtlDaemonCore/Supervisor/ServerSupervisor.swift +++ b/Sources/DevCtlDaemonCore/Supervisor/ServerSupervisor.swift @@ -13,16 +13,34 @@ public actor ServerSupervisor { private let logStore: LogStore private var outTailer: SpoolTailer? private var consecutiveSuccesses = 0 + /** Committed port before override/rebind; status.declaredPort. */ + private var declaredPort: Int? + /** Error-stream tally for the current process, captured when the phase turns + terminal or unhealthy rather than recomputed per status call. Cleared at + spawn so a crash loop reports this incarnation, bracketed to the run's + start, and persisted so it survives a daemon restart. */ + private var errorSummary: ErrorSummary? private var everHealthy = false + /** What this run binds after override/rebind/materialization. */ + private var effectivePort: Int? private var healthTask: Task? private var lastExit: LastExit? private var lastHealthAt: Date? + private var lastDescendantSnapshot: [ProcessIdentity] = [] private let launcher: any ProcessLauncher + /** Resolved named secondaries for this run (status.ports). */ + private var namedPorts: [String: Int]? private var observedPort: Int? private let paths: DevCtlPaths private var phase: ServerPhase = .stopped private var pid: pid_t? + private var portClaim: PortClaim? + private var portConflict: PortConflict? private let prober: any HealthProber + /** Captured when the phase turns terminal, not recomputed per status call: + the log stops growing once the process is gone, so one read at the + transition is both cheaper and a truer snapshot of the failure. */ + private var recentLogTail: [String]? private let registry: Registry private var runningSpecHash: String? private var runTask: Task? @@ -34,6 +52,8 @@ public actor ServerSupervisor { /** Carries the stop()'s intent into recordOutcome: deliberate clears the resume-on-boot flag, a launchd drain keeps it. */ private var stopWasDeliberate = true + /** Durable why evidence across ensure truncate / daemon rehydrate. */ + private var terminalEvidence: [String]? public init( events: EventStore? = nil, @@ -46,16 +66,21 @@ public actor ServerSupervisor { ) { self.events = events self.launcher = launcher - self.logStore = LogStore(currentURL: paths.structuredLogFile(project: projectPath, server: spec.name)) + /** Match Registry's normalized state keys (`/var` vs `/private/var`). */ + let project = canonicalProjectPath(projectPath) + self.logStore = LogStore(currentURL: paths.structuredLogFile(project: project, server: spec.name)) self.paths = paths self.prober = prober - self.projectPath = projectPath + self.projectPath = project self.registry = registry self.spec = spec + let id = serverID(project: project, name: spec.name) if let persisted = AtomicFile.loadDefensively(StateFile.self, from: paths.stateFile)? - .servers[serverID(project: projectPath, name: spec.name)] { + .servers[id] { + self.errorSummary = persisted.errorSummary self.lastExit = persisted.lastExit self.spawnError = persisted.spawnError + self.terminalEvidence = persisted.terminalEvidence if persisted.phase == .crashed || persisted.phase == .failed { self.phase = persisted.phase } @@ -66,6 +91,22 @@ public actor ServerSupervisor { spec = newSpec } + /** Port metadata for status/agents. Call after materializing the spawn spec. */ + public func setPortMeta( + claim: PortClaim? = nil, + declaredPort: Int?, effectivePort: Int?, portConflict: PortConflict? = nil + ) { + self.declaredPort = declaredPort + self.effectivePort = effectivePort + self.portClaim = claim + self.namedPorts = claim.flatMap { $0.named.isEmpty ? nil : $0.named } + self.portConflict = portConflict + } + + public func clearBoundPortMeta() { + portConflict = nil + } + /** Starts the server if not already starting/running; otherwise joins the in-flight attempt. Returns once a pid exists or the spawn has failed; the phase stays `starting` until the healthcheck passes. */ @@ -85,8 +126,12 @@ public actor ServerSupervisor { phase = .starting stopRequested = false spawnError = nil + errorSummary = nil + terminalEvidence = nil everHealthy = false observedPort = nil + recentLogTail = nil + lastDescendantSnapshot = [] consecutiveFailures = 0 consecutiveSuccesses = 0 runningSpecHash = Self.specHash(spec) @@ -198,13 +243,17 @@ public actor ServerSupervisor { stopRequested = true stopWasDeliberate = deliberate phase = .stopping + /** Capture before any signal: after the grace window the pid number may + name a different process, and SIGKILL must not follow a recycled id. */ + let rootIdentity = ProcessTree.identity(of: target) let snapshotResult = ProcessTree.descendants(of: target) if case .failed(let code) = snapshotResult { DevCtlLog.supervisor.error( "descendant sweep failed before SIGTERM (errno \(code)); group-only teardown") } let snapshot = snapshotResult.identities - ProcessTree.signalTree(rootPid: target, descendants: snapshot, signal: SIGTERM) + ProcessTree.signalTree( + descendants: snapshot, rootPid: target, signal: SIGTERM) let deadline = ContinuousClock.now.advanced(by: .seconds(graceSeconds)) while ContinuousClock.now < deadline { if runTask == nil { break } @@ -223,9 +272,13 @@ public actor ServerSupervisor { byPid[identity.pid] = identity } let escalation = Array(byPid.values) - if kill(target, 0) == 0 { + if let rootIdentity, + ProcessTree.shouldSignal( + snapshotted: rootIdentity, live: ProcessTree.identity(of: target)) + { ProcessTree.signalTree( - rootPid: target, descendants: escalation, signal: SIGKILL, revalidate: true) + descendants: escalation, revalidate: true, rootIdentity: rootIdentity, + rootPid: target, signal: SIGKILL) } else { ProcessTree.escalateIndividuals(escalation) } @@ -236,8 +289,16 @@ public actor ServerSupervisor { public func status() -> ServerStatus { let check = EffectiveHealthcheck.resolve(spec: spec) let terminal = phase == .crashed || phase == .failed + /** The tail is captured once at the transition and served from memory. A + server rehydrated as crashed after a daemon restart has none in memory + (only errorSummary is persisted), so fall back to a live read there, + which matches the pre-capture behavior for that narrow case. */ + let tail = terminal ? (recentLogTail ?? spoolTail()) : nil + let evidence = terminal ? (terminalEvidence ?? tail) : nil return ServerStatus( - declaredPort: spec.port, + declaredPort: declaredPort ?? spec.port, + effectivePort: effectivePort ?? spec.port, + errorSummary: errorSummary, heads: spec.heads, healthcheck: check.kind, icon: spec.icon, @@ -247,11 +308,14 @@ public actor ServerSupervisor { observedPort: observedPort, phase: phase, pid: pid.map(Int.init), + portConflict: portConflict, + ports: namedPorts, project: projectPath, - recentLogTail: terminal ? spoolTail() : nil, + recentLogTail: tail, server: spec.name, spawnError: spawnError, specStale: specStaleFlag(), + terminalEvidence: evidence, uptimeSec: startedAt.map { Int(Date().timeIntervalSince($0)) }, url: spec.url ) @@ -294,6 +358,9 @@ public actor ServerSupervisor { private func recordProbe(success: Bool) { /** Probes landing after the process died must not resurrect state. */ guard phase == .starting || phase == .running || phase == .unhealthy else { return } + if pid != nil { + refreshDescendantSnapshot() + } let healthyAfter = spec.healthcheck?.healthyAfter ?? 1 let unhealthyAfter = spec.healthcheck?.unhealthyAfter ?? 3 if success { @@ -319,6 +386,10 @@ public actor ServerSupervisor { `starting` until the deadline callers chose, never `unhealthy`. */ if everHealthy, phase == .running, consecutiveFailures >= unhealthyAfter { phase = .unhealthy + /** The process is still writing, so snapshot the err tally at the + moment it degrades; a later recovery to running clears nothing, + so the count reflects the most recent unhealthy episode. */ + errorSummary = captureErrorSummary(since: startedAt) postHealthEvent(.unhealthy) } } @@ -336,13 +407,55 @@ public actor ServerSupervisor { } } - private func recordObservedPort(ports: [Int]) { + private func recordObservedPort(ports: [Int]) async { guard !ports.isEmpty else { return } - if let declared = spec.port, ports.contains(declared) { - observedPort = declared + let expected = effectivePort ?? spec.port + let claimPorts = Set(portClaim?.allPorts ?? expected.map { [$0] } ?? []) + if let expected, ports.contains(expected) { + observedPort = expected + } else if let claimed = ports.first(where: { claimPorts.contains($0) }) { + observedPort = claimed } else { observedPort = ports.first } + /** Strict bind: primary must match. Listeners on claimed secondaries are + expected for composites; anything outside the claim is drift. */ + guard let expected, let observed = observedPort, + phase == .running || phase == .starting + else { return } + if observed == expected || claimPorts.contains(observed) { + if observed == expected { return } + /** Secondary claimed port observed without primary: still require primary. */ + if ports.contains(expected) { return } + } + guard observed != expected else { return } + portConflict = PortConflict( + declaredPort: declaredPort ?? expected, + effectivePort: expected, + message: + "server listened on \(observed) instead of \(expected); add {port} to the command, set portEnv, or use --port / devctl.local.json", + state: .drift) + phase = .failed + spawnError = SpawnError(message: "port drift: expected \(expected), observed \(observed)") + errorSummary = captureErrorSummary(since: startedAt) + recentLogTail = spoolTail() + terminalEvidence = recentLogTail + healthTask?.cancel() + DevCtlLog.supervisor.error( + "port-drift \(spec.name)@\(projectPath) expected \(expected) observed \(observed)") + await events?.post( + kind: .failed, project: projectPath, server: spec.name, + detail: "port drift \(expected)->\(observed)") + let id = serverID(project: projectPath, name: spec.name) + let summary = errorSummary + let err = spawnError + let evidence = terminalEvidence + await registryUpdate(id: id) { entry in + entry.errorSummary = summary + entry.phase = .failed + entry.spawnError = err + entry.terminalEvidence = evidence + } } /** Log access for the router: queries and marks flow through the store so @@ -387,6 +500,7 @@ public actor ServerSupervisor { private func recordSpawn(pid childPid: pid_t, id: String) async { pid = childPid + refreshDescendantSnapshot() let spawnedAt = Date() startedAt = spawnedAt let out = SpoolTailer( @@ -402,6 +516,10 @@ public actor ServerSupervisor { await logStore.append(stream: .sys, text: "started pid=\(childPid)") await events?.post(kind: .started, project: projectPath, server: spec.name, detail: "pid \(childPid)") startHealthMonitor() + Task { [weak self] in + try? await Task.sleep(for: .milliseconds(100)) + await self?.refreshDescendantSnapshot() + } await registryUpdate(id: id) { entry in entry.lastExit = nil entry.phase = .starting @@ -413,6 +531,20 @@ public actor ServerSupervisor { settleSpawnWaiters() } + private func refreshDescendantSnapshot() { + guard let pid else { return } + lastDescendantSnapshot = ProcessTree.descendants(of: pid).identities + } + + /** Snapshot the err-stream tally for the run that just started at + `windowStart`. Reads only from that point forward, so a crash loop reports + the current incarnation rather than the whole log history. */ + private func captureErrorSummary(since windowStart: Date?) -> ErrorSummary? { + LogQuery.summarize( + current: paths.structuredLogFile(project: projectPath, server: spec.name), + streams: [.err], since: windowStart) + } + private func recordSpawnFailure(_ error: SpawnError, id: String) async { spawnError = error phase = .failed @@ -423,11 +555,15 @@ public actor ServerSupervisor { runTask = nil healthTask?.cancel() healthTask = nil + recentLogTail = spoolTail() + terminalEvidence = recentLogTail + let evidence = terminalEvidence await registryUpdate(id: id) { entry in entry.phase = .failed entry.pid = nil entry.spawnError = error entry.startedAt = nil + entry.terminalEvidence = evidence } settleSpawnWaiters() } @@ -445,10 +581,21 @@ public actor ServerSupervisor { case .signaled(let signal): lastExit = LastExit(at: Date(), code: nil, signal: signal) } + let windowStart = startedAt if let out = outTailer { await out.stop() } if let err = errTailer { await err.stop() } outTailer = nil errTailer = nil + /** After the final drain, so the lines that explain the exit are in the + log before the snapshots are taken. */ + recentLogTail = spoolTail() + terminalEvidence = recentLogTail + errorSummary = captureErrorSummary(since: windowStart) + if !stopRequested, let rootPid = pid { + ProcessTree.signalTree( + descendants: lastDescendantSnapshot, rootPid: rootPid, signal: SIGTERM) + } + lastDescendantSnapshot = [] pid = nil startedAt = nil observedPort = nil @@ -464,12 +611,17 @@ public actor ServerSupervisor { await events?.post( kind: finalPhase == .stopped ? .stopped : .crashed, project: projectPath, server: spec.name, detail: cause) + let errors = errorSummary + let evidence = finalPhase == .stopped ? nil : terminalEvidence + if finalPhase == .stopped { terminalEvidence = nil } await registryUpdate(id: id) { entry in + entry.errorSummary = errors entry.lastExit = exit entry.phase = finalPhase entry.pid = nil if retireIntent { entry.resumeOnBoot = nil } entry.startedAt = nil + entry.terminalEvidence = evidence } settleSpawnWaiters() } @@ -507,7 +659,7 @@ public actor ServerSupervisor { let records = LogQuery.run( current: paths.structuredLogFile(project: projectPath, server: spec.name), options: LogQueryOptions(streams: [.err, .out, .sys], tail: lines)) - return records.isEmpty ? nil : records.map { "\($0.stream.rawValue): \($0.text)" } + return records.isEmpty ? nil : records.map(\.contextLine) } private func waitForRunTaskCompletion() async { diff --git a/Sources/DevCtlKit/Agent/AgentContext.swift b/Sources/DevCtlKit/Agent/AgentContext.swift new file mode 100644 index 0000000..805063b --- /dev/null +++ b/Sources/DevCtlKit/Agent/AgentContext.swift @@ -0,0 +1,127 @@ +import Darwin +import Foundation + +/** The harness-agnostic session-context block: fenced plain text any agent + harness can inject at session start (and after compaction) so a new or reset + session relearns what devctl is running. Pure over an already-fetched status + list, so it is unit-tested here; the socket fetch and the daemon guards live + in the CLI. + + Every string this emits is devctl's own: a phase, a port, a count, a + timestamp, a strerror name. A line of child output or a configured command + string never appears, because both are attacker-influenceable and would reach + an agent's context unattributed. Surfacing that an error stream is piling up + is safe (the count and timestamps are arithmetic); surfacing the lines + themselves is not, so the block points the agent at `devctl why`, which it + runs itself and can attribute. */ +public enum AgentContext { + public static let maxLength = 2400 + + /** Nil when there is nothing to say: an untrusted project (the hook advertises + only trusted ones) or no registered servers. Otherwise the fenced block, + truncated to `maxLength` with the closing tag preserved. */ + public static func render(list: ServerListResult) -> String? { + guard list.trusted == true, !list.servers.isEmpty else { return nil } + /** Bad-state servers lead: the block is length-capped and truncates from + the end, so the servers an agent must act on cannot sit behind the + healthy ones. Alphabetical within each group for a stable read. */ + let ordered = list.servers.sorted { lhs, rhs in + let lb = isBadState(lhs), rb = isBadState(rhs) + if lb != rb { return lb } + return lhs.server < rhs.server + } + var lines: [String] = [""] + lines.append( + "This project's dev servers are managed by devctl (daemon-supervised; they and their logs survive session compaction and restarts). Prefer devctl over launching servers directly.") + if let project = ordered.first?.project, CheckoutIdentity.isLinkedWorktree(project: project) { + let preferred = CheckoutIdentity.preferredSubdomain(project: project, committedHost: nil) + lines.append( + "This checkout is a git worktree of \(preferred); use the URLs below, not the main checkout's host.") + } + for server in ordered { + lines.append(bullet(for: server)) + if let conflict = server.portConflict { + lines.append(" warning: \(conflict.message)") + } + if isBadState(server) { + if let summary = server.errorSummary { + lines.append(" \(errorLine(summary))") + } + lines.append(" run: devctl why \(server.server) --json") + } + } + lines.append( + "Useful: devctl ensure (idempotent start) · devctl wait --healthy · devctl why (root cause) · devctl logs --since-mark --json · devctl mark \"text\" · devctl events --since 10m · devctl lock --no-pause -- …. All support --json.") + lines.append( + "While you work, monitor devctl itself: if it misbehaves, surprises you, or a missing capability slows you down, flag it (a line in ~/code/devctl/BACKLOG.md, or tell the user) rather than silently working around it.") + lines.append("") + let text = lines.joined(separator: "\n") + return text.count > maxLength ? String(text.prefix(maxLength)) + "\n" : text + } + + /** A server that warrants attention: broken outright, degraded, or running + against a spec that has since changed on disk. */ + private static func isBadState(_ server: ServerStatus) -> Bool { + switch server.phase { + case .crashed, .failed, .unhealthy: + return true + case .running, .starting, .stopped, .stopping: + return server.specStale == true || server.portConflict?.state == .held + || server.portConflict?.state == .drift + } + } + + private static func bullet(for server: ServerStatus) -> String { + var parts = ["- \(server.server): \(server.phase.rawValue)"] + if let url = server.url { parts.append(url) } + if let heads = server.heads, !heads.isEmpty { + let rendered = heads.sorted { $0.key < $1.key } + .map { "\($0.key) \($0.value)" } + .joined(separator: ", ") + parts.append("heads: \(rendered)") + } + if let port = server.effectivePort ?? server.observedPort ?? server.declaredPort { + parts.append("port \(port)") + } + if let ports = server.ports, !ports.isEmpty { + let rendered = ports.sorted { $0.key < $1.key } + .map { "\($0.key) \($0.value)" } + .joined(separator: ", ") + parts.append("ports: \(rendered)") + } + if let conflict = server.portConflict, conflict.state == .rebound { + parts.append("rebinding: worktree checkout, not the main origin") + } + switch server.phase { + case .crashed: + if let exit = server.lastExit { + let cause = exit.code.map { "exit \($0)" } ?? exit.signal.map { "signal \($0)" } ?? "unknown" + parts.append("last exit \(cause) at \(JSONCoding.formatISO8601(exit.at))") + } + case .failed: + /** strerror is devctl's own naming of the OS error, never the + spawnError message, which can echo the configured command. */ + parts.append("spawn failed: \(spawnCause(server.spawnError))") + case .unhealthy: + let since = server.lastHealthAt.map { JSONCoding.formatISO8601($0) } ?? "startup" + parts.append("last healthy \(since)") + case .running, .starting, .stopped, .stopping: + break + } + if server.specStale == true { parts.append("config changed since start") } + parts.append("log \(server.logPath)") + return parts.joined(separator: " · ") + } + + private static func spawnCause(_ error: SpawnError?) -> String { + guard let errno = error?.errno, errno != 0 else { return "could not start" } + return String(cString: strerror(Int32(errno))) + } + + private static func errorLine(_ summary: ErrorSummary) -> String { + if summary.count == 1 { + return "1 error line at \(JSONCoding.formatISO8601(summary.lastAt))" + } + return "\(summary.count) error lines since \(JSONCoding.formatISO8601(summary.firstAt)), latest \(JSONCoding.formatISO8601(summary.lastAt))" + } +} diff --git a/Sources/DevCtlKit/Agent/DiscoveryStanza.swift b/Sources/DevCtlKit/Agent/DiscoveryStanza.swift index 7421c63..3a54f16 100644 --- a/Sources/DevCtlKit/Agent/DiscoveryStanza.swift +++ b/Sources/DevCtlKit/Agent/DiscoveryStanza.swift @@ -12,6 +12,6 @@ public enum DiscoveryStanza { naming/host conventions sentence is constant either way. */ public static func render(serverNames: [String]) -> String { let example = serverNames.first ?? "" - return "- This project is registered with devctl (devservers.json). Prefer `devctl ensure \(example)` / `devctl status` / `devctl logs \(example)` over launching the server directly. Name servers after the project (not `web`); use `.localhost` hosts, not bare `localhost`." + return "- This project is registered with devctl (devservers.json). Prefer `devctl ensure \(example)` / `devctl status` / `devctl logs \(example)` over launching the server directly. Name servers after the project (not `web`); use `.localhost` hosts, not bare `localhost`. In a git worktree, ensure still manages the server: the live URL comes from `status` / session context (may be `worktree-*..localhost` on another port). If context warns about a port conflict, follow that URL or hint; do not edit `devservers.json` just to change ports; do not start an unmanaged `pnpm dev --port`." } } diff --git a/Sources/DevCtlKit/Config/LocalOverlay.swift b/Sources/DevCtlKit/Config/LocalOverlay.swift new file mode 100644 index 0000000..d38b1d7 --- /dev/null +++ b/Sources/DevCtlKit/Config/LocalOverlay.swift @@ -0,0 +1,97 @@ +import Foundation + +/** Gitignored per-checkout overlay (`devctl.local.json`): a partial server map + merged onto committed specs so a worktree can override port/command/env/url + without dirtying tracked `devservers.json`. */ +public struct LocalOverlayFile: Codable, Equatable, Sendable { + public var servers: [String: LocalOverlayServer]? + + public init(servers: [String: LocalOverlayServer]? = nil) { + self.servers = servers + } +} + +public struct LocalOverlayServer: Codable, Equatable, Sendable { + public var command: [String]? + public var cwd: String? + public var env: [String: String]? + public var heads: [String: String]? + public var healthcheck: HealthCheckSpec? + public var host: String? + public var port: Int? + public var portEnv: String? + public var ports: [String: SecondaryPort]? + public var portSpan: Int? + public var shell: Bool? + public var url: String? + + public init( + command: [String]? = nil, + cwd: String? = nil, + env: [String: String]? = nil, + heads: [String: String]? = nil, + healthcheck: HealthCheckSpec? = nil, + host: String? = nil, + port: Int? = nil, + portEnv: String? = nil, + ports: [String: SecondaryPort]? = nil, + portSpan: Int? = nil, + shell: Bool? = nil, + url: String? = nil + ) { + self.command = command + self.cwd = cwd + self.env = env + self.heads = heads + self.healthcheck = healthcheck + self.host = host + self.port = port + self.portEnv = portEnv + self.ports = ports + self.portSpan = portSpan + self.shell = shell + self.url = url + } +} + +public enum LocalOverlay { + public static func overlayURL(project: String) -> URL { + URL(fileURLWithPath: project).appending(path: "devctl.local.json") + } + + public static func load(project: String) -> LocalOverlayFile? { + let url = overlayURL(project: project) + guard let data = try? Data(contentsOf: url) else { return nil } + return try? JSONCoding.decoder().decode(LocalOverlayFile.self, from: data) + } + + /** Merge overlay fields onto a validated spec. Overlay wins per field. */ + public static func apply(spec: ServerSpec, overlay: LocalOverlayServer?, project: String) -> ServerSpec { + guard let overlay else { return spec } + var next = spec + if let command = overlay.command { next.command = command } + if let cwd = overlay.cwd { next.cwd = cwd } + if let env = overlay.env { + var merged = next.env ?? [:] + for (key, value) in env { merged[key] = value } + next.env = merged + } + if let heads = overlay.heads { next.heads = heads } + if let healthcheck = overlay.healthcheck { next.healthcheck = healthcheck } + if let host = overlay.host { next.host = host } + if let port = overlay.port { next.port = port } + if let portEnv = overlay.portEnv { next.portEnv = portEnv } + if let ports = overlay.ports { next.ports = ports } + if let portSpan = overlay.portSpan { next.portSpan = portSpan } + if let shell = overlay.shell { next.shell = shell } + if let url = overlay.url { + next.url = url + } else if overlay.port != nil || overlay.host != nil { + let host = next.host ?? ProjectConfigLoader.defaultSlug(project: project) + ".localhost" + if let port = next.port { + next.url = "http://\(host):\(port)/" + } + } + return next + } +} diff --git a/Sources/DevCtlKit/Config/ProjectConfig.swift b/Sources/DevCtlKit/Config/ProjectConfig.swift index 6c61695..c5ccfd7 100644 --- a/Sources/DevCtlKit/Config/ProjectConfig.swift +++ b/Sources/DevCtlKit/Config/ProjectConfig.swift @@ -43,6 +43,9 @@ public struct ProjectFileServer: Codable, Equatable, Sendable { public var icon: String? public var locks: [String]? public var port: Int? + public var portEnv: String? + public var ports: [String: SecondaryPort]? + public var portSpan: Int? public var shell: Bool? public var url: String? public var waitFor: WaitTarget? @@ -58,6 +61,9 @@ public struct ProjectFileServer: Codable, Equatable, Sendable { icon: String? = nil, locks: [String]? = nil, port: Int? = nil, + portEnv: String? = nil, + ports: [String: SecondaryPort]? = nil, + portSpan: Int? = nil, shell: Bool? = nil, url: String? = nil, waitFor: WaitTarget? = nil @@ -72,6 +78,9 @@ public struct ProjectFileServer: Codable, Equatable, Sendable { self.icon = icon self.locks = locks self.port = port + self.portEnv = portEnv + self.ports = ports + self.portSpan = portSpan self.shell = shell self.url = url self.waitFor = waitFor @@ -168,22 +177,27 @@ public enum ProjectConfigLoader { for dep in entry.dependsOn ?? [] where config.servers[dep] == nil { view.errors.append("server '\(name)' depends on unknown server '\(dep)'") } - specs.append( - ServerSpec( - command: entry.command, - cwd: entry.cwd, - dependsOn: entry.dependsOn, - env: entry.env, - heads: entry.heads, - healthcheck: entry.healthcheck, - icon: iconPath, - locks: entry.locks, - name: name, - port: entry.port, - shell: entry.shell, - url: url, - waitFor: entry.waitFor - )) + let draft = ServerSpec( + command: entry.command, + cwd: entry.cwd, + dependsOn: entry.dependsOn, + env: entry.env, + heads: entry.heads, + healthcheck: entry.healthcheck, + host: serverHost, + icon: iconPath, + locks: entry.locks, + name: name, + port: entry.port, + portEnv: entry.portEnv, + ports: entry.ports, + portSpan: entry.portSpan, + shell: entry.shell, + url: url, + waitFor: entry.waitFor + ) + view.errors.append(contentsOf: PortClaim.configErrors(spec: draft)) + specs.append(draft) } switch DependencyGraph.waves(specs: specs) { case .success: diff --git a/Sources/DevCtlKit/DeepLink/DeepLink.swift b/Sources/DevCtlKit/DeepLink/DeepLink.swift index 186694a..498497e 100644 --- a/Sources/DevCtlKit/DeepLink/DeepLink.swift +++ b/Sources/DevCtlKit/DeepLink/DeepLink.swift @@ -162,6 +162,11 @@ public struct DeepLink: Sendable, Equatable, Codable { case 1: return .success(matches[0]) default: + /** Prefer the main checkout when a worktree shares the basename. */ + let mains = matches.filter { !CheckoutIdentity.isLinkedWorktree(project: $0) } + if mains.count == 1 { + return .success(mains[0]) + } let candidates = matches.sorted().joined(separator: ", ") return .failure( WireError( diff --git a/Sources/DevCtlKit/Logs/LogFormat.swift b/Sources/DevCtlKit/Logs/LogFormat.swift index b6348da..c0f229d 100644 --- a/Sources/DevCtlKit/Logs/LogFormat.swift +++ b/Sources/DevCtlKit/Logs/LogFormat.swift @@ -27,6 +27,13 @@ public struct LogRecord: Codable, Equatable, Sendable { "\(JSONCoding.formatISO8601(at))\t\(stream.rawValue)\t\(text)" } + /** Stream-tagged payload for a human or an agent reading a tail: the shared + spelling behind `devctl why` evidence and `recentLogTail`, which used to + prefix it two different ways. */ + public var contextLine: String { + "\(stream.rawValue): \(text)" + } + public static func parse(_ line: Substring) -> LogRecord? { let firstTab = line.firstIndex(of: "\t") guard let firstTab else { return nil } diff --git a/Sources/DevCtlKit/Logs/LogQuery.swift b/Sources/DevCtlKit/Logs/LogQuery.swift index afec192..52ac6f1 100644 --- a/Sources/DevCtlKit/Logs/LogQuery.swift +++ b/Sources/DevCtlKit/Logs/LogQuery.swift @@ -41,8 +41,50 @@ public enum LogQuery { return files } + /** Why a caller-supplied grep pattern will not compile, or nil when it will. + Callers validate before querying: a pattern the engine cannot compile must + not silently degrade into "no filter", because returning every line reads + exactly like a query that matched everything. */ + public static func grepRejection(_ pattern: String) -> String? { + do { + _ = try Regex(pattern) + return nil + } catch { + return String(describing: error) + } + } + + /** Counts records on the given streams and brackets them in time, without + building the array. The count and the two timestamps are devctl's own + arithmetic over the log, safe to put in an agent's context where the lines + themselves must never go. `since` bounds the window to one process's run + (current.log is append-only across spawns), and drives the same file-skip + and binary search `run` uses, so it does not scan a long history. */ + public static func summarize(current: URL, streams: Set, since: Date?) -> ErrorSummary? { + var count = 0 + var first: Date? + var last: Date? + for record in run(current: current, options: LogQueryOptions(since: since, streams: streams)) { + count += 1 + if first == nil { first = record.at } + last = record.at + } + guard count > 0, let first, let last else { return nil } + return ErrorSummary(count: count, firstAt: first, lastAt: last) + } + public static func run(current: URL, options: LogQueryOptions) -> [LogRecord] { - let grep = options.grep.flatMap { try? Regex($0) } + /** A pattern that will not compile filters nothing out, so it would + answer with the whole log. Fail closed and say so instead: callers + screen user input with grepRejection first. */ + var grep: Regex? + if let pattern = options.grep { + guard let compiled = try? Regex(pattern) else { + DevCtlLog.daemon.error("log query grep pattern does not compile: \(pattern)") + return [] + } + grep = compiled + } var records: [LogRecord] = [] for file in familyFiles(current: current) { /** Whole-file skip: a file whose last line predates `since` cannot diff --git a/Sources/DevCtlKit/Model/Models.swift b/Sources/DevCtlKit/Model/Models.swift index 41b52a7..aaa6ded 100644 --- a/Sources/DevCtlKit/Model/Models.swift +++ b/Sources/DevCtlKit/Model/Models.swift @@ -97,6 +97,10 @@ public struct ServerSpec: Codable, Equatable, Sendable { serving several surfaces on one port): display name -> URL. */ public var heads: [String: String]? public var healthcheck: HealthCheckSpec? + /** Resolved server host (committed, worktree-derived, or overlay). Kept on + the spec so port materialization can rebuild urls without re-reading + the project file. Optional so older ad-hoc registry entries keep parsing. */ + public var host: String? /** Absolute path to an icon image (resolved from the config's project-relative path); used for Spotlight thumbnails. */ public var icon: String? @@ -107,6 +111,13 @@ public struct ServerSpec: Codable, Equatable, Sendable { public var locks: [String]? public var name: String public var port: Int? + /** Child env var that receives the effective port (default `PORT`). */ + public var portEnv: String? + /** Named secondary listeners (relative offsets or absolute singletons). */ + public var ports: [String: SecondaryPort]? + /** Claim `effectivePort ..< effectivePort+portSpan` without naming each + secondary. Sugar for apps that derive children from the primary env. */ + public var portSpan: Int? /** Runs the command through `/bin/zsh -lc` for shells that need login env (nvm/mise). */ public var shell: Bool? public var url: String? @@ -119,10 +130,14 @@ public struct ServerSpec: Codable, Equatable, Sendable { env: [String: String]? = nil, heads: [String: String]? = nil, healthcheck: HealthCheckSpec? = nil, + host: String? = nil, icon: String? = nil, locks: [String]? = nil, name: String, port: Int? = nil, + portEnv: String? = nil, + ports: [String: SecondaryPort]? = nil, + portSpan: Int? = nil, shell: Bool? = nil, url: String? = nil, waitFor: WaitTarget? = nil @@ -133,10 +148,14 @@ public struct ServerSpec: Codable, Equatable, Sendable { self.env = env self.heads = heads self.healthcheck = healthcheck + self.host = host self.icon = icon self.locks = locks self.name = name self.port = port + self.portEnv = portEnv + self.ports = ports + self.portSpan = portSpan self.shell = shell self.url = url self.waitFor = waitFor @@ -167,9 +186,27 @@ public struct SpawnError: Codable, Equatable, Sendable { } } +/** Counted, never quoted: what devctl observed on a server's error stream during + the current process's life. Every field is devctl's own arithmetic over the + log, which is what makes it safe to put in an agent's session context where a + child's own bytes must never go. */ +public struct ErrorSummary: Codable, Equatable, Sendable { + public var count: Int + public var firstAt: Date + public var lastAt: Date + + public init(count: Int, firstAt: Date, lastAt: Date) { + self.count = count + self.firstAt = firstAt + self.lastAt = lastAt + } +} + /** The core status schema agents consume; documented in docs/cli-contract.md. */ public struct ServerStatus: Codable, Equatable, Sendable { public var declaredPort: Int? + public var effectivePort: Int? + public var errorSummary: ErrorSummary? public var heads: [String: String]? public var healthcheck: HealthCheckType public var icon: String? @@ -179,16 +216,23 @@ public struct ServerStatus: Codable, Equatable, Sendable { public var observedPort: Int? public var phase: ServerPhase public var pid: Int? + public var portConflict: PortConflict? + /** Resolved named secondary ports when the spec declares `ports`. */ + public var ports: [String: Int]? public var project: String public var recentLogTail: [String]? public var server: String public var spawnError: SpawnError? public var specStale: Bool? + /** Short spool evidence persisted across ensure/rehydrate for `why`. */ + public var terminalEvidence: [String]? public var uptimeSec: Int? public var url: String? public init( declaredPort: Int? = nil, + effectivePort: Int? = nil, + errorSummary: ErrorSummary? = nil, heads: [String: String]? = nil, healthcheck: HealthCheckType = .none, icon: String? = nil, @@ -198,15 +242,20 @@ public struct ServerStatus: Codable, Equatable, Sendable { observedPort: Int? = nil, phase: ServerPhase, pid: Int? = nil, + portConflict: PortConflict? = nil, + ports: [String: Int]? = nil, project: String, recentLogTail: [String]? = nil, server: String, spawnError: SpawnError? = nil, specStale: Bool? = nil, + terminalEvidence: [String]? = nil, uptimeSec: Int? = nil, url: String? = nil ) { self.declaredPort = declaredPort + self.effectivePort = effectivePort + self.errorSummary = errorSummary self.heads = heads self.healthcheck = healthcheck self.icon = icon @@ -216,11 +265,14 @@ public struct ServerStatus: Codable, Equatable, Sendable { self.observedPort = observedPort self.phase = phase self.pid = pid + self.portConflict = portConflict + self.ports = ports self.project = project self.recentLogTail = recentLogTail self.server = server self.spawnError = spawnError self.specStale = specStale + self.terminalEvidence = terminalEvidence self.uptimeSec = uptimeSec self.url = url } @@ -256,6 +308,16 @@ public struct EventRecord: Codable, Equatable, Sendable { } } +/** Whether the menu bar should banner an event. Expected daemon bounce markers + (`daemon-restart`) are forensics for the feed, not user alerts. */ +public enum CrashNotificationPolicy { + public static func shouldNotify(kind: EventKind, detail: String?) -> Bool { + guard kind == .crashed || kind == .failed else { return false } + if let detail, detail.hasPrefix("daemon-restart") { return false } + return true + } +} + /** One step in a `devctl why` diagnosis chain. */ public struct WhyFinding: Codable, Equatable, Sendable { public var evidence: [String] diff --git a/Sources/DevCtlKit/Model/PortConflict.swift b/Sources/DevCtlKit/Model/PortConflict.swift new file mode 100644 index 0000000..17da12a --- /dev/null +++ b/Sources/DevCtlKit/Model/PortConflict.swift @@ -0,0 +1,36 @@ +import Foundation + +/** Structured port-conflict signal agents branch on without parsing English. + Present on status when a conflict was handled (rebound), is latent (held while + stopped), or killed the run (drift). */ +public struct PortConflict: Codable, Equatable, Sendable { + public var declaredPort: Int + public var effectivePort: Int? + /** `server@project` for a managed holder, or a short unmanaged label. */ + public var holder: String? + public var message: String + public var state: PortConflictState + + public init( + declaredPort: Int, + effectivePort: Int? = nil, + holder: String? = nil, + message: String, + state: PortConflictState + ) { + self.declaredPort = declaredPort + self.effectivePort = effectivePort + self.holder = holder + self.message = message + self.state = state + } +} + +public enum PortConflictState: String, Codable, Sendable { + /** Sibling held the committed port; this run rebound and is serving elsewhere. */ + case rebound + /** Declared/effective port is held and this server is not up yet (or ensure refused). */ + case held + /** Child listened on a different port than effectivePort (Vite silent bump). */ + case drift +} diff --git a/Sources/DevCtlKit/Net/LoopbackProbe.swift b/Sources/DevCtlKit/Net/LoopbackProbe.swift new file mode 100644 index 0000000..45fd466 --- /dev/null +++ b/Sources/DevCtlKit/Net/LoopbackProbe.swift @@ -0,0 +1,63 @@ +import Darwin +import Foundation + +/** Is anything accepting connections on a loopback port right now. + + Lives in DevCtlKit because two callers need it and cannot share a home + otherwise: the daemon's health prober and the CLI's `doctor`, which cannot + import DevCtlDaemonCore (that target links swift-subprocess). The CLI used to + carry its own IPv4-only copy with a blocking connect and no deadline, which + reported a free port for anything bound to `::1` alone. */ +public enum LoopbackProbe { + /** Dev servers bind either stack: Vite 5+ listens on `::1` only, older + tooling on `127.0.0.1` only, so both loopback families are tried and + either answering counts as a live listener. */ + public static func isListening(port: Int, timeoutMs: Int = 300) -> Bool { + connects(port: port, timeoutMs: timeoutMs, family: AF_INET) + || connects(port: port, timeoutMs: timeoutMs, family: AF_INET6) + } + + /** Non-blocking connect with a poll deadline, so an unreachable port cannot + stall a status call for the kernel's full connect timeout. */ + private static func connects(port: Int, timeoutMs: Int, family: Int32) -> Bool { + let sock = socket(family, SOCK_STREAM, 0) + guard sock >= 0 else { return false } + defer { close(sock) } + let flags = fcntl(sock, F_GETFL) + _ = fcntl(sock, F_SETFL, flags | O_NONBLOCK) + var storage = sockaddr_storage() + let sockLen: socklen_t + if family == AF_INET6 { + var addr = sockaddr_in6() + addr.sin6_family = sa_family_t(AF_INET6) + addr.sin6_port = UInt16(port).bigEndian + addr.sin6_addr = in6addr_loopback + sockLen = socklen_t(MemoryLayout.size) + withUnsafeMutablePointer(to: &storage) { ptr in + ptr.withMemoryRebound(to: sockaddr_in6.self, capacity: 1) { $0.pointee = addr } + } + } else { + var addr = sockaddr_in() + addr.sin_family = sa_family_t(AF_INET) + addr.sin_port = UInt16(port).bigEndian + addr.sin_addr = in_addr(s_addr: inet_addr("127.0.0.1")) + sockLen = socklen_t(MemoryLayout.size) + withUnsafeMutablePointer(to: &storage) { ptr in + ptr.withMemoryRebound(to: sockaddr_in.self, capacity: 1) { $0.pointee = addr } + } + } + let result = withUnsafePointer(to: &storage) { ptr in + ptr.withMemoryRebound(to: sockaddr.self, capacity: 1) { sa in + connect(sock, sa, sockLen) + } + } + if result == 0 { return true } + guard errno == EINPROGRESS else { return false } + var pollTarget = pollfd(fd: sock, events: Int16(POLLOUT), revents: 0) + guard poll(&pollTarget, 1, Int32(timeoutMs)) == 1 else { return false } + var soError: Int32 = 0 + var soLen = socklen_t(MemoryLayout.size) + getsockopt(sock, SOL_SOCKET, SO_ERROR, &soError, &soLen) + return soError == 0 + } +} diff --git a/Sources/DevCtlKit/Net/PortClaim.swift b/Sources/DevCtlKit/Net/PortClaim.swift new file mode 100644 index 0000000..109860e --- /dev/null +++ b/Sources/DevCtlKit/Net/PortClaim.swift @@ -0,0 +1,163 @@ +import Foundation + +/** A named secondary listener relative to the effective primary, or an absolute + machine-singleton port. Exactly one of `offset` or `port` must be set. */ +public struct SecondaryPort: Codable, Equatable, Sendable { + public var env: String? + public var offset: Int? + public var port: Int? + + public init(env: String? = nil, offset: Int? = nil, port: Int? = nil) { + self.env = env + self.offset = offset + self.port = port + } + + /** Nil when the entry is well-formed; otherwise a config-check message. */ + public func validationError(name: String) -> String? { + switch (offset, port) { + case (nil, nil): + return "ports.\(name): set offset or port" + case (.some, .some): + return "ports.\(name): set offset or port, not both" + case (.some(let value), nil) where value < 0: + return "ports.\(name): offset must be >= 0" + case (nil, .some(let value)) where value < 1 || value > 65_535: + return "ports.\(name): port must be 1...65535" + default: + return nil + } + } +} + +/** Every port a spawn claims: primary, optional consecutive span sugar, and + named secondaries. Pure; allocate / settle / materialize all use this set. */ +public struct PortClaim: Equatable, Sendable { + /** Fixed ports that never shift with sibling rebind. */ + public var absolute: [Int] + /** Env key → resolved number for child injection (primary + named with env). */ + public var injections: [String: Int] + /** Named secondary → resolved number (for status / agents). */ + public var named: [String: Int] + public var primary: Int? + /** Primary plus every relative member (span sugar and named offsets). */ + public var relative: [Int] + + public init( + absolute: [Int] = [], + injections: [String: Int] = [:], + named: [String: Int] = [:], + primary: Int? = nil, + relative: [Int] = [] + ) { + self.absolute = absolute + self.injections = injections + self.named = named + self.primary = primary + self.relative = relative + } + + /** Unique sorted ports that must be free for this claim. */ + public var allPorts: [Int] { + Array(Set(relative + absolute)).sorted() + } + + /** Offsets from primary that must stay free when searching for a rebound + base (excludes primary itself as offset 0). Absolute ports are not + included: they never move. */ + public var relativeOffsets: [Int] { + guard let primary else { return [] } + return relative.compactMap { port in + let offset = port - primary + return offset >= 0 ? offset : nil + }.sorted() + } + + /** Build a claim from a materialized-ready spec and its effective primary. + `error` is a config-shaped message when span and named offsets collide. */ + public static func resolve(spec: ServerSpec, effectivePort: Int?) -> ( + claim: PortClaim?, error: String? + ) { + var relative: Set = [] + var absolute: Set = [] + var named: [String: Int] = [:] + var injections: [String: Int] = [:] + if let primary = effectivePort { + relative.insert(primary) + let envKey = spec.portEnv ?? "PORT" + injections[envKey] = primary + } + if let span = spec.portSpan { + guard span >= 1 else { + return (nil, "portSpan must be >= 1") + } + guard let primary = effectivePort else { + return (nil, "portSpan requires a primary port") + } + for offset in 0.. = [] + if let span = spec.portSpan, span > 1 { + spanOffsets = Set(1.. [String] { + var errors: [String] = [] + if let span = spec.portSpan, span < 1 { + errors.append("server '\(spec.name)': portSpan must be >= 1") + } + var spanOffsets: Set = [] + if let span = spec.portSpan, span > 1 { + spanOffsets = Set(1.. ServerSpec { + var next = spec + let host = effectiveHost ?? spec.host + if let host { next.host = host } + let resolved = PortClaim.resolve(spec: spec, effectivePort: effectivePort) + let claim = + resolved.claim + ?? PortClaim(primary: effectivePort, relative: effectivePort.map { [$0] } ?? []) + if let effectivePort { + next.port = effectivePort + } + if !claim.injections.isEmpty { + var env = next.env ?? [:] + for (key, value) in claim.injections { + env[key] = String(value) + } + if let host { + env["DEVCTL_HOST"] = host + } + next.env = env + } else if let effectivePort { + let envKey = spec.portEnv ?? "PORT" + var env = next.env ?? [:] + env[envKey] = String(effectivePort) + if let host { + env["DEVCTL_HOST"] = host + } + next.env = env + } + let portText = effectivePort.map(String.init) + let hostText = host + let previousHost = matchHost ?? spec.host + next.command = next.command.map { substitute($0, port: portText, host: hostText) } + if let url = next.url { + next.url = rewriteURL(url, port: effectivePort, host: host, matchHost: previousHost) + ?? substitute(url, port: portText, host: hostText) + } else if let effectivePort, let host { + next.url = "http://\(host):\(effectivePort)/" + } + if let heads = next.heads { + var rewritten: [String: String] = [:] + for (name, url) in heads { + rewritten[name] = + rewriteURL(url, port: effectivePort, host: host, matchHost: previousHost) + ?? substitute(url, port: portText, host: hostText) + } + next.heads = rewritten + } + if var health = next.healthcheck { + if let healthURL = health.url { + health.url = + rewriteURL(healthURL, port: effectivePort, host: host, matchHost: previousHost) + ?? substitute(healthURL, port: portText, host: hostText) + } + if let effectivePort, health.port != nil { + health.port = effectivePort + } + next.healthcheck = health + } + return next + } + + /** Replace `{port}` / `{host}` tokens in a single string. */ + public static func substitute(_ text: String, port: String?, host: String?) -> String { + var out = text + if let port { + out = out.replacingOccurrences(of: "{port}", with: port) + } + if let host { + out = out.replacingOccurrences(of: "{host}", with: host) + } + return out + } + + /** Prefer structured URL rewrite so an explicit committed URL follows the + effective port/host without requiring `{port}` tokens. Host is only + replaced when the URL's host exactly matches `matchHost` (so a head like + `admin.app.localhost` keeps its subdomain when only the port moves). */ + public static func rewriteURL( + _ raw: String, port: Int?, host: String?, matchHost: String? = nil + ) -> String? { + guard var components = URLComponents(string: raw) else { return nil } + if let host { + if let matchHost { + if components.host == matchHost { + components.host = host + } + } else { + components.host = host + } + } + if let port { components.port = port } + return components.string + } +} diff --git a/Sources/DevCtlKit/Paths/CheckoutIdentity.swift b/Sources/DevCtlKit/Paths/CheckoutIdentity.swift new file mode 100644 index 0000000..38b8cf3 --- /dev/null +++ b/Sources/DevCtlKit/Paths/CheckoutIdentity.swift @@ -0,0 +1,121 @@ +import Foundation + +/** Git checkout identity helpers for sibling port rebind and ephemeral worktree + hosts. Shells out to `git`; failures return nil so non-git projects keep the + pre-coexistence path. */ +public enum CheckoutIdentity { + /** Absolute path to the shared git directory, or nil if not a git checkout. */ + public static func gitCommonDir(project: String) -> String? { + git(project: project, args: ["rev-parse", "--git-common-dir"]).map { + canonicalProjectPath(absoluteGitPath($0, project: project)) + } + } + + /** True when this path is a linked worktree (git-dir differs from common-dir, + or `.git` is a file). Main checkouts and non-git trees return false. */ + public static func isLinkedWorktree(project: String) -> Bool { + let gitFile = URL(fileURLWithPath: project).appending(path: ".git") + var isDir: ObjCBool = false + if FileManager.default.fileExists(atPath: gitFile.path, isDirectory: &isDir), !isDir.boolValue { + return true + } + guard let common = gitCommonDir(project: project), + let gitDir = git(project: project, args: ["rev-parse", "--git-dir"]).map({ + canonicalProjectPath(absoluteGitPath($0, project: project)) + }) + else { return false } + return common != gitDir + } + + /** Preferred subdomain for the project family: committed host without + `.localhost`, else the main worktree path's slug, else this path's slug. */ + public static func preferredSubdomain(project: String, committedHost: String?) -> String { + if let committedHost { + let trimmed = committedHost.hasSuffix(".localhost") + ? String(committedHost.dropLast(".localhost".count)) + : committedHost + let leaf = trimmed.split(separator: ".").last.map(String.init) ?? trimmed + if !leaf.isEmpty { return sanitizeLabel(leaf) } + } + if let main = mainWorktreePath(project: project) { + return ProjectConfigLoader.defaultSlug(project: main) + } + return ProjectConfigLoader.defaultSlug(project: project) + } + + /** Ephemeral host for a linked worktree: `worktree-