Reconcile lost SDK controllers with durable stop and process exit evidence

PROJECT.md

@@ -1,5 +1,7 @@
 # FlyBot — Project State
 
+#2483 controller-loss recovery joins a resource-side settled stop receipt with exact process exit, or existing durable host exit evidence. Reconcile worker and lease together; physical exit grants no Task completion or next-role decision.
+
 #2483 restart recovery notifies the resident Chief through existing durable inspection requests. Current exit/retirement facts wake judgement; Chief chooses recovery and unknown exits remain held. Endpoint tests alone do not prove this notification chain.
 
 #2483: Developer Git mutation rights follow its bound isolated resource clone, independently of App credentials. Shared host Git metadata and read-only role policies keep their existing boundaries.
@@ -28,46 +30,17 @@ architecture lives in [docs/architecture.md](docs/architecture.md) and
 Current stage and priorities: [ROADMAP.md](ROADMAP.md). How decisions are made:
 [docs/governance.md](docs/governance.md). Dated decisions: [decisions/](decisions/README.md).
 
-## SDK authentication — 2026-11-04
-
-#2398 binds readiness and setup to the existing resource profile, preserving its
-paths and Task-role context. Known SDK authentication failures reach authRequired
-without becoming code verdicts. No OAuth broker, credential copying or live login.
-
-## Role App delivery — 2026-11-04
-
-#2365 opts Project roles into explicit App connections and dedicated short-lived
-credential helpers. Original candidate Effects observe SDK commits/PRs without
-resealing; actual App Review and host merge/Chief acceptance stay separate.
-No Owner fallback or live migration. [Decision](decisions/2026-11-04-role-github-apps.md)
-and [operator contract](docs/role-github-apps.md); live App acceptance requires authorized installations.
-
-## Follow-up reliability delivery — 2026-11-04
-
-Owner authorizes sequential #2336/#2337/#2334/#2346/#2338/#2347/#2331/#2342
-implementation, one PR each, tests/memory/docs and validated merge to the delivery
-branch. #2336 establishes owned reservation rollback and a durable pre-spawn fence;
-unknown execution, workspace/context and current authority remain protected.
-#2337 validates the actual SDK-resolved CLI and role profile before inference;
-configuration readiness is separate from model/test/exit evidence.
-#2334 separates executor dependency readiness from host test outcomes and preserves
-same-candidate retry evidence; remote unknown exit retains preparation occupancy.
-#2346 admits cwd-only Task-role restoration with current binding/exit proof and
-explicit resumed/reconstructed identity; clearing bindings never resets context.
-No live task/configuration/credential mutation or deployment is implied.
-
-## Execution recovery — 2026-11-04
-
-#2325 aligns actual reused SDK/resource checkouts with pinned baseline/seed facts,
-preserves drafts and admits explicit inspect/integrate inputs through existing
-Effects. Chief selects only approved repository sources. #2327 adds bounded, durable Chief protocol correction in the resident Session;
-exhaustion retains incomplete judgement without automatic escalation. Follow-ups:
-#2328 adds confirmed-idle wait suspension and FIFO continuation re-admission
-through the original queue; exit-unknown capacity remains held. #2330 adds durable scheduled inspection reconciliation, configured deadlines,
-exit-proven bounded recovery and health visibility; independent Task refusals do not
-abort other decisions. No automatic business escalation or live rollout. See
-[delivery contract](docs/subsystems/project-delivery.md). Live Task mutation and
-deployment remain separate from source/test delivery.
+## Runtime and recovery contracts
+
+Earlier implementation checkpoints are [preserved verbatim](docs/history/2026-11-05-controller-exit-memory-checkpoints.md).
+Current resource profile identity, Task-role context, CLI/configuration validation and
+owned pre-spawn reservations remain binding. Unknown worker/preparation exit keeps
+occupancy; dependency readiness and exact-candidate test evidence remain separate.
+Apps require explicit installations and scoped credentials with no Owner fallback;
+SDK candidate/PR adoption, App Review, host merge and Chief acceptance remain distinct.
+Recovery/correction reuses original queues and schedules, preserves drafts and current
+rights, and never chooses automatic business escalation. All canonical subsystem
+contracts remain active. No live task/config/credential change or deployment is implied.
 
 ## Established backend foundations
 

ROADMAP.md

@@ -1,5 +1,7 @@
 # FlyBot — Roadmap
 
+#2483 controller loss: verify local/Docker worker shutdown and same-generation lease reconciliation; legacy records need actual saved exit evidence. Explicit Chief baseline choice remains the next independent stage.
+
 #2483 restart: cover saved state reload through Chief choice and actual pool re-dispatch, including historical blocked claims, preserved drafts and exit uncertainty. No fixed recovery role, automatic Task acceptance or live deployment.
 
 #2483 resource Git parity: verify current Project/Attempt/resource/cwd and private Git metadata before granting Dev its clone `.git`; validate actual SDK mutations and read-only denial separately from live resource acceptance.

changelog.d/2483-controller-exit.fixed.md

@@ -0,0 +1 @@
+Reconcile lost SDK controllers using verified resource-side stop/process evidence and repair partial worker/lease releases while preserving unknown-exit fences.

core/src/host/project-harness-worker.mjs

@@ -6,16 +6,22 @@ import { readFileSync, existsSync } from 'node:fs'
 import { dirname, join } from 'node:path'
 import { fileURLToPath } from 'node:url'
 import { runHarnessTurn } from './project-harness-turn.js'
+import { registerSdkExit, acknowledgeSdkStop, SDK_EXIT_VERSION } from './project-sdk-exit.mjs'
+const exitIdentity = process.argv[2] ? JSON.parse(process.argv[2]) : null
+if (exitIdentity) await registerSdkExit(exitIdentity)
 const input = createInterface({ input: process.stdin, crlfDelay: Infinity })
 let githubRuntime
-const write = value => process.stdout.write(JSON.stringify(githubRuntime ? githubRuntime.redact(value) : value) + '\n')
+let outputClosed = false
+const write = value => { if (!outputClosed) process.stdout.write(JSON.stringify(githubRuntime ? githubRuntime.redact(value) : value) + '\n') }
+process.stdout.on('error', () => { outputClosed = true; void shutdown() })
 const answers = new Map(), credentials = new Map()
 let controller, sessionId, settings, busy = false, shuttingDown = false
 const completedTurns = new Map()
 const permission = request => new Promise(resolve => {
   const id = randomUUID(); answers.set(id, resolve); write({ type: 'permission', id, request })
 })
 async function turn(message) {
+  if (shuttingDown) return
   if (message.turnId && completedTurns.has(message.turnId)) { write(completedTurns.get(message.turnId)); return }
   if (busy) { write({ type: 'failure', reason: 'SDK Session is already answering' }); return }
   busy = true; controller = new AbortController()
@@ -58,10 +64,12 @@ async function shutdown() {
   for (const resolve of answers.values()) resolve({ allow: false })
   for (let n = 0; busy && n < 100; n++) await new Promise(resolve => setTimeout(resolve, 50))
   if (busy) return // No exit claim while the SDK has not settled; host retains capacity.
+  if (exitIdentity) await acknowledgeSdkStop(exitIdentity)
   write({ type: 'stopped', released: true })
   input.close(); process.stdin.pause(); setTimeout(() => process.exit(0), 50)
 }
 input.on('close', () => { void shutdown() })
+process.on('SIGTERM', () => { void shutdown() })
 function sdkVersion(name) {
   try {
     let directory = dirname(fileURLToPath(import.meta.resolve(name)))
@@ -73,4 +81,4 @@ function sdkVersion(name) {
   } catch { /* Missing SDK is an explicit failed handshake, not a simulated version. */ }
   return null
 }
-write({ type: 'ready', roleGithub: 'role-github/v1', receiptCorrection: 'input-correction/v1', pid: process.pid, sdk: { codex: sdkVersion('@openai/codex-sdk'), 'claude-agent': sdkVersion('@anthropic-ai/claude-agent-sdk') } })
+write({ type: 'ready', roleGithub: 'role-github/v1', receiptCorrection: 'input-correction/v1', exitRecovery: exitIdentity ? SDK_EXIT_VERSION : null, pid: process.pid, sdk: { codex: sdkVersion('@openai/codex-sdk'), 'claude-agent': sdkVersion('@anthropic-ai/claude-agent-sdk') } })