diff --git a/CHANGELOG.md b/CHANGELOG.md index cc15846..fd1ee09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), ## [Unreleased] +### Fixed +- Claude Code continuation `StopFailure` hooks now match recoverable model-down failures (`rate_limit`, `overloaded`, `server_error`, `max_output_tokens`, `unknown`), not only rate limits. Auth and billing failures stay excluded. Re-run onboarding to rewrite legacy `rate_limit`-only matchers. + ## [0.7.0] - 2026-08-08 ### Added diff --git a/docs/README.md b/docs/README.md index d117561..3b02cf6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -12,7 +12,7 @@ Task-shaped how-tos for the main flows. - [Private context seeding](guides/private-context-seeding.md) — carrying reviewed preferences into a new repo. - [Targeted brief workflow](guides/targeted-brief-workflow.md) — scoped task briefs vs cross-boundary packets. - [MCP setup](guides/mcp-setup.md) — server flags, available tools, agent instruction packs. -- [Continuation checkpoints](guides/continuation-checkpoints.md) — opt-in Claude task aggregation, compaction fallback, rate-limit recovery, and resume confirmation. +- [Continuation checkpoints](guides/continuation-checkpoints.md) — opt-in Claude task aggregation, compaction fallback, model-down StopFailure recovery, and resume confirmation. ## Reference diff --git a/docs/guides/continuation-checkpoints.md b/docs/guides/continuation-checkpoints.md index 5b9f9b6..20595c9 100644 --- a/docs/guides/continuation-checkpoints.md +++ b/docs/guides/continuation-checkpoints.md @@ -15,7 +15,7 @@ Modes: - `auto`: hook fallbacks plus automatic agent-drafted checkpoints at configured semantic triggers. - `off`: remove only the 3Notch-owned Claude hooks and disable the workflow. -The default hook set uses `SessionStart`, `TaskCreated`, `TaskCompleted`, `PostCompact`, and `StopFailure:rate_limit`. Add `--checkpoint-stop` only if you accept a hook after every Claude response: +The default hook set uses `SessionStart`, `TaskCreated`, `TaskCompleted`, `PostCompact`, and recoverable `StopFailure` handling (config token `StopFailure:rate_limit`). The Claude settings matcher covers `rate_limit`, `overloaded`, `server_error`, `max_output_tokens`, and `unknown` — the model-down cases where a deterministic fallback still helps. Auth and billing failures are excluded. Add `--checkpoint-stop` only if you accept a hook after every Claude response: ```bash notch onboard --yes --mcp claude-code --checkpoints prompt --checkpoint-stop @@ -30,7 +30,7 @@ Claude task hooks accumulate task IDs, subjects, descriptions, and completion st A packet is written only at a recovery boundary: - `PostCompact` uses Claude's documented `compact_summary`. -- `StopFailure:rate_limit` uses accumulated tasks and current Git state. +- Recoverable `StopFailure` events (`rate_limit`, `overloaded`, `server_error`, `max_output_tokens`, `unknown`) use accumulated tasks and current Git state. - Optional `Stop` uses the final assistant message, accumulated tasks, and Git state, and deduplicates unchanged state. Fallback packets include branch, short commit, dirty state, and changed-file names. They never copy file contents or artifacts automatically. Every fallback is marked `unreviewed`, scanned before write, and linked to the previous checkpoint in the same stream with `supersedes`. @@ -60,5 +60,6 @@ At the next session start, 3Notch offers the latest matching checkpoint once. It - `.notch/index/continuation/` is local derived state and gitignored. - Project checkpoints land in `.notch/outbox/` and may appear in Git; choose `private` sensitivity to use ignored `.notch/private/outbox/`. - Private mode intentionally adds `--include-private` to this project's 3Notch MCP server so the agent can list and load a checkpoint after you confirm. That server can read other private packets in the same local store through approved MCP calls. -- A rate-limit fallback cannot recover reasoning that was never reflected in a task, Git state, compact summary, final assistant message, or prior curated checkpoint. +- A model-down StopFailure fallback cannot recover reasoning that was never reflected in a task, Git state, compact summary, final assistant message, or prior curated checkpoint. +- Existing projects with a legacy `StopFailure` matcher of only `rate_limit` should rerun `notch onboard --yes --mcp claude-code --checkpoints ` so Claude Code wires the full recoverable matcher. - 3Notch never commits, pushes, imports, or transports a checkpoint automatically. diff --git a/site/public/docs/guides/continuation-checkpoints/index.html b/site/public/docs/guides/continuation-checkpoints/index.html index a98f8f8..36913a0 100644 --- a/site/public/docs/guides/continuation-checkpoints/index.html +++ b/site/public/docs/guides/continuation-checkpoints/index.html @@ -71,7 +71,7 @@

Continuation checkpoints.

-

Preserve enough selected project state to continue after a rate limit, context compaction, or intentional agent switch. Checkpoints are ordinary local packets—not transcripts or hidden model memory.

+

Preserve enough selected project state to continue after a rate limit, model-down failure, context compaction, or intentional agent switch. Checkpoints are ordinary local packets—not transcripts or hidden model memory.

Enable Claude Code checkpoints

$ notch onboard --yes --mcp claude-code --checkpoints prompt
@@ -82,7 +82,7 @@

What the recovery layer records

Fallbacks include branch, short commit, dirty state, and changed-file names. They do not copy file contents or artifacts automatically.

diff --git a/src/cli/mcp-config-claude-code.ts b/src/cli/mcp-config-claude-code.ts index 52070d6..1ee7d7a 100644 --- a/src/cli/mcp-config-claude-code.ts +++ b/src/cli/mcp-config-claude-code.ts @@ -2,6 +2,10 @@ import { copyFile, readFile, stat } from 'node:fs/promises'; import path from 'node:path'; import { notchMcpServerDefinition } from './mcp-instructions.js'; +import { + isRecoverableStopFailureMatcher, + RECOVERABLE_STOP_FAILURE_MATCHER, +} from '../core/continuation-service.js'; import { atomicWriteFile } from '../core/store-service.js'; import { NotchException } from '../types/errors.js'; import type { ClaudeCodeContinuationEvent, NotchConfig } from '../types/records.js'; @@ -273,7 +277,10 @@ function ownedContinuationEvents(settings: ClaudeCodeSettings): ClaudeCodeContin } function normalizeHookEvent(event: string, matcher: unknown): ClaudeCodeContinuationEvent | undefined { - if (event === 'StopFailure' && matcher === 'rate_limit') { + // Accept the current recoverable matcher, the legacy rate_limit-only matcher, + // and any ordered subset/superset of the recoverable error codes so doctor and + // resync still recognize owned hooks after upgrades. + if (event === 'StopFailure' && isRecoverableStopFailureMatcher(matcher)) { return 'StopFailure:rate_limit'; } @@ -296,7 +303,7 @@ function normalizeHookEvent(event: string, matcher: unknown): ClaudeCodeContinua function hookTarget(event: ClaudeCodeContinuationEvent): { hookEvent: string; matcher?: string } { if (event === 'StopFailure:rate_limit') { - return { hookEvent: 'StopFailure', matcher: 'rate_limit' }; + return { hookEvent: 'StopFailure', matcher: RECOVERABLE_STOP_FAILURE_MATCHER }; } if (event === 'SessionStart') { diff --git a/src/core/continuation-service.ts b/src/core/continuation-service.ts index 82b94e4..a0b12e3 100644 --- a/src/core/continuation-service.ts +++ b/src/core/continuation-service.ts @@ -71,6 +71,47 @@ const supportedEvents = new Set([ 'Stop', ]); +/** + * StopFailure error codes that mean "the model/session ended through no fault of + * the work." Claude Code also emits auth/billing failures; those stay excluded so + * a misconfigured account does not flood the store with recovery packets. + * + * Config still uses the historical event token `StopFailure:rate_limit`; the + * Claude settings matcher and runtime gate cover this full recoverable set. + */ +export const RECOVERABLE_STOP_FAILURE_ERRORS = [ + 'rate_limit', + 'overloaded', + 'server_error', + 'max_output_tokens', + 'unknown', +] as const; + +export type RecoverableStopFailureError = (typeof RECOVERABLE_STOP_FAILURE_ERRORS)[number]; + +export const RECOVERABLE_STOP_FAILURE_MATCHER = RECOVERABLE_STOP_FAILURE_ERRORS.join('|'); + +export function isRecoverableStopFailureError( + error: string | undefined, +): error is RecoverableStopFailureError { + return typeof error === 'string' + && (RECOVERABLE_STOP_FAILURE_ERRORS as readonly string[]).includes(error); +} + +export function isRecoverableStopFailureMatcher(matcher: unknown): boolean { + if (typeof matcher !== 'string' || matcher.length === 0) { + return false; + } + + if (matcher === RECOVERABLE_STOP_FAILURE_MATCHER || matcher === 'rate_limit') { + return true; + } + + const parts = matcher.split('|').filter((part) => part.length > 0); + return parts.length > 0 + && parts.every((part) => (RECOVERABLE_STOP_FAILURE_ERRORS as readonly string[]).includes(part)); +} + export async function runClaudeCodeHook(rawInput: unknown): Promise { try { const input = validateHookInput(rawInput); @@ -174,7 +215,7 @@ function validateHookInput(value: unknown): ClaudeHookInput { function eventEnabled(events: string[], input: ClaudeHookInput): boolean { if (input.hook_event_name === 'StopFailure') { - return input.error === 'rate_limit' && events.includes('StopFailure:rate_limit'); + return isRecoverableStopFailureError(input.error) && events.includes('StopFailure:rate_limit'); } return events.includes(input.hook_event_name); @@ -207,16 +248,17 @@ async function handleEvent( primarySummary: input.compact_summary.trim(), ...(input.trigger ? { trigger: input.trigger } : {}), }); - case 'StopFailure': + case 'StopFailure': { state.interruptedAt = new Date().toISOString(); - if (!hasMaterialRecoveryState(state, git)) { + if (!hasMaterialRecoveryState(state, git) || !isRecoverableStopFailureError(input.error)) { await writeSessionState(context, state); return {}; } return await flushCheckpoint(context, state, git, stream, { - event: 'rate-limit', - primarySummary: 'Claude Code stopped because the API rate limit was reached. This fallback contains only structured task events and repository state captured before the failure.', + event: stopFailureSourceEvent(input.error), + primarySummary: stopFailurePrimarySummary(input.error), }); + } case 'Stop': { const message = input.last_assistant_message?.trim(); if (!message) { @@ -336,12 +378,21 @@ function updateTask( state.activityAt = now; } +type CheckpointSourceEvent = + | 'post-compact' + | 'stop' + | 'rate-limit' + | 'overloaded' + | 'server-error' + | 'max-output' + | 'unknown'; + async function flushCheckpoint( context: LoadedConfig, state: ContinuationSessionState, git: GitSnapshot, stream: string, - source: { event: 'post-compact' | 'rate-limit' | 'stop'; primarySummary: string; trigger?: string }, + source: { event: CheckpointSourceEvent; primarySummary: string; trigger?: string }, ): Promise { return await withCheckpointWriteLock(context, async () => { const fingerprint = recoveryFingerprint(state, git, source); @@ -440,6 +491,27 @@ function continuationTitle(projectName: string, stream: string, event: string): return `Continuation: ${projectName}/${stream} (${event})`; } +function stopFailureSourceEvent(error: RecoverableStopFailureError): CheckpointSourceEvent { + // Avoid embedding secret-scanner trigger words (e.g. "token") in titles/tags. + if (error === 'max_output_tokens') { + return 'max-output'; + } + + return error.replaceAll('_', '-') as CheckpointSourceEvent; +} + +function stopFailurePrimarySummary(error: RecoverableStopFailureError): string { + const reason = { + rate_limit: 'the API rate limit was reached', + overloaded: 'the API reported it was overloaded', + server_error: 'the API returned a server error', + max_output_tokens: 'the model hit the max output length limit', + unknown: 'Claude Code reported an unknown stop failure', + }[error]; + + return `Claude Code stopped because ${reason}. This fallback contains only structured task events and repository state captured before the failure.`; +} + function recoveryFingerprint( state: ContinuationSessionState, git: GitSnapshot, diff --git a/tests/cli/onboard-mcp-claude-code.test.ts b/tests/cli/onboard-mcp-claude-code.test.ts index b559eff..51fb0bf 100644 --- a/tests/cli/onboard-mcp-claude-code.test.ts +++ b/tests/cli/onboard-mcp-claude-code.test.ts @@ -85,7 +85,7 @@ describe('notch onboard --mcp claude-code', () => { ])); expect(settings.hooks.StopFailure).toEqual([ { - matcher: 'rate_limit', + matcher: 'rate_limit|overloaded|server_error|max_output_tokens|unknown', hooks: [{ type: 'command', command: 'notch hook claude-code' }], }, ]); @@ -159,6 +159,41 @@ describe('notch onboard --mcp claude-code', () => { }); }); + it('rewrites a legacy rate_limit-only StopFailure matcher to the recoverable set on resync', async () => { + await withTempProject({}, async (project) => { + const first = await runCli([ + 'onboard', '--yes', '--mcp', 'claude-code', '--checkpoints', 'script', + ], { cwd: project.path }); + expect(first.exitCode).toBe(0); + + const settingsPath = path.join(project.path, '.claude/settings.local.json'); + const settings = JSON.parse(await readFile(settingsPath, 'utf8')) as { + hooks: Record; matcher?: string }>>; + }; + settings.hooks.StopFailure = [{ + matcher: 'rate_limit', + hooks: [{ type: 'command', command: 'notch hook claude-code' }], + }]; + await writeFile(settingsPath, `${JSON.stringify(settings, null, 2)}\n`, 'utf8'); + + const resynced = await runCli([ + 'onboard', '--yes', '--mcp', 'claude-code', '--checkpoints', 'script', + ], { cwd: project.path }); + expect(resynced.exitCode).toBe(0); + expect(resynced.stdout).toContain('Configured Claude Code continuation hooks'); + + const next = JSON.parse(await readFile(settingsPath, 'utf8')) as { + hooks: Record; matcher?: string }>>; + }; + expect(next.hooks.StopFailure).toEqual([ + { + matcher: 'rate_limit|overloaded|server_error|max_output_tokens|unknown', + hooks: [{ type: 'command', command: 'notch hook claude-code' }], + }, + ]); + }); + }); + it('rejects malformed Claude settings without rewriting them', async () => { await withTempProject({}, async (project) => { const settingsPath = path.join(project.path, '.claude/settings.local.json'); diff --git a/tests/e2e/continuation-checkpoints-smoke.test.ts b/tests/e2e/continuation-checkpoints-smoke.test.ts index f43b7a3..19fbf37 100644 --- a/tests/e2e/continuation-checkpoints-smoke.test.ts +++ b/tests/e2e/continuation-checkpoints-smoke.test.ts @@ -24,10 +24,15 @@ describe('continuation checkpoint e2e', () => { const settings = JSON.parse(await readFile( path.join(project.path, '.claude/settings.local.json'), 'utf8', - )) as { hooks: Record }; + )) as { hooks: Record> }; expect(Object.keys(settings.hooks)).toEqual(expect.arrayContaining([ 'SessionStart', 'TaskCreated', 'TaskCompleted', 'PostCompact', 'StopFailure', ])); + expect(settings.hooks.StopFailure).toEqual(expect.arrayContaining([ + expect.objectContaining({ + matcher: 'rate_limit|overloaded|server_error|max_output_tokens|unknown', + }), + ])); await execFileAsync('git', ['add', '.'], { cwd: project.path }); const trackedLocalSettings = await execFileAsync( diff --git a/tests/unit/continuation-service.test.ts b/tests/unit/continuation-service.test.ts index a2ad070..41a64f2 100644 --- a/tests/unit/continuation-service.test.ts +++ b/tests/unit/continuation-service.test.ts @@ -108,6 +108,98 @@ describe('continuation service', () => { expect(packets).toHaveLength(1); expect(packets[0]?.packet.tags).toContain('source-rate-limit'); expect(packets[0]?.packet.summary).toContain('Finish verification'); + expect(packets[0]?.packet.summary).toContain('API rate limit was reached'); + }); + }); + + it('writes model-down StopFailure fallbacks for overloaded, server_error, max_output_tokens, and unknown', async () => { + await withContinuationProject(async (projectPath) => { + const cases = [ + { + session_id: 'stop-overloaded', + error: 'overloaded', + tag: 'source-overloaded', + phrase: 'API reported it was overloaded', + subject: 'Finish overloaded recovery', + }, + { + session_id: 'stop-server-error', + error: 'server_error', + tag: 'source-server-error', + phrase: 'API returned a server error', + subject: 'Finish server-error recovery', + }, + { + session_id: 'stop-max-output', + error: 'max_output_tokens', + tag: 'source-max-output', + phrase: 'max output length limit', + subject: 'Finish max-output recovery', + }, + { + session_id: 'stop-unknown', + error: 'unknown', + tag: 'source-unknown', + phrase: 'unknown stop failure', + subject: 'Finish unknown recovery', + }, + ] as const; + + for (const testCase of cases) { + await runClaudeCodeHook(taskEvent( + projectPath, + 'TaskCreated', + `task-${testCase.error}`, + testCase.subject, + testCase.session_id, + )); + const recovery = await runClaudeCodeHook({ + session_id: testCase.session_id, + cwd: projectPath, + hook_event_name: 'StopFailure', + error: testCase.error, + last_assistant_message: `API Error: ${testCase.error}`, + transcript_path: '/does/not/exist', + }); + expect(recovery.systemMessage).toContain('wrote unreviewed continuation'); + } + + const context = await loadConfig({ cwd: projectPath }); + const packets = await listPackets(context, { direction: 'outbox' }); + expect(packets).toHaveLength(cases.length); + + for (const testCase of cases) { + const packet = packets.find((entry) => entry.packet.tags.includes(testCase.tag)); + expect(packet, `missing packet for ${testCase.error}`).toBeDefined(); + expect(packet?.packet.summary).toContain(testCase.phrase); + expect(packet?.packet.summary).toContain(testCase.subject); + } + }); + }); + + it('skips non-recoverable StopFailure errors such as authentication and billing failures', async () => { + await withContinuationProject(async (projectPath) => { + for (const error of ['authentication_failed', 'billing_error'] as const) { + await runClaudeCodeHook(taskEvent( + projectPath, + 'TaskCreated', + `task-${error}`, + `Work before ${error}`, + `session-${error}`, + )); + const result = await runClaudeCodeHook({ + session_id: `session-${error}`, + cwd: projectPath, + hook_event_name: 'StopFailure', + error, + last_assistant_message: `API Error: ${error}`, + transcript_path: '/does/not/exist', + }); + expect(result).toEqual({}); + } + + const context = await loadConfig({ cwd: projectPath }); + expect(await listPackets(context, { direction: 'outbox' })).toEqual([]); }); });