Recover interrupted store migrations with production crash evidence

CHANGELOG.md

@@ -10,6 +10,24 @@ after its public API and format compatibility policies are established.
 
 ### Added
 
+- Recovery receipts retain the exact observed namespace prefix and bound
+  migration intent digest, including observations of completed migration.
+
+- Recovery jointly verifies retained stages and canonical records during
+  adoption, refusing substituted canonical inodes before forward execution.
+
+- Late-stage refusals report receipt effects as receipts when no marker
+  artifact exists.
+
+- Migration recovery revalidates current writer authority before observing
+  residue, preventing stale expected intents from admitting a corrupt store.
+
+- README recovery and reader-fence gaps point to their current open owners.
+
+- Migration recovery planning, bounded filesystem residue admission, explicit
+  truncated-stage discard, and receipts listing resumed phases. The production
+  crash matrix now includes 68 migration process-death cases (Refs #108).
+
 - `FilesystemRetentionPublicationAuthority` executes the 17 ordered retention
   publication phases against a completely migrated version-2 root. It stages
   `root.next`, `manifest.next`, and `head.next` exclusively, verifies device
@@ -606,6 +624,20 @@ after its public API and format compatibility policies are established.
 
 ### Fixed
 
+- The migration transition-ledger guard rejects noncanonical line endings,
+  extra rows, misplaced discard claims, and counterfeit completion postures.
+  Recovery posture and namespace interruption checks use their exact columns.
+- The independent migration restart model propagates missing occurrence,
+  arithmetic, and extent-conversion failures instead of normalizing them.
+- Crash-case admission refuses out-of-range migration namespace occurrences
+  before child execution; variable segment-record occurrences remain valid.
+- Receipt-only migration residue before a complete namespace reports
+  `ReceiptBeforeMarker`, preserving the exact missing-prerequisite boundary.
+- Migration residue observation rejects non-regular entries with the typed
+  kind refusal before opening them, retaining the post-open kind recheck.
+- Migration resumption is internal to verified recovery admission; external
+  callers cannot bypass current authority verification and residue adoption.
+
 - Version-two reopen compares persisted device and inode coordinates while
   retaining mount identity as same-process migration evidence (#97).
   Simulated-remount reopen and exact moved-root refusals are regression-tested;

README.md

@@ -67,19 +67,28 @@ Keep is required to refuse all three, before mutating anything.
   files, replaced protocol directories, and every namespace or capacity
   violation before it writes anything. Each refusal is a typed value, not a
   string.
+- **Migration restart recovery.** Recovery verifies current authority,
+  classifies the observed prefix, and resumes an exact persisted migration
+  intent through the remaining phases. An incomplete pre-effect intent stage
+  is discarded and rebuilt from freshly verified current intent. Its crash
+  matrix kills real writer processes
+  at 68 before/during/after coordinates (`KEEP-CRASH-053`–`073`), preserving
+  every version-1 byte. Broader hostile restart combinations remain in #111.
 
 ## What it does not do yet
 
-Version 2 writes correctly from a clean start and, if it finds the residue of
-an interrupted publication, refuses rather than guesses. Nothing yet recovers
-that residue, and readers have no fence, so **an interrupted version-2
-publication waits for a human until #19 lands.** A version-1 store stays
-admitted until its owner migrates it; migrate only if you accept that wait.
+Version-2 retention publication writes from a clean start and refuses the
+residue of an interrupted retention publication. Retention publication recovery
+and the reader snapshot fence await integration from PR #99, so **an
+interrupted retention publication waits for explicit recovery.** Migration
+restart recovery is implemented and does not grant retention authority. A
+version-1 store stays admitted until its owner migrates it; migrate only if
+you accept that wait.
 
 | Gap | Tracked |
 | --- | --- |
-| Restart recovery for retention publication and migration | [#19](https://github.com/flyingrobots/keep/issues/19) |
-| Reader fence binding one consistent catalog + retention snapshot | [#19](https://github.com/flyingrobots/keep/issues/19) |
+| Retention publication restart recovery and broader migration corruption coverage | [PR #99](https://github.com/flyingrobots/keep/pull/99), [#111](https://github.com/flyingrobots/keep/issues/111) |
+| Reader fence binding one consistent catalog + retention snapshot | [PR #99](https://github.com/flyingrobots/keep/pull/99) |
 | Precise verification reports at explicit depths | [#20](https://github.com/flyingrobots/keep/issues/20) |
 | Garbage collection and identity-preserving compaction | [#21](https://github.com/flyingrobots/keep/issues/21) |
 | Bounded production ingestion through the durable store | [#82](https://github.com/flyingrobots/keep/issues/82) |

conformance/segment-store/v2/ORIGIN.md

@@ -6,6 +6,13 @@ The corpus was constructed on 2026-07-29 with:
 - `cargo 1.96.0 (30a34c682 2026-05-25)`; and
 - `b3sum 1.8.5`.
 
+`transitions.tsv` was added on 2026-09-30 by transcribing the 21 boundaries of
+`StoreMigrationPhase::ALL` and the recovery table in
+`docs/formats/segment-store-v2/migration-recovery.md`. No independent oracle
+constructs this ledger. Its committed-byte guard is `transition_laws.rs`
+under `xtask/tests/retention_store_v2_protocol_contract/`. The crash matrix
+checks runtime behavior at the same boundaries without reading the TSV.
+
 ## Independent inputs
 
 The oracle imports exact bytes only from these previously accepted fixtures:
@@ -60,6 +67,9 @@ Exact output:
 A temporary ignored Rust test wrote the initially reviewed TSV and hexadecimal
 artifacts from the handwritten oracle. That write path was removed immediately
 after materialization. The committed oracle is read-only and rejects drift.
+This construction claim covers the original format tables and hexadecimal
+records; the later handwritten `transitions.tsv` has the separate verification
+boundary described above.
 
 Changing any fixture requires a deliberate specification change, an updated
 definition or profile digest when affected, fresh independent construction,

conformance/segment-store/v2/README.md

@@ -14,6 +14,7 @@ migration, retention transition, or garbage collector exists.
 | `inventory.tsv` | Canonical one-segment, one-catalog migration inventory |
 | `migration-source.tsv` | Exact version-1 and derived migration coordinates |
 | `artifacts.tsv` | Golden artifact lengths, digests, checksums, and filenames |
+| `transitions.tsv` | One stable crash identifier per migration boundary, `KEEP-CRASH-053` to `-073` |
 | `format-marker.hex` | Canonical 96-byte `FORMAT` record |
 | `migration-intent.hex` | Canonical 256-byte migration intent |
 | `migration-receipt.hex` | Canonical 256-byte migration receipt |
@@ -55,20 +56,46 @@ The retention fixture uses namespace bytes `00 2f ff`, proving the namespace is
 opaque and not a path or Unicode string. Its one anchor combines the canonical
 one-zero `BlobId` and `LayoutId` values from the existing layout corpus.
 
+## Transition protocol
+
+`transitions.tsv` mirrors the version-1 table: one row per migration
+durability operation with its pre-state, interrupted-state classification,
+post-state, and recovery posture. `KEEP-CRASH-053`, `-062`, and `-068` are
+the only rows whose interruption may leave an incomplete pre-effect stage and
+therefore the only rows that plan a discard; `-072` and `-073` admit the
+complete migration.
+
+The `cargo xtask durability-crash-matrix --sequence migration` harness
+executes 68 canonical process-death cases at the same 21 boundaries: each at
+three positions plus one `during` case per admitted directory-prefix length
+for `KEEP-CRASH-060`. It kills an isolated writer process group, compares the
+restarted root against an independent expected-state model, requires the
+production planner to report the predicted recovery plan, and requires the
+recovered store to be one complete migration with every version-1 byte
+intact. Host power loss remains outside its claim.
+
 ## Verification
 
 Run:
 
 ```bash
 cargo test --manifest-path xtask/Cargo.toml \
   --test retention_store_v2_format_oracle
+cargo test --manifest-path xtask/Cargo.toml \
+  --test retention_store_v2_protocol_contract transition_laws
 ```
 
 The test-only oracle constructs every record from handwritten offsets and
 domain preimages, compares exact fixture bytes and tables, and imports no
 production version-2 codec. The repository protocol and documentation gates
 route this corpus separately.
 
-Passing this corpus is necessary but insufficient for issue #19. Production
-code still needs parser, corruption, property, model, crash, recovery,
+`transition_laws.rs` checks the handwritten transition ledger's committed
+shape, operation order, and recovery-posture claims. The format oracle does
+not construct or compare `transitions.tsv`; the crash matrix checks runtime
+behavior without reading its bytes.
+
+Passing this corpus is necessary but insufficient for migration recovery
+(#108), restart corruption (#111), or compatibility and fuzz coverage (#112).
+Production code still needs parser, corruption, property, model, crash, recovery,
 concurrency, fuzz, and public API evidence.

conformance/segment-store/v2/transitions.tsv

@@ -0,0 +1,23 @@
+keep.segment-store.transitions/v2
+crash_id	phase	operation	pre_state	interrupted_class	post_state	recovery_posture
+KEEP-CRASH-053	migration	write-intent-stage	admitted-version-one-store	absent-or-incomplete-intent-stage	complete-intent-stage	discard-incomplete-stage-or-resume-stage-sync
+KEEP-CRASH-054	migration	sync-intent-stage	complete-intent-stage	complete-intent-stage	durable-intent-stage	resume-stage-sync
+KEEP-CRASH-055	migration	link-intent	durable-intent-stage	durable-intent-stage-or-linked-intent	linked-intent	verify-no-clobber-link-and-resume-root-sync
+KEEP-CRASH-056	migration	sync-root-after-intent	linked-intent	linked-intent	durable-intent	resume-root-sync
+KEEP-CRASH-057	migration	remove-intent-stage	durable-intent	durable-intent-with-or-without-stage	durable-intent-alone	resume-cleanup-sync
+KEEP-CRASH-058	migration	sync-root-after-intent-cleanup	durable-intent-alone	durable-intent-alone	intent-cleanup-synchronized	resume-cleanup-sync
+KEEP-CRASH-059	migration	admit-reader-fence	intent-cleanup-synchronized	intent-with-or-without-reader-fence	reader-fence-admitted	resume-namespace-prefix
+KEEP-CRASH-060	migration	admit-namespace-prefix	reader-fence-admitted	directory-prefix-length-zero-to-six	namespace-prefix-admitted	resume-namespace-prefix-or-root-sync
+KEEP-CRASH-061	migration	sync-root-after-namespace	namespace-prefix-admitted	namespace-prefix-admitted	durable-namespace-prefix	resume-root-sync
+KEEP-CRASH-062	migration	write-marker-stage	durable-namespace-prefix	absent-or-incomplete-marker-stage	complete-marker-stage	discard-incomplete-stage-or-resume-stage-sync
+KEEP-CRASH-063	migration	sync-marker-stage	complete-marker-stage	complete-marker-stage	durable-marker-stage	resume-stage-sync
+KEEP-CRASH-064	migration	link-marker	durable-marker-stage	durable-marker-stage-or-linked-marker	linked-marker	verify-no-clobber-link-and-resume-root-sync
+KEEP-CRASH-065	migration	sync-root-after-marker	linked-marker	linked-marker	durable-marker	resume-root-sync
+KEEP-CRASH-066	migration	remove-marker-stage	durable-marker	durable-marker-with-or-without-stage	durable-marker-alone	resume-cleanup-sync
+KEEP-CRASH-067	migration	sync-root-after-marker-cleanup	durable-marker-alone	durable-marker-alone	marker-cleanup-synchronized	resume-cleanup-sync
+KEEP-CRASH-068	migration	write-receipt-stage	marker-cleanup-synchronized	absent-or-incomplete-receipt-stage	complete-receipt-stage	discard-incomplete-stage-or-resume-stage-sync
+KEEP-CRASH-069	migration	sync-receipt-stage	complete-receipt-stage	complete-receipt-stage	durable-receipt-stage	resume-stage-sync
+KEEP-CRASH-070	migration	link-receipt	durable-receipt-stage	durable-receipt-stage-or-linked-receipt	linked-receipt	verify-no-clobber-link-and-resume-root-sync
+KEEP-CRASH-071	migration	sync-root-after-receipt	linked-receipt	linked-receipt	durable-receipt	resume-root-sync
+KEEP-CRASH-072	migration	remove-receipt-stage	durable-receipt	durable-receipt-with-or-without-stage	complete-migration	admit-complete-migration
+KEEP-CRASH-073	migration	sync-root-after-receipt-cleanup	complete-migration	complete-migration	durable-complete-migration	admit-complete-migration

docs/formats/segment-store-v2/README.md

@@ -8,11 +8,12 @@ namespaces.
 
 ADR-0009 owns the cross-cutting retention and liveness decision. These pages
 own its durable representation. The one-way migration, version-two reopen, and
-forward retention publication are implemented with executable evidence;
-recovery of retained retention stages, reader fencing, and collection remain
-planned in issue #19, and the [requirements ledger](requirements.md) records
-exactly which requirements are proven. A version-1 store remains admitted until
-its owner migrates it.
+forward retention publication, partial-prefix migration recovery, and the
+68-case migration process-death matrix are implemented with executable
+evidence. Retention recovery and reader fencing await integration from PR #99;
+collection remains planned in #21. The [requirements ledger](requirements.md)
+records exactly which requirements are proven. A version-1 store remains
+admitted until its owner migrates it.
 
 ## Core laws
 
@@ -95,9 +96,12 @@ stages, replaced protocol directories, and every namespace or capacity
 violation before mutation, each as a typed `RetentionCurrentStateRefusal`.
 
 Not implemented: retention publication recovery and `KEEP-CRASH-036..052`
-process-death evidence, partial-prefix migration recovery and
-`KEEP-CRASH-053..073`, the reader fence, model-based transition evidence, and
-garbage collection. Issue #19 owns the first four and issue #21 the last.
+process-death evidence, the reader fence, model-based transition evidence, and
+garbage collection. Partial-prefix migration recovery and the 68-case
+`KEEP-CRASH-053..073` process-death matrix are implemented. Broader migration
+restart corruption and compatibility coverage remain in #111 and #112.
+PR #99 contains retention recovery and reader fencing awaiting integration;
+issue #21 owns garbage collection.
 Reopen compares only the restart-stable root coordinates, device and inode,
 against the intent; see
 [root identity across restart](recovery.md#root-identity-across-restart). A

docs/formats/segment-store-v2/migration-crash.md

@@ -24,15 +24,17 @@ For each pair, migration:
 
 The verified stage is linked without replacement. The canonical target is
 immutable. Recovery never truncates, replaces, or repairs it. An exact stage
-with an absent target resumes at the link. Exact stage and target bytes resume
-at the required synchronization or cleanup. Different bytes, a substituted
-inode, a link, or a wrong file kind refuse.
+with an absent target resumes at stage synchronization. Exact stage and target
+bytes resume at the required synchronization or cleanup. Different bytes, a
+substituted inode, a link, or a wrong file kind refuse.
 
 A pre-effect incomplete stage may be removed only when its canonical target and
 every later-ordered migration effect are absent and every earlier effect admits
-exactly. Recovery pins the stage, removes it, synchronizes the store root, and
-returns a typed discard report. Any later effect makes incomplete or corrupt
-stage bytes unrecoverable ambiguity.
+exactly. Migration recovery revalidates the present stage's regular kind and
+strictly incomplete length immediately before removal, removes it, synchronizes
+the store root, and returns a typed discard report. It does not retain an
+incomplete-stage handle. Any later effect makes incomplete or corrupt stage
+bytes unrecoverable ambiguity.
 
 The fixed stage is not authority. `migration.intent` becomes migration
 authority only after its canonical link and store-root synchronization.
@@ -103,8 +105,18 @@ prefix length. Restart must classify exact stages, canonical targets, namespace
 prefix, marker, receipt, and cleanup state without depending on a clock,
 filesystem iteration order, or file existence alone.
 
+Namespace occurrence coordinates are zero-based: `during` accepts zero
+through five, while `before` and `after` accept only zero. Invalid coordinates
+refuse at crash-case admission before a child starts. Segment record
+occurrences remain dependent on the write's record count.
+
 `StoreMigrationPhase::ALL` freezes the 21 boundaries above in exact order.
-Fresh writer-locked filesystem execution now implements that exact order and
-has deterministic in-process storage-fault and corruption laws. The
-before/during/after process-death matrix and restart classifier remain
-unimplemented; this page does not yet claim crash recovery.
+The production recovery planner and filesystem adapter execute the lawful
+remaining suffix. `cargo xtask durability-crash-matrix --sequence migration`
+runs 68 process-death cases: before/during/after each boundary, with all six
+directory-prefix occurrences at `KEEP-CRASH-060`. The existing CI matrix
+runs these cases in debug and release builds.
+
+These cases establish recovery after process death; they do not simulate power
+loss. Broader restart corruption and compatibility/fuzz coverage remain tracked
+by #111 and #112.

docs/formats/segment-store-v2/migration-recovery.md

@@ -58,10 +58,48 @@ The migration recovery boundary admits only these ordered prefixes:
 
 <!-- markdownlint-enable MD013 -->
 
+Every row also requires the admission checks in
+[Executable recovery boundary](#executable-recovery-boundary). An intent stage
+surviving a namespace effect, or a marker stage surviving a receipt effect,
+refuses as `StageAfterEffect`. When a stage and canonical target both exist,
+exact bytes and one shared device/inode identity are required before resumption.
+
 A partial migration retry revalidates intent and existing bytes, resumes at the
 first absent canonical step, and never replaces an entry. A missing predecessor,
 changed version-1 coordinate, out-of-order name, wrong kind or bytes, conflicting
 receipt, unknown entry, or changed root identity is unrecoverable ambiguity.
 
 Death before durable intent leaves v1 plus at most its non-authoritative stage.
 Death after durable intent leaves recovery-required v2 migration state.
+
+## Executable recovery boundary
+
+`FilesystemStoreMigrationAuthority::reopen_for_recovery` reacquires writer
+authority without granting version-1 publication admission.
+`recover_store_migration` revalidates the caller's freshly derived current
+intent, then observes bounded residue and applies
+`plan_store_migration_recovery` before adopting records or discarding stages.
+Nested directory membership is checked before mutating recovery. An intent
+stage surviving namespace creation, or a marker stage surviving receipt
+publication, is refused as `StageAfterEffect`.
+
+When both a stage and its canonical target exist, adoption verifies they share
+the same device and inode as well as exact bytes before any forward phase,
+including directory synchronization.
+
+The receipt retains the exact observed namespace prefix and bound intent
+digest, names the admitted plan, and lists the executed forward phases
+through `executed_phases()`. The plan records the earliest unproven boundary;
+it does not infer which synchronization calls completed before process death.
+For `VersionOne`, `intent_digest()` returns `None`: no migration intent was
+admitted. Every other plan reports the intent used by recovery. An incomplete
+pre-effect intent stage has no complete persisted intent, so its discard path
+uses the freshly verified current intent.
+Restart compares device and inode identity; mount identity is same-process
+evidence. Whenever an exact intent survives, recovery continues with its
+persisted bytes.
+
+The filesystem laws cover every forward prefix, every strict byte-prefix
+truncation of all three stages, unchanged version-1 bytes, and refusal before
+mutation for corrupt intent and unexpected nested residue. The complete
+restart corruption matrix remains tracked separately in #111.