Retention recovery, crash-matrix evidence, reader fence, and model-based transitions (item 6)

.github/workflows/ci.yml

@@ -25,6 +25,12 @@ jobs:
       - name: Install pinned toolchain
         run: rustup show
 
+      - name: Install Linux syscall observation tool
+        run: |
+          sudo apt-get update
+          sudo apt-get install --yes strace
+          strace --version
+
       - name: Install independent vector tool
         uses: taiki-e/install-action@41049aa56687c35e0afa74eed4f09cec4f9afabf # v2.85.2
         with:

README.md

@@ -51,10 +51,10 @@ Keep is required to refuse all three, before mutating anything.
   generation-versioned catalogs, and a fixed-width `HEAD` are published
   through an ordered protocol whose every step is a named crash point.
   Platform admission is Linux ext4, non-casefolded, one writer.
-- **Proven restart recovery for version 1.** The crash matrix kills real
-  writer processes at 105 before/during/after coordinates
-  (`KEEP-CRASH-001`–`035`) and verifies the store lands in exactly one
-  documented lawful state each time.
+- **Process-death recovery evidence.** The crash matrix kills real writer processes
+  at 156 before/during/after coordinates (`KEEP-CRASH-001`–`052`) and
+  verifies the store lands in exactly one documented lawful state each time,
+  for version-1 publication and for version-2 retention publication.
 - **Version-2 retention and migration, forward path.** Explicit retention
   roots, deterministic closure verification, a one-way 21-phase migration,
   and a 17-phase retention publication — all with production filesystem
@@ -77,18 +77,22 @@ Keep is required to refuse all three, before mutating anything.
 
 ## What it does not do yet
 
-Version-2 retention publication writes from a clean start and refuses the
-residue of an interrupted retention publication. Retention publication recovery
-and the reader snapshot fence await integration from PR #99, so **an
-interrupted retention publication waits for explicit recovery.** Migration
-restart recovery is implemented and does not grant retention authority. A
-version-1 store stays admitted until its owner migrates it; migrate only if
-you accept that wait.
+Version-2 retention recovery, fenced reader snapshots and model-based transitions are implemented in this branch but still require the correctness corrections and independent acceptance review tracked in PR #99.
+
+The retention process-death sequence checks the recovered head generation and exact selected-root bytes before retry; its [evidence receipt](docs/testing-evidence/retention-crash-reader-oracle.md) bounds that claim to the declared initial-publication crash coordinates.
+
+Incomplete retention stages are preserved and block publication pending explicit disposition; automatic disposal is deferred. The [landing ledger](docs/testing-evidence/retention-landing.md) tracks execution-failure reporting and final acceptance under the [approved recovery contract](docs/formats/segment-store-v2/retention-recovery.md).
+
+Complete orphans remain recovery-protected until explicit disposition lands with garbage collection (#21).
+
+Migration restart recovery is implemented and does not grant retention authority.
+
+A version-1 store stays admitted until its owner migrates it.
 
 | Gap | Tracked |
 | --- | --- |
-| Retention publication restart recovery and broader migration corruption coverage | [PR #99](https://github.com/flyingrobots/keep/pull/99), [#111](https://github.com/flyingrobots/keep/issues/111) |
-| Reader fence binding one consistent catalog + retention snapshot | [PR #99](https://github.com/flyingrobots/keep/pull/99) |
+| Retention recovery correctness remediation and broader migration corruption coverage | [PR #99](https://github.com/flyingrobots/keep/pull/99), [#111](https://github.com/flyingrobots/keep/issues/111) |
+| Fenced reader correctness remediation and independent acceptance | [PR #99](https://github.com/flyingrobots/keep/pull/99) |
 | Precise verification reports at explicit depths | [#20](https://github.com/flyingrobots/keep/issues/20) |
 | Garbage collection and identity-preserving compaction | [#21](https://github.com/flyingrobots/keep/issues/21) |
 | Bounded production ingestion through the durable store | [#82](https://github.com/flyingrobots/keep/issues/82) |
@@ -194,6 +198,22 @@ cargo xtask golden-file-worldline-check
 cargo xtask conformance-check
 ```
 
+Select a crash campaign with `--sequence NAME` using these exact CLI names:
+
+| Name | Campaign |
+| --- | --- |
+| `segment` | Segment publication |
+| `catalog` | Catalog publication |
+| `head` | Publication-head replacement |
+| `recovery-discard` | Explicit recovery evidence discard |
+| `initialization` | Writer-locked initialization |
+| `retention` | Retention root, manifest, and head publication |
+| `migration` | Version-one to version-two migration |
+
+```bash
+cargo xtask durability-crash-matrix --sequence retention
+```
+
 ## Design boundary
 
 Keep owns physical content storage: exact byte identity; chunking and

docs/adr/reader-fence-visibility.md

@@ -0,0 +1,15 @@
+# Reader fence visibility
+
+Status: accepted.
+
+Change kind: deliberate API surface reduction, with no runtime behavior change.
+
+`FilesystemRetentionSnapshot` owns the reader fence for the snapshot's lifetime; no public operation accepts, returns, or constructs the fence itself.
+
+The previously exported `ReaderFence` exposed an unusable implementation detail and invited an unnecessary compatibility obligation, so its declaration is now visible only within retention, its parent import is private, and its crate-root export is removed.
+
+This changes the unreleased source API without changing locking, snapshot lifetime, durable encoding, or recovery behavior.
+
+Static before/after evidence is the public declaration and two export sites at parent `429e3f7`; compiler validation and the existing reader-fence and snapshot runtime laws check that internal consumers continue to work.
+
+No new runtime assertion is added for visibility: source/API evidence establishes this change, while the existing behavioral laws retain their independent oracles and calibration evidence.

docs/adr/retention-bounded-recovery-landing.md

@@ -0,0 +1,15 @@
+# Bounded retention recovery landing
+
+Status: accepted maintainer decision for #99. This decision supersedes earlier retention automatic-discard requirements and claims, without changing durable record formats or the migration recovery protocol.
+
+Incomplete retained stages are preserved. Planning refuses before recovery effects with a precise demonstrated-corruption cause or a typed disposition-required result. Available checks finding no contradiction do not prove that a complete canonical record exists. The accepted availability cost is blocked publication until explicit disposition is designed; users must not be instructed to delete evidence blindly.
+
+Automatic incomplete-stage disposition and stronger completion feasibility belong to a focused follow-up. Adding a quarantine namespace, journal or durable format to this landing is rejected. Existing useful partial validation remains diagnostic evidence, not disposal authorization.
+
+The supported namespace mutation model consists of cooperating writers under Keep authority. No writer-lock isolation is promised against arbitrary concurrent raw mutation. Keep retains no-follow, identity, exact-byte, namespace and corruption checks, and rejects observed substitution. Retaining a handle and checking metadata do not make pathname unlink or rename identity-conditional.
+
+Pre-effect refusal initiates no recovery mutation. Execution failure is not rollback: completed earlier steps and the failing capability's known or uncertain effects must be distinguished, with original typed causes and durability uncertainty retained. No later recovery steps execute after failure; a new attempt must observe again. An empty completed-step list says nothing about whether the failing capability changed the namespace.
+
+Complete-stage cleanup verifies its source and surviving pool evidence while retaining the opened stage. Its guarantee is preservation of verified surviving pool evidence under the supported mutation model, not preservation of a stage pathname that cleanup intentionally removes. Directory synchronization failure cannot undo an earlier unlink, link or rename.
+
+The finite review scope is the existing recovery capabilities, shared stage operations, executor and error boundaries. The closure ledger and stable-candidate validation live in [retention-landing.md](../testing-evidence/retention-landing.md). Independent review assesses this approved contract; unrelated improvements are follow-ups. Implementation stops when the ledger closes, exact-head independent approval is recorded and required checks are green; human merge approval remains required.

docs/adr/retention-live-closure-admission.md

@@ -0,0 +1,15 @@
+# Retention live closure admission
+
+Status: Accepted
+
+Pure preflight proves reconstructability against its supplied snapshot, which may predate corruption or belong to another store. Canonical retained root bytes and matching publication records cannot establish that live segment bytes still reconstruct their anchors. Recovery and forward authority therefore share capability-relative catalog loading and the existing pure closure verifier before mutation.
+
+The loader receives the retained root directory capability directly rather than reopening an ambient pathname. It authenticates the exact current head, catalog, and selected segments. Forward publication additionally compares the resulting proof's catalog coordinates with its prepared proof, so a fresh but different catalog cannot launder a stale preparation. Recovery has no persisted original catalog coordinate in the root format and instead verifies the anchors against the current authenticated catalog. No format, root digest, or closure transcript encoding changes.
+
+The existing snapshot loader materializes all selected segments. Its aggregate retained-segment policy is independent of the root's closure physical-byte counter, which counts selected records rather than complete segment allocation and cannot bound unrelated catalog members. The explicit recovery policy admits caller-selected bounds; the default and forward authority use the protocol maximum segment length as a conservative aggregate ceiling, 1 GiB. Selection above that default refuses before mutation and may be admitted for recovery with an explicit larger policy. Catalog encoding, segment record counts, and decoded indexes retain their separate existing bounds. This is not a total RSS limit or a measured optimal budget.
+
+Reusing the authenticated snapshot and pure closure verifier preserves one semantic oracle. A new on-demand record loader could reduce memory and unrelated reads but would need independent admission and parity evidence; implementing an unchecked partial catalog or trusting old preflight would weaken the contract. This change prioritizes verification and explicit bounded refusal. It adds segment I/O and reconstruction work to forward verification and complete-stage recovery; no throughput or latency improvement is claimed. Clean or incomplete-root recovery avoids this loading work.
+
+Catalog restart and closure failures remain concrete error sources in the observation or current-verification boundary. Invalid staged framing, history, conflicting pool identity, and selected predecessor admission retain their existing earlier refusal points. The new proof is required for committed-stage cleanup as well as new publication; cleaning up after losing reconstructability would erase useful evidence.
+
+Runtime regressions cover missing and corrupted catalog or segment artifacts at root-link, manifest-link, and head-finalization boundaries, absent anchor membership, a physical closure-limit violation, and corruption after forward preflight. Explicit loading-policy boundary tests cover a budget below and exactly equal to the selected segment bytes. These experiments do not establish physical power-loss durability, every catalog size or profile, atomic unlink under concurrent replacement, or per-test resource ceilings.

docs/adr/retention-reader-catalog-errors.md

@@ -0,0 +1,19 @@
+# Retention reader catalog refusal boundary
+
+Status: accepted.
+
+The public fenced reader must distinguish catalog restart admission failures from failures to collect consistent head coordinates.
+
+Previously the filesystem source wrapped `CatalogRestartError` in `io::Error`, the collector wrapped it again, and the reader returned `FilesystemRetentionSnapshotError::View`; the documented `Catalog` variant was unreachable.
+
+The filesystem source now carries the catalog admission result as its collected view value, retaining the concrete restart error without converting it into I/O.
+
+After consistent collection, the public loader maps that admission error directly into `FilesystemRetentionSnapshotError::Catalog`, while coordinate and retention I/O errors remain `View` failures.
+
+A catalog refusal observed between moving heads is discarded with that attempt, just like a successful speculative view; the existing bounded collector retries, and only a stable pair selects the admission result.
+
+This uses the existing view-value abstraction instead of adding a breaking public error parameter to the port or storing a separate mutable error sentinel on the filesystem source.
+
+No format, identity preimage, writer protocol, or public signature changes; catalog refusal paths perform the collector's second coordinate read before reporting an admission error, and successful paths retain their existing allocation and I/O behavior.
+
+The public error regressions cover absent selected catalogs, absent selected segments, corrupt selected catalog bytes, and a corrupt coordinate HEAD; a controlled real-filesystem schedule additionally restores a missing catalog and publishes retention between the load and second coordinate read.

docs/adr/retention-reader-catalog-pinning.md

@@ -0,0 +1,15 @@
+# Retention reader catalog pinning
+
+Status: accepted.
+
+The retention reader must load catalog bytes from the same opened store directory that supplies its collected publication coordinates.
+
+Reopening the ambient path during collection can select a replacement directory while coordinate reads continue through the original directory capability, producing a refusal or a mixed view unrelated to the admitted root.
+
+The filesystem reader therefore uses the existing directory-based catalog restart loader with its pinned root capability and unchanged restart policy.
+
+This changes neither the durable format nor the public API and adds no catalog allocation or retry beyond the existing restart loader.
+
+The regression drives the production collection port with a real migrated store, renames that store after pinning, places an empty replacement at the ambient path, and requires the original catalog generation and digest.
+
+Full public-loader scheduling during admission, head-coordinate completeness, and catalog error classification remain separate obligations.

docs/adr/retention-reader-complete-coordinates.md

@@ -0,0 +1,17 @@
+# Complete retention reader coordinates
+
+Status: accepted.
+
+Reader double collection must compare every validated semantic field of both selected heads, including catalog length and retention manifest length and predecessor.
+
+The filesystem reader previously projected both heads to generation and digest, so a correctly checksummed catalog head with a changed admitted length could appear unchanged and select a view that the later head would not admit.
+
+Catalog coordinates now retain validated generation, length, and digest; retention coordinates retain the existing validated `RetentionHead` value, whose equality includes generation, manifest length, digest, and predecessor.
+
+Invalid head checksums and other decoding failures refuse through the existing source-error boundary; they are not retried as successfully decoded coordinate changes.
+
+This intentionally changes the unreleased public `RetentionViewCoordinates` field types, requiring port implementations to supply the missing validated coordinates; it does not change durable bytes, identity preimages, retry limits, or write/recovery protocols.
+
+Retaining exact encoded head bytes was rejected because the storage-independent collection port exchanges validated semantic values rather than codecs or filesystem representations.
+
+Full coordinate equality still cannot observe an arbitrary external replacement and restoration entirely between its reads, or establish that an arbitrary port's loaded value actually belongs to those coordinates; these are separate limits of the existing collection contract.

docs/adr/retention-reader-root-identity.md

@@ -0,0 +1,9 @@
+# Retention reader root identity
+
+Status: Accepted
+
+Joint admission of `FORMAT`, `migration.intent`, and `migration.receipt` establishes that those records agree with each other. It does not establish that the directory receiving the read request is the directory the migration intent names. Reader admission must compare the opened root's physical identity before accepting the version-two read capability.
+
+The reader uses the same root identity probe and `require_root_identity` predicate as writer admission. Restart-stable device and inode coordinates must match; mount-instance identity is not compared, preserving the established remount rule. A mismatch remains a concrete `FilesystemPlatformAdmissionError::RootIdentityChanged` source inside the reader admission error, with the coordinate and both values intact. No record, identity preimage, or format bytes change.
+
+The regression transplants a complete mutually consistent migration-record set from one controlled migrated directory into another on the same filesystem. Reading the recipient must refuse with the donor inode as expected and the recipient inode as observed. Ordinary migrated-root snapshot laws establish the positive path. This experiment does not exercise an actual device move or remount, and does not establish catalog-path pinning, every read interleaving, or platform admission beyond the existing identity probe.

docs/adr/retention-recovery-directory-binding.md

@@ -0,0 +1,32 @@
+# Bind explicit recovery to admitted protocol directories
+
+This decision owns directory binding at the retention recovery entry point.
+It implements the existing exact-capability invariant from ADR-0009 and
+addresses the directory-replacement finding on PR #99 for issue #19.
+
+Writer admission pins the store root, retention directory and both immutable
+pools. Retaining those handles avoids following a replacement directory, but
+does not prove that the live protocol names still identify the admitted
+directories. Recovery through an unreachable old handle can otherwise delete
+stage evidence while returning success for a different visible store.
+
+`FilesystemRetentionPublicationAuthority::recover` runs the existing
+device/inode guard before observing stages or executing recovery effects.
+Publication invokes this same entry point, so both paths apply one rule. A
+replacement returns the existing `ProtocolDirectoryReplaced` refusal through
+the recovery `Observe` boundary, preserving its source. No durable bytes,
+identity encoding, synchronization order or successful recovery classification
+changes.
+
+Keeping the guard only in publication was rejected because explicit recovery
+is a public production path. Reopening replacement directories was rejected
+because they did not supply the authority's admission proof. The check is
+synchronous capability-relative I/O with bounded extra directory opens and no
+record-body allocation.
+
+Three permanent laws rename and recreate retention, roots and manifests in
+turn, then require the exact refusal and unchanged retained root-stage bytes.
+The laws fail on head `c9277eadbcb22a5ba1330760a3b908178957d4e5` at the
+intended refusal assertion. They use filesystem unit fixtures; they establish
+post-admission binding rather than production platform admission or physical
+power-loss durability. Other recovery findings remain independently open.