diff --git a/CHANGELOG.md b/CHANGELOG.md index ce8a27af..f1f0d82b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,24 @@ after its public API and format compatibility policies are established. ### Added +- Recovery receipts retain the exact observed namespace prefix and bound + migration intent digest, including observations of completed migration. + +- Recovery jointly verifies retained stages and canonical records during + adoption, refusing substituted canonical inodes before forward execution. + +- Late-stage refusals report receipt effects as receipts when no marker + artifact exists. + +- Migration recovery revalidates current writer authority before observing + residue, preventing stale expected intents from admitting a corrupt store. + +- README recovery and reader-fence gaps point to their current open owners. + +- Migration recovery planning, bounded filesystem residue admission, explicit + truncated-stage discard, and receipts listing resumed phases. The production + crash matrix now includes 68 migration process-death cases (Refs #108). + - `FilesystemRetentionPublicationAuthority` executes the 17 ordered retention publication phases against a completely migrated version-2 root. It stages `root.next`, `manifest.next`, and `head.next` exclusively, verifies device @@ -606,6 +624,20 @@ after its public API and format compatibility policies are established. ### Fixed +- The migration transition-ledger guard rejects noncanonical line endings, + extra rows, misplaced discard claims, and counterfeit completion postures. + Recovery posture and namespace interruption checks use their exact columns. +- The independent migration restart model propagates missing occurrence, + arithmetic, and extent-conversion failures instead of normalizing them. +- Crash-case admission refuses out-of-range migration namespace occurrences + before child execution; variable segment-record occurrences remain valid. +- Receipt-only migration residue before a complete namespace reports + `ReceiptBeforeMarker`, preserving the exact missing-prerequisite boundary. +- Migration residue observation rejects non-regular entries with the typed + kind refusal before opening them, retaining the post-open kind recheck. +- Migration resumption is internal to verified recovery admission; external + callers cannot bypass current authority verification and residue adoption. + - Version-two reopen compares persisted device and inode coordinates while retaining mount identity as same-process migration evidence (#97). Simulated-remount reopen and exact moved-root refusals are regression-tested; diff --git a/README.md b/README.md index cb940e06..6c62ecce 100644 --- a/README.md +++ b/README.md @@ -67,19 +67,28 @@ Keep is required to refuse all three, before mutating anything. files, replaced protocol directories, and every namespace or capacity violation before it writes anything. Each refusal is a typed value, not a string. +- **Migration restart recovery.** Recovery verifies current authority, + classifies the observed prefix, and resumes an exact persisted migration + intent through the remaining phases. An incomplete pre-effect intent stage + is discarded and rebuilt from freshly verified current intent. Its crash + matrix kills real writer processes + at 68 before/during/after coordinates (`KEEP-CRASH-053`–`073`), preserving + every version-1 byte. Broader hostile restart combinations remain in #111. ## What it does not do yet -Version 2 writes correctly from a clean start and, if it finds the residue of -an interrupted publication, refuses rather than guesses. Nothing yet recovers -that residue, and readers have no fence, so **an interrupted version-2 -publication waits for a human until #19 lands.** A version-1 store stays -admitted until its owner migrates it; migrate only if you accept that wait. +Version-2 retention publication writes from a clean start and refuses the +residue of an interrupted retention publication. Retention publication recovery +and the reader snapshot fence await integration from PR #99, so **an +interrupted retention publication waits for explicit recovery.** Migration +restart recovery is implemented and does not grant retention authority. A +version-1 store stays admitted until its owner migrates it; migrate only if +you accept that wait. | Gap | Tracked | | --- | --- | -| Restart recovery for retention publication and migration | [#19](https://github.com/flyingrobots/keep/issues/19) | -| Reader fence binding one consistent catalog + retention snapshot | [#19](https://github.com/flyingrobots/keep/issues/19) | +| Retention publication restart recovery and broader migration corruption coverage | [PR #99](https://github.com/flyingrobots/keep/pull/99), [#111](https://github.com/flyingrobots/keep/issues/111) | +| Reader fence binding one consistent catalog + retention snapshot | [PR #99](https://github.com/flyingrobots/keep/pull/99) | | Precise verification reports at explicit depths | [#20](https://github.com/flyingrobots/keep/issues/20) | | Garbage collection and identity-preserving compaction | [#21](https://github.com/flyingrobots/keep/issues/21) | | Bounded production ingestion through the durable store | [#82](https://github.com/flyingrobots/keep/issues/82) | diff --git a/conformance/segment-store/v2/ORIGIN.md b/conformance/segment-store/v2/ORIGIN.md index 305c4d23..84f00f7c 100644 --- a/conformance/segment-store/v2/ORIGIN.md +++ b/conformance/segment-store/v2/ORIGIN.md @@ -6,6 +6,13 @@ The corpus was constructed on 2026-07-29 with: - `cargo 1.96.0 (30a34c682 2026-05-25)`; and - `b3sum 1.8.5`. +`transitions.tsv` was added on 2026-09-30 by transcribing the 21 boundaries of +`StoreMigrationPhase::ALL` and the recovery table in +`docs/formats/segment-store-v2/migration-recovery.md`. No independent oracle +constructs this ledger. Its committed-byte guard is `transition_laws.rs` +under `xtask/tests/retention_store_v2_protocol_contract/`. The crash matrix +checks runtime behavior at the same boundaries without reading the TSV. + ## Independent inputs The oracle imports exact bytes only from these previously accepted fixtures: @@ -60,6 +67,9 @@ Exact output: A temporary ignored Rust test wrote the initially reviewed TSV and hexadecimal artifacts from the handwritten oracle. That write path was removed immediately after materialization. The committed oracle is read-only and rejects drift. +This construction claim covers the original format tables and hexadecimal +records; the later handwritten `transitions.tsv` has the separate verification +boundary described above. Changing any fixture requires a deliberate specification change, an updated definition or profile digest when affected, fresh independent construction, diff --git a/conformance/segment-store/v2/README.md b/conformance/segment-store/v2/README.md index 7417eb21..c995c8b2 100644 --- a/conformance/segment-store/v2/README.md +++ b/conformance/segment-store/v2/README.md @@ -14,6 +14,7 @@ migration, retention transition, or garbage collector exists. | `inventory.tsv` | Canonical one-segment, one-catalog migration inventory | | `migration-source.tsv` | Exact version-1 and derived migration coordinates | | `artifacts.tsv` | Golden artifact lengths, digests, checksums, and filenames | +| `transitions.tsv` | One stable crash identifier per migration boundary, `KEEP-CRASH-053` to `-073` | | `format-marker.hex` | Canonical 96-byte `FORMAT` record | | `migration-intent.hex` | Canonical 256-byte migration intent | | `migration-receipt.hex` | Canonical 256-byte migration receipt | @@ -55,6 +56,24 @@ The retention fixture uses namespace bytes `00 2f ff`, proving the namespace is opaque and not a path or Unicode string. Its one anchor combines the canonical one-zero `BlobId` and `LayoutId` values from the existing layout corpus. +## Transition protocol + +`transitions.tsv` mirrors the version-1 table: one row per migration +durability operation with its pre-state, interrupted-state classification, +post-state, and recovery posture. `KEEP-CRASH-053`, `-062`, and `-068` are +the only rows whose interruption may leave an incomplete pre-effect stage and +therefore the only rows that plan a discard; `-072` and `-073` admit the +complete migration. + +The `cargo xtask durability-crash-matrix --sequence migration` harness +executes 68 canonical process-death cases at the same 21 boundaries: each at +three positions plus one `during` case per admitted directory-prefix length +for `KEEP-CRASH-060`. It kills an isolated writer process group, compares the +restarted root against an independent expected-state model, requires the +production planner to report the predicted recovery plan, and requires the +recovered store to be one complete migration with every version-1 byte +intact. Host power loss remains outside its claim. + ## Verification Run: @@ -62,6 +81,8 @@ Run: ```bash cargo test --manifest-path xtask/Cargo.toml \ --test retention_store_v2_format_oracle +cargo test --manifest-path xtask/Cargo.toml \ + --test retention_store_v2_protocol_contract transition_laws ``` The test-only oracle constructs every record from handwritten offsets and @@ -69,6 +90,12 @@ domain preimages, compares exact fixture bytes and tables, and imports no production version-2 codec. The repository protocol and documentation gates route this corpus separately. -Passing this corpus is necessary but insufficient for issue #19. Production -code still needs parser, corruption, property, model, crash, recovery, +`transition_laws.rs` checks the handwritten transition ledger's committed +shape, operation order, and recovery-posture claims. The format oracle does +not construct or compare `transitions.tsv`; the crash matrix checks runtime +behavior without reading its bytes. + +Passing this corpus is necessary but insufficient for migration recovery +(#108), restart corruption (#111), or compatibility and fuzz coverage (#112). +Production code still needs parser, corruption, property, model, crash, recovery, concurrency, fuzz, and public API evidence. diff --git a/conformance/segment-store/v2/transitions.tsv b/conformance/segment-store/v2/transitions.tsv new file mode 100644 index 00000000..cce218d3 --- /dev/null +++ b/conformance/segment-store/v2/transitions.tsv @@ -0,0 +1,23 @@ +keep.segment-store.transitions/v2 +crash_id phase operation pre_state interrupted_class post_state recovery_posture +KEEP-CRASH-053 migration write-intent-stage admitted-version-one-store absent-or-incomplete-intent-stage complete-intent-stage discard-incomplete-stage-or-resume-stage-sync +KEEP-CRASH-054 migration sync-intent-stage complete-intent-stage complete-intent-stage durable-intent-stage resume-stage-sync +KEEP-CRASH-055 migration link-intent durable-intent-stage durable-intent-stage-or-linked-intent linked-intent verify-no-clobber-link-and-resume-root-sync +KEEP-CRASH-056 migration sync-root-after-intent linked-intent linked-intent durable-intent resume-root-sync +KEEP-CRASH-057 migration remove-intent-stage durable-intent durable-intent-with-or-without-stage durable-intent-alone resume-cleanup-sync +KEEP-CRASH-058 migration sync-root-after-intent-cleanup durable-intent-alone durable-intent-alone intent-cleanup-synchronized resume-cleanup-sync +KEEP-CRASH-059 migration admit-reader-fence intent-cleanup-synchronized intent-with-or-without-reader-fence reader-fence-admitted resume-namespace-prefix +KEEP-CRASH-060 migration admit-namespace-prefix reader-fence-admitted directory-prefix-length-zero-to-six namespace-prefix-admitted resume-namespace-prefix-or-root-sync +KEEP-CRASH-061 migration sync-root-after-namespace namespace-prefix-admitted namespace-prefix-admitted durable-namespace-prefix resume-root-sync +KEEP-CRASH-062 migration write-marker-stage durable-namespace-prefix absent-or-incomplete-marker-stage complete-marker-stage discard-incomplete-stage-or-resume-stage-sync +KEEP-CRASH-063 migration sync-marker-stage complete-marker-stage complete-marker-stage durable-marker-stage resume-stage-sync +KEEP-CRASH-064 migration link-marker durable-marker-stage durable-marker-stage-or-linked-marker linked-marker verify-no-clobber-link-and-resume-root-sync +KEEP-CRASH-065 migration sync-root-after-marker linked-marker linked-marker durable-marker resume-root-sync +KEEP-CRASH-066 migration remove-marker-stage durable-marker durable-marker-with-or-without-stage durable-marker-alone resume-cleanup-sync +KEEP-CRASH-067 migration sync-root-after-marker-cleanup durable-marker-alone durable-marker-alone marker-cleanup-synchronized resume-cleanup-sync +KEEP-CRASH-068 migration write-receipt-stage marker-cleanup-synchronized absent-or-incomplete-receipt-stage complete-receipt-stage discard-incomplete-stage-or-resume-stage-sync +KEEP-CRASH-069 migration sync-receipt-stage complete-receipt-stage complete-receipt-stage durable-receipt-stage resume-stage-sync +KEEP-CRASH-070 migration link-receipt durable-receipt-stage durable-receipt-stage-or-linked-receipt linked-receipt verify-no-clobber-link-and-resume-root-sync +KEEP-CRASH-071 migration sync-root-after-receipt linked-receipt linked-receipt durable-receipt resume-root-sync +KEEP-CRASH-072 migration remove-receipt-stage durable-receipt durable-receipt-with-or-without-stage complete-migration admit-complete-migration +KEEP-CRASH-073 migration sync-root-after-receipt-cleanup complete-migration complete-migration durable-complete-migration admit-complete-migration diff --git a/docs/formats/segment-store-v2/README.md b/docs/formats/segment-store-v2/README.md index 0f70d682..71194b52 100644 --- a/docs/formats/segment-store-v2/README.md +++ b/docs/formats/segment-store-v2/README.md @@ -8,11 +8,12 @@ namespaces. ADR-0009 owns the cross-cutting retention and liveness decision. These pages own its durable representation. The one-way migration, version-two reopen, and -forward retention publication are implemented with executable evidence; -recovery of retained retention stages, reader fencing, and collection remain -planned in issue #19, and the [requirements ledger](requirements.md) records -exactly which requirements are proven. A version-1 store remains admitted until -its owner migrates it. +forward retention publication, partial-prefix migration recovery, and the +68-case migration process-death matrix are implemented with executable +evidence. Retention recovery and reader fencing await integration from PR #99; +collection remains planned in #21. The [requirements ledger](requirements.md) +records exactly which requirements are proven. A version-1 store remains +admitted until its owner migrates it. ## Core laws @@ -95,9 +96,12 @@ stages, replaced protocol directories, and every namespace or capacity violation before mutation, each as a typed `RetentionCurrentStateRefusal`. Not implemented: retention publication recovery and `KEEP-CRASH-036..052` -process-death evidence, partial-prefix migration recovery and -`KEEP-CRASH-053..073`, the reader fence, model-based transition evidence, and -garbage collection. Issue #19 owns the first four and issue #21 the last. +process-death evidence, the reader fence, model-based transition evidence, and +garbage collection. Partial-prefix migration recovery and the 68-case +`KEEP-CRASH-053..073` process-death matrix are implemented. Broader migration +restart corruption and compatibility coverage remain in #111 and #112. +PR #99 contains retention recovery and reader fencing awaiting integration; +issue #21 owns garbage collection. Reopen compares only the restart-stable root coordinates, device and inode, against the intent; see [root identity across restart](recovery.md#root-identity-across-restart). A diff --git a/docs/formats/segment-store-v2/migration-crash.md b/docs/formats/segment-store-v2/migration-crash.md index d81d5656..0afd372c 100644 --- a/docs/formats/segment-store-v2/migration-crash.md +++ b/docs/formats/segment-store-v2/migration-crash.md @@ -24,15 +24,17 @@ For each pair, migration: The verified stage is linked without replacement. The canonical target is immutable. Recovery never truncates, replaces, or repairs it. An exact stage -with an absent target resumes at the link. Exact stage and target bytes resume -at the required synchronization or cleanup. Different bytes, a substituted -inode, a link, or a wrong file kind refuse. +with an absent target resumes at stage synchronization. Exact stage and target +bytes resume at the required synchronization or cleanup. Different bytes, a +substituted inode, a link, or a wrong file kind refuse. A pre-effect incomplete stage may be removed only when its canonical target and every later-ordered migration effect are absent and every earlier effect admits -exactly. Recovery pins the stage, removes it, synchronizes the store root, and -returns a typed discard report. Any later effect makes incomplete or corrupt -stage bytes unrecoverable ambiguity. +exactly. Migration recovery revalidates the present stage's regular kind and +strictly incomplete length immediately before removal, removes it, synchronizes +the store root, and returns a typed discard report. It does not retain an +incomplete-stage handle. Any later effect makes incomplete or corrupt stage +bytes unrecoverable ambiguity. The fixed stage is not authority. `migration.intent` becomes migration authority only after its canonical link and store-root synchronization. @@ -103,8 +105,18 @@ prefix length. Restart must classify exact stages, canonical targets, namespace prefix, marker, receipt, and cleanup state without depending on a clock, filesystem iteration order, or file existence alone. +Namespace occurrence coordinates are zero-based: `during` accepts zero +through five, while `before` and `after` accept only zero. Invalid coordinates +refuse at crash-case admission before a child starts. Segment record +occurrences remain dependent on the write's record count. + `StoreMigrationPhase::ALL` freezes the 21 boundaries above in exact order. -Fresh writer-locked filesystem execution now implements that exact order and -has deterministic in-process storage-fault and corruption laws. The -before/during/after process-death matrix and restart classifier remain -unimplemented; this page does not yet claim crash recovery. +The production recovery planner and filesystem adapter execute the lawful +remaining suffix. `cargo xtask durability-crash-matrix --sequence migration` +runs 68 process-death cases: before/during/after each boundary, with all six +directory-prefix occurrences at `KEEP-CRASH-060`. The existing CI matrix +runs these cases in debug and release builds. + +These cases establish recovery after process death; they do not simulate power +loss. Broader restart corruption and compatibility/fuzz coverage remain tracked +by #111 and #112. diff --git a/docs/formats/segment-store-v2/migration-recovery.md b/docs/formats/segment-store-v2/migration-recovery.md index 63f1c19a..033a183d 100644 --- a/docs/formats/segment-store-v2/migration-recovery.md +++ b/docs/formats/segment-store-v2/migration-recovery.md @@ -58,6 +58,12 @@ The migration recovery boundary admits only these ordered prefixes: +Every row also requires the admission checks in +[Executable recovery boundary](#executable-recovery-boundary). An intent stage +surviving a namespace effect, or a marker stage surviving a receipt effect, +refuses as `StageAfterEffect`. When a stage and canonical target both exist, +exact bytes and one shared device/inode identity are required before resumption. + A partial migration retry revalidates intent and existing bytes, resumes at the first absent canonical step, and never replaces an entry. A missing predecessor, changed version-1 coordinate, out-of-order name, wrong kind or bytes, conflicting @@ -65,3 +71,35 @@ receipt, unknown entry, or changed root identity is unrecoverable ambiguity. Death before durable intent leaves v1 plus at most its non-authoritative stage. Death after durable intent leaves recovery-required v2 migration state. + +## Executable recovery boundary + +`FilesystemStoreMigrationAuthority::reopen_for_recovery` reacquires writer +authority without granting version-1 publication admission. +`recover_store_migration` revalidates the caller's freshly derived current +intent, then observes bounded residue and applies +`plan_store_migration_recovery` before adopting records or discarding stages. +Nested directory membership is checked before mutating recovery. An intent +stage surviving namespace creation, or a marker stage surviving receipt +publication, is refused as `StageAfterEffect`. + +When both a stage and its canonical target exist, adoption verifies they share +the same device and inode as well as exact bytes before any forward phase, +including directory synchronization. + +The receipt retains the exact observed namespace prefix and bound intent +digest, names the admitted plan, and lists the executed forward phases +through `executed_phases()`. The plan records the earliest unproven boundary; +it does not infer which synchronization calls completed before process death. +For `VersionOne`, `intent_digest()` returns `None`: no migration intent was +admitted. Every other plan reports the intent used by recovery. An incomplete +pre-effect intent stage has no complete persisted intent, so its discard path +uses the freshly verified current intent. +Restart compares device and inode identity; mount identity is same-process +evidence. Whenever an exact intent survives, recovery continues with its +persisted bytes. + +The filesystem laws cover every forward prefix, every strict byte-prefix +truncation of all three stages, unchanged version-1 bytes, and refusal before +mutation for corrupt intent and unexpected nested residue. The complete +restart corruption matrix remains tracked separately in #111. diff --git a/docs/formats/segment-store-v2/recovery.md b/docs/formats/segment-store-v2/recovery.md index 110231d2..646c0b4e 100644 --- a/docs/formats/segment-store-v2/recovery.md +++ b/docs/formats/segment-store-v2/recovery.md @@ -62,8 +62,9 @@ corpus `definition.tsv` bytes. The format-marker digest is BLAKE3-256 of `CanonicalStoreMigrationIntent` retains typed intent coordinates; `CanonicalStoreMigrationReceipt` binds completion; admitted record types verify both. `StoreMigrationStorage` names all 21 durability capabilities; `execute_store_migration` verifies current authority first and returns only after final synchronization. `FilesystemStoreMigrationInventoryReader` inventories version-1 bytes under -retained writer authority. The fresh writer executes once; partial-prefix -recovery is absent; version-1 reopen and recovery both refuse a migrated root. +retained writer authority. The fresh writer executes once; separate migration +recovery plans and executes the lawful remaining suffix after process death. +Version-1 reopen and version-1 recovery both refuse a migrated root. ## Reader fence @@ -160,12 +161,13 @@ The restart-stable root identity is therefore the pair `(device, file)`: same process; that comparison catches a root swapped underneath a running migration and never crosses a restart. - `FilesystemVersionTwoAdmission::reopen` compares device and - file only and refuse with `RootIdentityChanged { coordinate: Device | File, + file only and refuses with `RootIdentityChanged { coordinate: Device | File, .. }`. A remounted store admits; a store copied to another device or restored into a different directory refuses. -Future partial-prefix recovery must use the same restart comparison; -this decision does not implement that recovery engine. +Partial-prefix migration recovery uses the same restart comparison; its +authority and resumption protocol are specified in +[the executable recovery boundary](migration-recovery.md#executable-recovery-boundary). The mount coordinate stays in the record as the migration-time observation. It remains evidence for same-process migration checks, not restart authority. @@ -212,9 +214,11 @@ recovery instead. Direct version-2 initialization is undefined. The exact offsets and fixtures are requirement `KEEP-MIGRATION-002`. The fresh writer emits only those canonical records; success is not restart evidence. -A migrated store is admitted for forward publication, but partial-prefix -recovery and `KEEP-MIGRATION-007` process-death evidence remain absent, so an -interrupted migration waits for recovery instead of continuing. +A migrated store is admitted for forward publication. Partial-prefix recovery +now plans and executes the lawful remaining migration suffix under writer +authority; the 68-case migration process-death matrix supplies restart evidence. +Broader hostile restart and compatibility coverage remain tracked in #111 and +issue #112. ## Retention publication recovery @@ -247,6 +251,12 @@ transitive member, reappeared stage, conflicting pool entry, or other corruption is a typed refusal. A complete valid orphan remains recovery-protected until explicit disposition. +This retention protocol requires pinning the incomplete regular file. The +separate [migration discard path](migration-crash.md#fixed-stage-law) +revalidates the current entry's regular kind and incomplete length before +removal without retaining an incomplete-stage handle. These are distinct +protocol boundaries; retention recovery awaits integration from PR #99. + The retention crash points are: | Identifier | Boundary | diff --git a/docs/formats/segment-store-v2/requirements.md b/docs/formats/segment-store-v2/requirements.md index 2ab9fc40..172bcb89 100644 --- a/docs/formats/segment-store-v2/requirements.md +++ b/docs/formats/segment-store-v2/requirements.md @@ -28,14 +28,14 @@ case is not evidence. | ID | Requirement | Evidence | Status | | --- | --- | --- | --- | -| `KEEP-MIGRATION-001` | Exact version-1 stores remain admitted until a durable migration artifact exists | compatibility fixtures | Planned in #19 | +| `KEEP-MIGRATION-001` | Exact version-1 stores remain admitted until a durable migration artifact exists | `tests/store_migration_recovery.rs` admits the artifact-free version-one observation; `src/adapters/store_migration/filesystem_migration_recovery_tests.rs` verifies every forward prefix under a fresh authority | Implemented; compatibility expansion in #112 | | `KEEP-MIGRATION-002` | Format marker, intent, and receipt have complete fixed byte tables, named domains, bounds, checksums, deterministic store identity, and exact initial-state digests | exact admission in `tests/store_format_marker.rs`, `tests/store_migration_intent.rs`, and `tests/store_migration_receipt.rs`; canonical construction in `tests/store_migration_intent_encoding.rs` and `tests/store_migration_receipt_encoding.rs`; seeded `migration_format` fuzz target | Implemented | | `KEEP-MIGRATION-003` | Migration revalidates version-1 head, catalog, pools, root identity (all three coordinates within the migrating process; device and file across restart), and writer authority before mutation | bounded canonical pool inventory in `tests/store_migration_inventory.rs`; writer-locked filesystem pool admission in `filesystem_inventory_*_tests`; exact authority observation and drift refusal in `filesystem_migration_authority_tests`; verification-first execution in `tests/store_migration_execution.rs`; fresh filesystem integration and post-publication drift refusal in `filesystem_migration_storage_tests`; a version-one store still holding a retained stage refuses before the intent is observed in `filesystem_migration_storage_tests` | Implemented | -| `KEEP-MIGRATION-004` | Every partial migration prefix continues idempotently under writer authority, comparing only the restart-stable root coordinates (device and file) against the persisted intent | restart-stable coordinate laws in `filesystem_version_two_admission_tests` and complete reopen laws in `filesystem_migration_remount_tests`; state-machine and recovery tests remain | Planned in #19 | -| `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | forward-execution stage preservation, byte-equal inode-substitution, out-of-order-prefix, and post-publication drift laws in `filesystem_migration_storage_tests`; unknown `retention` entries, non-digest namespace directories, and noncanonical pool names refuse before any retention stage is written in `filesystem_retention_namespace_tests`; restart corruption and mutation matrix remains | In progress in #19 | -| `KEEP-MIGRATION-006` | Migration never rewrites or deletes admitted version-1 immutable bytes | exact segment, catalog, and head before/after witness in `filesystem_migration_storage_tests`; restart-path evidence remains | In progress in #19 | -| `KEEP-MIGRATION-007` | Process death around every intent stage, canonical link, namespace prefix, marker stage, receipt stage, cleanup, and synchronization boundary reaches a documented lawful state | ordered phases and capabilities in `tests/store_migration_phase.rs` and `tests/store_migration_storage.rs`; exact phase-failure execution in `tests/store_migration_execution.rs`; production 21-phase forward execution in `filesystem_migration_storage_tests`; `KEEP-CRASH-053..=073` process-death matrix remains | In progress in #19 | -| `KEEP-MIGRATION-008` | Version-1 admission refuses every version-2 or partial-migration artifact after migration begins | `FORMAT` refusal before mutation in `filesystem_migration_authority_tests`; exact version-1 reopen refusal of a migrated root and separate version-2 namespace admission in `filesystem_initialization_namespace`; version-2 reopen returns a distinct `FilesystemVersionTwoAdmission` that no version-1 publisher can consume (pinned by `tests/version_two_admission_contract.rs`), admits every version-2 protocol directory under the Linux profile, and jointly admits the exact marker, intent, and receipt before returning writer authority, with aliased-directory, corrupt, oversized, and mutually inconsistent record refusals in `filesystem_version_two_admission_tests` and `filesystem_platform_profile_tests`; remaining compatibility and fuzz matrix | In progress in #19 | +| `KEEP-MIGRATION-004` | Every partial migration prefix continues idempotently under writer authority, comparing only the restart-stable root coordinates (device and file) against the persisted intent | restart-stable coordinate laws in `filesystem_version_two_admission_tests` and complete reopen laws in `filesystem_migration_remount_tests`; `tests/store_migration_recovery.rs` and `tests/store_migration_recovery_order.rs` freeze planner and ordering laws; `src/adapters/store_migration/filesystem_migration_recovery_tests.rs` covers every forward prefix; `src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs` covers every strict fixed-stage truncation | Implemented in #108 | +| `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | forward-execution stage preservation, byte-equal inode-substitution, out-of-order-prefix, and post-publication drift laws in `filesystem_migration_storage_tests`; unknown `retention` entries, non-digest namespace directories, and noncanonical pool names refuse before any retention stage is written in `filesystem_retention_namespace_tests`; restart corruption and mutation matrix remains | In progress in #111 | +| `KEEP-MIGRATION-006` | Migration never rewrites or deletes admitted version-1 immutable bytes | exact segment, catalog, and head witnesses in `src/adapters/store_migration/filesystem_migration_storage_tests.rs`, `src/adapters/store_migration/filesystem_migration_recovery_tests.rs`, and `src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs`; subprocess restart witnesses in `cargo xtask durability-crash-matrix --sequence migration` | Implemented in #108 | +| `KEEP-MIGRATION-007` | Process death around every intent stage, canonical link, namespace prefix, marker stage, receipt stage, cleanup, and synchronization boundary reaches a documented lawful state | ordered phases and capabilities in `tests/store_migration_phase.rs` and `tests/store_migration_storage.rs`; exact phase-failure execution in `tests/store_migration_execution.rs`; production 21-phase forward execution in `filesystem_migration_storage_tests`; `cargo xtask durability-crash-matrix --sequence migration` runs 68 production subprocess cases at `KEEP-CRASH-053..=073`, debug and release | Implemented in #108 | +| `KEEP-MIGRATION-008` | Version-1 admission refuses every version-2 or partial-migration artifact after migration begins | `FORMAT` refusal before mutation in `filesystem_migration_authority_tests`; exact version-1 reopen refusal of a migrated root and separate version-2 namespace admission in `filesystem_initialization_namespace`; version-2 reopen returns a distinct `FilesystemVersionTwoAdmission` that no version-1 publisher can consume (pinned by `tests/version_two_admission_contract.rs`), admits every version-2 protocol directory under the Linux profile, and jointly admits the exact marker, intent, and receipt before returning writer authority, with aliased-directory, corrupt, oversized, and mutually inconsistent record refusals in `filesystem_version_two_admission_tests` and `filesystem_platform_profile_tests`; remaining compatibility and fuzz matrix | In progress in #112 | diff --git a/src/adapters/filesystem_exact_record.rs b/src/adapters/filesystem_exact_record.rs index 14beda76..bda29df7 100644 --- a/src/adapters/filesystem_exact_record.rs +++ b/src/adapters/filesystem_exact_record.rs @@ -71,6 +71,16 @@ pub(super) enum ExactRecordError { Refused(ExactRecordRefusal), } +impl ExactRecordError { + /// Preserves the original operational source or typed semantic refusal. + pub(super) fn into_io(self) -> io::Error { + match self { + Self::Io(source) => source, + refused @ Self::Refused(_) => io::Error::new(io::ErrorKind::InvalidData, refused), + } + } +} + impl fmt::Display for ExactRecordRefusal { fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { formatter.write_str(match self { @@ -205,6 +215,42 @@ pub(super) fn link_without_replacement( } } +/// Opens the regular record `name` read-only, following no links and never +/// blocking, for callers that retain the handle and verify it by identity. +pub(super) fn open_regular(directory: &Dir, name: &str) -> io::Result { + open_read(directory, name) +} + +/// Reads at most `bound` bytes of the regular file `name`, or `None` if absent. +/// +/// Residue observers use this to see an incomplete, exact, or overlong record +/// as it is; a present entry that is not a regular file refuses by kind. +pub(super) fn read_bounded_optional( + directory: &Dir, + name: &str, + bound: usize, +) -> Result>, ExactRecordError> { + match directory.symlink_metadata(name) { + Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(None), + Err(source) => return Err(source.into()), + Ok(metadata) if !metadata.is_file() => { + return Err(ExactRecordRefusal::KindOrLength.into()); + } + Ok(_) => {} + } + let file = match open_read(directory, name) { + Ok(file) => file, + Err(source) if source.kind() == io::ErrorKind::NotFound => return Ok(None), + Err(source) => return Err(source.into()), + }; + if !file.metadata()?.is_file() { + return Err(ExactRecordRefusal::KindOrLength.into()); + } + let mut bytes = Vec::new(); + file.take(exact_length(bound)?).read_to_end(&mut bytes)?; + Ok(Some(bytes)) +} + fn open_read(directory: &Dir, name: &str) -> io::Result { let mut options = OpenOptions::new(); options.read(true).follow(FollowSymlinks::No).nonblock(true); diff --git a/src/adapters/filesystem_initialization_namespace.rs b/src/adapters/filesystem_initialization_namespace.rs index d1be904a..6b71be03 100644 --- a/src/adapters/filesystem_initialization_namespace.rs +++ b/src/adapters/filesystem_initialization_namespace.rs @@ -202,3 +202,35 @@ fn ambiguous_namespace() -> io::Error { "store root is not an empty or partial canonical initialization namespace", ) } + +/// Admits a published version-1 root carrying any subset of migration +/// residue, for migration recovery only. +/// +/// The five published names are required with their kinds; every migration +/// record, stage, the reader fence, and the three protocol directories are +/// optional but must have their kinds; anything else refuses. Which subsets +/// are lawful is the recovery planner's decision, not this admission's. +pub(super) fn admit_migrating(directory: &Dir) -> io::Result<()> { + admit_required_file(directory, LOCK_NAME)?; + admit_required_directory(directory, STAGING_NAME)?; + admit_required_directory(directory, SEGMENTS_NAME)?; + admit_required_directory(directory, CATALOGS_NAME)?; + admit_required_file(directory, HEAD_NAME)?; + for name in [ + READER_LOCK_NAME, + MARKER_NAME, + "FORMAT.next", + INTENT_NAME, + "migration.intent.next", + RECEIPT_NAME, + "migration.receipt.next", + ] { + admit_optional_file(directory, name)?; + } + for name in [RETENTION_NAME, GC_NAME, RECOVERY_NAME] { + admit_optional_directory(directory, name)?; + } + let mut allowed: Vec<&str> = PUBLISHED_NAMES.to_vec(); + allowed.extend_from_slice(&VERSION_TWO_MARKERS); + admit_membership(directory, &allowed) +} diff --git a/src/adapters/store_migration.rs b/src/adapters/store_migration.rs index a04b98e5..0f33fec5 100644 --- a/src/adapters/store_migration.rs +++ b/src/adapters/store_migration.rs @@ -44,8 +44,28 @@ mod filesystem_migration_fixed_artifact; mod filesystem_migration_namespace; mod filesystem_migration_namespace_directory; mod filesystem_migration_reader_fence; +mod filesystem_migration_recovery; +mod filesystem_migration_recovery_refusal; +pub use filesystem_migration_recovery_refusal::{ + FilesystemMigrationRecoveryRefusal, FilesystemMigrationResidueKind, +}; +#[cfg(test)] +mod filesystem_migration_current_recovery_tests; +#[cfg(test)] +mod filesystem_migration_pair_admission_tests; +#[cfg(test)] +mod filesystem_migration_receipt_evidence_tests; +#[cfg(test)] +mod filesystem_migration_recovery_tests; +#[cfg(test)] +mod filesystem_migration_recovery_truncation_tests; #[cfg(test)] mod filesystem_migration_remount_tests; +#[cfg(feature = "repository-tasks")] +mod filesystem_migration_repository_tasks; +mod filesystem_migration_residue; +#[cfg(test)] +mod filesystem_migration_residue_kind_tests; mod filesystem_migration_storage; #[cfg(test)] mod filesystem_migration_storage_tests; @@ -85,6 +105,18 @@ mod migration_receipt_encoder; mod migration_receipt_format; mod migration_receipt_initial_state; mod migration_record_bytes; +mod migration_stage_decode_error; +pub use migration_stage_decode_error::StoreMigrationStageDecodeError; +mod migration_namespace_prefix; +mod migration_recovery_ambiguity; +mod migration_recovery_ambiguity_display; +mod migration_recovery_execution; +mod migration_recovery_plan; +pub use migration_namespace_prefix::StoreMigrationNamespacePrefix; +mod migration_recovery_planner; +mod migration_recovery_residue; +mod migration_recovery_storage; +mod migration_resumption; mod migration_storage; mod migration_synchronization_mask; mod store_identifier; @@ -127,6 +159,14 @@ pub use migration_inventory_hasher::StoreMigrationInventoryHasher; pub use migration_phase::StoreMigrationPhase; pub use migration_receipt_decode_error::StoreMigrationReceiptDecodeError; pub(super) use migration_receipt_format::ENCODED_LENGTH as MIGRATION_RECEIPT_LENGTH; +pub use migration_recovery_ambiguity::{StoreMigrationEffect, StoreMigrationRecoveryAmbiguity}; +pub use migration_recovery_execution::{ + StoreMigrationRecoveryError, StoreMigrationRecoveryReceipt, recover_store_migration, +}; +pub use migration_recovery_plan::{StoreMigrationFixedStage, StoreMigrationRecoveryPlan}; +pub use migration_recovery_planner::plan_store_migration_recovery; +pub use migration_recovery_residue::{MIGRATION_NAMESPACE_PREFIX, StoreMigrationResidue}; +pub use migration_recovery_storage::StoreMigrationRecoveryStorage; pub use migration_storage::StoreMigrationStorage; pub use migration_synchronization_mask::MigrationSynchronizationMask; pub use store_identifier::StoreIdentifier; diff --git a/src/adapters/store_migration/filesystem_inventory_reader.rs b/src/adapters/store_migration/filesystem_inventory_reader.rs index 7d989ab6..2519a9ce 100644 --- a/src/adapters/store_migration/filesystem_inventory_reader.rs +++ b/src/adapters/store_migration/filesystem_inventory_reader.rs @@ -50,6 +50,19 @@ impl FilesystemStoreMigrationInventoryReader { policy: SegmentReadPolicy, ) -> Result { let (lock, root_identity) = admission.into_parts(); + Self::open_locked(lock, root_identity, policy) + } + + /// Pins both immutable pools under an already-held writer lock. + /// + /// Migration recovery uses this so a root carrying migration residue never + /// mints a version-1 platform admission the version-1 publisher could + /// consume. + pub(super) fn open_locked( + lock: FilesystemWriterLock, + root_identity: FilesystemRootIdentity, + policy: SegmentReadPolicy, + ) -> Result { let root = lock.clone_directory() .map_err(|source| FilesystemMigrationInventoryError::Io { diff --git a/src/adapters/store_migration/filesystem_migration_authority.rs b/src/adapters/store_migration/filesystem_migration_authority.rs index e894bcb4..44221757 100644 --- a/src/adapters/store_migration/filesystem_migration_authority.rs +++ b/src/adapters/store_migration/filesystem_migration_authority.rs @@ -38,6 +38,7 @@ const HEAD_LENGTH: u64 = 128; #[must_use] pub struct FilesystemStoreMigrationAuthority { inventory: FilesystemStoreMigrationInventoryReader, + pub(super) namespace: MigrationNamespacePolicy, pub(super) fixed_stage: Option, pub(super) published_intent: Option, pub(super) published_marker: Option, @@ -61,13 +62,24 @@ impl FilesystemStoreMigrationAuthority { ) -> Result { let inventory = FilesystemStoreMigrationInventoryReader::open(admission, policy) .map_err(|source| Error::Inventory { source })?; - Ok(Self { + Ok(Self::with_policy( inventory, + MigrationNamespacePolicy::Published, + )) + } + + pub(super) const fn with_policy( + inventory: FilesystemStoreMigrationInventoryReader, + namespace: MigrationNamespacePolicy, + ) -> Self { + Self { + inventory, + namespace, fixed_stage: None, published_intent: None, published_marker: None, published_receipt: None, - }) + } } /// Observes one canonical intent from exact current version-1 authority. @@ -136,8 +148,15 @@ impl FilesystemStoreMigrationAuthority { /// Recovery must complete before the intent is observed. fn verify_namespace(&self) -> Result<(), Error> { let root = self.inventory.root(); - filesystem_initialization_namespace::admit_published(root) - .map_err(|source| Error::Namespace { source })?; + match self.namespace { + MigrationNamespacePolicy::Published => { + filesystem_initialization_namespace::admit_published(root) + } + MigrationNamespacePolicy::Migrating => { + filesystem_initialization_namespace::admit_migrating(root) + } + } + .map_err(|source| Error::Namespace { source })?; let staging = root .open_dir_nofollow(STAGING_NAME) .map_err(|source| Error::Namespace { source })?; @@ -206,3 +225,12 @@ impl FilesystemStoreMigrationAuthority { self.inventory.root() } } + +/// Which root namespaces an authority admits when it observes. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum MigrationNamespacePolicy { + /// Exactly the published version-1 namespace: a fresh migration. + Published, + /// The published version-1 namespace plus any migration residue: recovery. + Migrating, +} diff --git a/src/adapters/store_migration/filesystem_migration_authority_error.rs b/src/adapters/store_migration/filesystem_migration_authority_error.rs index ca639f70..3dfb37f0 100644 --- a/src/adapters/store_migration/filesystem_migration_authority_error.rs +++ b/src/adapters/store_migration/filesystem_migration_authority_error.rs @@ -3,7 +3,9 @@ use std::io; use super::{FilesystemMigrationInventoryError, StoreMigrationIntentDigest}; -use crate::adapters::{CatalogDecodeError, CatalogRestartError, PublicationHeadDecodeError}; +use crate::adapters::{ + CatalogDecodeError, CatalogRestartError, PublicationHeadDecodeError, WriterLockAcquireError, +}; use crate::{CatalogDigest, CatalogGeneration, CatalogLength}; /// Published version-1 artifact observed while establishing migration authority. @@ -39,6 +41,16 @@ pub enum StoreRootIdentityCoordinate { /// Failure to observe or revalidate exact filesystem migration authority. #[derive(Debug)] pub enum FilesystemMigrationAuthorityError { + /// The production platform refused the store root. + Platform { + /// Preserved platform source. + source: io::Error, + }, + /// The existing writer lock could not be acquired. + WriterLock { + /// Preserved lock refusal. + source: WriterLockAcquireError, + }, /// Complete immutable-pool inventory could not be admitted. Inventory { /// Preserved inventory refusal. diff --git a/src/adapters/store_migration/filesystem_migration_authority_error_display.rs b/src/adapters/store_migration/filesystem_migration_authority_error_display.rs index 9fc8c14d..f794346b 100644 --- a/src/adapters/store_migration/filesystem_migration_authority_error_display.rs +++ b/src/adapters/store_migration/filesystem_migration_authority_error_display.rs @@ -32,6 +32,12 @@ impl fmt::Display for StoreRootIdentityCoordinate { impl fmt::Display for FilesystemMigrationAuthorityError { fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { match self { + Self::Platform { .. } => { + formatter.write_str("filesystem migration platform admission was refused") + } + Self::WriterLock { .. } => { + formatter.write_str("filesystem migration writer lock was refused") + } Self::Inventory { .. } => { formatter.write_str("filesystem migration inventory was refused") } @@ -96,7 +102,10 @@ impl Error for FilesystemMigrationAuthorityError { fn source(&self) -> Option<&(dyn Error + 'static)> { match self { Self::Inventory { source } => Some(source), - Self::Namespace { source } | Self::RootIdentity { source } => Some(source), + Self::WriterLock { source } => Some(source), + Self::Platform { source } + | Self::Namespace { source } + | Self::RootIdentity { source } => Some(source), Self::Artifact { source, .. } => Some(source), Self::Head { source } => Some(source), Self::Catalog { source, .. } => Some(source), diff --git a/src/adapters/store_migration/filesystem_migration_current_recovery_tests.rs b/src/adapters/store_migration/filesystem_migration_current_recovery_tests.rs new file mode 100644 index 00000000..a5c981e2 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_current_recovery_tests.rs @@ -0,0 +1,50 @@ +//! This module owns recovery's current-authority validation before admission. + +use std::error::Error; +use std::fs; +use std::io; + +use super::filesystem_migration_test_fixture::{maximum_policy, open_authority}; +use super::{ + FilesystemMigrationAuthorityError, FilesystemStoreMigrationAuthority, recover_store_migration, +}; + +#[test] +fn a_corrupt_current_head_cannot_receive_a_version_one_recovery_receipt() +-> Result<(), Box> { + let (sandbox, authority) = open_authority("migration-recovery-current-head")?; + let expected = authority.observe_intent()?; + drop(authority); + let path = sandbox.path().join("HEAD"); + let mut bytes = fs::read(&path)?; + *bytes.last_mut().ok_or("HEAD is empty")? ^= 1; + fs::write(&path, &bytes)?; + let mut recovered = FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + sandbox.path(), + maximum_policy(), + )?; + let error = recover_store_migration(&mut recovered, &expected) + .err() + .ok_or("corrupt HEAD received a version-one recovery receipt")?; + assert!(matches!( + error, + super::StoreMigrationRecoveryError::CurrentVerification { .. } + )); + let source = error + .source() + .and_then(|source| source.downcast_ref::()) + .ok_or("current-state failure lost its I/O boundary")?; + assert!(matches!( + source + .get_ref() + .and_then(|source| source.downcast_ref::()), + Some(FilesystemMigrationAuthorityError::Head { + source: crate::adapters::PublicationHeadDecodeError::ChecksumMismatch { .. } + }) + )); + assert_eq!(fs::read(&path)?, bytes); + assert!(!sandbox.path().join("migration.intent.next").exists()); + drop(recovered); + sandbox.remove()?; + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_fixed_artifact.rs b/src/adapters/store_migration/filesystem_migration_fixed_artifact.rs index 50bca1e5..5a9f52b7 100644 --- a/src/adapters/store_migration/filesystem_migration_fixed_artifact.rs +++ b/src/adapters/store_migration/filesystem_migration_fixed_artifact.rs @@ -4,10 +4,13 @@ use std::io::{self, Write}; use cap_std::fs::{Dir, File}; +use super::filesystem_migration_recovery_refusal::{ + FilesystemMigrationRecoveryRefusal as Refusal, invalid, +}; use super::{format_marker_decoder, migration_intent_format, migration_receipt_format}; use crate::adapters::filesystem_catalog_artifact; use crate::adapters::filesystem_exact_record::{ - self as exact_record, EntryIdentity, ExactRecordError, ExactRecordRefusal, + self as exact_record, EntryIdentity, ExactRecordError, }; #[derive(Clone, Copy, Debug, Eq, PartialEq)] @@ -18,7 +21,7 @@ pub(super) enum FilesystemMigrationFixedArtifact { } impl FilesystemMigrationFixedArtifact { - const fn stage_name(self) -> &'static str { + pub(super) const fn stage_name(self) -> &'static str { match self { Self::Intent => "migration.intent.next", Self::Marker => "FORMAT.next", @@ -26,7 +29,7 @@ impl FilesystemMigrationFixedArtifact { } } - const fn canonical_name(self) -> &'static str { + pub(super) const fn canonical_name(self) -> &'static str { match self { Self::Intent => "migration.intent", Self::Marker => "FORMAT", @@ -34,7 +37,7 @@ impl FilesystemMigrationFixedArtifact { } } - const fn encoded_length(self) -> usize { + pub(super) const fn encoded_length(self) -> usize { match self { Self::Intent => migration_intent_format::ENCODED_LENGTH, Self::Marker => format_marker_decoder::ENCODED_LENGTH, @@ -69,6 +72,31 @@ impl FilesystemMigrationFixedStage { }) } + /// Creates the stage exclusively and writes only `expected[..end]`, + /// leaving an unsynchronized incomplete pre-effect stage behind. The + /// handle is dropped: repository crash tasks kill the process next. + #[cfg(feature = "repository-tasks")] + pub(super) fn create_prefix( + root: &Dir, + artifact: FilesystemMigrationFixedArtifact, + expected: &[u8], + end: usize, + ) -> io::Result<()> { + require_length(artifact, expected)?; + let prefix = expected + .get(..end) + .filter(|prefix| prefix.len() < expected.len()) + .ok_or_else(|| { + invalid(Refusal::StagePrefix { + complete_length: expected.len(), + observed: end, + }) + })?; + let mut file = filesystem_catalog_artifact::create_exclusive(root, artifact.stage_name())?; + file.write_all(prefix)?; + file.flush() + } + pub(super) fn synchronize(&self, root: &Dir) -> io::Result<()> { self.require_handle()?; self.file.sync_all()?; @@ -170,29 +198,59 @@ fn verify_named_record( /// Maps a shared exact-record failure onto this protocol's refusal messages. fn migration_error(error: ExactRecordError) -> io::Error { - match error { - ExactRecordError::Io(source) => source, - ExactRecordError::Refused(refusal) => invalid_data(match refusal { - ExactRecordRefusal::LengthOverflow => "migration fixed-record length exceeded u64", - ExactRecordRefusal::KindOrLength | ExactRecordRefusal::KindLengthOrIdentity => { - "migration fixed-record kind, length, or identity disagreed" - } - ExactRecordRefusal::Bytes | ExactRecordRefusal::TrailingBytes => { - "migration fixed-record bytes disagreed" - } - ExactRecordRefusal::RemainedVisible => "removed migration stage remained visible", - }), - } + error.into_io() } fn require_length(artifact: FilesystemMigrationFixedArtifact, expected: &[u8]) -> io::Result<()> { if expected.len() == artifact.encoded_length() { Ok(()) } else { - Err(invalid_data("migration fixed-record length disagreed")) + Err(invalid(Refusal::RecordLength { + expected: artifact.encoded_length(), + observed: expected.len(), + })) } } fn invalid_data(message: &'static str) -> io::Error { io::Error::new(io::ErrorKind::InvalidData, message) } + +impl FilesystemMigrationFixedStage { + /// Reopens an existing exact stage by identity for a resumed migration. + pub(super) fn reopen_stage( + root: &Dir, + artifact: FilesystemMigrationFixedArtifact, + expected: &[u8], + ) -> io::Result { + Self::reopen(root, artifact, artifact.stage_name(), expected) + } + + /// Reopens an existing exact canonical record by identity, as the + /// published handle a resumed migration verifies against. + pub(super) fn reopen_canonical( + root: &Dir, + artifact: FilesystemMigrationFixedArtifact, + expected: &[u8], + ) -> io::Result { + Self::reopen(root, artifact, artifact.canonical_name(), expected) + } + + fn reopen( + root: &Dir, + artifact: FilesystemMigrationFixedArtifact, + name: &str, + expected: &[u8], + ) -> io::Result { + require_length(artifact, expected)?; + let file = exact_record::open_regular(root, name)?; + let identity = EntryIdentity::of_file(&file)?; + verify_named_record(root, name, expected, identity)?; + Ok(Self { + artifact, + expected: Box::from(expected), + identity, + file, + }) + } +} diff --git a/src/adapters/store_migration/filesystem_migration_namespace.rs b/src/adapters/store_migration/filesystem_migration_namespace.rs index 9b294eb8..0343d15a 100644 --- a/src/adapters/store_migration/filesystem_migration_namespace.rs +++ b/src/adapters/store_migration/filesystem_migration_namespace.rs @@ -94,14 +94,15 @@ pub(super) fn admit_reader_fence(root: &Dir) -> io::Result<()> { verify_reader_root(root) } +/// The number of directories the namespace prefix admits, in order. +pub(super) const PREFIX_DIRECTORY_COUNT: usize = 6; + pub(super) fn admit_namespace_prefix(root: &Dir) -> io::Result<()> { preflight_prefix(root)?; - let retention = PinnedMigrationDirectory::admit(root, RETENTION)?; - let roots = PinnedMigrationDirectory::admit(retention.directory(), ROOTS)?; - let manifests = PinnedMigrationDirectory::admit(retention.directory(), MANIFESTS)?; - let gc = PinnedMigrationDirectory::admit(root, GC)?; - let recovery = PinnedMigrationDirectory::admit(root, RECOVERY)?; - let dispositions = PinnedMigrationDirectory::admit(recovery.directory(), DISPOSITIONS)?; + let prefix = admit_directories(root, PREFIX_DIRECTORY_COUNT)?; + let (retention, roots, manifests, gc, recovery, dispositions) = prefix + .complete() + .ok_or_else(|| ambiguous("namespace prefix admission stopped short"))?; roots.verify(retention.directory())?; manifests.verify(retention.directory())?; dispositions.verify(recovery.directory())?; @@ -111,6 +112,77 @@ pub(super) fn admit_namespace_prefix(root: &Dir) -> io::Result<()> { verify_namespace_prefix(root) } +/// Admits only the first `count` prefix directories, in protocol order, and +/// stops without the final verification. Repository crash tasks use this to +/// leave a store at an exact directory-prefix length. +#[cfg(feature = "repository-tasks")] +pub(super) fn admit_namespace_prefix_partially(root: &Dir, count: usize) -> io::Result<()> { + if count > PREFIX_DIRECTORY_COUNT { + return Err(ambiguous("namespace prefix count exceeds the protocol")); + } + preflight_prefix(root)?; + admit_directories(root, count).map(|_prefix| ()) +} + +/// The prefix directories admitted so far, each pinned by its handle. +#[derive(Default)] +struct AdmittedPrefix { + retention: Option, + roots: Option, + manifests: Option, + gc: Option, + recovery: Option, + dispositions: Option, +} + +type CompletePrefix<'a> = ( + &'a PinnedMigrationDirectory, + &'a PinnedMigrationDirectory, + &'a PinnedMigrationDirectory, + &'a PinnedMigrationDirectory, + &'a PinnedMigrationDirectory, + &'a PinnedMigrationDirectory, +); + +impl AdmittedPrefix { + fn complete(&self) -> Option> { + Some(( + self.retention.as_ref()?, + self.roots.as_ref()?, + self.manifests.as_ref()?, + self.gc.as_ref()?, + self.recovery.as_ref()?, + self.dispositions.as_ref()?, + )) + } +} + +/// Admits the first `count` prefix directories in the normative order: +/// `retention`, `retention/roots`, `retention/manifests`, `gc`, `recovery`, +/// `recovery/dispositions`. Each admission synchronizes its parent. +fn admit_directories(root: &Dir, count: usize) -> io::Result { + let mut prefix = AdmittedPrefix::default(); + for ordinal in 0..count { + match ordinal { + 0 => prefix.retention = Some(PinnedMigrationDirectory::admit(root, RETENTION)?), + 1 => prefix.roots = Some(admit_child(prefix.retention.as_ref(), ROOTS)?), + 2 => prefix.manifests = Some(admit_child(prefix.retention.as_ref(), MANIFESTS)?), + 3 => prefix.gc = Some(PinnedMigrationDirectory::admit(root, GC)?), + 4 => prefix.recovery = Some(PinnedMigrationDirectory::admit(root, RECOVERY)?), + _ => prefix.dispositions = Some(admit_child(prefix.recovery.as_ref(), DISPOSITIONS)?), + } + } + Ok(prefix) +} + +fn admit_child( + parent: Option<&PinnedMigrationDirectory>, + name: &'static str, +) -> io::Result { + let parent = parent.ok_or_else(|| ambiguous("namespace prefix parent was not admitted"))?; + PinnedMigrationDirectory::admit(parent.directory(), name) +} + pub(super) fn verify_intent_root(root: &Dir) -> io::Result<()> { require_v1_and_intent(root)?; require_exact_membership(root, &BEFORE_READER) @@ -152,6 +224,11 @@ pub(super) fn verify_receipt_view(root: &Dir) -> io::Result<()> { fn preflight_prefix(root: &Dir) -> io::Result<()> { require_allowed_membership(root, &PREFIX_ROOT)?; require_base_namespace(root)?; + preflight_recovery_directories(root) +} + +/// Checks nested migration residue before any stage adoption or removal. +pub(super) fn preflight_recovery_directories(root: &Dir) -> io::Result<()> { let retention = optional_directory(root, RETENTION)?; let gc = optional_directory(root, GC)?; let recovery = optional_directory(root, RECOVERY)?; diff --git a/src/adapters/store_migration/filesystem_migration_pair_admission_tests.rs b/src/adapters/store_migration/filesystem_migration_pair_admission_tests.rs new file mode 100644 index 00000000..e6be821c --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_pair_admission_tests.rs @@ -0,0 +1,66 @@ +//! This module owns refusal of substituted stage/canonical pairs before resumption. + +use std::error::Error; +use std::fs; + +use super::filesystem_migration_recovery_tests::version_one_witness; +use super::filesystem_migration_test_fixture::{maximum_policy, open_authority}; +use super::migration_resumption::{MigrationRecords, execute_phase}; +use super::{ + FilesystemStoreMigrationAuthority, StoreMigrationPhase, StoreMigrationRecoveryError, + StoreMigrationStorage, recover_store_migration, +}; +use crate::adapters::filesystem_exact_record::{ExactRecordError, ExactRecordRefusal}; + +#[test] +fn substituted_canonical_pairs_refuse_during_adoption_before_forward_execution() +-> Result<(), Box> { + for (count, canonical, stage) in [ + (3, "migration.intent", "migration.intent.next"), + (12, "FORMAT", "FORMAT.next"), + (18, "migration.receipt", "migration.receipt.next"), + ] { + refuse_pair(count, canonical, stage)?; + } + Ok(()) +} + +fn refuse_pair(count: usize, canonical: &str, stage: &str) -> Result<(), Box> { + let (sandbox, mut authority) = open_authority("migration-pair-admission")?; + let intent = authority.observe_intent()?; + StoreMigrationStorage::verify_current(&mut authority, &intent)?; + let records = MigrationRecords::for_intent(&intent); + for phase in StoreMigrationPhase::ALL.iter().take(count) { + execute_phase(&mut authority, *phase, &records)?; + } + drop(authority); + let bytes = fs::read(sandbox.path().join(canonical))?; + fs::remove_file(sandbox.path().join(canonical))?; + fs::write(sandbox.path().join(canonical), &bytes)?; + let witness = version_one_witness(sandbox.path())?; + let mut recovered = FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + sandbox.path(), + maximum_policy(), + )?; + let expected = recovered.observe_intent()?; + let error = recover_store_migration(&mut recovered, &expected) + .err() + .ok_or("a substituted pair was admitted")?; + let StoreMigrationRecoveryError::Adoption { source } = error else { + return Err(format!("{canonical}: substitution reached a forward phase: {error}").into()); + }; + assert!(matches!( + source + .get_ref() + .and_then(|error| error.downcast_ref::()), + Some(ExactRecordError::Refused( + ExactRecordRefusal::KindLengthOrIdentity + )) + )); + assert_eq!(fs::read(sandbox.path().join(canonical))?, bytes); + assert_eq!(fs::read(sandbox.path().join(stage))?, bytes); + assert_eq!(version_one_witness(sandbox.path())?, witness); + drop(recovered); + sandbox.remove()?; + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_receipt_evidence_tests.rs b/src/adapters/store_migration/filesystem_migration_receipt_evidence_tests.rs new file mode 100644 index 00000000..355bde91 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_receipt_evidence_tests.rs @@ -0,0 +1,79 @@ +//! This module owns the law that recovery receipts preserve observed evidence. + +use std::error::Error; +use std::fs; + +use super::filesystem_migration_test_fixture::{maximum_policy, open_authority}; +use super::migration_resumption::{MigrationRecords, execute_phase}; +use super::{ + FilesystemStoreMigrationAuthority, StoreMigrationPhase, StoreMigrationStorage, + recover_store_migration, +}; + +const DIRECTORIES: [&str; 6] = [ + "retention", + "retention/roots", + "retention/manifests", + "gc", + "recovery", + "recovery/dispositions", +]; + +#[test] +fn recovery_receipts_distinguish_each_observed_namespace_prefix() -> Result<(), Box> { + let (sandbox, mut authority) = open_authority("migration-receipt-prefix-evidence")?; + let intent = authority.observe_intent()?; + StoreMigrationStorage::verify_current(&mut authority, &intent)?; + let records = MigrationRecords::for_intent(&intent); + for phase in StoreMigrationPhase::ALL.iter().take(7) { + execute_phase(&mut authority, *phase, &records)?; + } + drop(authority); + let mut previous = None; + for count in 0..=DIRECTORIES.len() { + for name in DIRECTORIES.iter().take(count) { + fs::create_dir(sandbox.path().join(name))?; + } + let mut recovered = + FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + sandbox.path(), + maximum_policy(), + )?; + let expected = recovered.observe_intent()?; + let receipt = recover_store_migration(&mut recovered, &expected)?; + assert_eq!(receipt.intent_digest(), Some(intent.digest())); + assert_eq!( + receipt.observed_namespace_prefix().len(), + count.checked_add(1).ok_or("prefix overflow")? + ); + let names: Vec<_> = std::iter::once("reader.lock") + .chain(DIRECTORIES.into_iter().take(count)) + .collect(); + assert_eq!( + receipt + .observed_namespace_prefix() + .names() + .collect::>(), + names + ); + if let Some(previous) = previous { + assert_ne!( + receipt, previous, + "distinct observed prefixes collapsed at {count}" + ); + } + previous = Some(receipt); + let complete = recover_store_migration(&mut recovered, &expected)?; + assert_eq!(complete.intent_digest(), Some(intent.digest())); + assert_eq!(complete.observed_namespace_prefix().len(), 7); + assert_eq!(complete.executed_phases().count(), 0); + drop(recovered); + fs::remove_file(sandbox.path().join("FORMAT"))?; + fs::remove_file(sandbox.path().join("migration.receipt"))?; + for name in DIRECTORIES.iter().rev() { + fs::remove_dir(sandbox.path().join(name))?; + } + } + sandbox.remove()?; + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_recovery.rs b/src/adapters/store_migration/filesystem_migration_recovery.rs new file mode 100644 index 00000000..faf00abc --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_recovery.rs @@ -0,0 +1,184 @@ +//! This module binds filesystem migration authority to the recovery port. + +use std::io; +use std::path::Path; + +use cap_std::fs::Dir; + +use super::FilesystemMigrationRecoveryRefusal as Refusal; +use super::filesystem_migration_authority::MigrationNamespacePolicy; +use super::filesystem_migration_authority_error::FilesystemMigrationAuthorityError as Error; +use super::filesystem_migration_fixed_artifact::{ + FilesystemMigrationFixedArtifact as FixedArtifact, FilesystemMigrationFixedStage, +}; +use super::filesystem_migration_recovery_refusal::{invalid, kind}; +use super::migration_resumption::MigrationRecords; +use super::{ + CanonicalStoreMigrationIntent, FilesystemStoreMigrationAuthority, + FilesystemStoreMigrationInventoryReader, StoreMigrationFixedStage, + StoreMigrationRecoveryStorage, StoreMigrationResidue, filesystem_migration_residue, +}; +use crate::adapters::filesystem_exact_record::{self as exact_record, ExactRecordError}; +use crate::adapters::{ + FilesystemWriterLock, SegmentReadPolicy, filesystem_catalog_artifact, + filesystem_initialization_namespace, filesystem_platform_profile, +}; + +impl FilesystemStoreMigrationAuthority { + /// Reacquires writer authority over a root that may carry migration residue. + /// + /// The call admits the production platform, acquires the existing writer + /// lock, and admits the published version-1 namespace plus any subset of + /// migration records, stages, the reader fence, and protocol directories. + /// It never produces a version-1 platform admission, so the version-1 + /// publisher can never run against a partly migrated root. The returned + /// authority observes with the migrating namespace policy and implements + /// [`StoreMigrationRecoveryStorage`]. It mutates nothing. + /// + /// # Errors + /// + /// Returns [`FilesystemMigrationAuthorityError`](Error) at the exact + /// platform, writer-lock, namespace, or pool refusal. + pub fn reopen_for_recovery( + store_root: &Path, + policy: SegmentReadPolicy, + ) -> Result { + let root = filesystem_platform_profile::open(store_root) + .map_err(|source| Error::Platform { source })?; + Self::recover_root(root, policy) + } + + #[cfg(test)] + pub(super) fn reopen_for_recovery_unchecked_for_tests( + store_root: &Path, + policy: SegmentReadPolicy, + ) -> Result { + let root = Dir::open_ambient_dir(store_root, cap_std::ambient_authority()) + .map_err(|source| Error::Platform { source })?; + Self::recover_root(root, policy) + } + + pub(super) fn recover_root(root: Dir, policy: SegmentReadPolicy) -> Result { + let lock = FilesystemWriterLock::try_acquire_in(root) + .map_err(|source| Error::WriterLock { source })?; + let directory = lock + .clone_directory() + .map_err(|source| Error::Namespace { source })?; + filesystem_initialization_namespace::admit_migrating(&directory) + .map_err(|source| Error::Namespace { source })?; + let root_identity = filesystem_platform_profile::root_identity(&directory) + .map_err(|source| Error::RootIdentity { source })?; + let inventory = + FilesystemStoreMigrationInventoryReader::open_locked(lock, root_identity, policy) + .map_err(|source| Error::Inventory { source })?; + Ok(Self::with_policy( + inventory, + MigrationNamespacePolicy::Migrating, + )) + } +} + +impl StoreMigrationRecoveryStorage for FilesystemStoreMigrationAuthority { + fn observe_residue(&mut self) -> io::Result { + filesystem_migration_residue::observe(self.root()) + } + + fn adopt_residue( + &mut self, + residue: &StoreMigrationResidue, + intent: &CanonicalStoreMigrationIntent, + ) -> io::Result<()> { + super::filesystem_migration_namespace::preflight_recovery_directories(self.root()) + .map_err(|source| { + io::Error::new(source.kind(), Refusal::NamespacePreflight { source }) + })?; + let records = MigrationRecords::for_intent(intent); + adopt_artifact( + self, + FixedArtifact::Intent, + residue.intent_stage.as_deref(), + residue.intent.as_deref(), + intent.encoded(), + )?; + adopt_artifact( + self, + FixedArtifact::Marker, + residue.marker_stage.as_deref(), + residue.marker.as_deref(), + records.marker.encoded(), + )?; + adopt_artifact( + self, + FixedArtifact::Receipt, + residue.receipt_stage.as_deref(), + residue.receipt.as_deref(), + records.receipt.encoded(), + ) + } + + fn discard_stage(&mut self, stage: StoreMigrationFixedStage) -> io::Result<()> { + let artifact = match stage { + StoreMigrationFixedStage::Intent => FixedArtifact::Intent, + StoreMigrationFixedStage::Marker => FixedArtifact::Marker, + StoreMigrationFixedStage::Receipt => FixedArtifact::Receipt, + }; + let root = self.root(); + exact_record::require_absent(root, artifact.canonical_name()).map_err(refusal)?; + let metadata = root.symlink_metadata(artifact.stage_name())?; + let complete = u64::try_from(artifact.encoded_length()).map_err(|source| { + invalid(Refusal::StageLengthOverflow { + length: artifact.encoded_length(), + source, + }) + })?; + if !metadata.is_file() || metadata.len() >= complete { + return Err(invalid(Refusal::StageNotIncomplete { + stage, + complete_length: complete, + observed_length: metadata.len(), + observed_kind: kind(&metadata), + })); + } + root.remove_file(artifact.stage_name())?; + exact_record::require_absent(root, artifact.stage_name()).map_err(refusal)?; + filesystem_catalog_artifact::synchronize_directory(root) + } +} + +/// Reopens the handles one artifact's residue implies: an exact stage becomes +/// the active stage (shared with its canonical target when both exist), a +/// canonical record alone becomes the published handle, and an incomplete +/// stage is left for the planner's discard. +fn adopt_artifact( + authority: &mut FilesystemStoreMigrationAuthority, + artifact: FixedArtifact, + stage: Option<&[u8]>, + canonical: Option<&[u8]>, + expected: &[u8], +) -> io::Result<()> { + let root = authority.root(); + if stage.is_some_and(|bytes| bytes == expected) { + if authority.fixed_stage.is_some() { + return Err(invalid(Refusal::MultipleExactStages)); + } + let reopened = FilesystemMigrationFixedStage::reopen_stage(root, artifact, expected)?; + if canonical.is_some() { + reopened.verify_linked(root)?; + } + authority.fixed_stage = Some(reopened); + return Ok(()); + } + if canonical.is_some() { + let reopened = FilesystemMigrationFixedStage::reopen_canonical(root, artifact, expected)?; + match artifact { + FixedArtifact::Intent => authority.published_intent = Some(reopened), + FixedArtifact::Marker => authority.published_marker = Some(reopened), + FixedArtifact::Receipt => authority.published_receipt = Some(reopened), + } + } + Ok(()) +} + +fn refusal(error: ExactRecordError) -> io::Error { + error.into_io() +} diff --git a/src/adapters/store_migration/filesystem_migration_recovery_refusal.rs b/src/adapters/store_migration/filesystem_migration_recovery_refusal.rs new file mode 100644 index 00000000..8da6e67c --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_recovery_refusal.rs @@ -0,0 +1,110 @@ +//! This module owns typed filesystem migration recovery refusals. + +use std::error::Error; +use std::fmt; +use std::num::TryFromIntError; + +use super::StoreMigrationFixedStage; + +/// Kind observed at a fixed migration protocol name without following links. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum FilesystemMigrationResidueKind { + /// A regular file. + RegularFile, + /// A directory. + Directory, + /// A link, device, pipe, or another unsupported kind. + Other, +} + +/// Semantic failure retained as an I/O payload during migration recovery. +#[derive(Debug)] +#[non_exhaustive] +pub enum FilesystemMigrationRecoveryRefusal { + /// Nested migration directories contain invalid or out-of-order residue. + NamespacePreflight { + /// The original filesystem or namespace failure. + source: std::io::Error, + }, + /// The record's bounded observation length cannot be represented. + ResidueBoundOverflow { + /// The canonical record bound. + length: usize, + }, + /// A persistent reader fence is not an empty regular file. + ReaderFence { + /// The observed entry kind; an empty regular file is required. + observed_kind: FilesystemMigrationResidueKind, + /// The observed length; zero is required. + observed_length: u64, + }, + /// A directory-prefix name is not a directory. + NamespaceKind { + /// The observed kind. + observed_kind: FilesystemMigrationResidueKind, + }, + /// The canonical stage bound does not fit the filesystem length. + StageLengthOverflow { + /// The bound that failed conversion. + length: usize, + /// The original conversion failure. + source: TryFromIntError, + }, + /// Only an incomplete regular pre-effect stage can be discarded. + StageNotIncomplete { + /// The stage being considered. + stage: StoreMigrationFixedStage, + /// The first complete length, excluded from discard. + complete_length: u64, + /// The observed entry length. + observed_length: u64, + /// The observed entry kind. + observed_kind: FilesystemMigrationResidueKind, + }, + /// More than one complete fixed stage would be adopted simultaneously. + MultipleExactStages, + /// A crash hook requested a non-strict byte prefix. + StagePrefix { + /// The complete length, excluded from strict prefixes. + complete_length: usize, + /// The requested prefix length. + observed: usize, + }, + /// The supplied fixed record does not have its canonical length. + RecordLength { + /// The canonical length. + expected: usize, + /// The supplied length. + observed: usize, + }, +} + +impl fmt::Display for FilesystemMigrationRecoveryRefusal { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "migration recovery refused: {self:?}") + } +} + +impl Error for FilesystemMigrationRecoveryRefusal { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::NamespacePreflight { source } => Some(source), + Self::StageLengthOverflow { source, .. } => Some(source), + _ => None, + } + } +} + +pub(super) fn kind(metadata: &cap_std::fs::Metadata) -> FilesystemMigrationResidueKind { + if metadata.is_file() { + FilesystemMigrationResidueKind::RegularFile + } else if metadata.is_dir() { + FilesystemMigrationResidueKind::Directory + } else { + FilesystemMigrationResidueKind::Other + } +} + +pub(super) fn invalid(refusal: FilesystemMigrationRecoveryRefusal) -> std::io::Error { + std::io::Error::new(std::io::ErrorKind::InvalidData, refusal) +} diff --git a/src/adapters/store_migration/filesystem_migration_recovery_tests.rs b/src/adapters/store_migration/filesystem_migration_recovery_tests.rs new file mode 100644 index 00000000..3dbd367f --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_recovery_tests.rs @@ -0,0 +1,201 @@ +//! Filesystem migration recovery laws: every forward prefix recovers. + +use std::error::Error; +use std::fs; +use std::io; +use std::path::Path; + +use super::filesystem_migration_test_fixture::{maximum_policy, open_authority}; +use super::migration_resumption::{MigrationRecords, execute_phase}; +use super::{ + AdmittedStoreFormatMarker, AdmittedStoreMigrationIntent, AdmittedStoreMigrationReceipt, + FilesystemStoreMigrationAuthority, StoreMigrationFixedStage, StoreMigrationIntentDecodeError, + StoreMigrationPhase, StoreMigrationRecoveryAmbiguity, StoreMigrationRecoveryError, + StoreMigrationRecoveryPlan, StoreMigrationStorage, recover_store_migration, +}; + +/// Runs the first `count` forward phases in-process, drops the writer, and +/// recovers under a fresh authority. Every prefix must end in exactly one +/// complete migration whose records admit, with every version-1 byte intact. +#[test] +fn every_forward_prefix_recovers_to_one_complete_migration() -> Result<(), Box> { + for count in 0..=StoreMigrationPhase::ALL.len() { + let name = format!("filesystem-migration-recovery-prefix-{count}"); + let (sandbox, mut authority) = open_authority(&name)?; + let intent = authority.observe_intent()?; + let witness = version_one_witness(sandbox.path())?; + StoreMigrationStorage::verify_current(&mut authority, &intent)?; + let records = MigrationRecords::for_intent(&intent); + for phase in StoreMigrationPhase::ALL.iter().take(count) { + execute_phase(&mut authority, *phase, &records)?; + } + drop(authority); + + let mut recovered = + FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + sandbox.path(), + maximum_policy(), + )?; + let expected = recovered.observe_intent()?; + let receipt = recover_store_migration(&mut recovered, &expected) + .map_err(|error| format!("prefix {count}: {error}: {:?}", error.source()))?; + match receipt.plan() { + StoreMigrationRecoveryPlan::Resume { resume } + | StoreMigrationRecoveryPlan::DiscardStage { resume, .. } => { + assert_eq!(receipt.executed_phases().next(), Some(resume)); + assert_eq!( + receipt.executed_phases().last(), + Some(StoreMigrationPhase::SynchronizeRootAfterReceiptCleanup) + ); + } + _ => assert_eq!(receipt.executed_phases().count(), 0), + } + drop(recovered); + + match receipt.plan() { + StoreMigrationRecoveryPlan::VersionOne => { + assert_eq!(count, 0, "only an untouched store admits version one"); + assert!(!sandbox.path().join("migration.intent").exists()); + } + StoreMigrationRecoveryPlan::Complete => { + // The receipt is canonical from phase 20 on; the final root + // synchronization leaves nothing a residue can observe. + assert!(count >= 20, "prefix {count} reported complete"); + } + StoreMigrationRecoveryPlan::Resume { .. } => { + assert!(receipt.published().is_some(), "prefix {count} resumed"); + assert_complete_migration(sandbox.path(), &intent)?; + } + StoreMigrationRecoveryPlan::DiscardStage { .. } => { + return Err(format!("prefix {count} needed a discard for an exact stage").into()); + } + } + if count > 0 { + assert_complete_migration(sandbox.path(), &intent)?; + } + assert_eq!( + version_one_witness(sandbox.path())?, + witness, + "prefix {count}" + ); + sandbox.remove()?; + } + Ok(()) +} + +#[test] +fn a_truncated_intent_stage_is_discarded_and_the_migration_completes() -> Result<(), Box> +{ + let (sandbox, mut authority) = open_authority("filesystem-migration-recovery-truncated")?; + let intent = authority.observe_intent()?; + StoreMigrationStorage::verify_current(&mut authority, &intent)?; + StoreMigrationStorage::write_intent_stage(&mut authority, &intent)?; + drop(authority); + let stage = sandbox.path().join("migration.intent.next"); + let mut bytes = fs::read(&stage)?; + bytes.truncate(100); + fs::write(&stage, &bytes)?; + + let mut recovered = FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + sandbox.path(), + maximum_policy(), + )?; + let expected = recovered.observe_intent()?; + let receipt = recover_store_migration(&mut recovered, &expected)?; + drop(recovered); + + assert_eq!( + receipt.plan(), + StoreMigrationRecoveryPlan::DiscardStage { + stage: StoreMigrationFixedStage::Intent, + resume: StoreMigrationPhase::WriteIntentStage, + } + ); + assert_complete_migration(sandbox.path(), &intent)?; + sandbox.remove()?; + Ok(()) +} + +#[test] +fn a_corrupt_durable_intent_refuses_recovery_before_any_mutation() -> Result<(), Box> { + let (sandbox, mut authority) = open_authority("filesystem-migration-recovery-corrupt")?; + let intent = authority.observe_intent()?; + StoreMigrationStorage::verify_current(&mut authority, &intent)?; + let records = MigrationRecords::for_intent(&intent); + for phase in StoreMigrationPhase::ALL.iter().take(6) { + execute_phase(&mut authority, *phase, &records)?; + } + drop(authority); + let canonical = sandbox.path().join("migration.intent"); + let mut bytes = fs::read(&canonical)?; + let last = bytes.last_mut().ok_or("intent is empty")?; + *last ^= 1; + fs::write(&canonical, &bytes)?; + let before = fs::read_dir(sandbox.path())?.count(); + + let mut recovered = FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + sandbox.path(), + maximum_policy(), + )?; + let expected = recovered.observe_intent()?; + let error = recover_store_migration(&mut recovered, &expected) + .err() + .ok_or("a corrupt durable intent was recovered")?; + drop(recovered); + + assert!(matches!( + error, + StoreMigrationRecoveryError::Ambiguity { + source: StoreMigrationRecoveryAmbiguity::IntentUndecodable { + source: StoreMigrationIntentDecodeError::ChecksumMismatch { .. } + } + } + )); + assert_eq!(fs::read_dir(sandbox.path())?.count(), before); + assert!(!sandbox.path().join("reader.lock").exists()); + sandbox.remove()?; + Ok(()) +} + +pub(super) fn assert_complete_migration( + root: &Path, + intent: &super::CanonicalStoreMigrationIntent, +) -> Result<(), Box> { + let intent_bytes = fs::read(root.join("migration.intent"))?; + let marker_bytes = fs::read(root.join("FORMAT"))?; + let receipt_bytes = fs::read(root.join("migration.receipt"))?; + let admitted = AdmittedStoreMigrationIntent::decode(&intent_bytes)?; + let marker = AdmittedStoreFormatMarker::decode(&marker_bytes)?; + let _receipt = AdmittedStoreMigrationReceipt::decode(&receipt_bytes, &admitted, &marker)?; + assert_eq!(admitted.encoded(), intent.encoded()); + for stage in [ + "migration.intent.next", + "FORMAT.next", + "migration.receipt.next", + ] { + assert!(!root.join(stage).exists(), "{stage} survived recovery"); + } + for directory in [ + "retention/roots", + "retention/manifests", + "gc", + "recovery/dispositions", + ] { + assert!(root.join(directory).is_dir(), "{directory} is absent"); + } + assert_eq!(fs::metadata(root.join("reader.lock"))?.len(), 0); + Ok(()) +} + +pub(super) fn version_one_witness(root: &Path) -> io::Result)>> { + let mut witness = vec![("HEAD".to_owned(), fs::read(root.join("HEAD"))?)]; + for pool in ["segments", "catalogs"] { + for entry in fs::read_dir(root.join(pool))? { + let entry = entry?; + let name = format!("{pool}/{}", entry.file_name().to_string_lossy()); + witness.push((name, fs::read(entry.path())?)); + } + } + witness.sort(); + Ok(witness) +} diff --git a/src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs b/src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs new file mode 100644 index 00000000..3a1c4ec0 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs @@ -0,0 +1,123 @@ +//! This module owns exhaustive strict-stage-prefix recovery and preflight laws. + +use std::error::Error; +use std::fs; + +use super::filesystem_migration_recovery_tests::{assert_complete_migration, version_one_witness}; +use super::filesystem_migration_test_fixture::{maximum_policy, open_authority}; +use super::migration_resumption::{MigrationRecords, execute_phase}; +use super::{ + FilesystemMigrationRecoveryRefusal, FilesystemStoreMigrationAuthority, + StoreMigrationFixedStage, StoreMigrationPhase, StoreMigrationRecoveryError, + StoreMigrationRecoveryPlan, StoreMigrationStorage, recover_store_migration, +}; + +#[test] +fn every_strict_fixed_stage_prefix_is_discarded_without_changing_version_one_bytes() +-> Result<(), Box> { + for (stage, phase, name, length) in [ + ( + StoreMigrationFixedStage::Intent, + StoreMigrationPhase::WriteIntentStage, + "migration.intent.next", + super::MIGRATION_INTENT_LENGTH, + ), + ( + StoreMigrationFixedStage::Marker, + StoreMigrationPhase::WriteMarkerStage, + "FORMAT.next", + super::FORMAT_MARKER_LENGTH, + ), + ( + StoreMigrationFixedStage::Receipt, + StoreMigrationPhase::WriteReceiptStage, + "migration.receipt.next", + super::MIGRATION_RECEIPT_LENGTH, + ), + ] { + for length in 0..length { + recover_truncation(stage, phase, name, length)?; + } + } + Ok(()) +} + +fn recover_truncation( + stage: StoreMigrationFixedStage, + phase: StoreMigrationPhase, + name: &str, + length: usize, +) -> Result<(), Box> { + let (sandbox, mut authority) = open_authority("migration-all-truncations")?; + let intent = authority.observe_intent()?; + let witness = version_one_witness(sandbox.path())?; + StoreMigrationStorage::verify_current(&mut authority, &intent)?; + let records = MigrationRecords::for_intent(&intent); + for current in StoreMigrationPhase::ALL { + execute_phase(&mut authority, current, &records)?; + if current == phase { + break; + } + } + drop(authority); + fs::OpenOptions::new() + .write(true) + .open(sandbox.path().join(name))? + .set_len(u64::try_from(length)?)?; + let mut recovered = FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + sandbox.path(), + maximum_policy(), + )?; + let expected = recovered.observe_intent()?; + let receipt = recover_store_migration(&mut recovered, &expected)?; + assert_eq!( + receipt.plan(), + StoreMigrationRecoveryPlan::DiscardStage { + stage, + resume: phase + } + ); + drop(recovered); + assert_complete_migration(sandbox.path(), &intent)?; + assert_eq!(version_one_witness(sandbox.path())?, witness); + sandbox.remove()?; + Ok(()) +} + +#[test] +fn unexpected_nested_residue_refuses_before_creating_a_marker_stage() -> Result<(), Box> +{ + let (sandbox, mut authority) = open_authority("migration-nested-refusal")?; + let intent = authority.observe_intent()?; + StoreMigrationStorage::verify_current(&mut authority, &intent)?; + let records = MigrationRecords::for_intent(&intent); + for phase in StoreMigrationPhase::ALL.iter().take(8) { + execute_phase(&mut authority, *phase, &records)?; + } + drop(authority); + fs::write(sandbox.path().join("gc/unexpected"), b"preserve")?; + let witness = version_one_witness(sandbox.path())?; + let mut recovered = FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_tests( + sandbox.path(), + maximum_policy(), + )?; + let expected = recovered.observe_intent()?; + let error = recover_store_migration(&mut recovered, &expected) + .err() + .ok_or("unexpected nested residue admitted")?; + let StoreMigrationRecoveryError::Adoption { source } = error else { + return Err("nested residue was not rejected before adoption".into()); + }; + assert!(matches!( + source + .get_ref() + .and_then(|error| error.downcast_ref::()), + Some(FilesystemMigrationRecoveryRefusal::NamespacePreflight { .. }) + )); + assert!(!sandbox.path().join("FORMAT.next").exists()); + assert_eq!(fs::read(sandbox.path().join("gc/unexpected"))?, b"preserve"); + assert_eq!(version_one_witness(sandbox.path())?, witness); + drop(recovered); + sandbox.remove()?; + Ok(()) +} diff --git a/src/adapters/store_migration/filesystem_migration_repository_tasks.rs b/src/adapters/store_migration/filesystem_migration_repository_tasks.rs new file mode 100644 index 00000000..3d91ff05 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_repository_tasks.rs @@ -0,0 +1,99 @@ +//! This module owns the migration authority's repository crash-task hooks. +//! +//! Repository process-death tasks drive the production migration protocol +//! from a store they built without platform admission, stop it at an exact +//! byte or directory prefix, and reopen it for recovery. Nothing here is a +//! separate protocol: each hook runs one production step short, or opens the +//! same authority without the Linux platform check. + +use std::io; +use std::path::Path; + +use cap_std::fs::Dir; + +use super::filesystem_migration_authority::MigrationNamespacePolicy; +use super::filesystem_migration_authority_error::FilesystemMigrationAuthorityError as Error; +use super::filesystem_migration_fixed_artifact::{ + FilesystemMigrationFixedArtifact as FixedArtifact, FilesystemMigrationFixedStage, +}; +use super::{ + FilesystemStoreMigrationAuthority, FilesystemStoreMigrationInventoryReader, + StoreMigrationFixedStage, filesystem_migration_namespace, +}; +use crate::adapters::{FilesystemPlatformAdmission, FilesystemWriterLock, SegmentReadPolicy}; + +impl FilesystemStoreMigrationAuthority { + /// Opens fresh migration authority over a store built without platform + /// admission, for repository process-death tasks. + /// + /// # Errors + /// + /// Returns the same admission and pool failures as [`Self::open`]. + #[doc(hidden)] + pub fn open_unchecked_for_repository_tasks( + lock: FilesystemWriterLock, + policy: SegmentReadPolicy, + ) -> Result { + let admission = FilesystemPlatformAdmission::unchecked_for_repository_tasks(lock) + .map_err(|source| Error::Platform { source })?; + let inventory = FilesystemStoreMigrationInventoryReader::open(admission, policy) + .map_err(|source| Error::Inventory { source })?; + Ok(Self::with_policy( + inventory, + MigrationNamespacePolicy::Published, + )) + } + + /// Reacquires recovery authority without platform admission, for + /// repository process-death tasks; otherwise exactly + /// [`Self::reopen_for_recovery`]. + /// + /// # Errors + /// + /// Returns the same writer-lock, namespace, and pool failures as + /// [`Self::reopen_for_recovery`]. + #[doc(hidden)] + pub fn reopen_for_recovery_unchecked_for_repository_tasks( + store_root: &Path, + policy: SegmentReadPolicy, + ) -> Result { + let root = Dir::open_ambient_dir(store_root, cap_std::ambient_authority()) + .map_err(|source| Error::Platform { source })?; + Self::recover_root(root, policy) + } + + /// Creates `stage` and writes a strict prefix of `record`, leaving an + /// incomplete pre-effect stage exactly as process death during the write + /// would. + /// + /// # Errors + /// + /// Returns the exact length, prefix-bound, creation, or write failure. + #[doc(hidden)] + pub fn write_fixed_stage_prefix_for_repository_tasks( + &mut self, + stage: StoreMigrationFixedStage, + record: &[u8], + prefix: usize, + ) -> io::Result<()> { + let artifact = match stage { + StoreMigrationFixedStage::Intent => FixedArtifact::Intent, + StoreMigrationFixedStage::Marker => FixedArtifact::Marker, + StoreMigrationFixedStage::Receipt => FixedArtifact::Receipt, + }; + FilesystemMigrationFixedStage::create_prefix(self.root(), artifact, record, prefix) + } + + /// Admits only the first `count` namespace-prefix directories in protocol + /// order, each with its parent synchronized, and stops before the final + /// prefix verification. + /// + /// # Errors + /// + /// Returns the exact preflight or admission failure, or refuses a count + /// beyond the six-directory prefix. + #[doc(hidden)] + pub fn admit_namespace_prefix_for_repository_tasks(&mut self, count: usize) -> io::Result<()> { + filesystem_migration_namespace::admit_namespace_prefix_partially(self.root(), count) + } +} diff --git a/src/adapters/store_migration/filesystem_migration_residue.rs b/src/adapters/store_migration/filesystem_migration_residue.rs new file mode 100644 index 00000000..0552d5c4 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_residue.rs @@ -0,0 +1,88 @@ +//! This module owns observing migration residue on one pinned root. + +use std::io; + +use cap_fs_ext::DirExt; +use cap_std::fs::Dir; + +use super::FilesystemMigrationRecoveryRefusal as Refusal; +use super::filesystem_migration_recovery_refusal::{invalid, kind}; +use super::{ + FORMAT_MARKER_LENGTH, MIGRATION_INTENT_LENGTH, MIGRATION_RECEIPT_LENGTH, StoreMigrationResidue, +}; +use crate::adapters::filesystem_exact_record::{self as exact_record, ExactRecordError}; + +const READER_LOCK: &str = "reader.lock"; + +/// Observes every fixed migration name without mutation. +/// +/// Records and stages are read without following links and bounded to one +/// byte more than their canonical length, so an overlong file stays +/// distinguishable from an exact one. A wrong kind refuses. +pub(super) fn observe(root: &Dir) -> io::Result { + Ok(StoreMigrationResidue { + intent_stage: bounded_file(root, "migration.intent.next", MIGRATION_INTENT_LENGTH)?, + intent: bounded_file(root, "migration.intent", MIGRATION_INTENT_LENGTH)?, + reader_fence: reader_fence(root)?, + namespace_prefix: namespace_prefix(root)?, + marker_stage: bounded_file(root, "FORMAT.next", FORMAT_MARKER_LENGTH)?, + marker: bounded_file(root, "FORMAT", FORMAT_MARKER_LENGTH)?, + receipt_stage: bounded_file(root, "migration.receipt.next", MIGRATION_RECEIPT_LENGTH)?, + receipt: bounded_file(root, "migration.receipt", MIGRATION_RECEIPT_LENGTH)?, + }) +} + +fn bounded_file(root: &Dir, name: &str, length: usize) -> io::Result>> { + let bound = length + .checked_add(1) + .ok_or_else(|| invalid(Refusal::ResidueBoundOverflow { length }))?; + exact_record::read_bounded_optional(root, name, bound).map_err(ExactRecordError::into_io) +} + +fn reader_fence(root: &Dir) -> io::Result { + match root.symlink_metadata(READER_LOCK) { + Err(source) if source.kind() == io::ErrorKind::NotFound => Ok(false), + Err(source) => Err(source), + Ok(metadata) if metadata.is_file() && metadata.len() == 0 => Ok(true), + Ok(metadata) => Err(invalid(Refusal::ReaderFence { + observed_kind: kind(&metadata), + observed_length: metadata.len(), + })), + } +} + +fn namespace_prefix(root: &Dir) -> io::Result<[bool; 6]> { + let retention = optional_directory(root, "retention")?; + let (roots, manifests) = match retention.as_ref() { + Some(retention) => ( + optional_directory(retention, "roots")?.is_some(), + optional_directory(retention, "manifests")?.is_some(), + ), + None => (false, false), + }; + let gc = optional_directory(root, "gc")?.is_some(); + let recovery = optional_directory(root, "recovery")?; + let dispositions = match recovery.as_ref() { + Some(recovery) => optional_directory(recovery, "dispositions")?.is_some(), + None => false, + }; + Ok([ + retention.is_some(), + roots, + manifests, + gc, + recovery.is_some(), + dispositions, + ]) +} + +fn optional_directory(parent: &Dir, name: &str) -> io::Result> { + match parent.symlink_metadata(name) { + Err(source) if source.kind() == io::ErrorKind::NotFound => Ok(None), + Err(source) => Err(source), + Ok(metadata) if metadata.is_dir() => parent.open_dir_nofollow(name).map(Some), + Ok(metadata) => Err(invalid(Refusal::NamespaceKind { + observed_kind: kind(&metadata), + })), + } +} diff --git a/src/adapters/store_migration/filesystem_migration_residue_kind_tests.rs b/src/adapters/store_migration/filesystem_migration_residue_kind_tests.rs new file mode 100644 index 00000000..476db544 --- /dev/null +++ b/src/adapters/store_migration/filesystem_migration_residue_kind_tests.rs @@ -0,0 +1,48 @@ +//! This module owns typed refusal of non-regular migration residue. + +#![cfg(unix)] + +use std::error::Error; +use std::fs; +use std::os::unix::fs::symlink; + +use super::StoreMigrationRecoveryStorage; +use super::filesystem_migration_test_fixture::open_authority; +use crate::adapters::filesystem_exact_record::{ExactRecordError, ExactRecordRefusal}; + +#[test] +fn symlink_residue_refuses_by_kind_before_reading_its_target() -> Result<(), Box> { + for name in [ + "migration.intent.next", + "migration.intent", + "FORMAT.next", + "FORMAT", + "migration.receipt.next", + "migration.receipt", + ] { + let (sandbox, mut authority) = open_authority("migration-residue-symlink")?; + let target = sandbox.path().join("HEAD"); + let before = fs::read(&target)?; + symlink("HEAD", sandbox.path().join(name))?; + let error = StoreMigrationRecoveryStorage::observe_residue(&mut authority) + .err() + .ok_or("symlink residue was admitted")?; + assert!( + matches!( + error + .get_ref() + .and_then(|source| source.downcast_ref::()), + Some(ExactRecordError::Refused(ExactRecordRefusal::KindOrLength)) + ), + "{name}: {error:?}" + ); + assert_eq!(fs::read(&target)?, before); + assert_eq!( + fs::read_link(sandbox.path().join(name))?, + std::path::Path::new("HEAD") + ); + drop(authority); + sandbox.remove()?; + } + Ok(()) +} diff --git a/src/adapters/store_migration/migration_namespace_prefix.rs b/src/adapters/store_migration/migration_namespace_prefix.rs new file mode 100644 index 00000000..0ccdf880 --- /dev/null +++ b/src/adapters/store_migration/migration_namespace_prefix.rs @@ -0,0 +1,47 @@ +//! This module owns a validated observation of the migration namespace prefix. + +use super::{MIGRATION_NAMESPACE_PREFIX, StoreMigrationRecoveryAmbiguity, StoreMigrationResidue}; + +/// The exact contiguous namespace prefix observed before recovery writes. +/// +/// This is observation evidence, not a durability claim or storage identity. +/// Only recovery constructs it after validating the residue's ordering. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct StoreMigrationNamespacePrefix { + extent: usize, +} + +impl StoreMigrationNamespacePrefix { + pub(super) fn observe( + residue: &StoreMigrationResidue, + ) -> Result { + residue + .namespace_extent() + .map(|extent| Self { extent }) + .map_err( + |(absent, present)| StoreMigrationRecoveryAmbiguity::NamespaceOutOfOrder { + absent, + present, + }, + ) + } + + /// Returns the number of observed names, including `reader.lock`. + pub const fn len(self) -> usize { + self.extent + } + + /// Returns whether no namespace effect was observed. + pub const fn is_empty(self) -> bool { + self.extent == 0 + } + + /// Iterates the exact observed protocol names in admission order. + /// + /// Iteration allocates nothing and performs no I/O. + pub fn names(self) -> impl Iterator { + std::iter::once("reader.lock") + .chain(MIGRATION_NAMESPACE_PREFIX) + .take(self.extent) + } +} diff --git a/src/adapters/store_migration/migration_recovery_ambiguity.rs b/src/adapters/store_migration/migration_recovery_ambiguity.rs new file mode 100644 index 00000000..015fa64c --- /dev/null +++ b/src/adapters/store_migration/migration_recovery_ambiguity.rs @@ -0,0 +1,86 @@ +//! This boundary module owns migration residue that no lawful plan resolves. + +use super::{ + StoreFormatMarkerDecodeError, StoreMigrationFixedStage, StoreMigrationIntentDecodeError, + StoreMigrationReceiptDecodeError, +}; + +/// A migration effect that can exist only after a durable intent. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum StoreMigrationEffect { + /// `reader.lock` or a prefix directory. + Namespace, + /// `FORMAT` or `FORMAT.next`. + Marker, + /// `migration.receipt` or `migration.receipt.next`. + Receipt, +} + +/// Residue that recovery refuses to interpret. +/// +/// Recovery never guesses which step produced conflicting evidence; every +/// variant names the exact conflict so a human can resolve it. +#[derive(Debug)] +pub enum StoreMigrationRecoveryAmbiguity { + /// A later effect exists although no durable intent does. + EffectBeforeIntent { + /// The earliest effect found. + effect: StoreMigrationEffect, + }, + /// A stage survived an effect that requires its prior removal. + StageAfterEffect { + /// The surviving stage. + stage: StoreMigrationFixedStage, + /// The later effect. + effect: StoreMigrationEffect, + }, + /// A complete-length stage does not decode as a canonical record. + StageUndecodable { + /// Preserved stage decoder failure. + source: super::StoreMigrationStageDecodeError, + /// The stage. + stage: StoreMigrationFixedStage, + }, + /// A stage is longer than its canonical record. + StageOverlong { + /// The stage. + stage: StoreMigrationFixedStage, + /// Observed byte length. + observed: usize, + }, + /// A complete stage and its canonical target, or a complete pre-effect + /// stage and the expected record, carry different bytes. + StageDiffers { + /// The stage. + stage: StoreMigrationFixedStage, + }, + /// `migration.intent` does not decode. + IntentUndecodable { + /// Preserved decode failure. + source: StoreMigrationIntentDecodeError, + }, + /// `migration.intent` binds a different version-1 store or root + /// (any coordinate but the mount identity). + IntentDiffers, + /// `reader.lock` and the directory prefix are not one contiguous prefix. + NamespaceOutOfOrder { + /// Index of the first absent slot (0 is `reader.lock`). + absent: usize, + /// Index of a later present slot. + present: usize, + }, + /// A marker artifact exists before the namespace prefix is complete. + MarkerBeforeNamespace, + /// `FORMAT` does not decode as the registered version-2 marker. + MarkerUndecodable { + /// Preserved decode failure. + source: StoreFormatMarkerDecodeError, + }, + /// A receipt artifact exists before `FORMAT`. + ReceiptBeforeMarker, + /// `migration.receipt` does not bind the observed intent and marker. + ReceiptUndecodable { + /// Preserved decode failure. + source: StoreMigrationReceiptDecodeError, + }, +} diff --git a/src/adapters/store_migration/migration_recovery_ambiguity_display.rs b/src/adapters/store_migration/migration_recovery_ambiguity_display.rs new file mode 100644 index 00000000..f29a23f2 --- /dev/null +++ b/src/adapters/store_migration/migration_recovery_ambiguity_display.rs @@ -0,0 +1,69 @@ +//! This boundary module owns migration-recovery ambiguity formatting. + +use std::error::Error; +use std::fmt; + +use super::StoreMigrationRecoveryAmbiguity; + +impl fmt::Display for StoreMigrationRecoveryAmbiguity { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::EffectBeforeIntent { effect } => write!( + formatter, + "migration {effect:?} effect exists without a durable intent" + ), + Self::StageAfterEffect { stage, effect } => write!( + formatter, + "migration {stage:?} stage survived later {effect:?} effect" + ), + Self::StageUndecodable { stage, .. } => { + write!( + formatter, + "complete migration {stage:?} stage does not decode" + ) + } + Self::StageOverlong { stage, observed } => write!( + formatter, + "migration {stage:?} stage has {observed} bytes, more than its record" + ), + Self::StageDiffers { stage } => { + write!( + formatter, + "migration {stage:?} stage bytes differ from their record" + ) + } + Self::IntentUndecodable { .. } => { + formatter.write_str("migration intent does not decode") + } + Self::IntentDiffers => { + formatter.write_str("migration intent binds a different store or root") + } + Self::NamespaceOutOfOrder { absent, present } => write!( + formatter, + "migration namespace slot {present} exists before slot {absent}" + ), + Self::MarkerBeforeNamespace => { + formatter.write_str("format marker exists before the namespace prefix") + } + Self::MarkerUndecodable { .. } => formatter.write_str("format marker does not decode"), + Self::ReceiptBeforeMarker => { + formatter.write_str("migration receipt exists before the format marker") + } + Self::ReceiptUndecodable { .. } => { + formatter.write_str("migration receipt does not bind the observed records") + } + } + } +} + +impl Error for StoreMigrationRecoveryAmbiguity { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::StageUndecodable { source, .. } => Some(source), + Self::IntentUndecodable { source } => Some(source), + Self::MarkerUndecodable { source } => Some(source), + Self::ReceiptUndecodable { source } => Some(source), + _ => None, + } + } +} diff --git a/src/adapters/store_migration/migration_recovery_execution.rs b/src/adapters/store_migration/migration_recovery_execution.rs new file mode 100644 index 00000000..1f79ce9b --- /dev/null +++ b/src/adapters/store_migration/migration_recovery_execution.rs @@ -0,0 +1,235 @@ +//! This boundary module owns ordered migration recovery: observe, plan, +//! adopt, discard, resume. + +use std::error::Error; +use std::fmt; +use std::io; + +use super::migration_resumption::resume_store_migration; +use super::{ + AdmittedStoreMigrationIntent, CanonicalStoreMigrationIntent, CanonicalStoreMigrationReceipt, + StoreMigrationError, StoreMigrationFixedStage, StoreMigrationRecoveryAmbiguity, + StoreMigrationRecoveryPlan, StoreMigrationRecoveryStorage, StoreMigrationResidue, + plan_store_migration_recovery, +}; + +/// What one recovery found and did. +#[must_use] +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct StoreMigrationRecoveryReceipt { + plan: StoreMigrationRecoveryPlan, + observed_prefix: super::StoreMigrationNamespacePrefix, + intent_digest: Option, + published: Option, +} + +impl StoreMigrationRecoveryReceipt { + /// Returns the plan the residue admitted. + pub const fn plan(&self) -> StoreMigrationRecoveryPlan { + self.plan + } + + /// Returns the exact namespace prefix observed before any recovery writes. + pub const fn observed_namespace_prefix(&self) -> super::StoreMigrationNamespacePrefix { + self.observed_prefix + } + + /// Returns the intent bound by this recovery, including complete observations. + /// + /// An untouched version-one observation has no migration intent. Resumed + /// recovery binds persisted intent bytes, or the newly published intent + /// after discarding an incomplete pre-effect stage. + pub const fn intent_digest(&self) -> Option { + self.intent_digest + } + + /// Returns the migration receipt this recovery published, when it ran + /// the forward protocol to completion. + pub const fn published(&self) -> Option<&CanonicalStoreMigrationReceipt> { + self.published.as_ref() + } + + /// Lists every forward phase this successful recovery executed, in order. + /// + /// A version-one admission or already-complete observation executed no + /// forward phases. Any pre-effect discard is recorded separately in + /// [`Self::plan`]. No allocation or I/O occurs during iteration. + pub fn executed_phases(&self) -> impl Iterator { + let first = match self.plan { + StoreMigrationRecoveryPlan::Resume { resume } + | StoreMigrationRecoveryPlan::DiscardStage { resume, .. } => Some(resume), + StoreMigrationRecoveryPlan::VersionOne | StoreMigrationRecoveryPlan::Complete => None, + }; + super::StoreMigrationPhase::ALL + .into_iter() + .skip_while(move |phase| Some(*phase) != first) + } +} + +/// Failure to recover one interrupted migration. +#[derive(Debug)] +pub enum StoreMigrationRecoveryError { + /// Current writer authority no longer reproduces the caller's expected intent. + CurrentVerification { + /// Preserved current-state refusal or operational failure. + source: io::Error, + }, + /// The residue could not be observed. + Observation { + /// Preserved storage failure. + source: io::Error, + }, + /// The residue matches no lawful recovery row. + Ambiguity { + /// The exact conflict. + source: StoreMigrationRecoveryAmbiguity, + }, + /// The exact stage and canonical handles could not be adopted. + Adoption { + /// Preserved storage failure. + source: io::Error, + }, + /// The incomplete pre-effect stage could not be removed. + Discard { + /// The stage. + stage: StoreMigrationFixedStage, + /// Preserved storage failure. + source: io::Error, + }, + /// A resumed forward phase refused. + Resumption { + /// Preserved phase failure. + source: StoreMigrationError, + }, +} + +/// Recovers one interrupted migration under writer authority. +/// +/// Current writer authority first revalidates `expected`. The residue is then +/// observed once and planned against `expected` (the intent the +/// version-1 store derives today, compared on every restart-stable +/// coordinate), and either admitted as version 1, reported complete, or +/// driven through the remaining forward phases with the persisted intent, +/// never the freshly derived one. Nothing is truncated, replaced, or +/// repaired; an incomplete pre-effect stage is the only artifact removed. +/// +/// # Errors +/// +/// Returns [`StoreMigrationRecoveryError`] at the exact boundary that refused. +/// +/// Resumption is internal: external callers cannot bypass recovery admission. +/// +/// ```compile_fail +/// use keep::resume_store_migration; +/// ``` +pub fn recover_store_migration( + storage: &mut impl StoreMigrationRecoveryStorage, + expected: &CanonicalStoreMigrationIntent, +) -> Result { + storage + .verify_current(expected) + .map_err(|source| StoreMigrationRecoveryError::CurrentVerification { source })?; + let residue = storage + .observe_residue() + .map_err(|source| StoreMigrationRecoveryError::Observation { source })?; + let expected_admitted = + AdmittedStoreMigrationIntent::decode(expected.encoded()).map_err(|source| { + StoreMigrationRecoveryError::Observation { + source: io::Error::new(io::ErrorKind::InvalidData, source), + } + })?; + let plan = plan_store_migration_recovery(&expected_admitted, &residue) + .map_err(|source| StoreMigrationRecoveryError::Ambiguity { source })?; + let observed_prefix = super::StoreMigrationNamespacePrefix::observe(&residue) + .map_err(|source| StoreMigrationRecoveryError::Ambiguity { source })?; + let persisted = persisted_intent(&residue, expected)?; + let intent_digest = + (plan != StoreMigrationRecoveryPlan::VersionOne).then(|| persisted.digest()); + let resume = match plan { + StoreMigrationRecoveryPlan::VersionOne | StoreMigrationRecoveryPlan::Complete => { + return Ok(StoreMigrationRecoveryReceipt { + plan, + observed_prefix, + intent_digest, + published: None, + }); + } + StoreMigrationRecoveryPlan::DiscardStage { stage, resume } => { + storage + .adopt_residue(&residue, &persisted) + .map_err(|source| StoreMigrationRecoveryError::Adoption { source })?; + storage + .discard_stage(stage) + .map_err(|source| StoreMigrationRecoveryError::Discard { stage, source })?; + resume + } + StoreMigrationRecoveryPlan::Resume { resume } => { + storage + .adopt_residue(&residue, &persisted) + .map_err(|source| StoreMigrationRecoveryError::Adoption { source })?; + resume + } + }; + let published = resume_store_migration(storage, &persisted, resume) + .map_err(|source| StoreMigrationRecoveryError::Resumption { source })?; + Ok(StoreMigrationRecoveryReceipt { + plan, + observed_prefix, + intent_digest, + published: Some(published), + }) +} + +/// The intent the resumed phases must publish: the durable one when it +/// exists, else the exact staged one, else the freshly derived one. +fn persisted_intent( + residue: &StoreMigrationResidue, + expected: &CanonicalStoreMigrationIntent, +) -> Result { + let staged = residue + .intent + .as_deref() + .or(residue.intent_stage.as_deref()); + let Some(bytes) = staged else { + return Ok(expected.clone()); + }; + match AdmittedStoreMigrationIntent::decode(bytes) { + Ok(admitted) => Ok(CanonicalStoreMigrationIntent::from_admitted(&admitted)), + Err(_) if residue.intent.is_none() => Ok(expected.clone()), + Err(source) => Err(StoreMigrationRecoveryError::Observation { + source: io::Error::new(io::ErrorKind::InvalidData, source), + }), + } +} + +impl fmt::Display for StoreMigrationRecoveryError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::CurrentVerification { .. } => { + formatter.write_str("current migration authority verification failed") + } + Self::Observation { .. } => formatter.write_str("migration residue observation failed"), + Self::Ambiguity { source } => { + write!(formatter, "migration residue is ambiguous: {source}") + } + Self::Adoption { .. } => formatter.write_str("migration residue adoption failed"), + Self::Discard { stage, .. } => { + write!(formatter, "migration {stage:?} stage discard failed") + } + Self::Resumption { source } => write!(formatter, "resumed migration failed: {source}"), + } + } +} + +impl Error for StoreMigrationRecoveryError { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::CurrentVerification { source } + | Self::Observation { source } + | Self::Adoption { source } + | Self::Discard { source, .. } => Some(source), + Self::Ambiguity { source } => Some(source), + Self::Resumption { source } => Some(source), + } + } +} diff --git a/src/adapters/store_migration/migration_recovery_plan.rs b/src/adapters/store_migration/migration_recovery_plan.rs new file mode 100644 index 00000000..92da9c15 --- /dev/null +++ b/src/adapters/store_migration/migration_recovery_plan.rs @@ -0,0 +1,40 @@ +//! This boundary module owns the lawful responses to migration residue. + +use super::StoreMigrationPhase; + +/// One fixed-name migration stage. +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum StoreMigrationFixedStage { + /// `migration.intent.next`. + Intent, + /// `FORMAT.next`. + Marker, + /// `migration.receipt.next`. + Receipt, +} + +/// The one lawful response to an observed migration residue. +/// +/// A plan is a statement about the residue; executing it is the recovery +/// storage's job. Every resume point is the earliest phase whose effect the +/// residue cannot prove, so re-running from it is idempotent. +#[must_use] +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum StoreMigrationRecoveryPlan { + /// No migration artifact exists: the exact version-1 store admits. + VersionOne, + /// Remove one incomplete pre-effect stage, then resume at `resume`. + DiscardStage { + /// The incomplete stage to remove. + stage: StoreMigrationFixedStage, + /// The forward phase to resume at after removal. + resume: StoreMigrationPhase, + }, + /// Resume the forward protocol at `resume`. + Resume { + /// The earliest phase the residue cannot prove complete. + resume: StoreMigrationPhase, + }, + /// The exact receipt is canonical and no stage remains. + Complete, +} diff --git a/src/adapters/store_migration/migration_recovery_planner.rs b/src/adapters/store_migration/migration_recovery_planner.rs new file mode 100644 index 00000000..521671c2 --- /dev/null +++ b/src/adapters/store_migration/migration_recovery_planner.rs @@ -0,0 +1,272 @@ +//! This boundary module owns storage-independent migration recovery planning. +//! +//! The planner turns one observed residue into the one lawful response the +//! recovery table in `migration-recovery.md` prescribes, or into the exact +//! ambiguity that refuses it. It reads no storage and mutates nothing. + +use super::StoreMigrationStageDecodeError as DecodeError; +use super::migration_recovery_ambiguity::StoreMigrationEffect as Effect; +use super::migration_recovery_residue::NAMESPACE_NAME_COUNT; +use super::{ + AdmittedStoreFormatMarker, AdmittedStoreMigrationIntent, AdmittedStoreMigrationReceipt, + FORMAT_MARKER_LENGTH, MIGRATION_INTENT_LENGTH, MIGRATION_RECEIPT_LENGTH, + StoreMigrationFixedStage as Stage, StoreMigrationPhase as Phase, + StoreMigrationRecoveryAmbiguity as Ambiguity, StoreMigrationRecoveryPlan as Plan, + StoreMigrationResidue, +}; + +/// Plans the one lawful recovery of `residue` against the intent the +/// version-1 store derives today. +/// +/// The expected intent is compared on every coordinate but the mount +/// identity, which is same-process evidence (see `recovery.md`, "Root +/// identity across restart"). +/// +/// # Errors +/// +/// Returns [`StoreMigrationRecoveryAmbiguity`](super::StoreMigrationRecoveryAmbiguity) +/// when the residue matches no table row. +pub fn plan_store_migration_recovery( + expected: &AdmittedStoreMigrationIntent<'_>, + residue: &StoreMigrationResidue, +) -> Result { + let Some(intent_bytes) = residue.intent.as_deref() else { + return plan_before_durable_intent(expected, residue); + }; + let intent = AdmittedStoreMigrationIntent::decode(intent_bytes) + .map_err(|source| Ambiguity::IntentUndecodable { source })?; + if !restart_stable_match(expected, &intent) { + return Err(Ambiguity::IntentDiffers); + } + if let Some(stage) = residue.intent_stage.as_deref() { + let later = if residue.has_namespace_effect() { + Some(Effect::Namespace) + } else if residue.marker_stage.is_some() || residue.marker.is_some() { + Some(Effect::Marker) + } else if residue.has_receipt_effect() { + Some(Effect::Receipt) + } else { + None + }; + if let Some(effect) = later { + return Err(Ambiguity::StageAfterEffect { + stage: Stage::Intent, + effect, + }); + } + return if stage == intent_bytes { + Ok(Plan::Resume { + resume: Phase::SynchronizeRootAfterIntent, + }) + } else { + Err(Ambiguity::StageDiffers { + stage: Stage::Intent, + }) + }; + } + plan_namespace(&intent, residue) +} + +/// Rows one and two: nothing, or an intent stage alone. +fn plan_before_durable_intent( + expected: &AdmittedStoreMigrationIntent<'_>, + residue: &StoreMigrationResidue, +) -> Result { + if residue.has_namespace_effect() { + return Err(Ambiguity::EffectBeforeIntent { + effect: Effect::Namespace, + }); + } + if residue.marker_stage.is_some() || residue.marker.is_some() { + return Err(Ambiguity::EffectBeforeIntent { + effect: Effect::Marker, + }); + } + if residue.has_receipt_effect() { + return Err(Ambiguity::EffectBeforeIntent { + effect: Effect::Receipt, + }); + } + let Some(stage) = residue.intent_stage.as_deref() else { + return Ok(Plan::VersionOne); + }; + match classify_stage(Stage::Intent, stage, MIGRATION_INTENT_LENGTH)? { + StageShape::Incomplete => Ok(Plan::DiscardStage { + stage: Stage::Intent, + resume: Phase::WriteIntentStage, + }), + StageShape::Complete => { + let staged = AdmittedStoreMigrationIntent::decode(stage).map_err(|source| { + Ambiguity::StageUndecodable { + stage: Stage::Intent, + source: DecodeError::Intent { source }, + } + })?; + if restart_stable_match(expected, &staged) { + Ok(Plan::Resume { + resume: Phase::SynchronizeIntentStage, + }) + } else { + Err(Ambiguity::StageDiffers { + stage: Stage::Intent, + }) + } + } + } +} + +/// Rows three and four: a durable intent, then `reader.lock` and the prefix. +fn plan_namespace( + intent: &AdmittedStoreMigrationIntent<'_>, + residue: &StoreMigrationResidue, +) -> Result { + let extent = residue + .namespace_extent() + .map_err(|(absent, present)| Ambiguity::NamespaceOutOfOrder { absent, present })?; + if extent < NAMESPACE_NAME_COUNT { + if residue.marker_stage.is_some() || residue.marker.is_some() { + return Err(Ambiguity::MarkerBeforeNamespace); + } + if residue.has_receipt_effect() { + return Err(Ambiguity::ReceiptBeforeMarker); + } + return Ok(Plan::Resume { + resume: if extent == 0 { + Phase::SynchronizeRootAfterIntentCleanup + } else { + Phase::AdmitNamespacePrefix + }, + }); + } + plan_marker(intent, residue) +} + +/// Rows five and six: the marker stage and the canonical marker. +fn plan_marker( + intent: &AdmittedStoreMigrationIntent<'_>, + residue: &StoreMigrationResidue, +) -> Result { + let Some(marker_bytes) = residue.marker.as_deref() else { + if residue.has_receipt_effect() { + return Err(Ambiguity::ReceiptBeforeMarker); + } + let Some(stage) = residue.marker_stage.as_deref() else { + return Ok(Plan::Resume { + resume: Phase::SynchronizeRootAfterNamespace, + }); + }; + return match classify_stage(Stage::Marker, stage, FORMAT_MARKER_LENGTH)? { + StageShape::Incomplete => Ok(Plan::DiscardStage { + stage: Stage::Marker, + resume: Phase::WriteMarkerStage, + }), + StageShape::Complete => { + AdmittedStoreFormatMarker::decode(stage) + .map(|_| ()) + .map_err(|source| Ambiguity::StageUndecodable { + source: DecodeError::Marker { source }, + stage: Stage::Marker, + })?; + Ok(Plan::Resume { + resume: Phase::SynchronizeMarkerStage, + }) + } + }; + }; + let marker = AdmittedStoreFormatMarker::decode(marker_bytes) + .map_err(|source| Ambiguity::MarkerUndecodable { source })?; + if let Some(stage) = residue.marker_stage.as_deref() { + if residue.has_receipt_effect() { + return Err(Ambiguity::StageAfterEffect { + stage: Stage::Marker, + effect: Effect::Receipt, + }); + } + return if stage == marker_bytes { + Ok(Plan::Resume { + resume: Phase::SynchronizeRootAfterMarker, + }) + } else { + Err(Ambiguity::StageDiffers { + stage: Stage::Marker, + }) + }; + } + plan_receipt(intent, &marker, residue) +} + +/// Row seven: the receipt stage and the canonical receipt. +fn plan_receipt( + intent: &AdmittedStoreMigrationIntent<'_>, + marker: &AdmittedStoreFormatMarker<'_>, + residue: &StoreMigrationResidue, +) -> Result { + let Some(receipt_bytes) = residue.receipt.as_deref() else { + let Some(stage) = residue.receipt_stage.as_deref() else { + return Ok(Plan::Resume { + resume: Phase::SynchronizeRootAfterMarkerCleanup, + }); + }; + return match classify_stage(Stage::Receipt, stage, MIGRATION_RECEIPT_LENGTH)? { + StageShape::Incomplete => Ok(Plan::DiscardStage { + stage: Stage::Receipt, + resume: Phase::WriteReceiptStage, + }), + StageShape::Complete => { + AdmittedStoreMigrationReceipt::decode(stage, intent, marker) + .map(|_| ()) + .map_err(|source| Ambiguity::StageUndecodable { + source: DecodeError::Receipt { source }, + stage: Stage::Receipt, + })?; + Ok(Plan::Resume { + resume: Phase::SynchronizeReceiptStage, + }) + } + }; + }; + AdmittedStoreMigrationReceipt::decode(receipt_bytes, intent, marker) + .map(|_| ()) + .map_err(|source| Ambiguity::ReceiptUndecodable { source })?; + match residue.receipt_stage.as_deref() { + None => Ok(Plan::Complete), + Some(stage) if stage == receipt_bytes => Ok(Plan::Resume { + resume: Phase::SynchronizeRootAfterReceipt, + }), + Some(_) => Err(Ambiguity::StageDiffers { + stage: Stage::Receipt, + }), + } +} + +enum StageShape { + Incomplete, + Complete, +} + +fn classify_stage(stage: Stage, bytes: &[u8], length: usize) -> Result { + match bytes.len().cmp(&length) { + std::cmp::Ordering::Less => Ok(StageShape::Incomplete), + std::cmp::Ordering::Equal => Ok(StageShape::Complete), + std::cmp::Ordering::Greater => Err(Ambiguity::StageOverlong { + stage, + observed: bytes.len(), + }), + } +} + +/// Every intent coordinate but the mount identity. +fn restart_stable_match( + expected: &AdmittedStoreMigrationIntent<'_>, + observed: &AdmittedStoreMigrationIntent<'_>, +) -> bool { + expected.catalog_generation() == observed.catalog_generation() + && expected.catalog_length() == observed.catalog_length() + && expected.catalog_digest() == observed.catalog_digest() + && expected.predecessor_catalog_digest() == observed.predecessor_catalog_digest() + && expected.inventory_digest() == observed.inventory_digest() + && expected.root_device_identity() == observed.root_device_identity() + && expected.root_file_identity() == observed.root_file_identity() + && expected.target_definition_digest() == observed.target_definition_digest() + && expected.store_identifier() == observed.store_identifier() +} diff --git a/src/adapters/store_migration/migration_recovery_residue.rs b/src/adapters/store_migration/migration_recovery_residue.rs new file mode 100644 index 00000000..90d0daf7 --- /dev/null +++ b/src/adapters/store_migration/migration_recovery_residue.rs @@ -0,0 +1,98 @@ +//! This boundary module owns the observed residue of one interrupted +//! migration. + +/// Names of the version-2 directory prefix in creation order, after +/// `reader.lock`. +pub const MIGRATION_NAMESPACE_PREFIX: [&str; 6] = [ + "retention", + "retention/roots", + "retention/manifests", + "gc", + "recovery", + "recovery/dispositions", +]; + +pub(super) const NAMESPACE_NAME_COUNT: usize = MIGRATION_NAMESPACE_PREFIX.len() + 1; + +/// Everything a migration may have left behind, as one observation. +/// +/// An observer reads each fixed name without following links and refuses a +/// wrong file kind, a link, or an unknown entry before producing this value, +/// so the planner sees only bytes and presence. Byte fields carry exactly the +/// bytes observed, bounded by the observer to one byte more than the record's +/// canonical length so overlong records stay distinguishable. +#[must_use] +#[derive(Clone, Debug, Default, Eq, PartialEq)] +pub struct StoreMigrationResidue { + /// Bytes of `migration.intent.next`, when present. + pub intent_stage: Option>, + /// Bytes of `migration.intent`, when present. + pub intent: Option>, + /// Whether the persistent `reader.lock` exists. + pub reader_fence: bool, + /// Presence of each directory in [`MIGRATION_NAMESPACE_PREFIX`] order. + pub namespace_prefix: [bool; MIGRATION_NAMESPACE_PREFIX.len()], + /// Bytes of `FORMAT.next`, when present. + pub marker_stage: Option>, + /// Bytes of `FORMAT`, when present. + pub marker: Option>, + /// Bytes of `migration.receipt.next`, when present. + pub receipt_stage: Option>, + /// Bytes of `migration.receipt`, when present. + pub receipt: Option>, +} + +impl StoreMigrationResidue { + /// The observation of a store with no migration artifact at all. + pub const VERSION_ONE: Self = Self { + intent_stage: None, + intent: None, + reader_fence: false, + namespace_prefix: [false; MIGRATION_NAMESPACE_PREFIX.len()], + marker_stage: None, + marker: None, + receipt_stage: None, + receipt: None, + }; + + /// Whether `reader.lock` or any prefix directory exists. + #[must_use] + pub fn has_namespace_effect(&self) -> bool { + self.reader_fence || self.namespace_prefix.iter().any(|present| *present) + } + + /// Whether any marker or receipt artifact or stage exists. + #[must_use] + pub const fn has_marker_or_receipt_effect(&self) -> bool { + self.marker_stage.is_some() + || self.marker.is_some() + || self.receipt_stage.is_some() + || self.receipt.is_some() + } + + /// Whether any receipt artifact or stage exists. + #[must_use] + pub const fn has_receipt_effect(&self) -> bool { + self.receipt_stage.is_some() || self.receipt.is_some() + } + + /// Length of the contiguous present prefix of `reader.lock` followed by + /// the six directories. + /// + /// # Errors + /// + /// Returns the index of the first absent slot and the index of a later + /// present slot when the prefix is not contiguous. + pub fn namespace_extent(&self) -> Result { + let slots = std::iter::once(self.reader_fence).chain(self.namespace_prefix); + let mut first_absent = None; + for (index, present) in slots.enumerate() { + match (first_absent, present) { + (None, false) => first_absent = Some(index), + (Some(absent), true) => return Err((absent, index)), + _ => {} + } + } + Ok(first_absent.unwrap_or(NAMESPACE_NAME_COUNT)) + } +} diff --git a/src/adapters/store_migration/migration_recovery_storage.rs b/src/adapters/store_migration/migration_recovery_storage.rs new file mode 100644 index 00000000..11b6133b --- /dev/null +++ b/src/adapters/store_migration/migration_recovery_storage.rs @@ -0,0 +1,47 @@ +//! This boundary module owns the blocking capabilities migration recovery +//! needs beyond the forward protocol. + +use std::io; + +use super::{ + CanonicalStoreMigrationIntent, StoreMigrationFixedStage, StoreMigrationResidue, + StoreMigrationStorage, +}; + +/// Blocking storage capabilities for recovering one interrupted migration. +/// +/// An implementation holds exclusive writer authority over a root that may +/// carry any lawful migration residue. It observes without mutation, adopts +/// the exact stage and canonical handles a resume point needs by reopening +/// them by device and inode identity, and removes only an incomplete +/// pre-effect stage the planner named. +pub trait StoreMigrationRecoveryStorage: StoreMigrationStorage { + /// Observes every fixed migration name without mutation. + /// + /// # Errors + /// + /// Returns the exact I/O failure, or a typed refusal for a wrong file + /// kind, a link, or an unknown entry. + fn observe_residue(&mut self) -> io::Result; + + /// Reopens, verifies, and retains every exact stage and canonical record + /// the residue holds, so later phases find the handles the forward + /// protocol would have retained. + /// + /// # Errors + /// + /// Returns the exact reopen, identity, or byte verification failure. + fn adopt_residue( + &mut self, + residue: &StoreMigrationResidue, + intent: &CanonicalStoreMigrationIntent, + ) -> io::Result<()>; + + /// Removes one incomplete pre-effect stage and synchronizes the root. + /// + /// # Errors + /// + /// Returns the exact removal or synchronization failure, or a typed + /// refusal when the stage's canonical target already exists. + fn discard_stage(&mut self, stage: StoreMigrationFixedStage) -> io::Result<()>; +} diff --git a/src/adapters/store_migration/migration_resumption.rs b/src/adapters/store_migration/migration_resumption.rs new file mode 100644 index 00000000..3f37b89a --- /dev/null +++ b/src/adapters/store_migration/migration_resumption.rs @@ -0,0 +1,162 @@ +//! This boundary module owns resuming an interrupted migration at one exact +//! phase. + +use std::io; + +use super::{ + CanonicalStoreFormatMarker, CanonicalStoreMigrationIntent, CanonicalStoreMigrationReceipt, + StoreMigrationError, StoreMigrationPhase, StoreMigrationStorage, +}; + +/// The three canonical records one migration publishes. +pub(super) struct MigrationRecords<'a> { + pub(super) intent: &'a CanonicalStoreMigrationIntent, + pub(super) marker: CanonicalStoreFormatMarker, + pub(super) receipt: CanonicalStoreMigrationReceipt, +} + +impl<'a> MigrationRecords<'a> { + pub(super) fn for_intent(intent: &'a CanonicalStoreMigrationIntent) -> Self { + let marker = CanonicalStoreFormatMarker::version_two(); + let receipt = CanonicalStoreMigrationReceipt::from_canonical(intent, &marker); + Self { + intent, + marker, + receipt, + } + } +} + +/// Resumes one migration at `from` and runs every later phase in order. +/// +/// The storage must already hold the handles the phases before `from` +/// established; a recovery storage adopts them from the observed residue. +/// No current-state verification runs here, because recovery verified the +/// persisted intent before planning. The returned receipt exists only after +/// the final store-root synchronization. +/// +/// # Errors +/// +/// Returns [`StoreMigrationError::Storage`] naming the exact phase that +/// refused. Failure returns no receipt. +pub(super) fn resume_store_migration( + storage: &mut impl StoreMigrationStorage, + intent: &CanonicalStoreMigrationIntent, + from: StoreMigrationPhase, +) -> Result { + let records = MigrationRecords::for_intent(intent); + let start = StoreMigrationPhase::ALL + .iter() + .position(|phase| *phase == from) + .unwrap_or(StoreMigrationPhase::ALL.len()); + for phase in StoreMigrationPhase::ALL.iter().skip(start) { + execute_phase(storage, *phase, &records)?; + } + Ok(records.receipt) +} + +/// Runs exactly one phase against the storage. +pub(super) fn execute_phase( + storage: &mut impl StoreMigrationStorage, + phase: StoreMigrationPhase, + records: &MigrationRecords<'_>, +) -> Result<(), StoreMigrationError> { + let result = match phase { + StoreMigrationPhase::WriteIntentStage + | StoreMigrationPhase::SynchronizeIntentStage + | StoreMigrationPhase::LinkIntent + | StoreMigrationPhase::SynchronizeRootAfterIntent + | StoreMigrationPhase::RemoveIntentStage + | StoreMigrationPhase::SynchronizeRootAfterIntentCleanup + | StoreMigrationPhase::AdmitReaderFence + | StoreMigrationPhase::AdmitNamespacePrefix + | StoreMigrationPhase::SynchronizeRootAfterNamespace => { + intent_and_namespace_phase(storage, phase, records.intent) + } + StoreMigrationPhase::WriteMarkerStage + | StoreMigrationPhase::SynchronizeMarkerStage + | StoreMigrationPhase::LinkMarker + | StoreMigrationPhase::SynchronizeRootAfterMarker + | StoreMigrationPhase::RemoveMarkerStage + | StoreMigrationPhase::SynchronizeRootAfterMarkerCleanup => { + marker_phase(storage, phase, &records.marker) + } + StoreMigrationPhase::WriteReceiptStage + | StoreMigrationPhase::SynchronizeReceiptStage + | StoreMigrationPhase::LinkReceipt + | StoreMigrationPhase::SynchronizeRootAfterReceipt + | StoreMigrationPhase::RemoveReceiptStage + | StoreMigrationPhase::SynchronizeRootAfterReceiptCleanup => { + receipt_phase(storage, phase, &records.receipt) + } + }; + result.map_err(|source| StoreMigrationError::Storage { phase, source }) +} + +fn intent_and_namespace_phase( + storage: &mut impl StoreMigrationStorage, + phase: StoreMigrationPhase, + intent: &CanonicalStoreMigrationIntent, +) -> io::Result<()> { + match phase { + StoreMigrationPhase::WriteIntentStage => storage.write_intent_stage(intent), + StoreMigrationPhase::SynchronizeIntentStage => storage.synchronize_intent_stage(), + StoreMigrationPhase::LinkIntent => storage.link_intent(intent), + StoreMigrationPhase::SynchronizeRootAfterIntent => storage.synchronize_root_after_intent(), + StoreMigrationPhase::RemoveIntentStage => storage.remove_intent_stage(), + StoreMigrationPhase::SynchronizeRootAfterIntentCleanup => { + storage.synchronize_root_after_intent_cleanup() + } + StoreMigrationPhase::AdmitReaderFence => storage.admit_reader_fence(), + StoreMigrationPhase::AdmitNamespacePrefix => storage.admit_namespace_prefix(), + StoreMigrationPhase::SynchronizeRootAfterNamespace => { + storage.synchronize_root_after_namespace() + } + _ => Err(wrong_section(phase)), + } +} + +fn marker_phase( + storage: &mut impl StoreMigrationStorage, + phase: StoreMigrationPhase, + marker: &CanonicalStoreFormatMarker, +) -> io::Result<()> { + match phase { + StoreMigrationPhase::WriteMarkerStage => storage.write_marker_stage(marker), + StoreMigrationPhase::SynchronizeMarkerStage => storage.synchronize_marker_stage(), + StoreMigrationPhase::LinkMarker => storage.link_marker(marker), + StoreMigrationPhase::SynchronizeRootAfterMarker => storage.synchronize_root_after_marker(), + StoreMigrationPhase::RemoveMarkerStage => storage.remove_marker_stage(), + StoreMigrationPhase::SynchronizeRootAfterMarkerCleanup => { + storage.synchronize_root_after_marker_cleanup() + } + _ => Err(wrong_section(phase)), + } +} + +fn receipt_phase( + storage: &mut impl StoreMigrationStorage, + phase: StoreMigrationPhase, + receipt: &CanonicalStoreMigrationReceipt, +) -> io::Result<()> { + match phase { + StoreMigrationPhase::WriteReceiptStage => storage.write_receipt_stage(receipt), + StoreMigrationPhase::SynchronizeReceiptStage => storage.synchronize_receipt_stage(), + StoreMigrationPhase::LinkReceipt => storage.link_receipt(receipt), + StoreMigrationPhase::SynchronizeRootAfterReceipt => { + storage.synchronize_root_after_receipt() + } + StoreMigrationPhase::RemoveReceiptStage => storage.remove_receipt_stage(), + StoreMigrationPhase::SynchronizeRootAfterReceiptCleanup => { + storage.synchronize_root_after_receipt_cleanup() + } + _ => Err(wrong_section(phase)), + } +} + +fn wrong_section(phase: StoreMigrationPhase) -> io::Error { + io::Error::new( + io::ErrorKind::InvalidInput, + format!("migration phase {phase} dispatched to the wrong section"), + ) +} diff --git a/src/adapters/store_migration/migration_stage_decode_error.rs b/src/adapters/store_migration/migration_stage_decode_error.rs new file mode 100644 index 00000000..1b4454b5 --- /dev/null +++ b/src/adapters/store_migration/migration_stage_decode_error.rs @@ -0,0 +1,48 @@ +//! This module owns the original decoder failure of a complete migration stage. + +use std::error::Error; +use std::fmt; + +use super::{ + StoreFormatMarkerDecodeError, StoreMigrationIntentDecodeError, StoreMigrationReceiptDecodeError, +}; + +/// The exact typed decoder failure of a complete fixed stage. +#[derive(Debug)] +pub enum StoreMigrationStageDecodeError { + /// An intent stage failed canonical admission. + Intent { + /// Original decoder failure. + source: StoreMigrationIntentDecodeError, + }, + /// A marker stage failed canonical admission. + Marker { + /// Original decoder failure. + source: StoreFormatMarkerDecodeError, + }, + /// A receipt stage failed canonical admission or record binding. + Receipt { + /// Original decoder failure. + source: StoreMigrationReceiptDecodeError, + }, +} + +impl fmt::Display for StoreMigrationStageDecodeError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + match self { + Self::Intent { source } => write!(formatter, "intent stage: {source}"), + Self::Marker { source } => write!(formatter, "marker stage: {source}"), + Self::Receipt { source } => write!(formatter, "receipt stage: {source}"), + } + } +} + +impl Error for StoreMigrationStageDecodeError { + fn source(&self) -> Option<&(dyn Error + 'static)> { + match self { + Self::Intent { source } => Some(source), + Self::Marker { source } => Some(source), + Self::Receipt { source } => Some(source), + } + } +} diff --git a/src/adapters/store_migration/rationale.md b/src/adapters/store_migration/rationale.md new file mode 100644 index 00000000..2aaeba51 --- /dev/null +++ b/src/adapters/store_migration/rationale.md @@ -0,0 +1,34 @@ +# Migration recovery admission + +Recovery must return the exact named bytes or refuse. A residue snapshot is +untrusted evidence, not writer authority. Planning precedes record adoption, +truncated-stage removal, and all forward writes. Nested directory membership +is checked before adoption so a planner-visible prefix cannot hide foreign +entries that would otherwise fail only after creating the next stage. + +The phase-resumption helper is private to recovery. Exposing it would let +callers manufacture a completion receipt without current-state verification, +residue planning, or adoption. The public recovery entry point owns that order. + +A retained intent stage and a later namespace effect cannot arise from the +ordered protocol: stage removal precedes namespace admission. The analogous +marker-stage/receipt combination also refuses. Treating either as a harmless +cleanup would erase evidence of an ambiguous write history. + +Current authority first verifies the caller's freshly derived expected intent, +including its live mount identity, against the presently pinned root and +version-1 evidence. Planning compares a persisted intent with that expected +intent on restart-stable coordinates; a changed device or inode returns +`IntentDiffers`, while a new mount coordinate alone does not. + +Resume the earliest unproven synchronization with the surviving exact intent's +persisted bytes. If only an incomplete pre-effect intent stage exists, discard +uses the freshly verified expected intent because no complete intent survived. +Mount identity remains a fence for one live authority, not a persisted restart +identity. No format bytes change. Sealed version-1 objects remain untouched. +A completed observation performs no migration writes and does not grant +retention publication authority. + +The 68-case subprocess matrix tests process death, not physical power loss. +Exhaustive strict-stage truncations and forward-prefix laws supplement it; +broader hostile restart combinations remain independently tracked in #111. diff --git a/src/lib.rs b/src/lib.rs index 6611c89b..2e0ffcef 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -38,9 +38,10 @@ //! and the complete synchronization mask. Writer-locked filesystem authority //! now executes one fresh forward migration through exact fixed-record and //! namespace transitions while retaining version-1 immutable bytes. -//! Partial-prefix migration recovery, filesystem retention execution, -//! immutable reader snapshots, and garbage collection remain intentionally -//! absent. +//! Partial-prefix recovery now plans and resumes lawful migration residue, +//! returning typed refusals and an ordered execution receipt. Filesystem +//! retention publication is available; retention restart recovery, immutable +//! reader snapshots, and garbage collection remain absent. #[cfg(test)] extern crate self as keep; @@ -147,6 +148,13 @@ pub use adapters::{ VerifiedRetentionClosure, execute_retention_publication, plan_retention_transition, preflight_retention_transition, prepare_retention_publication, verify_retention_closure, }; +pub use adapters::{ + FilesystemMigrationRecoveryRefusal, FilesystemMigrationResidueKind, MIGRATION_NAMESPACE_PREFIX, + StoreMigrationEffect, StoreMigrationFixedStage, StoreMigrationNamespacePrefix, + StoreMigrationRecoveryAmbiguity, StoreMigrationRecoveryError, StoreMigrationRecoveryPlan, + StoreMigrationRecoveryReceipt, StoreMigrationRecoveryStorage, StoreMigrationResidue, + StoreMigrationStageDecodeError, plan_store_migration_recovery, recover_store_migration, +}; pub use blob::{ BlobHashError, BlobHasher, BlobId, BlobLength, BlobReadError, ByteLength, ByteOffset, ByteRange, ByteRangeError, diff --git a/tests/store_migration_recovery.rs b/tests/store_migration_recovery.rs new file mode 100644 index 00000000..ae94cde8 --- /dev/null +++ b/tests/store_migration_recovery.rs @@ -0,0 +1,424 @@ +//! Storage-independent migration recovery planning laws. +//! +//! One law per row of the recovery table in `migration-recovery.md`, plus +//! one per ambiguity rule, each over the frozen version-2 records. + +mod support; + +use std::error::Error; + +use keep::{ + AdmittedStoreMigrationIntent, StoreMigrationEffect, StoreMigrationFixedStage as Stage, + StoreMigrationPhase as Phase, StoreMigrationRecoveryAmbiguity as Ambiguity, + StoreMigrationRecoveryPlan as Plan, StoreMigrationResidue, plan_store_migration_recovery, +}; + +use crate::support::{domain_hash, patch}; + +const INTENT: &str = include_str!("../conformance/segment-store/v2/migration-intent.hex"); +const MARKER: &str = include_str!("../conformance/segment-store/v2/format-marker.hex"); +const RECEIPT: &str = include_str!("../conformance/segment-store/v2/migration-receipt.hex"); +const INTENT_CHECKSUM_OFFSET: usize = 224; +const ROOT_DEVICE_OFFSET: usize = 136; +const ROOT_MOUNT_OFFSET: usize = 144; + +struct Records { + intent: Vec, + marker: Vec, + receipt: Vec, +} + +#[test] +fn no_artifact_admits_version_one() -> Result<(), Box> { + let records = records()?; + let expected = AdmittedStoreMigrationIntent::decode(&records.intent)?; + assert_eq!( + plan_store_migration_recovery(&expected, &StoreMigrationResidue::VERSION_ONE)?, + Plan::VersionOne + ); + Ok(()) +} + +#[test] +fn an_intent_stage_alone_resumes_when_exact_and_is_discarded_when_incomplete() +-> Result<(), Box> { + let records = records()?; + let expected = AdmittedStoreMigrationIntent::decode(&records.intent)?; + + let exact = StoreMigrationResidue { + intent_stage: Some(records.intent.clone()), + ..StoreMigrationResidue::VERSION_ONE + }; + assert_eq!( + plan_store_migration_recovery(&expected, &exact)?, + Plan::Resume { + resume: Phase::SynchronizeIntentStage + } + ); + + let mut truncated = records.intent.clone(); + truncated.truncate(100); + let incomplete = StoreMigrationResidue { + intent_stage: Some(truncated), + ..StoreMigrationResidue::VERSION_ONE + }; + assert_eq!( + plan_store_migration_recovery(&expected, &incomplete)?, + Plan::DiscardStage { + stage: Stage::Intent, + resume: Phase::WriteIntentStage, + } + ); + + let mut overlong = records.intent.clone(); + overlong.push(0); + let overlong = StoreMigrationResidue { + intent_stage: Some(overlong), + ..StoreMigrationResidue::VERSION_ONE + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &overlong), + Err(Ambiguity::StageOverlong { + stage: Stage::Intent, + observed: 257, + }) + )); + + let mut corrupt = records.intent.clone(); + let last = corrupt.last_mut().ok_or("intent is empty")?; + *last ^= 1; + let corrupt = StoreMigrationResidue { + intent_stage: Some(corrupt), + ..StoreMigrationResidue::VERSION_ONE + }; + let error = plan_store_migration_recovery(&expected, &corrupt) + .err() + .ok_or("corrupt stage admitted")?; + assert!(error.source().and_then(|source| source.source()).is_some()); + assert!(matches!( + error, + Ambiguity::StageUndecodable { + stage: Stage::Intent, + source: keep::StoreMigrationStageDecodeError::Intent { + source: keep::StoreMigrationIntentDecodeError::ChecksumMismatch { .. } + } + } + )); + Ok(()) +} + +#[test] +fn a_durable_intent_resumes_after_its_cleanup_and_binds_restart_stable_coordinates() +-> Result<(), Box> { + let records = records()?; + let expected = AdmittedStoreMigrationIntent::decode(&records.intent)?; + + let durable = StoreMigrationResidue { + intent: Some(records.intent.clone()), + ..StoreMigrationResidue::VERSION_ONE + }; + assert_eq!( + plan_store_migration_recovery(&expected, &durable)?, + Plan::Resume { + resume: Phase::SynchronizeRootAfterIntentCleanup + } + ); + + let with_stage = StoreMigrationResidue { + intent_stage: Some(records.intent.clone()), + ..durable.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &with_stage)?, + Plan::Resume { + resume: Phase::SynchronizeRootAfterIntent + } + ); + + let mut remounted = records.intent.clone(); + patch(&mut remounted, ROOT_MOUNT_OFFSET, &9_u64.to_be_bytes())?; + reseal_intent(&mut remounted)?; + let remounted = AdmittedStoreMigrationIntent::decode(&remounted)?; + assert_eq!( + plan_store_migration_recovery(&remounted, &durable)?, + Plan::Resume { + resume: Phase::SynchronizeRootAfterIntentCleanup + }, + "a rebooted root's new mount id must not reject the store's own intent" + ); + + let mut moved = records.intent.clone(); + patch(&mut moved, ROOT_DEVICE_OFFSET, &9_u64.to_be_bytes())?; + reseal_intent(&mut moved)?; + let moved = AdmittedStoreMigrationIntent::decode(&moved)?; + assert!(matches!( + plan_store_migration_recovery(&moved, &durable), + Err(Ambiguity::IntentDiffers) + )); + + let mut corrupt = durable; + if let Some(bytes) = corrupt.intent.as_mut() + && let Some(last) = bytes.last_mut() + { + *last ^= 1; + } + assert!(matches!( + plan_store_migration_recovery(&expected, &corrupt), + Err(Ambiguity::IntentUndecodable { + source: keep::StoreMigrationIntentDecodeError::ChecksumMismatch { .. } + }) + )); + Ok(()) +} + +#[test] +fn every_effect_needs_a_durable_intent_first() -> Result<(), Box> { + let records = records()?; + let expected = AdmittedStoreMigrationIntent::decode(&records.intent)?; + for (residue, effect) in [ + ( + StoreMigrationResidue { + reader_fence: true, + ..StoreMigrationResidue::VERSION_ONE + }, + StoreMigrationEffect::Namespace, + ), + ( + StoreMigrationResidue { + marker: Some(records.marker.clone()), + ..StoreMigrationResidue::VERSION_ONE + }, + StoreMigrationEffect::Marker, + ), + ( + StoreMigrationResidue { + receipt_stage: Some(records.receipt.clone()), + ..StoreMigrationResidue::VERSION_ONE + }, + StoreMigrationEffect::Receipt, + ), + ] { + assert!(matches!( + plan_store_migration_recovery(&expected, &residue), + Err(Ambiguity::EffectBeforeIntent { effect: observed }) if observed == effect + )); + } + Ok(()) +} + +#[test] +fn the_namespace_prefix_resumes_in_order_and_refuses_gaps() -> Result<(), Box> { + let records = records()?; + let expected = AdmittedStoreMigrationIntent::decode(&records.intent)?; + let durable = StoreMigrationResidue { + intent: Some(records.intent.clone()), + ..StoreMigrationResidue::VERSION_ONE + }; + + let partial = StoreMigrationResidue { + reader_fence: true, + namespace_prefix: [true, true, false, false, false, false], + ..durable.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &partial)?, + Plan::Resume { + resume: Phase::AdmitNamespacePrefix + } + ); + + let complete = StoreMigrationResidue { + reader_fence: true, + namespace_prefix: [true; 6], + ..durable.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &complete)?, + Plan::Resume { + resume: Phase::SynchronizeRootAfterNamespace + } + ); + + let gap = StoreMigrationResidue { + reader_fence: true, + namespace_prefix: [true, false, true, false, false, false], + ..durable.clone() + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &gap), + Err(Ambiguity::NamespaceOutOfOrder { + absent: 2, + present: 3, + }) + )); + + let no_fence = StoreMigrationResidue { + namespace_prefix: [true, false, false, false, false, false], + ..durable + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &no_fence), + Err(Ambiguity::NamespaceOutOfOrder { + absent: 0, + present: 1, + }) + )); + + let early_marker = StoreMigrationResidue { + marker_stage: Some(records.marker.clone()), + ..partial + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &early_marker), + Err(Ambiguity::MarkerBeforeNamespace) + )); + Ok(()) +} + +#[test] +fn the_marker_stage_and_marker_resume_at_their_first_unproven_phase() -> Result<(), Box> +{ + let records = records()?; + let expected = AdmittedStoreMigrationIntent::decode(&records.intent)?; + let namespace = StoreMigrationResidue { + intent: Some(records.intent.clone()), + reader_fence: true, + namespace_prefix: [true; 6], + ..StoreMigrationResidue::VERSION_ONE + }; + + let staged = StoreMigrationResidue { + marker_stage: Some(records.marker.clone()), + ..namespace.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &staged)?, + Plan::Resume { + resume: Phase::SynchronizeMarkerStage + } + ); + + let mut short = records.marker.clone(); + short.truncate(10); + let incomplete = StoreMigrationResidue { + marker_stage: Some(short), + ..namespace.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &incomplete)?, + Plan::DiscardStage { + stage: Stage::Marker, + resume: Phase::WriteMarkerStage, + } + ); + + let linked = StoreMigrationResidue { + marker: Some(records.marker.clone()), + marker_stage: Some(records.marker.clone()), + ..namespace.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &linked)?, + Plan::Resume { + resume: Phase::SynchronizeRootAfterMarker + } + ); + + let published = StoreMigrationResidue { + marker: Some(records.marker.clone()), + ..namespace.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &published)?, + Plan::Resume { + resume: Phase::SynchronizeRootAfterMarkerCleanup + } + ); + + let early_receipt = StoreMigrationResidue { + receipt: Some(records.receipt.clone()), + ..namespace + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &early_receipt), + Err(Ambiguity::ReceiptBeforeMarker) + )); + Ok(()) +} + +#[test] +fn the_receipt_completes_the_migration_only_when_exact_and_alone() -> Result<(), Box> { + let records = records()?; + let expected = AdmittedStoreMigrationIntent::decode(&records.intent)?; + let marked = StoreMigrationResidue { + intent: Some(records.intent.clone()), + reader_fence: true, + namespace_prefix: [true; 6], + marker: Some(records.marker.clone()), + ..StoreMigrationResidue::VERSION_ONE + }; + + let staged = StoreMigrationResidue { + receipt_stage: Some(records.receipt.clone()), + ..marked.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &staged)?, + Plan::Resume { + resume: Phase::SynchronizeReceiptStage + } + ); + + let linked = StoreMigrationResidue { + receipt: Some(records.receipt.clone()), + receipt_stage: Some(records.receipt.clone()), + ..marked.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &linked)?, + Plan::Resume { + resume: Phase::SynchronizeRootAfterReceipt + } + ); + + let complete = StoreMigrationResidue { + receipt: Some(records.receipt.clone()), + ..marked.clone() + }; + assert_eq!( + plan_store_migration_recovery(&expected, &complete)?, + Plan::Complete + ); + + let mut conflicting = records.receipt.clone(); + let last = conflicting.last_mut().ok_or("receipt is empty")?; + *last ^= 1; + let conflicting = StoreMigrationResidue { + receipt: Some(conflicting), + ..marked + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &conflicting), + Err(Ambiguity::ReceiptUndecodable { + source: keep::StoreMigrationReceiptDecodeError::ChecksumMismatch { .. } + }) + )); + Ok(()) +} + +fn records() -> Result> { + Ok(Records { + intent: support::decode_hex(INTENT.trim_end())?, + marker: support::decode_hex(MARKER.trim_end())?, + receipt: support::decode_hex(RECEIPT.trim_end())?, + }) +} + +fn reseal_intent(bytes: &mut [u8]) -> Result<(), Box> { + let preimage = bytes + .get(..INTENT_CHECKSUM_OFFSET) + .ok_or("intent lacks its checksum preimage")?; + let checksum = domain_hash(b"keep.store-migration-intent-checksum/v2\0", preimage); + patch(bytes, INTENT_CHECKSUM_OFFSET, &checksum)?; + Ok(()) +} diff --git a/tests/store_migration_recovery_order.rs b/tests/store_migration_recovery_order.rs new file mode 100644 index 00000000..43d0ba37 --- /dev/null +++ b/tests/store_migration_recovery_order.rs @@ -0,0 +1,121 @@ +//! This module owns refusal of migration stages surviving later effects. + +mod support; + +use keep::{ + AdmittedStoreMigrationIntent, MIGRATION_NAMESPACE_PREFIX, StoreMigrationEffect, + StoreMigrationFixedStage, StoreMigrationRecoveryAmbiguity, StoreMigrationResidue, + plan_store_migration_recovery, +}; +use std::error::Error; + +#[test] +fn an_intent_stage_cannot_survive_reader_fence_creation() -> Result<(), Box> { + let intent = support::decode_hex( + include_str!("../conformance/segment-store/v2/migration-intent.hex").trim(), + )?; + let expected = AdmittedStoreMigrationIntent::decode(&intent)?; + let residue = StoreMigrationResidue { + intent: Some(intent.clone()), + intent_stage: Some(intent.clone()), + reader_fence: true, + ..StoreMigrationResidue::VERSION_ONE + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &residue), + Err(StoreMigrationRecoveryAmbiguity::StageAfterEffect { + stage: StoreMigrationFixedStage::Intent, + effect: StoreMigrationEffect::Namespace + }) + )); + Ok(()) +} + +#[test] +fn a_marker_stage_cannot_survive_receipt_publication() -> Result<(), Box> { + let intent = support::decode_hex( + include_str!("../conformance/segment-store/v2/migration-intent.hex").trim(), + )?; + let marker = support::decode_hex( + include_str!("../conformance/segment-store/v2/format-marker.hex").trim(), + )?; + let receipt = support::decode_hex( + include_str!("../conformance/segment-store/v2/migration-receipt.hex").trim(), + )?; + let expected = AdmittedStoreMigrationIntent::decode(&intent)?; + let residue = StoreMigrationResidue { + intent: Some(intent.clone()), + reader_fence: true, + namespace_prefix: [true; 6], + marker: Some(marker.clone()), + marker_stage: Some(marker), + receipt: Some(receipt), + ..StoreMigrationResidue::VERSION_ONE + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &residue), + Err(StoreMigrationRecoveryAmbiguity::StageAfterEffect { + stage: StoreMigrationFixedStage::Marker, + effect: StoreMigrationEffect::Receipt + }) + )); + Ok(()) +} + +#[test] +fn a_receipt_only_later_effect_is_reported_as_receipt() -> Result<(), Box> { + let intent = support::decode_hex( + include_str!("../conformance/segment-store/v2/migration-intent.hex").trim(), + )?; + let receipt = support::decode_hex( + include_str!("../conformance/segment-store/v2/migration-receipt.hex").trim(), + )?; + let expected = AdmittedStoreMigrationIntent::decode(&intent)?; + let residue = StoreMigrationResidue { + intent: Some(intent.clone()), + intent_stage: Some(intent.clone()), + receipt: Some(receipt), + ..StoreMigrationResidue::VERSION_ONE + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &residue), + Err(StoreMigrationRecoveryAmbiguity::StageAfterEffect { + stage: StoreMigrationFixedStage::Intent, + effect: StoreMigrationEffect::Receipt, + }) + )); + Ok(()) +} + +#[test] +fn a_receipt_before_the_marker_reports_the_missing_marker_at_every_partial_namespace() +-> Result<(), Box> { + let intent = support::decode_hex( + include_str!("../conformance/segment-store/v2/migration-intent.hex").trim(), + )?; + let receipt = support::decode_hex( + include_str!("../conformance/segment-store/v2/migration-receipt.hex").trim(), + )?; + let expected = AdmittedStoreMigrationIntent::decode(&intent)?; + for extent in 0..=MIGRATION_NAMESPACE_PREFIX.len() { + for (receipt_stage, canonical) in + [(Some(receipt.clone()), None), (None, Some(receipt.clone()))] + { + let residue = StoreMigrationResidue { + intent: Some(intent.clone()), + reader_fence: extent > 0, + namespace_prefix: std::array::from_fn(|position: usize| { + position.checked_add(1).is_some_and(|next| next < extent) + }), + receipt_stage, + receipt: canonical, + ..StoreMigrationResidue::VERSION_ONE + }; + assert!(matches!( + plan_store_migration_recovery(&expected, &residue), + Err(StoreMigrationRecoveryAmbiguity::ReceiptBeforeMarker) + )); + } + } + Ok(()) +} diff --git a/tests/support/byte_patches.rs b/tests/support/byte_patches.rs new file mode 100644 index 00000000..beb80580 --- /dev/null +++ b/tests/support/byte_patches.rs @@ -0,0 +1,30 @@ +//! This module owns bounded fixture mutation and independent checksum preimages. + +use std::io; + +/// Patches one field without extending the fixture. +/// +/// # Errors +/// Returns a fixture error for overflow or an out-of-bounds field. +pub(crate) fn patch(bytes: &mut [u8], offset: usize, value: &[u8]) -> io::Result<()> { + let end = offset + .checked_add(value.len()) + .ok_or_else(|| io::Error::other("fixture offset overflow"))?; + bytes + .get_mut(offset..end) + .ok_or_else(|| io::Error::other("fixture field absent"))? + .copy_from_slice(value); + Ok(()) +} + +/// Hashes the fixture's named domain and canonical preimage independently. +/// +/// The caller supplies the exact registered, NUL-terminated domain. Its +/// terminator separates the fixed domain from the preimage; this helper hashes +/// their bytes unchanged and adds no length prefix or delimiter. +pub(crate) fn domain_hash(domain: &[u8], preimage: &[u8]) -> [u8; 32] { + let mut hasher = blake3::Hasher::new(); + hasher.update(domain); + hasher.update(preimage); + *hasher.finalize().as_bytes() +} diff --git a/tests/support/mod.rs b/tests/support/mod.rs index 9f143055..2a72bd69 100644 --- a/tests/support/mod.rs +++ b/tests/support/mod.rs @@ -6,6 +6,7 @@ reason = "shared fixtures are crate-visible across binary integration-test modules" )] +pub(crate) mod byte_patches; pub(crate) mod byte_readers; pub(crate) mod byte_writers; @@ -13,6 +14,7 @@ use std::io; use keep::{ChunkSpan, FastCdc}; +pub(crate) use byte_patches::{domain_hash, patch}; pub(crate) use byte_readers::{FailingReader, LyingReader, PartitionReader}; pub(crate) use byte_writers::{ FailingWriter, LyingWriter, PartitionWriter, PrefixThenFailWriter, ZeroWriter, diff --git a/xtask/src/durability_crash_case.rs b/xtask/src/durability_crash_case.rs index 3b267d47..71a1b10c 100644 --- a/xtask/src/durability_crash_case.rs +++ b/xtask/src/durability_crash_case.rs @@ -2,7 +2,7 @@ use crate::{ DurabilityCrashCaseError, DurabilityCrashOccurrence, DurabilityCrashPoint, - DurabilityCrashPosition, + DurabilityCrashPosition, DurabilityCrashSequence, }; /// One validated process-death coordinate in the durability crash matrix. @@ -21,12 +21,19 @@ impl DurabilityCrashCase { /// Returns [`DurabilityCrashCaseError::MissingOccurrence`] when a repeated /// transition lacks an occurrence, or /// [`DurabilityCrashCaseError::UnexpectedOccurrence`] when a non-repeated - /// transition receives one. + /// transition receives one. Namespace admission also returns + /// [`DurabilityCrashCaseError::OccurrenceOutOfRange`] outside its six + /// `during` occurrences or sole `before` and `after` occurrences. pub const fn new( point: DurabilityCrashPoint, position: DurabilityCrashPosition, occurrence: Option, ) -> Result { + if let Some(observed) = occurrence + && let Err(source) = validate_fixed_range(point, position, observed) + { + return Err(source); + } match (point.occurrence_counted(), occurrence) { (true, None) => Err(DurabilityCrashCaseError::MissingOccurrence { point }), (false, Some(observed)) => { @@ -41,11 +48,38 @@ impl DurabilityCrashCase { } /// Returns every canonical case in point-major, position-minor order. + /// + /// A boundary with more than one `during` occurrence contributes one + /// `during` case per occurrence, in occurrence order, between its + /// `before` and `after` cases. pub fn all() -> impl Iterator { DurabilityCrashPoint::ALL.into_iter().flat_map(|point| { DurabilityCrashPosition::ALL .into_iter() - .map(move |position| Self::canonical(point, position)) + .flat_map(move |position| Self::canonical_at(point, position)) + }) + } + + /// Returns every canonical case whose boundary belongs to `sequence`. + pub fn in_sequence(sequence: DurabilityCrashSequence) -> impl Iterator { + Self::all().filter(move |case| case.point().sequence() == sequence) + } + + fn canonical_at( + point: DurabilityCrashPoint, + position: DurabilityCrashPosition, + ) -> impl Iterator { + let occurrences = if position == DurabilityCrashPosition::During { + point.during_occurrences() + } else { + 1 + }; + (0..occurrences).map(move |ordinal| { + let mut case = Self::canonical(point, position); + if point.occurrence_counted() { + case.occurrence = Some(DurabilityCrashOccurrence::new(ordinal)); + } + case }) } @@ -79,3 +113,26 @@ impl DurabilityCrashCase { } } } + +const fn validate_fixed_range( + point: DurabilityCrashPoint, + position: DurabilityCrashPosition, + observed: DurabilityCrashOccurrence, +) -> Result<(), DurabilityCrashCaseError> { + if !matches!(point, DurabilityCrashPoint::MigrationAdmitNamespacePrefix) { + return Ok(()); + } + let exclusive_limit = if matches!(position, DurabilityCrashPosition::During) { + point.during_occurrences() + } else { + 1 + }; + if observed.get() >= exclusive_limit { + return Err(DurabilityCrashCaseError::OccurrenceOutOfRange { + point, + observed, + exclusive_limit, + }); + } + Ok(()) +} diff --git a/xtask/src/durability_crash_case_error.rs b/xtask/src/durability_crash_case_error.rs index 13c053e3..d044a0c5 100644 --- a/xtask/src/durability_crash_case_error.rs +++ b/xtask/src/durability_crash_case_error.rs @@ -8,6 +8,15 @@ use crate::{DurabilityCrashOccurrence, DurabilityCrashPoint}; /// A failure to construct a valid crash-matrix coordinate. #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub enum DurabilityCrashCaseError { + /// A coordinate exceeds a protocol-fixed occurrence range. + OccurrenceOutOfRange { + /// The transition with the fixed range. + point: DurabilityCrashPoint, + /// The rejected zero-based occurrence. + observed: DurabilityCrashOccurrence, + /// The exclusive upper bound for this position. + exclusive_limit: u32, + }, /// A repeated durability transition lacks its occurrence coordinate. MissingOccurrence { /// The repeated transition missing its coordinate. @@ -25,6 +34,16 @@ pub enum DurabilityCrashCaseError { impl fmt::Display for DurabilityCrashCaseError { fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { match self { + Self::OccurrenceOutOfRange { + point, + observed, + exclusive_limit, + } => write!( + formatter, + "{} occurrence {} must be less than {exclusive_limit}", + point.identifier(), + observed.get() + ), Self::MissingOccurrence { point } => { write!(formatter, "{} requires an occurrence", point.identifier()) } diff --git a/xtask/src/durability_crash_matrix.rs b/xtask/src/durability_crash_matrix.rs index aca8546e..c762b0e5 100644 --- a/xtask/src/durability_crash_matrix.rs +++ b/xtask/src/durability_crash_matrix.rs @@ -12,9 +12,11 @@ use std::path::Path; pub(crate) use error::DurabilityCrashMatrixError; use xtask::{ DurabilityCrashCase, DurabilityCrashOccurrence, DurabilityCrashPoint, DurabilityCrashPosition, + DurabilityCrashSequence, }; const CASE_ARGUMENT: &str = "--case"; +const SEQUENCE_ARGUMENT: &str = "--sequence"; pub(crate) fn run( repository_root: &Path, @@ -26,6 +28,14 @@ pub(crate) fn run( } return Ok(()); }; + if flag == OsStr::new(SEQUENCE_ARGUMENT) { + let sequence = parse_sequence(&mut arguments)?; + refuse_extra(&mut arguments)?; + for case in DurabilityCrashCase::in_sequence(sequence) { + run_case(repository_root, case)?; + } + return Ok(()); + } if flag != OsStr::new(CASE_ARGUMENT) { return Err(DurabilityCrashMatrixError::Usage); } @@ -44,6 +54,8 @@ pub(crate) fn run_child( child::run(case, Path::new(&case_root), Path::new(&readiness_socket)) } +/// Parses `POINT POSITION [OCCURRENCE]`; the occurrence is read only for an +/// occurrence-counted boundary and defaults to the first occurrence. fn parse_case( arguments: &mut impl Iterator, ) -> Result { @@ -59,13 +71,40 @@ fn parse_case( .ok_or(DurabilityCrashMatrixError::InvalidPositionEncoding)?; let position = DurabilityCrashPosition::from_identifier(position_text) .ok_or_else(|| DurabilityCrashMatrixError::UnknownPosition(position_text.into()))?; - let occurrence = point - .occurrence_counted() - .then_some(DurabilityCrashOccurrence::FIRST); + let occurrence = if point.occurrence_counted() { + Some(parse_occurrence(arguments)?) + } else { + None + }; DurabilityCrashCase::new(point, position, occurrence) .map_err(DurabilityCrashMatrixError::InvalidCase) } +fn parse_occurrence( + arguments: &mut impl Iterator, +) -> Result { + let Some(argument) = arguments.next() else { + return Ok(DurabilityCrashOccurrence::FIRST); + }; + let text = argument + .to_str() + .ok_or(DurabilityCrashMatrixError::InvalidOccurrenceEncoding)?; + text.parse() + .map(DurabilityCrashOccurrence::new) + .map_err(|_| DurabilityCrashMatrixError::UnknownOccurrence(text.into())) +} + +fn parse_sequence( + arguments: &mut impl Iterator, +) -> Result { + let argument = arguments.next().ok_or(DurabilityCrashMatrixError::Usage)?; + let text = argument + .to_str() + .ok_or(DurabilityCrashMatrixError::InvalidSequenceEncoding)?; + DurabilityCrashSequence::from_identifier(text) + .ok_or_else(|| DurabilityCrashMatrixError::UnknownSequence(text.into())) +} + fn refuse_extra( arguments: &mut impl Iterator, ) -> Result<(), DurabilityCrashMatrixError> { diff --git a/xtask/src/durability_crash_matrix/child.rs b/xtask/src/durability_crash_matrix/child.rs index 4c664853..fa474fa9 100644 --- a/xtask/src/durability_crash_matrix/child.rs +++ b/xtask/src/durability_crash_matrix/child.rs @@ -37,10 +37,15 @@ fn write_marker( } pub(super) fn marker(case: DurabilityCrashCase) -> Vec { - format!( - "{}\t{}\n", + let mut marker = format!( + "{}\t{}", case.point().identifier(), case.position().identifier() - ) - .into_bytes() + ); + if let Some(occurrence) = case.occurrence() { + marker.push('\t'); + marker.push_str(&occurrence.get().to_string()); + } + marker.push('\n'); + marker.into_bytes() } diff --git a/xtask/src/durability_crash_matrix/error.rs b/xtask/src/durability_crash_matrix/error.rs index d0a7ec46..4c76aaac 100644 --- a/xtask/src/durability_crash_matrix/error.rs +++ b/xtask/src/durability_crash_matrix/error.rs @@ -7,15 +7,18 @@ use std::error::Error; use std::io; use std::time::Duration; +use keep::StoreMigrationRecoveryPlan; use xtask::protocol_admission::HexError; use xtask::{ - DurabilityCrashCase, DurabilityCrashCaseError, DurabilityCrashPoint, DurabilityCrashPosition, + DurabilityCrashCase, DurabilityCrashCaseError, DurabilityCrashOccurrence, DurabilityCrashPoint, + DurabilityCrashPosition, }; pub(crate) enum DurabilityCrashMatrixError { Case { point: DurabilityCrashPoint, position: DurabilityCrashPosition, + occurrence: Option, source: Box, }, ArtifactBytesMismatch { @@ -50,8 +53,10 @@ pub(crate) enum DurabilityCrashMatrixError { artifact: &'static str, }, InvalidCase(DurabilityCrashCaseError), + InvalidOccurrenceEncoding, InvalidPointEncoding, InvalidPositionEncoding, + InvalidSequenceEncoding, InvalidReadinessSignal { observed: u8, }, @@ -78,6 +83,10 @@ pub(crate) enum DurabilityCrashMatrixError { PointSequenceMismatch { point: DurabilityCrashPoint, }, + RecoveryPlanMismatch { + expected: StoreMigrationRecoveryPlan, + observed: StoreMigrationRecoveryPlan, + }, RepeatedInventoryPath { path: String, }, @@ -93,8 +102,10 @@ pub(crate) enum DurabilityCrashMatrixError { expected: &'static str, observed: &'static str, }, + UnknownOccurrence(String), UnknownPoint(String), UnknownPosition(String), + UnknownSequence(String), Usage, Verification { phase: &'static str, @@ -127,6 +138,7 @@ impl DurabilityCrashMatrixError { Self::Case { point: case.point(), position: case.position(), + occurrence: case.occurrence(), source: Box::new(self), } } @@ -147,20 +159,25 @@ impl Error for DurabilityCrashMatrixError { | Self::FixtureLength { .. } | Self::FixtureRange | Self::FixtureTerminator { .. } + | Self::InvalidOccurrenceEncoding | Self::InvalidPointEncoding | Self::InvalidPositionEncoding + | Self::InvalidSequenceEncoding | Self::InvalidReadinessSignal { .. } | Self::InventoryMismatch { .. } | Self::HardLinkIdentityMismatch { .. } | Self::MissingVisibleRecord { .. } | Self::NonUnicodeStatePath | Self::PointSequenceMismatch { .. } + | Self::RecoveryPlanMismatch { .. } | Self::RepeatedInventoryPath { .. } | Self::SnapshotGenerationMismatch { .. } | Self::Timeout { .. } | Self::UnexpectedArtifactKind { .. } + | Self::UnknownOccurrence(_) | Self::UnknownPoint(_) | Self::UnknownPosition(_) + | Self::UnknownSequence(_) | Self::Usage => None, } } diff --git a/xtask/src/durability_crash_matrix/error/display.rs b/xtask/src/durability_crash_matrix/error/display.rs index 79f24e82..f269fa17 100644 --- a/xtask/src/durability_crash_matrix/error/display.rs +++ b/xtask/src/durability_crash_matrix/error/display.rs @@ -18,6 +18,7 @@ impl fmt::Display for DurabilityCrashMatrixError { | Self::HardLinkIdentityMismatch { .. } | Self::InventoryMismatch { .. } | Self::MissingVisibleRecord { .. } + | Self::RecoveryPlanMismatch { .. } | Self::RepeatedInventoryPath { .. } | Self::SnapshotGenerationMismatch { .. } | Self::UnexpectedArtifactKind { .. } => format_state(self, formatter), @@ -31,10 +32,14 @@ impl fmt::Display for DurabilityCrashMatrixError { | Self::FixtureTerminator { .. } => format_fixture(self, formatter), Self::Case { .. } | Self::InvalidCase(_) + | Self::InvalidOccurrenceEncoding | Self::InvalidPointEncoding | Self::InvalidPositionEncoding + | Self::InvalidSequenceEncoding + | Self::UnknownOccurrence(_) | Self::UnknownPoint(_) | Self::UnknownPosition(_) + | Self::UnknownSequence(_) | Self::Usage => format_command(self, formatter), Self::Io { .. } | Self::NonUnicodeStatePath @@ -82,6 +87,10 @@ fn format_state( "post-crash snapshot lacks visible record `{record}`" ) } + DurabilityCrashMatrixError::RecoveryPlanMismatch { expected, observed } => write!( + formatter, + "post-crash migration recovery planned {observed:?}, expected {expected:?}" + ), DurabilityCrashMatrixError::RepeatedInventoryPath { path } => { write!(formatter, "post-crash inventory repeated path `{path}`") } @@ -182,31 +191,51 @@ fn format_command( DurabilityCrashMatrixError::Case { point, position, + occurrence, source, - } => write!( - formatter, - "{} {}: {source}", - point.identifier(), - position.identifier() - ), + } => { + write!( + formatter, + "{} {}", + point.identifier(), + position.identifier() + )?; + if let Some(occurrence) = occurrence { + write!(formatter, " occurrence {}", occurrence.get())?; + } + write!(formatter, ": {source}") + } DurabilityCrashMatrixError::InvalidCase(error) => { write!(formatter, "invalid crash case: {error}") } + DurabilityCrashMatrixError::InvalidOccurrenceEncoding => { + formatter.write_str("crash occurrence is not valid Unicode") + } DurabilityCrashMatrixError::InvalidPointEncoding => { formatter.write_str("crash point is not valid Unicode") } DurabilityCrashMatrixError::InvalidPositionEncoding => { formatter.write_str("crash position is not valid Unicode") } + DurabilityCrashMatrixError::InvalidSequenceEncoding => { + formatter.write_str("crash sequence is not valid Unicode") + } + DurabilityCrashMatrixError::UnknownOccurrence(occurrence) => { + write!(formatter, "unknown crash occurrence `{occurrence}`") + } DurabilityCrashMatrixError::UnknownPoint(point) => { write!(formatter, "unknown crash point `{point}`") } DurabilityCrashMatrixError::UnknownPosition(position) => { write!(formatter, "unknown crash position `{position}`") } + DurabilityCrashMatrixError::UnknownSequence(sequence) => { + write!(formatter, "unknown crash sequence `{sequence}`") + } DurabilityCrashMatrixError::Usage => formatter.write_str( "usage: cargo xtask durability-crash-matrix \ - --case ", + [--case [] \ + | --sequence ]", ), _ => Err(fmt::Error), } diff --git a/xtask/src/durability_crash_matrix/process.rs b/xtask/src/durability_crash_matrix/process.rs index 8375ad6f..c3b4540c 100644 --- a/xtask/src/durability_crash_matrix/process.rs +++ b/xtask/src/durability_crash_matrix/process.rs @@ -67,9 +67,11 @@ fn spawn( .current_dir(repository_root) .arg("__durability-crash-child") .arg(case.point().identifier()) - .arg(case.position().identifier()) - .arg(case_root) - .arg(socket_path); + .arg(case.position().identifier()); + if let Some(occurrence) = case.occurrence() { + command.arg(occurrence.get().to_string()); + } + command.arg(case_root).arg(socket_path); let mut child = crate::bounded_process::spawn_in_process_group(&mut command) .map_err(|source| DurabilityCrashMatrixError::io("spawn crash child", source))?; match ProcessGroup::for_child(&child) { diff --git a/xtask/src/durability_crash_matrix/production_protocol.rs b/xtask/src/durability_crash_matrix/production_protocol.rs index 6453a94c..45fe2600 100644 --- a/xtask/src/durability_crash_matrix/production_protocol.rs +++ b/xtask/src/durability_crash_matrix/production_protocol.rs @@ -2,8 +2,10 @@ mod control; pub(super) mod fixture; -mod initialization; +pub(super) mod initialization; mod initialization_storage; +mod migration; +mod migration_storage; mod publication; mod publication_storage; mod recovery; @@ -41,6 +43,9 @@ pub(super) fn run( DurabilityCrashSequence::RecoveryDiscard => { recovery::run(&store_root, &mut control)?; } + DurabilityCrashSequence::Migration => { + migration::run(&store_root, &mut control)?; + } } Err(DurabilityCrashMatrixError::PointSequenceMismatch { point: case.point(), @@ -54,7 +59,7 @@ fn create_store_root(case_root: &Path) -> Result DurabilityCrashMatrixError { diff --git a/xtask/src/durability_crash_matrix/production_protocol/control.rs b/xtask/src/durability_crash_matrix/production_protocol/control.rs index 9d96a91e..72fdeb5b 100644 --- a/xtask/src/durability_crash_matrix/production_protocol/control.rs +++ b/xtask/src/durability_crash_matrix/production_protocol/control.rs @@ -3,7 +3,9 @@ use std::io::{self, Read, Write}; use std::os::unix::net::UnixStream; -use xtask::{DurabilityCrashCase, DurabilityCrashPoint, DurabilityCrashPosition}; +use xtask::{ + DurabilityCrashCase, DurabilityCrashOccurrence, DurabilityCrashPoint, DurabilityCrashPosition, +}; const READY: u8 = b'r'; @@ -55,6 +57,10 @@ impl CrashControl { (self.case.point() == point).then(|| self.case.position()) } + pub(super) const fn occurrence(&self) -> Option { + self.case.occurrence() + } + pub(super) fn await_process_death(&mut self) -> io::Result<()> { self.readiness.write_all(&[READY])?; let mut unexpected = [0_u8; 1]; diff --git a/xtask/src/durability_crash_matrix/production_protocol/initialization.rs b/xtask/src/durability_crash_matrix/production_protocol/initialization.rs index 6bd54667..d75d1e17 100644 --- a/xtask/src/durability_crash_matrix/production_protocol/initialization.rs +++ b/xtask/src/durability_crash_matrix/production_protocol/initialization.rs @@ -45,6 +45,6 @@ pub(super) fn restart_policy() -> Result SegmentReadPolicy { +pub(in crate::durability_crash_matrix) const fn segment_policy() -> SegmentReadPolicy { SegmentReadPolicy::new(SegmentRecordLimit::MAXIMUM, LayoutEntryLimit::MAXIMUM) } diff --git a/xtask/src/durability_crash_matrix/production_protocol/migration.rs b/xtask/src/durability_crash_matrix/production_protocol/migration.rs new file mode 100644 index 00000000..6b932783 --- /dev/null +++ b/xtask/src/durability_crash_matrix/production_protocol/migration.rs @@ -0,0 +1,39 @@ +//! This module owns execution of the production store-migration protocol. + +use std::path::Path; + +use keep::{FilesystemStoreMigrationAuthority, FilesystemWriterLock, execute_store_migration}; + +use super::control::CrashControl; +use super::initialization; +use super::migration_storage::CrashMigrationStorage; +use super::publication; +use super::{DurabilityCrashMatrixError, verification}; + +/// Publishes the Golden File Worldline version-1 store, then migrates it +/// through the production 21-phase protocol with the selected boundary gated +/// for process death. +/// +/// No version-1 gate fires for a migration case, so the publication +/// precondition runs to completion and releases its writer lock before the +/// migration authority reacquires it. +pub(super) fn run( + store_root: &Path, + control: &mut CrashControl, +) -> Result<(), DurabilityCrashMatrixError> { + publication::run(store_root, control)?; + let lock = FilesystemWriterLock::try_acquire(store_root) + .map_err(|source| verification("reacquire writer lock for migration", source))?; + let authority = FilesystemStoreMigrationAuthority::open_unchecked_for_repository_tasks( + lock, + initialization::segment_policy(), + ) + .map_err(|source| verification("open production migration authority", source))?; + let intent = authority + .observe_intent() + .map_err(|source| verification("observe production migration intent", source))?; + let mut storage = CrashMigrationStorage::new(authority, control); + execute_store_migration(&mut storage, &intent) + .map(|_receipt| ()) + .map_err(|source| verification("execute production store migration", source)) +} diff --git a/xtask/src/durability_crash_matrix/production_protocol/migration_stage_prefix_laws.rs b/xtask/src/durability_crash_matrix/production_protocol/migration_stage_prefix_laws.rs new file mode 100644 index 00000000..1bde7755 --- /dev/null +++ b/xtask/src/durability_crash_matrix/production_protocol/migration_stage_prefix_laws.rs @@ -0,0 +1,45 @@ +//! This module owns nonempty strict migration interruption-prefix laws. + +use std::error::Error; + +use keep::{ + AdmittedStoreFormatMarker, AdmittedStoreMigrationIntent, AdmittedStoreMigrationReceipt, +}; +use xtask::protocol_admission::{EmptyHex, decode_lower_hex}; + +use super::{INTENT_INTERRUPTION, MARKER_INTERRUPTION, RECEIPT_INTERRUPTION}; + +const INTENT: &str = include_str!("../../../../conformance/segment-store/v2/migration-intent.hex"); +const MARKER: &str = include_str!("../../../../conformance/segment-store/v2/format-marker.hex"); +const RECEIPT: &str = + include_str!("../../../../conformance/segment-store/v2/migration-receipt.hex"); + +#[test] +fn stage_interruptions_are_nonempty_strict_prefixes_of_admitted_records() +-> Result<(), Box> { + let intent_bytes = decode(INTENT)?; + let marker_bytes = decode(MARKER)?; + let receipt_bytes = decode(RECEIPT)?; + let intent = AdmittedStoreMigrationIntent::decode(&intent_bytes)?; + let marker = AdmittedStoreFormatMarker::decode(&marker_bytes)?; + let receipt = AdmittedStoreMigrationReceipt::decode(&receipt_bytes, &intent, &marker)?; + for (bytes, prefix) in [ + (intent.encoded(), INTENT_INTERRUPTION), + (marker.encoded(), MARKER_INTERRUPTION), + (receipt.encoded(), RECEIPT_INTERRUPTION), + ] { + assert!(prefix > 0, "an interruption must leave a nonempty prefix"); + assert!( + prefix < bytes.len(), + "an interruption must not complete the record" + ); + } + Ok(()) +} + +fn decode(text: &str) -> Result, Box> { + let hex = text + .strip_suffix('\n') + .ok_or("fixture has no final newline")?; + Ok(decode_lower_hex(hex, 256, EmptyHex::Refuse)?) +} diff --git a/xtask/src/durability_crash_matrix/production_protocol/migration_storage.rs b/xtask/src/durability_crash_matrix/production_protocol/migration_storage.rs new file mode 100644 index 00000000..ececb482 --- /dev/null +++ b/xtask/src/durability_crash_matrix/production_protocol/migration_storage.rs @@ -0,0 +1,315 @@ +//! This module owns crash injection around production store migration. + +use std::io; + +use keep::{ + CanonicalStoreFormatMarker, CanonicalStoreMigrationIntent, CanonicalStoreMigrationReceipt, + FilesystemStoreMigrationAuthority, StoreMigrationFixedStage, StoreMigrationStorage, +}; +use xtask::{DurabilityCrashPoint, DurabilityCrashPosition}; + +use super::control::{CrashControl, DuringTiming}; + +#[cfg(test)] +#[path = "migration_stage_prefix_laws.rs"] +mod stage_prefix_laws; + +// The fixture law below admits all three records and requires these fixed +// interruption offsets to remain nonempty strict prefixes of their encodings. +const INTENT_INTERRUPTION: usize = 128; +const MARKER_INTERRUPTION: usize = 48; +const RECEIPT_INTERRUPTION: usize = 128; + +pub(super) struct CrashMigrationStorage<'control> { + inner: FilesystemStoreMigrationAuthority, + control: &'control mut CrashControl, +} + +impl<'control> CrashMigrationStorage<'control> { + pub(super) const fn new( + inner: FilesystemStoreMigrationAuthority, + control: &'control mut CrashControl, + ) -> Self { + Self { inner, control } + } +} + +impl StoreMigrationStorage for CrashMigrationStorage<'_> { + fn verify_current(&mut self, intent: &CanonicalStoreMigrationIntent) -> io::Result<()> { + StoreMigrationStorage::verify_current(&mut self.inner, intent) + } + + fn write_intent_stage(&mut self, intent: &CanonicalStoreMigrationIntent) -> io::Result<()> { + execute_write( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationWriteIntentStage, + |inner| inner.write_intent_stage(intent), + |inner| { + inner.write_fixed_stage_prefix_for_repository_tasks( + StoreMigrationFixedStage::Intent, + intent.encoded(), + INTENT_INTERRUPTION, + ) + }, + ) + } + + fn synchronize_intent_stage(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeIntentStage, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_intent_stage, + ) + } + + fn link_intent(&mut self, intent: &CanonicalStoreMigrationIntent) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationLinkIntent, + DuringTiming::After, + |inner| inner.link_intent(intent), + ) + } + + fn synchronize_root_after_intent(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeRootAfterIntent, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_root_after_intent, + ) + } + + fn remove_intent_stage(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationRemoveIntentStage, + DuringTiming::After, + FilesystemStoreMigrationAuthority::remove_intent_stage, + ) + } + + fn synchronize_root_after_intent_cleanup(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeRootAfterIntentCleanup, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_root_after_intent_cleanup, + ) + } + + fn admit_reader_fence(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationAdmitReaderFence, + DuringTiming::After, + FilesystemStoreMigrationAuthority::admit_reader_fence, + ) + } + + fn admit_namespace_prefix(&mut self) -> io::Result<()> { + let point = DurabilityCrashPoint::MigrationAdmitNamespacePrefix; + match self.control.position(point) { + None => self.inner.admit_namespace_prefix(), + Some(DurabilityCrashPosition::Before) => self.control.await_process_death(), + Some(DurabilityCrashPosition::During) => { + let reached = self + .control + .occurrence() + .map_or(1, |occurrence| occurrence.get().saturating_add(1)); + let reached = usize::try_from(reached).map_err(io::Error::other)?; + self.inner + .admit_namespace_prefix_for_repository_tasks(reached)?; + self.control.await_process_death() + } + Some(DurabilityCrashPosition::After) => { + self.inner.admit_namespace_prefix()?; + self.control.await_process_death() + } + } + } + + fn synchronize_root_after_namespace(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeRootAfterNamespace, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_root_after_namespace, + ) + } + + fn write_marker_stage(&mut self, marker: &CanonicalStoreFormatMarker) -> io::Result<()> { + execute_write( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationWriteMarkerStage, + |inner| inner.write_marker_stage(marker), + |inner| { + inner.write_fixed_stage_prefix_for_repository_tasks( + StoreMigrationFixedStage::Marker, + marker.encoded(), + MARKER_INTERRUPTION, + ) + }, + ) + } + + fn synchronize_marker_stage(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeMarkerStage, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_marker_stage, + ) + } + + fn link_marker(&mut self, marker: &CanonicalStoreFormatMarker) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationLinkMarker, + DuringTiming::After, + |inner| inner.link_marker(marker), + ) + } + + fn synchronize_root_after_marker(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeRootAfterMarker, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_root_after_marker, + ) + } + + fn remove_marker_stage(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationRemoveMarkerStage, + DuringTiming::After, + FilesystemStoreMigrationAuthority::remove_marker_stage, + ) + } + + fn synchronize_root_after_marker_cleanup(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeRootAfterMarkerCleanup, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_root_after_marker_cleanup, + ) + } + + fn write_receipt_stage(&mut self, receipt: &CanonicalStoreMigrationReceipt) -> io::Result<()> { + execute_write( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationWriteReceiptStage, + |inner| inner.write_receipt_stage(receipt), + |inner| { + inner.write_fixed_stage_prefix_for_repository_tasks( + StoreMigrationFixedStage::Receipt, + receipt.encoded(), + RECEIPT_INTERRUPTION, + ) + }, + ) + } + + fn synchronize_receipt_stage(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeReceiptStage, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_receipt_stage, + ) + } + + fn link_receipt(&mut self, receipt: &CanonicalStoreMigrationReceipt) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationLinkReceipt, + DuringTiming::After, + |inner| inner.link_receipt(receipt), + ) + } + + fn synchronize_root_after_receipt(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeRootAfterReceipt, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_root_after_receipt, + ) + } + + fn remove_receipt_stage(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationRemoveReceiptStage, + DuringTiming::After, + FilesystemStoreMigrationAuthority::remove_receipt_stage, + ) + } + + fn synchronize_root_after_receipt_cleanup(&mut self) -> io::Result<()> { + execute( + &mut self.inner, + self.control, + DurabilityCrashPoint::MigrationSynchronizeRootAfterReceiptCleanup, + DuringTiming::Before, + FilesystemStoreMigrationAuthority::synchronize_root_after_receipt_cleanup, + ) + } +} + +fn execute( + inner: &mut FilesystemStoreMigrationAuthority, + control: &mut CrashControl, + point: DurabilityCrashPoint, + during: DuringTiming, + operation: impl FnOnce(&mut FilesystemStoreMigrationAuthority) -> io::Result, +) -> io::Result { + control.before(point, during)?; + let result = operation(inner)?; + control.after(point, during)?; + Ok(result) +} + +fn execute_write( + inner: &mut FilesystemStoreMigrationAuthority, + control: &mut CrashControl, + point: DurabilityCrashPoint, + complete: impl FnOnce(&mut FilesystemStoreMigrationAuthority) -> io::Result<()>, + interrupted: impl FnOnce(&mut FilesystemStoreMigrationAuthority) -> io::Result<()>, +) -> io::Result<()> { + match control.position(point) { + None => complete(inner), + Some(DurabilityCrashPosition::Before) => control.await_process_death(), + Some(DurabilityCrashPosition::During) => { + interrupted(inner)?; + control.await_process_death() + } + Some(DurabilityCrashPosition::After) => { + complete(inner)?; + control.await_process_death() + } + } +} diff --git a/xtask/src/durability_crash_matrix/restart.rs b/xtask/src/durability_crash_matrix/restart.rs index 423efc81..8ee7c51a 100644 --- a/xtask/src/durability_crash_matrix/restart.rs +++ b/xtask/src/durability_crash_matrix/restart.rs @@ -1,6 +1,8 @@ //! This module owns independent post-process-death store verification. mod expectation; +mod migration; +mod migration_expectation; mod semantic; use std::collections::BTreeSet; @@ -11,12 +13,15 @@ use std::path::{Path, PathBuf}; use super::DurabilityCrashMatrixError; use super::production_protocol::fixture::GoldenFixture; use expectation::ExpectedStoreState; -use xtask::DurabilityCrashCase; +use xtask::{DurabilityCrashCase, DurabilityCrashSequence}; pub(super) fn verify( store_root: &Path, case: DurabilityCrashCase, ) -> Result<(), DurabilityCrashMatrixError> { + if case.point().sequence() == DurabilityCrashSequence::Migration { + return migration::verify(store_root, case); + } let expected = ExpectedStoreState::for_case(case)?; let observed_paths = inventory(store_root)?; if observed_paths != expected.paths() { @@ -47,7 +52,7 @@ pub(super) fn verify( Ok(()) } -fn inventory(store_root: &Path) -> Result, DurabilityCrashMatrixError> { +pub(super) fn inventory(store_root: &Path) -> Result, DurabilityCrashMatrixError> { let mut paths = BTreeSet::new(); let mut pending = vec![PathBuf::new()]; while let Some(relative_parent) = pending.pop() { diff --git a/xtask/src/durability_crash_matrix/restart/expectation.rs b/xtask/src/durability_crash_matrix/restart/expectation.rs index 1b6be39c..c4a779a0 100644 --- a/xtask/src/durability_crash_matrix/restart/expectation.rs +++ b/xtask/src/durability_crash_matrix/restart/expectation.rs @@ -64,6 +64,11 @@ impl ExpectedStoreState { DurabilityCrashSequence::Head => sequence::head(case), DurabilityCrashSequence::RecoveryDiscard => sequence::recovery(case), DurabilityCrashSequence::Initialization => sequence::initialization(case), + DurabilityCrashSequence::Migration => { + Err(DurabilityCrashMatrixError::PointSequenceMismatch { + point: case.point(), + }) + } } } diff --git a/xtask/src/durability_crash_matrix/restart/migration.rs b/xtask/src/durability_crash_matrix/restart/migration.rs new file mode 100644 index 00000000..ce219195 --- /dev/null +++ b/xtask/src/durability_crash_matrix/restart/migration.rs @@ -0,0 +1,122 @@ +//! This module owns independent post-process-death migration verification: +//! the exact residue, the predicted recovery plan, the production recovery, +//! and the complete migration it must leave behind. + +use std::fs; +use std::path::Path; + +use keep::{ + FilesystemStoreMigrationAuthority, FilesystemWriterLock, StoreMigrationRecoveryPlan as Plan, + execute_store_migration, recover_store_migration, +}; +use xtask::DurabilityCrashCase; + +use super::migration_expectation::{MigrationExpectation, complete_paths}; +use crate::durability_crash_matrix::DurabilityCrashMatrixError; +use crate::durability_crash_matrix::production_protocol::fixture::{ + CATALOG_POOL_PATH, GoldenFixture, SEGMENT_POOL_PATH, +}; +use crate::durability_crash_matrix::production_protocol::initialization::segment_policy; +use crate::durability_crash_matrix::production_protocol::verification; + +pub(super) fn verify( + store_root: &Path, + case: DurabilityCrashCase, +) -> Result<(), DurabilityCrashMatrixError> { + let expected = MigrationExpectation::for_case(case)?; + require_inventory(store_root, expected.paths())?; + verify_version_one_bytes(store_root)?; + + let plan = recover(store_root)?; + if plan != expected.plan() { + return Err(DurabilityCrashMatrixError::RecoveryPlanMismatch { + expected: expected.plan(), + observed: plan, + }); + } + if plan == Plan::VersionOne { + migrate_forward(store_root)?; + } + + require_inventory(store_root, &complete_paths())?; + verify_version_one_bytes(store_root)?; + let settled = recover(store_root)?; + if settled == Plan::Complete { + Ok(()) + } else { + Err(DurabilityCrashMatrixError::RecoveryPlanMismatch { + expected: Plan::Complete, + observed: settled, + }) + } +} + +fn require_inventory( + store_root: &Path, + expected: &std::collections::BTreeSet, +) -> Result<(), DurabilityCrashMatrixError> { + let observed = super::inventory(store_root)?; + if &observed == expected { + Ok(()) + } else { + Err(DurabilityCrashMatrixError::InventoryMismatch { + expected: expected.clone(), + observed, + }) + } +} + +/// Reacquires writer authority the way a restarted process would and runs +/// the production recovery once, reporting the plan the residue admitted. +fn recover(store_root: &Path) -> Result { + let mut authority = + FilesystemStoreMigrationAuthority::reopen_for_recovery_unchecked_for_repository_tasks( + store_root, + segment_policy(), + ) + .map_err(|source| verification("reopen migration for recovery", source))?; + let expected = authority + .observe_intent() + .map_err(|source| verification("observe recovery migration intent", source))?; + let receipt = recover_store_migration(&mut authority, &expected) + .map_err(|source| verification("recover production migration", source))?; + Ok(receipt.plan()) +} + +/// The forward retry after an untouched version-1 store admits. +fn migrate_forward(store_root: &Path) -> Result<(), DurabilityCrashMatrixError> { + let lock = FilesystemWriterLock::try_acquire(store_root) + .map_err(|source| verification("reacquire writer lock for forward retry", source))?; + let mut authority = FilesystemStoreMigrationAuthority::open_unchecked_for_repository_tasks( + lock, + segment_policy(), + ) + .map_err(|source| verification("open forward-retry migration authority", source))?; + let intent = authority + .observe_intent() + .map_err(|source| verification("observe forward-retry migration intent", source))?; + execute_store_migration(&mut authority, &intent) + .map(|_receipt| ()) + .map_err(|source| verification("execute forward-retry migration", source)) +} + +/// Migration never rewrites a version-1 byte: the pools and `HEAD` remain +/// the Golden File Worldline fixtures before and after recovery. +fn verify_version_one_bytes(store_root: &Path) -> Result<(), DurabilityCrashMatrixError> { + for (relative, fixture) in [ + (SEGMENT_POOL_PATH, GoldenFixture::segment()?), + (CATALOG_POOL_PATH, GoldenFixture::catalog()?), + ("HEAD", GoldenFixture::head()?), + ] { + let observed = fs::read(store_root.join(relative)) + .map_err(|source| DurabilityCrashMatrixError::io("read version-1 artifact", source))?; + if observed != fixture.bytes() { + return Err(DurabilityCrashMatrixError::artifact_bytes( + relative, + fixture.bytes(), + &observed, + )); + } + } + Ok(()) +} diff --git a/xtask/src/durability_crash_matrix/restart/migration_expectation.rs b/xtask/src/durability_crash_matrix/restart/migration_expectation.rs new file mode 100644 index 00000000..6a54ed9f --- /dev/null +++ b/xtask/src/durability_crash_matrix/restart/migration_expectation.rs @@ -0,0 +1,246 @@ +//! This module owns the independent expected state after migration process +//! death: the exact root inventory and the one lawful recovery plan. +//! +//! The rules here restate the recovery table in +//! `docs/formats/segment-store-v2/migration-recovery.md` without reading any +//! production classifier, so a planner defect cannot hide behind itself. + +use std::collections::BTreeSet; + +use keep::{StoreMigrationFixedStage, StoreMigrationPhase, StoreMigrationRecoveryPlan as Plan}; +use xtask::{ + DurabilityCrashCase, DurabilityCrashCaseError, DurabilityCrashPoint, DurabilityCrashPosition, +}; + +use crate::durability_crash_matrix::DurabilityCrashMatrixError; +use crate::durability_crash_matrix::production_protocol::fixture::{ + CATALOG_POOL_PATH, SEGMENT_POOL_PATH, +}; + +pub(super) const INTENT_STAGE: &str = "migration.intent.next"; +pub(super) const INTENT: &str = "migration.intent"; +pub(super) const READER_LOCK: &str = "reader.lock"; +pub(super) const MARKER_STAGE: &str = "FORMAT.next"; +pub(super) const MARKER: &str = "FORMAT"; +pub(super) const RECEIPT_STAGE: &str = "migration.receipt.next"; +pub(super) const RECEIPT: &str = "migration.receipt"; + +/// The namespace prefix in admission order; each entry is one `during` +/// occurrence of `KEEP-CRASH-060`. +const PREFIX_DIRECTORIES: [&str; 6] = [ + "retention", + "retention/roots", + "retention/manifests", + "gc", + "recovery", + "recovery/dispositions", +]; + +pub(super) struct MigrationExpectation { + paths: BTreeSet, + plan: Plan, +} + +impl MigrationExpectation { + pub(super) fn for_case(case: DurabilityCrashCase) -> Result { + let step = migration_step(case.point())?; + let partial = partial_stage(case); + let done = if case.position() == DurabilityCrashPosition::After + || (case.position() == DurabilityCrashPosition::During && atomic(case.point())) + { + step.saturating_add(1) + } else { + step + }; + let prefix_reached = prefix_reached(case)?; + let mut paths = version_one_paths(); + if (1..5).contains(&done) { + paths.insert(INTENT_STAGE.into()); + } + if done >= 3 { + paths.insert(INTENT.into()); + } + if done >= 7 { + paths.insert(READER_LOCK.into()); + } + let directories = if done >= 8 { + PREFIX_DIRECTORIES.len() + } else { + prefix_reached.unwrap_or(0) + }; + paths.extend( + PREFIX_DIRECTORIES + .iter() + .take(directories) + .map(|path| (*path).into()), + ); + if (10..14).contains(&done) { + paths.insert(MARKER_STAGE.into()); + } + if done >= 12 { + paths.insert(MARKER.into()); + } + if (16..20).contains(&done) { + paths.insert(RECEIPT_STAGE.into()); + } + if done >= 18 { + paths.insert(RECEIPT.into()); + } + if let Some(stage) = partial { + paths.insert(stage_name(stage).into()); + } + let plan = partial.map_or_else( + || plan_after(done, prefix_reached), + |stage| Plan::DiscardStage { + stage, + resume: write_phase(stage), + }, + ); + Ok(Self { paths, plan }) + } + + pub(super) const fn paths(&self) -> &BTreeSet { + &self.paths + } + + pub(super) const fn plan(&self) -> Plan { + self.plan + } +} + +/// The exact version-1 store the migration starts from. +pub(super) fn version_one_paths() -> BTreeSet { + [ + "writer.lock", + "staging", + "segments", + "catalogs", + "HEAD", + SEGMENT_POOL_PATH, + CATALOG_POOL_PATH, + ] + .into_iter() + .map(Into::into) + .collect() +} + +/// The exact root after one complete migration: no stage remains. +pub(super) fn complete_paths() -> BTreeSet { + let mut paths = version_one_paths(); + paths.extend( + [INTENT, READER_LOCK, MARKER, RECEIPT] + .into_iter() + .chain(PREFIX_DIRECTORIES) + .map(Into::into), + ); + paths +} + +fn migration_step(point: DurabilityCrashPoint) -> Result { + DurabilityCrashPoint::MIGRATION + .into_iter() + .position(|candidate| candidate == point) + .ok_or(DurabilityCrashMatrixError::PointSequenceMismatch { point }) +} + +/// Boundaries whose effect is one link, unlink, or exclusive creation, so +/// `during` cannot leave a partial effect. +const fn atomic(point: DurabilityCrashPoint) -> bool { + matches!( + point, + DurabilityCrashPoint::MigrationLinkIntent + | DurabilityCrashPoint::MigrationRemoveIntentStage + | DurabilityCrashPoint::MigrationAdmitReaderFence + | DurabilityCrashPoint::MigrationLinkMarker + | DurabilityCrashPoint::MigrationRemoveMarkerStage + | DurabilityCrashPoint::MigrationLinkReceipt + | DurabilityCrashPoint::MigrationRemoveReceiptStage + ) +} + +/// Process death during a stage write leaves an incomplete pre-effect stage. +fn partial_stage(case: DurabilityCrashCase) -> Option { + if case.position() != DurabilityCrashPosition::During { + return None; + } + match case.point() { + DurabilityCrashPoint::MigrationWriteIntentStage => Some(StoreMigrationFixedStage::Intent), + DurabilityCrashPoint::MigrationWriteMarkerStage => Some(StoreMigrationFixedStage::Marker), + DurabilityCrashPoint::MigrationWriteReceiptStage => Some(StoreMigrationFixedStage::Receipt), + _ => None, + } +} + +/// Process death during namespace admission leaves the first `occurrence + 1` +/// prefix directories, each with its parent synchronized. +fn prefix_reached(case: DurabilityCrashCase) -> Result, DurabilityCrashMatrixError> { + if case.point() != DurabilityCrashPoint::MigrationAdmitNamespacePrefix + || case.position() != DurabilityCrashPosition::During + { + return Ok(None); + } + let point = case.point(); + let observed = case + .occurrence() + .ok_or(DurabilityCrashMatrixError::InvalidCase( + DurabilityCrashCaseError::MissingOccurrence { point }, + ))?; + let reached = observed + .get() + .checked_add(1) + .ok_or(DurabilityCrashMatrixError::InvalidCase( + DurabilityCrashCaseError::OccurrenceOutOfRange { + point, + observed, + exclusive_limit: point.during_occurrences(), + }, + ))?; + usize::try_from(reached) + .map(Some) + .map_err(|source| DurabilityCrashMatrixError::Verification { + phase: "represent reached migration namespace prefix", + source: Box::new(source), + }) +} + +const fn stage_name(stage: StoreMigrationFixedStage) -> &'static str { + match stage { + StoreMigrationFixedStage::Intent => INTENT_STAGE, + StoreMigrationFixedStage::Marker => MARKER_STAGE, + StoreMigrationFixedStage::Receipt => RECEIPT_STAGE, + } +} + +const fn write_phase(stage: StoreMigrationFixedStage) -> StoreMigrationPhase { + match stage { + StoreMigrationFixedStage::Intent => StoreMigrationPhase::WriteIntentStage, + StoreMigrationFixedStage::Marker => StoreMigrationPhase::WriteMarkerStage, + StoreMigrationFixedStage::Receipt => StoreMigrationPhase::WriteReceiptStage, + } +} + +/// The recovery-table row for a residue in which the first `done` phases +/// completed and nothing else happened: resume at the earliest phase the +/// residue cannot prove. +const fn plan_after(done: usize, prefix_reached: Option) -> Plan { + let resume = match done { + 0 => return Plan::VersionOne, + 1 | 2 => StoreMigrationPhase::SynchronizeIntentStage, + 3 | 4 => StoreMigrationPhase::SynchronizeRootAfterIntent, + 5 | 6 => StoreMigrationPhase::SynchronizeRootAfterIntentCleanup, + 7 => match prefix_reached { + Some(reached) if reached == PREFIX_DIRECTORIES.len() => { + StoreMigrationPhase::SynchronizeRootAfterNamespace + } + _ => StoreMigrationPhase::AdmitNamespacePrefix, + }, + 8 | 9 => StoreMigrationPhase::SynchronizeRootAfterNamespace, + 10 | 11 => StoreMigrationPhase::SynchronizeMarkerStage, + 12 | 13 => StoreMigrationPhase::SynchronizeRootAfterMarker, + 14 | 15 => StoreMigrationPhase::SynchronizeRootAfterMarkerCleanup, + 16 | 17 => StoreMigrationPhase::SynchronizeReceiptStage, + 18 | 19 => StoreMigrationPhase::SynchronizeRootAfterReceipt, + _ => return Plan::Complete, + }; + Plan::Resume { resume } +} diff --git a/xtask/src/durability_crash_point.rs b/xtask/src/durability_crash_point.rs index 2ed8d6a5..555ed7ec 100644 --- a/xtask/src/durability_crash_point.rs +++ b/xtask/src/durability_crash_point.rs @@ -13,6 +13,41 @@ pub enum DurabilityCrashSequence { RecoveryDiscard, /// Writer-locked store initialization. Initialization, + /// One-way version-1 to version-2 store migration. + Migration, +} + +impl DurabilityCrashSequence { + /// Every sequence in stable protocol order. + pub const ALL: [Self; 6] = [ + Self::Segment, + Self::Catalog, + Self::Head, + Self::RecoveryDiscard, + Self::Initialization, + Self::Migration, + ]; + + /// Returns the stable identifier used by the crash-matrix command line. + #[must_use] + pub const fn identifier(self) -> &'static str { + match self { + Self::Segment => "segment", + Self::Catalog => "catalog", + Self::Head => "head", + Self::RecoveryDiscard => "recovery-discard", + Self::Initialization => "initialization", + Self::Migration => "migration", + } + } + + /// Parses one exact sequence identifier. + #[must_use] + pub fn from_identifier(identifier: &str) -> Option { + Self::ALL + .into_iter() + .find(|sequence| sequence.identifier() == identifier) + } } /// One stable process-death boundary in the durable segment-store protocol. @@ -88,11 +123,54 @@ pub enum DurabilityCrashPoint { CreateCatalogPoolDirectory, /// Synchronize the store root after initialization. SynchronizeRootAfterInitialization, + /// Write the complete canonical `migration.intent.next`. + MigrationWriteIntentStage, + /// Synchronize `migration.intent.next`. + MigrationSynchronizeIntentStage, + /// Link the synchronized intent stage to `migration.intent`. + MigrationLinkIntent, + /// Synchronize the store root after the intent link. + MigrationSynchronizeRootAfterIntent, + /// Remove the retained `migration.intent.next`. + MigrationRemoveIntentStage, + /// Synchronize the store root after intent-stage cleanup. + MigrationSynchronizeRootAfterIntentCleanup, + /// Create or exactly admit the persistent reader fence. + MigrationAdmitReaderFence, + /// Create or exactly admit the canonical version-2 directory prefix; the + /// occurrence names the directory-prefix length reached. + MigrationAdmitNamespacePrefix, + /// Synchronize the store root after namespace admission. + MigrationSynchronizeRootAfterNamespace, + /// Write the complete canonical `FORMAT.next`. + MigrationWriteMarkerStage, + /// Synchronize `FORMAT.next`. + MigrationSynchronizeMarkerStage, + /// Link the synchronized marker stage to `FORMAT`. + MigrationLinkMarker, + /// Synchronize the store root after the marker link. + MigrationSynchronizeRootAfterMarker, + /// Remove the retained `FORMAT.next`. + MigrationRemoveMarkerStage, + /// Synchronize the store root after marker-stage cleanup. + MigrationSynchronizeRootAfterMarkerCleanup, + /// Write the complete canonical `migration.receipt.next`. + MigrationWriteReceiptStage, + /// Synchronize `migration.receipt.next`. + MigrationSynchronizeReceiptStage, + /// Link the synchronized receipt stage to `migration.receipt`. + MigrationLinkReceipt, + /// Synchronize the store root after the receipt link. + MigrationSynchronizeRootAfterReceipt, + /// Remove the retained `migration.receipt.next`. + MigrationRemoveReceiptStage, + /// Synchronize the store root after receipt-stage cleanup. + MigrationSynchronizeRootAfterReceiptCleanup, } impl DurabilityCrashPoint { /// Every crash boundary in stable protocol order. - pub const ALL: [Self; 35] = [ + pub const ALL: [Self; 56] = [ Self::CreateSegmentStage, Self::WriteSegmentHeader, Self::AppendSegmentRecord, @@ -128,8 +206,59 @@ impl DurabilityCrashPoint { Self::CreateSegmentPoolDirectory, Self::CreateCatalogPoolDirectory, Self::SynchronizeRootAfterInitialization, + Self::MigrationWriteIntentStage, + Self::MigrationSynchronizeIntentStage, + Self::MigrationLinkIntent, + Self::MigrationSynchronizeRootAfterIntent, + Self::MigrationRemoveIntentStage, + Self::MigrationSynchronizeRootAfterIntentCleanup, + Self::MigrationAdmitReaderFence, + Self::MigrationAdmitNamespacePrefix, + Self::MigrationSynchronizeRootAfterNamespace, + Self::MigrationWriteMarkerStage, + Self::MigrationSynchronizeMarkerStage, + Self::MigrationLinkMarker, + Self::MigrationSynchronizeRootAfterMarker, + Self::MigrationRemoveMarkerStage, + Self::MigrationSynchronizeRootAfterMarkerCleanup, + Self::MigrationWriteReceiptStage, + Self::MigrationSynchronizeReceiptStage, + Self::MigrationLinkReceipt, + Self::MigrationSynchronizeRootAfterReceipt, + Self::MigrationRemoveReceiptStage, + Self::MigrationSynchronizeRootAfterReceiptCleanup, ]; + /// The migration boundaries in `StoreMigrationPhase::ALL` order. + pub const MIGRATION: [Self; 21] = [ + Self::MigrationWriteIntentStage, + Self::MigrationSynchronizeIntentStage, + Self::MigrationLinkIntent, + Self::MigrationSynchronizeRootAfterIntent, + Self::MigrationRemoveIntentStage, + Self::MigrationSynchronizeRootAfterIntentCleanup, + Self::MigrationAdmitReaderFence, + Self::MigrationAdmitNamespacePrefix, + Self::MigrationSynchronizeRootAfterNamespace, + Self::MigrationWriteMarkerStage, + Self::MigrationSynchronizeMarkerStage, + Self::MigrationLinkMarker, + Self::MigrationSynchronizeRootAfterMarker, + Self::MigrationRemoveMarkerStage, + Self::MigrationSynchronizeRootAfterMarkerCleanup, + Self::MigrationWriteReceiptStage, + Self::MigrationSynchronizeReceiptStage, + Self::MigrationLinkReceipt, + Self::MigrationSynchronizeRootAfterReceipt, + Self::MigrationRemoveReceiptStage, + Self::MigrationSynchronizeRootAfterReceiptCleanup, + ]; + + /// The number of directories the migration namespace prefix admits; each + /// is one `during` occurrence of + /// [`Self::MigrationAdmitNamespacePrefix`]. + pub const NAMESPACE_PREFIX_DIRECTORIES: u32 = 6; + /// Parses one exact stable crash identifier. #[must_use] pub fn from_identifier(identifier: &str) -> Option { @@ -177,12 +306,49 @@ impl DurabilityCrashPoint { | Self::CreateSegmentPoolDirectory | Self::CreateCatalogPoolDirectory | Self::SynchronizeRootAfterInitialization => DurabilityCrashSequence::Initialization, + Self::MigrationWriteIntentStage + | Self::MigrationSynchronizeIntentStage + | Self::MigrationLinkIntent + | Self::MigrationSynchronizeRootAfterIntent + | Self::MigrationRemoveIntentStage + | Self::MigrationSynchronizeRootAfterIntentCleanup + | Self::MigrationAdmitReaderFence + | Self::MigrationAdmitNamespacePrefix + | Self::MigrationSynchronizeRootAfterNamespace + | Self::MigrationWriteMarkerStage + | Self::MigrationSynchronizeMarkerStage + | Self::MigrationLinkMarker + | Self::MigrationSynchronizeRootAfterMarker + | Self::MigrationRemoveMarkerStage + | Self::MigrationSynchronizeRootAfterMarkerCleanup + | Self::MigrationWriteReceiptStage + | Self::MigrationSynchronizeReceiptStage + | Self::MigrationLinkReceipt + | Self::MigrationSynchronizeRootAfterReceipt + | Self::MigrationRemoveReceiptStage + | Self::MigrationSynchronizeRootAfterReceiptCleanup => { + DurabilityCrashSequence::Migration + } } } /// Reports whether tests may select a repeated occurrence. #[must_use] pub const fn occurrence_counted(self) -> bool { - matches!(self, Self::AppendSegmentRecord) + matches!( + self, + Self::AppendSegmentRecord | Self::MigrationAdmitNamespacePrefix + ) + } + + /// Returns how many distinct `during` occurrences the canonical matrix + /// runs for this boundary: one directory-prefix length per occurrence + /// for the migration namespace prefix, otherwise exactly one. + #[must_use] + pub const fn during_occurrences(self) -> u32 { + match self { + Self::MigrationAdmitNamespacePrefix => Self::NAMESPACE_PREFIX_DIRECTORIES, + _ => 1, + } } } diff --git a/xtask/src/durability_crash_point_identity.rs b/xtask/src/durability_crash_point_identity.rs index 173afa74..7959b43b 100644 --- a/xtask/src/durability_crash_point_identity.rs +++ b/xtask/src/durability_crash_point_identity.rs @@ -42,6 +42,27 @@ impl DurabilityCrashPoint { Self::CreateSegmentPoolDirectory => "KEEP-CRASH-033", Self::CreateCatalogPoolDirectory => "KEEP-CRASH-034", Self::SynchronizeRootAfterInitialization => "KEEP-CRASH-035", + Self::MigrationWriteIntentStage => "KEEP-CRASH-053", + Self::MigrationSynchronizeIntentStage => "KEEP-CRASH-054", + Self::MigrationLinkIntent => "KEEP-CRASH-055", + Self::MigrationSynchronizeRootAfterIntent => "KEEP-CRASH-056", + Self::MigrationRemoveIntentStage => "KEEP-CRASH-057", + Self::MigrationSynchronizeRootAfterIntentCleanup => "KEEP-CRASH-058", + Self::MigrationAdmitReaderFence => "KEEP-CRASH-059", + Self::MigrationAdmitNamespacePrefix => "KEEP-CRASH-060", + Self::MigrationSynchronizeRootAfterNamespace => "KEEP-CRASH-061", + Self::MigrationWriteMarkerStage => "KEEP-CRASH-062", + Self::MigrationSynchronizeMarkerStage => "KEEP-CRASH-063", + Self::MigrationLinkMarker => "KEEP-CRASH-064", + Self::MigrationSynchronizeRootAfterMarker => "KEEP-CRASH-065", + Self::MigrationRemoveMarkerStage => "KEEP-CRASH-066", + Self::MigrationSynchronizeRootAfterMarkerCleanup => "KEEP-CRASH-067", + Self::MigrationWriteReceiptStage => "KEEP-CRASH-068", + Self::MigrationSynchronizeReceiptStage => "KEEP-CRASH-069", + Self::MigrationLinkReceipt => "KEEP-CRASH-070", + Self::MigrationSynchronizeRootAfterReceipt => "KEEP-CRASH-071", + Self::MigrationRemoveReceiptStage => "KEEP-CRASH-072", + Self::MigrationSynchronizeRootAfterReceiptCleanup => "KEEP-CRASH-073", } } } diff --git a/xtask/tests/durability_crash_case_contract.rs b/xtask/tests/durability_crash_case_contract.rs index 87b671e4..755ac076 100644 --- a/xtask/tests/durability_crash_case_contract.rs +++ b/xtask/tests/durability_crash_case_contract.rs @@ -6,40 +6,47 @@ use std::error::Error; use xtask::{ DurabilityCrashCase, DurabilityCrashCaseError, DurabilityCrashOccurrence, DurabilityCrashPoint, - DurabilityCrashPosition, + DurabilityCrashPosition, DurabilityCrashSequence, }; #[test] -fn every_crash_point_has_exactly_three_ordered_process_death_cases() -> Result<(), Box> { +fn every_crash_point_has_three_ordered_positions_and_one_during_case_per_occurrence() +-> Result<(), Box> { let cases: Vec<_> = DurabilityCrashCase::all().collect(); - let expected = DurabilityCrashPoint::ALL - .len() - .checked_mul(DurabilityCrashPosition::ALL.len()) - .ok_or("crash-matrix case count overflow")?; - - assert_eq!(cases.len(), expected); - for (point_index, point) in DurabilityCrashPoint::ALL.into_iter().enumerate() { - for (position_index, position) in DurabilityCrashPosition::ALL.into_iter().enumerate() { - let index = point_index - .checked_mul(DurabilityCrashPosition::ALL.len()) - .and_then(|base| base.checked_add(position_index)) - .ok_or("crash-matrix index overflow")?; - let case = cases.get(index).ok_or("missing canonical crash case")?; - assert_eq!(case.point(), point); - assert_eq!(case.position(), position); - assert_eq!( - case.occurrence(), - point + let mut expected = Vec::new(); + for point in DurabilityCrashPoint::ALL { + for position in DurabilityCrashPosition::ALL { + let occurrences = if position == DurabilityCrashPosition::During { + point.during_occurrences() + } else { + 1 + }; + for ordinal in 0..occurrences { + let occurrence = point .occurrence_counted() - .then_some(DurabilityCrashOccurrence::FIRST) - ); + .then_some(DurabilityCrashOccurrence::new(ordinal)); + expected.push(DurabilityCrashCase::new(point, position, occurrence)?); + } } } + + assert_eq!(cases, expected); + // 56 boundaries at three positions, plus five extra namespace-prefix + // lengths for `KEEP-CRASH-060`. + assert_eq!(cases.len(), 173); + let migration: Vec<_> = + DurabilityCrashCase::in_sequence(DurabilityCrashSequence::Migration).collect(); + assert_eq!(migration.len(), 68); + assert!( + migration + .iter() + .all(|case| case.point().sequence() == DurabilityCrashSequence::Migration) + ); Ok(()) } #[test] -fn occurrence_coordinates_exist_only_for_record_append() -> Result<(), Box> { +fn occurrence_coordinates_exist_only_for_counted_boundaries() -> Result<(), Box> { let occurrence = DurabilityCrashOccurrence::new(7); let counted = DurabilityCrashCase::new( @@ -97,3 +104,33 @@ fn identifiers_and_positions_round_trip_without_aliases() { } assert_eq!(DurabilityCrashPosition::from_identifier("between"), None); } + +#[test] +fn namespace_occurrences_are_admitted_only_within_the_protocol_fixed_range() +-> Result<(), Box> { + let point = DurabilityCrashPoint::MigrationAdmitNamespacePrefix; + for position in DurabilityCrashPosition::ALL { + let exclusive_limit = if position == DurabilityCrashPosition::During { + DurabilityCrashPoint::NAMESPACE_PREFIX_DIRECTORIES + } else { + 1 + }; + for ordinal in 0..exclusive_limit { + let occurrence = DurabilityCrashOccurrence::new(ordinal); + let case = DurabilityCrashCase::new(point, position, Some(occurrence))?; + assert_eq!(case.occurrence(), Some(occurrence)); + } + for ordinal in [exclusive_limit, u32::MAX] { + let observed = DurabilityCrashOccurrence::new(ordinal); + assert_eq!( + DurabilityCrashCase::new(point, position, Some(observed)), + Err(DurabilityCrashCaseError::OccurrenceOutOfRange { + point, + observed, + exclusive_limit, + }) + ); + } + } + Ok(()) +} diff --git a/xtask/tests/durability_crash_documentation.rs b/xtask/tests/durability_crash_documentation.rs index 8949f2d3..78193699 100644 --- a/xtask/tests/durability_crash_documentation.rs +++ b/xtask/tests/durability_crash_documentation.rs @@ -6,6 +6,10 @@ const ROOT_README: &str = include_str!("../../README.md"); const RECOVERY: &str = include_str!("../../docs/formats/segment-store-v1/recovery.md"); const REQUIREMENTS: &str = include_str!("../../docs/formats/segment-store-v1/requirements.md"); const CORPUS_README: &str = include_str!("../../conformance/segment-store/v1/README.md"); +const V2_CORPUS_README: &str = include_str!("../../conformance/segment-store/v2/README.md"); +const MIGRATION_CRASH: &str = + include_str!("../../docs/formats/segment-store-v2/migration-crash.md"); +const V2_REQUIREMENTS: &str = include_str!("../../docs/formats/segment-store-v2/requirements.md"); #[test] fn living_documentation_routes_the_complete_crash_matrix_and_its_limits() { @@ -14,6 +18,12 @@ fn living_documentation_routes_the_complete_crash_matrix_and_its_limits() { (RECOVERY, "## Process-death crash matrix"), (REQUIREMENTS, "`KEEP-RECOVERY-021`"), (CORPUS_README, "105 canonical process-death cases"), + (V2_CORPUS_README, "68 canonical process-death cases"), + ( + MIGRATION_CRASH, + "cargo xtask durability-crash-matrix --sequence migration", + ), + (V2_REQUIREMENTS, "`KEEP-MIGRATION-007`"), ] { assert!( document.contains(claim), @@ -24,4 +34,9 @@ fn living_documentation_routes_the_complete_crash_matrix_and_its_limits() { "Process-death injection, retention, compaction, and garbage collection remain planned." )); assert!(RECOVERY.contains("does not simulate host power loss")); + assert!( + !MIGRATION_CRASH + .contains("this page claims in-process recovery, not process-death recovery") + ); + assert!(!ROOT_README.contains("Process-death evidence for migration recovery")); } diff --git a/xtask/tests/durability_crash_point_contract.rs b/xtask/tests/durability_crash_point_contract.rs index 1945b010..06c21cea 100644 --- a/xtask/tests/durability_crash_point_contract.rs +++ b/xtask/tests/durability_crash_point_contract.rs @@ -4,7 +4,7 @@ use xtask::{DurabilityCrashPoint, DurabilityCrashSequence}; -use DurabilityCrashSequence::{Catalog, Head, Initialization, RecoveryDiscard, Segment}; +use DurabilityCrashSequence::{Catalog, Head, Initialization, Migration, RecoveryDiscard, Segment}; const EXPECTED: &[(DurabilityCrashPoint, &str, DurabilityCrashSequence)] = &[ ( @@ -162,6 +162,111 @@ const EXPECTED: &[(DurabilityCrashPoint, &str, DurabilityCrashSequence)] = &[ "KEEP-CRASH-035", Initialization, ), + ( + DurabilityCrashPoint::MigrationWriteIntentStage, + "KEEP-CRASH-053", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeIntentStage, + "KEEP-CRASH-054", + Migration, + ), + ( + DurabilityCrashPoint::MigrationLinkIntent, + "KEEP-CRASH-055", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeRootAfterIntent, + "KEEP-CRASH-056", + Migration, + ), + ( + DurabilityCrashPoint::MigrationRemoveIntentStage, + "KEEP-CRASH-057", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeRootAfterIntentCleanup, + "KEEP-CRASH-058", + Migration, + ), + ( + DurabilityCrashPoint::MigrationAdmitReaderFence, + "KEEP-CRASH-059", + Migration, + ), + ( + DurabilityCrashPoint::MigrationAdmitNamespacePrefix, + "KEEP-CRASH-060", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeRootAfterNamespace, + "KEEP-CRASH-061", + Migration, + ), + ( + DurabilityCrashPoint::MigrationWriteMarkerStage, + "KEEP-CRASH-062", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeMarkerStage, + "KEEP-CRASH-063", + Migration, + ), + ( + DurabilityCrashPoint::MigrationLinkMarker, + "KEEP-CRASH-064", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeRootAfterMarker, + "KEEP-CRASH-065", + Migration, + ), + ( + DurabilityCrashPoint::MigrationRemoveMarkerStage, + "KEEP-CRASH-066", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeRootAfterMarkerCleanup, + "KEEP-CRASH-067", + Migration, + ), + ( + DurabilityCrashPoint::MigrationWriteReceiptStage, + "KEEP-CRASH-068", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeReceiptStage, + "KEEP-CRASH-069", + Migration, + ), + ( + DurabilityCrashPoint::MigrationLinkReceipt, + "KEEP-CRASH-070", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeRootAfterReceipt, + "KEEP-CRASH-071", + Migration, + ), + ( + DurabilityCrashPoint::MigrationRemoveReceiptStage, + "KEEP-CRASH-072", + Migration, + ), + ( + DurabilityCrashPoint::MigrationSynchronizeRootAfterReceiptCleanup, + "KEEP-CRASH-073", + Migration, + ), ]; #[test] @@ -173,7 +278,7 @@ fn crash_boundaries_have_one_contiguous_stable_vocabulary() { } #[test] -fn only_record_append_selects_an_occurrence() { +fn only_record_append_and_namespace_prefix_select_an_occurrence() { let occurrence_counted: Vec<_> = DurabilityCrashPoint::ALL .into_iter() .filter(|point| point.occurrence_counted()) @@ -181,6 +286,58 @@ fn only_record_append_selects_an_occurrence() { assert_eq!( occurrence_counted, - [DurabilityCrashPoint::AppendSegmentRecord] + [ + DurabilityCrashPoint::AppendSegmentRecord, + DurabilityCrashPoint::MigrationAdmitNamespacePrefix + ] ); } + +#[test] +fn the_namespace_prefix_runs_one_during_case_per_directory() { + for point in DurabilityCrashPoint::ALL { + let expected = if point == DurabilityCrashPoint::MigrationAdmitNamespacePrefix { + DurabilityCrashPoint::NAMESPACE_PREFIX_DIRECTORIES + } else { + 1 + }; + assert_eq!( + point.during_occurrences(), + expected, + "{}", + point.identifier() + ); + } + assert_eq!(DurabilityCrashPoint::NAMESPACE_PREFIX_DIRECTORIES, 6); +} + +#[test] +fn migration_boundaries_follow_the_twenty_one_phases_in_order() { + let migration: Vec<_> = DurabilityCrashPoint::ALL + .into_iter() + .filter(|point| point.sequence() == Migration) + .collect(); + + assert_eq!(migration, DurabilityCrashPoint::MIGRATION); + assert_eq!(migration.len(), keep::StoreMigrationPhase::ALL.len()); + assert_eq!( + DurabilityCrashPoint::MIGRATION.map(DurabilityCrashPoint::identifier), + std::array::from_fn::<_, 21, _>(|index| { + let ordinal = 53 + index; + let identifier = format!("KEEP-CRASH-{ordinal:03}"); + DurabilityCrashPoint::from_identifier(&identifier) + .map_or("missing", DurabilityCrashPoint::identifier) + }) + ); +} + +#[test] +fn sequences_round_trip_their_command_line_identifiers() { + for sequence in DurabilityCrashSequence::ALL { + assert_eq!( + DurabilityCrashSequence::from_identifier(sequence.identifier()), + Some(sequence) + ); + } + assert_eq!(DurabilityCrashSequence::from_identifier("retention"), None); +} diff --git a/xtask/tests/durability_crash_production_contract.rs b/xtask/tests/durability_crash_production_contract.rs index 7c5cbf6a..eba7adcc 100644 --- a/xtask/tests/durability_crash_production_contract.rs +++ b/xtask/tests/durability_crash_production_contract.rs @@ -15,6 +15,7 @@ fn crash_children_execute_every_claimed_production_protocol() -> Result<(), Box< "durability_crash_matrix/production_protocol/initialization.rs", "durability_crash_matrix/production_protocol/publication.rs", "durability_crash_matrix/production_protocol/recovery.rs", + "durability_crash_matrix/production_protocol/migration.rs", ] .into_iter() .map(|path| fs::read_to_string(source_root.join(path))) @@ -27,6 +28,7 @@ fn crash_children_execute_every_claimed_production_protocol() -> Result<(), Box< "publish_catalog_generation(", "initialize_store(", "execute_recovery_stage_discard(", + "execute_store_migration(", ] { assert!( protocol.contains(required), diff --git a/xtask/tests/retention_store_v2_conformance_contract.rs b/xtask/tests/retention_store_v2_conformance_contract.rs index 38f7a601..9c947f34 100644 --- a/xtask/tests/retention_store_v2_conformance_contract.rs +++ b/xtask/tests/retention_store_v2_conformance_contract.rs @@ -17,6 +17,7 @@ const REQUIRED_PATHS: &[&str] = &[ "inventory.tsv", "migration-source.tsv", "artifacts.tsv", + "transitions.tsv", "format-marker.hex", "migration-intent.hex", "migration-receipt.hex", diff --git a/xtask/tests/retention_store_v2_protocol_contract.rs b/xtask/tests/retention_store_v2_protocol_contract.rs index 0847610f..d3e7fbe9 100644 --- a/xtask/tests/retention_store_v2_protocol_contract.rs +++ b/xtask/tests/retention_store_v2_protocol_contract.rs @@ -8,6 +8,8 @@ mod closure_contract_laws; mod migration_contract_laws; #[path = "retention_store_v2_protocol_contract/parser_fuzz_laws.rs"] mod parser_fuzz_laws; +#[path = "retention_store_v2_protocol_contract/transition_laws.rs"] +mod transition_laws; use std::fs; use std::io; diff --git a/xtask/tests/retention_store_v2_protocol_contract/transition_laws.rs b/xtask/tests/retention_store_v2_protocol_contract/transition_laws.rs new file mode 100644 index 00000000..d29f0934 --- /dev/null +++ b/xtask/tests/retention_store_v2_protocol_contract/transition_laws.rs @@ -0,0 +1,166 @@ +//! This module owns canonical migration transition-ledger admission laws. + +use keep::StoreMigrationPhase; + +const TRANSITIONS: &str = include_str!("../../../conformance/segment-store/v2/transitions.tsv"); +const HEADER: &str = "keep.segment-store.transitions/v2\n\ + crash_id\tphase\toperation\tpre_state\tinterrupted_class\t\ + post_state\trecovery_posture\n"; + +/// The operation column in `StoreMigrationPhase::ALL` order. +const OPERATIONS: [&str; 21] = [ + "write-intent-stage", + "sync-intent-stage", + "link-intent", + "sync-root-after-intent", + "remove-intent-stage", + "sync-root-after-intent-cleanup", + "admit-reader-fence", + "admit-namespace-prefix", + "sync-root-after-namespace", + "write-marker-stage", + "sync-marker-stage", + "link-marker", + "sync-root-after-marker", + "remove-marker-stage", + "sync-root-after-marker-cleanup", + "write-receipt-stage", + "sync-receipt-stage", + "link-receipt", + "sync-root-after-receipt", + "remove-receipt-stage", + "sync-root-after-receipt-cleanup", +]; + +#[derive(Debug, Eq, PartialEq)] +enum LedgerRefusal { + Encoding, + Header, + RowCount { observed: usize }, + Fields { row: usize }, + Coordinate { row: usize }, + Posture { row: usize }, + NamespaceExtent, +} + +#[test] +fn migration_transition_coordinates_are_complete_and_ordered() { + assert_eq!(OPERATIONS.len(), StoreMigrationPhase::ALL.len()); + assert_eq!(admit(TRANSITIONS), Ok(())); +} + +#[test] +fn transition_encoding_rejects_crlf_and_missing_final_newline() -> Result<(), String> { + let crlf = TRANSITIONS.replacen("resume-stage-sync\n", "resume-stage-sync\r\n", 1); + assert_eq!(admit(&crlf), Err(LedgerRefusal::Encoding)); + let unterminated = TRANSITIONS + .strip_suffix('\n') + .ok_or("fixture has no newline")?; + assert_eq!(admit(unterminated), Err(LedgerRefusal::Encoding)); + Ok(()) +} + +#[test] +fn discard_posture_cannot_be_hidden_in_another_column() { + let misplaced = TRANSITIONS.replacen( + "admitted-version-one-store\tabsent-or-incomplete-intent-stage\t\ + complete-intent-stage\tdiscard-incomplete-stage-or-resume-stage-sync", + "discard-incomplete-stage\tabsent-or-incomplete-intent-stage\t\ + complete-intent-stage\tresume-stage-sync", + 1, + ); + assert_ne!(misplaced, TRANSITIONS); + assert_eq!(admit(&misplaced), Err(LedgerRefusal::Posture { row: 0 })); +} + +#[test] +fn completion_posture_requires_its_exact_field_value() { + let altered = TRANSITIONS.replacen( + "\tadmit-complete-migration\n", + "\tunknown-admit-complete-migration\n", + 1, + ); + assert_eq!(admit(&altered), Err(LedgerRefusal::Posture { row: 19 })); +} + +#[test] +fn extra_transition_rows_refuse_before_row_admission() { + let extra = format!("{TRANSITIONS}foreign\n"); + assert_eq!(admit(&extra), Err(LedgerRefusal::RowCount { observed: 22 })); +} + +fn admit(document: &str) -> Result<(), LedgerRefusal> { + if !document.is_ascii() || document.contains('\r') || !document.ends_with('\n') { + return Err(LedgerRefusal::Encoding); + } + if !document.starts_with(HEADER) { + return Err(LedgerRefusal::Header); + } + let rows: Vec<_> = document.lines().skip(2).collect(); + if rows.len() != OPERATIONS.len() { + return Err(LedgerRefusal::RowCount { + observed: rows.len(), + }); + } + for (offset, row) in rows.iter().enumerate() { + admit_row(offset, row)?; + } + Ok(()) +} + +fn admit_row(offset: usize, row: &str) -> Result<(), LedgerRefusal> { + let fields: Vec<_> = row.split('\t').collect(); + let [id, phase, operation, pre, interrupted, post, posture]: [&str; 7] = fields + .try_into() + .map_err(|_| LedgerRefusal::Fields { row: offset })?; + if [id, phase, operation, pre, interrupted, post, posture] + .into_iter() + .any(str::is_empty) + { + return Err(LedgerRefusal::Fields { row: offset }); + } + let ordinal = offset + .checked_add(53) + .ok_or(LedgerRefusal::Coordinate { row: offset })?; + if id != format!("KEEP-CRASH-{ordinal:03}") + || phase != "migration" + || Some(&operation) != OPERATIONS.get(offset) + { + return Err(LedgerRefusal::Coordinate { row: offset }); + } + if Some(posture) != expected_posture(operation) { + return Err(LedgerRefusal::Posture { row: offset }); + } + if id == "KEEP-CRASH-060" && interrupted != "directory-prefix-length-zero-to-six" { + return Err(LedgerRefusal::NamespaceExtent); + } + Ok(()) +} + +fn expected_posture(operation: &str) -> Option<&'static str> { + match operation { + "write-intent-stage" | "write-marker-stage" | "write-receipt-stage" => { + Some("discard-incomplete-stage-or-resume-stage-sync") + } + "sync-intent-stage" | "sync-marker-stage" | "sync-receipt-stage" => { + Some("resume-stage-sync") + } + "link-intent" | "link-marker" | "link-receipt" => { + Some("verify-no-clobber-link-and-resume-root-sync") + } + "sync-root-after-intent" + | "sync-root-after-marker" + | "sync-root-after-receipt" + | "sync-root-after-namespace" => Some("resume-root-sync"), + "remove-intent-stage" + | "remove-marker-stage" + | "sync-root-after-intent-cleanup" + | "sync-root-after-marker-cleanup" => Some("resume-cleanup-sync"), + "admit-reader-fence" => Some("resume-namespace-prefix"), + "admit-namespace-prefix" => Some("resume-namespace-prefix-or-root-sync"), + "remove-receipt-stage" | "sync-root-after-receipt-cleanup" => { + Some("admit-complete-migration") + } + _ => None, + } +}