Test: verify migration compatibility and fuzz recovery transitions (#112)

CHANGELOG.md

@@ -8,6 +8,8 @@ after its public API and format compatibility policies are established.
 
 ## [Unreleased]
 
+- Migration compatibility laws preserve version-one bytes and reject version-one authority at every forward prefix; public version/flag refusal tests and bounded seeded recovery-planner fuzzing extend transition evidence (#112).
+
 - Completed migration recovery now verifies the version-two namespace before reporting success, refusing unknown reserved GC/recovery entries without effects while preserving published retention state (#111). Recovery storage implementors must supply the new read-only `verify_complete` capability.
 
 - Migration restart laws preserve complete filesystem witnesses when rejecting damaged records and pools, conflicting or substituted stages, invalid ordering, copied-root identity, changed inventory, foreign receipts, unknown names and wrong kinds (#111).

docs/formats/segment-store-v2/requirements.md

@@ -35,7 +35,7 @@ case is not evidence.
 | `KEEP-MIGRATION-005` | Unknown, out-of-order, substituted, corrupt, conflicting, or changed evidence is unrecoverable ambiguity | Forward-execution laws remain. Filesystem restart record, pair, ordering, root, pool and namespace laws assert exact existing refusal boundaries and preserve complete names, device/inode identities and bytes; see [restart matrix](../../testing-evidence/migration-restart-matrix.md) for scenarios, calibration, diagnostic limits and validation ownership. | Implemented in #111 candidate, including reserved complete-state namespace refusal; final review and integration pending |
 | `KEEP-MIGRATION-006` | Migration never rewrites or deletes admitted version-1 immutable bytes | exact segment, catalog, and head witnesses in `src/adapters/store_migration/filesystem_migration_storage_tests.rs`, `src/adapters/store_migration/filesystem_migration_recovery_tests.rs`, and `src/adapters/store_migration/filesystem_migration_recovery_truncation_tests.rs`; subprocess restart witnesses in `cargo xtask durability-crash-matrix --sequence migration` | Implemented in #108 |
 | `KEEP-MIGRATION-007` | Process death around every intent stage, canonical link, namespace prefix, marker stage, receipt stage, cleanup, and synchronization boundary reaches a documented lawful state | ordered phases and capabilities in `tests/store_migration_phase.rs` and `tests/store_migration_storage.rs`; exact phase-failure execution in `tests/store_migration_execution.rs`; production 21-phase forward execution in `filesystem_migration_storage_tests`; `cargo xtask durability-crash-matrix --sequence migration` runs 68 production subprocess cases at `KEEP-CRASH-053..=073`, debug and release | Implemented in #108 |
-| `KEEP-MIGRATION-008` | Version-1 admission refuses every version-2 or partial-migration artifact after migration begins | `FORMAT` refusal before mutation in `filesystem_migration_authority_tests`; exact version-1 reopen refusal of a migrated root and separate version-2 namespace admission in `filesystem_initialization_namespace`; version-2 reopen returns a distinct `FilesystemVersionTwoAdmission` that no version-1 publisher can consume (pinned by `tests/version_two_admission_contract.rs`), admits every version-2 protocol directory under the Linux profile, and jointly admits the exact marker, intent, and receipt before returning writer authority, with aliased-directory, corrupt, oversized, and mutually inconsistent record refusals in `filesystem_version_two_admission_tests` and `filesystem_platform_profile_tests`; remaining compatibility and fuzz matrix | In progress in #112 |
+| `KEEP-MIGRATION-008` | Version-1 admission refuses every version-2 or partial-migration artifact after migration begins | Every forward-prefix compatibility law preserves v1 HEAD/catalog/segment bytes and refuses v1 authority after migration effects; public decoder laws pin unsupported versions and every mandatory flag bit; bounded seeded migration parser/recovery-planner fuzzing explores valid and malformed transitions. Existing jointly bound version-two admission and root-identity laws remain. See [compatibility evidence](../../testing-evidence/migration-compatibility-fuzz.md) for coverage, calibration and limits. | Implemented |
 
 <!-- markdownlint-enable MD013 -->
 

docs/testing-evidence/migration-compatibility-fuzz.md

@@ -0,0 +1,50 @@
+# Migration compatibility and recovery fuzz evidence
+
+This change addresses #112 and KEEP-MIGRATION-008 from base main `6051abb25a9fd33ae7ee0de5614514b709a4d82a`. Change kind: missing runtime verification and stronger fuzz exploration, with no production behavior change. Owner: `@flyingrobots`. This branch is independent of the #111 restart ambiguity PR and does not claim mainline integration before merging.
+
+## Claims and owners
+
+`filesystem_migration_compatibility_tests::every_migration_prefix_preserves_v1_bytes_but_refuses_v1_authority` executes every complete forward-phase prefix in fresh filesystem storage. Before migration begins, version-one authority reopens and verifies the same intent. After any migration effect, reopening requires the existing exact `FilesystemPlatformAdmissionError::Namespace` / `InvalidData` refusal. After the attempt, every original version-one HEAD, segment and catalog name/byte pair must remain unchanged. Existing recovery-prefix and process-death laws retain responsibility for interrupted operations and restart recovery.
+
+`tests/store_migration_compatibility.rs` admits the golden migration records through public decoders and tests unsupported versions and every single mandatory flag bit in marker, intent and receipt. Expected coordinates come from the format's big-endian version and flags fields; precise `UnsupportedVersion` and `UnsupportedFlags` precede checksum refusal. Existing `tests/segment_header.rs`, `tests/segment_header/mutation_laws.rs`, `tests/catalog/header_laws.rs` and `tests/publication_head.rs` own version-one grammar/version/flag refusals.
+
+`migration_format` retains record parser fuzzing and adds bounded `plan_store_migration_recovery` exploration. Its properties require version-one success exactly when no migration evidence exists, precise effects-before-intent refusal, and complete success only with a full namespace, no stages and jointly admitted records. The complete-record property reuses product decoders; it checks agreement between planner and admission, not an independent implementation of the binary grammar. Other planner outcomes still receive robustness exploration rather than a complete reference-model oracle.
+
+## Corpus and bounds
+
+Checked-in seed recipes cover record admission, malformed marker version, unsupported intent flags, corrupt receipt checksum, valid migration prefixes, complete migration, corrupt intent, an effect before durable intent, a stage surviving an effect, a namespace hole and a receipt preceding its marker. They are materialized by `cargo xtask prepare-fuzz-corpus` and exercised through the existing migration target in smoke and scheduled workflows. No seed-count assertion establishes correctness.
+
+Recovery selector 3 is a fuzz-only envelope, not a durable format: a 256-byte expected intent, record-presence byte, namespace-presence byte and six length-prefixed record payloads. The envelope lengths use little-endian u16; embedded product records retain their canonical big-endian fields. Each payload is limited to 513 bytes, allowing overlong records while bounding payload allocation to 3,078 bytes. A complete envelope consumes at most 3,349 bytes including its selector; trailing fuzz-input bytes are ignored. This is an exploration bound, not a new storage limit. The checked-in campaign also enforces its own input, timeout and RSS bounds.
+
+The fixed local run uses pinned nightly `nightly-2026-07-24`, cargo-fuzz 0.13.2, seed 112, 20,000 executions, a 4,096-byte input cap, five-second per-input timeout and 1,024 MiB RSS limit. The named starting corpus is archived before launch; generated descendants are retained separately from the checked-in recipes. Replay `cargo +nightly-2026-07-24 fuzz run migration_format <restored-corpus> -- -runs=20000 -seed=112 -max_len=4096 -timeout=5 -rss_limit_mb=1024`. The instrumented binary exports `__asan_init`, and the release fuzz calibration demonstrates active assertions. No product crash or new real counterexample was found in that bounded run; it is not an exhaustive correctness claim.
+
+## Calibration and validation
+
+Separate copied-source/build mutations produce behavioral RED: admitting migrating namespaces as v1 incorrectly succeeds at the first migration prefix; damaging HEAD after the final migration phase fails byte preservation; bypassing marker version/flag checks changes the precise failures to checksum errors; forcing version-one success for a retained intent stage fails the fuzz oracle on its named seed. Mutations are excluded from the candidate. The initial test-authoring mistake using little-endian field patches and compiler/lint corrections are retained in logs but are not product RED evidence.
+
+Focused runtime tests pass in debug and release, and the bounded seeded fuzz run passes. The PR records immutable candidate coordinates, full workspace checks, migration crash campaign results and hosted CI; no earlier SHA's green result transfers automatically. Stable Rust is pinned to 1.96.0. Rust runs use copied Docker sources on Linux aarch64, not a writable host checkout.
+
+The filesystem law is medium-size and uses repository-only platform admission to isolate compatibility; it does not prove production platform eligibility. Decoder examples are small. Fuzzing invokes parser/planner runtime, not filesystem recovery execution, power-loss simulation or arbitrary writer interleavings. Existing platform, crash and recovery owners retain those claims. Ordinary-test resource enforcement gaps remain as disclosed in the testing enforcement profile.
+
+The seed materialization tool test no longer freezes global or per-target case counts. Mainline #172 now supplies the stronger materialization witness: a named emitted partial-seal counterexample reaches the real recovery classifier and produces its exact typed refusal, followed by deterministic repeated materialization. The integration preserves that witness instead of restoring the weaker nonempty-output assertion; migration product confidence comes from actual decoder, planner and filesystem outcomes. Deletion criterion: incidental inventory totals did not guard a product contract and rejected legitimate added exploration. Retire the new laws only when their compatibility contract disappears or stronger evidence subsumes it.
+
+## Landing calibration closure
+
+The fresh review of merged candidate `6e7e2d6a9c965f0240e006c7d45a5bc3c7010d37`, tracked tree `437bc1971fc8a98f2d039e446b3f5008979974f9`, found that the original Marker failures stopped execution before the independent Intent/Receipt checks, and the original VersionOne control did not reach the other fuzz properties. This is a proof gap, not a product bug. One bounded batch now demonstrates the distinct assertions below without changing their expected outcomes or the candidate's runtime code.
+
+| Control | Violated observation and intended runtime failure | Receipt |
+| --- | --- | --- |
+| [Intent header admission](migration-compatibility-fuzz/intent.patch) | Skip only Intent version/flag checks; unchanged Marker admission passes, then the Intent assertions observe checksum failures instead of exact version/flag refusals. | [RED](migration-compatibility-fuzz/intent-red.txt) |
+| [Receipt header admission](migration-compatibility-fuzz/receipt.patch) | Skip only Receipt version/flag checks; earlier Marker/Intent assertions pass, and Receipt's exact refusals fail. | [RED](migration-compatibility-fuzz/receipt-red.txt) |
+| [Pre-intent effect coordinate](migration-compatibility-fuzz/pre-intent.patch) | After the actual planner runs, replace its Namespace effect refusal with Marker; the unchanged VersionOne property passes, then the exact pre-intent assertion fails. | [RED](migration-compatibility-fuzz/pre-intent-red.txt) |
+| [Complete namespace](migration-compatibility-fuzz/namespace.patch) | After actual Complete planning, remove the observed reader fence; the full-namespace assertion fails. | [RED](migration-compatibility-fuzz/namespace-red.txt) |
+| [Complete stage absence](migration-compatibility-fuzz/stage.patch) | After actual Complete planning, add observed staged evidence; the namespace assertion passes, then stage absence fails. | [RED](migration-compatibility-fuzz/stage-red.txt) |
+| [Complete record admission](migration-compatibility-fuzz/records.patch) | After actual Complete planning, replace the receipt observation with invalid bytes; earlier properties pass, then joint record admission fails. | [RED](migration-compatibility-fuzz/records-red.txt) |
+
+The four fuzz controls calibrate the unchanged oracles by perturbing observations after calling the real planner. They are deliberately inconsistent observations, not claims of production planner defects or real on-disk counterexamples. The earlier filesystem authority, byte-preservation, Marker and VersionOne controls remain valid for their original scope. No mutation score or one-control-per-bit claim is made. An initial launcher expected the wrong source-path prefix in the fuzz panic text and stopped after the valid pre-intent failure; the raw assertion failure is retained, the log-matching prefix was corrected, and only the three remaining controls were launched. That launcher mismatch is not runtime RED.
+
+Apply one patch at a time with `git apply --unidiff-zero` to a fresh copy of the source above. For Intent/Receipt, run `cargo test --all-features --locked --test store_migration_compatibility`. For fuzz controls, first materialize the unchanged candidate's named seeds with `cargo xtask prepare-fuzz-corpus`, then run `cargo +nightly-2026-07-24 fuzz run migration_format <seed-file> -- -runs=1 -seed=112 -max_len=4096 -timeout=5 -rss_limit_mb=1024`. Use `recovery-effect-before-intent` for the pre-intent control and `recovery-complete` for the other three. Each experiment uses its own copied source/build directory; no mutation enters the PR.
+
+The unchanged candidate then passes [public decoder debug/release GREEN](migration-compatibility-fuzz/restored-headers-green.txt) and a fresh [20,000-run seeded fuzz GREEN](migration-compatibility-fuzz/restored-fuzz-green.txt). The latter's named starting corpus is archived before launch; generated descendants are separate retained execution artifacts. Both receipts pin the original tracked tree. Full copied-Docker validation also completes successfully at that tree, and all four hosted jobs in run `37155153324` pass. Final evidence-only successors still require their own hosted/static checks and exact-head review; those are recorded in the PR.
+
+Committed logs normalize only isolated container source/build prefixes and line-end/trailing-empty whitespace. Original raw logs, exact variants, source archive, starting corpus and traced launchers remain retained by the author. This closure adds no runtime behavior, exploration scope, physical-power-loss evidence or resource-enforcement claim beyond the boundaries above.

docs/testing-evidence/migration-compatibility-fuzz/intent-red.txt

@@ -0,0 +1,73 @@
++ cd <isolated-build>
++ CARGO_TARGET_DIR=<isolated-build>
++ cargo test --all-features --locked --test store_migration_compatibility
+   Compiling rustix v1.1.4
+   Compiling linux-raw-sys v0.12.1
+   Compiling io-lifetimes v2.0.4
+   Compiling bitflags v2.13.1
+   Compiling io-lifetimes v3.0.1
+   Compiling proc-macro2 v1.0.107
+   Compiling io-extras v0.19.0
+   Compiling cap-primitives v4.0.2
+   Compiling shlex v2.0.1
+   Compiling quote v1.0.47
+   Compiling once_cell v1.21.4
+   Compiling find-msvc-tools v0.1.9
+   Compiling unicode-ident v1.0.24
+   Compiling ambient-authority v0.0.2
+   Compiling maybe-owned v0.3.4
+   Compiling ipnet v2.12.0
+   Compiling cap-std v4.0.2
+   Compiling cap-fs-ext v4.0.2
+   Compiling clap_lex v1.1.0
+   Compiling cfg-if v1.0.4
+   Compiling cc v1.3.0
+   Compiling libc v0.2.186
+   Compiling anstyle v1.0.14
+   Compiling arrayref v0.3.9
+   Compiling arrayvec v0.7.8
+   Compiling constant_time_eq v0.4.2
+   Compiling condtype v1.3.0
+   Compiling regex-lite v0.1.9
+   Compiling allocation-counter v0.8.1
+   Compiling clap_builder v4.6.2
+   Compiling syn v2.0.119
+   Compiling blake3 v1.8.5
+   Compiling clap v4.6.4
+   Compiling rustix-linux-procfs v0.1.1
+   Compiling fs-set-times v0.20.3
+   Compiling divan-macros v0.1.21
+   Compiling keep v0.0.0 (<isolated-build>)
+   Compiling divan v0.1.21
+    Finished `test` profile [unoptimized + debuginfo] target(s) in 3.37s
+     Running tests/store_migration_compatibility.rs (<isolated-build>/debug/deps/store_migration_compatibility-fbb66f7c5fa9ff33)
+
+running 2 tests
+test unsupported_migration_versions_refuse_with_exact_coordinates ... FAILED
+test every_unknown_mandatory_flag_refuses_without_downgrade ... FAILED
+
+failures:
+
+---- unsupported_migration_versions_refuse_with_exact_coordinates stdout ----
+
+thread 'unsupported_migration_versions_refuse_with_exact_coordinates' (2069453) panicked at tests/store_migration_compatibility.rs:36:9:
+assertion `left == right` failed
+  left: Some(ChecksumMismatch { expected: [133, 68, 127, 122, 231, 209, 183, 129, 12, 27, 150, 206, 179, 58, 125, 26, 91, 117, 149, 153, 171, 142, 116, 148, 187, 207, 21, 50, 125, 116, 53, 155], observed: [123, 236, 16, 204, 140, 30, 239, 90, 176, 232, 232, 182, 163, 50, 64, 187, 162, 145, 37, 45, 66, 99, 20, 125, 241, 52, 6, 46, 183, 13, 63, 31] })
+ right: Some(UnsupportedVersion { expected: 2, observed: 0 })
+note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
+
+---- every_unknown_mandatory_flag_refuses_without_downgrade stdout ----
+
+thread 'every_unknown_mandatory_flag_refuses_without_downgrade' (2069452) panicked at tests/store_migration_compatibility.rs:74:9:
+assertion `left == right` failed
+  left: Some(ChecksumMismatch { expected: [146, 191, 58, 134, 238, 87, 46, 110, 169, 158, 196, 30, 241, 211, 86, 58, 134, 88, 230, 64, 167, 53, 64, 238, 64, 23, 23, 129, 132, 179, 158, 13], observed: [123, 236, 16, 204, 140, 30, 239, 90, 176, 232, 232, 182, 163, 50, 64, 187, 162, 145, 37, 45, 66, 99, 20, 125, 241, 52, 6, 46, 183, 13, 63, 31] })
+ right: Some(UnsupportedFlags { observed: 1 })
+
+
+failures:
+    every_unknown_mandatory_flag_refuses_without_downgrade
+    unsupported_migration_versions_refuse_with_exact_coordinates
+
+test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
+
+error: test failed, to rerun pass `--test store_migration_compatibility`

docs/testing-evidence/migration-compatibility-fuzz/intent.patch

@@ -0,0 +1,8 @@
+--- a/src/adapters/store_migration/migration_intent_decoder.rs
++++ b/src/adapters/store_migration/migration_intent_decoder.rs
+@@ -52 +52 @@ fn validate_fixed_fields(encoded: &[u8]) -> Result<(), StoreMigrationIntentDecod
+-    if version != format::VERSION {
++    if false && version != format::VERSION {
+@@ -66 +66 @@ fn validate_fixed_fields(encoded: &[u8]) -> Result<(), StoreMigrationIntentDecod
+-    if flags != 0 {
++    if false && flags != 0 {