Skip to content

Docs: reconcile living v1 pages with implemented recovery - #136

Merged
flyingrobots merged 8 commits into
mainfrom
docs/69-v1-current-behavior
Oct 2, 2026
Merged

flyingrobots merged 8 commits into
mainfrom
docs/69-v1-current-behavior

Conversation

@flyingrobots

Copy link
Copy Markdown
Owner

The living v1 format pages described implemented initialization, admission, and recovery as future work. This PR reconciles those claims with main's implementation and existing requirement/test anchors, while retaining historical issue references.

Evidence and approach

Checked the production initialize/reopen producers, gated repository-harness admission, canonical publication checks, recovery classifiers, and KEEP-RECOVERY-001–021 ledger. The documentation now names implemented behavior and the 105-case process-death matrix, without treating it as power-loss evidence. Whole-byte stage classification is described as the existing design rather than an absent streaming implementation.

The new documentation contract fails against main with the stale future-recovery claim and passes after correction in debug and release. Existing implementation-posture law also passes.

Validation

Pinned Rust 1.96.0: targeted documentation contracts in debug and release, formatting, workspace/all-target/all-feature Clippy with warnings denied, documentation integrity, source structure, dependency audit and policy checks. Existing runtime evidence was inspected; no fresh crash-matrix run is claimed for this prose change.

Compatibility and scope

No runtime, format, API, identity, durability, recovery, performance, or security changes. Reorganizing unrelated documentation and implementing v2 behavior are excluded. A new ADR or benchmark is unnecessary because this records existing decisions and evidence. Leaving stale future claims was rejected because it misstates the source boundary for migration work.

This branch starts directly at origin/main. Closes #69. Refs #132.

Replace stale future-work claims with current initialization, admission, publication, restart, and recovery evidence. Keep historical issue references and distinguish process death from power loss.

Red/green documentation contracts pass in debug and release after reproducing stale claims. Pinned fmt, Clippy, documentation/source checks, dependency audit and policy checks passed. Refs #69, #132.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 28 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 6d5daad0-eb31-4e81-893f-03b63244a877

📥 Commits

Reviewing files that changed from the base of the PR and between 2311d80 and 153bf05.

📒 Files selected for processing (6)
  • CHANGELOG.md
  • docs/formats/segment-store-v1/README.md
  • docs/formats/segment-store-v1/publication.md
  • docs/formats/segment-store-v1/recovery.md
  • docs/formats/segment-store-v1/requirements.md
  • xtask/tests/segment_store_implementation_documentation.rs
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Autopilot is currently an internal CodeRabbit preview.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@flyingrobots

Copy link
Copy Markdown
Owner Author

Code Lawyer current-head findings (c009895). Cc @codex.

Severity File/lines Issue Evidence Acceptance check
P4 docs/formats/segment-store-v1/recovery.md:60–63 Whole-byte classification incorrectly attributes materialization to the inventory reader FilesystemRecoveryInventoryReader::fingerprint_stage streams to evidence without content-sized allocation; it has no stage-byte materialization method. Callers supply classifier bytes; the segment resumer materializes through its pinned writable stage separately. Static regression rejects the false producer claim; prose distinguishes fingerprinting, caller-supplied classification bytes, and the resumer's materialization.
P4 docs/formats/segment-store-v1/recovery.md:121–124 Unqualified no-argument crash command now executes 173 cases, while its description says 105 xtask/src/durability_crash_matrix.rs:26–30 iterates all cases; PR #138 added 68 migration cases. The 105 v1 subset remains valid. State that the full command includes the 105 v1 subset plus additional version-two cases; keep the v1 point/count evidence precise.

These are documentation/integration defects; neither finding demonstrates a runtime recovery defect. Independent agy review is in progress before remediation.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

@flyingrobots

Copy link
Copy Markdown
Owner Author

Adversarial Code Review: PR 136 (docs/69-v1-current-behavior)

Repository: flyingrobots/keep
Target Branch: main at 2311d805e9bbcb1743fc5cdc46adb324641b9f14
PR Branch: docs/69-v1-current-behavior
PR Head: c0098953dceb78650d50db0547e715e71d8bc978
Initial PR Commit: 0e3dfbe3125831ab9abe527c69024573edc1e6de
Merge Base: f49cff732cf7a6e1b472decba9e4c4130990559e
Review Status: Complete Read-Only Adversarial Audit
Binding Verdict: REQUEST CHANGES


Executive Summary

PR 136 reconciles living version-1 documentation pages (README.md, publication.md, recovery.md, requirements.md), CHANGELOG.md, and adds an enforcement test in xtask/tests/segment_store_implementation_documentation.rs to reflect that issue #17 (initialization, platform admission, explicit recovery, and the 105-case process-death matrix) is fully implemented rather than future planned work.

The merge commit c009895 merged main at 2311d80 (incorporating PR 140, PR 145, PR 135, PR 137, and PR 138). Clean semantic integration with all merged main commits was verified: no conflicting files exist outside CHANGELOG.md, no unintended regressions or file deletions occurred, and all invariant constraints (including forbidden source basenames, host-independent benchmark sources, and bounded staging memory) remain intact.

However, an ultra-strict line-by-line verification of the changed statements against the underlying Rust source code and test fixtures revealed a P2 factual inaccuracy in normative recovery documentation (docs/formats/segment-store-v1/recovery.md), an editorial omission regarding transitive publication-view admission (P3), and technical phrasing imprecision regarding crash-matrix decorators (P4).


Findings

Finding 1 (P2): Factual inaccuracy regarding stage byte materialization by the inventory reader

  • Severity: P2 (Major documentation defect: false specification of adapter capability and dataflow).

  • Location: docs/formats/segment-store-v1/recovery.md:58-62

  • Code in PR:

    Every classifier consumes complete, protocol-bounded stage bytes that the inventory
    reader materializes after fingerprinting; the ledger's classification rows
    (`KEEP-RECOVERY-010`, `KEEP-RECOVERY-011`) are whole-byte by design, and no
    classifier streams from a filesystem handle.
  • Concrete Failure Scenario:
    An engineer, integrator, auditor, or subagent reading recovery.md relies on the assertion that "the inventory reader materializes [stage bytes] after fingerprinting". When designing recovery tooling or integrating storage adapters, they inspect FilesystemRecoveryInventoryReader to find byte-materialization APIs or assume the inventory reader buffers complete stages in memory.

    In reality:

    1. FilesystemRecoveryInventoryReader owns only namespace directory enumeration (read) and bounded streaming fingerprinting (fingerprint_stage). It has no API or method to materialize stage bytes or return buffer slices.
    2. FilesystemRecoveryInventoryReader::fingerprint_stage streams via fingerprint_recovery_stage using an 8 KiB buffer without retaining bytes in memory. Line 121 explicitly documents: "The synchronous call allocates no content-sized memory, may block on filesystem I/O, and performs no protocol mutation."
    3. When stage bytes are materialized in filesystem storage (such as during segment continuation), materialization is performed by ObservedRecoveryStage::materialize_and_position within the stage storage adapter, not by the inventory reader.
    4. For semantic stage assessment (assess_recovery_stage), caller-supplied bytes must first be admitted against prior observation evidence via admit_recovery_stage_bytes.
    5. The doc itself states in line 52 that classifiers process "complete caller-supplied stage bytes", creating an internal contradiction with line 59's claim that the inventory reader materializes them.
  • Suggested Fix:
    Update docs/formats/segment-store-v1/recovery.md to remove the claim that the inventory reader materializes stage bytes. For example:

    Every classifier consumes complete, protocol-bounded stage bytes admitted against
    prior fingerprint evidence (`admit_recovery_stage_bytes`); the ledger's classification
    rows (`KEEP-RECOVERY-010`, `KEEP-RECOVERY-011`) are whole-byte by design, and no
    classifier streams directly from a filesystem handle.

Finding 2 (P3): Silent removal and lack of traceability for transitive publication-view admission

  • Severity: P3 (Moderate documentation gap: dropped requirement claim without explanation).
  • Location: docs/formats/segment-store-v1/recovery.md:55-58
  • Code in PR:
    -distinguishing exact truncation from complete canonical bytes. Transitive
    -publication-view admission and filesystem-streaming semantic classification
    -remain unimplemented.
    +distinguishing exact truncation from complete canonical bytes. Every
    +classifier consumes complete, protocol-bounded stage bytes that the inventory
    +reader materializes after fingerprinting; the ledger's classification rows
    +(`KEEP-RECOVERY-010`, `KEEP-RECOVERY-011`) are whole-byte by design, and no
    +classifier streams from a filesystem handle.
  • Context & Evidence:
    In earlier issue Build the durable crash-injection and recovery matrix #17 iterations, recovery.md recorded that "Transitive publication-view admission and filesystem-streaming semantic classification remain unimplemented."
    To pass the newly introduced test in xtask/tests/segment_store_implementation_documentation.rs:48 (assert!(!document.contains("remain unimplemented"))), the entire sentence was deleted and replaced.
    While the replacement correctly notes that filesystem-streaming semantic classification was rejected in favor of whole-byte classification by design, it completely dropped any mention or explanation of transitive publication-view admission.
    Transitive catalog snapshot verification during head.next recovery is an implemented protocol requirement (KEEP-RECOVERY-017 & KEEP-RECOVERY-018, implemented in plan_recovery_next_head_finalization).
  • Suggested Fix:
    Clarify how transitive publication-view admission is handled under the implemented architecture (e.g. explicitly citing that candidate head.next recovery verifies the complete transitive catalog snapshot under KEEP-RECOVERY-017/018 before finalization).

Finding 3 (P4): Imprecise description of crash-matrix decorators vs harnesses


Mandatory Verification Checklist

1. Runtime Paths Traced (file:line to file:line)

Runtime Behavior Production Path Test / Repository Task Path Rule Parity & Verification
Store Initialization FilesystemPlatformAdmission::initialize -> FilesystemInitializationStorage::admit -> initialize_store FilesystemPlatformAdmission::initialize_unchecked_for_tests; RepositoryInitializationStorage::admit_unchecked Both production and test paths execute the identical 6-phase state machine (KEEP-RECOVERY-002) and root synchronization. Test/task path bypasses only the strict ext4 platform profile via lenient capability opening.
Store Reopen FilesystemPlatformAdmission::reopen -> filesystem_platform_profile::open -> reopen_root FilesystemPlatformAdmission::reopen_unchecked_for_tests; FilesystemVersionTwoAdmission::reopen (v2) Both verify exact root directory contents (writer.lock, staging, segments, catalogs, and regular HEAD). Mutates nothing. v2 reopen additionally enforces device/inode coordinates against migration.intent (PR 137).
Catalog Publisher Opening FilesystemCatalogPublisher::open consuming FilesystemPlatformAdmission FilesystemCatalogPublisher::open_unchecked_for_repository_tasks; open_unchecked_for_tests Production requires FilesystemPlatformAdmission whose private fields can only be initialized by Keep. Repository task path uses feature = "repository-tasks". Both pin root, staging, segments, and catalogs without following links.
Stage Residue Refusal during Publication publish_catalog_generation -> filesystem_catalog_current::verify_current tests/catalog_filesystem_publication.rs Invariant parity verified: unowned current.seg, any head.next, or any current.cat causes immediate refusal (ErrorKind::AlreadyExists) before mutation. Empty immutable pools required when HEAD is absent.
Recovery Stage Fingerprinting FilesystemRecoveryInventoryReader::fingerprint_stage -> filesystem_recovery_stage::fingerprint -> fingerprint_recovery_stage tests/recovery_stage_fingerprint.rs Streaming 8 KiB buffer under KEEP:RECOVERY:STAGE\0 BLAKE3 domain. Verified zero heap byte materialization. Verified post-read namespace and entry checks.
Stage Byte Admission & Assessment admit_recovery_stage_bytes -> assess_recovery_stage tests/recovery_stage_assessment/ Materialized bytes admitted only when stage, length, and recomputed fingerprint match prior evidence. Dispatches to pure whole-byte classifiers (classify_recovery_segment_stage, classify_recovery_catalog_stage, classify_recovery_next_head_stage).
Segment Continuation execute_recovery_segment_resume -> FilesystemRecoverySegmentResumer::open_reusable tests/recovery_segment_resume.rs Only path that materializes stage bytes on disk via materialize_and_position (filesystem_recovery_stage.rs:57). Re-fingerprints and positions handle at append boundary without rewriting prefix.
Next-Head Recovery Finalization plan_recovery_next_head_finalization -> RecoveryNextHeadFinalizationStorage tests/recovery_next_head_finalization.rs Enforces candidate snapshot generation, catalog length, and digest agreement with candidate head (KEEP-RECOVERY-017). Synchronizes candidate before atomic rename (KEEP-RECOVERY-018).
Living Docs Assertion living_v1_pages_no_longer_assign_shipped_recovery_to_a_future_issue N/A (xtask integration test) Scans FORMAT_README, PUBLICATION, RECOVERY, and REQUIREMENTS for 7 specific stale planning strings. Statically verified that all 7 assertions pass on the tree.

2. Merges Audited (SHA and Integration Invariants)

  • Merge Commit: c0098953dceb78650d50db0547e715e71d8bc978
  • Parent 1: 0e3dfbe3125831ab9abe527c69024573edc1e6de (docs branch)
  • Parent 2: 2311d805e9bbcb1743fc5cdc46adb324641b9f14 (main integrating PR 138)
  • Merge Base: f49cff732cf7a6e1b472decba9e4c4130990559e

Invariants verified across merged commits:

  1. PR 140 (eec39ba / 971f03f — ambient-CPU-independent source law):
    • Verified that no benchmark source files or CPU model checks were touched or disrupted by PR 136.
  2. PR 145 (88f35c4 / 05f7ef6 — forbidden source filenames):
  3. PR 135 (07bf0b8 / e43ad0e, b5def4a — bounded reference-staging memory contract):
    • Staging memory contract tests and laws in src/reference/ remain untouched.
  4. PR 137 (200cfc8 / b3bd395, a4ff000, b691da7 — restart device/inode vs live mount identity):
    • Version 2 reopen invariant comparing device/inode coordinates across restart remains intact in src/adapters/filesystem_version_two_admission.rs.
  5. PR 138 (2311d80 / c2414bf..bb1e6f6 — partial-prefix migration recovery & 68 crash cases):
    • Verified that the migration crash cases and transition ledger remain completely untouched.
  6. Conflict Resolution & Semantic Equivalence:
    • git diff 2311d80 c009895 -- . ':(exclude)CHANGELOG.md' ':(exclude)docs/formats/segment-store-v1' ':(exclude)xtask/tests/segment_store_implementation_documentation.rs' is completely empty (0 diff).
    • In CHANGELOG.md, the merge cleanly incorporated the PR 136 entry alongside the main entries under ### Fixed.

3. Constants and Evidence Coordinates Checked

Constant / Coordinate Documented Location Tree Source / Evidence Binding Threshold & Evaluation
Domain Separator KEEP:RECOVERY:STAGE\0 docs/formats/segment-store-v1/requirements.md:125, 129 src/adapters/recovery/recovery_stage_fingerprinter.rs:10 Exact byte sequence: b"KEEP:RECOVERY:STAGE\0" verified.
Streaming Fingerprint Buffer 8,192 bytes (8 KiB) docs/formats/segment-store-v1/requirements.md:125 src/adapters/recovery/recovery_stage_fingerprinter.rs:11 Constant BUFFER_LENGTH: usize = 8_192. Zero content-sized heap allocation verified.
Catalog Stage Interruption 176 bytes docs/formats/segment-store-v1/publication.md:231 xtask/src/durability_crash_matrix/production_protocol/publication_storage.rs:14 Constant CATALOG_INTERRUPTION: usize = 176 verified.
Head Stage Interruption 64 bytes docs/formats/segment-store-v1/publication.md:253 xtask/src/durability_crash_matrix/production_protocol/publication_storage.rs:15 Constant HEAD_INTERRUPTION: usize = 64 verified.
Crash Restart Byte Limit 1,048,576 bytes (1 MiB) KEEP-RECOVERY-021 xtask/src/durability_crash_matrix/production_protocol/initialization.rs:14 Constant RESTART_BYTE_LIMIT: u64 = 1_048_576 verified.
Catalog Pool Digest Fixture Conformance vectors tests/catalog.rs:10 04b82519b0399baefd0b9c0f32a871052e4c47e3a00226ab03b21661470f7320 verified across 7 separate fixture sites.

4. Doc Figures and Counts Checked

Count / Figure Document Claim Raw Tree Evidence Verification Result
v1 Process-Death Crash Cases: 105 docs/formats/segment-store-v1/README.md:11, requirements.md:138, 195, recovery.md:123, conformance/segment-store/v1/README.md:108 35 v1 points (KEEP-CRASH-001–KEEP-CRASH-035) × 3 positions (Before, During, After) = 105 cases. Tested in xtask/tests/durability_crash_documentation.rs:20. Verified Exact.
v2 Migration Crash Cases: 68 conformance/segment-store/v2/README.md:95, docs/formats/segment-store-v2/migration-crash.md:7 xtask/tests/durability_crash_case_contract.rs:39: assert_eq!(migration.len(), 68) Verified Exact.
Total Repository Crash Cases: 173 N/A (Internal matrix bound) xtask/tests/durability_crash_case_contract.rs:36: assert_eq!(cases.len(), 173) (105 v1 + 68 v2 migration = 173). Verified Exact.
v1 Recovery Requirements: 21 docs/formats/segment-store-v1/requirements.md:118-138 Table rows KEEP-RECOVERY-001 through KEEP-RECOVERY-021. Verified Exact (21 contiguous rows).
v1 Crash Points: 35 docs/formats/segment-store-v1/requirements.md:118 KEEP-CRASH-001 through KEEP-CRASH-035 in xtask/src/durability_crash_point.rs. Verified Exact (35 points).
Forbidden Source Basenames: 9 CHANGELOG.md:657 xtask/src/source_structure/forbidden_filename.rs:7-17 Verified Exact (9 names).
Stale Claim Test Count: 7 xtask/tests/segment_store_implementation_documentation.rs:40-51 7 tuples tested in for (document, stale_claim). Verified Exact.

5. Repository Standards Compliance

  • Pure Rust Project (AGENTS.md line 13): Compliant (no python scripts).
  • Code Size & Line Limits (AGENTS.md lines 49-57):
    • xtask/tests/segment_store_implementation_documentation.rs: 58 physical lines (Target: 200, Max: 500).
    • New test function living_v1_pages_no_longer_assign_shipped_recovery_to_a_future_issue: 19 lines (Target: 20, Max: 60).
    • Maximum nesting depth: 2 (loop + assert).
    • Function parameters: 0.
  • Deny unwrap, panic, todo (AGENTS.md line 16): Compliant (standard assert! used in test).
  • Markdown Formatting & Line Lengths (Documentation Standards §7.2, §8):
    • All modified prose lines in markdown files are <= 80 columns.
    • git diff --check origin/main..HEAD executed and reported 0 whitespace errors.
    • Wide table rows use standard <!-- markdownlint-disable MD013 -->.
  • Rust Formatting (rustfmt.toml):
    • max_width = 100. The two added lines in xtask/tests/segment_store_implementation_documentation.rs are 94 and 88 columns, strictly compliant.

6. Review Execution & Coverage Ledger

In accordance with mandatory protocol:

  • Checks Executed:
    • git diff 2311d80..HEAD (full diff inspection)
    • git log and git diff on merge commit c009895 against both parents 0e3dfbe and 2311d80
    • git merge-base 0e3dfbe 2311d80
    • git diff --check origin/main..HEAD (whitespace and line-ending verification)
    • String search and line-length calculation across all modified lines
    • Static evaluation of xtask/tests/segment_store_implementation_documentation.rs against all 4 inspected documents
  • Checks Inspected Statically (Read-Only):
    • Rust source implementations in src/adapters/filesystem_recovery_inventory_reader.rs, src/adapters/recovery/, src/adapters/filesystem_store_initializer.rs, src/adapters/filesystem_platform_admission.rs, src/adapters/filesystem_catalog_publisher.rs, src/adapters/filesystem_recovery_stage.rs, and xtask/src/durability_crash_matrix/
    • All 173 crash case definitions and counts in xtask/tests/durability_crash_case_contract.rs and conformance/
  • Checks Skipped / Unavailable:
    • Host execution of cargo test / cargo clippy / docker was explicitly skipped per reviewer instructions ("Do not run host tests. Primary agent handles Docker validation."). Static inspection is not dynamic execution.
    • Physical power loss was not simulated; as documented in recovery.md and requirements.md, process-death injection is distinct from host power-loss evidence.

Verdict

REQUEST CHANGES

Summary of Required Actions Before Approval

  1. Correct docs/formats/segment-store-v1/recovery.md:58-62 to remove the inaccurate statement that the inventory reader materializes stage bytes. State instead that stage bytes are caller/adapter materialized and admitted against prior fingerprint evidence via admit_recovery_stage_bytes.
  2. Clarify in docs/formats/segment-store-v1/recovery.md how transitive publication-view admission is handled (i.e. governed during head.next recovery under KEEP-RECOVERY-017/018 via CatalogSnapshot verification).
  3. Refine docs/formats/segment-store-v1/publication.md:137-140 to accurately describe that test harnesses obtain the unchecked publisher and wrap it in fault-injecting decorators.

Primary reconciliation: finding 1 is confirmed and deduplicated with the previously posted materialization finding. Finding 3 is confirmed. Finding 2's claim that transitive verification was entirely dropped is superseded by the existing “Leftover next head” section, which already names the exact CatalogSnapshot, planner, executor, filesystem finalizer, and complete transitive current/candidate checks. I will add a local cross-reference and KEEP-RECOVERY-017/018 anchors to improve the opening section's traceability without claiming the behavior was missing. The primary audit also found that the unqualified crash command runs all 173 cases while this v1 page describes only 105; that separate integration finding will be corrected. Cc @codex.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

@flyingrobots

Copy link
Copy Markdown
Owner Author

Additional Code Lawyer findings while checking #69's acceptance criterion that every living v1 page describes current main. Cc @codex.

Severity Location Verified issue Acceptance
P4 requirements.md KEEP-SEGMENT-008/009 Evidence names tests/segment_filesystem_stage.rs, which does not exist. The four filesystem laws live in src/adapters/filesystem_segment_stage_tests.rs. Correct both paths and verify the referenced source exists; preserve IDs and requirement ownership.
P4 requirements.md KEEP-SEGMENT-005 The status claims an implemented prohibition on mutable sealed-stage exposure without disclosing the repository-tasks map_stage escape already demonstrated and tracked in #146. Preserve the normative immutability requirement and explicitly mark the known repository-task exception pending #146; do not pretend the runtime defect is fixed here.

The runtime #146 fix stays a separate coherent PR. These changes correct its evidence/status representation, not its acceptance criterion.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

@flyingrobots

Copy link
Copy Markdown
Owner Author

Code Lawyer Activity Summary — current head 153bf05

Item Severity/source File Commit RED → GREEN evidence Outcome
Materialization attributed to inventory reader P4 primary / P2 agy recovery.md; documentation law 5f511c6 New static law fails at c009895 in debug/release; passes after producer correction in both profiles Fixed
Full crash command described as only v1 cases P4 primary recovery.md; documentation law 4791ba0 New static law fails at c009895 in debug/release; passes after explicit subset qualification Fixed
Harness construction conflated with decorators P4 agy publication.md 4e78c84 Docker static check rejects old producer wording; corrected harness/decorator distinction passes Fixed
Transitive verification overview traceability P3 agy, reconciled recovery.md 0cdec01 Existing Leftover next head section already proves prior description; new local link absent before, present after; 017/018 match planner/finalizer Clarified; claim of entirely missing behavior disputed with existing section evidence
Nonexistent filesystem test anchors P4 primary requirements.md c86119a Docker confirms old path absent and stale references present, then stale refs absent and actual module exists Fixed
Undisclosed repository-task sealed-stage escape P4 primary requirements.md 153bf05 Status caveat absent before, present after; source still exposes feature-gated map_stage Documentation fixed; runtime #146 remains open

The full initial agy feedback/checklist was posted before remediation. All fixes are focused commits and published. No review threads exist to resolve; exhausted discovery contains only global comments.

Passed: pre-remediation exact-head full Docker workspace debug/release, doctests, fmt, both Clippy profiles, source policy; updated targeted documentation laws debug/release and fmt; updated eight-page Markdown lint (0.23.2). The first attempted Docker copy targeted the loop mount directly and failed before the new tests were installed; it is setup failure, not RED evidence. Actual RED results above came after copying through the container's visible temporary directory.

Pending: independent agy repeat review with the mandatory current-head Verification Checklist, clean-bundle dedicated-target full workspace revalidation, and new-head hosted CI. No merge approval is claimed yet. Format/API/runtime bytes are unchanged. No new benchmark or physical power-loss evidence is claimed. Runtime #146 and broader unfinished roadmap requirements remain open. Cc @codex.

@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

@flyingrobots

Copy link
Copy Markdown
Owner Author

Adversarial Code Review: PR 136 (docs/69-v1-current-behavior)

Repository: flyingrobots/keep
Target Branch: main at 2311d805e9bbcb1743fc5cdc46adb324641b9f14
PR Branch: docs/69-v1-current-behavior
PR Head: 153bf057377ad4495e88e83a8f82da4dcf00fc3d
Initial PR Commit: 0e3dfbe3125831ab9abe527c69024573edc1e6de
Mainline Merge Commit: c0098953dceb78650d50db0547e715e71d8bc978
Common Merge Base: f49cff732cf7a6e1b472decba9e4c4130990559e
Review Status: Complete Read-Only Adversarial Audit (Post-Remediation Verification)
Binding Verdict: APPROVE


Executive Summary

PR 136 reconciles living version-1 documentation pages (README.md, publication.md, recovery.md, and requirements.md), updates CHANGELOG.md, and adds living documentation posture tests in xtask/tests/segment_store_implementation_documentation.rs to reflect that issue #17 (initialization, platform admission, explicit recovery, and the 105-case process-death matrix) is fully implemented rather than future planned work.

Following the initial REQUEST CHANGES verdict on c009895, the PR branch received six focused remediation commits:

  1. 5f511c6 corrects recovery stage materialization ownership, removing the claim that the inventory reader materializes stage bytes, and adds a static regression test.
  2. 4791ba0 clarifies that the 105 v1 crash cases are a subset of the full 173-case matrix executed by the unqualified command, and adds an enforcement test.
  3. 4e78c84 refines the publication documentation to distinguish test harnesses (which obtain the unchecked publisher) from decorators (which inject faults).
  4. 0cdec01 links candidate head.next transitive publication-view admission to the "Leftover next head" section and requirements KEEP-RECOVERY-017/018.
  5. c86119a repairs broken evidence paths for KEEP-SEGMENT-008 and 009, pointing to the existing src/adapters/filesystem_segment_stage_tests.rs.
  6. 153bf05 discloses the pending repository-tasks sealed-stage capability escape tracked in issue Prevent post-seal writable stage escape through receipt mapping #146 while preserving the normative immutability requirement.

All six commits, the merge commit c009895, and every changed line against main have been exhaustively verified against the underlying Rust source code, tests, and conformance artifacts. Zero unresolved defects remain.


Prior Findings Audit & Resolution Verification

Prior Finding ID Initial Severity Verified Issue & Root Cause Remediation Commit & Mechanism Head Verification Status
Finding 1 (Prior agy P2 / Self-finding) P2 / P4 docs/formats/segment-store-v1/recovery.md:58-62 stated that the inventory reader materializes stage bytes. FilesystemRecoveryInventoryReader::fingerprint_stage is streaming and allocates zero content memory. Materialization is performed on disk by FilesystemRecoverySegmentResumer::open_reusable via materialize_and_position. 5f511c6 rewrote the passage to state that classifiers consume caller-supplied bytes, the inventory reader returns fingerprint evidence without retaining stage bytes, and the segment resumer separately materializes its pinned writable stage. Added regression test recovery_documentation_does_not_assign_materialization_to_the_inventory_reader in xtask/tests/segment_store_implementation_documentation.rs:47-53. VERIFIED RESOLVED. Doc matches code; test passes.
Finding 2 (Prior agy P3) P3 docs/formats/segment-store-v1/recovery.md:55-58 silently removed the mention of transitive publication-view admission without linking to its implemented boundary. 0cdec01 added an explicit cross-reference in recovery.md:71-75 linking transitive publication-view admission for candidate head.next to the "Leftover next head" section and requirements KEEP-RECOVERY-017 / KEEP-RECOVERY-018. VERIFIED RESOLVED. Traceability restored and verified against plan_recovery_next_head_finalization.
Finding 3 (Prior agy P4) P4 docs/formats/segment-store-v1/publication.md:137-140 stated that "fault-injecting decorators obtain an unchecked value". In reality, test harnesses obtain the unchecked publisher and pass it to wrapping decorators. 4e78c84 refined phrasing: "The crash matrix harness opens an unchecked publisher only behind the repository-tasks Cargo feature, then wraps the publisher in fault-injecting decorators." VERIFIED RESOLVED. Strictly matches initialization.rs:38 and publication_storage.rs:17-29.
Finding 4 (Primary Self-finding) P4 docs/formats/segment-store-v1/recovery.md:121-124 stated that cargo xtask durability-crash-matrix executes 105 canonical cases. With PR 138 merged, the no-argument command executes 173 cases (including 68 v2 migration cases). 4791ba0 updated recovery.md:130-133 to state that the 105 version-one cases are a subset of the complete command, which also executes version-two migration cases. Added regression test version_one_crash_evidence_is_distinguished_from_the_complete_command in xtask/tests/segment_store_implementation_documentation.rs:11-17. VERIFIED RESOLVED. Strictly matches durability_crash_matrix.rs:25-30 and durability_crash_case_contract.rs:36-39.
Finding 5 (Primary Ledger Finding) P4 In docs/formats/segment-store-v1/requirements.md, KEEP-SEGMENT-008 and 009 cited tests/segment_filesystem_stage.rs, a nonexistent file path. c86119a corrected the path to src/adapters/filesystem_segment_stage_tests.rs. VERIFIED RESOLVED. Target file exists and contains the four filesystem laws.
Finding 6 (Primary Ledger Finding) P4 In docs/formats/segment-store-v1/requirements.md, KEEP-SEGMENT-005 claimed implemented status without disclosing the repository-tasks feature escape SealedSegment::map_stage tracked in issue #146. 153bf05 updated the status column of KEEP-SEGMENT-005 to: "Production API implemented in #15; repository-task escape tracked in #146". VERIFIED RESOLVED. Discloses the capability escape accurately without falsely claiming runtime fix #146 here.

Mandatory Verification Checklist

1. Runtime Paths Traced (file:line to file:line)

Runtime Behavior Production Path Test / Repository Task Path Rule Parity & Invariant Verification
Store Initialization FilesystemPlatformAdmission::initialize -> FilesystemInitializationStorage::admit -> initialize_store FilesystemPlatformAdmission::initialize_unchecked_for_tests; RepositoryInitializationStorage::admit_unchecked Both production and test paths execute the identical 6-phase state machine (KEEP-RECOVERY-002) and root synchronization. Test/task path bypasses only the strict ext4 platform profile via lenient capability opening.
Store Reopen FilesystemPlatformAdmission::reopen -> filesystem_platform_profile::open -> reopen_root FilesystemPlatformAdmission::reopen_unchecked_for_tests; FilesystemVersionTwoAdmission::reopen (v2) Both verify exact root directory contents (writer.lock, staging, segments, catalogs, and regular HEAD). Mutates nothing. v2 reopen additionally enforces device/inode coordinates against migration.intent (PR 137).
Catalog Publisher Opening FilesystemCatalogPublisher::open consuming FilesystemPlatformAdmission FilesystemCatalogPublisher::open_unchecked_for_repository_tasks; open_unchecked_for_tests Production requires FilesystemPlatformAdmission whose private fields can only be initialized by Keep. Repository task path uses feature = "repository-tasks". Both pin root, staging, segments, and catalogs without following links.
Stage Residue Refusal during Publication publish_catalog_generation -> filesystem_catalog_current::verify_current tests/catalog_filesystem_publication.rs Invariant parity verified: unowned current.seg, any head.next, or any current.cat causes immediate refusal (ErrorKind::AlreadyExists) before mutation. Empty immutable pools required when HEAD is absent.
Recovery Stage Fingerprinting FilesystemRecoveryInventoryReader::fingerprint_stage -> filesystem_recovery_stage::fingerprint -> fingerprint_recovery_stage tests/recovery_stage_fingerprint.rs Streaming 8 KiB buffer under KEEP:RECOVERY:STAGE\0 BLAKE3 domain. Verified zero heap byte materialization. Verified post-read namespace and entry checks.
Stage Byte Admission & Assessment admit_recovery_stage_bytes -> assess_recovery_stage tests/recovery_stage_assessment.rs Materialized bytes admitted only when stage, length, and recomputed fingerprint match prior evidence. Dispatches to pure whole-byte classifiers (classify_recovery_segment_stage, classify_recovery_catalog_stage, classify_recovery_next_head_stage).
Segment Continuation & Materialization execute_recovery_segment_resume -> FilesystemRecoverySegmentResumer::open_reusable tests/recovery_segment_resume.rs The only recovery path that materializes stage bytes on disk via materialize_and_position. Re-fingerprints and positions handle at append boundary without rewriting prefix.
Next-Head Recovery Finalization plan_recovery_next_head_finalization -> RecoveryNextHeadFinalizationStorage -> FilesystemRecoveryNextHeadFinalizer tests/recovery_next_head_finalization.rs Enforces candidate snapshot generation, catalog length, and digest agreement with candidate head (KEEP-RECOVERY-017). Synchronizes candidate before atomic rename (KEEP-RECOVERY-018).
Sealed Stage Handle Immutability SealedSegment::close SealedSegment::map_stage (#[cfg(feature = "repository-tasks")]) Production public API consumes the sealed stage and exposes no mutable handle (KEEP-SEGMENT-005). Repository-task escape allows mapping internal stage decorators, tracked in #146.
Living Documentation Posture living_v1_pages_no_longer_assign_shipped_recovery_to_a_future_issue recovery_documentation_does_not_assign_materialization_to_the_inventory_reader; version_one_crash_evidence_is_distinguished_from_the_complete_command Scans FORMAT_README, PUBLICATION, RECOVERY, and REQUIREMENTS for 7 stale planning strings, false inventory materialization claims, and unqualified 105-case scope claims. Statically verified that all assertions evaluate to true.

2. Merges Audited (SHA and Integration Invariants)

Invariants verified across merged commits:

  1. PR 140 (eec39ba — ambient-CPU-independent source law):
    • Verified: PR 136 makes zero modifications to benches/, xtask/src/benchmark/, or CPU model checking code.
  2. PR 145 (88f35c4 — forbidden source filenames):
    • Verified against xtask/src/source_structure/forbidden_filename.rs:7-17. PR 136 introduces no new files, and none of the nine forbidden basenames (utils.rs, helpers.rs, common.rs, misc.rs, shared.rs, manager.rs, service.rs, types.rs, models.rs) exist in the PR diff.
  3. PR 135 (07bf0b8 — bounded reference-staging memory contract):
    • Verified: PR 136 makes zero modifications to src/reference/ or reference memory allocation test suites.
  4. PR 137 (200cfc8 — restart device/inode vs live mount identity):
    • Verified: PR 136 leaves src/adapters/filesystem_version_two_admission.rs untouched; device/inode restart verification remains fully active.
  5. PR 138 (2311d80 — partial-prefix migration recovery & 68 crash cases):
    • Verified: PR 136 leaves migration recovery in src/adapters/store_migration/ and conformance/segment-store/v2/ untouched. The 68 migration crash cases are explicitly accounted for in commit 4791ba0.
  6. Conflict Resolution & Changelog Invariants:
    • git diff 2311d80 c009895 -- . ':(exclude)CHANGELOG.md' ':(exclude)docs/formats/segment-store-v1' ':(exclude)xtask/tests/segment_store_implementation_documentation.rs' is completely empty (0 diff).
    • In CHANGELOG.md:624-660, merge commit c009895 cleanly combined PR 136 entries with main's PR 138 entries under ### Fixed, preserving both histories without dropping any entry.

3. Constants and Evidence Coordinates Checked

Constant / Coordinate Documented Location Tree Source / Evidence Binding Threshold & Evaluation
Domain Separator KEEP:RECOVERY:STAGE\0 docs/formats/segment-store-v1/requirements.md:125, 129, recovery.md:38, 278 src/adapters/recovery/recovery_stage_fingerprinter.rs:10 Exact byte literal b"KEEP:RECOVERY:STAGE\0" verified.
Streaming Fingerprint Buffer 8,192 bytes (8 KiB) docs/formats/segment-store-v1/requirements.md:125 src/adapters/recovery/recovery_stage_fingerprinter.rs:11 Constant BUFFER_LENGTH: usize = 8_192. Zero heap allocation verified.
Max Recovery Inventory Entry Limit 2,097,152 / 2,097,153 docs/formats/segment-store-v1/recovery.md:15-18, catalog.md:169 xtask/tests/segment_store_protocol_contract/recovery_laws.rs:8 MAX_RECOVERY_INVENTORY_ENTRY_COUNT = 2_097_152. Refusal on first excess entry reports 2,097,153. Verified.
Catalog Stage Interruption 176 bytes docs/formats/segment-store-v1/publication.md:231 xtask/src/durability_crash_matrix/production_protocol/publication_storage.rs:14 Constant CATALOG_INTERRUPTION: usize = 176 verified.
Head Stage Interruption 64 bytes docs/formats/segment-store-v1/publication.md:253 xtask/src/durability_crash_matrix/production_protocol/publication_storage.rs:15 Constant HEAD_INTERRUPTION: usize = 64 verified.
Crash Restart Byte Limit 1,048,576 bytes (1 MiB) KEEP-RECOVERY-021 xtask/src/durability_crash_matrix/production_protocol/initialization.rs:14 Constant RESTART_BYTE_LIMIT: u64 = 1_048_576 verified.
Publication Head Length 128 bytes docs/formats/segment-store-v1/recovery.md:268, 302 src/adapters/publication_head.rs Constant PUBLICATION_HEAD_LENGTH: usize = 128 verified.
Catalog Pool Digest Fixture Conformance vectors tests/catalog.rs:10 04b82519b0399baefd0b9c0f32a871052e4c47e3a00226ab03b21661470f7320 verified across 7 separate fixture sites.

4. Doc Figures and Counts Checked

Count / Figure Document Claim Raw Tree Evidence Verification Result
v1 Process-Death Crash Cases: 105 docs/formats/segment-store-v1/README.md:11, requirements.md:138, 195, recovery.md:130, conformance/segment-store/v1/README.md:108 35 v1 points (KEEP-CRASH-001–KEEP-CRASH-035) × 3 positions (Before, During, After) = 105 cases. Tested in xtask/tests/durability_crash_documentation.rs:20. Verified Exact.
v2 Migration Crash Cases: 68 conformance/segment-store/v2/README.md:68-75, docs/formats/segment-store-v2/migration-crash.md:7 xtask/tests/durability_crash_case_contract.rs:39: assert_eq!(migration.len(), 68). Verified Exact.
Total Crash Cases Executed by Default Command: 173 docs/formats/segment-store-v1/recovery.md:130-133 xtask/tests/durability_crash_case_contract.rs:36: assert_eq!(cases.len(), 173) (105 v1 + 68 v2 migration = 173). Verified Exact.
v1 Recovery Requirements: 21 docs/formats/segment-store-v1/requirements.md:118-138 Table rows KEEP-RECOVERY-001 through KEEP-RECOVERY-021. Verified Exact (21 contiguous rows).
v1 Crash Points: 35 docs/formats/segment-store-v1/requirements.md:118 KEEP-CRASH-001 through KEEP-CRASH-035 in xtask/src/durability_crash_point.rs. Verified Exact (35 points).
Forbidden Source Basenames: 9 CHANGELOG.md:657 xtask/src/source_structure/forbidden_filename.rs:7-17. Verified Exact (9 names).
Stale Claim Test Tuples: 7 xtask/tests/segment_store_implementation_documentation.rs:56-67 7 tuples tested in for (document, stale_claim). Verified Exact.
PR 136 Changelog Items: 7 CHANGELOG.md:627-650 7 distinct bullet points detailing each doc fix. Verified Exact.

5. Repository Standards Compliance

  • Pure Rust Project (AGENTS.md:13): Strictly compliant (no python scripts).
  • Code Size & Line Limits (AGENTS.md:49-57):
    • xtask/tests/segment_store_implementation_documentation.rs: 74 physical lines (Target: 200, Max: 500).
    • Test functions:
      • version_one_crash_evidence_is_distinguished_from_the_complete_command: 6 lines (Target: 20, Max: 60).
      • recovery_documentation_does_not_assign_materialization_to_the_inventory_reader: 6 lines (Target: 20, Max: 60).
      • living_v1_pages_no_longer_assign_shipped_recovery_to_a_future_issue: 19 lines (Target: 20, Max: 60).
    • Maximum nesting depth: 2 (loop + assert).
    • Function parameters: 0.
  • Deny unwrap, panic, todo (AGENTS.md:16): Strictly compliant (assert! used in test functions).
  • Markdown Formatting & Line Lengths:
    • git diff --check origin/main..HEAD executed and reported 0 whitespace errors.
    • All modified prose lines in markdown files are <= 80 columns.
    • Wide table rows use standard <!-- markdownlint-disable MD013 -->.
  • Rust Formatting (rustfmt.toml):
    • max_width = 100. All modified lines in xtask/tests/segment_store_implementation_documentation.rs are <= 93 columns.
  • Link Target Validity:

6. Review Execution & Coverage Ledger


Verdict

APPROVE

Primary Code Lawyer final Activity Summary and merge judgment

This supersedes pending gates in the earlier Activity Summary. All six focused findings are addressed at 153bf05. The transitive-view concern was reconciled as an overview traceability improvement, since the existing later section already described that implementation. Runtime #146 remains open and is now disclosed accurately.

Passed on a clean Git-bundle clone at this exact SHA with its own Cargo target directory in Docker: full workspace/all-feature debug and release suites including doctests; formatting; debug and release workspace/all-target/all-feature Clippy with -D warnings; source-structure policy. Updated eight-page Markdown lint passes. All four hosted jobs pass in CI run 36964001601, including documentation/refusal/link/whitespace gates, fuzz smoke and dependency policy. agy completed with APPROVE and the full mandatory checklist above. Exhausted GitHub review discovery has no reviews and no review threads; CodeRabbit remains explicitly rate-limited, not an approval.

MERGE GATE: OPEN under the user's explicit authorization for clean agy review plus green validation in place of the unavailable skill-only two-reviewer/cooldown gates. Actual repository protections remain enforced. No runtime/format/API changes or new benchmark, crash-matrix execution, or physical power-loss evidence are claimed for this documentation PR. Cc @codex.

@flyingrobots
flyingrobots merged commit 99551ec into main Oct 2, 2026
5 checks passed
@flyingrobots
flyingrobots deleted the docs/69-v1-current-behavior branch October 2, 2026 04:25
@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create an environment for this repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refresh segment-store v1 living documentation after recovery delivery

1 participant