Skip to content

comline-ir-formats: frozen format 3 and the format gate (step 1.2 of #268) - #280

Merged
Kinflou merged 8 commits into
masterfrom
feat/ir-formats-gate
Nov 14, 2026
Merged

Kinflou merged 8 commits into
masterfrom
feat/ir-formats-gate

Conversation

@Kinflou

@Kinflou Kinflou commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Step 1.2 of #268 (IR migrations). The format stays 3; nothing here migrates, deletes or changes a store.

What is in it

comline-ir-formats (core/formats/, publish = false, no dependency on comline-core; deps are serde, ciborium, and arbitrary behind a feature). One directory per format, src/v3/, with the four files the design names:

  • types.rs: the frozen copy of format 3's types (the WIP commit's file, re-checked against master: no persisted type has changed since 07d86da), plus UnknownRecord and ProducedBy.
  • codec.rs: the only code that reads format 3's bytes: decode_unit, decode_config, decode_docs, decode_history, decode_unknown, decode_produced_by. A reader never turns bytes it can't read into an empty value.
  • SCHEMA.md: the document for format 3 (store layout, objects, every blob and entry, absence readings, the ledger table). Source for every format's document; the CAS guide keeps describing only the current one.
  • ledger.toml: 22 rows, 3.01 to 3.22: the twenty additions from the design's table, then the two new entries.

The seal. core/formats/MANIFEST: one line per superseded format, v<N> blake3:<hex> # <why>; the hash is over every file under src/v<N>/ (relative path, a zero byte, length, bytes, in path order). Empty today: format 3 is current and is sealed when format 4 is introduced. cargo xtask check-seal fails when a seal differs, when a superseded format isn't sealed, and when the current one is. cargo xtask seal-format <N> --why <text> writes or moves a seal and refuses without a reason.

live == frozen(current) (core/tests/migrations/live_frozen.rs). (a) every object of the stores built for the fixture packages (with a release and a second build, so commits have parents and a release): the live type and the frozen copy decode it and each re-encodes it to the same bytes; a root or dependency tree entry the frozen copy doesn't know fails with "this is a ledger row". (b) values of the frozen types drawn with Arbitrary (proptest, 512 cases each for schema units, config units, docs, facts, commit, tree, blob): the frozen copy's bytes decode in the live type and re-encode to the same bytes. (c) CURRENT == CURRENT_IR_FORMAT_VERSION, and the entry names agree with core's.

Golden fresh-build hashes. Corpus under core/tests/migrations/corpus/ (tour, headers, unverified with @unverified at argument, field, type and protocol scope plus handshake = "none" and a wire, and dependencies: app, acme with an encoding, base with another) plus tests/fixtures/packages/test. A fresh build of each writes the objects core/tests/migrations/expected/<name>.txt records: one line per entry of the root tree, recursively, <hash> <path>. The standard library's schema tree (compiled into a build that uses std) is not recorded, so a change to std doesn't move them. The dependencies fixture needs deps. Regenerate with COMLINE_UPDATE_GOLDEN=1 cargo test -p comline-core --features deps --test mod migrations::golden. ir_roundtrip_tests now reads its TOUR from the corpus file instead of holding a second copy.

Reserved entries and fail-closed readers. Root-tree entry unknown (CBOR [UnknownRecord { path, step, reason, original? }]) and produced_by (CBOR ProducedBy { core, spec }), both deny_unknown_fields, ledger rows 3.21 and 3.22. Nothing writes them. Core's cas::entries reads them: read_unknown, require_known (an error VersionHasUnknowns for a non-empty list, and an error for an entry that can't be decoded, isn't a blob or has a field this build doesn't know), read_produced_by (None means unknown, not the reading core). doc::read_docs, compat::read_history and compare call require_known, so no consumer takes such a version for complete.

cargo xtask check-format-bump [--base <rev>] (xtask/, alias in .cargo/config.toml, workflow .github/workflows/format-gate.yml, which passes the PR's base sha). Always: the current format's directory has its four files, and the seals hold. With --base, against the merge base:

  • CURRENT_IR_FORMAT_VERSION went up: it must be by one; v<C>/ has step.rs defining UNKNOWN, fn up and fn down; v<C>/SCHEMA.md starts # IR format <C>; format C-1 is sealed; every ledger row of C-1 names a fixture that exists under core/tests/migrations/stores/format-<C-1>/. Format 3's rows have none yet, so the format 4 pull request can't pass until cargo xtask ir-fixtures (step 1.5) exists and has run, which is the intent.
  • no bump: a row can't be removed; a new row needs SCHEMA.md and types.rs or codec.rs changed in the same change; moved golden hashes need a new row.
    The xtask's own tests build throw-away git repositories and show each failure.

Decisions to confirm

  1. Layout. The crate is core/formats/ (package comline-ir-formats), the format directory is core/formats/src/v3/, the seal is core/formats/MANIFEST. The design says formats/v<C>/ and formats/MANIFEST; this follows the WIP branch's core/formats/src/v3/. A bump's step lives in v<C>/step.rs.
  2. Entry names and shape. unknown and produced_by (one blob holding both versions, not two entries); JSON pointers for path; step as "3-4". The design left the names open.
  3. No Known<T> yet. read_produced_by returns Option (None is unknown). Known<T> belongs with the first format that uses it (format 4).
  4. A version with unknowns is refused, not read modulo unknown. The design's finer rules (strictest reading, unseen, compare modulo unknown) need the engine and a step that writes one; until then the strict reading is the only safe one.
  5. Ledger rows. The 20 historical rows are copied from the design's table (ids 3.NN, in its order); the two new ones are a row each for unknown and produced_by. fixture is optional and absent on all of them for now.
  6. Golden file format. One line per root-tree entry, recursively, no root tree hash and no commit (a root hash would move with std). The "moved golden hashes need a ledger row" rule is enforced by the xtask on the PR diff, not by the test: the test only says they moved.
  7. The CAS guide's stamp-table row (from the bump check in the design) can't be checked from here: the guide is in the docs repo. The check requires the # IR format <C> heading of SCHEMA.md instead. Say if the guide check should come back some other way.
  8. Random values in live == frozen are drawn from the frozen types (Arbitrary behind the crate's arbitrary feature, a dev-dependency of core only), not from the live ones: that catches a frozen copy that is wrong about a field, and the fixtures catch the live side. A value nested deeper than the CBOR reader's recursion limit is skipped.
  9. seal-format and check-seal are two xtask commands beyond the one the issue names; the design asks for a seal that moves only by a command that names why.
  10. Publishing. The crate is publish = false as designed, and comline-core depends on it with a path. cargo publish -p comline-core fails until the crate is published or the dependency is made optional.
  11. Not done: CODEOWNERS for the seal and the bump check (there is no CODEOWNERS file), and the format gate as a required check (branch protection).

@Kinflou
Kinflou marked this pull request as ready for review November 14, 2026 17:40
@Kinflou
Kinflou merged commit 01d647c into master Nov 14, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant