Skip to content

IR migrations: migrate every stored format accurately and provably #268

Open

Description

@Kinflou

Proposal: ComlineProject/docs#171 (draft), docs/docs/proposals/ir-migrations/. Feeds #146 (reading older IR after 1.0) and #222 (granular @unverified, which needs IR format 4, item 3).

Why. ensure_compatible_store (core/src/package/build/cas/object_store.rs) deletes the whole .comline/ when its FORMAT_VERSION differs from CURRENT_IR_FORMAT_VERSION (3), deps-cache/ included, and it does the same to a store that is newer than the binary. Every package loses the history that compat and the changelog compare against.

Scope (decided). The formats that exist today will all be reset when 1.0 arrives, and the numbering restarts at format 0; the 1.0 format is the first with a guaranteed migration path. This issue is about the migration system and a proof that it is sound, not about migrating today's formats. Format 4 (#222 chains) is the first real client and proves the system before 1.0.

The rule (decided). Nothing is deleted or changed without the user doing so. No command migrates, resets, moves or deletes a store on its own. A command that finds an older store stops and names the command to run (comline store migrate, with --check as the dry run, or comline store reset); on a terminal it may ask whether to run that exact command and does nothing without a yes; in CI it fails with the message. Backups are removed only by comline store prune. Read commands may read an old store through an in-memory migration, which changes nothing on disk. A newer store is refused and never touched. Binaries from before this work are out of scope.

Decisions

  • Steps are N -> N+1, total and deterministic, over frozen format-N types. Every step has a down witness (down(up(v)) == v), checked in tests and at migration time on the user's own store.
  • Whatever N never recorded is an explicit unknown (typed Known<T>, or an unknown entry that keeps the original). Generators and doc fail closed: "unknown (migrated from format N)".
  • Atomic and resumable: write .comline.migrating/, verify, rename the old store to .comline.format-N.bak/, rename the new one in. The backup stays until comline store prune. comline store reset moves a store with no path to a backup after the user confirms that exact action; deps-cache/ is kept.
  • A Git merge of stores in two formats is refused: "migrate both sides first".
  • A dependency pin that verifies through the witness is accepted with a one-line notice suggesting a re-pin; never silent, never a failure. Pins are never rewritten.
  • Each stored version records the core and spec versions that wrote it.
  • Frozen formats live in the comline-ir-formats crate; the wasm IR-only build carries it only for in-memory, read-only migration.
  • 1.0: a warning release, then a 1.0 release that refuses pre-1.0 stores, names the reset command, and stamps 1.0/0. How long a bridge release is supported is decided at 1.0.
  • Proof: real stores built by the real core of each format, migrate(old build) == forget(fresh build), differential property tests, test-only formats for composition and lossy steps, laws (totality, determinism, idempotence, composition, hash stability, convergence), and a CI gate that fails a format bump without its step, fixtures and document.

Rollout (framework, then format 4 as the first client, then the 1.0 refusal and reset)

1. The framework (format stays 3)

  • A newer store is an error; an older store is an error naming comline store migrate or comline store reset; nothing is changed automatically. Core and CLI message. Removes the wipe; released first.
  • comline-ir-formats crate: frozen format-3 types, SCHEMA.md, ledger, formats/MANIFEST seal, the live == frozen(current) test, golden fresh-build hashes for the fixture packages. Ledger rows for the unknown entry and the core and spec versions that wrote a version, with fail-closed readers (nothing writes them yet).
  • Engine and comline store (status, migrate [--check] [--assume-format N] [--map], verify, prune, reset): container reader, plan, journal, verification, swap, backup, the terminal prompt.
  • Proof harness on test-only formats (T0 to T1 to T2): composition, a lossy step with an untyped unknown, a typed unknown, steps wrong on purpose (the gate must reject them), crash injection; the laws.
  • Fixture generator cargo xtask ir-fixtures (scripted histories, PRODUCED_BY, lockfile, fake clock) and the weekly --check.
  • CI gate: seal, live == frozen, golden fresh build, xtask check-format-bump, pin gate.
  • Differential property test (oracle generator extended with histories; the previous format's core out of process) and the store_migrate fuzz target.

2. Format 4, the first client (#222 item 3)

  • The new frozen shape, step 3->4 (scope read from stored markers; chains and package-scope pins Unknown), fixtures built by format 3's closing core, and SCHEMA.md, in one pull request, as the gate requires.
  • Pin bridge: a pin made under format N verifies through the witness, with the notice; golden-pins.json from the previous core.
  • The first build after a migration compares modulo unknown (no commit for an unchanged tree).
  • Acknowledgements pinned to chains are re-asked: the build names each protocol and prints the entry; nothing in config.idp is rewritten.
  • Cross-package chains: recomputed from stored inputs for old versions in diff and the changelog; unseen extended to paths and chains; the guarantee oracle (Guarantee safety: property test against an oracle, and a fuzz target (#223) #244) and its fuzz target also run over migrated stores.

3. The 1.0 refusal and reset

4. Tooling

  • doc cache keyed on the IR format; comline new ignores .comline.*.bak/.
  • inspect, doc --version and the language server show what is unseen; generators and doc fail closed on unknowns.
  • Docs: the CAS guide loses the deletion warning and describes the stamp, backups and comline store.

Status: accepted, decided 2026-11-13. Nothing open.

  • SCHEMA.md in core is the source for every format's document; the CAS guide describes only the current format.
  • At the 3->4 transition an unpinned accept_unverified entry is re-asked, not accepted: the build stops and names comline accept, and config.idp is never rewritten automatically.

Related: #146, #182 (a damaged store), #219, #220, #222, #244, #251, #261, #262.

Activity

  1. added
    P1Next
    versioningVersions and compatibility
    designNeeds a design decision
    on Nov 12, 2026
  2. Kinflou commented on Nov 14, 2026

    @Kinflou
    MemberAuthor

    Related: #278 proposes splitting the IR into its own crate, comline-ir. It would be done together with step 1.2's comline-ir-formats crate, so core's workspace is rearranged once.

  3. moved this from Todo to Design in Comline Development Roadmapon Nov 15, 2026
  4. Kinflou commented on Nov 15, 2026

    @Kinflou
    MemberAuthor

    From #146 (closed into this issue): it also asked that a newer comline read what an older one wrote for config.idp, not only the stored IR. This design covers the stored formats; config.idp as a source language is not covered by it. Whether that needs its own rule or stays with language editions (specification_version) is still undecided, so it is noted here rather than lost.

  5. Kinflou commented on Nov 15, 2026

    @Kinflou
    MemberAuthor

    Audit 2026-11-16: ticked what is merged. Step 1.1: core#269 (a store in another format is an error and nothing is deleted). Step 1.2: core#280, the comline-ir-formats crate (core/formats) with format 3's frozen types, SCHEMA.md, the ledger (including the unknown and produced_by entries), the seal in MANIFEST, the live == frozen test, golden fresh-build hashes, and xtask check-format-bump and check-seal with the format-gate workflow. The CI gate box is left unticked because I did not confirm the pin gate part. Still open: the engine and comline store, the proof harness on test-only formats, cargo xtask ir-fixtures, the differential property test and store_migrate fuzz target, and steps 2 to 4. Also note #280's own list of what it left (CODEOWNERS, making the gate a required check).

  6. added
    1.0-blockerMust be done before the 1.0 announcement (docs conduct/readiness.md)
    on Nov 15, 2026
  7. added this to the 1.0 milestone on Nov 15, 2026
  8. Kinflou commented on Nov 16, 2026

    @Kinflou
    MemberAuthor

    Noted from core#297 (qualified references): the frozen IR still holds one Import per name a whole-namespace use brings in. Older dependencies' IR needs them, but packages compiled under the Rust-model rule don't read them. Dropping them is an IR migration, so it belongs here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    1.0-blockerMust be done before the 1.0 announcement (docs conduct/readiness.md)P1NextdesignNeeds a design decisionversioningVersions and compatibility

    Type

    No type

    Projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions