Repository navigation
Description
Proposal: ComlineProject/docs#171 (draft), docs/docs/proposals/ir-migrations/. Feeds #146 (reading older IR after 1.0) and #222 (granular @unverified, which needs IR format 4, item 3).
Why. ensure_compatible_store (core/src/package/build/cas/object_store.rs) deletes the whole .comline/ when its FORMAT_VERSION differs from CURRENT_IR_FORMAT_VERSION (3), deps-cache/ included, and it does the same to a store that is newer than the binary. Every package loses the history that compat and the changelog compare against.
Scope (decided). The formats that exist today will all be reset when 1.0 arrives, and the numbering restarts at format 0; the 1.0 format is the first with a guaranteed migration path. This issue is about the migration system and a proof that it is sound, not about migrating today's formats. Format 4 (#222 chains) is the first real client and proves the system before 1.0.
The rule (decided). Nothing is deleted or changed without the user doing so. No command migrates, resets, moves or deletes a store on its own. A command that finds an older store stops and names the command to run (comline store migrate, with --check as the dry run, or comline store reset); on a terminal it may ask whether to run that exact command and does nothing without a yes; in CI it fails with the message. Backups are removed only by comline store prune. Read commands may read an old store through an in-memory migration, which changes nothing on disk. A newer store is refused and never touched. Binaries from before this work are out of scope.
Decisions
- Steps are
N -> N+1, total and deterministic, over frozen format-N types. Every step has adownwitness (down(up(v)) == v), checked in tests and at migration time on the user's own store. - Whatever N never recorded is an explicit unknown (typed
Known<T>, or anunknownentry that keeps the original). Generators anddocfail closed: "unknown (migrated from format N)". - Atomic and resumable: write
.comline.migrating/, verify, rename the old store to.comline.format-N.bak/, rename the new one in. The backup stays untilcomline store prune.comline store resetmoves a store with no path to a backup after the user confirms that exact action;deps-cache/is kept. - A Git merge of stores in two formats is refused: "migrate both sides first".
- A dependency pin that verifies through the witness is accepted with a one-line notice suggesting a re-pin; never silent, never a failure. Pins are never rewritten.
- Each stored version records the core and spec versions that wrote it.
- Frozen formats live in the
comline-ir-formatscrate; the wasm IR-only build carries it only for in-memory, read-only migration. - 1.0: a warning release, then a 1.0 release that refuses pre-1.0 stores, names the reset command, and stamps
1.0/0. How long a bridge release is supported is decided at 1.0. - Proof: real stores built by the real core of each format, migrate(old build) == forget(fresh build), differential property tests, test-only formats for composition and lossy steps, laws (totality, determinism, idempotence, composition, hash stability, convergence), and a CI gate that fails a format bump without its step, fixtures and document.
Rollout (framework, then format 4 as the first client, then the 1.0 refusal and reset)
1. The framework (format stays 3)
- A newer store is an error; an older store is an error naming
comline store migrateorcomline store reset; nothing is changed automatically. Core and CLI message. Removes the wipe; released first. -
comline-ir-formatscrate: frozen format-3 types,SCHEMA.md, ledger,formats/MANIFESTseal, thelive == frozen(current)test, golden fresh-build hashes for the fixture packages. Ledger rows for theunknownentry and the core and spec versions that wrote a version, with fail-closed readers (nothing writes them yet). - Engine and
comline store(status,migrate [--check] [--assume-format N] [--map],verify,prune,reset): container reader, plan, journal, verification, swap, backup, the terminal prompt. - Proof harness on test-only formats (T0 to T1 to T2): composition, a lossy step with an untyped unknown, a typed unknown, steps wrong on purpose (the gate must reject them), crash injection; the laws.
- Fixture generator
cargo xtask ir-fixtures(scripted histories,PRODUCED_BY, lockfile, fake clock) and the weekly--check. - CI gate: seal,
live == frozen, golden fresh build,xtask check-format-bump, pin gate. - Differential property test (oracle generator extended with histories; the previous format's core out of process) and the
store_migratefuzz target.
2. Format 4, the first client (#222 item 3)
- The new frozen shape, step 3->4 (scope read from stored markers; chains and package-scope pins
Unknown), fixtures built by format 3's closing core, andSCHEMA.md, in one pull request, as the gate requires. - Pin bridge: a pin made under format N verifies through the witness, with the notice;
golden-pins.jsonfrom the previous core. - The first build after a migration compares modulo unknown (no commit for an unchanged tree).
- Acknowledgements pinned to chains are re-asked: the build names each protocol and prints the entry; nothing in
config.idpis rewritten. - Cross-package chains: recomputed from stored inputs for old versions in
diffand the changelog;unseenextended to paths and chains; the guarantee oracle (Guarantee safety: property test against an oracle, and a fuzz target (#223) #244) and its fuzz target also run over migrated stores.
3. The 1.0 refusal and reset
- A warning release that says, without changing anything, that 1.0 will refuse pre-1.0 stores.
- The 1.0 release refuses a pre-1.0 store, names
comline store reset, and stamps1.0/0. - From then on every format change is a step, with fixtures kept for ever (IR and config formats stable, or versioned with readers for older ones #146).
4. Tooling
-
doccache keyed on the IR format;comline newignores.comline.*.bak/. -
inspect,doc --versionand the language server show what isunseen; generators anddocfail closed on unknowns. - Docs: the CAS guide loses the deletion warning and describes the stamp, backups and
comline store.
Status: accepted, decided 2026-11-13. Nothing open.
SCHEMA.mdin core is the source for every format's document; the CAS guide describes only the current format.- At the 3->4 transition an unpinned
accept_unverifiedentry is re-asked, not accepted: the build stops and namescomline accept, andconfig.idpis never rewritten automatically.
Related: #146, #182 (a damaged store), #219, #220, #222, #244, #251, #261, #262.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDesign
Activity
A store in another IR format is an error; nothing deletes it (step 1.…
Kinflou commentedon Nov 14, 2026
Related: #278 proposes splitting the IR into its own crate,
comline-ir. It would be done together with step 1.2'scomline-ir-formatscrate, so core's workspace is rearranged once.comline-ir-formats: format 3's codec, SCHEMA.md, ledger and seal; xta…
Format gate: live == frozen(3), golden fresh-build hashes, fail-close…
comline-ir-formats: frozen format 3 and the format gate (step 1.2 of #…
Kinflou commentedon Nov 15, 2026
From #146 (closed into this issue): it also asked that a newer comline read what an older one wrote for config.idp, not only the stored IR. This design covers the stored formats; config.idp as a source language is not covered by it. Whether that needs its own rule or stays with language editions (specification_version) is still undecided, so it is noted here rather than lost.
Kinflou commentedon Nov 15, 2026
Audit 2026-11-16: ticked what is merged. Step 1.1: core#269 (a store in another format is an error and nothing is deleted). Step 1.2: core#280, the comline-ir-formats crate (core/formats) with format 3's frozen types, SCHEMA.md, the ledger (including the unknown and produced_by entries), the seal in MANIFEST, the live == frozen test, golden fresh-build hashes, and xtask check-format-bump and check-seal with the format-gate workflow. The CI gate box is left unticked because I did not confirm the pin gate part. Still open: the engine and comline store, the proof harness on test-only formats, cargo xtask ir-fixtures, the differential property test and store_migrate fuzz target, and steps 2 to 4. Also note #280's own list of what it left (CODEOWNERS, making the gate a required check).
Kinflou commentedon Nov 16, 2026
Noted from core#297 (qualified references): the frozen IR still holds one
Importper name a whole-namespaceusebrings in. Older dependencies' IR needs them, but packages compiled under the Rust-model rule don't read them. Dropping them is an IR migration, so it belongs here.