diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index f996eb7..83e75fa 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -137,6 +137,24 @@ read in order: scope that guarantee has, checked against what core computes. Revisits part of the guarantee-marker decision. *accepted 2026-11-12; not built yet.* +#### [Migrating stored IR across format versions](ir-migrations/index.md) + +How the version history in `.comline/` survives a change of the IR format +instead of being deleted: each format written down, a migration as a chain of +total, deterministic steps over frozen copies of the formats that write what an +old format never recorded as an explicit unknown, a new store verified beside +the old one before it replaces it, and a proof (a corpus of real stores built by +the historical cores, an equivalence against a fresh build, differential +property tests and a CI gate) that fails a format bump without its step. +Seven parts, read in order: [the formats](ir-migrations/formats.md), +[steps](ir-migrations/steps.md), +[running a migration](ir-migrations/running.md), +[proving it](ir-migrations/proof.md), +[hashes, pins and history](ir-migrations/hashes-and-history.md), +[beyond the local store](ir-migrations/beyond-the-store.md) and +[format 4 as the first client](ir-migrations/format-4.md). *accepted 2026-11-13; tracked in +[core#268](https://github.com/ComlineProject/core/issues/268).* + ### Connections - [Connections](connections.md): what a connection carries beyond one client diff --git a/docs/docs/proposals/ir-migrations/beyond-the-store.md b/docs/docs/proposals/ir-migrations/beyond-the-store.md new file mode 100644 index 0000000..d55224f --- /dev/null +++ b/docs/docs/proposals/ir-migrations/beyond-the-store.md @@ -0,0 +1,92 @@ +# Beyond the local store + +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) + +**The question.** The local `.comline/` is not the only place frozen IR, or a hash of it, is kept or relied on. What +does each of the others need, and what does a published package need when there are some? + +## Every artifact + +| Artifact | Holds | Migration story | +|---|---|---| +| **`.comline/` store** (`objects/`, `refs/`, `FORMAT_VERSION`) | every version, as frozen IR | [Migrated](running.md), whole, with the old one kept; before 1.0, where no step exists, [the user resets it](#the-10-reset) into a backup | +| **`.comline/deps-cache/`** | git checkouts of dependencies: source, not IR | Not migrated, not part of the store. **Carried across the swap**: today's wipe deletes it, and an offline machine can't fetch it again | +| **A dependency's own committed `.comline/`** | its version history; the only thing core reads of it is the newest commit's `version` (`resolved_version`) | Read with the format-independent [container reader](formats.md#what-a-store-holds). Today any failure reads as `0.0.0-unbuilt`; it should say the store couldn't be read | +| **Dependency pins** in `config.idp` | scheme v2 hash over IR bytes | Never rewritten. [Bridged](hashes-and-history.md#pins) | +| **Acknowledgements** in `config.idp`: a consumer's `accept_unverified`, a package's `unverified = { … }` | the set of guarantees accepted, per protocol | Never rewritten. [Re-asked](format-4.md#acknowledgements) when they are pinned to something the old format didn't record | +| **Generated code** | `schema_ir_hash` as the handshake's `ir_hash` | Derived, not migrated: regenerate. The hash follows the units' bytes, so a format change that alters them is visible at a connection. The pull request says so, and the [gate](proof.md#the-gate) lists it | +| **The generator protocol** | each request carries `ir_format` and the frozen units of the schemas | The units a generator gets from an old version are the migrated ones, in the current format. What to do with unknowns: [format 4](format-4.md#consumers) | +| **`comline doc` output and cache** (`.comline-doc.json`) | pages rendered per version, keyed by commit hash and the renderer | Derived. A migration changes commit hashes, so it re-renders once; the key should also hold the IR format | +| **The doc model** (`schemas/doc-model.json`) | a JSON schema, checked by a test | Derived, versioned on its own | +| **The language server, the playground, the editor** | no store; the server resolves dependencies without fetching and excludes `.comline/` from its watching | Nothing to migrate. Each ships a core, and that core's format is what it compiles against | +| **Published packages, the registry** | none yet: [no `publish` step exists](../../guides/ir/cas.md#what-to-commit), the registry server is on hold and core rejects registry dependencies | [Below](#published-packages-and-the-registry) | + +There is no `.cmln` archive or equivalent in any repository. Core has a `PublishRegistry` config unit (where a package +publishes) and nothing that builds an archive. + +## Published packages and the registry + +When there is something to publish, the [offline proposal](https://github.com/ComlineProject/docs/pull/163) already +decides that **identity is the hash and location is replaceable**: a dependency is its pin, and caches and bundles are +keyed by it. The rules for IR follow from that. + +- **Migrate on read, never rewrite what is published.** A published archive is addressed by its hash, and perhaps + signed. Rewriting it is making another artifact. A consumer that meets an archive whose store is in an older + format reads it through an in-memory migration, as `comline diff` reads an old local store. Its result, if cached, is + cached beside the archive and keyed by the archive's hash and the target format. +- **An archive says its format outside its payload,** in its manifest, so that a consumer can refuse a newer one + without decoding it, with the same error as a local store: the format, and the comline that reads it. +- **What a consumer needs from a dependency is its source.** A dependency is compiled from source, and its pin is checked + against that compile. The stored IR is only for comparing versions (compatibility, the changelog). So a version + too old to migrate costs a comparison, "can't tell", and never a build. +- **The registry holds, and never migrates.** It serves old archives as they were published. Migration is the + consumer's, in the consumer's core. +- **A pin never depends on which format an archive's store is in.** That is the requirement the + [pin bridge](hashes-and-history.md#pins) meets. + +## The 1.0 reset + +**Decided:** the formats that exist today are all reset when 1.0 arrives, and the numbering restarts: the 1.0 format is +**format 0** of a new numbering, and the first with a guaranteed path forward. Nothing before it is migrated. And, like +everything here, **nothing happens on its own**: + +1. **A warning release first.** The release before 1.0 says, once per command that finds a pre-1.0 store, that 1.0 will + refuse it and which command to run. It changes nothing. +2. **The 1.0 release refuses a pre-1.0 store.** Any command that finds one stops, names `comline store reset`, and + changes nothing. The reset is the same [user command](running.md#reset) as before 1.0: it renames the store to + `.comline.format-N.bak/` after the user confirms that exact action, keeps `deps-cache/`, and starts a new store at + the next build, stamped format 0 of the 1.0 numbering. The backup stays until `comline store prune`. +3. **The stamp has an epoch.** Pre-1.0 stamps are the integers 0 to 4, and a 1.0 `0` or `3` must not be read as one of + them. So the 1.0 stamp reads `1.0/0`, and a bare integer is by definition pre-1.0. +4. **The frozen types of the pre-1.0 formats are not kept:** the refusal needs only the stamp and the container, which + are format-independent ([formats](formats.md#what-a-store-holds)). +5. Every format change after it is a step, with a document, fixtures and a seal, as the gate requires. The system is + already in use by then: format 4's step is its first run. + +## After 1.0 + +[core#146](https://github.com/ComlineProject/core/issues/146) asks that from 1.0 a newer comline reads what an older one +wrote, "either freeze them or keep a reader for every older version, with tests". This proposal is the second: the +framework *is* the readers, and its fixtures and laws are the tests. + +| | Before 1.0 | From 1.0 | +|---|---|---| +| **Formats that migrate** | none promised: format 3 for the first client, the rest reset | every format released at or after 1.0, for ever. A released format's frozen types, steps, fixtures and seal are never removed | +| **A store with no path** | refused; the user's `comline store reset` moves it to a backup | refused, and the error names the **bridge release**: the last comline that still migrates it. There is no reset | +| **Dropping a format** | any minor release | a major one, announced one minor earlier; the bridge release stays installable | +| **Cost of a bump** | a step and its fixtures | the same, **and permanent**: it is carried for ever | +| **Additive changes** | a [ledger](formats.md#format-3-is-a-family) row each | a ledger row each, a fixture, and the same lines in the frozen copy | + +Because each bump is permanent from 1.0, bumps are worth batching: the policy proposed is that a format bump is +planned, not incidental. Cheap changes are additive ones, and the gate makes those cheap. + +A *version* is not a format. Package version 0.3.0, built long ago in a format that still migrates, is migrated like +every version of its store: nothing is dropped, and it can be compared at the precision it recorded, nothing finer. + +The spec side needs nothing of this: the spec is append-only for anything frozen +([the vocabulary](../shared-vocabulary/catalog.md#versioning)), so an old version reads under a newer spec without a step. +Only the IR format does. + +## Decided here + +- **How long a bridge release stays supported is decided at 1.0,** not before. diff --git a/docs/docs/proposals/ir-migrations/format-4.md b/docs/docs/proposals/ir-migrations/format-4.md new file mode 100644 index 0000000..30d917a --- /dev/null +++ b/docs/docs/proposals/ir-migrations/format-4.md @@ -0,0 +1,108 @@ +# Format 4 as the first client + +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) + +**The question.** [Granular `@unverified`](../shared-vocabulary/granular-unverified.md), item 3 of +[core#222](https://github.com/ComlineProject/core/issues/222), changes what a version freezes. What does the 3→4 step +do with every format-3 version already stored, and what is the package author asked? + +Format 4 is not designed here. It is the shape that proposal accepted, and this page is what the step does with it. +Anything that depends on the final shape is marked. + +**It is the first real client of the system,** and exists to prove it before 1.0, because today's formats won't be +migrated ([scope](index.md#scope-the-system-not-the-history)). One pull request exercises, on a real change: the frozen +copy and ledger, a step with `up`, `down` and an `unknown` list, a typed unknown, the square against a fresh format-4 +build, the laws, the pin bridge, and the gate. The test-only formats of [the proof](proof.md#test-only-formats) cover +what one step can't (composition, an untyped unknown); the two together are the system's proof. A format-3 store that +exists when the framework ships is migrated for real (when the user runs `comline store migrate`), and one older than that is not migrated: the user resets it with `comline store reset`. + +## What changes + +Format 3 freezes what core worked out as a **set**, and where it was written: + +- a protocol's `parameters` hold `Unverified(Vec)` beside the `@unverified` reason, and, since + [#265](https://github.com/ComlineProject/core/pull/265), markers on arguments, functions, fields and types are + properties in *their* parameters; +- the package's own `unverified` is a config unit, a `ConfigValue` as written; +- a declared encoding records what it `establishes` and whether it `has_vectors`. + +Format 4 freezes, per the decision, **(guarantee, scope path, chain to each cause, reason)**, so that a comparison sees +narrowing and widening and an acknowledgement is pinned to what it covered. + +## The step + +| Item | In format 3 | In format 4 | What `up` writes | +|---|---|---|---| +| each given-up guarantee | a member of a protocol's set | a record per guarantee | one record per member; the `guarantee` is carried | +| its **scope** | where the marker is written: the protocol, or an inner property since #265 | a scope path | the path of the marker that covers it, read from the stored units. Before #265 only a protocol could carry one, so the protocol is **known**, not guessed | +| the **chain** to each cause | not stored | the hops, package by package | **`Unknown`**, typed, with the reason "format 3 records the set, not the path to its causes" | +| its **reason** | one string on the plain marker, or a dictionary as written | a reason per guarantee | a plain marker's string is copied to each guarantee it covers, as written; a dictionary's entries as written | +| a package-default cause, acknowledged in `config.idp` | the `Unverified` config unit, as written | pinned to the setting that causes it | the unit carried; its pin to the setting is **`Unknown`** | +| a declared encoding's guarantees | `establishes`, `has_vectors`, as they were (the strict reading where absent) | the same | carried | + +Two things it does **not** do. It doesn't compute the chains: that needs the guarantee engine and the catalog, which +are live code and change, and a step is frozen. And it doesn't write an empty chain list or a one-hop chain to a +guarantee whose causes weren't recorded; that would read as "no other cause". + +The step is `up`, `down` and an `unknown` list, like any. `down` drops the typed unknowns and the scope paths that +equal the old protocol-level reading, and restores the old set, so the witness holds. The `unknown` list is the +two rows above, and the square compares the migration with a fresh format-4 build whose chains and package pins are +forgotten, with each scope that the old format could place left alone. + +## Chains in the old versions + +Recomputation, not the stored record, answers any question that needs a chain. A version stores its inputs (types, +a dependency's schemas and, since #262, its config), and the current core can compute the chains from those under +the current spec ([history](hashes-and-history.md#history)). So: + +- `diff` and the changelog **recompute chains for both versions** and compare those. A migrated version compared with an + unchanged successor shows no change, even though its stored chain is unknown. +- The chains stay unknown only where an input is missing: a dependency's config in a version older than #262, a type + whose information an older format never kept. Those are `unseen`, as everything else that can't be seen is + ([history](hashes-and-history.md#what-a-comparison-does)). +- "Narrowing a marker isn't a major change; widening one or adding a cause is" is a rule over *recomputed* causes for + these versions, so it needs no stored chain to hold. + +## Acknowledgements + +Three things are called an acknowledgement, and the migration treats them differently, because only one of them is in +the store: + +| | Where it is | At format 4 | +|---|---|---| +| **The author's marker** (`@unverified …`) | the source | Unchanged, and still valid: the plain form acknowledges what core computes at its scope. The frozen record of it is rebuilt by each build | +| **A consumer's acceptance** (`accept_unverified = { Billing = […] }`) | the consumer's `config.idp` | Pinned to chains in format 4. An existing entry is a set and has no pin, so it matches nothing, and the build **asks again**: it refuses, names each protocol, shows what is now recorded (the guarantee, the chain, the origin's reason), and prints the entry. `comline accept` is the existing way to write it | +| **A package's own `unverified = { … }`** | its `config.idp`, next to the setting | Pinned to the setting. The same re-asking applies, once, for a package that wrote it | + +This is the intended behaviour of [core#222](https://github.com/ComlineProject/core/issues/222): "a new cause, or a swapped +type upstream, asks again at every hop". A format change that makes every existing acceptance ask once is the same +event, because the pin it lacked is the pin the invariant needs. **Nothing is rewritten.** The CLI has a precedent +that is a good model: when bounds on types arrived, `build` listed every position, and the CLI *offered* package +defaults with maxima the author typed, never guessed +(`bounds_migration.rs`). Here the offer is each acceptance, with what it now covers, and the author decides. + +**Decided:** at the 3→4 transition an unpinned `accept_unverified` entry is **re-asked, not accepted**. The build stops and +names `comline accept`, which records the acceptance again with its chain. `config.idp` is never rewritten automatically. +This differs from old dependency pins, which [verify through `down` and are accepted with a notice](hashes-and-history.md#pins): +a pin that verifies through `down` proves the content is unchanged, but an unpinned acceptance proves nothing about which +causes it covered. + +## Consumers + +What reads a version with unknown chains, and what each does: + +| Consumer | With unknown chains | +|---|---| +| `build`, the guarantee check | recomputes from inputs; the unknown is never what it judges on | +| `diff`, `doc --version`, the changelog | recompute; list what is `unseen` | +| `inspect` | shows the recomputed chain, marked "recomputed, not recorded" | +| the generated `@unverified` docs ([generation#93](https://github.com/ComlineProject/generation/issues/93)) | the chain as recomputed. Where it can't be, it fails closed as below | +| a generator, or `doc`, given an old version | **fails closed**: it stops with "unknown (migrated from format N)" for what it would print from an unknown, naming the version and the path. A generator is never handed a placeholder that looks complete | + +## Fixtures and tests for 3→4 + +The corpus ([proof](proof.md#the-fixture-corpus)) needs, for this step: the chain graphs of the oracle, with the inner scopes +(`66feecb`) and the package-scope `unverified` (`9ea142e`), built by format 3's closing core. The square is then: the format-3 store of a graph, migrated, against the format-4 build of +the same graph with chains and package pins forgotten. The differential test draws random graphs and histories from +the oracle's generators, the same ones that test the invariant, so the property that no guarantee goes +unacknowledged is checked over migrated stores too (the cross-package chains part of stage 2 of the rollout). diff --git a/docs/docs/proposals/ir-migrations/formats.md b/docs/docs/proposals/ir-migrations/formats.md new file mode 100644 index 0000000..1db01d9 --- /dev/null +++ b/docs/docs/proposals/ir-migrations/formats.md @@ -0,0 +1,129 @@ +# The formats + +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) + +**The question.** What exactly is in a store, which of it depends on the IR format, and what does a format +number promise about it? + +## What a store holds + +| Path | Holds | Depends on the format | +|---|---|---| +| `objects//` | One object: LZ4 over CBOR, addressed by the blake3 of the CBOR. Three kinds: `Blob { content }`, `Tree { entries: [{ mode, name, hash }] }`, `Commit { tree, parents, author, timestamp, message, version, release? }`. | The *container* does not. See below. | +| a blob's `content` | One frozen schema unit (`FrozenUnit`), or a version's `config` (a list of config units), `docs`, `facts` (`History`), a dependency's `manifest`, `config` and `dependency_configs`. | **Yes.** This is what a format version describes. | +| `refs/heads/main` | The hash of the newest commit, hex. After a Git merge, conflict markers around two. | No | +| `FORMAT_VERSION` | The stamp. | It is the stamp. | +| `deps-cache/` | Git checkouts of dependencies. A cache, ignored by Git. | No, and not part of the store | + +A version's root tree has one `schema_` tree per schema file, a `dep_` tree per vendored dependency +(`schema_`, `manifest`, `config`, `dependency_configs`), and the blobs `config`, `docs` and `facts`. The +[CAS guide](../../guides/ir/cas.md) describes this for format 3. + +**Proposed: the container is not a format.** The object layout (CBOR, LZ4, blake3, `Blob`, `Tree`, `Commit`, the +ref file) is the same in every CBOR format so far, and a format version covers the payloads inside it. Three things follow: + +- A reader of the container needs no format knowledge. The CLI and core can list versions, read each commit's + `version` and `release`, and walk parents in any store they can find the container of. Today + `resolved_version` reads a dependency's committed `.comline/` this way and falls back to `0.0.0-unbuilt` on any + failure, which hides a store it merely couldn't read. +- A change to the container (the hash, the compression, the ref layout) is also a format change, and a step like + any other. Its model is the whole store. +- Every hash above a changed payload changes with it, because parents, tree entries and `release.since` are all + hashes. See [hashes](hashes-and-history.md#object-hashes). + +## History of the formats + +Background only: these formats will be reset at 1.0 ([scope](index.md#scope-the-system-not-the-history)), and are +listed because they are why a migration system is needed. + +| Format | Stamp | On `master` | What it is | +|---|---|---|---| +| **0** | none | 2026-01-10 to 2026-10-11 | bincode, which tags fields and variants by position. About 20 commits reshaped the persisted types, none marked. | +| **1** | `1` | 2026-10-11 (`f71ef96`) | Name-tagged CBOR. The stamp and `ensure_compatible_store` are introduced. About 15 hours on `master`. | +| **2** | `2` | 2026-10-12 (`dbdb13f`) | `Property.expression: Option` becomes `value: ConfigValue`: a field renamed and retyped. | +| **3** | `3` | 2026-10-18 (`eda0684`) | `KindValue` text becomes `TypeRef`, `Literal` and `Condition`. | + +Two of these bumps lost information that the next format needed: format 2's text froze `f32` and `f64` as `float` and +`T[8]` as `T[]`, and format 1's `expression` had no quotes. Nothing could have recovered them, which is the case the +unknown machinery in [steps](steps.md#unknown) exists for, and the one the test-only formats in +[the proof](proof.md#test-only-formats) imitate. + +## Format 3 is a family + +The doc comment on `CURRENT_IR_FORMAT_VERSION` says growing `FrozenUnit` needs no bump. Since 2026-10-18 these have +been added without one, and so a "format 3" store can lack any of them. The 3→4 step, the first client, takes format 3 +as it stands when the framework ships (the last row), and a store from before that is not migrated (the user can reset it, like any pre-1.0 store): + +| Added | Commit | Absent reads as | +|---|---|---| +| validator call `span`, `arg_spans` | `0983015`, 10-18 | unknown span, `(0, 0)` | +| type alias `parameters` | `8243d98`, 10-18 | none | +| `Assert.message_span`, `uninterpolated`; validator `validate_span` | `ad2b3f5`, 10-19 | none | +| `Unverified(Vec)` in a protocol's parameters; the `Guarantee` enum | `bd9d9f4`, 10-20 | no recorded set | +| `Commit.release` | `af0371f`, 10-20 | a release only if the version has no pre-release part | +| config units `Handshake`, `HandshakeParams` | `6571780`, 10-20 | the default handshake **of the reading core** | +| config unit `Wire` | `031d74c`, 10-25 | the default format and framing **of the reading core** | +| `ConfigValue::Dict` | `a5d6f85`, 10-25 | a variant older readers can't decode | +| config unit `Encoding` | `f643d51`, 10-31 | no declaration | +| root `docs` blob | `038591e`, 11-01 | no `//!` headers | +| root `facts` blob; config unit `PackageId` | `f873d98`, 11-01 | computed when read; no id | +| `Encoding.establishes`, `has_vectors` | `4ba7c57`, 11-01 | the strict reading: nothing established, no vectors | +| `PrimitiveType::Char`, `Array.max` | `4eb6228`, 11-02 | a variant older readers can't decode; no maximum | +| `Guarantee::BoundedSize` | `2b6d636`, 11-02 | a variant older readers can't decode | +| facts' `number`, `retired`, `numbered` | `8ec27df`, 11-03 | `numbered = false`: computed when read | +| `Encoding.grammar`, `calls`, `directions` | `4da05ab`, 11-07 | none | +| facts' per-protocol `encodings`; the defaults moved to `datagram@3` and `comline@2` | `bbbd957`, `9a6482e`, 11-08 | resolved when read, with what an absent value *was* (`IMPLICIT_FRAMING`, `IMPLICIT_HANDSHAKE`) | +| config unit `Unverified` | `9ea142e`, 11-10 | none | +| `dep_/config`, `dependency_configs` | `965e781`, 11-12 | **unseen**: not empty, not defaulted | +| `FrozenArgument.parameters` (markers on arguments) | `66feecb`, 11-12 | none | + +The rule, as written, is sound for the direction it was written for: a new core reads an old store. It has four +gaps, and a migration framework has to close them: + +1. **A store holds commits of several ledger levels.** Each build writes what its core knew. The absent-entry + reading is per commit and per entry, so "format 3" says nothing about which of the rows above a given version + has. +2. **The rule is one-directional.** A shipped binary that meets a new *variant* (`BoundedSize`, `Dict`, `Char`) fails + to decode. One that meets a new *field* drops it without a word. Neither is checked. +3. **Some additions change what old bytes mean.** An old version's `Unverified(Vec)` has no + `bounded_size`, because the guarantee didn't exist; under today's spec the same schema gives it up. An absent + `wire.framing` meant `datagram@2` when it was built and `datagram@3` now, so a reader has to say which. +4. **Nothing checks it.** `f71ef96` records the decision not to add a golden-hash check, "can revisit if it becomes a + real recurring problem". Twenty additions in 25 days is that. + +## Each format is written down + +**Proposed:** a format is not "whatever the types were at that commit". Each format N has a directory in a new +`formats` crate, and these four things in it: + +| File | Holds | +|---|---| +| `SCHEMA.md` | Every blob, tree and entry; every field, its type and meaning; what an absent entry or field means (the *absence reading*); the ledger. The source of truth for the format. | +| `types.rs` | Frozen copies of the format's types: serde derives and nothing else, with no behaviour, no dependency on `comline-core`, and every `#[serde(...)]` attribute that decides the bytes. | +| `codec.rs` | `decode` and `encode` between bytes and the format's model. The only code that touches format-N bytes. | +| `ledger.toml` | One row per addition made without a bump: the commit, the date, what was added, the absence reading, and the fixture that has it. | + +A format is **sealed** when the next one is introduced: `formats/MANIFEST` records the blake3 of each sealed +directory, and CI fails if one changes. The current format is not sealed, and it is held to the live types by a +test instead: `live ≡ frozen(current)`, meaning that for every fixture and for random values, the core's types and +the frozen copy encode to the same bytes and decode each other's. A change to the live types without the same +change in the frozen copy fails it, and so does a change that alters a fixture's bytes. + +This replaces the doc comment's rule with a checked one. Without a bump, the only changes allowed are: + +- a new entry in a root or dependency tree whose absence reads as unknown; +- a new field with `#[serde(default)]` and `skip_serializing_if` for the default, so that content which doesn't use + it keeps its bytes; +- a new enum variant, if nothing existing needs it. + +Each one adds a ledger row and the same lines to the frozen copy, in the same pull request. Any other change is a +bump. The `skip_serializing_if` condition is the one the code already follows ("so a schema without bounds freezes +to the same bytes as before them"), because the bytes of a frozen unit are also what a [dependency pin +hashes](hashes-and-history.md#pins). + +## Decided here + +- **No second number** (a ledger level) beside the format. It would only let a binary from before this proposal refuse to + write into a store whose additions it would drop, and those binaries are out of scope. +- **`SCHEMA.md` in core is the source for every format's document.** The [CAS guide](../../guides/ir/cas.md) describes only + the current format, says so, and links to the source for the others. diff --git a/docs/docs/proposals/ir-migrations/hashes-and-history.md b/docs/docs/proposals/ir-migrations/hashes-and-history.md new file mode 100644 index 0000000..4c6fc6e --- /dev/null +++ b/docs/docs/proposals/ir-migrations/hashes-and-history.md @@ -0,0 +1,141 @@ +# Hashes, pins and history + +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) + +**The question.** A migration rewrites what hashes cover. Which hashes may change, which must not, how does a +dependency pin keep meaning what it meant, and what do compatibility checks and the changelog say about a history +that was migrated? + +## The hashes that depend on the IR + +| Hash | Covers | Computed | Under a migration | +|---|---|---|---| +| **Object** (blob, tree, commit) | the object's serialized bytes | at write | changes exactly where it has to: [below](#object-hashes) | +| **Dependency pin**, `hash = "blake3:…"` in `config.idp` | scheme v2 ([core#261](https://github.com/ComlineProject/core/pull/261)): the CBOR of every schema's frozen units, and the dependency's frozen config units, sorted, minus `PublishRegistry` and `Dependency` | from a fresh compile of the dependency, checked against the pin | not in the store, **never rewritten**, [bridged](#pins) | +| **`manifest`** blob | the content hash verified when a dependency was vendored, as text | at build | carried verbatim. Nothing in core reads it today | +| **`schema_ir_hash`**, the handshake's `ir_hash` | the CBOR of one schema's units, folded to 64 bits | embedded in generated code | follows the units' bytes. Not stored; generated code isn't migrated | +| **`shape_hash`, `param_hash`** | a protocol's canonical wire shape; an encoding's `(kind, name, params)` | when facts are read | **must not change**: peers negotiate on them | + +The last is the point of the table: only the first three are *stored*, and only the first changes by construction. + +## Object hashes + +A hash covers bytes, and a commit holds its tree's hash and its parents'. So a changed payload changes its blob, every +tree on the way up, its commit, and every later commit, because each holds its parent's hash. `release.since` is a +hash too, and so are the second parents a reconcile leaves. + +- **What may change:** a payload whose encoding changed, and everything above it. For a format that touches only + `schema_` blobs, that is those blobs, the trees above them, and the whole commit chain. +- **What must not:** a payload whose bytes are the same in N and C keeps its hash, and is not rewritten. A step that + moves one field of one unit changes that unit's blob and no other. +- **Who maps them:** the engine, bottom-up, from the model, where references are [version ids](steps.md#the-model). + A step author never sees a hash. The old-to-new map is a bijection on commits, and is checked. +- **Outside the store:** a CI log, a note, or `comline doc`'s cache (`.comline-doc.json` is keyed by commit hash) may hold + an old one. `comline store migrate --map` writes the map, and caches should key on the IR format as well. + +## Pins + +A pin is hashed from **IR bytes**: `schema_units_cbor` is `to_cbor` of the units. So a format change that alters those +bytes for a dependency changes its pin, and the scheme tag (`comline-dep-content-v2`) doesn't know about the IR format. +Core#261 handled the case of a scheme change, and its message ("this pin was made by an older comline") is the +precedent. A *format* change is the same event with no message, and the earlier format changes very probably did it +(not checked) without anyone being told. + +Two things make that unacceptable for migrations. A migration that turns every pin in a repository into a mismatch is a +migration that breaks the build it was meant to save. And the [offline proposal](https://github.com/ComlineProject/docs/pull/163) +makes the hash the identity of a dependency: caches and signed bundles are keyed by it, so a hash that moves with core's +internal format would invalidate both. + +**Decided:** a pin made under format N, by scheme S, **still verifies under C when the dependency's content can be +expressed in N**, and a pin is never accepted otherwise. + +1. To check a pin that doesn't match, core takes the dependency's compiled model and applies `down` through each step + back to N. If every step's witness is defined on it, the content has nothing N couldn't record. It encodes the + result with N's codec, hashes it as scheme S did, and compares it with the pin. +2. **A match is exact or it isn't one.** Equality of the hash, of the content N would have recorded. Nothing is weaker: + the content has no information the pin didn't cover, so every guarantee the pin gave is still given. +3. **If the content uses something N lacks** (a dependency that now has chain markers, with format 4), the witness is + not defined, no older pin can match, and the error says what is new: "this pin was made under IR format 3; the + dependency now records chains that it couldn't have covered. Review it and re-pin: `hash = "…"`". The command that + writes a re-pin exists already as a request (`comline add --repin`, or the future `update`, core#261). +4. **`config.idp` is never rewritten** by a migration or by this check. +5. Cost is only on a mismatch, as `mismatch_message` is today, and only through the formats that still have steps. + +A package that doesn't use what format 4 added therefore keeps every pin through the bump, with no action, and the +tests show it ([proof](proof.md#laws)): `golden-pins.json` holds each fixture dependency's pin as the *historical* core +computed it, and the current core must verify it through the bridge. A package that does use it is asked, once, and +told why. + +| | A. Verify through the witness (**proposed**) | B. A pin form independent of the IR | C. Re-pin on every format change | +|---|---|---|---| +| An unchanged dependency, after a bump | keeps its pin | keeps its pin | **every pin breaks** | +| A new scheme | no | yes: a v3, so every pin changes once | no | +| A second definition of "content" to keep in step with the IR | no | yes | no | +| Proof | golden pins from the old cores | golden pins from the form | none needed, and none possible | +| Costs | `down` must be defined on a compile, not only on a stored version | the form must also carry new information the day the IR does | the offline proposal's bundles and caches | + +**Decided:** a pin that verifies through the witness is **accepted, with a one-line notice** that suggests a re-pin +("this pin was made under IR format N and still verifies; re-pin to move to the current one: `hash = "…"`"). It is never +silent and never a failure. Accepting is sound by the argument above: the content has nothing the pin didn't cover. The +re-pin is the user's, by hand or with `comline add --repin`; nothing rewrites `config.idp`. Option B stays available if +the bridge turns out too costly to keep through many formats. + +The pin gate in [the proof](proof.md#the-gate) makes the move visible: a change that moves the *fresh* pin of a fixture +dependency has to say so in the pull request. + +## History + +Compatibility, `diff` and the changelog compare versions, and a migrated history is not the same as a freshly built +one in what it records. The principle is: + +> **Compare from inputs; recompute outputs; keep what only history can say.** + +| What a version holds | Examples | In a migrated history | +|---|---|---| +| **Inputs** | the schema units, the vendored dependencies' schemas and configs, `config`, `docs` | carried. If a version never kept one, it is **unseen**, not empty | +| **Outputs of a rule** | a protocol's given-up guarantees and their causes, the encodings in play, a shape hash | recomputed when read, by the core that reads, under the current spec. Where stored (for pinning an acknowledgement, or to be quick), the stored value is a record and not an input to comparisons | +| **History itself** | `version`, `release` records, a protocol's `line`, a function's `since`, wire numbers | carried as data. They depend on the chain, not one version, and they are promises that peers may already hold | + +Version numbers in particular are **kept** even where today's rules would have numbered the history differently. A +migration is not a rebuild, and a version that was released as `0.4.0` was. + +### What a comparison does + +- **Both sides are recomputed under the same, current spec.** [Granular `@unverified`](../shared-vocabulary/granular-unverified.md#a-newer-spec-that-finds-an-existing-loss) + says `diff` recomputes both versions under the newer spec, so a loss the spec learned to see is in both and is no + change. A format-3 version whose protocol never had `bounded_size` in its frozen set (the guarantee didn't exist) is + not "narrower" than today's: it is recomputed, and gives it up like its successor does. +- **The spec and core that wrote a version** are, today, recorded nowhere: the decision in the vocabulary is that each + frozen version records the spec it was frozen against, and that isn't built. **Decided:** each stored version records + **the core version and the spec version that wrote it**, as root-tree entries (absent meaning unknown), in the first + release of the framework, so every version built after it has them. A version without them, a migrated one included, reads as `unknown`, is + recomputed under the current spec like any older one, and a ledger row can be told from the record instead of + guessed. +- **What can't be recomputed is `unseen`.** If a rule needs an input the version lacks (a dependency's config, a + type's width), the comparison lists it in the same `Comparison::unseen` that [core#262](https://github.com/ComlineProject/core/pull/262) + added: the version and the path, with the reason from the `unknown` entry. A rule that needs a fact it can't see reads + the strictest (the same way `Encodings` does when it isn't `complete`), and the bump is **at least a patch**: a + comparison that couldn't see part of what it compares doesn't say that nothing changed. It can be higher, since + "strictest" can be a breaking change: a type of unknown width against a known one is possibly breaking. +- **The changelog** shows the same: a change it found, and, as lines of their own, what it could not compare, in the words of + the reason. `comline doc --version V` for a version with unknowns **fails closed** on what it would print from an unknown, as `doc` does for + any consumer ([format 4](format-4.md#consumers)). + +| Case | Result | +|---|---| +| A package migrated from format 3, source unchanged | Nothing found; see [the first build](#the-first-build-after-a-migration) | +| The spec has since added a guarantee an old protocol gives up | Recomputed on both sides: no change, no false major | +| A version from before dependency configs were kept (core#262) | `unseen: Dependency("acme")`, at least a patch: as now | +| A version that went through a lossy step, with an array whose length was lost, compared to today's | the path listed as unseen; possibly breaking | +| A version whose `release` says `0.4.0` | `0.4.0`, whatever the current rules would number it | + +### The first build after a migration + +A migrated head lacks what the current core knows, so a fresh build of the unchanged working tree differs from it, +by exactly the unknowns. Without care, the first `comline build` makes a commit, with a bump of at least a patch, to +say that nothing changed. + +**Proposed:** the build's "same as before" test compares **modulo unknown**: if the working tree equals the head at +every field the head knows, nothing is committed and no version moves. What the build learned is not written back, +since history is append-only, and is not needed: it is recomputed when read. Any real change is compared with the +unknowns listed as unseen, as above. diff --git a/docs/docs/proposals/ir-migrations/index.md b/docs/docs/proposals/ir-migrations/index.md new file mode 100644 index 0000000..bd49b65 --- /dev/null +++ b/docs/docs/proposals/ir-migrations/index.md @@ -0,0 +1,162 @@ +# Migrating stored IR across format versions + +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · feeds +[core#146](https://github.com/ComlineProject/core/issues/146) (reading older IR after 1.0) and the +[granular `@unverified`](../shared-vocabulary/granular-unverified.md) work, which is IR format 4 · seven parts, +read in this order: [the formats](formats.md), [steps](steps.md), [running a migration](running.md), +[proving it](proof.md), [hashes, pins and history](hashes-and-history.md), +[beyond the local store](beyond-the-store.md), [format 4 as the first client](format-4.md) + +**The question.** `.comline/` holds every version a package has had, and compatibility checks, the changelog and +`comline doc --version` all read it. When core changes the shape of what it stores, `ensure_compatible_store` +deletes the directory, so the version history is lost with each bump. **Can the formats from 1.0 on be migrated, each +to the next, so that nobody can tell the result from a store the current core built, except where the old format never +recorded something, and so that this is shown by tests instead of asserted?** + +## Scope: the system, not the history + +**Decided (the user, 2026-11-13):** the formats that exist today will all be reset when 1.0 arrives, and the numbering +restarts at format 0. So this proposal prepares **the migration system and shows it is sound**: the model, frozen +per-format types, steps, the unknown and fail-closed machinery, atomic and idempotent running, the laws, the proof +harness and the CI gate. It does not promise to migrate today's formats (the unstamped bincode store, formats 1 and 2, +or the family that is format 3), and it is fine if one of them can't be migrated reliably. The 1.0 format is the first +with a guaranteed path forward. + +- **Before 1.0,** a store in an older format with no step to the current one is **never touched by a command on its own.** + The command stops and names `comline store reset`, which the user runs, and which moves the store to a backup and + deletes nothing. A store *newer* than the binary is refused. `deps-cache/` is kept. See + [running](running.md#what-each-command-does-with-each-stamp). +- **The system is proven before 1.0,** because 1.0 must not be its first test. The 3→4 step for the chains of + [core#222](https://github.com/ComlineProject/core/issues/222) is the first real client, and test-only formats prove + composition and lossy steps. See [format 4](format-4.md) and [proof](proof.md). +- **The 1.0 reset:** a warning release, then a 1.0 release that refuses pre-1.0 stores and names the reset command. See + [beyond the store](beyond-the-store.md#the-10-reset). + +## What it looks like today + +| Part | Today | +|---|---| +| **The stamp** | `.comline/FORMAT_VERSION` holds `CURRENT_IR_FORMAT_VERSION` (3). `ensure_compatible_store` compares with `!=`: anything else, a missing stamp included, deletes all of `.comline/`, `deps-cache/` too. | +| **Who deletes** | `comline build` does it with a warning in the log. The read commands (`diff`, `doc`, `generate`, `inspect`, `release`) refuse, and say that `build` "starts the version history over". An older binary deletes a store that is *newer* than it, because the check is `!=`. | +| **What a number means** | Four on-disk shapes have existed: an unstamped bincode store and formats 1, 2 and 3. Format 3 is not one shape: [twenty additions](formats.md#format-3-is-a-family) have landed in it without a bump, which is why the rule needs a check. | +| **What the rule is** | Bump only for a rename, a removal or a retype; anything else is "additive". The rule is in a doc comment. Nothing checks it: the commit that introduced the stamp deferred a golden-hash check "if it becomes a real recurring problem". | +| **Hashes that cover IR bytes** | Every object's address, a dependency pin (`hash = "blake3:…"`, scheme v2, [core#261](https://github.com/ComlineProject/core/pull/261)) and `schema_ir_hash`. A format change moves them. | +| **Proof** | Round-trip tests over real packages, an oracle property test and fuzz targets for the guarantee check, differential tests in the runtime. None says anything about an old store. | + +The next change, format 4 ([core#222](https://github.com/ComlineProject/core/issues/222)), would wipe every store. +This proposal comes first so that it doesn't, and so that by 1.0 the machinery has already carried a real change. + +## What this proposes + +1. **[Formats written down](formats.md).** Each format has a document (blobs, trees, entries, field meanings, what + an absent entry means), frozen copies of its types, and a ledger of every addition made without a bump. The + additive rule is tightened and checked. +2. **[Steps](steps.md).** A migration is a chain of single steps `N → N+1`. A step is a total, deterministic + function over frozen format-N types, with a written list of what N cannot know. Those things are written as an + explicit **unknown**, never as a default that looks complete. Every step also has an inverse witness, so no + step can drop what a version recorded. +3. **[Running a migration](running.md).** The new store is written beside the old one, verified, and swapped in; + the old one is kept. A migration can be checked without writing (`comline store migrate --check`), resumed, and + repeated. It runs only when the user runs it: every other command stops and names it. A store newer than the binary + is an error, never a deletion. +4. **[Proof](proof.md).** A corpus of real stores built by the real core of each format, an equivalence stated field by + field, differential property tests against the previous format's core, test-only formats for composition and lossy + steps, laws (totality, determinism, idempotence, composition, hash stability), and a CI gate that fails a format bump + without its step, fixtures and document. +5. **[Hashes, pins and history](hashes-and-history.md).** What may and may not change in a hash, how a dependency + pin survives a format bump without being weakened, and how compatibility and the changelog read a migrated + history. +6. **[Beyond the local store](beyond-the-store.md).** Every other place stored IR or its hashes appear, the 1.0 reset, + and the policy for old formats after 1.0. +7. **[Format 4 as the first client](format-4.md),** which proves the system end to end before 1.0. + +## Rules that hold whatever the details + +- **Nothing is deleted or changed without the user doing so.** No command migrates, resets, moves or deletes a store + as a side effect. A command that finds an older store stops, says what it found and names the command to run + (`comline store migrate`, with `--check` as the dry run, or `comline store reset`). On a terminal it may ask whether + to run that exact command and does nothing without a yes; in CI it fails with the message. Backups are removed only by + `comline store prune`. The one thing a command may do on its own is read an old store through an in-memory migration, + because that changes nothing on disk. +- **A store is never deleted to change its format.** Migration replaces deletion; where migration is impossible + the old store stays, and the error says what to do. +- **A step is a pure function of one format's data.** No clock, no environment, no network, no randomness. Two + people migrating the same store get the same bytes. +- **What a version recorded is kept.** What its format never recorded is unknown, and unknown fails closed + everywhere it is read: [#251](https://github.com/ComlineProject/core/issues/251)'s rule for a dependency's config, + and the fail-closed rule of the [shared vocabulary](../shared-vocabulary/granular-unverified.md#fail-closed). +- **Nothing is replaced until it is verified,** and what is replaced is kept until someone removes it. A store that + can't be migrated stays as it is until the user resets it into a backup. +- **Accurate means tested against a fresh build.** A migrated version equals what the current core would have built + from the same source, field for field, except for the fields the step declared unknown. +- **A format change brings its step, its fixtures and its document in the same pull request,** or CI fails. + +## What was decided + +Settled with the user, and written into the pages: + +| | Decision | +|---|---| +| Unit of migration | the whole store, version by version, all or nothing | +| Steps | `N → N+1`, typed on frozen copies; `up`, `down` and a list of unknowns. **Every step has the `down` witness**, checked in tests and at migration time on the user's own store | +| Unknown | typed `Known` where the new type has room, otherwise an `unknown` entry in the root tree that keeps the original value. Generators and `doc` **fail closed** on one: "unknown (migrated from format N)" | +| Automatic changes | **none.** Migration and reset run only by the user's command; other commands stop and name it (`comline store migrate`, `--check` as the dry run; `comline store reset`). CI fails with the message | +| Atomicity | write `.comline.migrating/`, verify, rename the old store to `.comline.format-N.bak/`, rename the new one in | +| Backup | `.comline.format-N.bak/`, kept until the user runs `comline store prune` | +| Reset | `comline store reset`, an explicit command that moves the store to a backup after the user confirms that exact action; `deps-cache/` is kept | +| Newer store | an error naming the format and the comline that reads it; never touched | +| Read commands | read an old store through an in-memory migration; they write nothing | +| Git merge of stores in two formats | refused: "migrate both sides first" | +| Pins | verified through the witness; an old pin that verifies is **accepted with a one-line notice** suggesting a re-pin, never silently and never a failure | +| History | comparisons recompute derived facts from stored inputs under the current spec; what can't be recomputed is `unseen`. Each stored version records the core and spec versions that wrote it | +| Frozen formats | the `comline-ir-formats` crate; the IR-only (`wasm`) build carries it only for in-memory, read-only migration | +| Gate | `formats/MANIFEST`, `live ≡ frozen(current)`, fixtures, and a bump check; see [the gate](proof.md#the-gate) | +| Today's formats | not migrated; the user resets them; at 1.0 the numbering restarts | +| 1.0 | a warning release, then a 1.0 release that refuses pre-1.0 stores, names the reset command, and stamps `1.0/0` | +| Bridge releases | how long one is supported is decided at 1.0 | +| First client | the 3→4 step of #222's chains, end to end, before 1.0 | +| Format documents | `SCHEMA.md` in core is the source for every format's document; the CAS guide describes only the current format | +| Unpinned acceptances at 3→4 | an unpinned `accept_unverified` entry is re-asked, not accepted: the build stops and names `comline accept`; `config.idp` is never rewritten automatically | + +Binaries from before this proposal delete a store whose stamp they don't expect. They are out of scope. + +## Rollout + +Staged pull requests: the framework, then format 4 as its first client, then the 1.0 refusal and reset. Each is a +checklist in [core#268](https://github.com/ComlineProject/core/issues/268). + +**1. The framework** (core, format stays 3): + +1. *A newer store is an error; an older store is an error naming `comline store migrate` or `comline store reset`; + nothing is changed automatically.* Core and CLI message. This removes the wipe and is released first. +2. *Format documents and the gate.* A `formats` crate with the frozen format-3 types, the ledger, the seal, the + `live ≡ frozen(3)` test, and golden hashes for the fixture packages. This stops further no-bump drift before it + grows. It also adds, as ledger rows, the entries nothing writes yet (`unknown`, and the core and spec versions that + wrote a version), with readers that fail closed, before any step or format writes them. +3. *The engine and `comline store`:* container reader, plan, journal, verification, swap, backup, `--check`, `reset`, + `prune`, and the prompt a command shows on a terminal. +4. *The proof harness,* proved on **test-only formats** before any real step exists: composition (N→N+2), a lossy step + with unknowns, the witness, crash injection, the laws. And the fixture generator. + +**2. Format 4, the first client** (core): the new frozen shape of +[core#222](https://github.com/ComlineProject/core/issues/222) item 3, step 3→4, its fixtures (stores built by format +3's closing core) and format document, the pin bridge, and acknowledgements re-asked, all in one pull request because +the gate requires it. It runs the square, the laws and the gate on a real change. Then cross-package chains: +recomputation over stored dependency IR, `unseen` extended to chains, and the oracle and fuzz target over migrated +stores. + +**3. The 1.0 refusal and reset** (core, CLI): a warning release, then the 1.0 release, which refuses a pre-1.0 store, +names `comline store reset`, and stamps `1.0/0`. From then on every format change is a step. See +[the 1.0 reset](beyond-the-store.md#the-10-reset). + +**4. Tooling** (CLI, language server, generators, docs): + +- `doc` caches key on the IR format. +- The language server and `inspect` show `unseen`. +- Generators and `doc` fail closed on unknowns. +- The guide on [the CAS](../../guides/ir/cas.md) loses its warning about deletion and describes the stamp, backups and + `comline store`. + +## Still open + +Nothing open. diff --git a/docs/docs/proposals/ir-migrations/proof.md b/docs/docs/proposals/ir-migrations/proof.md new file mode 100644 index 0000000..09a63fa --- /dev/null +++ b/docs/docs/proposals/ir-migrations/proof.md @@ -0,0 +1,207 @@ +# Proving a migration + +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) + +**The question.** "The migration is accurate" has to be a statement a test makes, for every old format and every +future one, and a format change has to be unable to land without the tests that make it. What are they? + +## What accurate means + +A migrated version has to be what the current core would have built from the same source, except where the old +format could not know something. As a square that must commute: + +```text + a package's source, at each point in its history + ┌────────────────────┴────────────────────┐ + build with the core of format N build with the current core (format C) + │ │ + a store of format N a store of format C + │ │ + migrate N → C forget N → C + │ │ + └──────────────── must be equal ──────────┘ +``` + +`forget` is not a second implementation of the step. It acts on a *current* value and does one thing: it replaces the +paths on the steps' `unknown` lists with an unknown, as `up` would have, and touches nothing else. It is derived from +the lists, so it is a few lines whatever the format. Three kinds of mistake each break the equality: + +| The step | The equality fails because | +|---|---| +| computes a wrong value for a field N could know | migrated has a value, forget leaves the fresh one, and they differ | +| invents a value for a field N could not know | migrated has a value, forget has an unknown | +| writes an unknown for a field N could know | migrated has an unknown, forget leaves the fresh value | + +## What is compared + +Equality is on **root trees**, not commits. A tree's hash covers everything in it, so equal tree hashes are equal +bytes, and the bytes are canonical (CBOR with ordered maps, the same value always encodes the same). It is compared +per part: + +| Part of a version | Compared | Why | +|---|---|---| +| `schema_` trees, `config`, `docs`, a dependency's `schema_`, `config` and `dependency_configs` | **exactly,** after `forget`, as bytes | what the format records | +| a path on a step's `unknown` list | the migrated placeholder and `unknown` entry against `forget`'s | N never recorded it | +| `facts` | **exactly,** except the differences in the fixture's `known-differences.toml` | a record of what the build computed under the rules then; a rule that changed since moves a recomputed fact, and the file names the commit that changed it | +| a commit's `version`, `release` and parent structure | exactly, through the old-to-new map | user data | +| a commit's `author`, `timestamp`, `message` and its **hash** | **not compared** | provenance the fresh build can't share: a different time and user, and the hash follows from them | +| a dependency's `manifest` | not compared with the fresh build; compared with the **old** store's | it records the content hash verified at build time, under the pin scheme then in force | +| `deps-cache/`, backups | not part of a store | | + +For the comparison to be fair, a fixture's history must be one on which today's rules number the versions as the +old core did. Where they don't, the fixture records why. That is the proof's one admitted gap, and it is a list a +reviewer reads. + +## The fixture corpus + +**The sources.** A fixed set of packages, committed once and never edited except to add: + +- `tour`, the schema in `core/tests/cas/ir_roundtrip_tests.rs` that uses every construct the grammar has; +- the packages in `core/tests/fixtures/packages`, and `std`; +- a package with `//!` headers; +- a package with a path dependency that declares encodings, a `wire`, `unverified` and `settings`, and one whose + dependency has a dependency of its own (so `dependency_configs` is there); +- packages with `@unverified` at each scope and with `handshake = "none"`; +- the chain graphs of the oracle ([core#244](https://github.com/ComlineProject/core/pull/244)): A's type reached by B, reached by C. + +Each is a **scripted history**: snapshots of the sources and the actions between them (`build`, `release`, +`build --reconcile`), with at least a breaking change, a release, a rebuild that changes nothing (so makes no +commit), and, for one package, two heads left by a Git merge. They live in `core/tests/migrations/corpus/`. + +**The stores.** For each format that has a step out of it, the real `.comline/` that the real core of that format built +from each script, committed under `core/tests/migrations/stores/format-//`, with a `PRODUCED_BY` file (the +revision, the digest of the lockfile, the toolchain, the command). A store is a few objects per version, so the corpus +stays small. A format that gains a [ledger row](formats.md#format-3-is-a-family) while it is current gets a second set +for the row, so a store of each level is covered. + +Before 1.0 the only format with a step out of it is 3, for the 3→4 client: its fixtures are built by the core at the +closing revision of format 3, the one the framework ships on. The CLI revision that pins it is the easy route, +because the CLI keeps a committed `Cargo.lock`. From 1.0 on, **every released format** gets its fixtures when it +is released, built by that release's core, and keeps them for ever ([after 1.0](beyond-the-store.md#after-10)). + +### Regenerating + +`cargo xtask ir-fixtures --format N` checks the recorded revision out into a worktree outside the repository, builds +it with its lockfile and `--locked`, and runs each script. Core's only non-determinism is the author (`USER`, or +`whoami`) and the timestamp (`SystemTime`), so the generator sets the first and runs under a fake clock +(`faketime`, on Linux). A store regenerated that way is **byte for byte** the committed one, and `--check` says so. +The committed bytes are the fixtures; regeneration is what shows they are what that core made. A weekly job runs +`--check` for every recorded format, so a yanked crate or a vanished revision shows within a week and not when a +fixture is needed. The committed stores don't depend on it. + +Producing a format's fixtures is therefore part of releasing a format, and the [gate](#the-gate) requires them. + +### Test-only formats + +Format 4 alone can't prove everything: it has one step, and its unknowns are typed. The harness also has formats that +exist only in the test tree, `T0 → T1 → T2`, with real frozen types, codecs and steps, never written by a release: + +| What | How it is built in | What it proves | +|---|---|---| +| a chain of two steps | `T0 → T1` and `T1 → T2`, run alone and composed | the **composition** law, N→N+2 against N→N+1→N+2 and against the square | +| a lossy step | `T1` retypes a field so that `T0`'s value can't be recovered, and the new type has no unknown variant | the `unknown` entry, the placeholder, the original kept, and `forget` | +| a typed unknown | `T2` adds a `Known` field | the typed carrier, and that every reader has to handle it | +| a step that is wrong on purpose | each of the three mistakes in [the square](#what-accurate-means), and a step that drops a field or uses a clock | the harness **fails**: the gate is tested by seeing it reject a bad step | +| crashes | the engine on `T0` stores, killed after every write and rename | resume and idempotence | + +They need no old cores: a `T0` store is built by the `T0` writer in the test tree. They are the way the system is proven +before there is a second real format. + +## Differential property tests + +Fixtures are fixed histories. The differential test draws random ones. + +- **The generator** is `core/oracle`, extended: `Spec` already renders a package graph to `config.idp` and `.ids` + files and has no dependency on core. Add **histories**: a sequence of edits (add a field, swap a type, add or + remove a dependency, change a handshake or a bound, release), with proptest strategies for the property tests + and `Arbitrary` for the fuzzer, as for the guarantee oracle. +- **The old side runs out of process.** The CLI (or maker) built at the recorded revision of the previous format is run over the + rendered package, and its `.comline/` is read back. The binary is built once per `(revision, lockfile, toolchain)` + and cached in CI. The only exchange is the file system. +- **The current side runs in the test:** the current core builds the same history, `migrate` runs on the old store, and + the two are compared as [above](#what-is-compared). +- **Budget:** 256 cases by default, like the oracle test, and a few per PR if that is too slow to run on each; more + nightly (`PROPTEST_CASES`). A failure shrinks to the shortest history that disagrees, and the test writes it as a new + corpus entry. + +### An old core as a renamed dependency + +The brief asks whether the old core can be a renamed dev-dependency, `comline-core-v3 = { package = "comline-core", +git = …, rev = … }`, so the test calls it in process. Cargo allows two sources of one package under different names. +**It is feasible, and I'd not use it**, for these reasons, none of which has been tried: + +- Both parsers are linked into one test binary. If a generated grammar exports C symbols that don't depend on the + revision, they collide at link time. Core's `Cargo.toml` mentions `tree-sitter-c2rust`, a pure-Rust variant, so this may not + occur, but it needs trying. +- Core has no `links` key, but its dependencies might, and the resolver unifies versions and features across both + copies. Old revisions have no lockfile, so they resolve against today's, which is the problem fixtures solve. +- Every old rev wants its own toolchain, and `HEAD` pins `1.99.1`. +- The harness is per revision: the build entry points changed between them (`process_changes`, `build_with`, the + `deps` feature). +- The test links one more copy of core per format, forever. + +Out of process costs a process spawn per version and removes all of it. If the in-process route does work for the +latest format, it can replace the binary for that one. + +### A fuzz target + +`store_migrate` in `core/fuzz`, in the nightly matrix beside `unverified_oracle`: arbitrary bytes go through each +format's codec and the chain. It must never panic; it may return an error only when the bytes don't decode; and when +they do, the result must encode, decode, and satisfy the witness. A second form draws values of the frozen types +with `Arbitrary`, which covers what no real build produces, and is how totality is tested over the whole format. + +## Laws + +| Law | Statement | Tested by | +|---|---|---| +| **Totality** | `up` returns a value for every decodable `Version_N`, and it encodes | the `Arbitrary` form of the fuzz target and a proptest over frozen N types, not only reachable ones; every fixture | +| **Determinism** | the same input gives the same bytes on any platform, thread count and version order | run twice and compare; migrate the fixtures on Linux, macOS and Windows and compare digests with committed golden ones; shuffle version order and threads | +| **Information preserved** | `down(up(v)) = v`: no step dropped anything | the witness, over fixtures and random values, and on every real store at [run time](running.md#verifying) | +| **Idempotence** | migrating a migrated store changes nothing; an interrupted migration, resumed, gives the same files | run twice; fault injection that crashes after each write and each rename, then resumes | +| **Composition** | `migrate(N→K) = migrate(M→K) ∘ migrate(N→M)`, and equals `forget` over a fresh build | for each `N < M < K`: migrate to M, write it, continue; compare with straight through, and with the square | +| **Hashes carried** | a payload with the same bytes keeps its hash; each protocol's `shape_hash` and `param_hash`, its wire numbers, and every `version` and `release` are the same after | compare through the old-to-new map | +| **Convergence** | two independent migrations of one store give the same files | different thread counts and order, compared byte for byte | +| **Pins** | a dependency's pin made by the old core still verifies through the witness, or the format change declares that it doesn't | golden pins per format, [below](hashes-and-history.md#pins) | + +## The gate + +A format change that has no step, no fixtures or no document can't merge. It is a set of checks, most of them +running on every pull request because the stores are committed: + +| Check | Fails when | +|---|---| +| **Seal** | a sealed format's directory differs from `formats/MANIFEST`, so a frozen type, a codec or a schema document changed after its format was superseded | +| **`live ≡ frozen(current)`** | the core's types and the frozen copy of the current format encode differently, or can't decode each other's, for a fixture or a random value | +| **Golden fresh build** | a fresh build of each fixture source has different tree hashes from `expected/`, unless the pull request adds a [ledger](formats.md#each-format-is-written-down) row (and the frozen copy) or bumps | +| **Bump check** (`xtask check-format-bump`) | `CURRENT_IR_FORMAT_VERSION` changed, and the pull request lacks any of: `formats/v/` with its four files; a step `C-1 → C` with `unknown`, `up` and `down`; fixtures for every ledger row of `C-1`; the previous format's seal in the manifest; a row in the CAS guide's stamp table | +| **The square, the laws** | any of the above tests fails. A step with no tests can't be there: the bump check requires the fixtures, and the square runs on them | +| **Pin gate** | a golden pin of a fixture dependency moved, and the pull request doesn't declare it ([hashes](hashes-and-history.md#pins)) | +| **Spec** | the vendored spec files differ from `tests/spec/REV`: the existing check, because a step's `forget` reads the catalog | + +What it catches: + +| The mistake | Caught by | +|---|---| +| A field renamed or retyped without a bump | the fixtures of the current format stop decoding; `live ≡ frozen` | +| A field added without `skip_serializing_if` | the golden fresh build moves for a package that doesn't use it, so a row (and a pin note) is required | +| A bump without a step, or a step without fixtures | the bump check | +| A wrong, invented or needlessly unknown value | the square | +| A field dropped | the witness | +| A `HashMap`, a clock or a thread in a step | determinism and convergence | +| An edit to a frozen format | the seal | + +The seal and the bump check should be under CODEOWNERS, and a seal can be moved only by a command that names why, so that +moving one is a deliberate edit and not a side effect. Weekly: the fixtures `--check` against their recorded revisions. Nightly: +the differential test at a larger budget and the fuzz target. + +## What this proves, and what it doesn't + +It proves that on the corpus and on generated histories, each step does exactly what its `unknown` list says: it +derives what N knew, and nothing else, and it loses nothing. It does not prove it for a store the corpus doesn't +resemble. + +The trust base is small and named: the frozen types (each fixture is decoded and re-encoded to the same bytes, which +checks the copies against what real cores wrote), the `unknown` lists a reviewer reads, the old cores built from +their revisions, and the generator's reach. The ledger's granularity is the corpus's: a store written between two +rows by an unreleased build is in no fixture, and a row is added when one is found. `comline store verify` runs +the structural checks on the store of anyone who runs it, which is how a store the corpus didn't predict gets found. diff --git a/docs/docs/proposals/ir-migrations/running.md b/docs/docs/proposals/ir-migrations/running.md new file mode 100644 index 0000000..654ac6e --- /dev/null +++ b/docs/docs/proposals/ir-migrations/running.md @@ -0,0 +1,169 @@ +# Running a migration + +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) + +**The question.** How does a store change format without any moment at which history is lost, however the process +ends, and how does someone find out what it would do first? + +## The rule + +**Nothing is deleted or changed without the user doing so.** No command moves, rewrites, resets or deletes a store as a +side effect. A command that finds a store it can't use as it is *stops*, says what it found and names the command that +fixes it, and does nothing until the user runs that command. The only exception is reading: a read command may read an +old store through an in-memory migration, because that changes nothing on disk. + +## What each command does with each stamp + +`ensure_compatible_store` stops deleting. With the stamp *N* and a binary that writes *C*: + +| The store | A read command (`diff`, `doc`, `generate`, `inspect`) | A write command (`build`, `release`, …) | +|---|---|---| +| N = C | proceeds | proceeds | +| N < C, and a chain of steps reaches C | reads through an **in-memory** migration and writes nothing | **stops**: "the store is format N; run `comline store migrate` (`--check` shows what it would do)". On a terminal it asks whether to run exactly that, and does nothing unless the user says yes; without a terminal (CI) it fails with that message | +| N < C, and no chain does (pre-1.0 only) | an error naming the format and the [options](steps.md#what-cannot-be-migrated) | **stops**, naming `comline store reset`. It never offers to reset inline | +| unstamped, or damaged | an error naming the format or the file | the same error; nothing is changed | +| **N > C** | an error: the store is format N, this comline reads up to C, and *comline ≥ X* reads N | the same error; nothing written | +| stamps or refs in conflict because Git merged stores in different formats | an error ([see Git](#git)) | the same | + +Reading an old store needs no write because a migration is a pure function: a read command can run the chain on the +versions it needs and drop the result. That makes `comline diff`, `doc --version` and `generate` work on a store nobody +has migrated yet, at the cost of the memory and time of the chain, which a cache can hide. + +Today's behaviors of deleting an older store and of deleting a newer one are both gone. Binaries from before this +proposal still delete a store whose stamp they don't expect; they are out of scope. + +## The procedure + +`comline store migrate`, which is also what `build` runs if the user confirms it: + +1. **Claim the store.** `mkdir .comline.migrating/` beside `.comline/`. It fails if the directory exists, which is the + lock. If it exists from a crash, see [below](#crashes). The migrating directory is on the same filesystem as + `.comline/` so that the renames in step 5 are atomic. +2. **Plan, in memory.** Read the container: the refs, then every commit reachable from them by parents, second parents + included, each object checked against its hash. Decode each version under N, run the chain, and encode the result + to bytes, writing nothing. The plan lists the versions, the steps, the objects that change, and every unknown with its + path and reason. `--check` stops here, after the [checks](#verifying) that don't need the disk. +3. **Write the new store** into `.comline.migrating/`: `objects/` (payloads that didn't change are linked or copied under + the same hash; everything else is new), `refs/heads/main` with each head mapped, and `FORMAT_VERSION` **last**. A + `JOURNAL` there records the old head hashes, the stamp and assumed format, and the state: `writing`, then `ready`. + `deps-cache/` is not read or written: it isn't part of the store. +4. **Verify** the new directory as if it were a store, [below](#verifying). A failure keeps the directory, marks it + `FAILED` with the reason, leaves `.comline/` as it was, and exits nonzero. +5. **Swap.** Set the journal `ready`. Rename `.comline/` to `.comline.format-N.bak/`. Rename `.comline.migrating/` to + `.comline/`. Move `deps-cache/` from the backup into the new `.comline/`: offline machines depend on it, and today's + wipe deletes it with everything else. Write the receipt and remove the journal. + +Each rename is atomic. Between the two there is no `.comline/`, and that state is recognized and completed, not +mistaken for "never built". + +### Crashes + +The state is in the paths and the journal alone, so every step is resumable. Another command that finds an interrupted +migration stops and names `comline store migrate`, which resumes it; nothing resumes on its own. + +| Interrupted | What is on disk | `comline store migrate` | +|---|---|---| +| steps 1 to 3 | `.comline/` intact, `.comline.migrating/` partial, journal `writing` | If the journal's old heads equal the store's, it resumes: writes are content-addressed, so repeating one writes the same bytes. Otherwise the store changed meanwhile: it stops and names the partial directory, which the user removes, or has `--restart` remove. | +| after verify, before the first rename | journal `ready` | completes step 5 | +| between the two renames | no `.comline/`, a backup and a ready directory | renames the ready directory into place | +| after both, before `deps-cache/` moved | new store in place, the cache in the backup | moves it | +| after | complete | nothing to do | + +Running the whole command again on a migrated store is a no-op: the stamp is C. That is the idempotence of the +operation. The idempotence of the *function* is a [law](proof.md#laws). + +### Verifying + +Before the swap, on the new directory, every one of these is checked, and any failure stops the migration: + +1. Every object decompresses, matches its hash, and every reference (tree entry, parent, `release.since`, ref) + resolves. +2. Every version decodes under C, and encoding it again gives the same bytes: the encoding is canonical. +3. The old-to-new map is a bijection on commits, parents map to parents, and for each commit the `version`, `release` + record, `author`, `timestamp` and `message` are the same as before. +4. The witness holds: `down` of every migrated version, through every step, equals the version as it was. + Nothing a version recorded was lost. See [the witness](steps.md#the-witness). +5. The unknowns written are exactly the ones the chain declares. A step that wrote an undeclared one, or didn't write a + declared one, fails here. +6. A payload whose bytes are the same in N and C kept its hash. + +Reported, not failed: the `facts` of each version recomputed by the current core and compared with the carried +ones. A difference means a rule changed since the version was built, and is listed so that it isn't a surprise +later. The existing `compat` tests treat a recomputed history the same way. + +## The backup + +**Decided:** the old store is renamed to `.comline.format-N.bak/` beside the new one, and **kept until the user removes +it** with `comline store prune`, which removes a backup only when the store that replaced it passes `comline store +verify`. Nothing deletes one on its own, after a time or a number of builds. `comline store status` lists them. + +It is a **sibling** and not a directory inside `.comline/`: `.comline/` is committed to Git, while a sibling shows in +`git status` as untracked, so it is visible and nobody commits it without choosing to. `comline new` can add it to the +`.gitignore` it already writes for `deps-cache/`; a migration never edits anyone's files. Objects the migration didn't +change are hard links, so a backup costs little. + +## Reset + +`comline store reset` is the only way a store with no path to the current format is replaced, and it is the user's +command: + +- It renames `.comline/` to `.comline.format-N.bak/` and starts a new store at the next build. It deletes nothing, and + `deps-cache/` is kept. +- On a terminal the user is asked to confirm **that exact action** (it names the store, the backup and the format), and + nothing proceeds without it. Without a terminal it needs `--force`. +- No other command resets. `build` on such a store stops and names this one. + +(`comline reset`, which deletes `.comline/` outright behind a typed confirmation, is today's command and unchanged.) + +## Checking first + +```text +comline store migrate --check +``` + +Does steps 1 to 2 and the in-memory part of the checks, and writes nothing: no `.comline.migrating/`, no lock. It prints +what would happen and exits with distinct codes for *nothing to do*, *would migrate* and *would be refused*, so it runs in CI +as a guard ("the checked-in store is current") and by hand before an upgrade. + +```text +store format 3, 41 versions (0.1.0-dev.1 … 0.9.0) + would migrate 3 -> 4: 41 versions, 118 objects rewritten, 1,204 unchanged + unknown after migration: 52 (chains: 52) + 0.7.0 protocols/Billing/given_up/bounded_size/chain format 3 records the set, not the path to its cause + … + nothing deleted: the old store is kept as .comline.format-3.bak +``` + +(An illustration of the content, not a format.) + +## The CLI + +Only the touchpoints are named here. + +| Command | Does | +|---|---| +| `comline store status` | The stamp, the versions, whether a migration is needed, the unknowns per version, the backups there are. | +| `comline store migrate [--check] [--assume-format N] [--map FILE]` | The procedure above. `--map` writes the old-to-new hashes, for anything outside the store that recorded one. | +| `comline store verify` | The [checks](#verifying) on the current store. Also what `core#182` asks for: a damaged store says which file, and what to do. | +| `comline store prune` | Removes verified backups, when the user runs it. | +| `comline store reset` | [Moves a store with no path aside](#reset), on the user's confirmation. | +| `comline store replay` | *Proposed:* [rebuild from Git](steps.md#what-cannot-be-migrated). | +| `comline build`, `comline release` | **Stop** when the store is older, and name `comline store migrate` or `reset`; on a terminal they ask whether to run the migration, and do nothing without a yes. They never change a store on their own. | + +There is no flag or variable that makes a command migrate or reset by itself: a machine that must migrate runs +`comline store migrate`. + +## Git + +`.comline/` is committed, so a migration is a change in a repository, and two people can make it. Migrating is +deterministic: a step is a pure function and the engine hashes bottom-up, so migrating the same old commit gives the +same new hashes in every clone. Two developers who migrate separately produce the same files, and Git merges them +without a conflict. This is why the receipt (`.comline/MIGRATIONS`, one line per migration) holds no time, no author and +no core version: the same on every machine, so it never conflicts either. New hashes are new files, so the only files +two sides can conflict on are `FORMAT_VERSION` and `refs/heads/main`. + +**Decided:** a Git merge of stores in two formats is **refused.** A command that finds a conflicted stamp whose sides +are different formats stops with "migrate both sides first": check out each branch, run `comline store migrate`, then +merge. Because migration converges, the merge is then the ordinary one, and the ordinary +[reconcile](../../guides/ir/cas.md#branches) joins the heads, as it does today. Nothing is resolved on the user's +behalf. diff --git a/docs/docs/proposals/ir-migrations/steps.md b/docs/docs/proposals/ir-migrations/steps.md new file mode 100644 index 0000000..6ca491d --- /dev/null +++ b/docs/docs/proposals/ir-migrations/steps.md @@ -0,0 +1,167 @@ +# Steps + +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) + +**The question.** What is a migration made of, so that it is accurate for every old format, never loses what a +version recorded, and never claims to know what the old format did not? + +## The model + +A migration moves a whole store from format N to the current format C, one **step** at a time. Steps never touch +bytes or hashes. They work on a **model** of one version, which the format's codec decodes from the store and +encodes back: + +```text +bytes ──decode_N──▶ Version_N ──step N→N+1──▶ Version_N+1 ──…──▶ Version_C ──encode_C──▶ bytes + (frozen types of N) (frozen types of C) +``` + +A `Version_N` holds the commit's own fields (`version`, `release`, `author`, `timestamp`, `message`), the decoded +payloads (schemas, vendored dependencies, `config`, `docs`, `facts`) and the `unknown` list described +[below](#unknown). A reference to another version, a parent or a `release.since`, is a **version id** (its place in +the chain), not a hash. The engine turns ids back into hashes when it encodes, so a step author never handles a +hash and can't break the Merkle structure. See [hashes](hashes-and-history.md#object-hashes). + +The intermediate formats never reach the disk. A store goes from N to C in memory, version by version, and is written +once. + +## A step + +A step `N → N+1` is three things, written against the frozen types of N and N+1 +([formats](formats.md#each-format-is-written-down)), never against today's types: + +| Part | What it is | +|---|---| +| `up` | A function from a `Version_N` to a `Version_N+1`. **Total:** it returns a value for every value the format's types can express, not only for what a real build produces. **Deterministic:** the same input, the same output, on every platform. It has no I/O, clock, environment, randomness, thread or hash-map order. Every map in the model is ordered. | +| `down` | The **witness**: a function from the image of `up` back to `Version_N`, with `down(up(v)) = v` exactly. See [the witness](#the-witness). | +| `unknown` | The list of paths in `Version_N+1` that `up` writes as unknown because N never recorded them, each with the reason. Written down in the format's `SCHEMA.md` and read by the tests. | + +A step may read only the version it is given and, for facts that live in history, the versions before it in the same +chain. It may not recompute what the *current* core would compute (a guarantee set, a wire number, a shape hash): that +is code that changes, and a step is frozen. What a version recorded is carried across as data, and what has to be +recomputed is recomputed on read, by whichever core is reading. For example the `facts` blob is a record of what +the build computed under the rules in force; a step carries it, and `history_of_commit` already recomputes it when it +is absent or unnumbered. + +!!! note "Steps may not delete user data" + `up` keeps every field of N. A field that N+1 renamed or retyped is mapped. A field that N+1 has no place for is + kept in the `unknown` list with its original value (see below). The witness checks this. + +## Unknown + +Where N+1 needs something N never recorded, `up` writes an **unknown**. It never writes a default that looks +complete: a framing the author never chose, a length `T[]` for an array whose length was lost, an empty set of +causes for a guarantee whose causes weren't stored. This is the rule +[core#251](https://github.com/ComlineProject/core/issues/251) set for dependency configs ("read as unknown, never as +defaults that look complete") and the [vocabulary's](../shared-vocabulary/granular-unverified.md#fail-closed) for a +dependency that can't be seen, applied to a whole version. + +An unknown has two possible carriers. **Proposed: both**, in this order of preference. + +1. **Typed, in the new format.** Where N+1's own type has room for it, the field is `Known`: a value, or + `Unknown { reason }`. The compiler then forces every reader to handle it. Format 4's chains are the example + ([format 4](format-4.md#the-step)). A format that adds a field is written so that anything a later step might have + to leave unknown is already a `Known`. +2. **An `unknown` entry in the root tree,** for a retype whose new type has no room: a blob listing, for each + unknown, its `path` into the model, the `step` that wrote it, the `reason`, and the `original`, the bytes N + recorded, where N recorded something. The field itself holds a placeholder that the entry marks as one. This is + what lets a step exist where the new type has no "unknown" variant and can't gain one without a bump: the entry + keeps the old value (say, a `float` that N froze without a width) beside a placeholder. Format 3's `TypeRef` is an + example of a type with no room, and a test-only format has one on purpose + ([proof](proof.md#test-only-formats)). + +`unknown` is an additive entry, a row in format 3's ledger with the absence reading *none*. A version without it +has no unknowns. + +**Ordering constraint.** A reader that doesn't know the entry would read a store that carries unknowns as complete. So +the readers of `unknown` ship before any step that writes one. See [the rollout](index.md#rollout). + +### Reading an unknown + +Consumers don't see the entry or the placeholder. The accessors on a version return `Known`, with no `Deref` and +no default, so an unknown can't be forgotten. Every use then has a rule, and it is the strictest reading: + +| A consumer needs | The field is unknown | Result | +|---|---|---| +| a protocol's given-up guarantees | the set or a cause | counts as given up; the build that depends on it refuses | +| a protocol's framing or format | the declaration | the strictest facts, as `Encodings` does when `complete` is false (core#262) | +| a bound | the maximum | unbounded | +| a type, compared | its width or length | breaking, or "can't tell" if the rule has that | +| a span | where it was written | no span; diagnostics name the version instead | +| generating code or documentation | anything the output prints | **fails closed**: the generator or `doc` stops with "unknown (migrated from format N)", [format 4](format-4.md#consumers) | + +A comparison that read an unknown says so: it lists the version and the path in the same `unseen` list core#262 +added, and its bump is at least a patch. See [history](hashes-and-history.md#history). + +Some things are never unknown, because without them a version isn't one: its `version`, its `release` record, its +parents, and the schema units' names. A step that can't produce them fails, and the migration refuses. + +## The witness + +`down(up(v)) = v` is the proof, on every real version, that a step lost nothing. **Decided: every step has one.** It is +cheap to write for the steps that are likely: one that wraps a value, renames a field or adds a field with an unknown +has a trivial inverse, and a step that moves a value into the `unknown` list keeps the original there so `down` can put +it back. A step that can't state its inverse is exactly the one to look at. + +It is checked in two places. The tests run it over fixtures and random values ([proof](proof.md#laws)). And the engine +runs it over every version of **the user's own store** that it migrates, before it swaps anything +([running](running.md#verifying)), so a store the corpus didn't predict is still checked. The witness also has a second +use, for [pins](hashes-and-history.md#pins). + +## Chains + +`migrate(N → C) = up(C-1→C) ∘ … ∘ up(N→N+1)`. A later step may resolve an unknown an earlier one wrote, only from +data in the version. The unknowns of a chain are the union of its steps' lists minus what a later step resolved, +and the engine writes that list. + +The engine has one chain per store and one version at a time. It does not support skipping steps: a "shortcut" from +1 to 3 is a second implementation of the same function, and the composition law ([proof](proof.md#laws)) exists to +catch the two disagreeing. + +## What cannot be migrated + +Today's formats are out of scope ([index](index.md#scope-the-system-not-the-history)): they reset at 1.0. Until then, +a store the system can't migrate is never touched by a command on its own. The command that finds it stops and says why; +the user decides, and the only thing that moves it is the user's `comline store reset`, which renames it to a backup. + +| Store | Why | What a command does | +|---|---|---| +| A store in a pre-1.0 format with no step to the current one (the unstamped bincode store, formats 1 and 2, older rows of format 3) | Nobody writes those steps. Format 0 is positional: a decode of the wrong layout can succeed and give a different variant, and nothing in the store can check a decode against. | Stops, naming `comline store reset` ([running](running.md#reset)): the store is renamed to `.comline.format-N.bak/`, a new one starts at the next build, `deps-cache/` is kept | +| An unstamped store that decodes as CBOR | The stamp was lost, so the format isn't known. | Stops, unless the user gives one: `comline store migrate --assume-format N`. The decode must succeed for every object | +| A damaged store (a missing or corrupt object, [#182](https://github.com/ComlineProject/core/issues/182)) | A migration needs every object. | Stops, naming the file and how it failed | +| A store newer than the binary | The binary can't read it. | Stops, naming the format and the comline that reads it | + +A history can later be rebuilt from Git with `comline store replay` *(proposed; may not be worth building)*. It builds a +**new** history with the current core, one build per release tag or commit the user chooses, never claims to be the old +one, and numbers versions by the current rules. + +## Support + +| Format | Status | +|---|---| +| 1.0 and later | every format migrates to the next: that is the guarantee this system exists to give | +| 3 (as it stands when the framework ships) | 3→4, the first client: it carries real stores, and exists to prove the system | +| 0 to 2, and older rows of 3 | not migrated; the user resets them (`comline store reset`), and the 1.0 release refuses them | +| test-only formats | in the test tree, never written by a release: [the proof](proof.md#test-only-formats) | + +## Where the code lives + +**Decided:** a workspace member `comline-ir-formats` (`publish = false` at first) beside `oracle/`, holding one module +per format with its `SCHEMA.md`, types, codec and ledger, and the steps between them. It has no dependency on +`comline-core`, the way the oracle has none, so that a frozen type can't reach for today's. A step `N → N+1` lives +beside the frozen N+1 types. Those exist from the day N+1 is introduced, and are kept equal to the live types by the +`live ≡ frozen(current)` test until N+2 seals them. + +Core depends on the crate to read old stores. Test-only formats live in its test tree, not in the shipped modules. The +engine and the CLI commands are in core. The crate's only dependencies are `serde`, `ciborium`, `blake3` and +`lz4_flex`, so the IR-only build, which the `wasm32` consumers use, can carry it: only for in-memory, read-only +migration, never to write a store. + +## Decided here + +- **New formats use `Known`** for everything they add that a later step might have to leave unknown. The `unknown` + entry is for retypes that have no room. +- **The `unknown` entry's paths are JSON pointers** into the model: the frozen types already derive serde, and a path then + reads the same in the format's document and in a test. +- **The `formats` crate also carries the unknown readers and the steps for in-memory, read-only migration in the + IR-only build (`wasm`),** and never the engine that writes. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 18fe4b0..4bf409a 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -229,6 +229,15 @@ nav: - 'proposals/shared-vocabulary/index.md' - 'The catalog': 'proposals/shared-vocabulary/catalog.md' - 'Granular @unverified': 'proposals/shared-vocabulary/granular-unverified.md' + - 'Migrating stored IR': + - 'proposals/ir-migrations/index.md' + - 'The formats': 'proposals/ir-migrations/formats.md' + - 'Steps': 'proposals/ir-migrations/steps.md' + - 'Running a migration': 'proposals/ir-migrations/running.md' + - 'Proving it': 'proposals/ir-migrations/proof.md' + - 'Hashes, pins and history': 'proposals/ir-migrations/hashes-and-history.md' + - 'Beyond the local store': 'proposals/ir-migrations/beyond-the-store.md' + - 'Format 4': 'proposals/ir-migrations/format-4.md' extra_css: - stylesheets/nav.css?v=23 # bump on every nav.css change — it is not cache-busted otherwise