From cfaf603d5fadbed5a55ef09389863e4cfca678df Mon Sep 17 00:00:00 2001 From: Kinflou Date: Fri, 13 Nov 2026 03:42:04 +0800 Subject: [PATCH 1/4] proposals: migrating stored IR across format versions (draft) Seven pages under proposals/ir-migrations: the formats and the additive ledger, migration steps with explicit unknowns, running a migration safely, the proof plan and CI gate, hashes, pins and history, artifacts beyond the local store, and format 4 as the first client. Tracked in core#268. --- docs/docs/proposals/index.md | 18 ++ .../ir-migrations/beyond-the-store.md | 78 +++++++ docs/docs/proposals/ir-migrations/format-4.md | 100 +++++++++ docs/docs/proposals/ir-migrations/formats.md | 135 +++++++++++ .../ir-migrations/hashes-and-history.md | 140 ++++++++++++ docs/docs/proposals/ir-migrations/index.md | 140 ++++++++++++ docs/docs/proposals/ir-migrations/proof.md | 209 ++++++++++++++++++ docs/docs/proposals/ir-migrations/running.md | 171 ++++++++++++++ docs/docs/proposals/ir-migrations/steps.md | 171 ++++++++++++++ docs/mkdocs.yml | 9 + 10 files changed, 1171 insertions(+) create mode 100644 docs/docs/proposals/ir-migrations/beyond-the-store.md create mode 100644 docs/docs/proposals/ir-migrations/format-4.md create mode 100644 docs/docs/proposals/ir-migrations/formats.md create mode 100644 docs/docs/proposals/ir-migrations/hashes-and-history.md create mode 100644 docs/docs/proposals/ir-migrations/index.md create mode 100644 docs/docs/proposals/ir-migrations/proof.md create mode 100644 docs/docs/proposals/ir-migrations/running.md create mode 100644 docs/docs/proposals/ir-migrations/steps.md diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index f996eb7..498b373 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -137,6 +137,24 @@ read in order: scope that guarantee has, checked against what core computes. Revisits part of the guarantee-marker decision. *accepted 2026-11-12; not built yet.* +#### [Migrating stored IR across format versions](ir-migrations/index.md) + +How the version history in `.comline/` survives a change of the IR format +instead of being deleted: each format written down, a migration as a chain of +total, deterministic steps over frozen copies of the formats that write what an +old format never recorded as an explicit unknown, a new store verified beside +the old one before it replaces it, and a proof (a corpus of real stores built by +the historical cores, an equivalence against a fresh build, differential +property tests and a CI gate) that fails a format bump without its step. +Seven parts, read in order: [the formats](ir-migrations/formats.md), +[steps](ir-migrations/steps.md), +[running a migration](ir-migrations/running.md), +[proving it](ir-migrations/proof.md), +[hashes, pins and history](ir-migrations/hashes-and-history.md), +[beyond the local store](ir-migrations/beyond-the-store.md) and +[format 4 as the first client](ir-migrations/format-4.md). *draft; tracked in +[core#268](https://github.com/ComlineProject/core/issues/268).* + ### Connections - [Connections](connections.md): what a connection carries beyond one client diff --git a/docs/docs/proposals/ir-migrations/beyond-the-store.md b/docs/docs/proposals/ir-migrations/beyond-the-store.md new file mode 100644 index 0000000..5c8fb2b --- /dev/null +++ b/docs/docs/proposals/ir-migrations/beyond-the-store.md @@ -0,0 +1,78 @@ +# Beyond the local store + +Status: **draft** · part of [Migrating stored IR across format versions](index.md) + +**The question.** The local `.comline/` is not the only place frozen IR, or a hash of it, is kept or relied on. What +does each of the others need, and what does a published package need when there are some? + +## Every artifact + +| Artifact | Holds | Migration story | +|---|---|---| +| **`.comline/` store** (`objects/`, `refs/`, `FORMAT_VERSION`) | every version, as frozen IR | [Migrated](running.md), whole, with the old one kept | +| **`.comline/deps-cache/`** | git checkouts of dependencies: source, not IR | Not migrated, not part of the store. **Carried across the swap**: today's wipe deletes it, and an offline machine can't fetch it again | +| **A dependency's own committed `.comline/`** | its version history; the only thing core reads of it is the newest commit's `version` (`resolved_version`) | Read with the format-independent [container reader](formats.md#what-a-store-holds). Today any failure reads as `0.0.0-unbuilt`; it should say the store couldn't be read | +| **Dependency pins** in `config.idp` | scheme v2 hash over IR bytes | Never rewritten. [Bridged](hashes-and-history.md#pins) | +| **Acknowledgements** in `config.idp`: a consumer's `accept_unverified`, a package's `unverified = { … }` | the set of guarantees accepted, per protocol | Never rewritten. [Re-asked](format-4.md#acknowledgements) when they are pinned to something the old format didn't record | +| **Generated code** | `schema_ir_hash` as the handshake's `ir_hash` | Derived, not migrated: regenerate. The hash follows the units' bytes, so a format change that alters them is visible at a connection. The pull request says so, and the [gate](proof.md#the-gate) lists it | +| **The generator protocol** | each request carries `ir_format` and the frozen units of the schemas | The units a generator gets from an old version are the migrated ones, in the current format. What to do with unknowns: [format 4](format-4.md#consumers) | +| **`comline doc` output and cache** (`.comline-doc.json`) | pages rendered per version, keyed by commit hash and the renderer | Derived. A migration changes commit hashes, so it re-renders once; the key should also hold the IR format | +| **The doc model** (`schemas/doc-model.json`) | a JSON schema, checked by a test | Derived, versioned on its own | +| **The language server, the playground, the editor** | no store; the server resolves dependencies without fetching and excludes `.comline/` from its watching | Nothing to migrate. Each ships a core, and that core's format is what it compiles against | +| **Published packages, the registry** | none yet: [no `publish` step exists](../../guides/ir/cas.md#what-to-commit), the registry server is on hold and core rejects registry dependencies | [Below](#published-packages-and-the-registry) | + +There is no `.cmln` archive or equivalent in any repository. Core has a `PublishRegistry` config unit (where a package +publishes) and nothing that builds an archive. + +## Published packages and the registry + +When there is something to publish, the [offline proposal](https://github.com/ComlineProject/docs/pull/163) already +decides that **identity is the hash and location is replaceable**: a dependency is its pin, and caches and bundles are +keyed by it. The rules for IR follow from that. + +- **Migrate on read, never rewrite what is published.** A published archive is addressed by its hash, and perhaps + signed. Rewriting it is making another artifact. A consumer that meets an archive whose store is in an older + format reads it through an in-memory migration, as `comline diff` reads an old local store. Its result, if cached, is + cached beside the archive and keyed by the archive's hash and the target format. +- **An archive says its format outside its payload,** in its manifest, so that a consumer can refuse a newer one + without decoding it, with the same error as a local store: the format, and the comline that reads it. +- **What a consumer needs from a dependency is its source.** A dependency is compiled from source, and its pin is checked + against that compile. The stored IR is only for comparing versions (compatibility, the changelog). So a version + too old to migrate costs a comparison, "can't tell", and never a build. +- **The registry holds, and never migrates.** It serves old archives as they were published. Migration is the + consumer's, in the consumer's core. +- **A pin never depends on which format an archive's store is in.** That is the requirement the + [pin bridge](hashes-and-history.md#pins) meets. + +## After 1.0 + +[core#146](https://github.com/ComlineProject/core/issues/146) asks that from 1.0 a newer comline reads what an older one +wrote, "either freeze them or keep a reader for every older version, with tests". This proposal is the second: the +framework *is* the readers, and its fixtures and laws are the tests. + +| | Before 1.0 | From 1.0 | +|---|---|---| +| **Formats that migrate** | the supported ones: 3, and 1 and 2 if they get built | every format released at or after 1.0, for ever. A released format's frozen types, steps, fixtures and seal are never removed | +| **A format below the floor** | refused, the store kept, with the options of [what cannot be migrated](steps.md#what-cannot-be-migrated) | the same, and the error names the **bridge release**: the last comline that still migrates it | +| **Moving the floor** | any minor release | a major one, announced one minor earlier; the bridge release stays installable | +| **Cost of a bump** | a step and its fixtures | the same, **and permanent**: it is carried for ever | +| **Additive changes** | a [ledger](formats.md#format-3-is-a-family) row each | a ledger row each, a fixture, and the same lines in the frozen copy | + +Because each bump is permanent from 1.0, bumps are worth batching: the policy proposed is that a format bump is +planned, not incidental. Cheap changes are additive ones, and the gate makes those cheap. + +A *version* is not a format. Package version 0.3.0, built long ago in a format still on the support list, is migrated +like every version of its store: nothing is dropped, and it can be compared at the precision it recorded, nothing +finer. A version in a store below the floor is one the store still has, in the backup, and one `comline store replay` +can rebuild from Git. + +The spec side needs nothing of this: the spec is append-only for anything frozen +([the vocabulary](../shared-vocabulary/catalog.md#versioning)), so an old version reads under a newer spec without a step. +Only the IR format does. + +## Still open here + +- **The floor now.** Whether formats 1 and 2 are worth building: they are the only lossy steps that exist before format + 4, and they exercise the unknown machinery on real data, but format 1 existed fifteen hours and no CLI built it. + Leaning: build 2→3, which a released CLI did build for six days, and decide 1→2 when the engine exists. +- **How long a bridge release stays supported,** after 1.0. diff --git a/docs/docs/proposals/ir-migrations/format-4.md b/docs/docs/proposals/ir-migrations/format-4.md new file mode 100644 index 0000000..f241082 --- /dev/null +++ b/docs/docs/proposals/ir-migrations/format-4.md @@ -0,0 +1,100 @@ +# Format 4 as the first client + +Status: **draft** · part of [Migrating stored IR across format versions](index.md) + +**The question.** [Granular `@unverified`](../shared-vocabulary/granular-unverified.md), item 3 of +[core#222](https://github.com/ComlineProject/core/issues/222), changes what a version freezes. What does the 3→4 step +do with every format-3 version already stored, and what is the package author asked? + +Format 4 is not designed here. It is the shape that proposal accepted, and this page is what the step does with it. +Anything that depends on the final shape is marked. + +## What changes + +Format 3 freezes what core worked out as a **set**, and where it was written: + +- a protocol's `parameters` hold `Unverified(Vec)` beside the `@unverified` reason, and, since + [#265](https://github.com/ComlineProject/core/pull/265), markers on arguments, functions, fields and types are + properties in *their* parameters; +- the package's own `unverified` is a config unit, a `ConfigValue` as written; +- a declared encoding records what it `establishes` and whether it `has_vectors`. + +Format 4 freezes, per the decision, **(guarantee, scope path, chain to each cause, reason)**, so that a comparison sees +narrowing and widening and an acknowledgement is pinned to what it covered. + +## The step + +| Item | In format 3 | In format 4 | What `up` writes | +|---|---|---|---| +| each given-up guarantee | a member of a protocol's set | a record per guarantee | one record per member; the `guarantee` is carried | +| its **scope** | where the marker is written: the protocol, or an inner property since #265 | a scope path | the path of the marker that covers it, read from the stored units. Before #265 only a protocol could carry one, so the protocol is **known**, not guessed | +| the **chain** to each cause | not stored | the hops, package by package | **`Unknown`**, typed, with the reason "format 3 records the set, not the path to its causes" | +| its **reason** | one string on the plain marker, or a dictionary as written | a reason per guarantee | a plain marker's string is copied to each guarantee it covers, as written; a dictionary's entries as written | +| a package-default cause, acknowledged in `config.idp` | the `Unverified` config unit, as written | pinned to the setting that causes it | the unit carried; its pin to the setting is **`Unknown`** | +| a declared encoding's guarantees | `establishes`, `has_vectors`, as they were (the strict reading where absent) | the same | carried | + +Two things it does **not** do. It doesn't compute the chains: that needs the guarantee engine and the catalog, which +are live code and change, and a step is frozen. And it doesn't write an empty chain list or a one-hop chain to a +guarantee whose causes weren't recorded; that would read as "no other cause". + +The step is `up`, `down` and an `unknown` list, like any. `down` drops the typed unknowns and the scope paths that +equal the old protocol-level reading, and restores the old set, so the witness holds. The `unknown` list is the +two rows above, and the square compares the migration with a fresh format-4 build whose chains and package pins are +forgotten, with each scope that the old format could place left alone. + +## Chains in the old versions + +Recomputation, not the stored record, answers any question that needs a chain. A version stores its inputs (types, +a dependency's schemas and, since #262, its config), and the current core can compute the chains from those under +the current spec ([history](hashes-and-history.md#history)). So: + +- `diff` and the changelog **recompute chains for both versions** and compare those. A migrated version compared with an + unchanged successor shows no change, even though its stored chain is unknown. +- The chains stay unknown only where an input is missing: a dependency's config in a version older than #262, a type + whose width format 2 lost. Those are `unseen`, as everything else that can't be seen is + ([history](hashes-and-history.md#what-a-comparison-does)). +- "Narrowing a marker isn't a major change; widening one or adding a cause is" is a rule over *recomputed* causes for + these versions, so it needs no stored chain to hold. + +## Acknowledgements + +Three things are called an acknowledgement, and the migration treats them differently, because only one of them is in +the store: + +| | Where it is | At format 4 | +|---|---|---| +| **The author's marker** (`@unverified …`) | the source | Unchanged, and still valid: the plain form acknowledges what core computes at its scope. The frozen record of it is rebuilt by each build | +| **A consumer's acceptance** (`accept_unverified = { Billing = […] }`) | the consumer's `config.idp` | Pinned to chains in format 4. An existing entry is a set and has no pin, so it matches nothing, and the build **asks again**: it refuses, names each protocol, shows what is now recorded (the guarantee, the chain, the origin's reason), and prints the entry. `comline accept` is the existing way to write it | +| **A package's own `unverified = { … }`** | its `config.idp`, next to the setting | Pinned to the setting. The same re-asking applies, once, for a package that wrote it | + +This is the intended behaviour of [core#222](https://github.com/ComlineProject/core/issues/222): "a new cause, or a swapped +type upstream, asks again at every hop". A format change that makes every existing acceptance ask once is the same +event, because the pin it lacked is the pin the invariant needs. **Nothing is rewritten.** The CLI has a precedent +that is a good model: when bounds on types arrived, `build` listed every position, and the CLI *offered* package +defaults with maxima the author typed, never guessed +(`bounds_migration.rs`). Here the offer is each acceptance, with what it now covers, and the author decides. + +*Open:* whether a transition period accepts an unpinned entry with a warning, as pins might +([hashes](hashes-and-history.md#pins)). The safety invariant argues against it: an acceptance that isn't pinned to a +chain is exactly what the invariant is there to remove. + +## Consumers + +What reads a version with unknown chains, and what each does: + +| Consumer | With unknown chains | +|---|---| +| `build`, the guarantee check | recomputes from inputs; the unknown is never what it judges on | +| `diff`, `doc --version`, the changelog | recompute; list what is `unseen` | +| `inspect` | shows the recomputed chain, marked "recomputed, not recorded" | +| the generated `@unverified` docs ([generation#93](https://github.com/ComlineProject/generation/issues/93)) | the chain as recomputed. Where it can't be, "origin not recorded" | +| a generator given an old version | *Open.* The unit it is given has the typed unknown. Refusing a version that has one is the strictest reading, and a flag could allow it. Leaning: pass it, since a generator that prints a chain must print "not recorded", and add the field to the protocol, where it can ignore it | + +## Fixtures and tests for 3→4 + +The corpus ([proof](proof.md#the-fixture-corpus)) needs, for this step: the chain graphs of the oracle built by the cores +of each format-3 ledger row from `bd9d9f4` on, with the inner scopes (`66feecb`) and the package-scope `unverified` +(`9ea142e`) in the later rows. The square is then: the format-3 store of a graph, migrated, against the format-4 build of +the same graph with chains and package pins forgotten. The differential test draws random graphs and histories from +the oracle's generators, the same ones that test the invariant, so the property that no guarantee goes +unacknowledged is checked over migrated stores too (stage 3 of the rollout). diff --git a/docs/docs/proposals/ir-migrations/formats.md b/docs/docs/proposals/ir-migrations/formats.md new file mode 100644 index 0000000..1ba0b90 --- /dev/null +++ b/docs/docs/proposals/ir-migrations/formats.md @@ -0,0 +1,135 @@ +# The formats + +Status: **draft** · part of [Migrating stored IR across format versions](index.md) + +**The question.** What exactly is in a store, which of it depends on the IR format, and what does a format +number promise about it? + +## What a store holds + +| Path | Holds | Depends on the format | +|---|---|---| +| `objects//` | One object: LZ4 over CBOR, addressed by the blake3 of the CBOR. Three kinds: `Blob { content }`, `Tree { entries: [{ mode, name, hash }] }`, `Commit { tree, parents, author, timestamp, message, version, release? }`. | The *container* does not. See below. | +| a blob's `content` | One frozen schema unit (`FrozenUnit`), or a version's `config` (a list of config units), `docs`, `facts` (`History`), a dependency's `manifest`, `config` and `dependency_configs`. | **Yes.** This is what a format version describes. | +| `refs/heads/main` | The hash of the newest commit, hex. After a Git merge, conflict markers around two. | No | +| `FORMAT_VERSION` | The stamp. | It is the stamp. | +| `deps-cache/` | Git checkouts of dependencies. A cache, ignored by Git. | No, and not part of the store | + +A version's root tree has one `schema_` tree per schema file, a `dep_` tree per vendored dependency +(`schema_`, `manifest`, `config`, `dependency_configs`), and the blobs `config`, `docs` and `facts`. The +[CAS guide](../../guides/ir/cas.md) describes this for format 3. + +**Proposed: the container is not a format.** The object layout (CBOR, LZ4, blake3, `Blob`, `Tree`, `Commit`, the +ref file) is the same in formats 1 to 3, and a format version covers the payloads inside it. Three things follow: + +- A reader of the container needs no format knowledge. The CLI and core can list versions, read each commit's + `version` and `release`, and walk parents in any store they can find the container of. Today + `resolved_version` reads a dependency's committed `.comline/` this way and falls back to `0.0.0-unbuilt` on any + failure, which hides a store it merely couldn't read. +- A change to the container (the hash, the compression, the ref layout) is also a format change, and a step like + any other. Its model is the whole store. +- Every hash above a changed payload changes with it, because parents, tree entries and `release.since` are all + hashes. See [hashes](hashes-and-history.md#object-hashes). + +## History of the formats + +| Format | Stamp | On `master` | What it is | +|---|---|---|---| +| **0** | none | 2026-01-10 (`534da82`, the CAS), config blob 2026-08-30, dependencies vendored 2026-09-30, until 2026-10-11 | bincode, which tags struct fields and enum variants by position, no names on the wire. About 20 commits reshaped the persisted types in this period, none of them marked. | +| **1** | `1` | 2026-10-11 (`f71ef96`, PR #84) | Name-tagged CBOR. `ensure_compatible_store` and the stamp are introduced. `FrozenUnit::Parameter` removed. | +| **2** | `2` | 2026-10-12 (`dbdb13f`, PR #87) | `Property`'s `expression: Option`, the written value with its quotes stripped, becomes `value: ConfigValue`, which keeps the written form. A field renamed and retyped. | +| **3** | `3` | 2026-10-18 (`eda0684`, PR #108) | Types stop being `KindValue` text: `TypeRef` (`Primitive`, `Named`, `Array { element, length }`, `Union`, `Unit`), `Literal` and `Condition`. `Field`/`Constant` `kind_value` becomes `ty` and a literal; `EnumVariant` holds its name; `Assert` gains `expr`. | + +Notes that matter to migration: + +- **Format 1 lived about 15 hours on `master`.** Core's CI was green at its first merge (`d2763cd`, 2026-10-11) and at + format 2's (`59587ba`, 2026-10-12). No CLI revision pinned core in format 1. +- **Format 2's lossiness is real.** `KindValue` text froze `T[8]` and `T[]` both as `T[]`, and `f32` and `f64` both + as `float`. A 2→3 step cannot recover either. Format 1's `expression` had no quotes: `@framing = "datagram"` and + `@framing = datagram` read the same, so a 1→2 step cannot say which form was written. +- **The CLI pinned format 0 until 2026-10-11 (`f934a66`, core `f035379`), format 2 from 2026-10-12 (`086cd32`) to + 2026-10-18 (`00ee242`), and format 3 from 2026-10-18 (`1455192`).** Those revisions have a committed + `Cargo.lock`, and the git revisions their `Cargo.toml` names (core, generation, comline-rust, comline-typescript) + all still exist. [The proof](proof.md#building-the-old-revs) uses + them. + +## Format 3 is a family + +The doc comment on `CURRENT_IR_FORMAT_VERSION` says growing `FrozenUnit` needs no bump. Since 2026-10-18 these have +been added without one, and so a "format 3" store can lack any of them: + +| Added | Commit | Absent reads as | +|---|---|---| +| validator call `span`, `arg_spans` | `0983015`, 10-18 | unknown span, `(0, 0)` | +| type alias `parameters` | `8243d98`, 10-18 | none | +| `Assert.message_span`, `uninterpolated`; validator `validate_span` | `ad2b3f5`, 10-19 | none | +| `Unverified(Vec)` in a protocol's parameters; the `Guarantee` enum | `bd9d9f4`, 10-20 | no recorded set | +| `Commit.release` | `af0371f`, 10-20 | a release only if the version has no pre-release part | +| config units `Handshake`, `HandshakeParams` | `6571780`, 10-20 | the default handshake **of the reading core** | +| config unit `Wire` | `031d74c`, 10-25 | the default format and framing **of the reading core** | +| `ConfigValue::Dict` | `a5d6f85`, 10-25 | a variant older readers can't decode | +| config unit `Encoding` | `f643d51`, 10-31 | no declaration | +| root `docs` blob | `038591e`, 11-01 | no `//!` headers | +| root `facts` blob; config unit `PackageId` | `f873d98`, 11-01 | computed when read; no id | +| `Encoding.establishes`, `has_vectors` | `4ba7c57`, 11-01 | the strict reading: nothing established, no vectors | +| `PrimitiveType::Char`, `Array.max` | `4eb6228`, 11-02 | a variant older readers can't decode; no maximum | +| `Guarantee::BoundedSize` | `2b6d636`, 11-02 | a variant older readers can't decode | +| facts' `number`, `retired`, `numbered` | `8ec27df`, 11-03 | `numbered = false`: computed when read | +| `Encoding.grammar`, `calls`, `directions` | `4da05ab`, 11-07 | none | +| facts' per-protocol `encodings`; the defaults moved to `datagram@3` and `comline@2` | `bbbd957`, `9a6482e`, 11-08 | resolved when read, with what an absent value *was* (`IMPLICIT_FRAMING`, `IMPLICIT_HANDSHAKE`) | +| config unit `Unverified` | `9ea142e`, 11-10 | none | +| `dep_/config`, `dependency_configs` | `965e781`, 11-12 | **unseen**: not empty, not defaulted | +| `FrozenArgument.parameters` (markers on arguments) | `66feecb`, 11-12 | none | + +The rule, as written, is sound for the direction it was written for: a new core reads an old store. It has four +gaps, and a migration framework has to close them: + +1. **A store holds commits of several ledger levels.** Each build writes what its core knew. The absent-entry + reading is per commit and per entry, so "format 3" says nothing about which of the rows above a given version + has. +2. **The rule is one-directional.** A shipped binary that meets a new *variant* (`BoundedSize`, `Dict`, `Char`) fails + to decode. One that meets a new *field* drops it without a word. Neither is checked. +3. **Some additions change what old bytes mean.** An old version's `Unverified(Vec)` has no + `bounded_size`, because the guarantee didn't exist; under today's spec the same schema gives it up. An absent + `wire.framing` meant `datagram@2` when it was built and `datagram@3` now, so a reader has to say which. +4. **Nothing checks it.** `f71ef96` records the decision not to add a golden-hash check, "can revisit if it becomes a + real recurring problem". Twenty additions in 25 days is that. + +## Each format is written down + +**Proposed:** a format is not "whatever the types were at that commit". Each format N has a directory in a new +`formats` crate, and these four things in it: + +| File | Holds | +|---|---| +| `SCHEMA.md` | Every blob, tree and entry; every field, its type and meaning; what an absent entry or field means (the *absence reading*); the ledger. The source of truth for the format. | +| `types.rs` | Frozen copies of the format's types: serde derives and nothing else, with no behaviour, no dependency on `comline-core`, and every `#[serde(...)]` attribute that decides the bytes. | +| `codec.rs` | `decode` and `encode` between bytes and the format's model. The only code that touches format-N bytes. | +| `ledger.toml` | One row per addition made without a bump: the commit, the date, what was added, the absence reading, and the fixture that has it. | + +A format is **sealed** when the next one is introduced: `formats/MANIFEST` records the blake3 of each sealed +directory, and CI fails if one changes. The current format is not sealed, and it is held to the live types by a +test instead: `live ≡ frozen(current)`, meaning that for every fixture and for random values, the core's types and +the frozen copy encode to the same bytes and decode each other's. A change to the live types without the same +change in the frozen copy fails it, and so does a change that alters a fixture's bytes. + +This replaces the doc comment's rule with a checked one. Without a bump, the only changes allowed are: + +- a new entry in a root or dependency tree whose absence reads as unknown; +- a new field with `#[serde(default)]` and `skip_serializing_if` for the default, so that content which doesn't use + it keeps its bytes; +- a new enum variant, if nothing existing needs it. + +Each one adds a ledger row and the same lines to the frozen copy, in the same pull request. Any other change is a +bump. The `skip_serializing_if` condition is the one the code already follows ("so a schema without bounds freezes +to the same bytes as before them"), because the bytes of a frozen unit are also what a [dependency pin +hashes](hashes-and-history.md#pins). + +## Still open here + +- **Whether `SCHEMA.md` is the source and the [CAS guide](../../guides/ir/cas.md) links to it, or the guide stays the + description of the current format and `SCHEMA.md` is generated beside it.** Leaning: `SCHEMA.md` in core is the + source; the guide describes the current format and says so. +- **Whether an additive change should raise a second number** (a ledger level, beside the format) so that an older + binary refuses to *write* into a store whose commits use additions it would drop. This would turn gap 2 into an + error. The cost is a second stamp to keep in step. diff --git a/docs/docs/proposals/ir-migrations/hashes-and-history.md b/docs/docs/proposals/ir-migrations/hashes-and-history.md new file mode 100644 index 0000000..ace3b1d --- /dev/null +++ b/docs/docs/proposals/ir-migrations/hashes-and-history.md @@ -0,0 +1,140 @@ +# Hashes, pins and history + +Status: **draft** · part of [Migrating stored IR across format versions](index.md) + +**The question.** A migration rewrites what hashes cover. Which hashes may change, which must not, how does a +dependency pin keep meaning what it meant, and what do compatibility checks and the changelog say about a history +that was migrated? + +## The hashes that depend on the IR + +| Hash | Covers | Computed | Under a migration | +|---|---|---|---| +| **Object** (blob, tree, commit) | the object's serialized bytes | at write | changes exactly where it has to: [below](#object-hashes) | +| **Dependency pin**, `hash = "blake3:…"` in `config.idp` | scheme v2 ([core#261](https://github.com/ComlineProject/core/pull/261)): the CBOR of every schema's frozen units, and the dependency's frozen config units, sorted, minus `PublishRegistry` and `Dependency` | from a fresh compile of the dependency, checked against the pin | not in the store, **never rewritten**, [bridged](#pins) | +| **`manifest`** blob | the content hash verified when a dependency was vendored, as text | at build | carried verbatim. Nothing in core reads it today | +| **`schema_ir_hash`**, the handshake's `ir_hash` | the CBOR of one schema's units, folded to 64 bits | embedded in generated code | follows the units' bytes. Not stored; generated code isn't migrated | +| **`shape_hash`, `param_hash`** | a protocol's canonical wire shape; an encoding's `(kind, name, params)` | when facts are read | **must not change**: peers negotiate on them | + +The last is the point of the table: only the first three are *stored*, and only the first changes by construction. + +## Object hashes + +A hash covers bytes, and a commit holds its tree's hash and its parents'. So a changed payload changes its blob, every +tree on the way up, its commit, and every later commit, because each holds its parent's hash. `release.since` is a +hash too, and so are the second parents a reconcile leaves. + +- **What may change:** a payload whose encoding changed, and everything above it. For a format that touches only + `schema_` blobs, that is those blobs, the trees above them, and the whole commit chain. +- **What must not:** a payload whose bytes are the same in N and C keeps its hash, and is not rewritten. A step that + moves one field of one unit changes that unit's blob and no other. +- **Who maps them:** the engine, bottom-up, from the model, where references are [version ids](steps.md#the-model). + A step author never sees a hash. The old-to-new map is a bijection on commits, and is checked. +- **Outside the store:** a CI log, a note, or `comline doc`'s cache (`.comline-doc.json` is keyed by commit hash) may hold + an old one. `comline store migrate --map` writes the map, and caches should key on the IR format as well. + +## Pins + +A pin is hashed from **IR bytes**: `schema_units_cbor` is `to_cbor` of the units. So a format change that alters those +bytes for a dependency changes its pin, and the scheme tag (`comline-dep-content-v2`) doesn't know about the IR format. +Core#261 handled the case of a scheme change, and its message ("this pin was made by an older comline") is the +precedent. A *format* change is the same event with no message, and the earlier format changes very probably did it +(not checked) without anyone being told. + +Two things make that unacceptable for migrations. A migration that turns every pin in a repository into a mismatch is a +migration that breaks the build it was meant to save. And the [offline proposal](https://github.com/ComlineProject/docs/pull/163) +makes the hash the identity of a dependency: caches and signed bundles are keyed by it, so a hash that moves with core's +internal format would invalidate both. + +**Proposed:** a pin made under format N, by scheme S, **still verifies under C when the dependency's content can be +expressed in N**, and a pin is never accepted otherwise. + +1. To check a pin that doesn't match, core takes the dependency's compiled model and applies `down` through each step + back to N. If every step's witness is defined on it, the content has nothing N couldn't record. It encodes the + result with N's codec, hashes it as scheme S did, and compares it with the pin. +2. **A match is exact or it isn't one.** Equality of the hash, of the content N would have recorded. Nothing is weaker: + the content has no information the pin didn't cover, so every guarantee the pin gave is still given. +3. **If the content uses something N lacks** (a dependency that now has chain markers, with format 4), the witness is + not defined, no older pin can match, and the error says what is new: "this pin was made under IR format 3; the + dependency now records chains that it couldn't have covered. Review it and re-pin: `hash = "…"`". The command that + writes a re-pin exists already as a request (`comline add --repin`, or the future `update`, core#261). +4. **`config.idp` is never rewritten** by a migration or by this check. +5. Cost is only on a mismatch, as `mismatch_message` is today, and only through the formats at or above the support + floor. + +A package that doesn't use what format 4 added therefore keeps every pin through the bump, with no action, and the +tests show it ([proof](proof.md#laws)): `golden-pins.json` holds each fixture dependency's pin as the *historical* core +computed it, and the current core must verify it through the bridge. A package that does use it is asked, once, and +told why. + +| | A. Verify through the witness (**proposed**) | B. A pin form independent of the IR | C. Re-pin on every format change | +|---|---|---|---| +| An unchanged dependency, after a bump | keeps its pin | keeps its pin | **every pin breaks** | +| A new scheme | no | yes: a v3, so every pin changes once | no | +| A second definition of "content" to keep in step with the IR | no | yes | no | +| Proof | golden pins from the old cores | golden pins from the form | none needed, and none possible | +| Costs | `down` must be defined on a compile, not only on a stored version | the form must also carry new information the day the IR does | the offline proposal's bundles and caches | + +*Open:* whether a pin that verifies through the witness is **accepted silently,** accepted with a warning that names the +format ("re-pin to move to the current one"), or rejected with the same message as case 3 (the re-pin being a one-line +change that is then explicit). Accepting is sound by the argument above, and rejecting is the stricter and costs +every repository one edit per bump. Leaning: accept with a warning once the bridge has the golden pins behind it. +Option B stays available if the bridge turns out too costly to keep through many formats. + +The pin gate in [the proof](proof.md#the-gate) makes the move visible: a change that moves the *fresh* pin of a fixture +dependency has to say so in the pull request. + +## History + +Compatibility, `diff` and the changelog compare versions, and a migrated history is not the same as a freshly built +one in what it records. The principle is: + +> **Compare from inputs; recompute outputs; keep what only history can say.** + +| What a version holds | Examples | In a migrated history | +|---|---|---| +| **Inputs** | the schema units, the vendored dependencies' schemas and configs, `config`, `docs` | carried. If a version never kept one, it is **unseen**, not empty | +| **Outputs of a rule** | a protocol's given-up guarantees and their causes, the encodings in play, a shape hash | recomputed when read, by the core that reads, under the current spec. Where stored (for pinning an acknowledgement, or to be quick), the stored value is a record and not an input to comparisons | +| **History itself** | `version`, `release` records, a protocol's `line`, a function's `since`, wire numbers | carried as data. They depend on the chain, not one version, and they are promises that peers may already hold | + +Version numbers in particular are **kept** even where today's rules would have numbered the history differently. A +migration is not a rebuild, and a version that was released as `0.4.0` was. + +### What a comparison does + +- **Both sides are recomputed under the same, current spec.** [Granular `@unverified`](../shared-vocabulary/granular-unverified.md#a-newer-spec-that-finds-an-existing-loss) + says `diff` recomputes both versions under the newer spec, so a loss the spec learned to see is in both and is no + change. A format-3 version whose protocol never had `bounded_size` in its frozen set (the guarantee didn't exist) is + not "narrower" than today's: it is recomputed, and gives it up like its successor does. +- **The spec a version was frozen against** is, today, recorded nowhere: the decision in the vocabulary is that each + frozen version records it, and that isn't built. A version without it reads as `unknown`, and is recomputed under + the current spec like any older one. **Proposed:** record it as a root-tree entry, absent meaning unknown, in the + first release of the framework, so that every version built after it has it. *Open:* whether a commit also records the + core that wrote it (`produced_by`), which would tell a ledger row without guessing. +- **What can't be recomputed is `unseen`.** If a rule needs an input the version lacks (a dependency's config, a + type's width), the comparison lists it in the same `Comparison::unseen` that [core#262](https://github.com/ComlineProject/core/pull/262) + added: the version and the path, with the reason from the `unknown` entry. A rule that needs a fact it can't see reads + the strictest (the same way `Encodings` does when it isn't `complete`), and the bump is **at least a patch**: a + comparison that couldn't see part of what it compares doesn't say that nothing changed. It can be higher, since + "strictest" can be a breaking change: a type of unknown width against a known one is possibly breaking. +- **The changelog** shows the same: a change it found, and, as lines of their own, what it could not compare, in the words of + the reason. `comline doc --version V` for a version with unknowns says which parts are. + +| Case | Result | +|---|---| +| A package migrated from format 3, source unchanged | Nothing found; see [the first build](#the-first-build-after-a-migration) | +| The spec has since added a guarantee an old protocol gives up | Recomputed on both sides: no change, no false major | +| A version from before dependency configs were kept (core#262) | `unseen: Dependency("acme")`, at least a patch: as now | +| A format-2 version with an array of unknown length, compared to today's | the path listed as unseen; possibly breaking | +| A version whose `release` says `0.4.0` | `0.4.0`, whatever the current rules would number it | + +### The first build after a migration + +A migrated head lacks what the current core knows, so a fresh build of the unchanged working tree differs from it, +by exactly the unknowns. Without care, the first `comline build` makes a commit, with a bump of at least a patch, to +say that nothing changed. + +**Proposed:** the build's "same as before" test compares **modulo unknown**: if the working tree equals the head at +every field the head knows, nothing is committed and no version moves. What the build learned is not written back, +since history is append-only, and is not needed: it is recomputed when read. Any real change is compared with the +unknowns listed as unseen, as above. diff --git a/docs/docs/proposals/ir-migrations/index.md b/docs/docs/proposals/ir-migrations/index.md new file mode 100644 index 0000000..47ceb60 --- /dev/null +++ b/docs/docs/proposals/ir-migrations/index.md @@ -0,0 +1,140 @@ +# Migrating stored IR across format versions + +Status: **draft** · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · feeds +[core#146](https://github.com/ComlineProject/core/issues/146) (reading older IR after 1.0) and the +[granular `@unverified`](../shared-vocabulary/granular-unverified.md) work, which is IR format 4 · seven parts, +read in this order: [the formats](formats.md), [steps](steps.md), [running a migration](running.md), +[proving it](proof.md), [hashes, pins and history](hashes-and-history.md), +[beyond the local store](beyond-the-store.md), [format 4 as the first client](format-4.md) + +**The question.** `.comline/` holds every version a package has had, and compatibility checks, the changelog and +`comline doc --version` all read it. When core changes the shape of what it stores, `ensure_compatible_store` +deletes the directory, so the version history is lost with each bump. **Can every stored format, past and future, +be migrated to the current one, so that nobody can tell the result from a store the current core built, except +where the old format never recorded something, and so that this is shown by tests instead of asserted?** + +## What it looks like today + +| Part | Today | +|---|---| +| **The stamp** | `.comline/FORMAT_VERSION` holds `CURRENT_IR_FORMAT_VERSION` (3). `ensure_compatible_store` compares with `!=`: anything else, a missing stamp included, deletes all of `.comline/`, `deps-cache/` too. | +| **Who deletes** | `comline build` does it with a warning in the log. The read commands (`diff`, `doc`, `generate`, `inspect`, `release`) refuse, and say that `build` "starts the version history over". An older binary deletes a store that is *newer* than it, because the check is `!=`. | +| **What a number means** | Four on-disk shapes have existed: an unstamped bincode store and formats 1, 2 and 3. Format 3 is not one shape: [twenty additions](formats.md#format-3-is-a-family) have landed in it without a bump. | +| **What the rule is** | Bump only for a rename, a removal or a retype; anything else is "additive". The rule is in a doc comment. Nothing checks it: the commit that introduced the stamp deferred a golden-hash check "if it becomes a real recurring problem". | +| **Hashes that cover IR bytes** | Every object's address, a dependency pin (`hash = "blake3:…"`, scheme v2, [core#261](https://github.com/ComlineProject/core/pull/261)) and `schema_ir_hash`. A format change moves them. | +| **Proof** | Round-trip tests over real packages, an oracle property test and fuzz targets for the guarantee check, differential tests in the runtime. None says anything about an old store. | + +Format 3 is the first format with a population of stores worth saving, and the next change, format 4 +([core#222](https://github.com/ComlineProject/core/issues/222)), would wipe all of them. This proposal comes first +so that it doesn't. + +## What this proposes + +1. **[Formats written down](formats.md).** Each format has a document (blobs, trees, entries, field meanings, what + an absent entry means), frozen copies of its types, and a ledger of every addition made without a bump. The + additive rule is tightened and checked. +2. **[Steps](steps.md).** A migration is a chain of single steps `N → N+1`. A step is a total, deterministic + function over frozen format-N types, with a written list of what N cannot know. Those things are written as an + explicit **unknown**, never as a default that looks complete. Every step also has an inverse witness, so no + step can drop what a version recorded. +3. **[Running a migration](running.md).** The new store is written beside the old one, verified, and swapped in; + the old one is kept. A migration can be checked without writing (`comline store migrate --check`), resumed, and + repeated. A store newer than the binary is an error, never a deletion. +4. **[Proof](proof.md).** A corpus of real stores built by the real historical cores, an equivalence stated field by + field, differential property tests against an old core, laws (totality, determinism, idempotence, composition, + hash stability), and a CI gate that fails a format bump without its step, fixtures and document. +5. **[Hashes, pins and history](hashes-and-history.md).** What may and may not change in a hash, how a dependency + pin survives a format bump without being weakened, and how compatibility and the changelog read a migrated + history. +6. **[Beyond the local store](beyond-the-store.md).** Every other place stored IR or its hashes appear, and the + policy for very old formats after 1.0. +7. **[Format 4 as the first client](format-4.md).** + +## Rules that hold whatever the details + +- **A store is never deleted to change its format.** Migration replaces deletion; where migration is impossible + the old store stays, and the error says what to do. +- **A step is a pure function of one format's data.** No clock, no environment, no network, no randomness. Two + people migrating the same store get the same bytes. +- **What a version recorded is kept.** What its format never recorded is unknown, and unknown fails closed + everywhere it is read: [#251](https://github.com/ComlineProject/core/issues/251)'s rule for a dependency's config, + and the fail-closed rule of the [shared vocabulary](../shared-vocabulary/granular-unverified.md#fail-closed). +- **Nothing is replaced until it is verified,** and what is replaced is kept until someone removes it. +- **Accurate means tested against a fresh build.** A migrated version equals what the current core would have built + from the same source, field for field, except for the fields the step declared unknown. +- **A format change brings its step, its fixtures and its document in the same pull request,** or CI fails. + +## What is proposed, in one table + +| | Proposed | +|---|---| +| Unit of migration | the whole store, version by version, all or nothing | +| Steps | `N → N+1`, typed on frozen copies; `up`, `down` (witness), and a list of unknowns | +| Unknown | typed `Known` where the new type has room, otherwise an `unknown` entry in the root tree that keeps the original value | +| Atomicity | write `.comline.migrating/`, verify, rename the old store to `.comline.format-N.bak/`, rename the new one in | +| Backup | a sibling directory, kept until `comline store prune`; never deleted automatically | +| Newer store | an error naming the format and the comline that reads it | +| Read commands | read an old store through an in-memory migration; they write nothing | +| Pins | verified through the witness: a pin made under format N still verifies if the dependency's content is representable in N | +| History | comparisons recompute derived facts from stored inputs under the current spec; what can't be recomputed is `unseen` | +| Gate | `formats/MANIFEST`, `live ≡ frozen(current)`, fixtures, and a bump check; see [the gate](proof.md#the-gate) | +| Support | formats 1 to 3 migrate; the unstamped bincode store does not, and is refused with options | + +## Rollout + +Staged pull requests, in the agreed order: the framework, then format 4, then cross-package chains, then tooling. +Each stage is useful on its own, and each is a checklist in [core#268](https://github.com/ComlineProject/core/issues/268). + +**1. The framework** (core, format stays 3): + +1. *A newer store is an error, not a wipe.* Core and the CLI message. A small pull request, released **before** + anything ships format 4, so that no binary in circulation deletes a format-4 store. +2. *Format documents and the gate, with no migration yet.* A `formats` crate with the frozen format-3 types, the + ledger, the seal, the `live ≡ frozen(3)` test, and golden hashes for the fixture packages. This stops further + no-bump drift before it grows. It also adds, as ledger rows, the two entries nothing writes yet, `unknown` and the + spec version a version was frozen against, with readers that fail closed, and it ships in a release with step 1 + **before** any step or format that writes either one. +3. *The fixture corpus and its generator,* with format-3 stores built by the historical revs at each ledger row, a + format-2 and a format-0 store, and the weekly job that proves the old revs still build. +4. *The engine and `comline store`:* container reader, plan, journal, verification, swap, backup, `--check`, with + the laws proved on a test-only toy format before any real step exists. +5. *Steps 2→3 and 1→2,* each with its fixtures: the first real lossy steps, exercising the unknown machinery. + +**2. Format 4** (core): the new frozen shape of [core#222](https://github.com/ComlineProject/core/issues/222) item 3, +step 3→4, its fixtures and format document, and the pin bridge, all in one pull request because the gate requires it. + +**3. Cross-package chains** (core): the recomputation of chains over stored dependency IR, `unseen` extended to +chains, and the oracle and the fuzz target run over migrated stores. + +**4. Tooling** (CLI, language server, generators, docs): + +- `build` and `release` migrate on their own, with the plan printed. +- `doc` caches key on the IR format. +- The language server and `inspect` show `unseen`. +- The guide on [the CAS](../../guides/ir/cas.md) loses its warning about deletion. +- Generators learn what a version with unknowns means. +- `comline store replay` for stores that can't be migrated. + +## Still open + +Collected from the pages; each is where it is discussed. + +- Whether every step needs the inverse witness or only the ones that could lose something + ([steps](steps.md#the-witness)). +- Whether `build` and `release` migrate by default or need an explicit `comline store migrate` + ([running](running.md#the-cli)). +- Where the backup lives and for how long, and whether format 4 should also move the store to a new directory name + so that an older binary can't delete it ([running](running.md#older-binaries)). +- What happens when Git merges two branches whose stores are in different formats + ([running](running.md#git)). +- Whether a pin that verifies through the witness is accepted silently, with a warning, or only by an explicit + re-pin ([hashes and history](hashes-and-history.md#pins)). +- Whether to spend the effort on formats 1 and 2, and where the support floor is before and after 1.0 + ([beyond the store](beyond-the-store.md#after-10)). +- Where the frozen formats live: a crate, a module, and whether the IR-only (`wasm`) build carries them + ([steps](steps.md#where-the-code-lives)). +- Whether a commit should record the core and spec versions that wrote it + ([hashes and history](hashes-and-history.md#history)). +- What generators and `doc` do with a version that has unknowns ([format 4](format-4.md#consumers)). +- Whether `comline store replay` (rebuilding a history from Git) is worth building for the stores that can't be + migrated ([steps](steps.md#what-cannot-be-migrated)). diff --git a/docs/docs/proposals/ir-migrations/proof.md b/docs/docs/proposals/ir-migrations/proof.md new file mode 100644 index 0000000..1bf1e7a --- /dev/null +++ b/docs/docs/proposals/ir-migrations/proof.md @@ -0,0 +1,209 @@ +# Proving a migration + +Status: **draft** · part of [Migrating stored IR across format versions](index.md) + +**The question.** "The migration is accurate" has to be a statement a test makes, for every old format and every +future one, and a format change has to be unable to land without the tests that make it. What are they? + +## What accurate means + +A migrated version has to be what the current core would have built from the same source, except where the old +format could not know something. As a square that must commute: + +```text + a package's source, at each point in its history + ┌────────────────────┴────────────────────┐ + build with the core of format N build with the current core (format C) + │ │ + a store of format N a store of format C + │ │ + migrate N → C forget N → C + │ │ + └──────────────── must be equal ──────────┘ +``` + +`forget` is not a second implementation of the step. It acts on a *current* value and does one thing: it replaces the +paths on the steps' `unknown` lists with an unknown, as `up` would have, and touches nothing else. It is derived from +the lists, so it is a few lines whatever the format. Three kinds of mistake each break the equality: + +| The step | The equality fails because | +|---|---| +| computes a wrong value for a field N could know | migrated has a value, forget leaves the fresh one, and they differ | +| invents a value for a field N could not know | migrated has a value, forget has an unknown | +| writes an unknown for a field N could know | migrated has an unknown, forget leaves the fresh value | + +## What is compared + +Equality is on **root trees**, not commits. A tree's hash covers everything in it, so equal tree hashes are equal +bytes, and the bytes are canonical (CBOR with ordered maps, the same value always encodes the same). It is compared +per part: + +| Part of a version | Compared | Why | +|---|---|---| +| `schema_` trees, `config`, `docs`, a dependency's `schema_`, `config` and `dependency_configs` | **exactly,** after `forget`, as bytes | what the format records | +| a path on a step's `unknown` list | the migrated placeholder and `unknown` entry against `forget`'s | N never recorded it | +| `facts` | **exactly,** except the differences in the fixture's `known-differences.toml` | a record of what the build computed under the rules then; a rule that changed since moves a recomputed fact, and the file names the commit that changed it | +| a commit's `version`, `release` and parent structure | exactly, through the old-to-new map | user data | +| a commit's `author`, `timestamp`, `message` and its **hash** | **not compared** | provenance the fresh build can't share: a different time and user, and the hash follows from them | +| a dependency's `manifest` | not compared with the fresh build; compared with the **old** store's | it records the content hash verified at build time, under the pin scheme then in force | +| `deps-cache/`, backups | not part of a store | | + +For the comparison to be fair, a fixture's history must be one on which today's rules number the versions as the +old core did. Where they don't, the fixture records why. That is the proof's one admitted gap, and it is a list a +reviewer reads. + +## The fixture corpus + +**The sources.** A fixed set of packages, committed once and never edited except to add: + +- `tour`, the schema in `core/tests/cas/ir_roundtrip_tests.rs` that uses every construct the grammar has; +- the packages in `core/tests/fixtures/packages`, and `std`; +- a package with `//!` headers; +- a package with a path dependency that declares encodings, a `wire`, `unverified` and `settings`, and one whose + dependency has a dependency of its own (so `dependency_configs` is there); +- packages with `@unverified` at each scope and with `handshake = "none"`; +- the chain graphs of the oracle ([core#244](https://github.com/ComlineProject/core/pull/244)): A's type reached by B, reached by C. + +Each is a **scripted history**: snapshots of the sources and the actions between them (`build`, `release`, +`build --reconcile`), with at least a breaking change, a release, a rebuild that changes nothing (so makes no +commit), and, for one package, two heads left by a Git merge. They live in `core/tests/migrations/corpus/`. + +**The stores.** For each format and each [ledger row](formats.md#format-3-is-a-family) of it, the real +`.comline/` that the real core of that era built from each script, committed under +`core/tests/migrations/stores/format-///`, with a `PRODUCED_BY` file (the revision, the digest of +the lockfile, the toolchain, the command). A store is a few objects per version, so the corpus stays small. + +| Format | Built by | Notes | +|---|---|---| +| 0 | the CLI at `f934a66` (core `f035379`), lockfile committed | a store to be **refused**; the test is that nothing misreads it | +| 1 | core at `d2763cd`, plus a small patch that adds a build example | no CLI pinned it | +| 2 | the CLI at `086cd32` (core `6101ef8`), and at `00ee242` (core `30dc8ed`) | first and last of the seven pins in this format | +| 3 | one revision per ledger row, the CLI where it pinned one (`1455192` for `b04836d`, `102a476` for `bd9d9f4`), core with a patch where it didn't | `HEAD` at the time of the format-4 pull request is the last row | + +### Regenerating + +`cargo xtask ir-fixtures --format N` checks the recorded revision out into a worktree outside the repository, builds +it with its lockfile and `--locked`, and runs each script. Core's only non-determinism is the author (`USER`, or +`whoami`) and the timestamp (`SystemTime`), so the generator sets the first and runs under a fake clock +(`faketime`, on Linux). A store regenerated that way is **byte for byte** the committed one, and `--check` says so. +The committed bytes are the fixtures; regeneration is what shows they are what that core made. + +### Building the old revs + +I haven't built anything for this proposal. What can be read from Git: + +- **Core's CI was green at the first merge of each format:** `d2763cd` on 2026-10-11 (format 1), `59587ba` on + 2026-10-12 (format 2), `b04836d` on 2026-10-18 (format 3). The commits inside them have no runs of their own. +- **Core had no `Cargo.lock` at any of them** (it is a library; the file is ignored), and `rust-toolchain.toml` says + `stable`. A build today resolves every caret dependency to its newest release. It will probably build, and its + bytes depend on `ciborium`, `lz4_flex` and `blake3`, so a fixture needs a lockfile. Core-only revisions (format 1, + and the ledger rows no CLI pinned) get one generated once and committed in `PRODUCED_BY`. +- **The CLI revisions have a committed `Cargo.lock`,** no `rust-toolchain.toml`, and name four git dependencies (core, + generation, comline-rust, comline-typescript) at revisions that **all still exist** (checked for `f934a66`, + `086cd32`, `00ee242` and `1455192`). With `--locked` they are the best-reproducible route, on the stable toolchain + from that revision's date. +- **What can still go wrong:** a yanked crate, a rewritten history, a newer toolchain rejecting old code. None is + known. + +So the first thing that stage 1 does is run it, once, and record the result. A weekly job (`old-revs.yml`, plus a manual +trigger) rebuilds every `PRODUCED_BY` revision, so that a vanished revision or a yanked crate shows within a week and +not when a fixture is needed. The committed stores don't depend on it. It protects regeneration, not the tests. + +## Differential property tests + +Fixtures are fixed histories. The differential test draws random ones. + +- **The generator** is `core/oracle`, extended: `Spec` already renders a package graph to `config.idp` and `.ids` + files and has no dependency on core. Add **histories**: a sequence of edits (add a field, swap a type, add or + remove a dependency, change a handshake or a bound, release), with proptest strategies for the property tests + and `Arbitrary` for the fuzzer, as for the guarantee oracle. +- **The old side runs out of process.** The CLI (or maker) built at the pinned revision of each format is run over the + rendered package, and its `.comline/` is read back. The binary is built once per `(revision, lockfile, toolchain)` + and cached in CI. The only exchange is the file system. +- **The current side runs in the test:** the current core builds the same history, `migrate` runs on the old store, and + the two are compared as [above](#what-is-compared). +- **Budget:** 256 cases by default, like the oracle test, and a few per PR if that is too slow to run on each; more + nightly (`PROPTEST_CASES`). A failure shrinks to the shortest history that disagrees, and the test writes it as a new + corpus entry. + +### An old core as a renamed dependency + +The brief asks whether the old core can be a renamed dev-dependency, `comline-core-v3 = { package = "comline-core", +git = …, rev = … }`, so the test calls it in process. Cargo allows two sources of one package under different names. +**It is feasible, and I'd not use it**, for these reasons, none of which has been tried: + +- Both parsers are linked into one test binary. If a generated grammar exports C symbols that don't depend on the + revision, they collide at link time. Core's `Cargo.toml` mentions `tree-sitter-c2rust`, a pure-Rust variant, so this may not + occur, but it needs trying. +- Core has no `links` key, but its dependencies might, and the resolver unifies versions and features across both + copies. Old revisions have no lockfile, so they resolve against today's, which is the problem fixtures solve. +- Every old rev wants its own toolchain, and `HEAD` pins `1.99.1`. +- The harness is per revision: the build entry points changed between them (`process_changes`, `build_with`, the + `deps` feature). +- The test links one more copy of core per format, forever. + +Out of process costs a process spawn per version and removes all of it. If the in-process route does work for the +latest format, it can replace the binary for that one. + +### A fuzz target + +`store_migrate` in `core/fuzz`, in the nightly matrix beside `unverified_oracle`: arbitrary bytes go through each +format's codec and the chain. It must never panic; it may return an error only when the bytes don't decode; and when +they do, the result must encode, decode, and satisfy the witness. A second form draws values of the frozen types +with `Arbitrary`, which covers what no real build produces, and is how totality is tested over the whole format. + +## Laws + +| Law | Statement | Tested by | +|---|---|---| +| **Totality** | `up` returns a value for every decodable `Version_N`, and it encodes | the `Arbitrary` form of the fuzz target and a proptest over frozen N types, not only reachable ones; every fixture | +| **Determinism** | the same input gives the same bytes on any platform, thread count and version order | run twice and compare; migrate the fixtures on Linux, macOS and Windows and compare digests with committed golden ones; shuffle version order and threads | +| **Information preserved** | `down(up(v)) = v`: no step dropped anything | the witness, over fixtures and random values, and on every real store at [run time](running.md#verifying) | +| **Idempotence** | migrating a migrated store changes nothing; an interrupted migration, resumed, gives the same files | run twice; fault injection that crashes after each write and each rename, then resumes | +| **Composition** | `migrate(N→K) = migrate(M→K) ∘ migrate(N→M)`, and equals `forget` over a fresh build | for each `N < M < K`: migrate to M, write it, continue; compare with straight through, and with the square | +| **Hashes carried** | a payload with the same bytes keeps its hash; each protocol's `shape_hash` and `param_hash`, its wire numbers, and every `version` and `release` are the same after | compare through the old-to-new map | +| **Convergence** | two independent migrations of one store give the same files | different thread counts and order, compared byte for byte | +| **Pins** | a dependency's pin made by the old core still verifies through the witness, or the format change declares that it doesn't | golden pins per format, [below](hashes-and-history.md#pins) | + +## The gate + +A format change that has no step, no fixtures or no document can't merge. It is a set of checks, most of them +running on every pull request because the stores are committed: + +| Check | Fails when | +|---|---| +| **Seal** | a sealed format's directory differs from `formats/MANIFEST`, so a frozen type, a codec or a schema document changed after its format was superseded | +| **`live ≡ frozen(current)`** | the core's types and the frozen copy of the current format encode differently, or can't decode each other's, for a fixture or a random value | +| **Golden fresh build** | a fresh build of each fixture source has different tree hashes from `expected/`, unless the pull request adds a [ledger](formats.md#each-format-is-written-down) row (and the frozen copy) or bumps | +| **Bump check** (`xtask check-format-bump`) | `CURRENT_IR_FORMAT_VERSION` changed, and the pull request lacks any of: `formats/v/` with its four files; a step `C-1 → C` with `unknown`, `up` and `down`; fixtures for every ledger row of `C-1`; the previous format's seal in the manifest; a row in the CAS guide's stamp table | +| **The square, the laws** | any of the above tests fails. A step with no tests can't be there: the bump check requires the fixtures, and the square runs on them | +| **Pin gate** | a golden pin of a fixture dependency moved, and the pull request doesn't declare it ([hashes](hashes-and-history.md#pins)) | +| **Spec** | the vendored spec files differ from `tests/spec/REV`: the existing check, because a step's `forget` reads the catalog | + +What it catches: + +| The mistake | Caught by | +|---|---| +| A field renamed or retyped without a bump | the fixtures of the current format stop decoding; `live ≡ frozen` | +| A field added without `skip_serializing_if` | the golden fresh build moves for a package that doesn't use it, so a row (and a pin note) is required | +| A bump without a step, or a step without fixtures | the bump check | +| A wrong, invented or needlessly unknown value | the square | +| A field dropped | the witness | +| A `HashMap`, a clock or a thread in a step | determinism and convergence | +| An edit to a frozen format | the seal | + +The seal and the bump check should be under CODEOWNERS, and a seal can be moved only by a command that names why, so that +moving one is a deliberate edit and not a side effect. Weekly: the old revisions build and `--check` passes. Nightly: +the differential test at a larger budget and the fuzz target. + +## What this proves, and what it doesn't + +It proves that on the corpus and on generated histories, each step does exactly what its `unknown` list says: it +derives what N knew, and nothing else, and it loses nothing. It does not prove it for a store the corpus doesn't +resemble. + +The trust base is small and named: the frozen types (each fixture is decoded and re-encoded to the same bytes, which +checks the copies against what real cores wrote), the `unknown` lists a reviewer reads, the old cores built from +their revisions, and the generator's reach. The ledger's granularity is the corpus's: a store written between two +rows by an unreleased build is in no fixture, and a row is added when one is found. `comline store verify` runs +the structural checks on the store of anyone who runs it, which is how a store the corpus didn't predict gets found. diff --git a/docs/docs/proposals/ir-migrations/running.md b/docs/docs/proposals/ir-migrations/running.md new file mode 100644 index 0000000..0ad7e12 --- /dev/null +++ b/docs/docs/proposals/ir-migrations/running.md @@ -0,0 +1,171 @@ +# Running a migration + +Status: **draft** · part of [Migrating stored IR across format versions](index.md) + +**The question.** How does a store change format without any moment at which history is lost, however the process +ends, and how does someone find out what it would do first? + +## What each command does with each stamp + +`ensure_compatible_store` stops deleting. With the stamp *N* and a binary that writes *C*: + +| The store | A read command (`diff`, `doc`, `generate`, `inspect`) | A write command (`build`, `release`) | +|---|---|---| +| N = C | proceeds | proceeds | +| N < C, and N is supported | reads through an **in-memory** migration and writes nothing | migrates, then proceeds | +| N < C, N unsupported, or unstamped | an error that names the format and the [options](steps.md#what-cannot-be-migrated) | the same error | +| **N > C** | an error: the store is format N, this comline reads up to C, and *comline ≥ X* reads N | the same error; nothing written | +| a stamp that isn't a number, or has conflict markers on one side | an error naming the file ([see Git](#git)) | the same | + +Reading an old store needs no write because a migration is a pure function: a read command can run the chain on the +versions it needs and drop the result. That makes `comline diff`, `doc --version` and `generate` work on a store nobody +has migrated yet, at the cost of the memory and time of the chain, which a cache can hide. + +Two of today's behaviors go: deleting an older store, and deleting a newer one. The second is the more dangerous. +`stored != Some(CURRENT)` is true for a store that is ahead of the binary. + +## The procedure + +`comline store migrate`, and the same code behind `build` and `release`: + +1. **Claim the store.** `mkdir .comline.migrating/` beside `.comline/`. It fails if the directory exists, which is the + lock. If it exists from a crash, see [below](#crashes). The migrating directory is on the same filesystem as + `.comline/` so that the renames in step 5 are atomic. +2. **Plan, in memory.** Read the container: the refs, then every commit reachable from them by parents, second parents + included, each object checked against its hash. Decode each version under N, run the chain, and encode the result + to bytes, writing nothing. The plan lists the versions, the steps, the objects that change, and every unknown with its + path and reason. `--check` stops here, after the [checks](#verifying) that don't need the disk. +3. **Write the new store** into `.comline.migrating/`: `objects/` (payloads that didn't change are linked or copied under + the same hash; everything else is new), `refs/heads/main` with each head mapped, and `FORMAT_VERSION` **last**. A + `JOURNAL` there records the old head hashes, the stamp and assumed format, and the state: `writing`, then `ready`. + `deps-cache/` is not read or written: it isn't part of the store. +4. **Verify** the new directory as if it were a store, [below](#verifying). A failure keeps the directory, marks it + `FAILED` with the reason, leaves `.comline/` as it was, and exits nonzero. +5. **Swap.** Set the journal `ready`. Rename `.comline/` to `.comline.format-N.bak/`. Rename `.comline.migrating/` to + `.comline/`. Move `deps-cache/` from the backup into the new `.comline/`: offline machines depend on it, and today's + wipe deletes it with everything else. Write the receipt and remove the journal. + +Each rename is atomic. Between the two there is no `.comline/`, and that state is recognized and completed, not +mistaken for "never built". + +### Crashes + +The state is in the paths and the journal alone, so every step is resumable by the next run of any command that opens +the store: + +| Interrupted | What is on disk | The next run | +|---|---|---| +| steps 1 to 3 | `.comline/` intact, `.comline.migrating/` partial, journal `writing` | If the journal's old heads equal the store's, it resumes: writes are content-addressed, so repeating one writes the same bytes. Otherwise the store changed meanwhile, so the partial directory is deleted and it starts again. | +| after verify, before the first rename | journal `ready` | completes step 5 | +| between the two renames | no `.comline/`, a backup and a ready directory | renames the ready directory into place | +| after both, before `deps-cache/` moved | new store in place, the cache in the backup | moves it | +| after | complete | nothing to do | + +Running the whole command again on a migrated store is a no-op: the stamp is C. That is the idempotence of the +operation. The idempotence of the *function* is a [law](proof.md#laws). + +### Verifying + +Before the swap, on the new directory, every one of these is checked, and any failure stops the migration: + +1. Every object decompresses, matches its hash, and every reference (tree entry, parent, `release.since`, ref) + resolves. +2. Every version decodes under C, and encoding it again gives the same bytes: the encoding is canonical. +3. The old-to-new map is a bijection on commits, parents map to parents, and for each commit the `version`, `release` + record, `author`, `timestamp` and `message` are the same as before. +4. The witness holds: `down` of every migrated version, through every step, equals the version as it was. + Nothing a version recorded was lost. See [the witness](steps.md#the-witness). +5. The unknowns written are exactly the ones the chain declares. A step that wrote an undeclared one, or didn't write a + declared one, fails here. +6. A payload whose bytes are the same in N and C kept its hash. + +Reported, not failed: the `facts` of each version recomputed by the current core and compared with the carried +ones. A difference means a rule changed since the version was built, and is listed so that it isn't a surprise +later. The existing `compat` tests treat a recomputed history the same way. + +## The backup + +The old store is renamed to `.comline.format-N.bak/` beside the new one. **Kept until someone removes it:** +`comline store prune` removes backups, and only when the store that replaced them passes `comline store verify`. +Nothing deletes one on its own, after a time or after a number of builds, and `build` mentions it once. + +It is a **sibling** and not a directory inside `.comline/` for two reasons. The shipped binaries delete `.comline/` +on a stamp they don't expect, so a backup inside would go with it. And `.comline/` is committed to Git while a sibling +shows in `git status` as untracked, so it is visible and nobody commits it without choosing to. `comline new` can add +it to the `.gitignore` it already writes for `deps-cache/`; the migration itself does not edit anyone's files. + +*Open:* the name and place, and whether disk use warrants a default expiry. Leaning: no expiry. A store is small, +objects the migration didn't change are hard links, and the backup is the only copy of the old format. + +## Checking first + +```text +comline store migrate --check +``` + +Does steps 1 to 2 and the in-memory part of the checks, and writes nothing: no `.comline.migrating/`, no lock. It prints +what would happen and exits with distinct codes for *nothing to do*, *would migrate* and *would be refused*, so it runs in CI +as a guard ("the checked-in store is current") and by hand before an upgrade. + +```text +store format 3, 41 versions (0.1.0-dev.1 … 0.9.0) + would migrate 3 -> 4: 41 versions, 118 objects rewritten, 1,204 unchanged + unknown after migration: 52 (chains: 52) + 0.7.0 protocols/Billing/given_up/bounded_size/chain format 3 records the set, not the path to its cause + … + nothing deleted: the old store is kept as .comline.format-3.bak +``` + +(An illustration of the content, not a format.) + +## The CLI + +Only the touchpoints are named here. + +| Command | Does | +|---|---| +| `comline store status` | The stamp, the versions, whether a migration is needed, the unknowns per version, the backups there are. | +| `comline store migrate [--check] [--assume-format N] [--map FILE]` | The procedure above. `--map` writes the old-to-new hashes, for anything outside the store that recorded one. | +| `comline store verify` | The [checks](#verifying) on the current store. Also what `core#182` asks for: a damaged store says which file, and what to do. | +| `comline store prune` | Removes verified backups. | +| `comline store replay` | *Proposed:* [rebuild from Git](steps.md#what-cannot-be-migrated). | +| `comline build`, `comline release` | Migrate first when the store is older, and say so. | + +`build` deleted the store without asking; migrating it without asking is strictly safer, and the plan is printed. *Open:* +whether the default should instead be to stop and ask. A store is committed to Git, so a migration is a large diff the +author didn't write, which is an argument for asking once. `--no-migrate`, or `COMLINE_STORE_MIGRATE=never`, makes an +old store an error for the machine that must not write. + +## Git + +`.comline/` is committed, so a migration is a change in a repository, and two people can make it. Three properties make +that safe, and one is open. + +- **Convergence.** A step is deterministic and the engine hashes bottom-up, so migrating the same old commit gives the + same new hashes in every clone. Two developers who migrate separately produce the same files, and Git merges them + without a conflict. This is why the receipt (`.comline/MIGRATIONS`, one line per migration) holds no time, no + author and no core version: the same on every machine, so it never conflicts either. +- **The objects merge by name.** New hashes are new files. The only files two sides can conflict on are + `FORMAT_VERSION` and `refs/heads/main`. +- **A branch that didn't migrate** brings old-format objects and its own head. `store migrate` accepts a conflicted + stamp whose sides are both supported formats and a conflicted ref (`Heads::Two`), migrates the older side's chain, + which yields the same objects the other side already has for the commits they share, and leaves the ordinary + [reconcile](../../guides/ir/cas.md#branches) to join the heads, as it does today. + +*Open:* the conflicted stamp is parsed by hand, and what a developer sees when the two sides disagree is a conflict in +a one-line file. The error needs to say what to do. Whether the engine should resolve it itself is not settled. + +### Older binaries + +A binary that predates this proposal deletes any store whose stamp isn't its own, a newer one included. Nothing in +the store can stop that, so the order of releases does: + +1. The pull request that makes a newer store an error ships in a **release first**, and the docs and the CLI say that + every machine that touches a repository has to have it before format 4 lands. +2. Even then, a binary older than the release deletes a format-4 store. Git has the history, and the backup, being a + sibling, survives it. + +*Open:* the stronger fix is to put the format-4 store somewhere an older binary doesn't look, such as `.comline/` for +format 3 and a new directory name for 4. An older binary then finds no store and starts a new one without touching the +new. It costs a rename across the docs, the CLI, the language server's exclusion list and `.gitignore`. It might be the +price of making this once and for all safe, and it can only be done at a format boundary. diff --git a/docs/docs/proposals/ir-migrations/steps.md b/docs/docs/proposals/ir-migrations/steps.md new file mode 100644 index 0000000..7ad5af1 --- /dev/null +++ b/docs/docs/proposals/ir-migrations/steps.md @@ -0,0 +1,171 @@ +# Steps + +Status: **draft** · part of [Migrating stored IR across format versions](index.md) + +**The question.** What is a migration made of, so that it is accurate for every old format, never loses what a +version recorded, and never claims to know what the old format did not? + +## The model + +A migration moves a whole store from format N to the current format C, one **step** at a time. Steps never touch +bytes or hashes. They work on a **model** of one version, which the format's codec decodes from the store and +encodes back: + +```text +bytes ──decode_N──▶ Version_N ──step N→N+1──▶ Version_N+1 ──…──▶ Version_C ──encode_C──▶ bytes + (frozen types of N) (frozen types of C) +``` + +A `Version_N` holds the commit's own fields (`version`, `release`, `author`, `timestamp`, `message`), the decoded +payloads (schemas, vendored dependencies, `config`, `docs`, `facts`) and the `unknown` list described +[below](#unknown). A reference to another version, a parent or a `release.since`, is a **version id** (its place in +the chain), not a hash. The engine turns ids back into hashes when it encodes, so a step author never handles a +hash and can't break the Merkle structure. See [hashes](hashes-and-history.md#object-hashes). + +The intermediate formats never reach the disk. A store goes from N to C in memory, version by version, and is written +once. + +## A step + +A step `N → N+1` is three things, written against the frozen types of N and N+1 +([formats](formats.md#each-format-is-written-down)), never against today's types: + +| Part | What it is | +|---|---| +| `up` | A function from a `Version_N` to a `Version_N+1`. **Total:** it returns a value for every value the format's types can express, not only for what a real build produces. **Deterministic:** the same input, the same output, on every platform. It has no I/O, clock, environment, randomness, thread or hash-map order. Every map in the model is ordered. | +| `down` | The **witness**: a function from the image of `up` back to `Version_N`, with `down(up(v)) = v` exactly. See [the witness](#the-witness). | +| `unknown` | The list of paths in `Version_N+1` that `up` writes as unknown because N never recorded them, each with the reason. Written down in the format's `SCHEMA.md` and read by the tests. | + +A step may read only the version it is given and, for facts that live in history, the versions before it in the same +chain. It may not recompute what the *current* core would compute (a guarantee set, a wire number, a shape hash): that +is code that changes, and a step is frozen. What a version recorded is carried across as data, and what has to be +recomputed is recomputed on read, by whichever core is reading. For example the `facts` blob is a record of what +the build computed under the rules in force; a step carries it, and `history_of_commit` already recomputes it when it +is absent or unnumbered. + +!!! note "Steps may not delete user data" + `up` keeps every field of N. A field that N+1 renamed or retyped is mapped. A field that N+1 has no place for is + kept in the `unknown` list with its original value (see below). The witness checks this. + +## Unknown + +Where N+1 needs something N never recorded, `up` writes an **unknown**. It never writes a default that looks +complete: a framing the author never chose, a length `T[]` for an array whose length was lost, an empty set of +causes for a guarantee whose causes weren't stored. This is the rule +[core#251](https://github.com/ComlineProject/core/issues/251) set for dependency configs ("read as unknown, never as +defaults that look complete") and the [vocabulary's](../shared-vocabulary/granular-unverified.md#fail-closed) for a +dependency that can't be seen, applied to a whole version. + +An unknown has two possible carriers. **Proposed: both**, in this order of preference. + +1. **Typed, in the new format.** Where N+1's own type has room for it, the field is `Known`: a value, or + `Unknown { reason }`. The compiler then forces every reader to handle it. Format 4's chains are the example + ([format 4](format-4.md#the-step)). A format that adds a field is written so that anything a later step might have + to leave unknown is already a `Known`. +2. **An `unknown` entry in the root tree,** for a retype whose new type has no room: a blob listing, for each + unknown, its `path` into the model, the `step` that wrote it, the `reason`, and the `original`, the bytes N + recorded, where N recorded something. The field itself holds a placeholder that the entry marks as one. This is + what lets a 2→3 step exist at all: format 3's `TypeRef` has no "unknown" variant and can't gain one without a + bump, while the entry keeps the old `float` text beside a placeholder. + +`unknown` is an additive entry, a row in format 3's ledger with the absence reading *none*. A version without it +has no unknowns. + +**Ordering constraint.** A shipped format-3 binary would read a store migrated into format 3, with unknowns in the +entry, as complete. So the readers of `unknown` ship, in a release, before any step that writes one. See +[the rollout](index.md#rollout). + +### Reading an unknown + +Consumers don't see the entry or the placeholder. The accessors on a version return `Known`, with no `Deref` and +no default, so an unknown can't be forgotten. Every use then has a rule, and it is the strictest reading: + +| A consumer needs | The field is unknown | Result | +|---|---|---| +| a protocol's given-up guarantees | the set or a cause | counts as given up; the build that depends on it refuses | +| a protocol's framing or format | the declaration | the strictest facts, as `Encodings` does when `complete` is false (core#262) | +| a bound | the maximum | unbounded | +| a type, compared | its width or length | breaking, or "can't tell" if the rule has that | +| a span | where it was written | no span; diagnostics name the version instead | +| generating code or documentation | anything the output prints | refused or marked, [see format 4](format-4.md#consumers) | + +A comparison that read an unknown says so: it lists the version and the path in the same `unseen` list core#262 +added, and its bump is at least a patch. See [history](hashes-and-history.md#history). + +Some things are never unknown, because without them a version isn't one: its `version`, its `release` record, its +parents, and the schema units' names. A step that can't produce them fails, and the migration refuses. + +## The witness + +`down(up(v)) = v` is the proof, on every real version, that a step lost nothing. It is cheap to write for the steps +that are likely: one that wraps a value, renames a field or adds a field with an unknown has a trivial inverse, and a +step that moves a value into the `unknown` list keeps the original there so `down` can put it back. + +The engine runs the witness over every version of a store it migrates, before it swaps anything +([running](running.md#verifying)). The tests run it over fixtures and random values ([proof](proof.md#laws)). The +witness also has a second use, for [pins](hashes-and-history.md#pins). + +*Open:* whether every step has to provide one, or only the ones that can lose something. Leaning: every step, since +a step that can't state its inverse is exactly the one to look at. + +## Chains + +`migrate(N → C) = up(C-1→C) ∘ … ∘ up(N→N+1)`. A later step may resolve an unknown an earlier one wrote, only from +data in the version. The unknowns of a chain are the union of its steps' lists minus what a later step resolved, +and the engine writes that list. + +The engine has one chain per store and one version at a time. It does not support skipping steps: a "shortcut" from +1 to 3 is a second implementation of the same function, and the composition law ([proof](proof.md#laws)) exists to +catch the two disagreeing. + +## What cannot be migrated + +| Store | Why | What happens | +|---|---|---| +| **Format 0**, unstamped bincode | Positional: a bincode decode of the wrong layout can succeed and give a different variant, so the data can't be told from garbage. About 20 layouts existed in nine months and none was marked. A commit's timestamp says when it was built, not which core built it. Nothing in the store, whose hashes cover bytes and not meaning, can check a decode against. | Refused. The store is kept. The error names the options below. | +| An unstamped store that decodes as CBOR | The stamp was lost, so the format isn't known. | Refused, unless the user gives one: `--assume-format N`. It is recorded in the receipt, and the decode must succeed for every object. | +| A damaged store (a missing or corrupt object, [#182](https://github.com/ComlineProject/core/issues/182)) | A migration needs every object. | Refused, naming the file and how it failed. | +| A store whose stamp says N but whose objects don't decode as N | A binary that predates the stamp, or a half-finished edit. | Refused, naming the first object. | + +For a store that can't be migrated, the options are the ones that exist already, plus one: + +- **Keep it and start a new history,** `comline reset`'s result without the deletion: the old store is renamed + aside, as a backup is. +- **`comline store replay`** *(proposed; may not be worth building)*. Rebuilds a history from the project's Git + history with the current core, one build per release tag or commit chosen by the user. The result is a **new** + history: it never claims to be the old one, its versions are numbered by the current rules, and the command says + so. It is the way back for format 0 and for any store the framework refuses. + +## Support + +| Format | Migrates to | Lossy? | Fixtures | +|---|---|---|---| +| 3 | 4 | the chains and scopes format 4 records ([format 4](format-4.md)), and whatever a version's ledger level lacks | one per [ledger](formats.md#format-3-is-a-family) row | +| 2 | 3 | yes: float width, array length, written form of a value | built by the CLI pinned at `00ee242` | +| 1 | 2 | yes: the written form of an annotation's value | built from core's `d2763cd`, with a small patch | +| 0 | nothing | n/a | one, to test that it is *refused* | + +Steps 2→3 and 1→2 are the only real lossy steps that exist before format 4, and so they are where the unknown +machinery gets tested on real data. That is the reason to write them even though few stores are in these formats. +Format 1 existed for about fifteen hours on `master` and no released CLI built it. + +## Where the code lives + +**Proposed:** a workspace member `comline-ir-formats` (`publish = false` at first) beside `oracle/`, holding one module +per format with its `SCHEMA.md`, types, codec and ledger, and the steps between them. It has no dependency on +`comline-core`, the way the oracle has none, so that a frozen type can't reach for today's. A step `N → N+1` lives +beside the frozen N+1 types. Those exist from the day N+1 is introduced, and are kept equal to the live types by the +`live ≡ frozen(current)` test until N+2 seals them. + +Core depends on the crate to read old stores. The engine and the CLI commands are in core. The crate's only +dependencies are `serde`, `ciborium`, `blake3` and `lz4_flex`, so it can be in the IR-only build, which is the one +the `wasm32` consumers use. Whether it *should* be there, since the build's point is to be +small, is open. + +## Still open here + +- **Whether `Known` is also written in formats that already exist,** that is, whether the live types of format 3 + should gain a typed unknown in the next format instead of relying on the entry. Format 4 can, and should, for what + it adds. +- **A path syntax for the `unknown` entry.** Leaning: a JSON-pointer into the model, since the frozen types already + derive serde and the path then reads the same in the format's document and in a test. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 18fe4b0..4bf409a 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -229,6 +229,15 @@ nav: - 'proposals/shared-vocabulary/index.md' - 'The catalog': 'proposals/shared-vocabulary/catalog.md' - 'Granular @unverified': 'proposals/shared-vocabulary/granular-unverified.md' + - 'Migrating stored IR': + - 'proposals/ir-migrations/index.md' + - 'The formats': 'proposals/ir-migrations/formats.md' + - 'Steps': 'proposals/ir-migrations/steps.md' + - 'Running a migration': 'proposals/ir-migrations/running.md' + - 'Proving it': 'proposals/ir-migrations/proof.md' + - 'Hashes, pins and history': 'proposals/ir-migrations/hashes-and-history.md' + - 'Beyond the local store': 'proposals/ir-migrations/beyond-the-store.md' + - 'Format 4': 'proposals/ir-migrations/format-4.md' extra_css: - stylesheets/nav.css?v=23 # bump on every nav.css change — it is not cache-busted otherwise From b78d722c9ec3bf1d2cb7b3c4b4bcb35f9b178fda Mon Sep 17 00:00:00 2001 From: Kinflou Date: Fri, 13 Nov 2026 03:52:31 +0800 Subject: [PATCH 2/4] proposals/ir-migrations: scope to the migration system; today's formats reset at 1.0 Today's formats are out of scope and reset to a backup before 1.0, with the numbering restarting at format 0 at 1.0. Format 4 is the first real client; test-only formats prove composition and lossy steps. Drops the per-format fixture plans for formats 0 to 2 and the old-revs build job. --- .../ir-migrations/beyond-the-store.md | 40 +++++-- docs/docs/proposals/ir-migrations/format-4.md | 16 ++- docs/docs/proposals/ir-migrations/formats.md | 34 +++--- .../ir-migrations/hashes-and-history.md | 5 +- docs/docs/proposals/ir-migrations/index.md | 101 +++++++++++------- docs/docs/proposals/ir-migrations/proof.md | 58 +++++----- docs/docs/proposals/ir-migrations/running.md | 8 +- docs/docs/proposals/ir-migrations/steps.md | 52 ++++----- 8 files changed, 177 insertions(+), 137 deletions(-) diff --git a/docs/docs/proposals/ir-migrations/beyond-the-store.md b/docs/docs/proposals/ir-migrations/beyond-the-store.md index 5c8fb2b..ae043c6 100644 --- a/docs/docs/proposals/ir-migrations/beyond-the-store.md +++ b/docs/docs/proposals/ir-migrations/beyond-the-store.md @@ -9,7 +9,7 @@ does each of the others need, and what does a published package need when there | Artifact | Holds | Migration story | |---|---|---| -| **`.comline/` store** (`objects/`, `refs/`, `FORMAT_VERSION`) | every version, as frozen IR | [Migrated](running.md), whole, with the old one kept | +| **`.comline/` store** (`objects/`, `refs/`, `FORMAT_VERSION`) | every version, as frozen IR | [Migrated](running.md), whole, with the old one kept; before 1.0, [reset into a backup](#the-10-reset) where no step exists | | **`.comline/deps-cache/`** | git checkouts of dependencies: source, not IR | Not migrated, not part of the store. **Carried across the swap**: today's wipe deletes it, and an offline machine can't fetch it again | | **A dependency's own committed `.comline/`** | its version history; the only thing core reads of it is the newest commit's `version` (`resolved_version`) | Read with the format-independent [container reader](formats.md#what-a-store-holds). Today any failure reads as `0.0.0-unbuilt`; it should say the store couldn't be read | | **Dependency pins** in `config.idp` | scheme v2 hash over IR bytes | Never rewritten. [Bridged](hashes-and-history.md#pins) | @@ -44,6 +44,29 @@ keyed by it. The rules for IR follow from that. - **A pin never depends on which format an archive's store is in.** That is the requirement the [pin bridge](hashes-and-history.md#pins) meets. +## The 1.0 reset + +**Decided:** the formats that exist today are all reset when 1.0 arrives, and the numbering restarts: the 1.0 format is +**format 0** of a new numbering, and the first with a guaranteed path forward. Nothing before it is migrated. + +**Proposed:** it is one release, so that there is one moment and one rule. + +1. The release finds a pre-1.0 store, in any format, and **resets it into a backup**: `.comline.format-N.bak/`, the + same mechanism as [any reset](running.md#what-each-command-does-with-each-stamp), never a deletion. `deps-cache/` is + kept. A new store starts, and stamps the 1.0 format 0. +2. It is told apart from the old numbers: pre-1.0 stamps are the integers 0 to 4, and a 1.0 `0` or `3` must not be read + as one of them. So the 1.0 stamp carries an **epoch** (`FORMAT_VERSION` reads, say, `1.0/0`), and a bare integer + is by definition pre-1.0. The new directory name that [running](running.md#older-binaries) floats would do the + same job. *Open.* +3. The frozen types of the pre-1.0 formats are **not** kept: the reset is the only thing that touches them, and it + needs the stamp and the container, which are format-independent ([formats](formats.md#what-a-store-holds)). +4. Every format change after it is a step, with a document, fixtures and a seal, as the gate requires. The system is + already in use by then: format 4's step is its first run. + +*Open:* the release's shape. One release that only resets, so that its behaviour is easy to state and to find, or the +1.0 release itself. Leaning: the 1.0 release resets, and the release before it warns, once, that a reset is coming and +names the backup. + ## After 1.0 [core#146](https://github.com/ComlineProject/core/issues/146) asks that from 1.0 a newer comline reads what an older one @@ -52,19 +75,17 @@ framework *is* the readers, and its fixtures and laws are the tests. | | Before 1.0 | From 1.0 | |---|---|---| -| **Formats that migrate** | the supported ones: 3, and 1 and 2 if they get built | every format released at or after 1.0, for ever. A released format's frozen types, steps, fixtures and seal are never removed | -| **A format below the floor** | refused, the store kept, with the options of [what cannot be migrated](steps.md#what-cannot-be-migrated) | the same, and the error names the **bridge release**: the last comline that still migrates it | -| **Moving the floor** | any minor release | a major one, announced one minor earlier; the bridge release stays installable | +| **Formats that migrate** | none promised: format 3 for the first client, the rest reset | every format released at or after 1.0, for ever. A released format's frozen types, steps, fixtures and seal are never removed | +| **A store with no path** | reset into a backup | refused, never reset, and the error names the **bridge release**: the last comline that still migrates it | +| **Dropping a format** | any minor release | a major one, announced one minor earlier; the bridge release stays installable | | **Cost of a bump** | a step and its fixtures | the same, **and permanent**: it is carried for ever | | **Additive changes** | a [ledger](formats.md#format-3-is-a-family) row each | a ledger row each, a fixture, and the same lines in the frozen copy | Because each bump is permanent from 1.0, bumps are worth batching: the policy proposed is that a format bump is planned, not incidental. Cheap changes are additive ones, and the gate makes those cheap. -A *version* is not a format. Package version 0.3.0, built long ago in a format still on the support list, is migrated -like every version of its store: nothing is dropped, and it can be compared at the precision it recorded, nothing -finer. A version in a store below the floor is one the store still has, in the backup, and one `comline store replay` -can rebuild from Git. +A *version* is not a format. Package version 0.3.0, built long ago in a format that still migrates, is migrated like +every version of its store: nothing is dropped, and it can be compared at the precision it recorded, nothing finer. The spec side needs nothing of this: the spec is append-only for anything frozen ([the vocabulary](../shared-vocabulary/catalog.md#versioning)), so an old version reads under a newer spec without a step. @@ -72,7 +93,4 @@ Only the IR format does. ## Still open here -- **The floor now.** Whether formats 1 and 2 are worth building: they are the only lossy steps that exist before format - 4, and they exercise the unknown machinery on real data, but format 1 existed fifteen hours and no CLI built it. - Leaning: build 2→3, which a released CLI did build for six days, and decide 1→2 when the engine exists. - **How long a bridge release stays supported,** after 1.0. diff --git a/docs/docs/proposals/ir-migrations/format-4.md b/docs/docs/proposals/ir-migrations/format-4.md index f241082..4816e91 100644 --- a/docs/docs/proposals/ir-migrations/format-4.md +++ b/docs/docs/proposals/ir-migrations/format-4.md @@ -9,6 +9,13 @@ do with every format-3 version already stored, and what is the package author as Format 4 is not designed here. It is the shape that proposal accepted, and this page is what the step does with it. Anything that depends on the final shape is marked. +**It is the first real client of the system,** and exists to prove it before 1.0, because today's formats won't be +migrated ([scope](index.md#scope-the-system-not-the-history)). One pull request exercises, on a real change: the frozen +copy and ledger, a step with `up`, `down` and an `unknown` list, a typed unknown, the square against a fresh format-4 +build, the laws, the pin bridge, and the gate. The test-only formats of [the proof](proof.md#test-only-formats) cover +what one step can't (composition, an untyped unknown); the two together are the system's proof. A format-3 store that +exists when the framework ships is migrated for real, and one older than that is reset. + ## What changes Format 3 freezes what core worked out as a **set**, and where it was written: @@ -51,7 +58,7 @@ the current spec ([history](hashes-and-history.md#history)). So: - `diff` and the changelog **recompute chains for both versions** and compare those. A migrated version compared with an unchanged successor shows no change, even though its stored chain is unknown. - The chains stay unknown only where an input is missing: a dependency's config in a version older than #262, a type - whose width format 2 lost. Those are `unseen`, as everything else that can't be seen is + whose information an older format never kept. Those are `unseen`, as everything else that can't be seen is ([history](hashes-and-history.md#what-a-comparison-does)). - "Narrowing a marker isn't a major change; widening one or adding a cause is" is a rule over *recomputed* causes for these versions, so it needs no stored chain to hold. @@ -92,9 +99,8 @@ What reads a version with unknown chains, and what each does: ## Fixtures and tests for 3→4 -The corpus ([proof](proof.md#the-fixture-corpus)) needs, for this step: the chain graphs of the oracle built by the cores -of each format-3 ledger row from `bd9d9f4` on, with the inner scopes (`66feecb`) and the package-scope `unverified` -(`9ea142e`) in the later rows. The square is then: the format-3 store of a graph, migrated, against the format-4 build of +The corpus ([proof](proof.md#the-fixture-corpus)) needs, for this step: the chain graphs of the oracle, with the inner scopes +(`66feecb`) and the package-scope `unverified` (`9ea142e`), built by format 3's closing core. The square is then: the format-3 store of a graph, migrated, against the format-4 build of the same graph with chains and package pins forgotten. The differential test draws random graphs and histories from the oracle's generators, the same ones that test the invariant, so the property that no guarantee goes -unacknowledged is checked over migrated stores too (stage 3 of the rollout). +unacknowledged is checked over migrated stores too (the cross-package chains part of stage 2 of the rollout). diff --git a/docs/docs/proposals/ir-migrations/formats.md b/docs/docs/proposals/ir-migrations/formats.md index 1ba0b90..5aa1a0c 100644 --- a/docs/docs/proposals/ir-migrations/formats.md +++ b/docs/docs/proposals/ir-migrations/formats.md @@ -20,7 +20,7 @@ A version's root tree has one `schema_` tree per schema file, a `dep_` [CAS guide](../../guides/ir/cas.md) describes this for format 3. **Proposed: the container is not a format.** The object layout (CBOR, LZ4, blake3, `Blob`, `Tree`, `Commit`, the -ref file) is the same in formats 1 to 3, and a format version covers the payloads inside it. Three things follow: +ref file) is the same in every CBOR format so far, and a format version covers the payloads inside it. Three things follow: - A reader of the container needs no format knowledge. The CLI and core can list versions, read each commit's `version` and `release`, and walk parents in any store they can find the container of. Today @@ -33,30 +33,26 @@ ref file) is the same in formats 1 to 3, and a format version covers the payload ## History of the formats +Background only: these formats will be reset at 1.0 ([scope](index.md#scope-the-system-not-the-history)), and are +listed because they are why a migration system is needed. + | Format | Stamp | On `master` | What it is | |---|---|---|---| -| **0** | none | 2026-01-10 (`534da82`, the CAS), config blob 2026-08-30, dependencies vendored 2026-09-30, until 2026-10-11 | bincode, which tags struct fields and enum variants by position, no names on the wire. About 20 commits reshaped the persisted types in this period, none of them marked. | -| **1** | `1` | 2026-10-11 (`f71ef96`, PR #84) | Name-tagged CBOR. `ensure_compatible_store` and the stamp are introduced. `FrozenUnit::Parameter` removed. | -| **2** | `2` | 2026-10-12 (`dbdb13f`, PR #87) | `Property`'s `expression: Option`, the written value with its quotes stripped, becomes `value: ConfigValue`, which keeps the written form. A field renamed and retyped. | -| **3** | `3` | 2026-10-18 (`eda0684`, PR #108) | Types stop being `KindValue` text: `TypeRef` (`Primitive`, `Named`, `Array { element, length }`, `Union`, `Unit`), `Literal` and `Condition`. `Field`/`Constant` `kind_value` becomes `ty` and a literal; `EnumVariant` holds its name; `Assert` gains `expr`. | - -Notes that matter to migration: - -- **Format 1 lived about 15 hours on `master`.** Core's CI was green at its first merge (`d2763cd`, 2026-10-11) and at - format 2's (`59587ba`, 2026-10-12). No CLI revision pinned core in format 1. -- **Format 2's lossiness is real.** `KindValue` text froze `T[8]` and `T[]` both as `T[]`, and `f32` and `f64` both - as `float`. A 2→3 step cannot recover either. Format 1's `expression` had no quotes: `@framing = "datagram"` and - `@framing = datagram` read the same, so a 1→2 step cannot say which form was written. -- **The CLI pinned format 0 until 2026-10-11 (`f934a66`, core `f035379`), format 2 from 2026-10-12 (`086cd32`) to - 2026-10-18 (`00ee242`), and format 3 from 2026-10-18 (`1455192`).** Those revisions have a committed - `Cargo.lock`, and the git revisions their `Cargo.toml` names (core, generation, comline-rust, comline-typescript) - all still exist. [The proof](proof.md#building-the-old-revs) uses - them. +| **0** | none | 2026-01-10 to 2026-10-11 | bincode, which tags fields and variants by position. About 20 commits reshaped the persisted types, none marked. | +| **1** | `1` | 2026-10-11 (`f71ef96`) | Name-tagged CBOR. The stamp and `ensure_compatible_store` are introduced. About 15 hours on `master`. | +| **2** | `2` | 2026-10-12 (`dbdb13f`) | `Property.expression: Option` becomes `value: ConfigValue`: a field renamed and retyped. | +| **3** | `3` | 2026-10-18 (`eda0684`) | `KindValue` text becomes `TypeRef`, `Literal` and `Condition`. | + +Two of these bumps lost information that the next format needed: format 2's text froze `f32` and `f64` as `float` and +`T[8]` as `T[]`, and format 1's `expression` had no quotes. Nothing could have recovered them, which is the case the +unknown machinery in [steps](steps.md#unknown) exists for, and the one the test-only formats in +[the proof](proof.md#test-only-formats) imitate. ## Format 3 is a family The doc comment on `CURRENT_IR_FORMAT_VERSION` says growing `FrozenUnit` needs no bump. Since 2026-10-18 these have -been added without one, and so a "format 3" store can lack any of them: +been added without one, and so a "format 3" store can lack any of them. The 3→4 step, the first client, takes format 3 +as it stands when the framework ships (the last row), and a store from before that is reset, like any pre-1.0 store: | Added | Commit | Absent reads as | |---|---|---| diff --git a/docs/docs/proposals/ir-migrations/hashes-and-history.md b/docs/docs/proposals/ir-migrations/hashes-and-history.md index ace3b1d..eaf1543 100644 --- a/docs/docs/proposals/ir-migrations/hashes-and-history.md +++ b/docs/docs/proposals/ir-migrations/hashes-and-history.md @@ -59,8 +59,7 @@ expressed in N**, and a pin is never accepted otherwise. dependency now records chains that it couldn't have covered. Review it and re-pin: `hash = "…"`". The command that writes a re-pin exists already as a request (`comline add --repin`, or the future `update`, core#261). 4. **`config.idp` is never rewritten** by a migration or by this check. -5. Cost is only on a mismatch, as `mismatch_message` is today, and only through the formats at or above the support - floor. +5. Cost is only on a mismatch, as `mismatch_message` is today, and only through the formats that still have steps. A package that doesn't use what format 4 added therefore keeps every pin through the bump, with no action, and the tests show it ([proof](proof.md#laws)): `golden-pins.json` holds each fixture dependency's pin as the *historical* core @@ -125,7 +124,7 @@ migration is not a rebuild, and a version that was released as `0.4.0` was. | A package migrated from format 3, source unchanged | Nothing found; see [the first build](#the-first-build-after-a-migration) | | The spec has since added a guarantee an old protocol gives up | Recomputed on both sides: no change, no false major | | A version from before dependency configs were kept (core#262) | `unseen: Dependency("acme")`, at least a patch: as now | -| A format-2 version with an array of unknown length, compared to today's | the path listed as unseen; possibly breaking | +| A version that went through a lossy step, with an array whose length was lost, compared to today's | the path listed as unseen; possibly breaking | | A version whose `release` says `0.4.0` | `0.4.0`, whatever the current rules would number it | ### The first build after a migration diff --git a/docs/docs/proposals/ir-migrations/index.md b/docs/docs/proposals/ir-migrations/index.md index 47ceb60..aa7a4a7 100644 --- a/docs/docs/proposals/ir-migrations/index.md +++ b/docs/docs/proposals/ir-migrations/index.md @@ -9,9 +9,26 @@ read in this order: [the formats](formats.md), [steps](steps.md), [running a mig **The question.** `.comline/` holds every version a package has had, and compatibility checks, the changelog and `comline doc --version` all read it. When core changes the shape of what it stores, `ensure_compatible_store` -deletes the directory, so the version history is lost with each bump. **Can every stored format, past and future, -be migrated to the current one, so that nobody can tell the result from a store the current core built, except -where the old format never recorded something, and so that this is shown by tests instead of asserted?** +deletes the directory, so the version history is lost with each bump. **Can the formats from 1.0 on be migrated, each +to the next, so that nobody can tell the result from a store the current core built, except where the old format never +recorded something, and so that this is shown by tests instead of asserted?** + +## Scope: the system, not the history + +**Decided (the user, 2026-11-13):** the formats that exist today will all be reset when 1.0 arrives, and the numbering +restarts at format 0. So this proposal prepares **the migration system and shows it is sound**: the model, frozen +per-format types, steps, the unknown and fail-closed machinery, atomic and idempotent running, the laws, the proof +harness and the CI gate. It does not promise to migrate today's formats (the unstamped bincode store, formats 1 and 2, +or the family that is format 3), and it is fine if one of them can't be migrated reliably. The 1.0 format is the first +with a guaranteed path forward. + +- **Before 1.0,** a store in an older format may be **reset**, but safely: moved to a backup and never deleted, a + store *newer* than the binary refused and never touched, and `deps-cache/` kept. See + [running](running.md#what-each-command-does-with-each-stamp). +- **The system is proven before 1.0,** because 1.0 must not be its first test. The 3→4 step for the chains of + [core#222](https://github.com/ComlineProject/core/issues/222) is the first real client, and test-only formats prove + composition and lossy steps. See [format 4](format-4.md) and [proof](proof.md). +- **The 1.0 reset** is one release. See [beyond the store](beyond-the-store.md#the-10-reset). ## What it looks like today @@ -19,14 +36,13 @@ where the old format never recorded something, and so that this is shown by test |---|---| | **The stamp** | `.comline/FORMAT_VERSION` holds `CURRENT_IR_FORMAT_VERSION` (3). `ensure_compatible_store` compares with `!=`: anything else, a missing stamp included, deletes all of `.comline/`, `deps-cache/` too. | | **Who deletes** | `comline build` does it with a warning in the log. The read commands (`diff`, `doc`, `generate`, `inspect`, `release`) refuse, and say that `build` "starts the version history over". An older binary deletes a store that is *newer* than it, because the check is `!=`. | -| **What a number means** | Four on-disk shapes have existed: an unstamped bincode store and formats 1, 2 and 3. Format 3 is not one shape: [twenty additions](formats.md#format-3-is-a-family) have landed in it without a bump. | +| **What a number means** | Four on-disk shapes have existed: an unstamped bincode store and formats 1, 2 and 3. Format 3 is not one shape: [twenty additions](formats.md#format-3-is-a-family) have landed in it without a bump, which is why the rule needs a check. | | **What the rule is** | Bump only for a rename, a removal or a retype; anything else is "additive". The rule is in a doc comment. Nothing checks it: the commit that introduced the stamp deferred a golden-hash check "if it becomes a real recurring problem". | | **Hashes that cover IR bytes** | Every object's address, a dependency pin (`hash = "blake3:…"`, scheme v2, [core#261](https://github.com/ComlineProject/core/pull/261)) and `schema_ir_hash`. A format change moves them. | | **Proof** | Round-trip tests over real packages, an oracle property test and fuzz targets for the guarantee check, differential tests in the runtime. None says anything about an old store. | -Format 3 is the first format with a population of stores worth saving, and the next change, format 4 -([core#222](https://github.com/ComlineProject/core/issues/222)), would wipe all of them. This proposal comes first -so that it doesn't. +The next change, format 4 ([core#222](https://github.com/ComlineProject/core/issues/222)), would wipe every store. +This proposal comes first so that it doesn't, and so that by 1.0 the machinery has already carried a real change. ## What this proposes @@ -40,15 +56,16 @@ so that it doesn't. 3. **[Running a migration](running.md).** The new store is written beside the old one, verified, and swapped in; the old one is kept. A migration can be checked without writing (`comline store migrate --check`), resumed, and repeated. A store newer than the binary is an error, never a deletion. -4. **[Proof](proof.md).** A corpus of real stores built by the real historical cores, an equivalence stated field by - field, differential property tests against an old core, laws (totality, determinism, idempotence, composition, - hash stability), and a CI gate that fails a format bump without its step, fixtures and document. +4. **[Proof](proof.md).** A corpus of real stores built by the real core of each format, an equivalence stated field by + field, differential property tests against the previous format's core, test-only formats for composition and lossy + steps, laws (totality, determinism, idempotence, composition, hash stability), and a CI gate that fails a format bump + without its step, fixtures and document. 5. **[Hashes, pins and history](hashes-and-history.md).** What may and may not change in a hash, how a dependency pin survives a format bump without being weakened, and how compatibility and the changelog read a migrated history. -6. **[Beyond the local store](beyond-the-store.md).** Every other place stored IR or its hashes appear, and the - policy for very old formats after 1.0. -7. **[Format 4 as the first client](format-4.md).** +6. **[Beyond the local store](beyond-the-store.md).** Every other place stored IR or its hashes appear, the 1.0 reset, + and the policy for old formats after 1.0. +7. **[Format 4 as the first client](format-4.md),** which proves the system end to end before 1.0. ## Rules that hold whatever the details @@ -59,7 +76,8 @@ so that it doesn't. - **What a version recorded is kept.** What its format never recorded is unknown, and unknown fails closed everywhere it is read: [#251](https://github.com/ComlineProject/core/issues/251)'s rule for a dependency's config, and the fail-closed rule of the [shared vocabulary](../shared-vocabulary/granular-unverified.md#fail-closed). -- **Nothing is replaced until it is verified,** and what is replaced is kept until someone removes it. +- **Nothing is replaced until it is verified,** and what is replaced is kept until someone removes it. A store that + can't be migrated is reset into a backup, never deleted. - **Accurate means tested against a fresh build.** A migrated version equals what the current core would have built from the same source, field for field, except for the fields the step declared unknown. - **A format change brings its step, its fixtures and its document in the same pull request,** or CI fails. @@ -78,33 +96,38 @@ so that it doesn't. | Pins | verified through the witness: a pin made under format N still verifies if the dependency's content is representable in N | | History | comparisons recompute derived facts from stored inputs under the current spec; what can't be recomputed is `unseen` | | Gate | `formats/MANIFEST`, `live ≡ frozen(current)`, fixtures, and a bump check; see [the gate](proof.md#the-gate) | -| Support | formats 1 to 3 migrate; the unstamped bincode store does not, and is refused with options | +| Today's formats | not migrated; reset to a backup at 1.0, and before it where a step doesn't exist | +| First client | the 3→4 step of #222's chains, end to end, before 1.0 | ## Rollout -Staged pull requests, in the agreed order: the framework, then format 4, then cross-package chains, then tooling. -Each stage is useful on its own, and each is a checklist in [core#268](https://github.com/ComlineProject/core/issues/268). +Staged pull requests: the framework, then format 4 as its first client, then the 1.0 reset. Each is a checklist in +[core#268](https://github.com/ComlineProject/core/issues/268). **1. The framework** (core, format stays 3): -1. *A newer store is an error, not a wipe.* Core and the CLI message. A small pull request, released **before** - anything ships format 4, so that no binary in circulation deletes a format-4 store. -2. *Format documents and the gate, with no migration yet.* A `formats` crate with the frozen format-3 types, the - ledger, the seal, the `live ≡ frozen(3)` test, and golden hashes for the fixture packages. This stops further - no-bump drift before it grows. It also adds, as ledger rows, the two entries nothing writes yet, `unknown` and the - spec version a version was frozen against, with readers that fail closed, and it ships in a release with step 1 - **before** any step or format that writes either one. -3. *The fixture corpus and its generator,* with format-3 stores built by the historical revs at each ledger row, a - format-2 and a format-0 store, and the weekly job that proves the old revs still build. -4. *The engine and `comline store`:* container reader, plan, journal, verification, swap, backup, `--check`, with - the laws proved on a test-only toy format before any real step exists. -5. *Steps 2→3 and 1→2,* each with its fixtures: the first real lossy steps, exercising the unknown machinery. - -**2. Format 4** (core): the new frozen shape of [core#222](https://github.com/ComlineProject/core/issues/222) item 3, -step 3→4, its fixtures and format document, and the pin bridge, all in one pull request because the gate requires it. - -**3. Cross-package chains** (core): the recomputation of chains over stored dependency IR, `unseen` extended to -chains, and the oracle and the fuzz target run over migrated stores. +1. *A newer store is an error, not a wipe, and an older one is reset into a backup, not deleted.* Core and CLI message. + A small pull request, released **before** anything ships format 4, so that no binary in circulation deletes a + format-4 store. +2. *Format documents and the gate.* A `formats` crate with the frozen format-3 types, the ledger, the seal, the + `live ≡ frozen(3)` test, and golden hashes for the fixture packages. This stops further no-bump drift before it + grows. It also adds, as ledger rows, the two entries nothing writes yet, `unknown` and the spec version a version was + frozen against, with readers that fail closed. It ships in a release with step 1 **before** any step or format that + writes either one. +3. *The engine and `comline store`:* container reader, plan, journal, verification, swap, backup, `--check`. +4. *The proof harness,* proved on **test-only formats** before any real step exists: composition (N→N+2), a lossy step + with unknowns, the witness, crash injection, the laws. And the fixture generator. + +**2. Format 4, the first client** (core): the new frozen shape of +[core#222](https://github.com/ComlineProject/core/issues/222) item 3, step 3→4, its fixtures (stores built by format +3's closing core) and format document, the pin bridge, and acknowledgements re-asked, all in one pull request because +the gate requires it. It runs the square, the laws and the gate on a real change. Then cross-package chains: +recomputation over stored dependency IR, `unseen` extended to chains, and the oracle and fuzz target over migrated +stores. + +**3. The 1.0 reset** (core, CLI): one release resets every pre-1.0 store into a backup and stamps format 0 of the +1.0 numbering. From then on every format change is a step. See +[the 1.0 reset](beyond-the-store.md#the-10-reset). **4. Tooling** (CLI, language server, generators, docs): @@ -113,7 +136,6 @@ chains, and the oracle and the fuzz target run over migrated stores. - The language server and `inspect` show `unseen`. - The guide on [the CAS](../../guides/ir/cas.md) loses its warning about deletion. - Generators learn what a version with unknowns means. -- `comline store replay` for stores that can't be migrated. ## Still open @@ -129,12 +151,11 @@ Collected from the pages; each is where it is discussed. ([running](running.md#git)). - Whether a pin that verifies through the witness is accepted silently, with a warning, or only by an explicit re-pin ([hashes and history](hashes-and-history.md#pins)). -- Whether to spend the effort on formats 1 and 2, and where the support floor is before and after 1.0 - ([beyond the store](beyond-the-store.md#after-10)). +- **How the 1.0 reset is carried out:** one release that resets with a backup and stamps the 1.0 format 0, and how + that stamp can't be confused with a pre-1.0 `0` to `4` + ([beyond the store](beyond-the-store.md#the-10-reset)). - Where the frozen formats live: a crate, a module, and whether the IR-only (`wasm`) build carries them ([steps](steps.md#where-the-code-lives)). - Whether a commit should record the core and spec versions that wrote it ([hashes and history](hashes-and-history.md#history)). - What generators and `doc` do with a version that has unknowns ([format 4](format-4.md#consumers)). -- Whether `comline store replay` (rebuilding a history from Git) is worth building for the stores that can't be - migrated ([steps](steps.md#what-cannot-be-migrated)). diff --git a/docs/docs/proposals/ir-migrations/proof.md b/docs/docs/proposals/ir-migrations/proof.md index 1bf1e7a..ef4610b 100644 --- a/docs/docs/proposals/ir-migrations/proof.md +++ b/docs/docs/proposals/ir-migrations/proof.md @@ -68,17 +68,16 @@ Each is a **scripted history**: snapshots of the sources and the actions between `build --reconcile`), with at least a breaking change, a release, a rebuild that changes nothing (so makes no commit), and, for one package, two heads left by a Git merge. They live in `core/tests/migrations/corpus/`. -**The stores.** For each format and each [ledger row](formats.md#format-3-is-a-family) of it, the real -`.comline/` that the real core of that era built from each script, committed under -`core/tests/migrations/stores/format-///`, with a `PRODUCED_BY` file (the revision, the digest of -the lockfile, the toolchain, the command). A store is a few objects per version, so the corpus stays small. +**The stores.** For each format that has a step out of it, the real `.comline/` that the real core of that format built +from each script, committed under `core/tests/migrations/stores/format-//`, with a `PRODUCED_BY` file (the +revision, the digest of the lockfile, the toolchain, the command). A store is a few objects per version, so the corpus +stays small. A format that gains a [ledger row](formats.md#format-3-is-a-family) while it is current gets a second set +for the row, so a store of each level is covered. -| Format | Built by | Notes | -|---|---|---| -| 0 | the CLI at `f934a66` (core `f035379`), lockfile committed | a store to be **refused**; the test is that nothing misreads it | -| 1 | core at `d2763cd`, plus a small patch that adds a build example | no CLI pinned it | -| 2 | the CLI at `086cd32` (core `6101ef8`), and at `00ee242` (core `30dc8ed`) | first and last of the seven pins in this format | -| 3 | one revision per ledger row, the CLI where it pinned one (`1455192` for `b04836d`, `102a476` for `bd9d9f4`), core with a patch where it didn't | `HEAD` at the time of the format-4 pull request is the last row | +Before 1.0 the only format with a step out of it is 3, for the 3→4 client: its fixtures are built by the core at the +closing revision of format 3, the one the framework ships on. The CLI revision that pins it is the easy route, +because the CLI keeps a committed `Cargo.lock`. From 1.0 on, **every released format** gets its fixtures when it +is released, built by that release's core, and keeps them for ever ([after 1.0](beyond-the-store.md#after-10)). ### Regenerating @@ -86,28 +85,27 @@ the lockfile, the toolchain, the command). A store is a few objects per version, it with its lockfile and `--locked`, and runs each script. Core's only non-determinism is the author (`USER`, or `whoami`) and the timestamp (`SystemTime`), so the generator sets the first and runs under a fake clock (`faketime`, on Linux). A store regenerated that way is **byte for byte** the committed one, and `--check` says so. -The committed bytes are the fixtures; regeneration is what shows they are what that core made. +The committed bytes are the fixtures; regeneration is what shows they are what that core made. A weekly job runs +`--check` for every recorded format, so a yanked crate or a vanished revision shows within a week and not when a +fixture is needed. The committed stores don't depend on it. + +Producing a format's fixtures is therefore part of releasing a format, and the [gate](#the-gate) requires them. -### Building the old revs +### Test-only formats -I haven't built anything for this proposal. What can be read from Git: +Format 4 alone can't prove everything: it has one step, and its unknowns are typed. The harness also has formats that +exist only in the test tree, `T0 → T1 → T2`, with real frozen types, codecs and steps, never written by a release: -- **Core's CI was green at the first merge of each format:** `d2763cd` on 2026-10-11 (format 1), `59587ba` on - 2026-10-12 (format 2), `b04836d` on 2026-10-18 (format 3). The commits inside them have no runs of their own. -- **Core had no `Cargo.lock` at any of them** (it is a library; the file is ignored), and `rust-toolchain.toml` says - `stable`. A build today resolves every caret dependency to its newest release. It will probably build, and its - bytes depend on `ciborium`, `lz4_flex` and `blake3`, so a fixture needs a lockfile. Core-only revisions (format 1, - and the ledger rows no CLI pinned) get one generated once and committed in `PRODUCED_BY`. -- **The CLI revisions have a committed `Cargo.lock`,** no `rust-toolchain.toml`, and name four git dependencies (core, - generation, comline-rust, comline-typescript) at revisions that **all still exist** (checked for `f934a66`, - `086cd32`, `00ee242` and `1455192`). With `--locked` they are the best-reproducible route, on the stable toolchain - from that revision's date. -- **What can still go wrong:** a yanked crate, a rewritten history, a newer toolchain rejecting old code. None is - known. +| What | How it is built in | What it proves | +|---|---|---| +| a chain of two steps | `T0 → T1` and `T1 → T2`, run alone and composed | the **composition** law, N→N+2 against N→N+1→N+2 and against the square | +| a lossy step | `T1` retypes a field so that `T0`'s value can't be recovered, and the new type has no unknown variant | the `unknown` entry, the placeholder, the original kept, and `forget` | +| a typed unknown | `T2` adds a `Known` field | the typed carrier, and that every reader has to handle it | +| a step that is wrong on purpose | each of the three mistakes in [the square](#what-accurate-means), and a step that drops a field or uses a clock | the harness **fails**: the gate is tested by seeing it reject a bad step | +| crashes | the engine on `T0` stores, killed after every write and rename | resume and idempotence | -So the first thing that stage 1 does is run it, once, and record the result. A weekly job (`old-revs.yml`, plus a manual -trigger) rebuilds every `PRODUCED_BY` revision, so that a vanished revision or a yanked crate shows within a week and -not when a fixture is needed. The committed stores don't depend on it. It protects regeneration, not the tests. +They need no old cores: a `T0` store is built by the `T0` writer in the test tree. They are the way the system is proven +before there is a second real format. ## Differential property tests @@ -117,7 +115,7 @@ Fixtures are fixed histories. The differential test draws random ones. files and has no dependency on core. Add **histories**: a sequence of edits (add a field, swap a type, add or remove a dependency, change a handshake or a bound, release), with proptest strategies for the property tests and `Arbitrary` for the fuzzer, as for the guarantee oracle. -- **The old side runs out of process.** The CLI (or maker) built at the pinned revision of each format is run over the +- **The old side runs out of process.** The CLI (or maker) built at the recorded revision of the previous format is run over the rendered package, and its `.comline/` is read back. The binary is built once per `(revision, lockfile, toolchain)` and cached in CI. The only exchange is the file system. - **The current side runs in the test:** the current core builds the same history, `migrate` runs on the old store, and @@ -193,7 +191,7 @@ What it catches: | An edit to a frozen format | the seal | The seal and the bump check should be under CODEOWNERS, and a seal can be moved only by a command that names why, so that -moving one is a deliberate edit and not a side effect. Weekly: the old revisions build and `--check` passes. Nightly: +moving one is a deliberate edit and not a side effect. Weekly: the fixtures `--check` against their recorded revisions. Nightly: the differential test at a larger budget and the fuzz target. ## What this proves, and what it doesn't diff --git a/docs/docs/proposals/ir-migrations/running.md b/docs/docs/proposals/ir-migrations/running.md index 0ad7e12..beae0ee 100644 --- a/docs/docs/proposals/ir-migrations/running.md +++ b/docs/docs/proposals/ir-migrations/running.md @@ -12,8 +12,9 @@ ends, and how does someone find out what it would do first? | The store | A read command (`diff`, `doc`, `generate`, `inspect`) | A write command (`build`, `release`) | |---|---|---| | N = C | proceeds | proceeds | -| N < C, and N is supported | reads through an **in-memory** migration and writes nothing | migrates, then proceeds | -| N < C, N unsupported, or unstamped | an error that names the format and the [options](steps.md#what-cannot-be-migrated) | the same error | +| N < C, and a chain of steps reaches C | reads through an **in-memory** migration and writes nothing | migrates, then proceeds | +| N < C, and no chain does (pre-1.0 only) | an error that says the store is old and will be reset, and the [options](steps.md#what-cannot-be-migrated) | **resets**: the store is renamed to `.comline.format-N.bak/` and a new one starts; asks first on a terminal; `deps-cache/` is kept | +| unstamped, or damaged | an error naming the format or the file | the same error; nothing is reset or deleted | | **N > C** | an error: the store is format N, this comline reads up to C, and *comline ≥ X* reads N | the same error; nothing written | | a stamp that isn't a number, or has conflict markers on one side | an error naming the file ([see Git](#git)) | the same | @@ -21,7 +22,8 @@ Reading an old store needs no write because a migration is a pure function: a re versions it needs and drop the result. That makes `comline diff`, `doc --version` and `generate` work on a store nobody has migrated yet, at the cost of the memory and time of the chain, which a cache can hide. -Two of today's behaviors go: deleting an older store, and deleting a newer one. The second is the more dangerous. +Two of today's behaviors go: deleting an older store (it is moved to a backup instead), and deleting a newer one (it +is refused). The second is the more dangerous. `stored != Some(CURRENT)` is true for a store that is ahead of the binary. ## The procedure diff --git a/docs/docs/proposals/ir-migrations/steps.md b/docs/docs/proposals/ir-migrations/steps.md index 7ad5af1..65c268c 100644 --- a/docs/docs/proposals/ir-migrations/steps.md +++ b/docs/docs/proposals/ir-migrations/steps.md @@ -65,14 +65,16 @@ An unknown has two possible carriers. **Proposed: both**, in this order of prefe 2. **An `unknown` entry in the root tree,** for a retype whose new type has no room: a blob listing, for each unknown, its `path` into the model, the `step` that wrote it, the `reason`, and the `original`, the bytes N recorded, where N recorded something. The field itself holds a placeholder that the entry marks as one. This is - what lets a 2→3 step exist at all: format 3's `TypeRef` has no "unknown" variant and can't gain one without a - bump, while the entry keeps the old `float` text beside a placeholder. + what lets a step exist where the new type has no "unknown" variant and can't gain one without a bump: the entry + keeps the old value (say, a `float` that N froze without a width) beside a placeholder. Format 3's `TypeRef` is an + example of a type with no room, and a test-only format has one on purpose + ([proof](proof.md#test-only-formats)). `unknown` is an additive entry, a row in format 3's ledger with the absence reading *none*. A version without it has no unknowns. -**Ordering constraint.** A shipped format-3 binary would read a store migrated into format 3, with unknowns in the -entry, as complete. So the readers of `unknown` ship, in a release, before any step that writes one. See +**Ordering constraint.** A shipped format-3 binary would read a store that carries unknowns in the +entry as complete. So the readers of `unknown` ship, in a release, before any step that writes one. See [the rollout](index.md#rollout). ### Reading an unknown @@ -120,34 +122,32 @@ catch the two disagreeing. ## What cannot be migrated +Today's formats are out of scope ([index](index.md#scope-the-system-not-the-history)): they reset at 1.0. Until then, +and for any store the system can't migrate, one rule holds: **it is reset into a backup, never deleted.** + | Store | Why | What happens | |---|---|---| -| **Format 0**, unstamped bincode | Positional: a bincode decode of the wrong layout can succeed and give a different variant, so the data can't be told from garbage. About 20 layouts existed in nine months and none was marked. A commit's timestamp says when it was built, not which core built it. Nothing in the store, whose hashes cover bytes and not meaning, can check a decode against. | Refused. The store is kept. The error names the options below. | -| An unstamped store that decodes as CBOR | The stamp was lost, so the format isn't known. | Refused, unless the user gives one: `--assume-format N`. It is recorded in the receipt, and the decode must succeed for every object. | -| A damaged store (a missing or corrupt object, [#182](https://github.com/ComlineProject/core/issues/182)) | A migration needs every object. | Refused, naming the file and how it failed. | -| A store whose stamp says N but whose objects don't decode as N | A binary that predates the stamp, or a half-finished edit. | Refused, naming the first object. | +| A store in a pre-1.0 format with no step to the current one (the unstamped bincode store, formats 1 and 2, older rows of format 3) | Nobody writes those steps. Format 0 is positional: a decode of the wrong layout can succeed and give a different variant, and nothing in the store can check a decode against. | Reset: renamed to `.comline.format-N.bak/`, a new store starts at `0.1.0-dev.1`. The command says so and names the backup. `deps-cache/` is kept | +| An unstamped store that decodes as CBOR | The stamp was lost, so the format isn't known. | Refused, unless the user gives one: `--assume-format N`. The decode must succeed for every object | +| A damaged store (a missing or corrupt object, [#182](https://github.com/ComlineProject/core/issues/182)) | A migration needs every object. | Refused, naming the file and how it failed. Nothing is reset: a damaged store is for the user to look at | +| A store newer than the binary | The binary can't read it. | Refused, naming the format and the comline that reads it. Never touched | -For a store that can't be migrated, the options are the ones that exist already, plus one: +A reset needs the user's say where there is history worth asking about: on a terminal it asks (default no) and +otherwise needs `--reset-old-store`, as `comline reset` does for deletion. The backup is what makes it safe to ask +less. -- **Keep it and start a new history,** `comline reset`'s result without the deletion: the old store is renamed - aside, as a backup is. -- **`comline store replay`** *(proposed; may not be worth building)*. Rebuilds a history from the project's Git - history with the current core, one build per release tag or commit chosen by the user. The result is a **new** - history: it never claims to be the old one, its versions are numbered by the current rules, and the command says - so. It is the way back for format 0 and for any store the framework refuses. +A history can later be rebuilt from Git with `comline store replay` *(proposed; may not be worth building)*. It builds a +**new** history with the current core, one build per release tag or commit the user chooses, never claims to be the old +one, and numbers versions by the current rules. ## Support -| Format | Migrates to | Lossy? | Fixtures | -|---|---|---|---| -| 3 | 4 | the chains and scopes format 4 records ([format 4](format-4.md)), and whatever a version's ledger level lacks | one per [ledger](formats.md#format-3-is-a-family) row | -| 2 | 3 | yes: float width, array length, written form of a value | built by the CLI pinned at `00ee242` | -| 1 | 2 | yes: the written form of an annotation's value | built from core's `d2763cd`, with a small patch | -| 0 | nothing | n/a | one, to test that it is *refused* | - -Steps 2→3 and 1→2 are the only real lossy steps that exist before format 4, and so they are where the unknown -machinery gets tested on real data. That is the reason to write them even though few stores are in these formats. -Format 1 existed for about fifteen hours on `master` and no released CLI built it. +| Format | Status | +|---|---| +| 1.0 and later | every format migrates to the next: that is the guarantee this system exists to give | +| 3 (as it stands when the framework ships) | 3→4, the first client: it carries real stores, and exists to prove the system | +| 0 to 2, and older rows of 3 | reset at 1.0, and before it where no step exists | +| test-only formats | in the test tree, never written by a release: [the proof](proof.md#test-only-formats) | ## Where the code lives @@ -157,7 +157,7 @@ per format with its `SCHEMA.md`, types, codec and ledger, and the steps between beside the frozen N+1 types. Those exist from the day N+1 is introduced, and are kept equal to the live types by the `live ≡ frozen(current)` test until N+2 seals them. -Core depends on the crate to read old stores. The engine and the CLI commands are in core. The crate's only +Core depends on the crate to read old stores. Test-only formats live in its test tree, not in the shipped modules. The engine and the CLI commands are in core. The crate's only dependencies are `serde`, `ciborium`, `blake3` and `lz4_flex`, so it can be in the IR-only build, which is the one the `wasm32` consumers use. Whether it *should* be there, since the build's point is to be small, is open. From 7bafd8dca9bbe8b9a3785482492bd2e131369d0f Mon Sep 17 00:00:00 2001 From: Kinflou Date: Fri, 13 Nov 2026 04:11:42 +0800 Subject: [PATCH 3/4] proposals/ir-migrations: nothing changes without the user; record the decisions Migration and reset run only by explicit user commands (comline store migrate, comline store reset); other commands stop and name them. Every step has a down witness checked at migration time. Old pins that verify are accepted with a notice. Git merges across formats are refused. The 1.0 release refuses pre-1.0 stores, with an epoch stamp. Drops the older-binary hazard discussion and most open questions. --- .../ir-migrations/beyond-the-store.md | 40 +++--- docs/docs/proposals/ir-migrations/format-4.md | 10 +- docs/docs/proposals/ir-migrations/formats.md | 15 ++- .../ir-migrations/hashes-and-history.md | 26 ++-- docs/docs/proposals/ir-migrations/index.md | 100 +++++++-------- docs/docs/proposals/ir-migrations/running.md | 114 +++++++++--------- docs/docs/proposals/ir-migrations/steps.md | 66 +++++----- 7 files changed, 182 insertions(+), 189 deletions(-) diff --git a/docs/docs/proposals/ir-migrations/beyond-the-store.md b/docs/docs/proposals/ir-migrations/beyond-the-store.md index ae043c6..e63ad2a 100644 --- a/docs/docs/proposals/ir-migrations/beyond-the-store.md +++ b/docs/docs/proposals/ir-migrations/beyond-the-store.md @@ -9,7 +9,7 @@ does each of the others need, and what does a published package need when there | Artifact | Holds | Migration story | |---|---|---| -| **`.comline/` store** (`objects/`, `refs/`, `FORMAT_VERSION`) | every version, as frozen IR | [Migrated](running.md), whole, with the old one kept; before 1.0, [reset into a backup](#the-10-reset) where no step exists | +| **`.comline/` store** (`objects/`, `refs/`, `FORMAT_VERSION`) | every version, as frozen IR | [Migrated](running.md), whole, with the old one kept; before 1.0, where no step exists, [the user resets it](#the-10-reset) into a backup | | **`.comline/deps-cache/`** | git checkouts of dependencies: source, not IR | Not migrated, not part of the store. **Carried across the swap**: today's wipe deletes it, and an offline machine can't fetch it again | | **A dependency's own committed `.comline/`** | its version history; the only thing core reads of it is the newest commit's `version` (`resolved_version`) | Read with the format-independent [container reader](formats.md#what-a-store-holds). Today any failure reads as `0.0.0-unbuilt`; it should say the store couldn't be read | | **Dependency pins** in `config.idp` | scheme v2 hash over IR bytes | Never rewritten. [Bridged](hashes-and-history.md#pins) | @@ -47,26 +47,22 @@ keyed by it. The rules for IR follow from that. ## The 1.0 reset **Decided:** the formats that exist today are all reset when 1.0 arrives, and the numbering restarts: the 1.0 format is -**format 0** of a new numbering, and the first with a guaranteed path forward. Nothing before it is migrated. - -**Proposed:** it is one release, so that there is one moment and one rule. - -1. The release finds a pre-1.0 store, in any format, and **resets it into a backup**: `.comline.format-N.bak/`, the - same mechanism as [any reset](running.md#what-each-command-does-with-each-stamp), never a deletion. `deps-cache/` is - kept. A new store starts, and stamps the 1.0 format 0. -2. It is told apart from the old numbers: pre-1.0 stamps are the integers 0 to 4, and a 1.0 `0` or `3` must not be read - as one of them. So the 1.0 stamp carries an **epoch** (`FORMAT_VERSION` reads, say, `1.0/0`), and a bare integer - is by definition pre-1.0. The new directory name that [running](running.md#older-binaries) floats would do the - same job. *Open.* -3. The frozen types of the pre-1.0 formats are **not** kept: the reset is the only thing that touches them, and it - needs the stamp and the container, which are format-independent ([formats](formats.md#what-a-store-holds)). -4. Every format change after it is a step, with a document, fixtures and a seal, as the gate requires. The system is +**format 0** of a new numbering, and the first with a guaranteed path forward. Nothing before it is migrated. And, like +everything here, **nothing happens on its own**: + +1. **A warning release first.** The release before 1.0 says, once per command that finds a pre-1.0 store, that 1.0 will + refuse it and which command to run. It changes nothing. +2. **The 1.0 release refuses a pre-1.0 store.** Any command that finds one stops, names `comline store reset`, and + changes nothing. The reset is the same [user command](running.md#reset) as before 1.0: it renames the store to + `.comline.format-N.bak/` after the user confirms that exact action, keeps `deps-cache/`, and starts a new store at + the next build, stamped format 0 of the 1.0 numbering. The backup stays until `comline store prune`. +3. **The stamp has an epoch.** Pre-1.0 stamps are the integers 0 to 4, and a 1.0 `0` or `3` must not be read as one of + them. So the 1.0 stamp reads `1.0/0`, and a bare integer is by definition pre-1.0. +4. **The frozen types of the pre-1.0 formats are not kept:** the refusal needs only the stamp and the container, which + are format-independent ([formats](formats.md#what-a-store-holds)). +5. Every format change after it is a step, with a document, fixtures and a seal, as the gate requires. The system is already in use by then: format 4's step is its first run. -*Open:* the release's shape. One release that only resets, so that its behaviour is easy to state and to find, or the -1.0 release itself. Leaning: the 1.0 release resets, and the release before it warns, once, that a reset is coming and -names the backup. - ## After 1.0 [core#146](https://github.com/ComlineProject/core/issues/146) asks that from 1.0 a newer comline reads what an older one @@ -76,7 +72,7 @@ framework *is* the readers, and its fixtures and laws are the tests. | | Before 1.0 | From 1.0 | |---|---|---| | **Formats that migrate** | none promised: format 3 for the first client, the rest reset | every format released at or after 1.0, for ever. A released format's frozen types, steps, fixtures and seal are never removed | -| **A store with no path** | reset into a backup | refused, never reset, and the error names the **bridge release**: the last comline that still migrates it | +| **A store with no path** | refused; the user's `comline store reset` moves it to a backup | refused, and the error names the **bridge release**: the last comline that still migrates it. There is no reset | | **Dropping a format** | any minor release | a major one, announced one minor earlier; the bridge release stays installable | | **Cost of a bump** | a step and its fixtures | the same, **and permanent**: it is carried for ever | | **Additive changes** | a [ledger](formats.md#format-3-is-a-family) row each | a ledger row each, a fixture, and the same lines in the frozen copy | @@ -91,6 +87,6 @@ The spec side needs nothing of this: the spec is append-only for anything frozen ([the vocabulary](../shared-vocabulary/catalog.md#versioning)), so an old version reads under a newer spec without a step. Only the IR format does. -## Still open here +## Decided here -- **How long a bridge release stays supported,** after 1.0. +- **How long a bridge release stays supported is decided at 1.0,** not before. diff --git a/docs/docs/proposals/ir-migrations/format-4.md b/docs/docs/proposals/ir-migrations/format-4.md index 4816e91..352e75e 100644 --- a/docs/docs/proposals/ir-migrations/format-4.md +++ b/docs/docs/proposals/ir-migrations/format-4.md @@ -14,7 +14,7 @@ migrated ([scope](index.md#scope-the-system-not-the-history)). One pull request copy and ledger, a step with `up`, `down` and an `unknown` list, a typed unknown, the square against a fresh format-4 build, the laws, the pin bridge, and the gate. The test-only formats of [the proof](proof.md#test-only-formats) cover what one step can't (composition, an untyped unknown); the two together are the system's proof. A format-3 store that -exists when the framework ships is migrated for real, and one older than that is reset. +exists when the framework ships is migrated for real (when the user runs `comline store migrate`), and one older than that is not migrated: the user resets it with `comline store reset`. ## What changes @@ -81,8 +81,8 @@ that is a good model: when bounds on types arrived, `build` listed every positio defaults with maxima the author typed, never guessed (`bounds_migration.rs`). Here the offer is each acceptance, with what it now covers, and the author decides. -*Open:* whether a transition period accepts an unpinned entry with a warning, as pins might -([hashes](hashes-and-history.md#pins)). The safety invariant argues against it: an acceptance that isn't pinned to a +*Open:* whether a transition period accepts an unpinned entry with a warning. Old *dependency pins* are +[accepted with a notice](hashes-and-history.md#pins), but the safety invariant argues against doing the same here: an acceptance that isn't pinned to a chain is exactly what the invariant is there to remove. ## Consumers @@ -94,8 +94,8 @@ What reads a version with unknown chains, and what each does: | `build`, the guarantee check | recomputes from inputs; the unknown is never what it judges on | | `diff`, `doc --version`, the changelog | recompute; list what is `unseen` | | `inspect` | shows the recomputed chain, marked "recomputed, not recorded" | -| the generated `@unverified` docs ([generation#93](https://github.com/ComlineProject/generation/issues/93)) | the chain as recomputed. Where it can't be, "origin not recorded" | -| a generator given an old version | *Open.* The unit it is given has the typed unknown. Refusing a version that has one is the strictest reading, and a flag could allow it. Leaning: pass it, since a generator that prints a chain must print "not recorded", and add the field to the protocol, where it can ignore it | +| the generated `@unverified` docs ([generation#93](https://github.com/ComlineProject/generation/issues/93)) | the chain as recomputed. Where it can't be, it fails closed as below | +| a generator, or `doc`, given an old version | **fails closed**: it stops with "unknown (migrated from format N)" for what it would print from an unknown, naming the version and the path. A generator is never handed a placeholder that looks complete | ## Fixtures and tests for 3→4 diff --git a/docs/docs/proposals/ir-migrations/formats.md b/docs/docs/proposals/ir-migrations/formats.md index 5aa1a0c..f0856d7 100644 --- a/docs/docs/proposals/ir-migrations/formats.md +++ b/docs/docs/proposals/ir-migrations/formats.md @@ -52,7 +52,7 @@ unknown machinery in [steps](steps.md#unknown) exists for, and the one the test- The doc comment on `CURRENT_IR_FORMAT_VERSION` says growing `FrozenUnit` needs no bump. Since 2026-10-18 these have been added without one, and so a "format 3" store can lack any of them. The 3→4 step, the first client, takes format 3 -as it stands when the framework ships (the last row), and a store from before that is reset, like any pre-1.0 store: +as it stands when the framework ships (the last row), and a store from before that is not migrated (the user can reset it, like any pre-1.0 store): | Added | Commit | Absent reads as | |---|---|---| @@ -121,11 +121,10 @@ bump. The `skip_serializing_if` condition is the one the code already follows (" to the same bytes as before them"), because the bytes of a frozen unit are also what a [dependency pin hashes](hashes-and-history.md#pins). -## Still open here +## Decided and open here -- **Whether `SCHEMA.md` is the source and the [CAS guide](../../guides/ir/cas.md) links to it, or the guide stays the - description of the current format and `SCHEMA.md` is generated beside it.** Leaning: `SCHEMA.md` in core is the - source; the guide describes the current format and says so. -- **Whether an additive change should raise a second number** (a ledger level, beside the format) so that an older - binary refuses to *write* into a store whose commits use additions it would drop. This would turn gap 2 into an - error. The cost is a second stamp to keep in step. +- **Decided:** no second number (a ledger level) beside the format. It would only let a binary from before this proposal + refuse to write into a store whose additions it would drop, and those binaries are out of scope. +- **Still open:** whether `SCHEMA.md` in core is the source and the [CAS guide](../../guides/ir/cas.md) links to it, or + the guide stays the description of the current format and `SCHEMA.md` is generated beside it. Leaning: `SCHEMA.md` is + the source, and the guide describes the current format and says so. diff --git a/docs/docs/proposals/ir-migrations/hashes-and-history.md b/docs/docs/proposals/ir-migrations/hashes-and-history.md index eaf1543..42afc00 100644 --- a/docs/docs/proposals/ir-migrations/hashes-and-history.md +++ b/docs/docs/proposals/ir-migrations/hashes-and-history.md @@ -46,7 +46,7 @@ migration that breaks the build it was meant to save. And the [offline proposal] makes the hash the identity of a dependency: caches and signed bundles are keyed by it, so a hash that moves with core's internal format would invalidate both. -**Proposed:** a pin made under format N, by scheme S, **still verifies under C when the dependency's content can be +**Decided:** a pin made under format N, by scheme S, **still verifies under C when the dependency's content can be expressed in N**, and a pin is never accepted otherwise. 1. To check a pin that doesn't match, core takes the dependency's compiled model and applies `down` through each step @@ -74,11 +74,11 @@ told why. | Proof | golden pins from the old cores | golden pins from the form | none needed, and none possible | | Costs | `down` must be defined on a compile, not only on a stored version | the form must also carry new information the day the IR does | the offline proposal's bundles and caches | -*Open:* whether a pin that verifies through the witness is **accepted silently,** accepted with a warning that names the -format ("re-pin to move to the current one"), or rejected with the same message as case 3 (the re-pin being a one-line -change that is then explicit). Accepting is sound by the argument above, and rejecting is the stricter and costs -every repository one edit per bump. Leaning: accept with a warning once the bridge has the golden pins behind it. -Option B stays available if the bridge turns out too costly to keep through many formats. +**Decided:** a pin that verifies through the witness is **accepted, with a one-line notice** that suggests a re-pin +("this pin was made under IR format N and still verifies; re-pin to move to the current one: `hash = "…"`"). It is never +silent and never a failure. Accepting is sound by the argument above: the content has nothing the pin didn't cover. The +re-pin is the user's, by hand or with `comline add --repin`; nothing rewrites `config.idp`. Option B stays available if +the bridge turns out too costly to keep through many formats. The pin gate in [the proof](proof.md#the-gate) makes the move visible: a change that moves the *fresh* pin of a fixture dependency has to say so in the pull request. @@ -105,11 +105,12 @@ migration is not a rebuild, and a version that was released as `0.4.0` was. says `diff` recomputes both versions under the newer spec, so a loss the spec learned to see is in both and is no change. A format-3 version whose protocol never had `bounded_size` in its frozen set (the guarantee didn't exist) is not "narrower" than today's: it is recomputed, and gives it up like its successor does. -- **The spec a version was frozen against** is, today, recorded nowhere: the decision in the vocabulary is that each - frozen version records it, and that isn't built. A version without it reads as `unknown`, and is recomputed under - the current spec like any older one. **Proposed:** record it as a root-tree entry, absent meaning unknown, in the - first release of the framework, so that every version built after it has it. *Open:* whether a commit also records the - core that wrote it (`produced_by`), which would tell a ledger row without guessing. +- **The spec and core that wrote a version** are, today, recorded nowhere: the decision in the vocabulary is that each + frozen version records the spec it was frozen against, and that isn't built. **Decided:** each stored version records + **the core version and the spec version that wrote it**, as root-tree entries (absent meaning unknown), in the first + release of the framework, so every version built after it has them. A version without them, a migrated one included, reads as `unknown`, is + recomputed under the current spec like any older one, and a ledger row can be told from the record instead of + guessed. - **What can't be recomputed is `unseen`.** If a rule needs an input the version lacks (a dependency's config, a type's width), the comparison lists it in the same `Comparison::unseen` that [core#262](https://github.com/ComlineProject/core/pull/262) added: the version and the path, with the reason from the `unknown` entry. A rule that needs a fact it can't see reads @@ -117,7 +118,8 @@ migration is not a rebuild, and a version that was released as `0.4.0` was. comparison that couldn't see part of what it compares doesn't say that nothing changed. It can be higher, since "strictest" can be a breaking change: a type of unknown width against a known one is possibly breaking. - **The changelog** shows the same: a change it found, and, as lines of their own, what it could not compare, in the words of - the reason. `comline doc --version V` for a version with unknowns says which parts are. + the reason. `comline doc --version V` for a version with unknowns **fails closed** on what it would print from an unknown, as `doc` does for + any consumer ([format 4](format-4.md#consumers)). | Case | Result | |---|---| diff --git a/docs/docs/proposals/ir-migrations/index.md b/docs/docs/proposals/ir-migrations/index.md index aa7a4a7..5d61a45 100644 --- a/docs/docs/proposals/ir-migrations/index.md +++ b/docs/docs/proposals/ir-migrations/index.md @@ -22,13 +22,15 @@ harness and the CI gate. It does not promise to migrate today's formats (the uns or the family that is format 3), and it is fine if one of them can't be migrated reliably. The 1.0 format is the first with a guaranteed path forward. -- **Before 1.0,** a store in an older format may be **reset**, but safely: moved to a backup and never deleted, a - store *newer* than the binary refused and never touched, and `deps-cache/` kept. See +- **Before 1.0,** a store in an older format with no step to the current one is **never touched by a command on its own.** + The command stops and names `comline store reset`, which the user runs, and which moves the store to a backup and + deletes nothing. A store *newer* than the binary is refused. `deps-cache/` is kept. See [running](running.md#what-each-command-does-with-each-stamp). - **The system is proven before 1.0,** because 1.0 must not be its first test. The 3→4 step for the chains of [core#222](https://github.com/ComlineProject/core/issues/222) is the first real client, and test-only formats prove composition and lossy steps. See [format 4](format-4.md) and [proof](proof.md). -- **The 1.0 reset** is one release. See [beyond the store](beyond-the-store.md#the-10-reset). +- **The 1.0 reset:** a warning release, then a 1.0 release that refuses pre-1.0 stores and names the reset command. See + [beyond the store](beyond-the-store.md#the-10-reset). ## What it looks like today @@ -55,7 +57,8 @@ This proposal comes first so that it doesn't, and so that by 1.0 the machinery h step can drop what a version recorded. 3. **[Running a migration](running.md).** The new store is written beside the old one, verified, and swapped in; the old one is kept. A migration can be checked without writing (`comline store migrate --check`), resumed, and - repeated. A store newer than the binary is an error, never a deletion. + repeated. It runs only when the user runs it: every other command stops and names it. A store newer than the binary + is an error, never a deletion. 4. **[Proof](proof.md).** A corpus of real stores built by the real core of each format, an equivalence stated field by field, differential property tests against the previous format's core, test-only formats for composition and lossy steps, laws (totality, determinism, idempotence, composition, hash stability), and a CI gate that fails a format bump @@ -69,6 +72,12 @@ This proposal comes first so that it doesn't, and so that by 1.0 the machinery h ## Rules that hold whatever the details +- **Nothing is deleted or changed without the user doing so.** No command migrates, resets, moves or deletes a store + as a side effect. A command that finds an older store stops, says what it found and names the command to run + (`comline store migrate`, with `--check` as the dry run, or `comline store reset`). On a terminal it may ask whether + to run that exact command and does nothing without a yes; in CI it fails with the message. Backups are removed only by + `comline store prune`. The one thing a command may do on its own is read an old store through an in-memory migration, + because that changes nothing on disk. - **A store is never deleted to change its format.** Migration replaces deletion; where migration is impossible the old store stays, and the error says what to do. - **A step is a pure function of one format's data.** No clock, no environment, no network, no randomness. Two @@ -77,44 +86,53 @@ This proposal comes first so that it doesn't, and so that by 1.0 the machinery h everywhere it is read: [#251](https://github.com/ComlineProject/core/issues/251)'s rule for a dependency's config, and the fail-closed rule of the [shared vocabulary](../shared-vocabulary/granular-unverified.md#fail-closed). - **Nothing is replaced until it is verified,** and what is replaced is kept until someone removes it. A store that - can't be migrated is reset into a backup, never deleted. + can't be migrated stays as it is until the user resets it into a backup. - **Accurate means tested against a fresh build.** A migrated version equals what the current core would have built from the same source, field for field, except for the fields the step declared unknown. - **A format change brings its step, its fixtures and its document in the same pull request,** or CI fails. -## What is proposed, in one table +## What was decided -| | Proposed | +Settled with the user, and written into the pages: + +| | Decision | |---|---| | Unit of migration | the whole store, version by version, all or nothing | -| Steps | `N → N+1`, typed on frozen copies; `up`, `down` (witness), and a list of unknowns | -| Unknown | typed `Known` where the new type has room, otherwise an `unknown` entry in the root tree that keeps the original value | +| Steps | `N → N+1`, typed on frozen copies; `up`, `down` and a list of unknowns. **Every step has the `down` witness**, checked in tests and at migration time on the user's own store | +| Unknown | typed `Known` where the new type has room, otherwise an `unknown` entry in the root tree that keeps the original value. Generators and `doc` **fail closed** on one: "unknown (migrated from format N)" | +| Automatic changes | **none.** Migration and reset run only by the user's command; other commands stop and name it (`comline store migrate`, `--check` as the dry run; `comline store reset`). CI fails with the message | | Atomicity | write `.comline.migrating/`, verify, rename the old store to `.comline.format-N.bak/`, rename the new one in | -| Backup | a sibling directory, kept until `comline store prune`; never deleted automatically | -| Newer store | an error naming the format and the comline that reads it | +| Backup | `.comline.format-N.bak/`, kept until the user runs `comline store prune` | +| Reset | `comline store reset`, an explicit command that moves the store to a backup after the user confirms that exact action; `deps-cache/` is kept | +| Newer store | an error naming the format and the comline that reads it; never touched | | Read commands | read an old store through an in-memory migration; they write nothing | -| Pins | verified through the witness: a pin made under format N still verifies if the dependency's content is representable in N | -| History | comparisons recompute derived facts from stored inputs under the current spec; what can't be recomputed is `unseen` | +| Git merge of stores in two formats | refused: "migrate both sides first" | +| Pins | verified through the witness; an old pin that verifies is **accepted with a one-line notice** suggesting a re-pin, never silently and never a failure | +| History | comparisons recompute derived facts from stored inputs under the current spec; what can't be recomputed is `unseen`. Each stored version records the core and spec versions that wrote it | +| Frozen formats | the `comline-ir-formats` crate; the IR-only (`wasm`) build carries it only for in-memory, read-only migration | | Gate | `formats/MANIFEST`, `live ≡ frozen(current)`, fixtures, and a bump check; see [the gate](proof.md#the-gate) | -| Today's formats | not migrated; reset to a backup at 1.0, and before it where a step doesn't exist | +| Today's formats | not migrated; the user resets them; at 1.0 the numbering restarts | +| 1.0 | a warning release, then a 1.0 release that refuses pre-1.0 stores, names the reset command, and stamps `1.0/0` | +| Bridge releases | how long one is supported is decided at 1.0 | | First client | the 3→4 step of #222's chains, end to end, before 1.0 | +Binaries from before this proposal delete a store whose stamp they don't expect. They are out of scope. + ## Rollout -Staged pull requests: the framework, then format 4 as its first client, then the 1.0 reset. Each is a checklist in -[core#268](https://github.com/ComlineProject/core/issues/268). +Staged pull requests: the framework, then format 4 as its first client, then the 1.0 refusal and reset. Each is a +checklist in [core#268](https://github.com/ComlineProject/core/issues/268). **1. The framework** (core, format stays 3): -1. *A newer store is an error, not a wipe, and an older one is reset into a backup, not deleted.* Core and CLI message. - A small pull request, released **before** anything ships format 4, so that no binary in circulation deletes a - format-4 store. +1. *A newer store is an error; an older store is an error naming `comline store migrate` or `comline store reset`; + nothing is changed automatically.* Core and CLI message. This removes the wipe and is released first. 2. *Format documents and the gate.* A `formats` crate with the frozen format-3 types, the ledger, the seal, the `live ≡ frozen(3)` test, and golden hashes for the fixture packages. This stops further no-bump drift before it - grows. It also adds, as ledger rows, the two entries nothing writes yet, `unknown` and the spec version a version was - frozen against, with readers that fail closed. It ships in a release with step 1 **before** any step or format that - writes either one. -3. *The engine and `comline store`:* container reader, plan, journal, verification, swap, backup, `--check`. + grows. It also adds, as ledger rows, the entries nothing writes yet (`unknown`, and the core and spec versions that + wrote a version), with readers that fail closed, before any step or format writes them. +3. *The engine and `comline store`:* container reader, plan, journal, verification, swap, backup, `--check`, `reset`, + `prune`, and the prompt a command shows on a terminal. 4. *The proof harness,* proved on **test-only formats** before any real step exists: composition (N→N+2), a lossy step with unknowns, the witness, crash injection, the laws. And the fixture generator. @@ -125,37 +143,23 @@ the gate requires it. It runs the square, the laws and the gate on a real change recomputation over stored dependency IR, `unseen` extended to chains, and the oracle and fuzz target over migrated stores. -**3. The 1.0 reset** (core, CLI): one release resets every pre-1.0 store into a backup and stamps format 0 of the -1.0 numbering. From then on every format change is a step. See +**3. The 1.0 refusal and reset** (core, CLI): a warning release, then the 1.0 release, which refuses a pre-1.0 store, +names `comline store reset`, and stamps `1.0/0`. From then on every format change is a step. See [the 1.0 reset](beyond-the-store.md#the-10-reset). **4. Tooling** (CLI, language server, generators, docs): -- `build` and `release` migrate on their own, with the plan printed. - `doc` caches key on the IR format. - The language server and `inspect` show `unseen`. -- The guide on [the CAS](../../guides/ir/cas.md) loses its warning about deletion. -- Generators learn what a version with unknowns means. +- Generators and `doc` fail closed on unknowns. +- The guide on [the CAS](../../guides/ir/cas.md) loses its warning about deletion and describes the stamp, backups and + `comline store`. ## Still open -Collected from the pages; each is where it is discussed. - -- Whether every step needs the inverse witness or only the ones that could lose something - ([steps](steps.md#the-witness)). -- Whether `build` and `release` migrate by default or need an explicit `comline store migrate` - ([running](running.md#the-cli)). -- Where the backup lives and for how long, and whether format 4 should also move the store to a new directory name - so that an older binary can't delete it ([running](running.md#older-binaries)). -- What happens when Git merges two branches whose stores are in different formats - ([running](running.md#git)). -- Whether a pin that verifies through the witness is accepted silently, with a warning, or only by an explicit - re-pin ([hashes and history](hashes-and-history.md#pins)). -- **How the 1.0 reset is carried out:** one release that resets with a backup and stamps the 1.0 format 0, and how - that stamp can't be confused with a pre-1.0 `0` to `4` - ([beyond the store](beyond-the-store.md#the-10-reset)). -- Where the frozen formats live: a crate, a module, and whether the IR-only (`wasm`) build carries them - ([steps](steps.md#where-the-code-lives)). -- Whether a commit should record the core and spec versions that wrote it - ([hashes and history](hashes-and-history.md#history)). -- What generators and `doc` do with a version that has unknowns ([format 4](format-4.md#consumers)). +Little is left: + +- Whether `SCHEMA.md` in core is the source for each format's document, with the CAS guide describing the current format + ([formats](formats.md#decided-and-open-here)). Leaning yes. +- Whether the 3→4 transition accepts an unpinned `accept_unverified` entry with a warning + ([format 4](format-4.md#acknowledgements)). Leaning no: the safety invariant is there to remove such an entry. diff --git a/docs/docs/proposals/ir-migrations/running.md b/docs/docs/proposals/ir-migrations/running.md index beae0ee..73e9229 100644 --- a/docs/docs/proposals/ir-migrations/running.md +++ b/docs/docs/proposals/ir-migrations/running.md @@ -5,30 +5,36 @@ Status: **draft** · part of [Migrating stored IR across format versions](index. **The question.** How does a store change format without any moment at which history is lost, however the process ends, and how does someone find out what it would do first? +## The rule + +**Nothing is deleted or changed without the user doing so.** No command moves, rewrites, resets or deletes a store as a +side effect. A command that finds a store it can't use as it is *stops*, says what it found and names the command that +fixes it, and does nothing until the user runs that command. The only exception is reading: a read command may read an +old store through an in-memory migration, because that changes nothing on disk. + ## What each command does with each stamp `ensure_compatible_store` stops deleting. With the stamp *N* and a binary that writes *C*: -| The store | A read command (`diff`, `doc`, `generate`, `inspect`) | A write command (`build`, `release`) | +| The store | A read command (`diff`, `doc`, `generate`, `inspect`) | A write command (`build`, `release`, …) | |---|---|---| | N = C | proceeds | proceeds | -| N < C, and a chain of steps reaches C | reads through an **in-memory** migration and writes nothing | migrates, then proceeds | -| N < C, and no chain does (pre-1.0 only) | an error that says the store is old and will be reset, and the [options](steps.md#what-cannot-be-migrated) | **resets**: the store is renamed to `.comline.format-N.bak/` and a new one starts; asks first on a terminal; `deps-cache/` is kept | -| unstamped, or damaged | an error naming the format or the file | the same error; nothing is reset or deleted | +| N < C, and a chain of steps reaches C | reads through an **in-memory** migration and writes nothing | **stops**: "the store is format N; run `comline store migrate` (`--check` shows what it would do)". On a terminal it asks whether to run exactly that, and does nothing unless the user says yes; without a terminal (CI) it fails with that message | +| N < C, and no chain does (pre-1.0 only) | an error naming the format and the [options](steps.md#what-cannot-be-migrated) | **stops**, naming `comline store reset`. It never offers to reset inline | +| unstamped, or damaged | an error naming the format or the file | the same error; nothing is changed | | **N > C** | an error: the store is format N, this comline reads up to C, and *comline ≥ X* reads N | the same error; nothing written | -| a stamp that isn't a number, or has conflict markers on one side | an error naming the file ([see Git](#git)) | the same | +| stamps or refs in conflict because Git merged stores in different formats | an error ([see Git](#git)) | the same | Reading an old store needs no write because a migration is a pure function: a read command can run the chain on the versions it needs and drop the result. That makes `comline diff`, `doc --version` and `generate` work on a store nobody has migrated yet, at the cost of the memory and time of the chain, which a cache can hide. -Two of today's behaviors go: deleting an older store (it is moved to a backup instead), and deleting a newer one (it -is refused). The second is the more dangerous. -`stored != Some(CURRENT)` is true for a store that is ahead of the binary. +Today's behaviors of deleting an older store and of deleting a newer one are both gone. Binaries from before this +proposal still delete a store whose stamp they don't expect; they are out of scope. ## The procedure -`comline store migrate`, and the same code behind `build` and `release`: +`comline store migrate`, which is also what `build` runs if the user confirms it: 1. **Claim the store.** `mkdir .comline.migrating/` beside `.comline/`. It fails if the directory exists, which is the lock. If it exists from a crash, see [below](#crashes). The migrating directory is on the same filesystem as @@ -52,12 +58,12 @@ mistaken for "never built". ### Crashes -The state is in the paths and the journal alone, so every step is resumable by the next run of any command that opens -the store: +The state is in the paths and the journal alone, so every step is resumable. Another command that finds an interrupted +migration stops and names `comline store migrate`, which resumes it; nothing resumes on its own. -| Interrupted | What is on disk | The next run | +| Interrupted | What is on disk | `comline store migrate` | |---|---|---| -| steps 1 to 3 | `.comline/` intact, `.comline.migrating/` partial, journal `writing` | If the journal's old heads equal the store's, it resumes: writes are content-addressed, so repeating one writes the same bytes. Otherwise the store changed meanwhile, so the partial directory is deleted and it starts again. | +| steps 1 to 3 | `.comline/` intact, `.comline.migrating/` partial, journal `writing` | If the journal's old heads equal the store's, it resumes: writes are content-addressed, so repeating one writes the same bytes. Otherwise the store changed meanwhile: it stops and names the partial directory, which the user removes, or has `--restart` remove. | | after verify, before the first rename | journal `ready` | completes step 5 | | between the two renames | no `.comline/`, a backup and a ready directory | renames the ready directory into place | | after both, before `deps-cache/` moved | new store in place, the cache in the backup | moves it | @@ -87,17 +93,27 @@ later. The existing `compat` tests treat a recomputed history the same way. ## The backup -The old store is renamed to `.comline.format-N.bak/` beside the new one. **Kept until someone removes it:** -`comline store prune` removes backups, and only when the store that replaced them passes `comline store verify`. -Nothing deletes one on its own, after a time or after a number of builds, and `build` mentions it once. +**Decided:** the old store is renamed to `.comline.format-N.bak/` beside the new one, and **kept until the user removes +it** with `comline store prune`, which removes a backup only when the store that replaced it passes `comline store +verify`. Nothing deletes one on its own, after a time or a number of builds. `comline store status` lists them. + +It is a **sibling** and not a directory inside `.comline/`: `.comline/` is committed to Git, while a sibling shows in +`git status` as untracked, so it is visible and nobody commits it without choosing to. `comline new` can add it to the +`.gitignore` it already writes for `deps-cache/`; a migration never edits anyone's files. Objects the migration didn't +change are hard links, so a backup costs little. + +## Reset + +`comline store reset` is the only way a store with no path to the current format is replaced, and it is the user's +command: -It is a **sibling** and not a directory inside `.comline/` for two reasons. The shipped binaries delete `.comline/` -on a stamp they don't expect, so a backup inside would go with it. And `.comline/` is committed to Git while a sibling -shows in `git status` as untracked, so it is visible and nobody commits it without choosing to. `comline new` can add -it to the `.gitignore` it already writes for `deps-cache/`; the migration itself does not edit anyone's files. +- It renames `.comline/` to `.comline.format-N.bak/` and starts a new store at the next build. It deletes nothing, and + `deps-cache/` is kept. +- On a terminal the user is asked to confirm **that exact action** (it names the store, the backup and the format), and + nothing proceeds without it. Without a terminal it needs `--force`. +- No other command resets. `build` on such a store stops and names this one. -*Open:* the name and place, and whether disk use warrants a default expiry. Leaning: no expiry. A store is small, -objects the migration didn't change are hard links, and the backup is the only copy of the old format. +(`comline reset`, which deletes `.comline/` outright behind a typed confirmation, is today's command and unchanged.) ## Checking first @@ -129,45 +145,25 @@ Only the touchpoints are named here. | `comline store status` | The stamp, the versions, whether a migration is needed, the unknowns per version, the backups there are. | | `comline store migrate [--check] [--assume-format N] [--map FILE]` | The procedure above. `--map` writes the old-to-new hashes, for anything outside the store that recorded one. | | `comline store verify` | The [checks](#verifying) on the current store. Also what `core#182` asks for: a damaged store says which file, and what to do. | -| `comline store prune` | Removes verified backups. | +| `comline store prune` | Removes verified backups, when the user runs it. | +| `comline store reset` | [Moves a store with no path aside](#reset), on the user's confirmation. | | `comline store replay` | *Proposed:* [rebuild from Git](steps.md#what-cannot-be-migrated). | -| `comline build`, `comline release` | Migrate first when the store is older, and say so. | +| `comline build`, `comline release` | **Stop** when the store is older, and name `comline store migrate` or `reset`; on a terminal they ask whether to run the migration, and do nothing without a yes. They never change a store on their own. | -`build` deleted the store without asking; migrating it without asking is strictly safer, and the plan is printed. *Open:* -whether the default should instead be to stop and ask. A store is committed to Git, so a migration is a large diff the -author didn't write, which is an argument for asking once. `--no-migrate`, or `COMLINE_STORE_MIGRATE=never`, makes an -old store an error for the machine that must not write. +There is no flag or variable that makes a command migrate or reset by itself: a machine that must migrate runs +`comline store migrate`. ## Git -`.comline/` is committed, so a migration is a change in a repository, and two people can make it. Three properties make -that safe, and one is open. - -- **Convergence.** A step is deterministic and the engine hashes bottom-up, so migrating the same old commit gives the - same new hashes in every clone. Two developers who migrate separately produce the same files, and Git merges them - without a conflict. This is why the receipt (`.comline/MIGRATIONS`, one line per migration) holds no time, no - author and no core version: the same on every machine, so it never conflicts either. -- **The objects merge by name.** New hashes are new files. The only files two sides can conflict on are - `FORMAT_VERSION` and `refs/heads/main`. -- **A branch that didn't migrate** brings old-format objects and its own head. `store migrate` accepts a conflicted - stamp whose sides are both supported formats and a conflicted ref (`Heads::Two`), migrates the older side's chain, - which yields the same objects the other side already has for the commits they share, and leaves the ordinary - [reconcile](../../guides/ir/cas.md#branches) to join the heads, as it does today. - -*Open:* the conflicted stamp is parsed by hand, and what a developer sees when the two sides disagree is a conflict in -a one-line file. The error needs to say what to do. Whether the engine should resolve it itself is not settled. - -### Older binaries - -A binary that predates this proposal deletes any store whose stamp isn't its own, a newer one included. Nothing in -the store can stop that, so the order of releases does: - -1. The pull request that makes a newer store an error ships in a **release first**, and the docs and the CLI say that - every machine that touches a repository has to have it before format 4 lands. -2. Even then, a binary older than the release deletes a format-4 store. Git has the history, and the backup, being a - sibling, survives it. - -*Open:* the stronger fix is to put the format-4 store somewhere an older binary doesn't look, such as `.comline/` for -format 3 and a new directory name for 4. An older binary then finds no store and starts a new one without touching the -new. It costs a rename across the docs, the CLI, the language server's exclusion list and `.gitignore`. It might be the -price of making this once and for all safe, and it can only be done at a format boundary. +`.comline/` is committed, so a migration is a change in a repository, and two people can make it. Migrating is +deterministic: a step is a pure function and the engine hashes bottom-up, so migrating the same old commit gives the +same new hashes in every clone. Two developers who migrate separately produce the same files, and Git merges them +without a conflict. This is why the receipt (`.comline/MIGRATIONS`, one line per migration) holds no time, no author and +no core version: the same on every machine, so it never conflicts either. New hashes are new files, so the only files +two sides can conflict on are `FORMAT_VERSION` and `refs/heads/main`. + +**Decided:** a Git merge of stores in two formats is **refused.** A command that finds a conflicted stamp whose sides +are different formats stops with "migrate both sides first": check out each branch, run `comline store migrate`, then +merge. Because migration converges, the merge is then the ordinary one, and the ordinary +[reconcile](../../guides/ir/cas.md#branches) joins the heads, as it does today. Nothing is resolved on the user's +behalf. diff --git a/docs/docs/proposals/ir-migrations/steps.md b/docs/docs/proposals/ir-migrations/steps.md index 65c268c..97d1318 100644 --- a/docs/docs/proposals/ir-migrations/steps.md +++ b/docs/docs/proposals/ir-migrations/steps.md @@ -73,9 +73,8 @@ An unknown has two possible carriers. **Proposed: both**, in this order of prefe `unknown` is an additive entry, a row in format 3's ledger with the absence reading *none*. A version without it has no unknowns. -**Ordering constraint.** A shipped format-3 binary would read a store that carries unknowns in the -entry as complete. So the readers of `unknown` ship, in a release, before any step that writes one. See -[the rollout](index.md#rollout). +**Ordering constraint.** A reader that doesn't know the entry would read a store that carries unknowns as complete. So +the readers of `unknown` ship before any step that writes one. See [the rollout](index.md#rollout). ### Reading an unknown @@ -89,7 +88,7 @@ no default, so an unknown can't be forgotten. Every use then has a rule, and it | a bound | the maximum | unbounded | | a type, compared | its width or length | breaking, or "can't tell" if the rule has that | | a span | where it was written | no span; diagnostics name the version instead | -| generating code or documentation | anything the output prints | refused or marked, [see format 4](format-4.md#consumers) | +| generating code or documentation | anything the output prints | **fails closed**: the generator or `doc` stops with "unknown (migrated from format N)", [format 4](format-4.md#consumers) | A comparison that read an unknown says so: it lists the version and the path in the same `unseen` list core#262 added, and its bump is at least a patch. See [history](hashes-and-history.md#history). @@ -99,16 +98,15 @@ parents, and the schema units' names. A step that can't produce them fails, and ## The witness -`down(up(v)) = v` is the proof, on every real version, that a step lost nothing. It is cheap to write for the steps -that are likely: one that wraps a value, renames a field or adds a field with an unknown has a trivial inverse, and a -step that moves a value into the `unknown` list keeps the original there so `down` can put it back. +`down(up(v)) = v` is the proof, on every real version, that a step lost nothing. **Decided: every step has one.** It is +cheap to write for the steps that are likely: one that wraps a value, renames a field or adds a field with an unknown +has a trivial inverse, and a step that moves a value into the `unknown` list keeps the original there so `down` can put +it back. A step that can't state its inverse is exactly the one to look at. -The engine runs the witness over every version of a store it migrates, before it swaps anything -([running](running.md#verifying)). The tests run it over fixtures and random values ([proof](proof.md#laws)). The -witness also has a second use, for [pins](hashes-and-history.md#pins). - -*Open:* whether every step has to provide one, or only the ones that can lose something. Leaning: every step, since -a step that can't state its inverse is exactly the one to look at. +It is checked in two places. The tests run it over fixtures and random values ([proof](proof.md#laws)). And the engine +runs it over every version of **the user's own store** that it migrates, before it swaps anything +([running](running.md#verifying)), so a store the corpus didn't predict is still checked. The witness also has a second +use, for [pins](hashes-and-history.md#pins). ## Chains @@ -123,18 +121,15 @@ catch the two disagreeing. ## What cannot be migrated Today's formats are out of scope ([index](index.md#scope-the-system-not-the-history)): they reset at 1.0. Until then, -and for any store the system can't migrate, one rule holds: **it is reset into a backup, never deleted.** +a store the system can't migrate is never touched by a command on its own. The command that finds it stops and says why; +the user decides, and the only thing that moves it is the user's `comline store reset`, which renames it to a backup. -| Store | Why | What happens | +| Store | Why | What a command does | |---|---|---| -| A store in a pre-1.0 format with no step to the current one (the unstamped bincode store, formats 1 and 2, older rows of format 3) | Nobody writes those steps. Format 0 is positional: a decode of the wrong layout can succeed and give a different variant, and nothing in the store can check a decode against. | Reset: renamed to `.comline.format-N.bak/`, a new store starts at `0.1.0-dev.1`. The command says so and names the backup. `deps-cache/` is kept | -| An unstamped store that decodes as CBOR | The stamp was lost, so the format isn't known. | Refused, unless the user gives one: `--assume-format N`. The decode must succeed for every object | -| A damaged store (a missing or corrupt object, [#182](https://github.com/ComlineProject/core/issues/182)) | A migration needs every object. | Refused, naming the file and how it failed. Nothing is reset: a damaged store is for the user to look at | -| A store newer than the binary | The binary can't read it. | Refused, naming the format and the comline that reads it. Never touched | - -A reset needs the user's say where there is history worth asking about: on a terminal it asks (default no) and -otherwise needs `--reset-old-store`, as `comline reset` does for deletion. The backup is what makes it safe to ask -less. +| A store in a pre-1.0 format with no step to the current one (the unstamped bincode store, formats 1 and 2, older rows of format 3) | Nobody writes those steps. Format 0 is positional: a decode of the wrong layout can succeed and give a different variant, and nothing in the store can check a decode against. | Stops, naming `comline store reset` ([running](running.md#reset)): the store is renamed to `.comline.format-N.bak/`, a new one starts at the next build, `deps-cache/` is kept | +| An unstamped store that decodes as CBOR | The stamp was lost, so the format isn't known. | Stops, unless the user gives one: `comline store migrate --assume-format N`. The decode must succeed for every object | +| A damaged store (a missing or corrupt object, [#182](https://github.com/ComlineProject/core/issues/182)) | A migration needs every object. | Stops, naming the file and how it failed | +| A store newer than the binary | The binary can't read it. | Stops, naming the format and the comline that reads it | A history can later be rebuilt from Git with `comline store replay` *(proposed; may not be worth building)*. It builds a **new** history with the current core, one build per release tag or commit the user chooses, never claims to be the old @@ -146,26 +141,27 @@ one, and numbers versions by the current rules. |---|---| | 1.0 and later | every format migrates to the next: that is the guarantee this system exists to give | | 3 (as it stands when the framework ships) | 3→4, the first client: it carries real stores, and exists to prove the system | -| 0 to 2, and older rows of 3 | reset at 1.0, and before it where no step exists | +| 0 to 2, and older rows of 3 | not migrated; the user resets them (`comline store reset`), and the 1.0 release refuses them | | test-only formats | in the test tree, never written by a release: [the proof](proof.md#test-only-formats) | ## Where the code lives -**Proposed:** a workspace member `comline-ir-formats` (`publish = false` at first) beside `oracle/`, holding one module +**Decided:** a workspace member `comline-ir-formats` (`publish = false` at first) beside `oracle/`, holding one module per format with its `SCHEMA.md`, types, codec and ledger, and the steps between them. It has no dependency on `comline-core`, the way the oracle has none, so that a frozen type can't reach for today's. A step `N → N+1` lives beside the frozen N+1 types. Those exist from the day N+1 is introduced, and are kept equal to the live types by the `live ≡ frozen(current)` test until N+2 seals them. -Core depends on the crate to read old stores. Test-only formats live in its test tree, not in the shipped modules. The engine and the CLI commands are in core. The crate's only -dependencies are `serde`, `ciborium`, `blake3` and `lz4_flex`, so it can be in the IR-only build, which is the one -the `wasm32` consumers use. Whether it *should* be there, since the build's point is to be -small, is open. +Core depends on the crate to read old stores. Test-only formats live in its test tree, not in the shipped modules. The +engine and the CLI commands are in core. The crate's only dependencies are `serde`, `ciborium`, `blake3` and +`lz4_flex`, so the IR-only build, which the `wasm32` consumers use, can carry it: only for in-memory, read-only +migration, never to write a store. -## Still open here +## Decided here -- **Whether `Known` is also written in formats that already exist,** that is, whether the live types of format 3 - should gain a typed unknown in the next format instead of relying on the entry. Format 4 can, and should, for what - it adds. -- **A path syntax for the `unknown` entry.** Leaning: a JSON-pointer into the model, since the frozen types already - derive serde and the path then reads the same in the format's document and in a test. +- **New formats use `Known`** for everything they add that a later step might have to leave unknown. The `unknown` + entry is for retypes that have no room. +- **The `unknown` entry's paths are JSON pointers** into the model: the frozen types already derive serde, and a path then + reads the same in the format's document and in a test. +- **The `formats` crate also carries the unknown readers and the steps for in-memory, read-only migration in the + IR-only build (`wasm`),** and never the engine that writes. From 40a7593a808337b20f0133573dd6bcdcb1c729e9 Mon Sep 17 00:00:00 2001 From: Kinflou Date: Fri, 13 Nov 2026 04:44:12 +0800 Subject: [PATCH 4/4] proposals/ir-migrations: accepted; SCHEMA.md in core is the source, unpinned acceptances are re-asked --- docs/docs/proposals/index.md | 2 +- .../proposals/ir-migrations/beyond-the-store.md | 2 +- docs/docs/proposals/ir-migrations/format-4.md | 10 ++++++---- docs/docs/proposals/ir-migrations/formats.md | 13 ++++++------- .../proposals/ir-migrations/hashes-and-history.md | 2 +- docs/docs/proposals/ir-migrations/index.md | 11 ++++------- docs/docs/proposals/ir-migrations/proof.md | 2 +- docs/docs/proposals/ir-migrations/running.md | 2 +- docs/docs/proposals/ir-migrations/steps.md | 2 +- 9 files changed, 22 insertions(+), 24 deletions(-) diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index 498b373..83e75fa 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -152,7 +152,7 @@ Seven parts, read in order: [the formats](ir-migrations/formats.md), [proving it](ir-migrations/proof.md), [hashes, pins and history](ir-migrations/hashes-and-history.md), [beyond the local store](ir-migrations/beyond-the-store.md) and -[format 4 as the first client](ir-migrations/format-4.md). *draft; tracked in +[format 4 as the first client](ir-migrations/format-4.md). *accepted 2026-11-13; tracked in [core#268](https://github.com/ComlineProject/core/issues/268).* ### Connections diff --git a/docs/docs/proposals/ir-migrations/beyond-the-store.md b/docs/docs/proposals/ir-migrations/beyond-the-store.md index e63ad2a..d55224f 100644 --- a/docs/docs/proposals/ir-migrations/beyond-the-store.md +++ b/docs/docs/proposals/ir-migrations/beyond-the-store.md @@ -1,6 +1,6 @@ # Beyond the local store -Status: **draft** · part of [Migrating stored IR across format versions](index.md) +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) **The question.** The local `.comline/` is not the only place frozen IR, or a hash of it, is kept or relied on. What does each of the others need, and what does a published package need when there are some? diff --git a/docs/docs/proposals/ir-migrations/format-4.md b/docs/docs/proposals/ir-migrations/format-4.md index 352e75e..30d917a 100644 --- a/docs/docs/proposals/ir-migrations/format-4.md +++ b/docs/docs/proposals/ir-migrations/format-4.md @@ -1,6 +1,6 @@ # Format 4 as the first client -Status: **draft** · part of [Migrating stored IR across format versions](index.md) +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) **The question.** [Granular `@unverified`](../shared-vocabulary/granular-unverified.md), item 3 of [core#222](https://github.com/ComlineProject/core/issues/222), changes what a version freezes. What does the 3→4 step @@ -81,9 +81,11 @@ that is a good model: when bounds on types arrived, `build` listed every positio defaults with maxima the author typed, never guessed (`bounds_migration.rs`). Here the offer is each acceptance, with what it now covers, and the author decides. -*Open:* whether a transition period accepts an unpinned entry with a warning. Old *dependency pins* are -[accepted with a notice](hashes-and-history.md#pins), but the safety invariant argues against doing the same here: an acceptance that isn't pinned to a -chain is exactly what the invariant is there to remove. +**Decided:** at the 3→4 transition an unpinned `accept_unverified` entry is **re-asked, not accepted**. The build stops and +names `comline accept`, which records the acceptance again with its chain. `config.idp` is never rewritten automatically. +This differs from old dependency pins, which [verify through `down` and are accepted with a notice](hashes-and-history.md#pins): +a pin that verifies through `down` proves the content is unchanged, but an unpinned acceptance proves nothing about which +causes it covered. ## Consumers diff --git a/docs/docs/proposals/ir-migrations/formats.md b/docs/docs/proposals/ir-migrations/formats.md index f0856d7..1db01d9 100644 --- a/docs/docs/proposals/ir-migrations/formats.md +++ b/docs/docs/proposals/ir-migrations/formats.md @@ -1,6 +1,6 @@ # The formats -Status: **draft** · part of [Migrating stored IR across format versions](index.md) +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) **The question.** What exactly is in a store, which of it depends on the IR format, and what does a format number promise about it? @@ -121,10 +121,9 @@ bump. The `skip_serializing_if` condition is the one the code already follows (" to the same bytes as before them"), because the bytes of a frozen unit are also what a [dependency pin hashes](hashes-and-history.md#pins). -## Decided and open here +## Decided here -- **Decided:** no second number (a ledger level) beside the format. It would only let a binary from before this proposal - refuse to write into a store whose additions it would drop, and those binaries are out of scope. -- **Still open:** whether `SCHEMA.md` in core is the source and the [CAS guide](../../guides/ir/cas.md) links to it, or - the guide stays the description of the current format and `SCHEMA.md` is generated beside it. Leaning: `SCHEMA.md` is - the source, and the guide describes the current format and says so. +- **No second number** (a ledger level) beside the format. It would only let a binary from before this proposal refuse to + write into a store whose additions it would drop, and those binaries are out of scope. +- **`SCHEMA.md` in core is the source for every format's document.** The [CAS guide](../../guides/ir/cas.md) describes only + the current format, says so, and links to the source for the others. diff --git a/docs/docs/proposals/ir-migrations/hashes-and-history.md b/docs/docs/proposals/ir-migrations/hashes-and-history.md index 42afc00..4c6fc6e 100644 --- a/docs/docs/proposals/ir-migrations/hashes-and-history.md +++ b/docs/docs/proposals/ir-migrations/hashes-and-history.md @@ -1,6 +1,6 @@ # Hashes, pins and history -Status: **draft** · part of [Migrating stored IR across format versions](index.md) +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) **The question.** A migration rewrites what hashes cover. Which hashes may change, which must not, how does a dependency pin keep meaning what it meant, and what do compatibility checks and the changelog say about a history diff --git a/docs/docs/proposals/ir-migrations/index.md b/docs/docs/proposals/ir-migrations/index.md index 5d61a45..bd49b65 100644 --- a/docs/docs/proposals/ir-migrations/index.md +++ b/docs/docs/proposals/ir-migrations/index.md @@ -1,6 +1,6 @@ # Migrating stored IR across format versions -Status: **draft** · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · feeds +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · feeds [core#146](https://github.com/ComlineProject/core/issues/146) (reading older IR after 1.0) and the [granular `@unverified`](../shared-vocabulary/granular-unverified.md) work, which is IR format 4 · seven parts, read in this order: [the formats](formats.md), [steps](steps.md), [running a migration](running.md), @@ -115,6 +115,8 @@ Settled with the user, and written into the pages: | 1.0 | a warning release, then a 1.0 release that refuses pre-1.0 stores, names the reset command, and stamps `1.0/0` | | Bridge releases | how long one is supported is decided at 1.0 | | First client | the 3→4 step of #222's chains, end to end, before 1.0 | +| Format documents | `SCHEMA.md` in core is the source for every format's document; the CAS guide describes only the current format | +| Unpinned acceptances at 3→4 | an unpinned `accept_unverified` entry is re-asked, not accepted: the build stops and names `comline accept`; `config.idp` is never rewritten automatically | Binaries from before this proposal delete a store whose stamp they don't expect. They are out of scope. @@ -157,9 +159,4 @@ names `comline store reset`, and stamps `1.0/0`. From then on every format chang ## Still open -Little is left: - -- Whether `SCHEMA.md` in core is the source for each format's document, with the CAS guide describing the current format - ([formats](formats.md#decided-and-open-here)). Leaning yes. -- Whether the 3→4 transition accepts an unpinned `accept_unverified` entry with a warning - ([format 4](format-4.md#acknowledgements)). Leaning no: the safety invariant is there to remove such an entry. +Nothing open. diff --git a/docs/docs/proposals/ir-migrations/proof.md b/docs/docs/proposals/ir-migrations/proof.md index ef4610b..09a63fa 100644 --- a/docs/docs/proposals/ir-migrations/proof.md +++ b/docs/docs/proposals/ir-migrations/proof.md @@ -1,6 +1,6 @@ # Proving a migration -Status: **draft** · part of [Migrating stored IR across format versions](index.md) +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) **The question.** "The migration is accurate" has to be a statement a test makes, for every old format and every future one, and a format change has to be unable to land without the tests that make it. What are they? diff --git a/docs/docs/proposals/ir-migrations/running.md b/docs/docs/proposals/ir-migrations/running.md index 73e9229..654ac6e 100644 --- a/docs/docs/proposals/ir-migrations/running.md +++ b/docs/docs/proposals/ir-migrations/running.md @@ -1,6 +1,6 @@ # Running a migration -Status: **draft** · part of [Migrating stored IR across format versions](index.md) +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) **The question.** How does a store change format without any moment at which history is lost, however the process ends, and how does someone find out what it would do first? diff --git a/docs/docs/proposals/ir-migrations/steps.md b/docs/docs/proposals/ir-migrations/steps.md index 97d1318..6ca491d 100644 --- a/docs/docs/proposals/ir-migrations/steps.md +++ b/docs/docs/proposals/ir-migrations/steps.md @@ -1,6 +1,6 @@ # Steps -Status: **draft** · part of [Migrating stored IR across format versions](index.md) +Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md) **The question.** What is a migration made of, so that it is accurate for every old format, never loses what a version recorded, and never claims to know what the old format did not?