Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/docs/proposals/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,24 @@ read in order:
scope that guarantee has, checked against what core computes. Revisits part
of the guarantee-marker decision. *accepted 2026-11-12; not built yet.*

#### [Migrating stored IR across format versions](ir-migrations/index.md)

How the version history in `.comline/` survives a change of the IR format
instead of being deleted: each format written down, a migration as a chain of
total, deterministic steps over frozen copies of the formats that write what an
old format never recorded as an explicit unknown, a new store verified beside
the old one before it replaces it, and a proof (a corpus of real stores built by
the historical cores, an equivalence against a fresh build, differential
property tests and a CI gate) that fails a format bump without its step.
Seven parts, read in order: [the formats](ir-migrations/formats.md),
[steps](ir-migrations/steps.md),
[running a migration](ir-migrations/running.md),
[proving it](ir-migrations/proof.md),
[hashes, pins and history](ir-migrations/hashes-and-history.md),
[beyond the local store](ir-migrations/beyond-the-store.md) and
[format 4 as the first client](ir-migrations/format-4.md). *accepted 2026-11-13; tracked in
[core#268](https://github.com/ComlineProject/core/issues/268).*

### Connections

- [Connections](connections.md): what a connection carries beyond one client
Expand Down
92 changes: 92 additions & 0 deletions docs/docs/proposals/ir-migrations/beyond-the-store.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Beyond the local store

Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md)

**The question.** The local `.comline/` is not the only place frozen IR, or a hash of it, is kept or relied on. What
does each of the others need, and what does a published package need when there are some?

## Every artifact

| Artifact | Holds | Migration story |
|---|---|---|
| **`.comline/` store** (`objects/`, `refs/`, `FORMAT_VERSION`) | every version, as frozen IR | [Migrated](running.md), whole, with the old one kept; before 1.0, where no step exists, [the user resets it](#the-10-reset) into a backup |
| **`.comline/deps-cache/`** | git checkouts of dependencies: source, not IR | Not migrated, not part of the store. **Carried across the swap**: today's wipe deletes it, and an offline machine can't fetch it again |
| **A dependency's own committed `.comline/`** | its version history; the only thing core reads of it is the newest commit's `version` (`resolved_version`) | Read with the format-independent [container reader](formats.md#what-a-store-holds). Today any failure reads as `0.0.0-unbuilt`; it should say the store couldn't be read |
| **Dependency pins** in `config.idp` | scheme v2 hash over IR bytes | Never rewritten. [Bridged](hashes-and-history.md#pins) |
| **Acknowledgements** in `config.idp`: a consumer's `accept_unverified`, a package's `unverified = { … }` | the set of guarantees accepted, per protocol | Never rewritten. [Re-asked](format-4.md#acknowledgements) when they are pinned to something the old format didn't record |
| **Generated code** | `schema_ir_hash` as the handshake's `ir_hash` | Derived, not migrated: regenerate. The hash follows the units' bytes, so a format change that alters them is visible at a connection. The pull request says so, and the [gate](proof.md#the-gate) lists it |
| **The generator protocol** | each request carries `ir_format` and the frozen units of the schemas | The units a generator gets from an old version are the migrated ones, in the current format. What to do with unknowns: [format 4](format-4.md#consumers) |
| **`comline doc` output and cache** (`.comline-doc.json`) | pages rendered per version, keyed by commit hash and the renderer | Derived. A migration changes commit hashes, so it re-renders once; the key should also hold the IR format |
| **The doc model** (`schemas/doc-model.json`) | a JSON schema, checked by a test | Derived, versioned on its own |
| **The language server, the playground, the editor** | no store; the server resolves dependencies without fetching and excludes `.comline/` from its watching | Nothing to migrate. Each ships a core, and that core's format is what it compiles against |
| **Published packages, the registry** | none yet: [no `publish` step exists](../../guides/ir/cas.md#what-to-commit), the registry server is on hold and core rejects registry dependencies | [Below](#published-packages-and-the-registry) |

There is no `.cmln` archive or equivalent in any repository. Core has a `PublishRegistry` config unit (where a package
publishes) and nothing that builds an archive.

## Published packages and the registry

When there is something to publish, the [offline proposal](https://github.com/ComlineProject/docs/pull/163) already
decides that **identity is the hash and location is replaceable**: a dependency is its pin, and caches and bundles are
keyed by it. The rules for IR follow from that.

- **Migrate on read, never rewrite what is published.** A published archive is addressed by its hash, and perhaps
signed. Rewriting it is making another artifact. A consumer that meets an archive whose store is in an older
format reads it through an in-memory migration, as `comline diff` reads an old local store. Its result, if cached, is
cached beside the archive and keyed by the archive's hash and the target format.
- **An archive says its format outside its payload,** in its manifest, so that a consumer can refuse a newer one
without decoding it, with the same error as a local store: the format, and the comline that reads it.
- **What a consumer needs from a dependency is its source.** A dependency is compiled from source, and its pin is checked
against that compile. The stored IR is only for comparing versions (compatibility, the changelog). So a version
too old to migrate costs a comparison, "can't tell", and never a build.
- **The registry holds, and never migrates.** It serves old archives as they were published. Migration is the
consumer's, in the consumer's core.
- **A pin never depends on which format an archive's store is in.** That is the requirement the
[pin bridge](hashes-and-history.md#pins) meets.

## The 1.0 reset

**Decided:** the formats that exist today are all reset when 1.0 arrives, and the numbering restarts: the 1.0 format is
**format 0** of a new numbering, and the first with a guaranteed path forward. Nothing before it is migrated. And, like
everything here, **nothing happens on its own**:

1. **A warning release first.** The release before 1.0 says, once per command that finds a pre-1.0 store, that 1.0 will
refuse it and which command to run. It changes nothing.
2. **The 1.0 release refuses a pre-1.0 store.** Any command that finds one stops, names `comline store reset`, and
changes nothing. The reset is the same [user command](running.md#reset) as before 1.0: it renames the store to
`.comline.format-N.bak/` after the user confirms that exact action, keeps `deps-cache/`, and starts a new store at
the next build, stamped format 0 of the 1.0 numbering. The backup stays until `comline store prune`.
3. **The stamp has an epoch.** Pre-1.0 stamps are the integers 0 to 4, and a 1.0 `0` or `3` must not be read as one of
them. So the 1.0 stamp reads `1.0/0`, and a bare integer is by definition pre-1.0.
4. **The frozen types of the pre-1.0 formats are not kept:** the refusal needs only the stamp and the container, which
are format-independent ([formats](formats.md#what-a-store-holds)).
5. Every format change after it is a step, with a document, fixtures and a seal, as the gate requires. The system is
already in use by then: format 4's step is its first run.

## After 1.0

[core#146](https://github.com/ComlineProject/core/issues/146) asks that from 1.0 a newer comline reads what an older one
wrote, "either freeze them or keep a reader for every older version, with tests". This proposal is the second: the
framework *is* the readers, and its fixtures and laws are the tests.

| | Before 1.0 | From 1.0 |
|---|---|---|
| **Formats that migrate** | none promised: format 3 for the first client, the rest reset | every format released at or after 1.0, for ever. A released format's frozen types, steps, fixtures and seal are never removed |
| **A store with no path** | refused; the user's `comline store reset` moves it to a backup | refused, and the error names the **bridge release**: the last comline that still migrates it. There is no reset |
| **Dropping a format** | any minor release | a major one, announced one minor earlier; the bridge release stays installable |
| **Cost of a bump** | a step and its fixtures | the same, **and permanent**: it is carried for ever |
| **Additive changes** | a [ledger](formats.md#format-3-is-a-family) row each | a ledger row each, a fixture, and the same lines in the frozen copy |

Because each bump is permanent from 1.0, bumps are worth batching: the policy proposed is that a format bump is
planned, not incidental. Cheap changes are additive ones, and the gate makes those cheap.

A *version* is not a format. Package version 0.3.0, built long ago in a format that still migrates, is migrated like
every version of its store: nothing is dropped, and it can be compared at the precision it recorded, nothing finer.

The spec side needs nothing of this: the spec is append-only for anything frozen
([the vocabulary](../shared-vocabulary/catalog.md#versioning)), so an old version reads under a newer spec without a step.
Only the IR format does.

## Decided here

- **How long a bridge release stays supported is decided at 1.0,** not before.
108 changes: 108 additions & 0 deletions docs/docs/proposals/ir-migrations/format-4.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
# Format 4 as the first client

Status: **accepted** · decided 2026-11-13 · tracked in [core#268](https://github.com/ComlineProject/core/issues/268) · part of [Migrating stored IR across format versions](index.md)

**The question.** [Granular `@unverified`](../shared-vocabulary/granular-unverified.md), item 3 of
[core#222](https://github.com/ComlineProject/core/issues/222), changes what a version freezes. What does the 3→4 step
do with every format-3 version already stored, and what is the package author asked?

Format 4 is not designed here. It is the shape that proposal accepted, and this page is what the step does with it.
Anything that depends on the final shape is marked.

**It is the first real client of the system,** and exists to prove it before 1.0, because today's formats won't be
migrated ([scope](index.md#scope-the-system-not-the-history)). One pull request exercises, on a real change: the frozen
copy and ledger, a step with `up`, `down` and an `unknown` list, a typed unknown, the square against a fresh format-4
build, the laws, the pin bridge, and the gate. The test-only formats of [the proof](proof.md#test-only-formats) cover
what one step can't (composition, an untyped unknown); the two together are the system's proof. A format-3 store that
exists when the framework ships is migrated for real (when the user runs `comline store migrate`), and one older than that is not migrated: the user resets it with `comline store reset`.

## What changes

Format 3 freezes what core worked out as a **set**, and where it was written:

- a protocol's `parameters` hold `Unverified(Vec<Guarantee>)` beside the `@unverified` reason, and, since
[#265](https://github.com/ComlineProject/core/pull/265), markers on arguments, functions, fields and types are
properties in *their* parameters;
- the package's own `unverified` is a config unit, a `ConfigValue` as written;
- a declared encoding records what it `establishes` and whether it `has_vectors`.

Format 4 freezes, per the decision, **(guarantee, scope path, chain to each cause, reason)**, so that a comparison sees
narrowing and widening and an acknowledgement is pinned to what it covered.

## The step

| Item | In format 3 | In format 4 | What `up` writes |
|---|---|---|---|
| each given-up guarantee | a member of a protocol's set | a record per guarantee | one record per member; the `guarantee` is carried |
| its **scope** | where the marker is written: the protocol, or an inner property since #265 | a scope path | the path of the marker that covers it, read from the stored units. Before #265 only a protocol could carry one, so the protocol is **known**, not guessed |
| the **chain** to each cause | not stored | the hops, package by package | **`Unknown`**, typed, with the reason "format 3 records the set, not the path to its causes" |
| its **reason** | one string on the plain marker, or a dictionary as written | a reason per guarantee | a plain marker's string is copied to each guarantee it covers, as written; a dictionary's entries as written |
| a package-default cause, acknowledged in `config.idp` | the `Unverified` config unit, as written | pinned to the setting that causes it | the unit carried; its pin to the setting is **`Unknown`** |
| a declared encoding's guarantees | `establishes`, `has_vectors`, as they were (the strict reading where absent) | the same | carried |

Two things it does **not** do. It doesn't compute the chains: that needs the guarantee engine and the catalog, which
are live code and change, and a step is frozen. And it doesn't write an empty chain list or a one-hop chain to a
guarantee whose causes weren't recorded; that would read as "no other cause".

The step is `up`, `down` and an `unknown` list, like any. `down` drops the typed unknowns and the scope paths that
equal the old protocol-level reading, and restores the old set, so the witness holds. The `unknown` list is the
two rows above, and the square compares the migration with a fresh format-4 build whose chains and package pins are
forgotten, with each scope that the old format could place left alone.

## Chains in the old versions

Recomputation, not the stored record, answers any question that needs a chain. A version stores its inputs (types,
a dependency's schemas and, since #262, its config), and the current core can compute the chains from those under
the current spec ([history](hashes-and-history.md#history)). So:

- `diff` and the changelog **recompute chains for both versions** and compare those. A migrated version compared with an
unchanged successor shows no change, even though its stored chain is unknown.
- The chains stay unknown only where an input is missing: a dependency's config in a version older than #262, a type
whose information an older format never kept. Those are `unseen`, as everything else that can't be seen is
([history](hashes-and-history.md#what-a-comparison-does)).
- "Narrowing a marker isn't a major change; widening one or adding a cause is" is a rule over *recomputed* causes for
these versions, so it needs no stored chain to hold.

## Acknowledgements

Three things are called an acknowledgement, and the migration treats them differently, because only one of them is in
the store:

| | Where it is | At format 4 |
|---|---|---|
| **The author's marker** (`@unverified …`) | the source | Unchanged, and still valid: the plain form acknowledges what core computes at its scope. The frozen record of it is rebuilt by each build |
| **A consumer's acceptance** (`accept_unverified = { Billing = […] }`) | the consumer's `config.idp` | Pinned to chains in format 4. An existing entry is a set and has no pin, so it matches nothing, and the build **asks again**: it refuses, names each protocol, shows what is now recorded (the guarantee, the chain, the origin's reason), and prints the entry. `comline accept` is the existing way to write it |
| **A package's own `unverified = { … }`** | its `config.idp`, next to the setting | Pinned to the setting. The same re-asking applies, once, for a package that wrote it |

This is the intended behaviour of [core#222](https://github.com/ComlineProject/core/issues/222): "a new cause, or a swapped
type upstream, asks again at every hop". A format change that makes every existing acceptance ask once is the same
event, because the pin it lacked is the pin the invariant needs. **Nothing is rewritten.** The CLI has a precedent
that is a good model: when bounds on types arrived, `build` listed every position, and the CLI *offered* package
defaults with maxima the author typed, never guessed
(`bounds_migration.rs`). Here the offer is each acceptance, with what it now covers, and the author decides.

**Decided:** at the 3→4 transition an unpinned `accept_unverified` entry is **re-asked, not accepted**. The build stops and
names `comline accept`, which records the acceptance again with its chain. `config.idp` is never rewritten automatically.
This differs from old dependency pins, which [verify through `down` and are accepted with a notice](hashes-and-history.md#pins):
a pin that verifies through `down` proves the content is unchanged, but an unpinned acceptance proves nothing about which
causes it covered.

## Consumers

What reads a version with unknown chains, and what each does:

| Consumer | With unknown chains |
|---|---|
| `build`, the guarantee check | recomputes from inputs; the unknown is never what it judges on |
| `diff`, `doc --version`, the changelog | recompute; list what is `unseen` |
| `inspect` | shows the recomputed chain, marked "recomputed, not recorded" |
| the generated `@unverified` docs ([generation#93](https://github.com/ComlineProject/generation/issues/93)) | the chain as recomputed. Where it can't be, it fails closed as below |
| a generator, or `doc`, given an old version | **fails closed**: it stops with "unknown (migrated from format N)" for what it would print from an unknown, naming the version and the path. A generator is never handed a placeholder that looks complete |

## Fixtures and tests for 3→4

The corpus ([proof](proof.md#the-fixture-corpus)) needs, for this step: the chain graphs of the oracle, with the inner scopes
(`66feecb`) and the package-scope `unverified` (`9ea142e`), built by format 3's closing core. The square is then: the format-3 store of a graph, migrated, against the format-4 build of
the same graph with chains and package pins forgotten. The differential test draws random graphs and histories from
the oracle's generators, the same ones that test the invariant, so the property that no guarantee goes
unacknowledged is checked over migrated stores too (the cross-package chains part of stage 2 of the rollout).
Loading
Loading