diff --git a/docs/docs/proposals/auto-documentation.md b/docs/docs/proposals/auto-documentation/index.md similarity index 82% rename from docs/docs/proposals/auto-documentation.md rename to docs/docs/proposals/auto-documentation/index.md index 28d7309..b14c51b 100755 --- a/docs/docs/proposals/auto-documentation.md +++ b/docs/docs/proposals/auto-documentation/index.md @@ -18,10 +18,10 @@ Two sources feed it, and the first one alone is already enough to produce something useful: - **The declared shape**, straight out of the frozen IR: unit names, field names - and [types](../guides/idl/types.md), enum variants, function signatures, error + and [types](../../guides/idl/types.md), enum variants, function signatures, error shapes, validator parameters, `@key=value` annotations. An entirely undocumented schema still yields a complete API listing. -- **[Docstrings](../guides/idl/docstrings.md)** — `///` on a declaration or a +- **[Docstrings](../../guides/idl/docstrings.md)** — `///` on a declaration or a field, the `/// @name:` form for fields and arguments, and `//!` headers documenting a module or (in `config.idp`) the package itself. These exist today and, as that page notes, change nothing about a build, a version or @@ -36,7 +36,7 @@ modules. Versioning is not hand-maintained: every `comline build` freezes an immutable version into `.comline/`, the bump is derived from the schema change itself, and -[`comline diff`](../reference/tools/cli/index.md) already reports what changed between two +[`comline diff`](../../reference/tools/cli/index.md) already reports what changed between two frozen versions. A documentation tool built on top of that can do what a conventional one cannot: @@ -51,14 +51,14 @@ conventional one cannot: This is the half that distinguishes the feature from "run a generator over the IR", and it follows from the -[CAS](../guides/ir/cas.md) and -[automatic versioning](../reference/versioning.md) already being in place. +[CAS](../../guides/ir/cas.md) and +[automatic versioning](../../reference/versioning.md) already being in place. ## Optional, and the project sets the rules It is a tool a project opts into, not a step in the pipeline. Nothing here happens unless asked for. The consumer's -[`comline.toml`](../reference/package/congregation/comline-toml.md) owns the switches, +[`comline.toml`](../../reference/package/congregation/comline-toml.md) owns the switches, the same way it already owns generation's output location, layout and mode. ## Decisions @@ -72,7 +72,7 @@ These were open questions until 2026-11-01. The rationale is in are per language, and documentation is not. - **Config surface: `comline.toml` only.** Nothing about docs is frozen into a version. Whether and how docs are made is the reader's choice, as with - [generation config](../design/consumer-generation-config.md). The author's + [generation config](../../design/consumer-generation-config.md). The author's part is the docstrings themselves. - **Output: a JSON model, rendered by separate programs.** - A versioned JSON **doc model** is the one contract, with its JSON Schema @@ -81,9 +81,9 @@ These were open questions until 2026-11-01. The rationale is in type that names another unit linked to where that unit is declared, each protocol's wire summary from `comline inspect`, and the changelog. - **Renderers** are external programs on the same stdio protocol as - [generators](../reference/generator-protocol.md), native or `.wasm`. + [generators](../../reference/generator-protocol.md), native or `.wasm`. - The first official renderer writes **Markdown with navigation** for - mkdocs/zensical sites. Rendering [`std`](../guides/idl/use.md#the-standard-library) + mkdocs/zensical sites. Rendering [`std`](../../guides/idl/use.md#the-standard-library) onto this site proves it. - **History: any version, and a changelog.** Any frozen version can be documented. The changelog comes from the same analysis as `comline diff`. @@ -94,7 +94,10 @@ These were open questions until 2026-11-01. The rationale is in - **Dependencies: linked when they say where their docs are.** A unit's docs link to a type it imported from another package when `comline.toml` maps that dependency to the URL of its docs. Without a mapping, the type shows as plain - text. + text. *Being extended* in + [Links, the overview and highlighting](links-overview-highlighting.md): a + dependency can declare its own docs URL, which amends "`comline.toml` + only" above. - **Templating: the renderer's business.** Theming belongs to each renderer. Core keeps no template surface, and the model is the only thing it keeps stable. @@ -112,11 +115,11 @@ These were open questions until 2026-11-01. The rationale is in It lands in the product documentation, not here: a Guide page on documenting your schemas, a `comline doc` entry in the -[CLI reference](../reference/tools/cli/index.md), its `comline.toml` keys in the -[congregation reference](../reference/package/congregation/index.md), and the doc +[CLI reference](../../reference/tools/cli/index.md), its `comline.toml` keys in the +[congregation reference](../../reference/package/congregation/index.md), and the doc model's JSON Schema. The obvious first demonstration is -[the standard library](../guides/idl/use.md#the-standard-library), which already +[the standard library](../../guides/idl/use.md#the-standard-library), which already documents itself with `//!` and `///` throughout: run the tool over `std` and publish the result. diff --git a/docs/docs/proposals/auto-documentation/links-overview-highlighting.md b/docs/docs/proposals/auto-documentation/links-overview-highlighting.md new file mode 100644 index 0000000..7a8cf4b --- /dev/null +++ b/docs/docs/proposals/auto-documentation/links-overview-highlighting.md @@ -0,0 +1,308 @@ +# Links, the overview and highlighting + +Status: **draft** · tracked in +[generation#144](https://github.com/ComlineProject/generation/issues/144) · a follow-up to the accepted +[Automatic documentation](index.md) · decided so far (2026-11-14): a +dependency declares where its docs are, with local pages kept beside it; the +overview shows the package's whole configuration, a private registry by name +only; built-ins link to the reference for the toolchain version that built +the package · it +**amends** the decision that docs config lives in `comline.toml` only (see +[below](#the-amended-decision)) + +**The question.** The renderers (`comline-doc-html`, `comline-doc-markdown`) +produce a working site. **How does it become easier to read and to move +around in?** That means four things: + +- links to a dependency's docs wherever they're published; +- an overview that shows how the package is configured; +- built-in types and values that link to their reference; +- highlighting everywhere code appears, not only in signatures. + +## Where things stand + +- **Dependency links are manual.** A dependency's types link to its published + docs only when the *consumer* maps it in `comline.toml` + (`[doc] links = { billing = "https://…" }`). `std` defaults to its hosted + docs (`STD_DOCS`). Every other dependency gets local pages + (`deps//.html`) holding only what the package reaches. +- **The model can't say more.** A `Dependency` carries its name and its + reached modules: no version, no source (registry, git, path), no docs + location. Registries don't record a docs location either. +- **The overview shows no configuration.** `index.html` has the title, the + `//!` header, Modules, Dependencies and a changelog link. The model's + `Package` is `name`, `version` and `docs`. Core's model builder + (`core/src/doc.rs`, `model()`) has the whole project config at hand, but + copies none of it. +- **Built-ins link nowhere.** A primitive (`Type::Primitive`) and the + literals `true`/`false` are plain text. The page they'd link to, + `reference/package/schema/types`, being written in the reference + restructure, is mostly empty headings (`## bool` has no text, and there's nothing on + floats or `()`). Its anchors don't match its names either: the guide links + to `types#string`, which doesn't exist. +- **Highlighting follows two paths.** The HTML renderer draws a type with + `ty_idl()` (highlighted) in signatures, constants and the module table, and + with `ty()` (a plain ``) everywhere else: field and argument tables, + a validator's "Applies to", "Alias of". Annotations, defaults, asserts, + messages, enum variants and settings are plain ``. The tokenizer + also treats `i8` to `i64` and `bytes` as built-ins, which the language + doesn't have. + +## 1. Links to a dependency's docs + +### Where the URL comes from + +In order, the first that applies: + +1. **The consumer's `[doc] links`**, as today: an override, for a mirror or + for docs the dependency doesn't declare. +2. **The dependency's own declaration** (decided): a `documentation` key in + *its* `config.idp`. It travels with the package, so it works the same + from a public registry, a private one, git or a path. +3. **The registry**, later: one that hosts docs itself (in the way docs.rs + does) states where in its API. Nothing reads this until a registry + offers it. +4. **No declaration:** local pages only, as today, so a link never leads + nowhere. + +`std` is the first case of 2: its declaration is the hosted docs, and +`STD_DOCS` goes away as a special case. + +### The declaration + +```idp +documentation = "https://docs.example.com/billing/{version}/{module}/#{anchor}" +``` + +A template with three placeholders: + +| Placeholder | Filled with | +| --- | --- | +| `{version}` | the version the consumer pins | +| `{module}` | the module's path, `::` written as `/` | +| `{anchor}` | the unit's anchor, its name in lowercase; dropped with the `#` before it for a link to a module | + +- **Without placeholders**, the value is a base, read as + `{base}/{module}/#{anchor}`: today's form for `[doc] links`, so existing + entries keep working. +- **The pinned version, not the latest.** A link from a package built against + `billing 1.2.0` opens `billing`'s 1.2.0 docs, which still describe the type + the package uses. A template without `{version}` links to whatever the + dependency publishes there, and the build warns. +- **Releases only.** A consumer pinned to a dependency's pre-release or dev + build gets local pages for it. That version's docs were never published. +- **Local pages are kept beside it** (decided). The site still renders the + dependency's pages for what the package reaches, so the docs read in full + offline. A type links to its local page, and each local page and the + overview's Dependencies entry link out to the published docs at the pinned + version (`↗ published docs`). The site works offline, and the full, + canonical docs are one click away. +- **Never fetched.** The build doesn't check that the URL answers, so docs + stay buildable offline. If the docs are private, the browser asks for the + login. A `comline doc --check-links` that tries every external link could + come later. + +### The amended decision + +The accepted proposal decided that "nothing about docs is frozen into a +version: whether and how docs are made is the reader's choice". The +`documentation` key keeps to the intent of that decision. **Where** published +docs live is a fact about the package, like its id, and only its author knows +it. **Whether and how** to render docs stays the reader's choice, in +`comline.toml`. The decision is amended to say so: + +> Rendering is configured by the reader, in `comline.toml` only. A package +> may state where its own published docs are, in `config.idp`. + +Changing `documentation` changes no wire and no API, so it's a patch. + +### In the model + +`Dependency` gains, additively (the model stays version 1): + +| Field | What | +| --- | --- | +| `version` | the pinned version | +| `source` | where it comes from: a registry's name, a git URL, or a path | +| `documentation` | the dependency's declared template, if any | + +Core fills them while compiling the dependency, which it already does. The +renderer applies the order above, with the consumer's `[doc] links` passed as +today. + +## 2. The overview + +### Its order + +1. **The title**, version and `//!` header, as today. +2. **Package:** its id and edition (`specification_version`). +3. **Settings:** the `config.idp` `settings`, then the package's wire + defaults (format, framing, handshake), each with where it comes from + (`config.idp` or core's default), as a protocol's wire summary shows them. +4. **Dependencies:** each with its pinned version, its source, a link to its + docs (by the order above), and the modules the package uses. +5. **Modules.** +6. **Code generation:** the languages and versions declared in + `code_generation`. +7. **Registries:** the registries in `publish_registries`, each by name, and + with its URL unless it's private ([below](#private-registries)). +8. **Changelog.** + +Code generation and registries are included (decided), after what a reader +of the API needs first. They say how the package is built and published, not +what it declares. + +### In the model + +`Package` gains, additively, a `config` read from `config.idp`: + +| Field | What | +| --- | --- | +| `id` | the package id | +| `edition` | the specification version | +| `settings` | the settings tree: key paths and values | +| `wire` | format, framing and handshake, each a value, its parameters and its source | +| `code_generation` | each language and its version | +| `registries` | each registry's name, and its URL unless it's private | + +### Private registries + +A private registry's URL can reveal an internal host name, so the overview +shows it by **the registry's visibility** (decided), which the author states +in `config.idp`: + +```idp +publish_registries = { + public = { url = "https://registry.comline.dev" } + internal = { url = "https://registry.corp.example", private = true } +} +``` + +A private registry shows its name only, and a public one its name and URL. The +model carries no URL for a private registry at all, so no renderer can show it +by mistake. A registry is public unless marked `private`. + +Like the `//!` header, it's read from the source, so documenting an older +version needs that version's `config.idp`. The proposal already notes the +same limitation [for headers](index.md#where-the-docs-come-from). + +## 3. Built-ins link to the reference + +**The target:** the hosted +types reference (`reference/package/schema/types`), with **one +anchor per built-in**, the name itself: + +| Written | Links to | +| --- | --- | +| `bool`, `true`, `false` | `#bool` | +| `u8` … `u128`, `s8` … `s128` | `#u8` … `#s128` | +| `f32`, `f64` | `#f32`, `#f64` | +| `char` | `#char` | +| `string` | `#string` | +| `()` | `#unit` | +| a bound, `[max N]` | the [bounds](../../guides/idl/types.md#bounds) section | + +- **The page comes first.** It needs real content and these anchors before + anything links to it. The guide's broken `types#string` link gets fixed by + the same work. +- **The toolchain's docs** (decided). A built-in links to the reference of + the **Comline version that built the package**. The model records it, as a + new top-level `toolchain` field (additive). The hosted base becomes a + template, `…/{toolchain}/reference/package/schema/types/#{anchor}`. This + needs **hosted docs versioned by toolchain release**, which the docs site + doesn't do yet (see [What it touches](#what-it-touches)). Until it does, + links go to the current docs, and the page says which toolchain built it. +- **Not by edition.** Editions were weighed and rejected. An edition scopes + only the schema language, so versioning the site by edition would either + branch the whole site for a change to one part of it, or split the language + reference into folders per edition. Toolchain releases are the axis the + rest of the site (CLI, generators, runtimes) versions by anyway. +- **The base is overridable** with `[doc] builtins = "…"`. That's its own key, + because `links` is keyed by package and `builtins` could be one's name. +- **Encodings and guarantees too.** A protocol's wire summary prints + `datagram@3`, `packed@1` and `comline@2`. Each links to its + [reference page](../../reference/wire/official-encodings.md), and each + guarantee row to where the guarantee is explained. +- **Quiet links.** A link inside highlighted code keeps the token's colour, + with no underline until hovered. Otherwise every table of types turns into a + column of links. + +## 4. Highlighting everywhere code appears + +**One renderer for types.** `ty()` becomes `ty_idl()` in a +``, so every type is highlighted and linked the same way, +wherever it appears. That alone fixes a validator's "Applies to", the field +and argument tables, and "Alias of". + +Then each place that shows code, through the same tokenizer: + +| Where | Today | Becomes | +| --- | --- | --- | +| Validator "Applies to" | plain `bool`, `Test`, `s8` | highlighted, linked | +| Field and argument **Type** columns | plain; `union(` is bare text, outside `` | highlighted, linked | +| "Alias of" | plain | highlighted, linked | +| A field's default, `= true` | plain | a literal, linked when `true`/`false` | +| "(optional)" after a type | plain text | the `optional` keyword | +| Annotations, `@timeout_ms = 100` | plain | the annotation and its value | +| Validator asserts, `value > 1` | plain | an expression: names, operators, literals | +| Validator and error messages, `{value.name}` | plain text | text, with each `{…}` marked as an expression | +| An error's ordinal | plain | a literal | +| Enum variants | plain | names | +| A settings unit's keys and values | plain | a key path, and a literal or name | +| Unit headings and the module table's Item, `struct Greeting` | the kind is plain text | the kind as a keyword | +| The wire summary's encodings | plain | linked ([above](#3-built-ins-link-to-the-reference)) | + +**The tokenizer** drops `i8` to `i64` and `bytes` from its built-ins. It +also gains a token for operators and one for an interpolation in a message. + +**Markdown** already highlights its fenced blocks (` ```comline `), but +plain Markdown can't highlight inline code. Sites built with mkdocs and +`pymdownx.inlinehilite`, as Comline's own docs are, can, with `` `#!ids bool` ``. +The Markdown renderer gains an option for it, +`[doc.markdown] inline_highlight = true`, off by default because anywhere +else `#!ids` would show as text. + +## What it touches + +- **core:** the model's `Package.config` and the `Dependency` fields; the + `documentation` key in `config.idp`; the doc model's JSON Schema. +- **cli:** `[doc] builtins` and `[doc.markdown] inline_highlight`; passing the + pinned versions; later, `comline doc --check-links`. +- **generation:** both renderers: the overview's order and sections, the + link order for dependencies, built-in and encoding links, one type renderer, + and the highlighting table above; the tokenizer's fixes. +- **docs:** the types reference written out with an anchor per built-in; the + guide's broken link; the [documenting guide](../../guides/documenting.md) + and the [`comline.toml` reference](../../reference/package/congregation/comline-toml.md) + for the new keys; `std` declaring its `documentation`; and **hosted docs + versioned by toolchain release**, so that built-ins link to the reference of + the toolchain that built a package. That's a site change of its own, worth + its own proposal. `mike` is already configured in `mkdocs.yml`, but the + deploy (`zensical build`, then `actions/deploy-pages` with an artifact) + doesn't use it. +- **core:** `private` on a `publish_registries` entry. + +## Decided + +2026-11-14: + +- **A dependency declares its docs** in its `config.idp`, as a template + ([above](#the-declaration)), and local pages are kept beside it, for + reading offline. +- **The overview shows the whole configuration,** code generation and + registries included. A private registry shows by name only. +- **Built-ins link to the reference of the toolchain that built the + package,** once the docs site is versioned by toolchain release. Not by + edition. +- **`std` links by its own pinned version,** like any dependency: its + `documentation` template carries `{version}`. +- **Design first:** implemented once this page is accepted. + +## Open questions + +- **Versioned hosted docs:** how the docs site publishes one version per + toolchain release: `mike` on a `gh-pages` branch, or a job that builds + release branches or tags into one artifact. It's needed once a toolchain + release line is worth keeping docs for. +- **The amended decision** ([above](#the-amended-decision)): approving the + new wording of the accepted proposal's config decision. diff --git a/docs/docs/proposals/connections.md b/docs/docs/proposals/connections.md index 763aba3..3b64636 100644 --- a/docs/docs/proposals/connections.md +++ b/docs/docs/proposals/connections.md @@ -22,7 +22,7 @@ directions, per-connection state, and several protocols on one connection. starts with a `u16` call id and a reply with a `u64` request id, so an unprompted frame from the server would be misread as a reply. - **Nothing generates server-to-client protocols.** `@provider = Client` - ([packet dispatch](packet-dispatch.md#direction-provider-and-consumer)) has + ([packet dispatch](packet-dispatch/index.md#direction-provider-and-consumer)) has no runtime to run on. So server push is three gaps: a frame kind, something reading the connection @@ -66,7 +66,7 @@ Decided with the user (2026-10-20): - **Broadcasting** to many connections stays application code: a loop over connections. 5. **What the wire needs:** a request/reply kind marker, as decided for - [request ids in both directions](packet-dispatch.md#request-ids-in-both-directions). + [request ids in both directions](packet-dispatch/index.md#request-ids-in-both-directions). - `jsonrpc@1` works unchanged: a message with `method` is a request. - Datagram needs a new version, `datagram@2`, with the marker. Reverse calls under `datagram@1` are a build error. @@ -104,7 +104,7 @@ Today a `Server` serves one `Dispatch`. A request names only its function, by position (datagram) or bare name (JSON-RPC), and the handshake covers one schema. Error ids are positions across a whole schema, so two schemas sharing a connection could give two different errors the same number. -[Packet dispatch](packet-dispatch.md) already assumes several protocols per +[Packet dispatch](packet-dispatch/index.md) already assumes several protocols per connection: its families are protocols with an id, and its phases are per connection. @@ -231,6 +231,6 @@ Not built yet: senders, the connection handle in handlers, `@provider = Client`, error ids scoped by protocol. - **Core:** `@provider` as a known annotation (from - [packet dispatch](packet-dispatch.md)), the build error for reverse calls + [packet dispatch](packet-dispatch/index.md)), the build error for reverse calls under `datagram@1`, and the optional declared set of protocols. - **Docs:** the runtime guide. diff --git a/docs/docs/proposals/dependency-protocols/capabilities.md b/docs/docs/proposals/dependency-protocols/capabilities.md new file mode 100644 index 0000000..44a1b98 --- /dev/null +++ b/docs/docs/proposals/dependency-protocols/capabilities.md @@ -0,0 +1,356 @@ +# Part 1 — what framings and handshakes state + +Status: **accepted** (2026-11-14), with the `one_way` fact and `limits` · tracked in +[core#276](https://github.com/ComlineProject/core/issues/276) · a +prerequisite of the [proposals](index.md) in this section · extends the [encoding profiles](../schema-compatibility.md#encoding-profiles) +of Schema compatibility + +**The question.** A profile states facts about its encoding, and the toolchain +turns those facts into decisions: which bump a change takes, what a build +refuses, and which guarantees a protocol keeps. **Do today's facts say +everything a framing and a handshake can and can't do?** They don't, and +carrying a dependency's protocols on a connection depends on the parts they +leave out. + +## What profiles state today + +| Fact | Applies to | Values | What reads it | +| --- | --- | --- | --- | +| `calls` | framing | `by_index`, `by_name`, `by_id` | the comparison: inserting or reordering functions | +| `directions` | framing | `both`, `one` | the build: `@provider = Client` / `Multiple` needs `both` | +| `errors` | framing | `by_ordinal`, `by_name` | the comparison: inserting or reordering errors | +| `payload` | framing | `opaque`, `json` | the build: the format's `output` must fit it | +| `schema_agreement` | handshake | `true`, `false` | guarantees: schema agreement | +| `encoding_agreement` | handshake | `true`, `false` | guarantees: encoding agreement | + +## What they leave unsaid + +Each gap below has a concrete consequence today. + +1. **How a frame names its protocol.** `calls` says how a request names its + *function*. Its `by_id` value is described as "by the protocol's and + function's wire numbers", so one fact answers two questions, and only for + one framing. Nothing states that `datagram@2` names a protocol by its + *position in the package*. As a result, the comparison has no rule for + inserting or reordering protocols. Under `datagram@2`, adding a protocol + to an earlier schema silently re-addresses every protocol after it, and + the version doesn't say so. +2. **What a protocol's address is unique in.** A number unique in one + package, one unique in a connection, and a name unique in a package can't + be told apart. This is the question this section exists for: whether two + packages' protocols can share a connection. +3. **How many packages a handshake covers.** `comline@1` hashes one + package's IR, and `comline@2` sends one package id. No fact says so. The + rule only shows up as a refusal (`package_mismatch`) at run time. +4. **Whether a handshake binds anything.** A framing whose numbers come from + the handshake can only run under a handshake that hands them out. No fact + lets the build check that pairing. +5. **What a handshake compares.** `comline@1` compares identity (one IR + hash), and `comline@2` compares compatibility (lines, shapes, numbers). + Both state `schema_agreement = true`. What that means for upgrades isn't + stated: under identity, every peer must be rebuilt from the same schemas, + while under compatibility a compatible older peer still connects. +6. **What a refusal takes down.** `comline@1` refuses the connection. + `comline@2` refuses one protocol or the connection, depending on its `scope` + parameter. Generated code has to handle a protocol being `Unavailable` only + in the second case, and nothing states it. +7. **What a change of encoding implies.** Changing a framing or handshake is + major, and moves every line on it. Nothing says what the package gains or + loses by the change, or what it newly requires, such as build history or + a package `id`. + +## Proposed facts + +Same rules as the existing ones: a small, fixed set, each value proven by +conformance vectors, fixed for a version, and settable by a parameter where the +encoding has one. + +### Framing + +| Fact | Description | Values | +| --- | --- | --- | +| `calls` (narrowed) | How a request names the **function** it calls, within its protocol. | `by_index`: its position · `by_name`: its name · `by_id`: its wire number | +| `protocols` (new) | How a request names the **protocol** it calls into. | `unnamed`: it doesn't, so the connection carries one protocol · `by_index`: its position among its package's protocols · `by_name`: its qualified name, `namespace.Protocol` · `by_id`: its wire number in its package · `by_binding`: a number the handshake binds for the connection · `in_call`: it doesn't, but the function's id is unique across every protocol on the connection, so it picks out the protocol too (a flat opcode table) | +| `correlation` (from [Packet dispatch](../packet-dispatch/index.md#comlines-own-packet-framing)) | How a reply is matched to its request. | `none`: there are no replies · `header_id`: by an id in the header | +| `one_way` (new) | How a frame tells apart a call that gets no reply. | `by_schema`: it doesn't; both ends know from the function's declaration · `in_frame`: the frame says so (a request kind, a notification) | +| `handles` (new, from [IPC](../ipc.md#who-ties-handles-to-a-frame)) | How a frame's OS handles are tied to it. | `none`: the framing can't carry any · `by_transport`: the transport delivers them with the frame · `counted`: the header counts them · `indexed`: the frame lists where each one is | + +`handles` is `by_transport` for `datagram@1` to `@4` and `jsonrpc@1`, and +`none` for `comline.packet@1`. `counted` and `indexed` are for platform +framings such as D-Bus and Binder. + +Narrowing `calls` changes only its description. No value and no profile +changes. + +**What a protocol's address is unique in** follows from `protocols`, so it +isn't a fact of its own: + +| `protocols` | The address is unique in | So a connection can carry | +| --- | --- | --- | +| `unnamed` | nothing: there's one protocol | one protocol | +| `in_call` | its function ids, per phase and direction, across the connection | one package's protocols, with ids unique across all of them | +| `by_index`, `by_id` | its package | one package's protocols | +| `by_name` | its package (namespaces are unique in a package, not across packages) | one package's protocols | +| `by_binding` | the connection | protocols from several packages, if the handshake lists them | + +Caps on how much a framing can address are numbers, not facts: see +[Limits](#limits). + +### Handshake + +| Fact | Description | Values | +| --- | --- | --- | +| `packages` | How many packages one connection's handshake covers. | `one` · `several` | +| `binds` | What the handshake assigns for the connection's frames to use. | `nothing` · `protocols`: a connection number per protocol | +| `compares` | What decides whether two ends may talk. | `identity`: both built from the same schemas · `facts`: each version's lines, shapes and numbers, so compatible versions connect | +| `refuses` | The least a refusal takes down. Set by a `scope` parameter where the handshake has one. | `connection` · `package` · `protocol` | + +## The official encodings, stated in full + +### Framings + +| framing | `calls` | `protocols` | `directions` | `errors` | `payload` | +| --- | --- | --- | --- | --- | --- | +| `datagram@1` | `by_index` | `unnamed` | `one` | `by_ordinal` | `opaque` | +| `datagram@2` | `by_index` | `by_index` | `both` | `by_ordinal` | `opaque` | +| `datagram@3` | `by_id` | `by_id` | `both` | `by_ordinal` | `opaque` | +| [`datagram@4`](datagram@4.md) | `by_id` | `by_binding` | `both` | `by_ordinal` | `opaque` | +| `jsonrpc@1`, `method_names = bare` | `by_name` | `unnamed` | `both` | `by_ordinal` | `json` | +| `jsonrpc@1`, `method_names = qualified` | `by_name` | `by_name` | `both` | `by_ordinal` | `json` | +| `comline.packet@1` | `by_id` | `by_id`, or `in_call` with `family = "none"` | `both` | `by_ordinal` | `opaque` | + +`comline.packet@1`'s parameters set its facts and limits: see +[What each parameter sets](../packet-dispatch/index.md#what-each-parameter-sets). +Under `correlation = none` every frame is a request, so `directions` is +`both` with no kind marker. + +`jsonrpc@1`'s `method_names` parameter is the first framing parameter that +sets a fact. Profiles already allow that. + +`one_way` is `by_schema` for `datagram@1` to `@3`, and `in_frame` for +`datagram@4` (a request kind of its own), `jsonrpc@1` (a notification) and +`comline.packet@1` with correlation (kind `02`). Without correlation, every +`comline.packet@1` packet is one-way. Only under `in_frame` can a receiver stay silent for a one-way +call it can't take, since under `by_schema` it can't tell the call was +one-way. [Return forms](../return-forms.md) builds its quiet and acknowledged +forms on this fact. + +### Handshakes + +| handshake | `packages` | `binds` | `compares` | `refuses` | agreements | +| --- | --- | --- | --- | --- | --- | +| none | (none checked) | `nothing` | (nothing) | (never) | neither | +| `comline@1` | `one` | `nothing` | `identity` | `connection` | schema, encoding | +| `comline@2` | `one` | `nothing` | `facts` | `protocol` or `connection` (`scope`) | schema, encoding | +| [`comline@3`](multi-package-connection.md#the-handshake-comline3) | `several` | `protocols` | `facts` | `protocol`, `package` or `connection` (`scope`) | schema, encoding | + +## Capabilities: what a pairing can do + +A **capability** is derived from a framing's and a handshake's facts. It's +never stated on its own, so it can't disagree with them. `comline inspect` +would show the capabilities per protocol, beside the guarantees it shows today. + +| Capability | Holds when | +| --- | --- | +| several protocols on one connection | `protocols` isn't `unnamed` | +| several packages on one connection | `protocols = by_binding`, and the handshake has `packages = several` and `binds = protocols`; or every id is fixed in the schema (`comline.packet@1`) and the build proves them disjoint across the packages | +| calls in both directions | `directions = both` | +| inserting or reordering functions keeps their address | `calls` is `by_name` or `by_id` | +| inserting or reordering protocols keeps their address | `protocols` is `by_name`, `by_id`, `by_binding` or `in_call` | +| a compatible older or newer peer connects | `compares = facts` | +| a refused protocol leaves the rest working | `refuses` is `protocol` or `package` | +| needs the package's build history to generate | `calls = by_id`, `protocols` is `by_id` or `by_binding`, or `compares = facts` | +| needs a package `id` | `compares = facts` | +| numbers are known before connecting | `protocols` isn't `by_binding` | + +For the official pairings: + +| | several protocols | several packages | both directions | reorder functions | reorder protocols | compatible peers | partial refusal | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `datagram@1` + `comline@1` | no | no | no | no | (one) | no | no | +| `datagram@2` + `comline@1` | yes | no | yes | no | **no** | no | no | +| `datagram@3` + `comline@2` | yes | no | yes | yes | yes | yes | yes | +| `datagram@4` + `comline@3` | yes | **yes** | yes | yes | yes | yes | yes | +| `jsonrpc@1` qualified + `comline@1` | yes | no | yes | yes | yes | no | no | + +## Rules the facts make checkable + +**Build errors**, each naming the facts that decide it: + +- `protocols = by_binding` without a handshake that has `binds = protocols`: + nothing would hand out the numbers. +- Protocols of several packages on one connection, unless `protocols = by_binding`, + `packages = several` and `binds = protocols` all hold, or every id is fixed + in the schema and proven disjoint across the packages, per phase and + direction ([packet dispatch](../packet-dispatch/index.md#packages-on-a-packet-connection)). +- Two protocols on a connection whose framing has `protocols = unnamed` + (`jsonrpc@1`'s ambiguous bare names already fail this way). +- `compares = facts` in a package without an `id`. It fails at generation + today, and would fail at build instead. +- The existing rules, restated against facts: a format's `output` must fit the + framing's `payload`, and `directions = one` allows no calls from the + provider where calls have replies. Under `correlation = none` there are + none, so packets travel both ways without a kind marker. + +**Version comparison**, with the new rows it gains: + +| Change | Bump | +| --- | --- | +| insert, remove or reorder protocols, so that one's position in the package moves | major where `protocols = by_index` (`datagram@2`); none elsewhere | +| a protocol's qualified name changes (namespace or protocol renamed) | major where `protocols = by_name`; none where it's `by_id` or `by_binding` | +| the set of a dependency's protocols on the package's connection | minor to add one, major to remove one, where `packages = several` | + +## What changing an encoding implies + +Changing a protocol's framing or handshake is already major, and moves its +line. With the facts above, the comparison and `comline inspect` can say +what the change does, beyond "the bytes differ": + +| Change | Gains | Loses | Newly needs | +| --- | --- | --- | --- | +| `datagram@2` → `datagram@3` | inserting or reordering functions and protocols keeps addresses | numbers readable from the schema alone | build history (wire numbers) | +| `datagram@3` → `datagram@4` | several packages on a connection | protocol numbers known before the handshake | a handshake with `binds = protocols` (`comline@3`) | +| `datagram@*` → `jsonrpc@1` | readable frames, and addresses that survive reordering | binary formats (`payload = json`), and several protocols unless `qualified` | a format whose `output` is `json` | +| `comline@1` → `comline@2` | compatible peers connect, and per-protocol refusal | nothing | a package `id`, build history, releases | +| `comline@2` → `comline@3` | several packages on a connection | nothing | a framing with `protocols = by_binding` for several packages | +| any → none | no handshake round trip | schema and encoding agreement (needs `@unverified`), several packages, any `by_binding` framing | nothing | + +The reported reason names what's lost, such as "major: `datagram@4` numbers +protocols per connection; peers on `datagram@3` are refused". It also names +what the package must now have, such as "needs `comline@3`". + +## Limits + +**The question.** A framing's fields have widths, and widths cap how much the +framing can address. **Should a profile state those caps?** Today none does: +no profile says "this framing addresses at most N functions". + +### Why it hasn't mattered, and why it will + +So far, every official framing's caps happen to equal core's: wire numbers are +`u16`, and so are the datagram framings' protocol and function fields. The +coincidence hides the gap. + +[Packet dispatch](../packet-dispatch/index.md) breaks it. `comline.packet@1` takes +an `id` width parameter, `id = "u8"` among them. Under it, a protocol's +257th function, or an `@id = 300`, has a number the frame can't hold. Nothing +in the build knows the framing's width, so nothing refuses it. A third-party +framing with narrow fields has the same problem. + +### Proposed: `limits` beside `facts` + +A profile gains a `limits` map. Each limit is a number, with `value` and +`proven_by` like a fact, proven by vectors at the limit. Like facts, a limit +is fixed for the encoding's version, and **a parameter can set it** +(`comline.packet@1`'s `id = "u8"` sets `functions_per_protocol` to 256). + +Limits aren't facts, because they're used differently. A fact is one of a +few values that rules compare between two versions. A limit is a number that +the build checks the package's counts against. + +| Limit | Applies to | What it caps | +| --- | --- | --- | +| `protocols` | framing | protocols one address space holds. The space is the one the `protocols` fact names: a package, or a connection under `by_binding` | +| `functions_per_protocol` | framing | the largest function number, plus one | +| `errors_per_schema` | framing | the largest error ordinal, plus one | +| `pending_requests` | framing | requests awaiting a reply at once, per direction | +| `frame_bytes` | framing | the largest frame, when the framing writes the frame's length itself (`comline.packet@1`'s `length`). Otherwise the transport delimits frames, and the cap is the transport's | +| `handles_per_frame` | framing | the most handles one frame carries, for a framing that counts or lists them (`handles = counted` or `indexed`). Under `by_transport` the cap is the transport's per message | +| `packages_per_connection` | handshake | packages one handshake lists | +| `protocols_per_connection` | handshake | protocols one handshake lists | + +Format limits (items in a collection, bytes in a string, enum variants, union +members) would take the same shape, as a follow-up. + +### The official framings' limits + +| framing | `protocols` | `functions_per_protocol` | `errors_per_schema` | `pending_requests` | +| --- | --- | --- | --- | --- | +| `datagram@1` | 1 | 65536 | 65536 | 2⁶⁴ | +| `datagram@2` | 65536, per package | 65536 | 65536 | 2⁶⁴ | +| `datagram@3` | 65536, per package | 65536 | 65536 | 2⁶⁴ | +| [`datagram@4`](datagram@4.md#limits) | 65536, per connection | 65536 | 65536 | 2³² | +| `jsonrpc@1` | (none) | (none: by name) | (none) | (none) | +| `comline.packet@1`, `id = "u8"`, `family = "u8"`, `correlation = { header_id = "u16" }` | 256, per package | 256 | per its error status (undecided) | 2¹⁶ | + +"(none)" means the encoding's bytes impose no cap. Core's own still applies: +wire numbers are `u16`, so a package has at most 65536 protocols, and a +protocol at most 65536 functions. + +### What reads them + +- **The build.** It checks every count against the smallest cap in play: + core's, the framing's and the handshake's. The error names the limit, the + encoding, and the parameter that set it: ``protocol `game::Login` has 300 + functions, and `comline.packet@1` with `id = "u8"` addresses at most 256``. + The same check applies to an `@id` above the cap. +- **The comparison.** A limit doesn't change bytes, so it takes no bump of + its own. Changing to an encoding or parameter with a lower cap is already + major. A change that would exceed a cap is a build error, never a bump. +- **`comline inspect`.** It shows headroom: "12 of 256 functions". +- **Runtimes.** `pending_requests` bounds a caller's table of requests in + flight. A caller at the cap waits, or refuses the call, instead of reusing + an id that's still pending. +- **Generators.** They can size what they emit, such as a `u8` dispatch index + under a `u8` framing. + +### What stays out + +A profile states only the caps its **encoding's bytes** impose. Others belong +elsewhere: + +| Limit | Where it belongs | +| --- | --- | +| a runtime's language, such as JSON numbers above 2⁵³ in TypeScript | the language's support tables, and the generator's feature claims | +| the largest frame, where the transport delimits frames | the transport's profile ([Transports](../transports.md)). A framing that writes the length itself states `frame_bytes` | +| core's `u16` wire numbers | core: [Number](../../reference/versioning.md#number) | +| a type's bound, such as `char[max 64]` | the schema | + +### Is it worth it? + +- **For:** + - It's the only thing that catches a narrow encoding's overflow at build + time, rather than as truncation or a refusal at run time. + - It documents capacity where people look for it. + - Third-party encodings state theirs the same way. + - It gives `comline inspect` headroom to show, and runtimes a bound for + requests in flight. +- **Against:** + - Every limit needs vectors. + - For the official datagram framings, the caps equal core's, so stating them + adds rows without adding checks until a narrower encoding exists. + - The build has to take the smallest of several caps. + +Recommended, on the rule in [What stays out](#what-stays-out): **only the +caps the encoding's bytes impose**. Without that rule, limits become a place +where runtime, transport and schema limits blur together. + +## Where they live + +As for the existing facts: + +- **Profiles:** each profile in the runtime's corpus gains the new facts, each + value proven by vectors. Examples: `protocols = by_index` by a request + addressing a protocol by its position, and `packages = several` by a + `comline@3` hello listing two packages whose protocols are both #0. +- **Core** keeps its pinned copy, drift-checked as today. +- **The catalog** in the [shared vocabulary](../shared-vocabulary/catalog.md) + gets an entry per fact. +- **[Official encodings](../../reference/wire/official-encodings.md)** + renders every fact, and the capabilities derived from them, from the + profiles. + +## Open questions + +- **What IPC adds.** The `handles` fact and the `handles_per_frame` limit + are adopted (2026-11-14, [above](#framing)). Platform framings need `calls = by_hash` (FIDL), `errors = in_result` + (Binder, Mojo, FIDL) and `correlation = synchronous` (Binder). D-Bus, + the first platform framing, addresses *instances*, which no fact covers yet. + All of this is laid out in [IPC, Framings](../ipc.md#framings). +- ~~**Does `one_way` belong here?**~~ Decided: yes, as the `one_way` fact + ([above](#framing)). +- **Error scope.** `errors = by_ordinal` counts within the error space of the + schema that declares the protocol. Every official framing names the + protocol in a reply or matches it by request id, so the space is always + known. A framing that does neither could only carry one schema's errors, + which may warrant a fact later. diff --git a/docs/docs/proposals/dependency-protocols/connection-per-package.md b/docs/docs/proposals/dependency-protocols/connection-per-package.md new file mode 100644 index 0000000..e8c0cef --- /dev/null +++ b/docs/docs/proposals/dependency-protocols/connection-per-package.md @@ -0,0 +1,81 @@ +# Proposal C — a connection per package + +Status: **accepted** (2026-11-14), complementary to [A](multi-package-connection.md) · +See the [section overview](index.md) for the question all three answer + +**The idea:** each package's protocols run on a connection of their own, with +that package's facts and `comline@2` exactly as written. A dependency's +protocol is served on the dependency's connection, never on the consumer's. + +## Shape + +```text +connection 1 comline@2, package localdeps 0.2.0 greeting::Foo #0 +connection 2 comline@2, package test 0.4.0 none::Thing #0 +``` + +Generators already emit a connection per package. A dependency's generated +module gets its own `accept`, `serve` and `connect`, which run `comline@2` with +that package's facts. So C needs no change to the wire, the runtime or the +handshake, only step 0 and the dependency's facts in the request (as +[A](multi-package-connection.md#what-it-touches) needs too). + +## What it gives + +- **One identity, checked against its own version.** Foo is localdeps' #0 + everywhere, and the handshake compares localdeps' versions. +- **Each package keeps its own encodings.** A dependency on `jsonrpc` and an + application on `datagram@3` both work, because neither has to fit the other's + connection. A can't do this. +- **Isolation.** A dependency's handlers can't see the application's + connection state. A refusal under `scope = package` ends only that + package's connection. Different trust can sit on different connections: a + public health check beside an authenticated API. +- **The consumer's version is untouched.** None of the dependency's protocols + are part of the consumer's connection. +- **Serving in parts across processes.** A dependency's protocols can be served + by another process or host, since its connection doesn't have to end where + the application's does. + +## What it costs + +- **More connections, and finding them.** N packages need N connections: + N transport setups, TLS sessions and handshakes. A peer must also know where + each package lives, through separate ports or addresses, or a transport that + multiplexes streams (QUIC, HTTP/2, WebSocket subprotocols). Multiplexing in + the transport is A one layer down, without a handshake that sees across + packages. +- **No shared connection state.** Extensions are per connection. A peer + authenticates on each one, and Foo's handler can't see the session that + test's connection established. +- **No order across packages.** A connection runs one call at a time, in + order. Calls on two connections race. [Phases](../packet-dispatch/index.md) are + per connection, so a phase can't span packages. +- **No calls back across packages.** With calls in both directions, Foo's + handler can call only the protocols on its own connection. It can't call + the application's `Client` protocol on the same peer. +- **Half-connected peers.** One connection can drop while the other stays up, + and the application has to handle a peer it can reach for one package but + not the other. + +## Partial serving + +Serving part of what the packages declare doesn't need C by itself. Within one +connection, both `comline@2` and [A](multi-package-connection.md) handle a +missing protocol as `unknown_protocol`, so a server can provide only some of +them. A can also make that choice per dependency protocol. + +C is the answer when the parts are **served in different places**: a +dependency's protocols in another process, or another host, or behind +different trust. Then they need their own connection anyway. + +## When it's the choice + +- A dependency's protocol whose framing, handshake or format differs from the + connection's. A refuses it, and C is where it goes. +- A deployment that wants isolation, separate trust, or the dependency served + elsewhere. + +Otherwise A is cheaper for the peer: one transport, one handshake, shared +state and order. Both should be supported, and generators already emit C's +per-package connections. A adds the shared one. diff --git a/docs/docs/proposals/dependency-protocols/consumer-numbers.md b/docs/docs/proposals/dependency-protocols/consumer-numbers.md new file mode 100644 index 0000000..9be53b4 --- /dev/null +++ b/docs/docs/proposals/dependency-protocols/consumer-numbers.md @@ -0,0 +1,80 @@ +# Proposal B — the consumer numbers its dependencies' protocols + +Status: **rejected** (2026-11-14) as a default, see [verdict](#verdict); an +explicit opt-in re-export may come back · +See the [section overview](index.md) for the question all three answer + +**The idea:** at `comline build` and `comline release`, the consumer's history +numbers each dependency protocol on its connection in the consumer's own +space. It uses the same rules as the consumer's own protocols: the next free +number when the protocol is first added, bound at release, and retired if +removed. The Hello stays one package's, so nothing on the wire changes. + +## Shape + +```text +test 0.1.0 none::Thing #0 +test 0.2.0 adds localdeps, reaches Foo → localdeps::greeting::Foo #1 in test's facts +``` + +- Foo's functions keep the dependency's function numbers. They're unique in + their protocol already. +- The entry is keyed by **(dependency id, the dependency's wire number)**, not + by path. A protocol that the dependency renames keeps its number there, so + it keeps its number here too. +- test's facts copy Foo's shape hash, and record its line as a **test** + release: the release at which test picked up a breaking change to Foo. + +## Where it holds + +- **Every peer built from the consumer agrees.** A client and a server both + generated from `test` read the same facts and get the same numbers. That's + the common case: an application whose API includes a library's protocol. +- **It's as stable as the consumer's own numbers.** Numbers are assigned, + bound at release and never reused, by the + [same rules](../../reference/versioning.md#number). The + [function-ids decision](../function-ids/annotation.md#as-accepted) rejected a + hidden ledger of ids because a fresh checkout could assign them differently. + Wire numbers have lived in history since then, so this adds no new kind of + risk. +- **No collisions.** Every dependency's protocols share the consumer's one + space. +- **It isn't unsafe on the wire.** A peer built from the dependency alone + sends the dependency's package id, and the handshake refuses it + (`package_mismatch`) before any call. So no frame can reach the wrong + function. What's lost is reach, not correctness. + +## Where it doesn't + +1. **Two identities.** Foo is protocol #0 of `localdeps` and #1 of `test`. A peer + built from `localdeps` alone, such as a generic health checker, can't call + Foo on test's connection. Two applications that both depend on `localdeps` + each give Foo their own number. The protocol stops being a contract + that other packages can reuse: every consumer re-publishes it. +2. **Checked against the wrong version.** The handshake compares test's + versions, so a change in `localdeps` reaches peers as a change in `test`. + test must release to move Foo's line. Two test releases built against the + same `localdeps` release are still judged by test's line for Foo. +3. **Numbers depend on history order.** Two branches that each add a + dependency both assign their protocols the next number. Reconciling them + renumbers whatever neither released. That's already true for the + consumer's own protocols, but adding a dependency now triggers it as well. +4. **Diamonds.** A protocol reached through two paths (`test → a → localdeps` + and `test → localdeps`) needs one entry. If the paths pin different + versions of `localdeps`, a rule has to pick whose shape and line the facts + copy. +5. **Dropping and re-adding a dependency** retires its protocols' numbers, and + a peer built in between sees `unknown_protocol`. That's consistent with the + consumer's own protocols, but surprising for a change to `config.idp`. + +## Verdict + +It works when the consumer means to **re-export** the protocol as part of its +own API, and every peer is built from the consumer. As a default it gives +every dependency protocol a second identity without anyone asking for one, and +it ties the protocol's compatibility to the consumer's releases. + +Its one advantage over [A](multi-package-connection.md) is no wire change. +That lasts only until `comline@3` ships. So it isn't recommended as the +mechanism. An explicit "serve this as ours" re-export, one entry at a time, +could be proposed later for packages that want exactly that. diff --git a/docs/docs/proposals/dependency-protocols/datagram@4.md b/docs/docs/proposals/dependency-protocols/datagram@4.md new file mode 100644 index 0000000..383cf2d --- /dev/null +++ b/docs/docs/proposals/dependency-protocols/datagram@4.md @@ -0,0 +1,311 @@ +# `datagram@4` + +Status: **accepted** (2026-11-14) with [Proposal A](multi-package-connection.md), +the fixed-width layout to be confirmed by benchmark · See +the [section overview](index.md) + +`datagram@4` is a binary framing for a connection that carries protocols from +**one or more packages**, with calls in both directions. A request names its +protocol by a **connection number** that the handshake binds, and its +function by the function's **wire number**. Both numbers are stable, so +inserting or reordering declarations never changes what a call addresses, and +two packages' protocols never collide. Every header field is **fixed-width** +and at a fixed offset. + +## Requirements + +- **A handshake that binds connection numbers** (`binds = protocols`): + [`comline@3`](multi-package-connection.md#the-handshake-comline3). With no + handshake, or one that binds nothing, there are no connection numbers, and + the build refuses the pairing. +- **Wire numbers** for every function: core's numbering, from the package's + build history ([Number](../../reference/versioning.md#number)). Each + function's number comes from the history of the package that declares its + protocol. +- **A transport that delimits frames.** Each transport message is exactly one + frame. Over TCP, the runtime's stream transport writes a `u32` LE length, + then the frame. A frame is held to the transport's limit, 16 MiB by default + in every runtime. +- **Any format.** The payload is opaque bytes, so any format whose `output` + is `bytes` or `json` fits. + +## Numbers + +Every number is little-endian, at a fixed width: + +| Number | Width | Meaning | +| --- | --- | --- | +| connection number | `u16` | the protocol's number on this connection, as the handshake binds it. Under `comline@3`, it's the protocol's position in the initiator's Hello. Packages are listed by ascending id and protocols by ascending wire number, so the first protocol listed is 0 | +| function number | `u16` | the function's wire number in its protocol, from its package's history: unique in the protocol, and never reused once retired | +| request id | `u32` | chosen by the caller, sequentially (see [Request ids](#request-ids)) | +| error ordinal | `u16` | the error's position in the error space of the schema that declares the protocol | + +## Frames + +Every frame starts with a **kind** byte, which decides the rest of its +layout. Any other value is malformed. + +| Kind | Frame | Header | +| --- | --- | --- | +| `00` | request, expecting a reply | 9 bytes | +| `01` | reply | 7 bytes | +| `02` | one-way request | 5 bytes | + +### Request (`00`) + +| Offset | Field | Bytes | Meaning | +| --- | --- | --- | --- | +| 0 | kind | `00` | a request that expects a reply | +| 1 | protocol | `u16` LE | the connection number of the protocol called into | +| 3 | function | `u16` LE | the function's wire number in that protocol | +| 5 | request id | `u32` LE | the id its reply will carry | +| 9 | arguments | the rest of the frame | the function's arguments, in declaration order, encoded by the connection's format as one value | + +### One-way request (`02`) + +| Offset | Field | Bytes | Meaning | +| --- | --- | --- | --- | +| 0 | kind | `02` | a request that gets no reply | +| 1 | protocol | `u16` LE | as above | +| 3 | function | `u16` LE | as above | +| 5 | arguments | the rest of the frame | as above | + +A function declared without `->` is one-way, and is always called with kind +`02`. Since nothing will answer, it carries no request id. A function declared +with `->` is always called with kind `00`. A request whose kind disagrees with +its function's declaration is answered as a call the receiver doesn't have +(see [Receiving](#receiving)). + +### Reply (`01`) + +| Offset | Field | Bytes | Meaning | +| --- | --- | --- | --- | +| 0 | kind | `01` | a reply | +| 1 | protocol | `u16` LE | the connection number of the request it answers | +| 3 | request id | `u32` LE | the id of the request it answers | +| 7 | outcome | the rest of the frame | one of the three below | + +The **outcome:** + +| Outcome | Bytes | Meaning | +| --- | --- | --- | +| ok | `00`, then the result | the function's return value, encoded by the connection's format. A function declared `-> ()` returns the format's empty value | +| error | `01`, the error ordinal (`u16` LE), then its body | an error from the function's `!` list. The body is the error's fields, encoded like a struct | +| fault | `02`, nothing after | the provider failed in a way the schema doesn't describe, or couldn't take the call | + +The reply's protocol isn't needed to match it, since the request id does that. +It's there so a reply can be attributed to its protocol from the frame alone, +in a capture or a log, and so a reply that names the wrong protocol shows up +as a bug. + +## Request ids + +- **Per direction.** Each end numbers the requests it sends. Both ends can use + the same id at once without confusion, because a reply always answers a + request of the end that receives it. +- **Sequential.** Each end starts at 0 and adds 1 per request expecting a + reply, wrapping from 4294967295 to 0. A caller that gives up on a request, + for example after a timeout, doesn't reuse its id until the counter wraps. + So a late reply can't be mistaken for a newer request's. +- **Matched by id.** A reply is matched to its request by request id. Its + protocol must be the request's. + +## Receiving + +| Received | What the receiver does | +| --- | --- | +| a frame that doesn't parse: unknown kind, or shorter than its kind's header | ends the connection: it's malformed | +| a request (`00`) for a protocol this end provides, to a request/response function it has | dispatches it, and replies | +| a one-way request (`02`) for a protocol this end provides, to a one-way function it has | dispatches it. It never replies | +| a request (`00`) it can't take: an unbound connection number, a protocol it doesn't provide or the handshake refused, a function the protocol doesn't have, or a one-way function | replies with a **fault** | +| a one-way request (`02`) it can't take, for the same reasons, or to a request/response function | drops it, and never replies | +| a reply whose request id awaits no reply, such as a late reply to a request its caller gave up on | drops it | +| a reply whose protocol isn't its request's, or whose outcome tag isn't `00`, `01` or `02` | ends the connection: it's malformed | +| an error ordinal the caller's schema doesn't have | gives the caller a remote error with that ordinal: the provider is newer | + +A well-behaved caller never sends the fourth and fifth rows. After the +handshake, a call to a function or protocol the peer lacks fails locally, +without reaching the wire. + +## Limits + +What the frame's widths allow. Stated in the [profile](#profile) as +[limits](capabilities.md#limits), and checked at build: + +| Limit | Value | From | +| --- | --- | --- | +| protocols per connection | 65536 | the `u16` connection number | +| functions per protocol | 65536 | the `u16` function number, as core's wire numbers | +| errors per schema | 65536 | the `u16` error ordinal | +| requests awaiting a reply, per direction | 4294967296 | the `u32` request id | + +The largest frame isn't the framing's limit. It's the transport's. + +## Examples + +A connection carrying `localdeps` (whose `greeting::Foo` is its protocol #0) +and `test` (whose `none::Thing` is its protocol #0). If `localdeps`'s id +sorts first, `Foo` is bound to 0 and `Thing` to 1. Arguments are in `json@1`. + +A call to `Thing`'s function 3, request id 42, arguments `[7,"x"]`: + +```text +00 0100 0300 2a000000 5b372c2278225d +``` + +A call to `Foo`'s function 0, request id 43, the same arguments: + +```text +00 0000 0000 2b000000 5b372c2278225d +``` + +A one-way call to `Thing`'s function 5, with no arguments (`[]`): + +```text +02 0100 0500 5b5d +``` + +Replies to the first call: ok with `7`, error 2 with an empty body, and a +fault: + +```text +01 0100 2a000000 00 37 +01 0100 2a000000 01 0200 7b7d +01 0100 2a000000 02 +``` + +Function 65535 of connection number 300 is `00 2c01 ffff …`. + +## Profile + +```json +{ + "name": "datagram@4", + "kind": "framing", + "facts": { + "calls": "by_id", + "protocols": "by_binding", + "directions": "both", + "errors": "by_ordinal", + "payload": "opaque" + }, + "limits": { + "protocols_per_connection": 65536, + "functions_per_protocol": 65536, + "errors_per_schema": 65536, + "pending_requests": 4294967296 + } +} +``` + +(The profile in the corpus gives each fact and limit with `value` and +`proven_by`.) The facts and limits are [described here](capabilities.md#proposed-facts). +The profile is fixed: a change to any of them would be `datagram@5`. + +## Pros and cons + +### Pros + +- **Several packages on one connection,** with no registry of ids across + packages. Two packages' protocol #0 never collide. +- **Stable addresses.** Inserting, reordering or renaming functions and + protocols never changes what a call addresses. +- **Fixed headers.** Every field is at a fixed offset. A receiver checks a + frame's length against its kind's header once, then reads each field with a + plain load, with no per-byte branches and no shortest-form or range checks. + Packet filters, eBPF and XDP programs, load balancers and dissectors can + match a protocol or function at a fixed offset. +- **Dense protocol numbers.** Connection numbers run 0, 1, 2…, so a receiver + dispatches with an array index. +- **Small headers.** A request is 9 bytes, a one-way request 5, and a reply 7. +- **One-way calls are on the wire.** A receiver stays silent for a one-way + call it can't take, so a misbehaving peer can't trigger a reply nobody + awaits. +- **Late replies are harmless.** Sequential request ids and dropping + unmatched replies let a caller time out without ending the connection. + +### Cons + +- **Frames aren't self-describing.** A capture or log of frames can't be read + without the connection's Hello, because the same connection number means + different protocols on different connections. Debugging and tracing tools + need the handshake, not just the packages' facts. +- **Frames can't be forwarded as they are.** A proxy relaying calls between + two connections that carry different package sets must rewrite the + protocol number in every request and reply. +- **No numbers at build time.** Generated code resolves a protocol after the + handshake. A sender caches it once per connection, so the per-call cost is + small, but it's no longer a constant the compiler folds. +- **Larger than varints for small numbers.** A request below 128 protocols + and functions would be 2 bytes shorter with varints. +- **Fields are unaligned.** The kind byte puts every field at an odd offset. + Every common target reads unaligned integers, but a strict-alignment target + has to copy them out first. +- **`u16` ceilings.** 65536 protocols per connection and functions per + protocol. Core's wire numbers have the same ceiling, so the framing adds no + new one. +- **The reply's protocol is redundant** for matching. It costs 2 bytes per + reply, kept for attribution and checking. +- **A request's kind must match its function's declaration.** A mismatch is + a bug that's caught, not one that's tolerated. + +### Why fixed widths + +Varints were considered for the connection and function numbers, and +rejected: + +| Layout | Request | One-way | Reply | Fields at fixed offsets | +| --- | --- | --- | --- | --- | +| Varint numbers, `u64` id | 11 to 15 bytes | (not marked) | 10 to 12 bytes | the kind only | +| Fixed `u16`s, `u64` id | 13 bytes | (not marked) | 11 bytes | all | +| **Fixed `u16`s, `u32` id, one-way kind (chosen)** | **9 bytes** | **5 bytes** | **7 bytes** | **all** | +| Same, with no protocol in replies | 9 bytes | 5 bytes | 5 bytes | all | + +- **Varints** put every later field at a variable offset. They need + shortest-form and range checks on every frame, and their decoder's + branches predict poorly once numbers pass 127. In return they save 2 bytes + per request, and nothing at all once numbers pass 127. +- **A `u32` request id** saves 4 bytes on every request and reply, more than + varints would. 2³² requests in flight per direction is more than any + connection will have, and sequential ids make the wrap safe. +- **The one-way kind** saves the request id on one-way calls, and closes a + hazard: without it, a receiver can't tell a call is one-way, answers an + unknown one with a fault, and the caller receives a reply to nothing. +- **The reply's protocol** stays, for the reasons [above](#reply-01). + +The choice should still be confirmed by measurement. The runtime's +[benchmark gates](../shared-runtime-core/verification.md) can time encoding, +decoding and dispatch for this layout beside `datagram@3`, at small and large +numbers. + +## What it can't do + +- **Address a protocol before the handshake.** Connection numbers exist only + once the handshake has bound them. Generated code names a protocol by its + package id and wire number, and looks up the connection number after + connecting. +- **Run without a binding handshake.** It requires `comline@3`, so it can't + be paired with `none`. +- **Report which request it couldn't read.** A malformed frame ends the + connection. +- **Exceed its [limits](#limits).** +- **Stream, cancel or fragment.** One request has one reply at most, and one + frame has one message. Each must fit the transport's frame limit. +- **Order calls.** The framing says nothing about order. The connection + decides it (by default, one call at a time per connection). + +## Proven by + +`framing-datagram4` vectors in the runtime's corpus, byte-exact: + +- a request, a one-way request and each reply outcome; +- numbers at 0, 255, 256 and 65535, and request ids at 0 and 4294967295; +- two packages whose protocols are both #0, called on one connection under + their connection numbers (`protocols = by_binding`, with + `handshake-comline3`); +- rejections: unknown kind, a frame shorter than its header for each kind, a + reply with an unknown outcome tag, and a reply naming the wrong protocol; +- a request to an unbound connection number, answered with a fault; a one-way + request to one, answered with nothing; a reply to no pending request, + dropped; +- the request id wrapping from 4294967295 to 0. diff --git a/docs/docs/proposals/dependency-protocols/index.md b/docs/docs/proposals/dependency-protocols/index.md new file mode 100644 index 0000000..fcf4f88 --- /dev/null +++ b/docs/docs/proposals/dependency-protocols/index.md @@ -0,0 +1,117 @@ +# A dependency's protocols on a connection + +Status: **decided** (2026-11-14): A and C, with Part 1 · tracked in +[core#277](https://github.com/ComlineProject/core/issues/277), step 0 in +[generation#143](https://github.com/ComlineProject/generation/issues/143) · +carries out the first decision of +[Connections](../connections.md#several-protocols-on-one-connection), which +`comline@2` was written without + +**The question.** A package's dependencies can declare protocols, and the +package can serve or call them. When a dependency's protocol runs on a +connection, **which numbers address it, and which version does the handshake +check it against?** + +## Where things stand + +- **Wire numbers are per package.** A protocol's + [number](../../reference/versioning.md#number) is unique in its own package, + and every package numbers its protocols from 0. So `test::none::Thing` and + `localdeps::greeting::Foo` can both be protocol 0. +- **A frame names a protocol by that number alone.** A + [`datagram@3`](../../reference/wire/framings/index.md#datagram3) request carries a + protocol number and a function number, but no package. +- **A handshake covers one package.** The + [`comline@2`](../../reference/wire/handshakes/comline@2.md) Hello carries + one package id and one version, then that package's protocols. Two + different ids are refused (`package_mismatch`). +- **Profiles don't say any of this.** No fact states how a frame names its + protocol, what that address is unique in, or how many packages a handshake + covers. [Part 1](capabilities.md) lists what's missing, and what each gap + causes today. +- **Connections decided otherwise.** On 2026-10-20 the Connections proposal + decided that an offer lists protocols by package id plus protocol, and that + the connection hands out its own numbers, so that "protocols from different + packages, a dependency's health check say, then share a connection with no + registry of ids across packages". `comline@2` was written with fixed + per-package numbers instead, which that page's + [What's built](../connections.md#whats-built) records as differing from the + decision. +- **Generation refuses every dependency protocol.** `facts::check` in + `generation/codegen` addresses every protocol in the request, the + dependency's included, and looks them up in the package's own facts. A + dependency that declares a protocol under the default framing and handshake + then fails `comline generate`, even when the package never uses that + protocol: + + ```text + protocol `localdeps/greeting::Foo` uses `datagram@3` with `comline@2`, which + address it by the wire numbers its package's history assigns, and the + package's facts have none for it + ``` + + `facts::hello` already skips a dependency's protocols ("not this package's + to declare"), so the two disagree. + +### Step 0, whichever is chosen + +A dependency's protocol that the package doesn't reach isn't on its connection. +Generation shouldn't address it, check it, or generate connection code for it. +This fixes the error above for every package that only uses a dependency's +types, and doesn't depend on anything below. + +## The pages + +| | In one line | +| --- | --- | +| [Part 1 — what framings and handshakes state](capabilities.md) | new profile facts (`protocols`; `packages`, `binds`, `compares`, `refuses`), profile **limits** (how many protocols, functions, errors and pending requests an encoding can address), the capabilities they derive, the rules they make checkable, and what changing an encoding implies | +| [Proposal A](multi-package-connection.md) | several packages on one connection: `comline@3` lists each package with its own id, version and numbers, and binds a connection number per protocol | +| [`datagram@4`](datagram@4.md) | the framing A uses: a protocol by its connection number, a function by its wire number, in a fixed-width header (9-byte request, 5-byte one-way request, 7-byte reply) | +| [Proposal B](consumer-numbers.md) | the consumer's history numbers each dependency protocol it carries in its own space, from when it was added | +| [Proposal C](connection-per-package.md) | each package's protocols run on a connection of their own | + +Part 1 stands on its own: its facts are needed whichever proposal is chosen. +The proposals don't all compete. **A** is the general answer, and the +Connections decision already commits to it. **C** is what a protocol gets +when it can't share a connection, and is also a deployment choice. **B** is +the real alternative to A: it avoids changing the wire, at the cost of giving +the protocol a second identity. + +## How they compare + +| | A: several packages | B: consumer's numbers | C: per package | +| --- | --- | --- | --- | +| Wire change | yes: `comline@3` + `datagram@4` | none | none | +| Identities per protocol | one, its package's | two: the dependency's and the consumer's | one | +| Checked against | the dependency's own version | the consumer's version | the dependency's own version | +| A peer built from the dependency alone can call it | yes | no: different package id | yes | +| State, order and calls both ways shared across packages | yes | yes | no | +| A dependency's handlers see the application's connection state | yes | yes | no | +| A slow dependency handler holds up the application | yes, by default | yes, by default | no | +| Each package keeps its own framing and format | no: one set per connection | no | yes | +| Protocol numbers known before connecting | no | yes | yes | +| Serving part of what the packages declare | per protocol, through the offer | per protocol, as today | per package by connection, then per protocol | +| The dependency's changes move the consumer's version | only for which protocols the connection carries | yes: the consumer re-releases each one | no | +| Peers learn dependency versions | yes, from the Hello | no | yes, per connection | + +## Decided + +Decided 2026-11-14: + +- **Step 0 now.** +- **[Part 1](capabilities.md) is accepted:** its facts, the `one_way` fact, + and `limits`, for caps the encoding's bytes impose. These are needed + whichever proposal applies, and close the `datagram@2` protocol-reordering + gap on their own. +- **[A](multi-package-connection.md) is accepted** for protocols that can + share a connection, as `comline@3` with [`datagram@4`](datagram@4.md). They + become the defaults in the next edition, and runtimes may answer a + `comline@2` initiator during the migration. +- **[C](connection-per-package.md) is accepted** for protocols that can't + share one, because their framing, handshake or format differs, and for + deployments that want isolation. +- **[B](consumer-numbers.md) is rejected** as a default. Re-exporting a + dependency's protocol as the consumer's own could come back as an explicit, + opt-in act. + +The points decided within A are listed on [its page](multi-package-connection.md#as-decided). diff --git a/docs/docs/proposals/dependency-protocols/multi-package-connection.md b/docs/docs/proposals/dependency-protocols/multi-package-connection.md new file mode 100644 index 0000000..ec27985 --- /dev/null +++ b/docs/docs/proposals/dependency-protocols/multi-package-connection.md @@ -0,0 +1,310 @@ +# Proposal A — several packages on one connection + +Status: **accepted** (2026-11-14), as [decided below](#as-decided) · builds on +[Part 1](capabilities.md) · See the [section overview](index.md) for the +question all three answer + +**The idea:** a connection carries protocols from several packages. The +handshake, `comline@3`, lists each package with its own id, version and +protocols, and binds each protocol a connection number. The framing, +[`datagram@4`](datagram@4.md), addresses a protocol by that number. Each +package's wire numbers, lines and versions stay its own, so a dependency's +protocol keeps its one identity wherever it's served. This is the first +decision of [Connections](../connections.md#several-protocols-on-one-connection), +written for `comline@2`'s successor. + +## Shape + +`test` serves its own `Thing` and its dependency's `Foo` on one connection: + +```text +packages on the connection wire numbers connection numbers +localdeps (id 0730…, 0.2.0) greeting::Foo #0 → 0 +test (id a5d4…, 0.4.0) none::Thing #0 → 1 +``` + +Both protocols are #0 in their own packages, and the connection tells them +apart. Function numbers stay their package's own: they're unique in their +protocol, so they never collide. + +In Part 1's terms, the pairing states `protocols = by_binding`, +`packages = several` and `binds = protocols`, which is what +[several packages on one connection](capabilities.md#capabilities-what-a-pairing-can-do) +requires. + +## Why a new framing + +`datagram@3`'s protocol field is a package wire number (`protocols = by_id`). +A field that's sometimes a wire number and sometimes a connection number +would be one profile stating two facts. A profile is fixed for its version, +so a different fact means a new version: `datagram@4` (`protocols = +by_binding`). It's specified in full on [its own page](datagram@4.md). + +## The handshake: `comline@3` + +`comline@3` checks compatibility from each version's facts, for every package +on the connection. + +### Messages + +Every message starts with `"CO"`, the version byte `03`, and a type byte. +Numbers and counts are shortest-form varints, and a hash is 16 bytes (the +first 128 bits of core's blake3 hash). A release is three varints (major, +minor, patch), and a package version is a release followed by its pre-release +(length varint and UTF-8, or `00` for none). Bytes after a message's last +field make it malformed. + +**Hello** (type `01`): + +| Field | Bytes | +| --- | --- | +| `exchange` | 1: `00` offer, `01` symmetric | +| encodings | the connection's format, framing and handshake: count, then each kind (1: `00` format, `01` framing, `02` handshake) and parameter hash (16), ascending | +| packages | count, then each: package id (16), package version, protocols; ascending by package id | +| protocols, per package | count, then each: wire number, line (a release), shape hash (16), function numbers (count, then numbers, ascending); ascending by wire number | + +**Answer** (type `02`, from the responder under `offer`): + +- `00` accept: then, for every package in the offer's order, the responder's + version of it (or `00` and no version when it doesn't have that package). + Then, for every offered protocol in connection-number order, a status: + - `00` accepted, or `01` accepted with the shape hash differing, each + followed by the offered functions the responder lacks (count + numbers); + - `02` refused, followed by the reason code. +- `01` refuse, followed by a refusal. + +**Verdict** (type `03`): + +- `00` accept, followed by the protocols this end refuses: count, then each + connection number + reason code, ascending; +- `01` refuse, followed by a refusal. + +**Refusal:** the reason code (1 byte), the refusing side (1 byte: `00` +initiator, `01` responder), then the connection number when the reason is +about one protocol. + +The exchanges are `comline@2`'s: under `offer`, the initiator sends Hello, +the responder sends Answer, and the initiator sends Verdict. Under +`symmetric`, both send Hello, then both send Verdict. + +### Connection numbers + +A protocol's **connection number** is its position in the initiator's Hello, +counting across packages in order. + +- Both ends know the numbers once the initiator's Hello has arrived, under + `offer` and `symmetric` alike, so no message has to carry them. Connections + had the server's answer hand them out. The answer lists protocols in the + offer's order, so the position *is* the number. +- A protocol only the responder has is `unknown_protocol`, and never needs + a number. + +### What decides + +1. Another handshake version (`"CO"` with a version byte other than `03`): + refused, `handshake_version`. +2. Different encodings, by parameter hash: refused, `encoding_mismatch`. +3. Per package both ends list, and per protocol of it both have, comparing + **that package's** versions: + - the lines differ: the protocol is refused, `line_mismatch`; + - same package version but a different shape hash: `same_version_differs` + decides; + - the consumer has functions the provider lacks: `newer_consumer` decides. +4. A protocol only one end has, including every protocol of a package only one + end lists: `unknown_protocol`, refused on its own. + +The reason codes are `comline@2`'s. `package_mismatch` (4) no longer arises, +and stays reserved because codes are never reused. + +### Parameters + +`comline@2`'s four, with one value added to `scope`: + +| Parameter | Values | Meaning | +| --- | --- | --- | +| `scope` | `protocol` \| `each_package` \| `package` | A refusal takes down the protocol alone, every protocol of its package, or the whole connection. `package` keeps `comline@2`'s meaning, the whole connection, so switching a config from `comline@2` to `comline@3` never silently narrows a refusal. `unknown_protocol` never takes down more than its protocol. | +| `newer_consumer` | `degrade` \| `refuse` | as `comline@2` | +| `same_version_differs` | `refuse` \| `warn` | as `comline@2`, per package | +| `exchange` | `offer` \| `symmetric` | as `comline@2`; in the parameter hash | +| `accept_previous` | `false` \| `true` | a policy: also answer a `comline@2` initiator ([migration](#for-the-wire-and-peers)). Not in the parameter hash: it changes no byte for `comline@3` peers. Valid only when the package has a release built on the old wire | + +## Which dependency protocols join + +- **Reached by `use`.** A dependency's protocol is on the package's + connection when a schema of the package `use`s it: + + ```ids + use localdeps::greeting::Foo + ``` + + That's the same rule [Dependencies](../schema-compatibility.md#dependencies) + uses for versions. Declaring a dependency puts nothing on the wire, and + `use` of a protocol is new: today `use` names types and other declarations. +- **All three encodings must match.** A dependency's protocol joins only when + its own resolved framing, handshake and format are the connection's, by + parameter hash in the dependency's facts. They're part of the dependency's + contract (changing them is major there), and its shape hash covers what it + puts on the wire in its own format, so the consumer can't re-host it on + others. Otherwise it gets a [connection of its own](connection-per-package.md), + and the build says which encoding differs. +- **One version per package per end.** An end lists each package once. If + the package reaches one dependency through two paths that pin different + versions, the build refuses it, naming both paths. +- **Every package needs an `id`.** `comline@3` identifies each package by its + id, and a dependency built without one can't join. +- **Releases only, for releases.** Numbers are + [bound at release](../../reference/versioning.md#number). `comline generate` + takes a dependency's numbers from its pinned version, and warns when that + version is a pre-release. `comline release` refuses a dependency at a + pre-release version whose protocols are on its connection, because + [consumers only see releases](../schema-compatibility.md#builds-and-releases). + +## Implications + +### For the wire and peers + +- **A wire change for every package that takes the defaults.** Switching the + defaults from `datagram@3` + `comline@2` to `datagram@4` + `comline@3` + moves every protocol's line, so the next release is a breaking one (minor + in 0.x). As with the last change of defaults, it should come with a new + edition. A version built before it reads as built, and stating `wire = + { framing = "datagram@3" }` and `handshake = "comline@2"` keeps the old + wire. +- **`comline@2` peers during the migration (dual-accept).** A package that + sets `accept_previous = true` also answers a `comline@2` initiator. An + initiator always speaks its own version, so only responders do this. + - **It switches the whole wire, not only the handshake.** The responder + reads the version byte (`02` or `03`) and runs that exchange. A `comline@2` + connection then runs as `comline@2` would: one package, `datagram@3`, + protocols by package wire number, and no dependency protocols. + - **It serves the old contract.** Switching the defaults moved every line, + so answering an old peer with the new release's facts would refuse every + protocol (`line_mismatch`). The old-wire path presents the **last release + built on the old wire**: its version, lines and Hello. Generated code + embeds those facts beside the new ones, read from history by the CLI. + - **It lives in the shared runtime core,** written once for every + language that binds it, beside the `comline@2` exchange the C ABI already + runs. A connection's framing is decided at connect time, as an enum of the + two connection types, so nothing branches per call afterwards. The ABI + gains a mode where the handshake chooses the framing (additive, at 0.x). + A host running without the core, such as TypeScript's fallback, refuses + old peers (`handshake_version`), and its support claim says so. + - **It's temporary.** `accept_previous` is dropped in a later edition. + - A package that doesn't set it refuses old peers (`handshake_version`). + A gateway translating `comline@2` to `comline@3` is possible for + operators, but it breaks end-to-end agreement, so it isn't the mechanism. +- **The Hello grows** by 16 bytes for each extra package id, plus a version + for each. It's sent once per connection. +- **Dependency versions become visible to peers.** The Hello carries the + version of every package on the connection, so a peer learns which version + of each dependency the other end runs. `comline@2` already reveals the + package's own version. Where that matters, the package keeps the + dependency off the connection ([C](connection-per-package.md)). +- **Partial connections become routine.** A peer built against an older + `localdeps` sees `Foo` refused (`line_mismatch`) while `Thing` works. + Generated code already returns `Unavailable` for a refused protocol, so a + caller must expect it for any dependency protocol. + +### For versions + +- **The consumer's version covers which protocols its connection carries.** + Adding a dependency's protocol is minor, and removing one is major, just like + adding or removing a function. Moving a dependency's protocol between this + connection and a [connection of its own](connection-per-package.md) is + major. +- **The protocol's own changes are the dependency's.** They move its line in + the dependency's history, and the handshake compares the dependency's + versions, so the consumer doesn't re-release them. Its reach into the + dependency's types counts as it does today. +- **Upgrading a dependency is a consumer change** only as far as it reaches: + a dependency release that moves `Foo`'s line makes `Foo` unavailable to + peers on the old line, without touching `Thing`. + +### For generated code and runtimes + +- **Protocol numbers aren't constants any more.** Generated code names a + protocol by (package id, wire number). The runtime maps that to a connection + number after the handshake, and each call pays one table lookup. Under + `datagram@3`, the number is a constant in the code. +- **A connection's providers and senders span packages.** `Services`, `serve` + and `connect` take the dependency's providers too. The dependency's + generated module exposes its providers and senders, and the consumer's + connection carries them. +- **Shared connection state.** A dependency's handlers see the connection's + extensions: authentication, session, whatever interceptors put there. That + includes the application's state. A dependency that shouldn't see it + belongs on its own connection. +- **Shared order.** A connection runs one call at a time by default, so a + slow dependency handler holds up the application's calls (head-of-line + blocking). The async driver's concurrency applies across packages. +- **Errors stay per schema.** A reply names its protocol, so an error ordinal + is read in the error space of the schema that declares that protocol, in + whichever package. +- **`comline@3` runs in every runtime.** That means the Rust and TypeScript + runtimes, and the shared core with its C ABI and WASM build + ([Shared runtime core](../shared-runtime-core/index.md)), still at 0.x, + where a new handshake is additive. + +### For the toolchain + +- **`comline generation` reads dependency history.** For each dependency + whose protocols are on the connection, it reads that dependency's facts + from its history at the pinned version. A dependency fetched without its + `.comline` history can't provide them, so publishing has to include the + facts. +- **`comline inspect`** shows the connection's packages, each protocol's + connection position, and the [capabilities](capabilities.md#capabilities-what-a-pairing-can-do). +- **Conformance** gains `handshake-comline3` and `framing-datagram4` vectors + and features, and each generator claims them. +- **JSON-RPC can't follow yet.** A qualified method name is + `namespace.Protocol/function`, unique in a package but not across packages, + so two packages' protocols can collide. Carrying several packages on + JSON-RPC would need a `method_names` value that names the package too: a + parameter that sets `protocols`, not a new version. + +## What it touches + +- **runtime:** `comline@3`'s messages, decisions and vectors; `datagram@4` + and its vectors; dual-accept in the shared core, with a connection decided + at connect time and an ABI mode where the handshake picks the framing, and + vectors for a `comline@3` responder answering a `comline@2` initiator; the table from (package id, wire number) to connection + number; `Negotiated::check_call` and the dispatch table keyed by connection + number; profiles with the [new facts](capabilities.md#proposed-facts). +- **generation:** `GenerateRequest` carries facts per package: the package's + own, plus one set per dependency reached. `check`, `addressing` and `hello` + look each protocol up in its own package's facts. Generated code names a + protocol by (package id, wire number) and resolves it after the handshake. + Under `accept_previous`, it also embeds the last old-wire release's facts. +- **cli:** `comline generation` reads and passes each reached dependency's + facts, and under `accept_previous` the last old-wire release's, and enforces + the release rule above. Publishing includes facts. +- **core:** `use` of a protocol, reaching a dependency's protocol through it, + the consumer's version rules + above, the build errors of [Part 1](capabilities.md#rules-the-facts-make-checkable), + and `comline@3` and `datagram@4` as official encodings. + +## As decided + +Decided 2026-11-14, with the [section](index.md#decided): + +- **Reach is by `use`** of the protocol ([above](#which-dependency-protocols-join)). +- **Framing, handshake and format must all match** for a dependency's + protocol to join. Otherwise it takes a connection of its own. +- **`scope` is `protocol` | `each_package` | `package`**, where `package` + keeps meaning the whole connection ([Parameters](#parameters)). +- **The default in the next edition** is `comline@3` with `datagram@4`. +- **Dual-accept** is opt-in per package (`accept_previous`, outside the + parameter hash), lives in the shared runtime core, and serves the last + old-wire release's facts, which generation embeds + ([above](#for-the-wire-and-peers)). +- **`use` of a protocol is a reach only,** for now: it puts the protocol on the + connection, and doesn't alias or re-export it. +- **`datagram@4`'s layout** is fixed widths, a `u32` request id, a one-way + kind, and the protocol kept in replies + ([why](datagram@4.md#why-fixed-widths)). The runtime's benchmark gates + should confirm it against `datagram@3`. + +## Open questions + +- **How `comline inspect` shows** which protocols a connection carries, + its own and its dependencies'. diff --git a/docs/docs/proposals/extensibility.md b/docs/docs/proposals/extensibility.md index 4e946df..cc06811 100644 --- a/docs/docs/proposals/extensibility.md +++ b/docs/docs/proposals/extensibility.md @@ -244,7 +244,7 @@ What decides a connection under `comline@2`: the line follows the major version; before 1.0 it follows the minor, as Cargo's `0.x` convention does. - **The rest is situational, so it's a parameter of the handshake,** - declared in `config.idp` like [framing instances](packet-dispatch.md#where-the-framings-parameters-are-written). + declared in `config.idp` like [framing instances](packet-dispatch/index.md#where-the-framings-parameters-are-written). The defaults are shown: ``` diff --git a/docs/docs/proposals/function-ids/index.md b/docs/docs/proposals/function-ids/index.md index c2911d2..710a6fe 100644 --- a/docs/docs/proposals/function-ids/index.md +++ b/docs/docs/proposals/function-ids/index.md @@ -9,7 +9,7 @@ > protocol. See > [its page](annotation.md#as-accepted). -**The question.** [Packet dispatch](../packet-dispatch.md#explicit-ids-available-everywhere) +**The question.** [Packet dispatch](../packet-dispatch/index.md#explicit-ids-available-everywhere) lets any function state an explicit, fixed id, and requires one under a packet framing. **How is the id written?** diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index 83e75fa..d443eb0 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -166,6 +166,7 @@ Seven parts, read in order: [the formats](ir-migrations/formats.md), built-in framing, a new packed C-struct-like format `packed@1` as the built-in format, and a generated `Services` struct with a package-level `serve`. ***implemented.*** + - [Return forms](return-forms.md): one-way, acknowledged (`-> ()`) and value returns, and quiet calls (errors only) as a future form; which each framing and transport supports, checked at build, and `()` only as a return. @@ -175,6 +176,31 @@ Seven parts, read in order: [the formats](ir-migrations/formats.md), message and whether it loses, reorders or duplicates; a "transport fit" guarantee checked only where declared. *draft.* +#### [A dependency's protocols on a connection](dependency-protocols/index.md) + +Which numbers address a dependency's protocol on a connection, and which +version the handshake checks it against. ***Decided*** 2026-11-14: A and C, +with Part 1, and step 0 (a protocol the package doesn't reach isn't +addressed): + +- [Part 1 — what framings and handshakes state](dependency-protocols/capabilities.md): + new profile facts for how a frame names its protocol, whether it marks + one-way calls, and how many packages a handshake covers; `limits` for the + caps an encoding's bytes impose; the capabilities they derive, and what + changing an encoding implies. ***accepted.*** +- [Proposal A — several packages on one connection](dependency-protocols/multi-package-connection.md): + `comline@3` lists each package with its own id, version and numbers, and + binds a connection number per protocol, which + [`datagram@4`](dependency-protocols/datagram@4.md) addresses calls by in a + fixed-width header. Dependency protocols join by `use`. ***accepted***, the + defaults from the next edition. +- [Proposal B — the consumer's numbers](dependency-protocols/consumer-numbers.md): + the consumer's history numbers each dependency protocol it carries. + *rejected as a default: a second identity per protocol.* +- [Proposal C — a connection per package](dependency-protocols/connection-per-package.md): + each package on its own connection. ***accepted***, complementary to A: for + differing encodings and isolation. + ### Systems - [IPC](ipc.md): interfaces between processes on one machine: IPC transports, @@ -187,7 +213,7 @@ Seven parts, read in order: [the formats](ir-migrations/formats.md), ### Packet protocols -- [Packet dispatch](packet-dispatch.md): message-oriented packet protocols +- [Packet dispatch](packet-dispatch/index.md): message-oriented packet protocols (families, explicit ids, phases, `@provider` for direction), with addressing in the schema, generated dispatch tables, and an official configurable framing. *under discussion, partly decided.* @@ -257,8 +283,10 @@ the language side is shipped and documented in the repository can be watched over time, and any published versions compared; each repository publishes to its own `reports` branch. ***accepted*** — `observatory`, all four stages shipped. -- [Automatic documentation](auto-documentation.md) — an optional tool that +- [Automatic documentation](auto-documentation/index.md) — an optional tool that documents a user's own schemas and packages — their structs, enums, protocols, errors — from the IR and its docstrings, with the CAS supplying per-version docs and a generated changelog. ***accepted*** — a JSON doc model with - external renderers, configured in `comline.toml` only. + external renderers, configured in `comline.toml` only. A follow-up, + [links, the overview and highlighting](auto-documentation/links-overview-highlighting.md), + is a *draft*. diff --git a/docs/docs/proposals/ipc.md b/docs/docs/proposals/ipc.md index daaa1cb..56d33b1 100644 --- a/docs/docs/proposals/ipc.md +++ b/docs/docs/proposals/ipc.md @@ -4,7 +4,9 @@ Status: **draft** · Handles are typed, one type per kind (decided 2026-11-02) · Builds on [Transports](transports.md) and [Connections](connections.md) · Handles answer part of the open [capabilities design](https://github.com/ComlineProject/docs/issues/98) · The -first step of [Comline for systems software](systems-software.md) +first step of [Comline for systems software](systems-software.md) · +[Framings](#framings) laid out against +[Dependency protocols, Part 1](dependency-protocols/capabilities.md) **The question:** what does Comline need to be a good interface between processes on one machine, beyond what a network connection needs? @@ -138,6 +140,154 @@ Adding rights later tightens existing protocols, a breaking change, so it's best settled before 1.0. If rights come, R2 is the recommendation: read-only sharing is the real security need, and its small set works everywhere. +## Framings + +What runs inside a connection over an IPC transport. A framing is described +by the facts and limits of +[Dependency protocols, Part 1](dependency-protocols/capabilities.md). IPC +asks for things no network framing has had to state, and a framing that +can't state them doesn't belong over IPC. + +There are three kinds: + +- Comline's own framings, run over IPC transports. +- Framings for **channels handed over in a call** (`remote P`, `receiver P`). +- Each **platform's own** framing, for talking to its services. + +### What IPC asks of a framing + +| Need | Why | What it asks | +| --- | --- | --- | +| **Handles belong to one frame** | the decoder checks that a message carries exactly the handles its body refers to | something must say which handles arrived with which frame: the transport or the framing ([below](#who-ties-handles-to-a-frame)) | +| **Low per-call overhead** | calls take microseconds, so the header is a large share of the work | fixed-width headers at fixed offsets, with no varints ([`datagram@4`](dependency-protocols/datagram@4.md#why-fixed-widths)) | +| **One-way calls and events** | OS services push events and notifications | one-way calls marked in the frame, so a receiver never answers one | +| **Calls in both directions** | callbacks into the client | `directions = both` | +| **Handles in replies** | `-> file` | a reply carries handles like a request does. A fault carries none, and whatever the provider meant to send is closed | +| **Channels created by calls** | `remote P` and `receiver P` | a framing and an agreement for the new channel ([below](#channels-handed-over-in-a-call)) | +| **A peer that dies** | the OS reports it | nothing from the framing: the transport closes the connection, and every pending call fails | + +### Who ties handles to a frame + +| | Who | How | Trade-off | +| --- | --- | --- | --- | +| **T1** | the transport | A message transport (`unix-seqpacket@1`, Zircon and Mach channels, Windows message pipes, `postmessage@1`) delivers handles with the message. A stream transport (`unix-stream@1`) sends a frame's handles in the same `sendmsg` as the frame, and its length prefix gains a handle count, so the receiver takes exactly that many | frames are the same bytes over TCP and IPC, and the framing states nothing new. Each stream transport needs the count | +| **T2** | the framing | the frame's header states how many handles it carries, as D-Bus's `UNIX_FDS` field does | every framing that runs over IPC grows a field, or gets an IPC version of its own | + +**Decided 2026-11-14: T1.** It's the model above, where handles travel beside the bytes: +the transport carries the handles, the format encodes their indices, and the +framing carries the bytes. Comline's own framings then state `handles = +by_transport`. A platform's framing that counts or lists handles states that +instead (see [Facts IPC adds](#facts-ipc-adds)). + +The most handles one message can carry is the transport's limit (253 file +descriptors per message on Linux, 64 handles on a Zircon channel), unless a +framing that counts them states a lower one. The build checks each call's +largest number of handles against it, which is why an array of handles needs +a maximum. + +### Comline's own framings over IPC + +| framing + handshake | Over IPC | Request / reply header | +| --- | --- | --- | +| [`datagram@4`](dependency-protocols/datagram@4.md) + `comline@3` | **the default:** fixed widths, one-way marked, several packages, compatible versions connect | 9 / 7 bytes, a one-way 5 | +| `datagram@3` + `comline@2` | works, for one package. Varint numbers cost branches on every call | 11 to 15 / 10 to 12 bytes | +| `jsonrpc@1` | for scripted peers: Node, browsers over `postmessage@1`, Electron | JSON text | +| `datagram@2` + `comline@1` | works, but every peer must be rebuilt from identical schemas, which partial updates break | 12 / 10 bytes, a protocol past 127 one more | + +The handshake costs one message each way, once per connection, and IPC +connections usually live as long as the process. + +### Channels handed over in a call + +A call returning `remote Session` hands the caller a new channel endpoint, +already typed by the schema. What runs on that channel: + +- **The framing and format are the parent connection's.** They're stated once, + by the package, not chosen per channel. +- **Every child channel starts with a handshake** (decided 2026-11-14). The + connection's handshake, `comline@3`, runs on it before the first call, with + a Hello of one package and one protocol. That protocol is connection number + 0. +- **Why not inherit the parent's agreement.** It's tempting: the parent's + handshake already agreed the schema that types the endpoint, and Mojo and + FIDL skip a handshake per pipe or channel. But endpoints can be + **forwarded**. An endpoint created between A and B, then sent on by B to C, + connects A and C directly, and the OS moves it, so B is out of the path. A + and C never agreed on anything, and **A can't tell** that its channel's + other end changed hands. Detecting forwarding would take a first frame that + may be a Hello or a call, and calls held until the check passes. Sending the + creator's facts along would ship data where a cheap check that can fail + does the job. A handshake on every child channel is correct whoever holds + the endpoint, and needs nobody to know. +- **What it costs:** one round trip per child channel before its first call, + with a small Hello. A design that creates many short-lived channels pays it + each time. +- **The protocol field is constant.** Under `datagram@4` a child channel's + protocol is always 0: 2 bytes per frame spent on nothing. That's cheaper + than a framing of its own. +- **Outliving the parent.** A child channel is its own channel, with its own + agreement, so it stays valid after the parent connection closes. + +### Platform IPC framings + +Each platform's IPC has its own message layout, so each is its own framing, +often with its own format. Like a legacy packet system +([Packet dispatch](packet-dispatch/index.md#describing-legacy-protocols)), it's an +encoding a package names to talk to the platform's services, and its profile +describes it in Part 1's terms: + +| Platform | Names the function | Names the protocol | Matches a reply | One-way | Handles | Errors | +| --- | --- | --- | --- | --- | --- | --- | +| **D-Bus** | by its member name | by its interface name, plus an object path | the reply's serial (`u32`) in the header | a `NO_REPLY_EXPECTED` flag | counted in the header (`UNIX_FDS`) | by error name | +| **Binder** (AIDL) | a transaction code, from declaration order unless stated | an interface token, per object | the reply comes back on the calling thread | `FLAG_ONEWAY` | objects and file descriptors in the parcel, listed by offset | in the reply parcel | +| **Mojo** | a method ordinal, from declaration order unless stated | per message pipe; an associated interface gets an id bound at run time | a request id (`u64`) | a flag: expects a response | with the message | in the response | +| **FIDL** over Zircon | a 64-bit hash of the method's fully qualified name | none: one protocol per channel | a transaction id (`u32`), 0 for one-way | transaction id 0 | with the message, up to 64 | in the response, as a result union | + +They confirm two of Part 1's values in practice: D-Bus is the first real +`errors = by_name`, and Mojo's associated interfaces are `protocols = +by_binding`. They also need values Part 1 doesn't have (below). + +### A shared-memory ring + +`memory-ring@1` carries frames in its slots. `datagram@4`'s frames fit +unchanged, and its fixed header can be read in place in shared memory. + +Handles can't cross the ring, so a frame carrying handles goes over the side +channel, and two channels lose the order between them. + +**Decided 2026-11-14: a marker in the ring.** A frame with handles is sent over +the side channel, and the ring holds a small marker in its place. A receiver +reading the ring stops at a marker until the matching frame has arrived on the +side channel, then carries on. Frames without handles, the common case, cost +nothing. (A sequence number on every frame was the alternative: simpler to +reason about, but bytes on every frame.) + +### Facts IPC adds + +To [Part 1](dependency-protocols/capabilities.md): + +| Addition | Kind | Values | Needed by | +| --- | --- | --- | --- | +| `handles` | framing fact | `none` (the framing can't carry any) · `by_transport` (the transport ties them to the frame) · `counted` (the header counts them) · `indexed` (the frame lists where each one is) | every framing over IPC; Comline's own state `by_transport`. **Adopted** into Part 1 (2026-11-14) | +| `handles_per_frame` | framing limit | a number, for framings that count or list handles | D-Bus, Binder. **Adopted** into Part 1 (2026-11-14) | +| `calls = by_hash` | a new value | a hash of the function's qualified name | FIDL | +| `errors = in_result` | a new value | errors travel in the response's payload, not the frame | Binder, Mojo, FIDL | +| `correlation = synchronous` | a new value | a reply is matched by the calling thread | Binder | + +One gap needs more than a value: **instances.** A D-Bus or Binder frame names +an *object* as well as its interface, so many objects serve the same +protocol on one connection. Comline has one provider per protocol per +connection. `remote P` endpoints cover most of the need, each object +being its own channel, but a platform framing that addresses objects needs a +way to say so. + +**D-Bus is the first platform framing** (decided 2026-11-14). It brings +`errors = by_name`, `handles = counted` with a `handles_per_frame` limit, and +**instances**, through object paths. Instances are therefore the next design +this needs, and no longer deferred: how a schema or a connection says which +object a call addresses, and how generated code serves several objects of one +protocol. + ## Who the peer is The OS reports the peer: its process id, user and group (`SO_PEERCRED` on @@ -185,7 +335,16 @@ in [Comline for systems software](systems-software.md#an-exact-layout-format). - ~~**Handles**~~: decided, H2. The spellings (`file`, `memory`, `channel`, `remote P`, `receiver P`) are as proposed. - **Rights** on handles: R1, R2 or R3 ([above](#rights)), settled later. -- **Which platform IPCs** get official profiles, and in what order. +- **Which platform IPCs** get official profiles: D-Bus first (decided + 2026-11-14, [above](#facts-ipc-adds)); the order after it is open. - **A shared-memory ring** of Comline's own, and its side channel for peer death. - **Requiring a peer identity check:** how a protocol states it. +- ~~**Who ties handles to a frame**~~: decided, the transport (T1) + ([above](#who-ties-handles-to-a-frame)). +- ~~**Forwarded endpoints**~~: decided, every child channel starts with a + handshake ([above](#channels-handed-over-in-a-call)). +- ~~**The ring's order**~~: decided, a marker in the ring + ([above](#a-shared-memory-ring)). +- **Instances:** how a call addresses one of several objects serving a + protocol. Needed for D-Bus, so it's next ([above](#facts-ipc-adds)). diff --git a/docs/docs/proposals/packet-dispatch/comline.packet@1.md b/docs/docs/proposals/packet-dispatch/comline.packet@1.md new file mode 100644 index 0000000..901e7ea --- /dev/null +++ b/docs/docs/proposals/packet-dispatch/comline.packet@1.md @@ -0,0 +1,266 @@ +# `comline.packet@1` + +Status: **draft** · the official packet framing of +[Packet dispatch](index.md#the-mechanism-plus-an-official-framing) · its +parameters' facts and limits are [decided there](index.md#what-each-parameter-sets) · +its layout [decided](#as-decided) 2026-11-14, tracked in +[core#123](https://github.com/ComlineProject/core/issues/123) · +moves to the [framings reference](../../reference/wire/framings/index.md) +once built + +`comline.packet@1` is Comline's configurable framing for **packet protocols**: +packets addressed by explicit ids, optionally grouped into families, with +phases, in both directions, with or without replies. One framing covers the +common header layouts through its parameters. A protocol Comline didn't +design, whose header no combination of parameters matches, gets a framing of +its own instead ([Packet dispatch](index.md#describing-legacy-protocols)). + +## Parameters + +Written in a named framing instance in `config.idp` +([where](index.md#where-the-framings-parameters-are-written)): + +```idp +framings = { + "game" = { + encoding = "comline.packet@1" + length = "varint" + family = "u8" + id = "u8" + endianness = "little" + correlation = { header_id = "u32", status = "u16" } + } +} +``` + +| Parameter | Values | Default | Controls | +| --- | --- | --- | --- | +| `length` | `none`, `u16`, `u32`, `varint` | `u32` | the length prefix; `none` leaves frames to the transport | +| `family` | `none`, `u8`, `u16` | `u8` | the family field; `none` makes ids unique across the phase's protocols | +| `id` | `u8`, `u16`, `u32`, `varint` | `u16` | the packet id field | +| `endianness` | `little`, `big` | `little` | every fixed-width integer in the header | +| `correlation` | `none`, or `{ header_id = "u16" \| "u32", status = "u8" \| "u16" }` | `none` | replies: the kind byte, the correlation id and the status | + +Every parameter is in the parameter hash, so two peers that differ in any of +them are refused by a handshake that checks encodings. + +## Requirements + +- **Explicit ids.** Every packet has an `@id`, and every protocol has one as + its family id unless `family = "none"`. The build refuses a packet without + one. +- **Ids that fit.** Each id fits its field, as the + [limits](#limits) state, checked at build. +- **Unique ids.** Ids are unique per phase and direction: among a family's + packets, or under `family = "none"`, among every packet active in the phase + and direction. +- **A transport.** With `length = "none"` the transport must delimit frames. + Otherwise any byte stream will do. +- **Any format** whose `output` is `bytes` or `json`. The payload is opaque. + +## Integers + +- **Fixed-width** fields (`u8`, `u16`, `u32`) are in the instance's + `endianness`. That covers the length prefix, family, id, correlation id and + status. +- **`varint`** fields are unsigned LEB128, always low bits first, whatever the + `endianness`. Only the shortest form is accepted, and a value above the + field's limit is malformed. Minecraft's VarInt reads the same way for + non-negative values. +- **The payload isn't affected.** It's encoded by the connection's format, + with the format's own byte order (`packed@1` is little-endian). A legacy + big-endian payload needs a big-endian format, not this parameter. + +## Frames + +A frame is a header followed by a payload. The header's fields come in this +order, each only when its parameter puts it there: + +| # | Field | Present when | Width | +| --- | --- | --- | --- | +| 1 | length | `length` isn't `none` | `length` | +| 2 | kind | `correlation` isn't `none` | 1 byte | +| 3 | family | `family` isn't `none`, and the frame is a packet (not a reply) | `family` | +| 4 | id | the frame is a packet (not a reply) | `id` | +| 5 | correlation id | the frame is a request or a reply | `header_id` | +| 6 | status | the frame is a reply | `status` | + +**The length** counts every byte after the length field itself, so a reader +knows where the frame ends before reading anything else. + +### Kinds + +Under `correlation = none` there's no kind byte. Every packet is one-way, and +replies don't exist. + +With correlation, the kind byte says what the frame is: + +| Kind | Frame | Fields after the kind | +| --- | --- | --- | +| `00` | request: a packet whose function has `->` | family, id, correlation id, arguments | +| `01` | reply | correlation id, status, outcome body | +| `02` | one-way packet: a function without `->` | family, id, arguments | +| `03` | control frame | control code, its body | + +A packet's kind must match its function's declaration. A mismatch is +malformed. + +### Replies + +A reply names no family and no id. It's matched to its request by the +correlation id alone, and decoded by the requester's record of that request, +never by the phase's tables. So a reply that arrives after a phase transition +still decodes ([Packet dispatch](index.md#how-a-connection-moves-between-phases)). + +**The status** is the reply's outcome: + +| Status | Outcome | Body | +| --- | --- | --- | +| 0 | ok | the result, encoded by the format | +| 1 | fault | nothing | +| 2 and up | the error with ordinal *status − 2* | the error's fields, encoded like a struct | + +A `u8` status allows 254 errors, and a `u16` 65534. + +### Correlation ids + +Each end numbers its own requests, starting at 0 and adding 1 per request, +wrapping at the width's maximum. Replies travel the other way, so the two +ends' numbers may overlap freely. A caller that gives up on a request doesn't +reuse its id until the counter wraps. A reply to no pending request is +dropped, which is what a late reply after a timeout is. + +### Control frames + +Frames the framing sends itself, outside the schema's id tables. There's one +today: the **phase acknowledgement**, the first frame a receiver sends in a new +phase, when the old phase had packets travelling back +([Packet dispatch](index.md#how-a-connection-moves-between-phases)). + +| Field | Bytes | Meaning | +| --- | --- | --- | +| control code | `00` | phase acknowledgement | +| phase | varint | the phase entered: its variant's position in the `@phases` enum | + +How a control frame is told apart depends on the instance: + +- **With correlation:** kind `03`. +- **Without correlation:** the **largest value of the family field** marks a + control frame (`0xFF` under `family = "u8"`), with no id after it. Under + `family = "none"`, the **largest value of the id field** does. That value is + reserved, so the [limits](#limits) are one lower. +- **Under `id = "varint"` with `family = "none"`** there's no largest value. + Value 0 is reserved instead, and ids start at 1. + +An acknowledgement for a phase other than the one the sender just entered is +malformed. + +## Layouts at a glance + +| Instance | One-way packet | Request | Reply | +| --- | --- | --- | --- | +| Minecraft-like: `length = varint`, `family = none`, `id = varint`, no correlation | len (varint) + id (varint) + payload | (none) | (none) | +| Game opcodes: `length = u16`, `family = u8`, `id = u8`, no correlation | 2 + 1 + 1 = **4 bytes** | (none) | (none) | +| RPC over packets: `length = u32`, `family = u8`, `id = u16`, `header_id = u32`, `status = u16` | 4 + 1 + 1 + 2 = **8 bytes** | 4 + 1 + 1 + 2 + 4 = **12 bytes** | 4 + 1 + 4 + 2 = **11 bytes** | +| Over a message transport: `length = none`, `family = u8`, `id = u8`, `header_id = u16`, `status = u8` | 1 + 1 + 1 = **3 bytes** | 1 + 1 + 1 + 2 = **5 bytes** | 1 + 2 + 1 = **4 bytes** | + +For example, the game-opcode instance (little-endian) sending family 2's +packet `0x10` with a 3-byte payload `aa bb cc`: + +```text +0500 02 10 aabbcc +``` + +The RPC instance replying to request 7 with error ordinal 1 and an empty +`packed@1` body: + +```text +07000000 01 07000000 0300 +``` + +## Receiving + +| Received | What the receiver does | +| --- | --- | +| a frame that doesn't parse: a short header, an overlong or out-of-range varint, an unknown kind, a length past `frame_bytes` | ends the connection: it's malformed | +| a length prefix that disagrees with the frame the transport delivered | ends the connection: it's malformed | +| a packet whose family and id are active in this phase and direction | dispatches it; a request gets a reply | +| a packet whose family or id isn't active in this phase and direction | ends the connection: an out-of-phase or unknown packet is a decode error ([decided](index.md#how-a-connection-moves-between-phases)) | +| a packet whose kind disagrees with its function (`00` for one-way, `02` for `->`) | ends the connection: it's malformed | +| a reply whose correlation id awaits no reply | drops it | +| a reply with a status past the caller's known errors | gives the caller a remote error: the provider is newer | +| a control frame with an unknown code, or an acknowledgement for the wrong phase | ends the connection: it's malformed | + +Unlike `datagram@4`, an unknown packet ends the connection instead of getting +a fault. A packet system's tables are fixed and agreed, so an unknown id means +the two ends are out of step, and nothing after it can be trusted. + +## Profile + +Its facts and limits depend on the parameters +([decided in Packet dispatch](index.md#what-each-parameter-sets)), stated in +[Part 1](../dependency-protocols/capabilities.md)'s terms: + +| Fact | Value | +| --- | --- | +| `calls` | `by_id` | +| `protocols` | `by_id`, or `in_call` under `family = "none"` | +| `directions` | `both` | +| `errors` | `by_ordinal` | +| `payload` | `opaque` | +| `correlation` | `none` or `header_id` | +| `one_way` | `in_frame` with correlation (kind `02`). Without it every packet is one-way | +| `handles` | `none` | + +### Limits + +| Limit | Value | +| --- | --- | +| `protocols` | 2⁸ or 2¹⁶ per package, by `family`, one less without correlation (a control value) | +| `functions_per_protocol` | 2⁸, 2¹⁶, or core's 2¹⁶ for `u32` and `varint`, by `id`. One less when the id field carries the control value. Under `family = "none"`, this caps all packets in a phase and direction together | +| `errors_per_schema` | 254 or 65534, by `status` | +| `pending_requests` | 2¹⁶ or 2³² per direction, by `header_id` | +| `frame_bytes` | 65535 or 4294967295 by `length` (the length counts the bytes after itself), or the transport's | + +## What it can't do + +- **Describe another system's header** unless a combination of parameters + matches it exactly. Minecraft-style headers fit, and most others need a + framing of their own. +- **Answer an unknown packet.** It ends the connection. +- **Carry several packages by binding.** Ids are fixed in the schema. A + dependency's packets join only when the build proves their ids disjoint + ([Packages on a packet connection](index.md#packages-on-a-packet-connection)). +- **Carry handles.** +- **Stream, cancel or fragment** a packet. + +## Proven by + +`framing-packet1` vectors, byte-exact, for each parameter value at least once: + +- every field at its widths and both byte orders, and varints at 127, 128, + 16383 and 16384; +- each kind, each status outcome, and each control frame, in instances with + and without correlation and family; +- rejections: short headers, overlong varints, unknown kinds, a mismatched + length, a packet's kind disagreeing with its function, an out-of-phase + packet, an acknowledgement for the wrong phase; +- correlation ids wrapping, and a late reply dropped; +- each limit at its maximum, and the reserved control value refused as a + packet id. + +## As decided + +Decided 2026-11-14, each as written above: + +| | Question | Decided | Alternatives weighed | +| --- | --- | --- | --- | +| **P1** | Header field order | length, kind, family, id, correlation id, status | kind before length; correlation id before family and id | +| **P2** | How a control frame is told apart without correlation | the largest family value, or the largest id value under `family = "none"`; value 0 under a varint id | a kind byte on every frame (1 byte per packet); a kind byte only when the schema has acknowledged transitions (the layout then depends on the schema) | +| **P3** | One-way packets under correlation | their own kind, `02`, with no correlation id | kind `00` with a correlation id nobody answers | +| **P4** | Whether a reply names its family and id | no: matched by correlation id, decoded by the requester's record | yes, as `datagram@4` keeps its protocol: attributable in captures, 2–6 bytes more per reply | +| **P5** | Status values | 0 ok, 1 fault, 2 and up errors | 0 ok, errors from 1, the largest value fault | +| **P6** | What `endianness` covers | every fixed-width header integer, the length included; never varints or the payload | the length prefix fixed little-endian; or the payload's format too | +| **P7** | An unknown packet in the active phase | ends the connection, as an out-of-phase one does | a fault reply for a correlated request, ending the connection only for one-way packets | +| **P8** | Correlation ids | sequential per direction, wrapping; late replies dropped | any id unique among pending requests | +| **P9** | Defaults | `length = u32`, `family = u8`, `id = u16`, `little`, no correlation | no defaults: every parameter written out in each instance | diff --git a/docs/docs/proposals/packet-dispatch.md b/docs/docs/proposals/packet-dispatch/index.md similarity index 75% rename from docs/docs/proposals/packet-dispatch.md rename to docs/docs/proposals/packet-dispatch/index.md index a6e61c9..77a9357 100644 --- a/docs/docs/proposals/packet-dispatch.md +++ b/docs/docs/proposals/packet-dispatch/index.md @@ -2,7 +2,7 @@ Status: **under discussion** · tracked in [core#123](https://github.com/ComlineProject/core/issues/123) · partly decided, -nothing implemented +nothing implemented · [`comline.packet@1`](comline.packet@1.md) specified, draft **The question:** how Comline describes **message-oriented packet protocols**, the slim systems that typically come from C, and generates their dispatch. In @@ -29,14 +29,14 @@ They vary along independent axes: 1. **Only the client sends unprompted.** Calls go from client to server, and one-way functions exist only in that direction. Server push is listed as - undesigned in the [target contract](../design/core-target-contract.md). + undesigned in the [target contract](../../design/core-target-contract.md). 2. **Function ids are implicit.** Datagram addresses a call by the function's position in its protocol. Packet systems need **explicit, fixed, possibly sparse** ids, because those ids *are* the contract. 3. **A connection serves one protocol,** so there are no families. 4. **There are no phases.** 5. **A framing can't read addressing from the schema.** Under - [the extensibility proposal](extensibility.md#what-a-format-cant-be), a + [the extensibility proposal](../extensibility.md#what-a-format-cant-be), a framing that needs more than the generated data model is a language feature, and packet addressing is exactly that. @@ -44,7 +44,7 @@ They vary along independent axes: **Addressing** is the family, the id, the direction and the phase. It goes in the schema: both ends must agree on it, so it's part of the contract and of the -[version comparison](schema-compatibility.md). Generators turn it into +[version comparison](../schema-compatibility.md). Generators turn it into **dispatch tables**, mapping (phase, direction, family, id) to a decoder and a handler, plus the phase transitions. **A framing only lays out the header bytes,** looking ids up in those tables. A legacy system's exact header is then @@ -65,7 +65,7 @@ and already used in `runtime-std-extra`'s `reverse_proxy.ids`: `@consumer` follows from it. Writing both is allowed, but a mismatch is an error. The other reading of the 2024 file, *how many* peers provide or consume (a fleet behind a reverse proxy), is dropped; if that need returns, - it belongs with [connections](connections.md). + it belongs with [connections](../connections.md). - **Server-to-client packets** are a protocol with `@provider = Client`. The client gets the dispatcher and the provider trait for it, and the server gets the sender. @@ -105,7 +105,7 @@ protocol LoginClientbound { - In the version comparison, changing an id is major, and adding a function with a new id is minor. - **The syntax is decided:** `@id = 0x2A` - ([Function ids](function-ids/index.md), + ([Function ids](../function-ids/index.md), [core#127](https://github.com/ComlineProject/core/issues/127)). Ids can be assigned on build and written into the source, and reusing a retired id is an error. @@ -114,7 +114,7 @@ protocol LoginClientbound { A family is a protocol with an explicit **family id**, so several protocols share one connection. That's the identifier the schema states, the choice the -[rename rules](schema-compatibility.md#renames) asked for: renaming a protocol +[rename rules](../schema-compatibility.md#renames) asked for: renaming a protocol or namespace changes code, not the wire. Whether the family id uses the same syntax as function ids follows from that decision. @@ -124,7 +124,7 @@ A phase states which protocols are active in each direction, and which packet moves the connection to the next phase. Both ends generate the same state machine. The version comparison sees phases. An out-of-phase packet is a decode error, never a misread under another phase's table. The syntax is -decided in [Phase syntax](phase-syntax/index.md): phases are an ordinary enum +decided in [Phase syntax](../phase-syntax/index.md): phases are an ordinary enum marked `@phases`, and protocols and packets state their own phase and transitions: @@ -167,7 +167,7 @@ Also decided (2026-10-20): - **A transition is tied to a packet in the schema, and sending that packet is the transition.** It's the function marked `@transition` - ([phase syntax](phase-syntax/enum-and-annotations.md)). Both ends then know + ([phase syntax](../phase-syntax/enum-and-annotations.md)). Both ends then know the switch point. In Rust the connection can be typed by its phase: sending `success` consumes a `Conn` and returns a `Conn`, so sending a later-phase packet too early doesn't compile. TypeScript checks @@ -217,7 +217,7 @@ Also decided (2026-10-20): read under it. - **A legacy transition that needs one and has none** is a build error, unless the protocol gives up an agreement guarantee with - [`@unverified`](extensibility.md#giving-up-a-guarantee): that accepts the + [`@unverified`](../extensibility.md#giving-up-a-guarantee): that accepts the race, an in-flight packet read under the wrong table. - **Replies still pending at a transition go first.** A transition waits until the replies owed to old-phase requests are sent, so they're ahead of it in @@ -226,7 +226,7 @@ Also decided (2026-10-20): table, so a reply arriving after the switch still decodes correctly. - **A client-chosen target** is an enum field of the transition packet (`@transition = next`), limited by a validator - ([phase syntax](phase-syntax/enum-and-annotations.md#as-accepted)). + ([phase syntax](../phase-syntax/enum-and-annotations.md#as-accepted)). - **An out-of-phase packet** is a decode error that ends the connection, reported as a framing fault. @@ -240,7 +240,7 @@ Packet protocols do both, often within one protocol and chosen per packet: - X11: some requests have replies and some don't, and events arrive unprompted. The return forms these map to (one-way, quiet, acknowledged, value), and which -framings carry each, are in [Return forms](return-forms.md). +framings carry each, are in [Return forms](../return-forms.md). Real protocols match a reply to its request in about five ways: @@ -264,7 +264,7 @@ datagram and JSON-RPC: `->` functions mix freely in one family, as MQTT and Kafka mix them per packet: the function's signature decides. - **How a reply is matched is a fact in the encoding's - [profile](schema-compatibility.md#encoding-profiles):** `correlation`. + [profile](../schema-compatibility.md#encoding-profiles):** `correlation`. `comline.packet@1` offers `none` or `header_id` (an id of a configurable width in the header), set as a [framing parameter](#where-the-framings-parameters-are-written). @@ -330,14 +330,14 @@ structs. The second level of dispatch, hand-written in most implementations, is generated. Unions travel untagged today, so this needs tagged unions with explicit tag values ([core#126](https://github.com/ComlineProject/core/issues/126); how they're -written is weighed in [Tagged unions](tagged-unions/index.md)), which serve +written is weighed in [Tagged unions](../tagged-unions/index.md)), which serve more than packets. ### The mechanism, plus an official framing - **For anyone:** addressing in the schema and generated dispatch tables. A custom framing (`com.acme.legacy@1`, mapped per target as in - [the extensibility proposal](extensibility.md#how-generated-code-reaches-one)) + [the extensibility proposal](../extensibility.md#how-generated-code-reaches-one)) reads them. - **Official:** `comline.packet@1`, a configurable header covering the common layouts: @@ -348,12 +348,15 @@ more than packets. - correlation: none, or a header id of a given width, which adds the request/reply kind marker and the reply status. - Its [profile](schema-compatibility.md#encoding-profiles) states that calls go - by explicit id, so reordering never matters under it. + Its [profile](../schema-compatibility.md#encoding-profiles) states that calls go + by explicit id, so reordering never matters under it. Its bytes are + specified on [its own page](comline.packet@1.md). Every parameter sets + facts or limits of that profile: see + [What each parameter sets](#what-each-parameter-sets). #### Where the framing's parameters are written -Leaning (recorded for discussion, not final): **named framing instances in +Decided 2026-11-14: **named framing instances in `config.idp`.** These extend the declared framings list ([core#117](https://github.com/ComlineProject/core/issues/117)) from plain names to named, configured entries: @@ -427,27 +430,113 @@ Decided 2026-10-20 ([core#117](https://github.com/ComlineProject/core/issues/117 - **Handshake:** packet systems usually bring their own ([runtime#18](https://github.com/ComlineProject/runtime/issues/18)), often as a phase of its own. One that doesn't establish schema agreement makes the - protocols [`@unverified`](extensibility.md#giving-up-a-guarantee). + protocols [`@unverified`](../extensibility.md#giving-up-a-guarantee). - **Dispatch becomes phase-aware:** the active tables switch when a transition packet passes, on both sides at the same point. +#### What each parameter sets + +Proposed, with +[Dependency protocols, Part 1](../dependency-protocols/capabilities.md), which +adds the facts and limits used here. `comline.packet@1` is one profile whose +parameters set its facts and limits. An instance is checked against the +profile its parameters produce, so a narrow header is caught at build, not +at run time. + +| Parameter | Values | Sets | +| --- | --- | --- | +| `id` | `u8`, `u16`, `u32`, `varint` | the limit `functions_per_protocol`: 256, 65536, or core's 65536 for `u32` and `varint` (wire numbers are `u16`). `calls = by_id` whatever the width | +| `family` | `none`, `u8`, `u16` | the fact `protocols`: `in_call` for `none`, `by_id` otherwise. The limit `protocols`: 256 or 65536 per package. Under `none`, `functions_per_protocol` caps every function on the connection together, per phase and direction | +| `correlation` | `none`, or `{ header_id = "u16" \| "u32", status = "u8" \| "u16" }`, `status` defaulting to `u16` | the fact `correlation`. The limit `pending_requests`: 2¹⁶ or 2³² per direction. The limit `errors_per_schema`: 254 under a `u8` status, 65534 under `u16` (one value is ok, one is fault). Under `none`, `->` and `!` are build errors, every frame is a request, and `directions` is `both`: with no replies, nothing needs telling apart | +| `length` | `none`, `u16`, `u32`, `varint` | the limit `frame_bytes`: 65535, 4294967295, or the transport's for `varint` and `none`. Under `none`, the transport delimits frames | +| `endianness` | `little`, `big` | no fact and no limit. It changes the bytes, so it's in the parameter hash like every parameter | + +A **family id** is the protocol's wire number, and a packet id the function's, +both written with `@id` and required. So `protocols = by_id` here means "by +the protocol's `@id`", which is unique in its package. + +**`protocols = in_call`** is the flat opcode table: the frame names no +protocol, and a packet's id alone picks it out among every protocol active in +the phase and direction. Ids must then be unique across those protocols, not +just within one, and the build checks that per phase and direction. + +**The error status** (decided 2026-11-14). A correlated reply's status +carries an error's ordinal, so its width sets `errors_per_schema`. It's part +of the `correlation` parameter, because it only exists when replies do, and it +defaults to `u16`, as in the other framings. Like every parameter it's a +framing setting, not a format: it changes the header, not how values are +encoded, and it's in the parameter hash, so peers that differ are refused. + +**Build errors** name the parameter that set the limit: + +```text +protocol `game::Play` has 300 packets in phase `Play`, serverbound, and the +instance "game" (`comline.packet@1`, `id = "u8"`) addresses at most 256 + +`@id = 0x1F4` on `game::Play.chat`: the instance "game" has `id = "u8"`, +whose largest id is 0xFF + +`login` and `chat` share id 0x03 in phase `Play`, serverbound: the instance +"game" has `family = "none"`, so ids are unique across every protocol in a +phase and direction +``` + +A fixed-width header and a varint one have the trade-offs set out for +[`datagram@4`](../dependency-protocols/datagram@4.md#why-fixed-widths). A varint +length or id is offered because legacy systems use them (Minecraft's lengths +and ids are varints). A system designed for Comline is better served by fixed +widths. + +#### Packages on a packet connection + +A packet framing's ids are fixed in the schema, and a legacy system fixes +them, so nothing is negotiated +([Connections](../connections.md#several-protocols-on-one-connection)). The +ids are unique in a package, and nothing binds numbers per connection +(`protocols` is `by_id` or `in_call`, never `by_binding`). + +A dependency's packet protocol, such as a shared keepalive family from a +library, can't share a connection by binding numbers. What remains is a +static check: its ids are explicit and fixed, so the build can prove they +don't collide with the package's own, per phase and direction. + +**Decided 2026-11-14: allowed, when the build proves them disjoint.** A +dependency's packet protocol joins when every one of its ids is explicit and +none collides with the package's, or with another joining dependency's, in any +phase and direction. A later release of the dependency that adds a colliding +id fails the consumer's build, never the wire. The build error names both +packets: + +```text +`keepalive::Ping.ping` (from `netlib`) and `game::Play.chat` share id 0x03 in +phase `Play`, serverbound: a dependency's packets join a `comline.packet@1` +connection only with ids disjoint from the package's +``` + +A dependency whose protocols can't be proven disjoint, because one lacks an +explicit id, or one uses another instance, gets a +[connection of its own](../dependency-protocols/connection-per-package.md). + ## Open questions - **Replies in legacy protocols:** payload echo (`@reply` / `@match`), `in_order` and `sequence` correlation, and errors as declared packets (see [Describing legacy protocols](#describing-legacy-protocols)), deferred until a real legacy schema needs them. -- **The framing's parameters:** leaning towards named framing instances in - `config.idp` (see - [Where the framing's parameters are written](#where-the-framings-parameters-are-written)). - Instance names and sharing across packages are decided. +- ~~**The framing's parameters**~~: decided, named framing instances in + `config.idp` ([Where the framing's parameters are written](#where-the-framings-parameters-are-written)). +- ~~**The error status width**~~: decided, part of `correlation`, `u16` by + default ([What each parameter sets](#what-each-parameter-sets)). +- ~~**A dependency's packet protocols on the package's connection**~~: + decided, allowed when their ids are proven disjoint + ([Packages on a packet connection](#packages-on-a-packet-connection)). ## Prerequisites - **Server-to-client sending:** what `@provider = Client` needs, and nothing - generates today. Designed in [Connections](connections.md#calls-in-both-directions). + generates today. Designed in [Connections](../connections.md#calls-in-both-directions). - **Several protocols on one connection.** Designed in - [Connections](connections.md#several-protocols-on-one-connection). + [Connections](../connections.md#several-protocols-on-one-connection). - **The handshake as a declared axis** ([runtime#18](https://github.com/ComlineProject/runtime/issues/18)), **`@unverified`** ([core#121](https://github.com/ComlineProject/core/issues/121)), diff --git a/docs/docs/proposals/phase-syntax/index.md b/docs/docs/proposals/phase-syntax/index.md index 6a14b68..0591602 100644 --- a/docs/docs/proposals/phase-syntax/index.md +++ b/docs/docs/proposals/phase-syntax/index.md @@ -6,7 +6,7 @@ > key shape (`protocol.provider`). Six follow-up points are settled on > [its page](enum-and-annotations.md#as-accepted). -**The question.** [Packet dispatch](../packet-dispatch.md#phases-live-in-the-schema) +**The question.** [Packet dispatch](../packet-dispatch/index.md#phases-live-in-the-schema) puts phases in the schema: which protocols are active in each phase, and which packets move a connection between phases. **How are phases written?** diff --git a/docs/docs/proposals/project-dashboard.md b/docs/docs/proposals/project-dashboard.md index 02620f3..bef3fa8 100644 --- a/docs/docs/proposals/project-dashboard.md +++ b/docs/docs/proposals/project-dashboard.md @@ -194,7 +194,7 @@ stack the tutorial already uses (Svelte 5, Kit 2, Vite 7). |---|---|---| | Charts | **uPlot** (MIT, about 50 KB) | It draws on canvas, so hundreds of series over thousands of versions stay smooth. SVG charting slows down at that size | | Tables | Svelte tables, sortable, with virtualized rows | A table of every benchmark between two versions keeps only its visible rows in the DOM | -| Assets | self-contained, no CDN | the same rule as the [HTML doc renderer](auto-documentation.md) | +| Assets | self-contained, no CDN | the same rule as the [HTML doc renderer](auto-documentation/index.md) | | Theme | light and dark, following the OS | the same as this site and the HTML doc renderer | ### Versions and comparison diff --git a/docs/docs/proposals/return-forms.md b/docs/docs/proposals/return-forms.md index f361372..5b34d2d 100644 --- a/docs/docs/proposals/return-forms.md +++ b/docs/docs/proposals/return-forms.md @@ -1,7 +1,7 @@ # Return forms Status: **accepted** 2026-11-02, not implemented · Quiet calls are a future -form (below) · Relates to [packet dispatch](packet-dispatch.md#replies-both-one-way-and-correlated), +form (below) · Relates to [packet dispatch](packet-dispatch/index.md#replies-both-one-way-and-correlated), [Transports](transports.md) and [bounds on types](type-bounds/max-only.md) **The question:** a function can return a value, return `()`, or return @@ -85,5 +85,5 @@ Suggested diagnostic codes: `validation::one-way-cannot-raise`, learns earlier quiet calls succeeded (memcached sends a no-op and waits for its reply). - **Packet framings:** whether a named reply packet (`@reply`) is how `-> T` - is written there, designed with [packet dispatch](packet-dispatch.md). + is written there, designed with [packet dispatch](packet-dispatch/index.md). - **The facts' names and values** in the framing profiles. diff --git a/docs/docs/proposals/systems-software.md b/docs/docs/proposals/systems-software.md index 1c4e564..1893992 100644 --- a/docs/docs/proposals/systems-software.md +++ b/docs/docs/proposals/systems-software.md @@ -188,7 +188,7 @@ the schema. It's needed for 4, and useful for 2. Bit fields, byte order per field and fixed offsets are a different kind of format from what Comline has. When 3 is taken up, it likely starts from -[packet dispatch](packet-dispatch.md)'s legacy framings. +[packet dispatch](packet-dispatch/index.md)'s legacy framings. ## Prior art diff --git a/docs/docs/proposals/tagged-unions/index.md b/docs/docs/proposals/tagged-unions/index.md index e9d5c22..ee22ca6 100644 --- a/docs/docs/proposals/tagged-unions/index.md +++ b/docs/docs/proposals/tagged-unions/index.md @@ -6,7 +6,7 @@ **The question.** A value that is *one of several shapes*, with a tag saying which, is everywhere: sub-packets picked by a sub-type byte -([packet dispatch](../packet-dispatch.md#sub-packets-are-tagged-unions)), an +([packet dispatch](../packet-dispatch/index.md#sub-packets-are-tagged-unions)), an entity action that is attack, interact or interact-at, a result that is a success or a failure. **Is that a feature of `enum` or of `union`?** diff --git a/docs/docs/proposals/transports.md b/docs/docs/proposals/transports.md index 9531003..506f7f4 100644 --- a/docs/docs/proposals/transports.md +++ b/docs/docs/proposals/transports.md @@ -77,7 +77,7 @@ each framing's specification: | `datagram@1` (lockstep) | reliable delivery; one call at a time, so order follows | | `datagram@2` | reliable delivery; **ordered delivery for a host that waits in order:** such a host treats "a reply that skips the innermost waiting call" as a framing error, so a reordering transport breaks it. An async host, matching replies by id, doesn't need order | | `jsonrpc@1` | message boundaries, or a length prefix; reliable delivery | -| Packet framings ([packet dispatch](packet-dispatch.md)) | stated per framing; a packet framing may provide its own reliability | +| Packet framings ([packet dispatch](packet-dispatch/index.md)) | stated per framing; a packet framing may provide its own reliability | ### Who declares, and why both diff --git a/docs/docs/proposals/type-bounds/max-only.md b/docs/docs/proposals/type-bounds/max-only.md index 2a8cf6d..a1e80e6 100644 --- a/docs/docs/proposals/type-bounds/max-only.md +++ b/docs/docs/proposals/type-bounds/max-only.md @@ -198,7 +198,7 @@ Raised 2026-11-02 while weighing D. **Decided** marks what the user settled; | 7 | `StringBounds` brings a hidden maximum: `max_chars` defaults to 1024, so `StringBounds(min_chars = 1)` on `char[max 2000]` refuses 1025 to 2000 characters after decoding | `StringBounds` stops carrying a maximum: `max_chars` defaults to `u32::MAX`, then is deprecated, since the type always has one (issue 6). std gains `NonEmpty` and `MinChars`. The build errors when a validator's minimum exceeds the type's maximum, and warns when a validator caps lower than the type | **decided** | | 8 | Validator minimums are enforced only when the receiver opts in, and not at all in TypeScript; and whether senders check | **Validators are enforced everywhere by default.** `validate_on_decode` defaults to checking; turning it off is an explicit give-up with a reason, recorded and visible like `@unverified`. This breaks Rust consumers relying on today's `off` default. TypeScript's support claim stays `partial` until it generates validators ([core#96](https://github.com/ComlineProject/core/issues/96), step 4d). **Senders check too:** an encoder never produces bytes a conforming decoder refuses, and conformance vectors test that encoders refuse out-of-bound values. The check is on the value before encoding, the same in every format; what depends on the format and framing is in [Senders, by format and framing](#senders-by-format-and-framing) | **decided**, senders included | | 9 | What a string's bound counts, and byte budgets | Strings are `char[]`, counted in characters: [Strings as `char[]`](char-strings.md). Byte budgets belong to the framing | **decided** | -| 10 | Adopting a maximum is breaking: adding one to an unbounded field tightens it, so a major version | Accepted before 1.0. Later, version comparison may weigh direction (via `@provider` from [packet dispatch](../packet-dispatch.md)), so a limit on a value only the server receives breaks nobody | **decided** | +| 10 | Adopting a maximum is breaking: adding one to an unbounded field tightens it, so a major version | Accepted before 1.0. Later, version comparison may weigh direction (via `@provider` from [packet dispatch](../packet-dispatch/index.md)), so a limit on a value only the server receives breaks nobody | **decided** | | 11 | A generator using fixed-capacity types (`ArrayString<32>`) changes its consumers' code when a maximum changes, beyond the wire | Fixed-capacity types are opt-in in a generator's configuration; the default is the language's normal heap type with a check | **decided** | | 12 | Every language pays for checking, and writes the check itself | Following the [shared runtime core](../shared-runtime-core/index.md): the core owns what is bytes and integers, not host values. **Decoding** works on bytes, so "strict UTF-8 and count characters up to N" can be a core primitive; it moves into a language's path only where instruction counts show the call is at least as fast, as in [stage 4](../shared-runtime-core/index.md#stage-4-moving-typescript). **Encoding** starts from a host string, which the core doesn't own. The rule itself is written once: [stage 3](../shared-runtime-core/generated-codecs.md)'s layout plan carries each maximum, and every generator renders the check from it, held together by the vectors and differential fuzzing. The runtime cost per language is what the [validator benchmarks](../validator-benchmarks.md) would measure | **decided** | | 13 | Transports have limits of their own (a UDP datagram holds about 65 KB), which no configuration states today | Optional transport declarations, by the schema's author and by the consumer, adding a "transport fit" guarantee only where declared: [Transports](../transports.md) | open: a separate draft, not blocking D | @@ -282,7 +282,7 @@ every format. What changes: | `datagram@1`, `datagram@2`, `jsonrpc@1` | nothing for the check: the body is encoded whole, then framed | | A stream transport (a `u32` length prefix) | the worst-case size against `max_frame` (issue 4) | | Datagrams over UDP | a hard ceiling of about 65 KB per datagram (issue 13) | -| Packet framings (`comline.packet@1`, legacy ones, from [packet dispatch](../packet-dispatch.md)) | their maximum packet size, a framing parameter, checked like a frame limit; with compression, against the uncompressed size | +| Packet framings (`comline.packet@1`, legacy ones, from [packet dispatch](../packet-dispatch/index.md)) | their maximum packet size, a framing parameter, checked like a frame limit; with compression, against the uncompressed size | | Streams (an [open design](https://github.com/ComlineProject/docs/issues/98)) | a stream isn't one value, so bounds don't apply; "at most N items per stream" would be the stream's own limit, decided in that design | How a refused value is reported depends on the framing: diff --git a/docs/docs/proposals/type-bounds/model.md b/docs/docs/proposals/type-bounds/model.md index db098ea..f224e00 100644 --- a/docs/docs/proposals/type-bounds/model.md +++ b/docs/docs/proposals/type-bounds/model.md @@ -212,7 +212,7 @@ A warning when a validator only repeats a bound A finer rule could weigh the direction a value travels: loosening a parameter only the server receives breaks nobody. That's open, and would ride on -`@provider` from [packet dispatch](../packet-dispatch.md). +`@provider` from [packet dispatch](../packet-dispatch/index.md). ### Generated code diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 4bf409a..746ada9 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -185,18 +185,29 @@ nav: - 'Stage 1 — verification': 'proposals/shared-runtime-core/verification.md' - 'Stage 2 — core interface': 'proposals/shared-runtime-core/core-interface.md' - 'Stage 3 — generated codecs': 'proposals/shared-runtime-core/generated-codecs.md' - - Automatic documentation: 'proposals/auto-documentation.md' + - 'Automatic documentation': + - 'proposals/auto-documentation/index.md' + - 'Links, overview, highlighting': 'proposals/auto-documentation/links-overview-highlighting.md' - Project dashboard: 'proposals/project-dashboard.md' - Schema compatibility: 'proposals/schema-compatibility.md' - Language target versions: 'proposals/target-versions.md' - Extending the call system: 'proposals/extensibility.md' - Connections: 'proposals/connections.md' + - Dependency protocols: + - 'proposals/dependency-protocols/index.md' + - 'Part 1 — capabilities': 'proposals/dependency-protocols/capabilities.md' + - 'Proposal A': 'proposals/dependency-protocols/multi-package-connection.md' + - 'datagram@4': 'proposals/dependency-protocols/datagram@4.md' + - 'Proposal B': 'proposals/dependency-protocols/consumer-numbers.md' + - 'Proposal C': 'proposals/dependency-protocols/connection-per-package.md' - Package wire defaults: 'proposals/package-wire-defaults.md' - Return forms: 'proposals/return-forms.md' - Transports: 'proposals/transports.md' - IPC: 'proposals/ipc.md' - Systems software: 'proposals/systems-software.md' - - Packet dispatch: 'proposals/packet-dispatch.md' + - 'Packet dispatch': + - 'proposals/packet-dispatch/index.md' + - 'comline.packet@1': 'proposals/packet-dispatch/comline.packet@1.md' - 'Function ids': - 'proposals/function-ids/index.md' - 'Proposal A': 'proposals/function-ids/annotation.md'