diff --git a/.github/workflows/std-drift.yml b/.github/workflows/std-drift.yml index 2da8001..28d1d82 100644 --- a/.github/workflows/std-drift.yml +++ b/.github/workflows/std-drift.yml @@ -1,6 +1,6 @@ name: Std drift -# The pages under `docs/docs/reference/schema/std/` are not written by hand: they are `comline +# The pages under `docs/docs/reference/package/schema/std/` are not written by hand: they are `comline # doc` run over the standard library package core embeds # (`core_stdlib/packages/std`), and the std block of `docs/mkdocs.yml` is its # generated nav. This regenerates both and fails on a difference, so a @@ -19,7 +19,7 @@ on: - cron: '15 7 * * 1' pull_request: paths: - - 'docs/docs/reference/schema/std/**' + - 'docs/docs/reference/package/schema/std/**' - 'docs/mkdocs.yml' - 'scripts/sync-std-docs.py' - '.github/workflows/std-drift.yml' diff --git a/docs/docs/conduct/language-implementation/index.md b/docs/docs/conduct/language-implementation/index.md index 2580ba4..cd3557a 100644 --- a/docs/docs/conduct/language-implementation/index.md +++ b/docs/docs/conduct/language-implementation/index.md @@ -21,4 +21,4 @@ you want the detail; this page just needed to say it exists. ## How is a language mantained -Test \ No newline at end of file +Test diff --git a/docs/docs/conduct/readiness.md b/docs/docs/conduct/readiness.md index 8e4baba..88055a0 100644 --- a/docs/docs/conduct/readiness.md +++ b/docs/docs/conduct/readiness.md @@ -69,7 +69,7 @@ Semver for a user's schema package means nothing until a break is defined. - The C ABI's structs, status codes and flags are final for 1.x, including `comline_connection_size`/`init` and whether a `comline@2` changes the event - struct ([C ABI](../reference/c-abi.md)). + struct ([C ABI](../reference/languages/c-abi.md)). - The IR and `config.idp` formats are stable, or versioned with a reader for every older version ([core#146](https://github.com/ComlineProject/core/issues/146)). @@ -84,7 +84,7 @@ Semver for a user's schema package means nothing until a break is defined. corpus and ops protocol; - the documentation model behind [`comline doc`](../reference/tools/cli/doc.md) (version 1 today); - - the [`comline.toml` schema](../reference/package/congregation/comline-toml.md) + - the [`comline.toml` schema](../reference/package/comline-toml.md) (`0.x` today); - the JSON from [`comline inspect --json`](../reference/tools/cli/inspect.md). - The extension API is part of 1.0, so it is designed, built and frozen diff --git a/docs/docs/design/consumer-generation-config.md b/docs/docs/design/consumer-generation-config.md index 9c77bb0..92082a3 100644 --- a/docs/docs/design/consumer-generation-config.md +++ b/docs/docs/design/consumer-generation-config.md @@ -3,7 +3,7 @@ Status: **core feature shipped** · `ComlineProject/core` + `ComlineProject/cli` · Relates to core#6, core#8, cli#1 | Piece | State | -|---|---| +| --- | --- | | Drop frozen `generation_path` + core-side codegen | ✅ merged (core#25) | | Drop frozen `package_versions` from the congregation | ✅ merged (core#26) | | `comline.toml` `[generate]`, `--out`/`--layout`/`--mode`, `generate` no longer freezes, `clean`/`new`/`watch` | ✅ merged (cli#10) | @@ -39,7 +39,7 @@ local path choice perturb the package's identity. ## Two files, two owners | File | Owner | Format | Frozen into CAS | Purpose | -|---|---|---|---|---| +| --- | --- | --- | --- | --- | | `config.idp` (congregation) | package **author** | `.idp` | yes (interpreted → `Vec`) | package identity + declarations: name, `specification_version`, `dependencies`, `publish_registries`, and the codegen **capability declaration** `code_generation.languages` = "this package *can* be generated as `rust#1.70`, `python#3.11`" | | `comline.toml` | package **consumer** | TOML | **no** — never enters freezing or CAS | *what this checkout generates, in what mode, and where* | @@ -205,7 +205,7 @@ schemas. What is *not* in a commit: or reordered mid-list. So the CAS store covers only **half** of a state-freezing story: version history -+ diff-driven auto-versioning, i.e. the "track past/present state" half. It does +- diff-driven auto-versioning, i.e. the "track past/present state" half. It does **not** yet cover the "make a rebuild reproducible" half — freezing configuration and pinning imported / dependency schemas. For that, a version's tree needs (at least) the frozen config committed back alongside the schema entries, and — once @@ -228,7 +228,7 @@ units that affect that API feed the bump. The rest are *recorded* (for reproducibility) but *not diffed*. | Frozen config unit | Change | Version bump | -|---|---|---| +| --- | --- | --- | | `Dependency` | **removed**, or **version changed** | **major** — conservative: without dependency resolution we cannot tell a compatible dep bump from a breaking one, and a dep's schema shapes are part of this package's effective surface. Refine later (dep-major → major, dep-minor → minor) once core#6 can diff dep schemas. | | `Dependency` | **added** | **minor** — additive; the schema content that uses the new dep already carries its own minor/major. | | `Dependency` | `uri` / `hash` only, same version | **patch** — provenance, not API. | diff --git a/docs/docs/design/validators.md b/docs/docs/design/validators.md index 989901e..9267647 100644 --- a/docs/docs/design/validators.md +++ b/docs/docs/design/validators.md @@ -257,7 +257,7 @@ left to right, parenthesising each comparison. Whether decode runs the checks is the **consumer's** call, not the schema's — it changes the shape and cost of generated decode (a clone per decode, no "parse now, validate later"), which is a codegen concern. So it's a `[generate]` key in -the consumer's [`comline.toml`](../reference/package/congregation/comline-toml.md), overridable per +the consumer's [`comline.toml`](../reference/package/comline-toml.md), overridable per `[[generate.target]]`, slotting into the existing `defaults → [generate] → [[generate.target]] → env → CLI` precedence: @@ -279,7 +279,7 @@ validate_on_decode = "collect" in its `TryFrom`); `"off"` leaves decode untouched. - Matching `COMLINE_GENERATE_VALIDATE_ON_DECODE` env var and `--validate-on-decode` flag, like the other `[generate]` keys. -- Folds into [`comline.toml`](../reference/package/congregation/comline-toml.md) when 4c ships. +- Folds into [`comline.toml`](../reference/package/comline-toml.md) when 4c ships. ### Decided diff --git a/docs/docs/developer/extending/encodings.md b/docs/docs/developer/extending/encodings.md index b5a96d9..0a78d32 100644 --- a/docs/docs/developer/extending/encodings.md +++ b/docs/docs/developer/extending/encodings.md @@ -94,9 +94,9 @@ still a [proposal](../../proposals/schema-compatibility.md#encoding-profiles). For the official profiles, see the [runtime's `profiles/`](https://github.com/ComlineProject/runtime/tree/main/runtime/conformance/profiles). -## 3. Write vectors, and prove an implementation +## 3. Writing vectors -**Vectors** are byte-exact cases. They sit beside the profile: the profile's +**Vectors** are byte-exact cases that prove an implementation. They sit beside the profile: the profile's path with `.json` replaced by `.vectors.json`, so `profiles/lines.json` has `profiles/lines.vectors.json`. diff --git a/docs/docs/getting-started.md b/docs/docs/getting-started.md index d35938a..2d70014 100755 --- a/docs/docs/getting-started.md +++ b/docs/docs/getting-started.md @@ -87,7 +87,7 @@ comline generation Targets come from `code_generation.languages` in `config.idp` (the scaffold declares `rust#1.85.0`, the Rust generator's floor); output lands under `generated/` by default. Change where -and how in [`comline.toml`](reference/package/congregation/comline-toml.md): +and how in [`comline.toml`](reference/package/comline-toml.md): ```toml title="comline.toml" [generate] diff --git a/docs/docs/guides/codegen/generating-code.md b/docs/docs/guides/codegen/generating-code.md index 8273807..4b7e900 100644 --- a/docs/docs/guides/codegen/generating-code.md +++ b/docs/docs/guides/codegen/generating-code.md @@ -267,7 +267,7 @@ using cross-schema references in `code` mode needs to declare matching `mod` statements itself. `std` is the exception, because -[`external_std`](../../reference/package/congregation/comline-toml.md#external_std) +[`external_std`](../../reference/package/comline-toml.md#external_std) is on by default in `lib` mode: a std type then resolves to `comline_std::http::Request` and the crate depends on `comline_std` rather than carrying a copy of it. @@ -293,7 +293,7 @@ An alias imports the name the other module actually exports nothing refers to produces no import at all. `std` again resolves to its own package when -[`external_std`](../../reference/package/congregation/comline-toml.md#external_std) +[`external_std`](../../reference/package/comline-toml.md#external_std) is on, by subpath import — the leading `std` segment drops, since `@comline/std` is its own package root: diff --git a/docs/docs/guides/codegen/index.md b/docs/docs/guides/codegen/index.md index 04c1a52..a3c058d 100755 --- a/docs/docs/guides/codegen/index.md +++ b/docs/docs/guides/codegen/index.md @@ -20,7 +20,7 @@ detail that the generated code is pleasant to use directly rather than only as a ## Using it `comline generation` is the command; [Generating code](generating-code.md) is the -full guide, and [`comline.toml`](../../reference/package/congregation/comline-toml.md) is where a +full guide, and [`comline.toml`](../../reference/package/comline-toml.md) is where a consumer sets the output location, layout and mode. Today there are `rust` and `typescript` generators, and both do `code` and diff --git a/docs/docs/guides/codegen/library-generation.md b/docs/docs/guides/codegen/library-generation.md index 442f58c..4524b92 100755 --- a/docs/docs/guides/codegen/library-generation.md +++ b/docs/docs/guides/codegen/library-generation.md @@ -7,7 +7,7 @@ the module layout. A planned third mode, `dylib`, would compile it to a library any language's [runtime](../runtime/index.md) can load. `comline generation`'s `mode` chooses how far generation goes. It is set per target -in [`comline.toml`](../../reference/package/congregation/comline-toml.md) (`[generate] mode` or +in [`comline.toml`](../../reference/package/comline-toml.md) (`[generate] mode` or `[[generate.target]] mode`). | `mode` | Output | Status | @@ -71,7 +71,7 @@ import { Greeting } from "my-api/types"; Comline's standard library is implicitly part of every package, so without help a `lib` build would vendor `std/http` and `std/validators` into every generated package. It doesn't: -[`external_std`](../../reference/package/congregation/comline-toml.md#external_std) +[`external_std`](../../reference/package/comline-toml.md#external_std) defaults to **on** in `lib` mode, so std's schemas are left out and the manifest depends on std's own pre-generated package instead — `comline_std` for rust, `@comline/std` for typescript. Your own nested @@ -89,7 +89,7 @@ with a message naming the pair. In either language, `/` keeps this target's package from colliding with another `lib` target sharing the same `out`; a target with `out` to itself can set -[`flatten = true`](../../reference/package/congregation/comline-toml.md#flatten) to skip that and +[`flatten = true`](../../reference/package/comline-toml.md#flatten) to skip that and land the package straight into `out`. `layout` plays no part in any of this — it only applies to `code` mode. Multi-version `lib` output is not done yet. diff --git a/docs/docs/guides/documenting.md b/docs/docs/guides/documenting.md index 4d482e1..1b048c1 100644 --- a/docs/docs/guides/documenting.md +++ b/docs/docs/guides/documenting.md @@ -144,7 +144,7 @@ the declarations your own use, and what those use in turn, never the whole dependency. Each dependency gets its own pages (`deps//`), and a type from it links there. -`std` links to its hosted docs instead, `https://comlineproject.github.io/docs/reference/schema/std/`. +`std` links to its hosted docs instead, `https://comlineproject.github.io/docs/reference/package/schema/std/`. To link another dependency to its published docs rather than render pages for it, give it a base URL in `comline.toml`: @@ -211,7 +211,7 @@ and every path it returns is checked before anything is written. ## Settings -Every flag has a `[doc]` key in [`comline.toml`](../reference/package/congregation/comline-toml.md#doc), +Every flag has a `[doc]` key in [`comline.toml`](../reference/package/comline-toml.md#doc), and the flag wins: | Key | Flag | Default | @@ -226,8 +226,8 @@ and the flag wins: ## The standard library, rendered When rendering documentation of a package that uses any standard library types, -the built documentation will link to [`std` pages](../reference/schema/std/index.md), meaning +the built documentation will link to [`std` pages](../reference/package/schema/std/index.md), meaning that the links will always lead to online A type from `std` in your own docs links to them by default, so `std::http::Request` in a field's type links -to `https://comlineproject.github.io/docs/reference/schema/std/http/#request`. +to `https://comlineproject.github.io/docs/reference/package/schema/std/http/#request`. diff --git a/docs/docs/guides/idl/annotations.md b/docs/docs/guides/idl/annotations.md index cfc8dc0..c417a8e 100644 --- a/docs/docs/guides/idl/annotations.md +++ b/docs/docs/guides/idl/annotations.md @@ -38,7 +38,7 @@ build. | `@deprecated` | a struct, a `type` alias | warns wherever it's used; the value is `true` or a message | [Types](types.md#type-aliases) | | `@settings` | any declaration | applies a named settings preset | [Settings](settings.md#named-presets-settings) | | `@framing` | a protocol | how calls are framed on the wire | [Protocols](protocol.md#annotations) | -| `@handshake` | a protocol | what runs before the first call, with [its parameters](../../reference/wire/handshakes/index.md#parameters) if it takes any | [Protocols](protocol.md#annotations) | +| `@handshake` | a protocol | what runs before the first call, with [its parameters](../../reference/wire/handshakes/comline@2.md#parameters) if it takes any | [Protocols](protocol.md#annotations) | | `@provider` | a protocol | which side implements it: `Server`, `Client` or `Multiple` | [Calls in both directions](protocol.md#calls-in-both-directions) | | `@consumer` | a protocol | the other side; follows from `@provider` | [Protocols](protocol.md#annotations) | | `@unverified` | a protocol | gives up a guarantee, with the reason | [Giving up a guarantee](protocol.md#giving-up-a-guarantee) | diff --git a/docs/docs/guides/idl/index.md b/docs/docs/guides/idl/index.md index e1aad2a..39f62da 100755 --- a/docs/docs/guides/idl/index.md +++ b/docs/docs/guides/idl/index.md @@ -1,6 +1,6 @@ # IDL / Schema -The **Interface Description Language** (IDL, also called the *schema*) is what you +The **Interface Description Language** (IDL, also called __Schema__) is what you write by hand. A schema file has the extension `.ids` and lives under a project's `src/` directory. It defines the data shapes and the protocols that carry them, independent of any language. diff --git a/docs/docs/guides/idl/protocol.md b/docs/docs/guides/idl/protocol.md index 77d55a3..f3138df 100755 --- a/docs/docs/guides/idl/protocol.md +++ b/docs/docs/guides/idl/protocol.md @@ -118,7 +118,7 @@ protocol Mail { need to write it, but if you do, it has to agree. - **`@handshake`** picks what runs on a connection before the first call: `"comline@2"` (the default) checks they're compatible, and takes - [parameters](../../reference/wire/handshakes/index.md#parameters), as in + [parameters](../../reference/wire/handshakes/comline@2.md#parameters), as in `{ name = "comline@2", scope = package }`; `"comline@1"` checks that both ends share the schema, format and framing; `"none"` sends and checks nothing. Without it, the protocol takes the package's `handshake` from `config.idp`, diff --git a/docs/docs/guides/idl/types.md b/docs/docs/guides/idl/types.md index 7db0e8d..aca24fe 100644 --- a/docs/docs/guides/idl/types.md +++ b/docs/docs/guides/idl/types.md @@ -8,7 +8,7 @@ Primitives: | unsigned integers | `u8` `u16` `u32` `u64` `u128` | | floats | `f32` `f64` | | boolean | `bool` | -| text | `string`, `char` (see below) | +| text | `string`, `char` (see the [reference](../../reference/package/schema/types.md#text-string-and-char)) | Plus any `struct` / `enum` name (scoped: `pkg::module::Type`), arrays (`Type[]`, `Type[max 100]` or fixed `Type[10]`), and unions (`union(TypeA TypeB)`). @@ -18,20 +18,6 @@ type or given by a [setting](settings.md#default-maximums). The examples in this carry maxima so they build as written; a bare `string` or `T[]` in one of your schemas builds only once a default is set. -## Text: `string` and `char` - -- **`char`** is one Unicode scalar value: a single character, not a byte or a - UTF-16 unit. Its literal is a string of exactly one character, `"a"`; any - other length is a build error. -- **`string`** is the text type: UTF-8, the built-in name for `char[]`. An - array of `char` is always text, so `char[max 64]` is a string of at most 64 - characters, not a list of 64 `char` values. -- **`str`** was an accepted spelling until it was removed. Writing it is now a - build error that tells you to use `string`. -- **`String`** (capitalised) is **not** Comline syntax. It is the Rust type that - `string` generates into, so you see it in generated Rust, never in a `.ids` - file. Writing it gets the same hint. - ## Bounds A **maximum** says how big a value may be, in its type, so a decoder can refuse @@ -50,17 +36,22 @@ struct Customer { - **`T[N]`** is exactly `N`. **`T[max N]`** is at most `N`. `T[]` writes no maximum, and needs one from a setting. + - **Only a maximum.** A minimum ("not empty", "at least one") is a [validator](validator.md#the-standard-validators): `NonEmpty`, `MinChars`, `MinItems`. So is a range of values, `IntRange`. + - **`N` is an integer or a [constant](const.md#sizes-from-constants):** `char[max NAME_MAX]`. It is at least 1 and at most `u32::MAX` (4294967295); `max 0`, an exact length of 0 and a negative size are build errors. + - **`max` is a keyword only right after `[`.** Anywhere else it is an ordinary name, so a field called `max` is fine. + - **Only strings and dynamic arrays take one.** Integers, floats, `bool`, enums, structs, unions and `()` don't. `string[max 10]` is ten strings, each at the package default; for a bounded string write `char[max N]`. + - **Any change to a maximum is breaking**: tightening breaks old senders and loosening breaks old receivers. diff --git a/docs/docs/guides/idl/use.md b/docs/docs/guides/idl/use.md index a745a35..fabd488 100644 --- a/docs/docs/guides/idl/use.md +++ b/docs/docs/guides/idl/use.md @@ -102,9 +102,9 @@ struct Call { `std` and each of its modules carry [module docs](docstrings.md#module-docs): hover `std` or `validators` in the editor to read them, with what's in each. -The full reference is on this site, [rendered from those docs](../../reference/schema/std/index.md) -by [`comline doc`](../documenting.md): [`std::http`](../../reference/schema/std/http.md) and -[`std::validators`](../../reference/schema/std/validators.md), each declaration with its fields +The full reference is on this site, [rendered from those docs](../../reference/package/schema/std/index.md) +by [`comline doc`](../documenting.md): [`std::http`](../../reference/package/schema/std/http.md) and +[`std::validators`](../../reference/package/schema/std/validators.md), each declaration with its fields and variants. A build includes only the std schemas a package imports. Like a dependency's, diff --git a/docs/docs/guides/idl/validator.md b/docs/docs/guides/idl/validator.md index 6f1bc14..a8133be 100644 --- a/docs/docs/guides/idl/validator.md +++ b/docs/docs/guides/idl/validator.md @@ -125,7 +125,7 @@ function there's no single value for it to check, so it's a build error. ## The standard validators -[`std::validators`](../../reference/schema/std/validators.md) ships these. A **maximum** belongs +[`std::validators`](../../reference/package/schema/std/validators.md) ships these. A **maximum** belongs in the type ([`char[max 32]`](types.md#bounds)), where a decoder checks it before allocating; validators check what a type can't say, a minimum or a range. @@ -225,7 +225,7 @@ in `comline.toml`; `"first"` stops at the first failure), and a provider refuses a call whose arguments fail them. Turning that off is a give-up with its reason, `validate_on_decode = { off = "why" }`; then nothing runs until you call `validate()`. See -[`validate_on_decode`](../../reference/package/congregation/comline-toml.md#validate_on_decode). +[`validate_on_decode`](../../reference/package/comline-toml.md#validate_on_decode). Properties are filled in when the code is generated: the argument you wrote, else the property's default, with constants resolved. Nothing about the diff --git a/docs/docs/guides/index.md b/docs/docs/guides/index.md index cbb2f80..57e3591 100644 --- a/docs/docs/guides/index.md +++ b/docs/docs/guides/index.md @@ -70,7 +70,6 @@ The [language guides](../languages/index.md) show it in Rust and TypeScript. - [Packages & dependencies](packages/index.md): the manifest, wire defaults, and importing other packages. + - [Documenting your schemas](documenting.md): pages and a site from your docstrings. -- [Extending Comline](../developer/extending/index.md): your own formats, framings, - handshakes, generators, transports and middleware. diff --git a/docs/docs/guides/ir/cas.md b/docs/docs/guides/ir/cas.md index e19b8af..9495263 100755 --- a/docs/docs/guides/ir/cas.md +++ b/docs/docs/guides/ir/cas.md @@ -347,9 +347,9 @@ hash, and each encoding's name and parameter hash (both blake3, as hex). See [Compatibility facts](../../reference/versioning.md#compatibility-facts) for what they mean and how they are compared. The `config.idp` keys are in the -[config.idp reference](../../reference/package/congregation/config-id.md), and where +[config.idp reference](../../reference/package/idp.md), and where generated code lands is set in -[`comline.toml`](../../reference/package/congregation/comline-toml.md). +[`comline.toml`](../../reference/package/comline-toml.md). ## Branches diff --git a/docs/docs/guides/packages/index.md b/docs/docs/guides/packages/index.md index 721dbc6..5712f5a 100644 --- a/docs/docs/guides/packages/index.md +++ b/docs/docs/guides/packages/index.md @@ -28,7 +28,7 @@ code_generation = { ``` Every key, its syntax and what it does is in the -[`config.idp` reference](../../reference/package/congregation/config-id.md). The ones +[`config.idp` reference](../../reference/package/idp.md). The ones with more to them have their own sections below: [wire defaults](#wire-defaults) and [dependencies](#dependencies). @@ -82,7 +82,7 @@ Where generated code lands, in what layout, for which versions — none of which belongs to the package author — is set by whoever runs `comline generation`, in a separate, non-frozen `comline.toml`. See [Generating code](../codegen/generating-code.md) and the -[`comline.toml` reference](../../reference/package/congregation/comline-toml.md). +[`comline.toml` reference](../../reference/package/comline-toml.md). ## Dependencies diff --git a/docs/docs/guides/runtime/index.md b/docs/docs/guides/runtime/index.md index be53df6..d4f3cb5 100755 --- a/docs/docs/guides/runtime/index.md +++ b/docs/docs/guides/runtime/index.md @@ -23,7 +23,7 @@ flowchart TB The protocol logic that has to be byte-identical everywhere (framing, request ids, the handshake's frames) is written once, in Rust: `comline-runtime` is the [shared runtime core](../../proposals/shared-runtime-core/index.md), and -its [C ABI](../../reference/c-abi.md) exposes it to other languages. The +its [C ABI](../../reference/languages/c-abi.md) exposes it to other languages. The TypeScript runtime loads it as WebAssembly for its built-in framings, and falls back to its own TypeScript implementation where the core can't load (a strict content security policy, say) or for a framing of your own; both paths behave diff --git a/docs/docs/languages/c/index.md b/docs/docs/languages/c/index.md new file mode 100644 index 0000000..e1acdcb --- /dev/null +++ b/docs/docs/languages/c/index.md @@ -0,0 +1,10 @@ +# C + +There is no C generator yet, so nothing turns a schema into C types and +codecs. What C has is the runtime's protocol core, `comline-runtime-ffi`, +through its [C ABI](../../reference/languages/c-abi.md): the handshakes, the +connection framings, envelopes, connection state and frame limits. Codecs, +transports and dispatch are the host's to write. + +It's the same core the TypeScript runtime loads through wasm, so a host on it +speaks the same bytes as the Rust and TypeScript runtimes. diff --git a/docs/docs/languages/index.md b/docs/docs/languages/index.md index 2ce8907..d5765ef 100644 --- a/docs/docs/languages/index.md +++ b/docs/docs/languages/index.md @@ -1,7 +1,7 @@ # Language Guides -The [IDL guide](../guides/idl/index.md) covers writing a schema; [Code -generation](../guides/codegen/index.md) covers turning it into source. This +The [IDL guide](../guides/idl/index.md) covered writing a schema; [Code +generation](../guides/codegen/index.md) covered turning it into source. This section is the third piece: actually *using* the generated code at run time — spawning a client, serving a protocol, picking framing and wire format, and (for Rust) fitting the runtime into a sync or async program. @@ -11,18 +11,21 @@ Each language gets its own tree. Pick one from the tabs on the left. ## Status | Language | Generator | Modes | Runtime | -|---|---|---|---| +| --- | --- | --- | --- | | [Rust](rust/index.md) | shipped | `code`, `lib` | `comline-runtime`, `no_std`-first; sync, or async with `async = true` | | [TypeScript](typescript/index.md) | shipped | `code`, `lib` | `@comline/runtime`, async/Promise-based | -| Python, Lua, Luau, C | none yet | — | — | +| [C](c/index.md) | none yet | — | `comline-runtime-ffi`, through the [C ABI](../reference/languages/c-abi.md) | +| Python, Lua, Luau | none yet | — | — | -Python, Lua, Luau and C appear in example `config.idp` manifests and design +Python, Lua and Luau appear in example `config.idp` manifests and design sketches, but no `comline-codegen-*` crate exists for them today — there is -nothing to generate and no runtime to link against. +nothing to generate and no runtime to link against. C has no generator yet +either, but it has a runtime to link against: the C ABI. ## What each runtime speaks -Both runtimes speak every built-in framing, `packed@1` and `json@1`, and are +Both runtimes speak the supported [built-in framings](../reference/wire/framings/index.md), +the supported [data formats](../reference/wire/formats/index.md) and are byte-compatible with each other. MessagePack is Rust only for now, and so is TCP: TypeScript ships an in-memory transport and you bring your own for anything else. The full matrix, proven against the conformance corpus, is in @@ -30,8 +33,8 @@ the [language reference](../reference/languages/index.md#runtimes-side-by-side). ## Before the per-language pages -Framing and wire format are two independent axes — which is laid out +Framing and wire format are two independent concerns, which are laid out language-agnostically in [Call systems & -serialization](../guides/runtime/call-system.md). Read that first if the -distinction isn't already clear; the per-language pages below are the -concrete API, not the concept. +serialization](../guides/runtime/call-system.md). Read about them first if the +distinction isn't clear enough, since the per-language pages in this section are +the concrete API. diff --git a/docs/docs/languages/rust/client-and-server.md b/docs/docs/languages/rust/client-and-server.md index 611ebed..073b621 100644 --- a/docs/docs/languages/rust/client-and-server.md +++ b/docs/docs/languages/rust/client-and-server.md @@ -187,14 +187,14 @@ Each generated error struct, and each error enum with variants, gets a the sentence you wrote in the schema is what `{}` and `.to_string()` produce, with `{self.field}` placeholders filled in: -```ids +```ids title="errors.ids" error NotFound { message = f"no entry for {self.key}" key: string } ``` -```rust +```rust title="user_impl.rs" NotFound { key: "k".into() }.to_string() // "no entry for k" ``` diff --git a/docs/docs/languages/rust/concurrency.md b/docs/docs/languages/rust/concurrency.md index ffb97d8..fd7b655 100644 --- a/docs/docs/languages/rust/concurrency.md +++ b/docs/docs/languages/rust/concurrency.md @@ -11,6 +11,20 @@ This page is about the lockstep `Client` and `Server` that a protocol on the inside, so everything below is a pattern for *your* code, not a feature of `comline-runtime`. +## Sync by default, async when asked + +The runtime is `no_std`-first and **sync** by default: a call blocks until +its reply arrives, and nothing inside the runtime spawns a thread or takes a +lock, so it works the same on a bare-metal target with no allocator as in a +normal `std` binary. + +Set [`async = true`](../../reference/package/comline-toml.md#async) in +`comline.toml` and connection protocols are generated for the runtime's +**async** driver instead: handlers are `async fn`s, the connection is a future +you spawn on any executor, and handles call the peer from any task +([Connections → Async](connections.md#async)). Protocols on the lockstep +`datagram` framing keep their blocking client and server either way. + ## Plain sync — the common case Most of what's shown elsewhere in this guide is already this: own a diff --git a/docs/docs/languages/rust/connections.md b/docs/docs/languages/rust/connections.md index 2d28c6c..14acfe9 100644 --- a/docs/docs/languages/rust/connections.md +++ b/docs/docs/languages/rust/connections.md @@ -2,11 +2,10 @@ Every protocol runs on a [connection](../../guides/idl/protocol.md#calls-in-both-directions) unless it -picks the lockstep `datagram` framing, in which case it gets a -[client and server](client-and-server.md) instead. This page uses this -schema: +picks the lockstep `datagram` framing, in which case it gets a [client and server](client-and-server.md) instead. + This page assumed the following schema: -```ids +```ids title="snippet.ids" protocol Chat { function send(text: string) -> u64; function count() -> u64; @@ -212,7 +211,7 @@ another task while the connection serves, generate for the async driver. ## Async -With `async = true` in [`comline.toml`](../../reference/package/congregation/comline-toml.md#async), +With `async = true` in [`comline.toml`](../../reference/package/comline-toml.md#async), connection protocols are generated for the runtime's `AsyncConnection` instead, on any executor: diff --git a/docs/docs/languages/rust/index.md b/docs/docs/languages/rust/index.md index 10806dc..36f2d0a 100644 --- a/docs/docs/languages/rust/index.md +++ b/docs/docs/languages/rust/index.md @@ -3,9 +3,9 @@ Status: shipped. `comline-codegen-rust` generates `code` mode (plain source files) and `lib` mode (a buildable crate) — see [Library generation](../../guides/codegen/library-generation.md) for what each mode -produces. Generated code depends on the **`comline-runtime`** crate -(MPL-2.0 — the piece that actually ships inside your program, as opposed -to the GPL-3.0 generator that only runs at build time). +produces. + +## Adding runtime to dependencies Not published to crates.io yet. Pin it by git rev, the same way the rest of the toolchain does (see `cli/Cargo.toml`'s own pattern if you want a @@ -16,19 +16,11 @@ worked example): comline-runtime = { git = "https://github.com/ComlineProject/runtime", rev = "…" } ``` -## Sync by default, async when asked +## Licensing -The runtime is `no_std`-first and **sync** by default: a call blocks until -its reply arrives, and nothing inside the runtime spawns a thread or takes a -lock, so it works the same on a bare-metal target with no allocator as in a -normal `std` binary. - -Set [`async = true`](../../reference/package/congregation/comline-toml.md#async) in -`comline.toml` and connection protocols are generated for the runtime's -**async** driver instead: handlers are `async fn`s, the connection is a future -you spawn on any executor, and handles call the peer from any task -([Connections → Async](connections.md#async)). Protocols on the lockstep -`datagram` framing keep their blocking client and server either way. +Generated code depends on the **`comline-runtime`** crate +(MPL-2.0 — the piece that actually ships inside your program, as opposed +to the GPL-3.0 generator that only runs at build time). ## Where to go next diff --git a/docs/docs/languages/typescript/index.md b/docs/docs/languages/typescript/index.md index 504929f..21211b8 100644 --- a/docs/docs/languages/typescript/index.md +++ b/docs/docs/languages/typescript/index.md @@ -22,7 +22,7 @@ it from async code is simply using it as it's already written, with ## On the shared core Part of the runtime is the Rust [shared runtime core](../../proposals/shared-runtime-core/index.md), -loaded through wasm (the [C ABI](../../reference/c-abi.md#wasm32-and-the-js-glue)), +loaded through wasm (the [C ABI](../../reference/languages/c-abi.md#wasm32-and-the-js-glue)), and part is still TypeScript. The core is vendored into the package, pinned to a `runtime` revision by `scripts/vendor-core.sh` ([ts#87](https://github.com/ComlineProject/comline-typescript/pull/87)). diff --git a/docs/docs/proposals/auto-documentation/index.md b/docs/docs/proposals/auto-documentation/index.md index b14c51b..3553c24 100755 --- a/docs/docs/proposals/auto-documentation/index.md +++ b/docs/docs/proposals/auto-documentation/index.md @@ -58,7 +58,7 @@ IR", and it follows from the It is a tool a project opts into, not a step in the pipeline. Nothing here happens unless asked for. The consumer's -[`comline.toml`](../../reference/package/congregation/comline-toml.md) owns the switches, +[`comline.toml`](../../reference/package/comline-toml.md) owns the switches, the same way it already owns generation's output location, layout and mode. ## Decisions @@ -116,7 +116,7 @@ These were open questions until 2026-11-01. The rationale is in It lands in the product documentation, not here: a Guide page on documenting your schemas, a `comline doc` entry in the [CLI reference](../../reference/tools/cli/index.md), its `comline.toml` keys in the -[congregation reference](../../reference/package/congregation/index.md), and the doc +[congregation reference](../../reference/package/index.md), and the doc model's JSON Schema. The obvious first demonstration is diff --git a/docs/docs/proposals/auto-documentation/links-overview-highlighting.md b/docs/docs/proposals/auto-documentation/links-overview-highlighting.md index 7a8cf4b..a478cd4 100644 --- a/docs/docs/proposals/auto-documentation/links-overview-highlighting.md +++ b/docs/docs/proposals/auto-documentation/links-overview-highlighting.md @@ -273,7 +273,7 @@ else `#!ids` would show as text. and the highlighting table above; the tokenizer's fixes. - **docs:** the types reference written out with an anchor per built-in; the guide's broken link; the [documenting guide](../../guides/documenting.md) - and the [`comline.toml` reference](../../reference/package/congregation/comline-toml.md) + and the [`comline.toml` reference](../../reference/package/comline-toml.md) for the new keys; `std` declaring its `documentation`; and **hosted docs versioned by toolchain release**, so that built-ins link to the reference of the toolchain that built a package. That's a site change of its own, worth diff --git a/docs/docs/proposals/dependency-protocols/index.md b/docs/docs/proposals/dependency-protocols/index.md index fcf4f88..043d6e7 100644 --- a/docs/docs/proposals/dependency-protocols/index.md +++ b/docs/docs/proposals/dependency-protocols/index.md @@ -19,7 +19,7 @@ check it against?** and every package numbers its protocols from 0. So `test::none::Thing` and `localdeps::greeting::Foo` can both be protocol 0. - **A frame names a protocol by that number alone.** A - [`datagram@3`](../../reference/wire/framings/index.md#datagram3) request carries a + [`datagram@3`](../../reference/wire/framings/datagram@3.md) request carries a protocol number and a function number, but no package. - **A handshake covers one package.** The [`comline@2`](../../reference/wire/handshakes/comline@2.md) Hello carries diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index d443eb0..c807376 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -74,7 +74,7 @@ reference. One proposal in stages, read in order: - [Stage 2 — core interface](shared-runtime-core/core-interface.md): the handshake, framings and connection state machine behind a stable C ABI and WASM, built from one crate. ***accepted***, steps 1–3 built: the core - changes, the C ABI at 0.1 ([reference](../reference/c-abi.md)), and its + changes, the C ABI at 0.1 ([reference](../reference/languages/c-abi.md)), and its vectors, fuzzing and benchmarks. The ABI stays `0.x` until the [1.0 announcement](../conduct/phases.md). - [Stage 3 — generated codecs](shared-runtime-core/generated-codecs.md): diff --git a/docs/docs/proposals/shared-runtime-core/core-interface.md b/docs/docs/proposals/shared-runtime-core/core-interface.md index 3f064b0..e639933 100644 --- a/docs/docs/proposals/shared-runtime-core/core-interface.md +++ b/docs/docs/proposals/shared-runtime-core/core-interface.md @@ -23,7 +23,7 @@ cost unavoidable, the page says so, and the Built 2026-10-25 and 2026-10-26, in [`runtime`](https://github.com/ComlineProject/runtime) except where noted. These are [rollout](#rollout) steps 1–3. The as-built API is the -[C ABI reference](../../reference/c-abi.md). +[C ABI reference](../../reference/languages/c-abi.md). **Step 1: the core changes** @@ -140,7 +140,7 @@ else on this page is as proposed. - An event sets `COMLINE_EVENT_ONE_WAY` only when the frame says so, never from a request id of 0. - **Which fields each framing reads** is stated per function and set out in - [the reference](../../reference/c-abi.md#what-each-framing-reads). + [the reference](../../reference/languages/c-abi.md#what-each-framing-reads). `datagram@2` reads `protocol` and `call_id`. `jsonrpc@1` reads `name`, writes an error's `error_id` as the JSON-RPC `code`, and writes a fault as `-32603`. - **JSON-RPC without `std` is confirmed.** `serde_json` with `raw_value` builds @@ -354,7 +354,7 @@ runtime/ ComlineProject/runtime ### The API -The header and the [C ABI reference](../../reference/c-abi.md) hold the +The header and the [C ABI reference](../../reference/languages/c-abi.md) hold the as-built API. The C sketch that stood here is gone. Its shape held: - a version, `comline_abi_version()`, and a header that carries it; @@ -779,4 +779,4 @@ Steps 1–3 are done ([What's built](#whats-built)). - the [runtime guide](../../guides/runtime/index.md)'s "no FFI boundary"; - [`core-target-contract.md`](../../design/core-target-contract.md) §4.7 (answered here) and §4.6's `no_std` layering table; - - a `comline.h` reference page: done, the [C ABI reference](../../reference/c-abi.md). + - a `comline.h` reference page: done, the [C ABI reference](../../reference/languages/c-abi.md). diff --git a/docs/docs/proposals/shared-runtime-core/index.md b/docs/docs/proposals/shared-runtime-core/index.md index a3f796d..4fd7699 100644 --- a/docs/docs/proposals/shared-runtime-core/index.md +++ b/docs/docs/proposals/shared-runtime-core/index.md @@ -113,7 +113,7 @@ Split the work by what can cross a language boundary cheaply. | Stage | Page | State | |---|---|---| | 1 | [Verification: differential fuzzing and benchmark gates](verification.md) | accepted, in progress | -| 2 | [One protocol core: a C ABI and WASM from one crate](core-interface.md) | accepted; rollout steps 1–3 built, and TypeScript binds it through wasm (stage 4); see the [C ABI reference](../../reference/c-abi.md) | +| 2 | [One protocol core: a C ABI and WASM from one crate](core-interface.md) | accepted; rollout steps 1–3 built, and TypeScript binds it through wasm (stage 4); see the [C ABI reference](../../reference/languages/c-abi.md) | | 3 | [Codecs and protocol glue generated from one plan](generated-codecs.md) | proposal | | 4 | [Move TypeScript onto the core where the counts allow](#stage-4-moving-typescript) | built: TypeScript runs on the core through wasm ([ts#87](https://github.com/ComlineProject/comline-typescript/pull/87), [ts#88](https://github.com/ComlineProject/comline-typescript/pull/88), [ts#89](https://github.com/ComlineProject/comline-typescript/pull/89)) | | 5 | Clean up the leftovers and the documents that contradict this | agreed | diff --git a/docs/docs/proposals/shared-runtime-core/verification.md b/docs/docs/proposals/shared-runtime-core/verification.md index b38ba9b..6152a9b 100644 --- a/docs/docs/proposals/shared-runtime-core/verification.md +++ b/docs/docs/proposals/shared-runtime-core/verification.md @@ -179,7 +179,7 @@ except where noted. - The job summary pairs each ABI call with its core twin, the crossing cost. It informs and isn't gated on its own. - runtime#60 cut the crossing: every ABI call is cheaper, none dearer. The - numbers are in the [C ABI reference](../../reference/c-abi.md#what-crossing-costs). + numbers are in the [C ABI reference](../../reference/languages/c-abi.md#what-crossing-costs). - **Required checks.** The `main` ruleset requires `ffi`, `ffi miri`, `wasm32 core` and the fuzz jobs `fuzz (ffi_feed)`, `fuzz (ffi_vs_core)` and `fuzz (stream_split)`. See [Branch rules](../../conduct/branch-rules.md). diff --git a/docs/docs/reference/conformance/fuzzing.md b/docs/docs/reference/conformance/fuzzing.md index b95eaca..22f057f 100644 --- a/docs/docs/reference/conformance/fuzzing.md +++ b/docs/docs/reference/conformance/fuzzing.md @@ -4,6 +4,8 @@ How the reference runtime is fuzzed, and how other runtimes are tested against it on random inputs. The source of truth is the runtime's [`runtime/fuzz/README.md`](https://github.com/ComlineProject/runtime/blob/main/runtime/fuzz/README.md). +For context, when **"reference runtime"** is mentioned, it is talking about the [reference runtime repository](https://github.com/ComlineProject/runtime/tree/main/runtime/reference). + The Rust runtime is the reference every other runtime is checked against, so its decoders must be strict, canonical and panic-free. Two kinds of test check that: @@ -70,12 +72,6 @@ A runtime that isn't the reference compares itself with it through `comline-reference`, which answers any [ops protocol](wire.md#the-ops-protocol) question. -comline-typescript's nightly workflow (03:17 UTC) installs `comline-reference` -at the corpus's `VECTORS_REV` and runs its property tests -(`runtime/test/prop/`) against it at many more runs than a pull request's. -Connection logic is compared through the `session` op. A failure opens or -comments on an issue. Coverage-guided fuzzing (Jazzer.js) isn't set up yet. - ## Outputs | Report | Written by | Published | diff --git a/docs/docs/reference/generator-protocol.md b/docs/docs/reference/generator-protocol.md index 818c6b3..dde245d 100644 --- a/docs/docs/reference/generator-protocol.md +++ b/docs/docs/reference/generator-protocol.md @@ -4,15 +4,14 @@ How the CLI talks to a code generator, and to a documentation renderer. For a walkthrough, see [Generators](../developer/extending/generators.md); to point a target at one, see the [CLI reference](tools/cli/index.md#commands) and -[`comline.toml`](package/congregation/comline-toml.md#generator). +[`comline.toml`](package/comline-toml.md#generator). The Rust definitions are in `codegen/src/protocol.rs` of the [generation](https://github.com/ComlineProject/generation) repository. ## Version -**Protocol version: `0`** (`PROTOCOL_VERSION`). While Comline is in its early -days, the protocol may change incompatibly, and the number moves when it does. +**Protocol version: `0`** (`PROTOCOL_VERSION`). An additive change keeps the number: a new field with a default, or a new request that only goes to a program whose `Description` says it handles it (`Render` was added this way). @@ -21,6 +20,10 @@ The protocol is also pinned to an **IR format** (`IR_FORMAT`), the stamp on `.comline/` that `comline-core` owns. A generator accepts the IR formats it lists and refuses others. +!!! info "Early days" + While Comline is in early days, the protocol may change incompatibly, + and the protocol version number moves when it does. + ## Transport - One request is written to the generator's **stdin**, then stdin closes. diff --git a/docs/docs/reference/c-abi.md b/docs/docs/reference/languages/c-abi.md similarity index 97% rename from docs/docs/reference/c-abi.md rename to docs/docs/reference/languages/c-abi.md index ed316b3..3c2a381 100644 --- a/docs/docs/reference/c-abi.md +++ b/docs/docs/reference/languages/c-abi.md @@ -6,13 +6,13 @@ for the `comline@1` and `comline@2` handshakes, the connection framings transports, async and dispatch stay in each host. - **ABI version 0.2.** It is not frozen; it freezes when the project announces - 1.0 ([phases](../conduct/phases.md)). See [Version](#version). + 1.0 ([phases](../../conduct/phases.md)). See [Version](#version). - **Source:** [`runtime/ffi`](https://github.com/ComlineProject/runtime/tree/main/runtime/ffi) in the runtime repository. - **Header:** [`include/comline.h`](https://github.com/ComlineProject/runtime/blob/main/runtime/ffi/include/comline.h). cbindgen 0.29.2 generates it from `src/lib.rs`, and it is checked in. Its comments are the doc comments of the Rust source. -- **Why it exists and what was decided:** [one protocol core](../proposals/shared-runtime-core/core-interface.md). +- **Why it exists and what was decided:** [one protocol core](../../proposals/shared-runtime-core/core-interface.md). This page was checked against runtime `main` at `b101fe9`. Where it and the header disagree, the header is right. @@ -32,7 +32,7 @@ header disagree, the header is right. - **Before 1.0, a minor counts as a major.** A host built against `0.2` accepts only `0.2`. The JS glue does exactly this. - **The freeze comes with the project's 1.0 announcement,** not with an ABI - milestone of its own ([phases](../conduct/phases.md)). + milestone of its own ([phases](../../conduct/phases.md)). - The major changes when a signature, a struct layout or the meaning of a code changes. The minor changes when functions, codes or flag bits are added. - Structs are frozen within a major. New data comes through new functions or new @@ -61,7 +61,7 @@ connection is open from the start, and `start` writes nothing. ### `comline@2` -[`comline@2`](wire/handshakes/index.md#comline2) has several messages, so its +[`comline@2`](../wire/handshakes/comline@2.md) has several messages, so its life differs in steps 1 to 3: 1. `comline_connection_new_comline2`, with `handshake` set to @@ -201,12 +201,12 @@ A decoded `comline@1` frame, written by `comline_handshake_check`. 32 bytes. A `comline@2` connection's facts and parameters, read once by `comline_connection_new_comline2`. Each parameter's default is 0, and -[the handshake's reference](wire/handshakes/index.md#parameters) says what it means. +[the handshake's reference](../wire/handshakes/comline@2.md#parameters) says what it means. 24 bytes on a 64-bit target and 16 on wasm32. | Field | Type | Meaning | |---|---|---| -| `hello`, `hello_len` | `const uint8_t *`, `size_t` | This end's facts, as [an encoded hello message](wire/handshakes/index.md#messages). A generator can write them out as a constant. They're put in canonical order, and the `exchange` byte is replaced by `exchange` below. | +| `hello`, `hello_len` | `const uint8_t *`, `size_t` | This end's facts, as [an encoded hello message](../wire/handshakes/comline@2.md#messages). A generator can write them out as a constant. They're put in canonical order, and the `exchange` byte is replaced by `exchange` below. | | `side` | `uint8_t` | `COMLINE_SIDE_INITIATOR` (0) or `COMLINE_SIDE_RESPONDER` (1). | | `scope` | `uint8_t` | `COMLINE_SCOPE_PROTOCOL` (0) or `COMLINE_SCOPE_PACKAGE` (1). | | `newer_consumer` | `uint8_t` | `COMLINE_NEWER_CONSUMER_DEGRADE` (0) or `COMLINE_NEWER_CONSUMER_REFUSE` (1). | @@ -220,7 +220,7 @@ Why `comline@2` refused a connection. 8 bytes. | Field | Type | Meaning | |---|---|---| -| `reason` | `uint8_t` | `COMLINE_REASON_*`, the [reason codes](wire/handshakes/index.md#reason-codes). | +| `reason` | `uint8_t` | `COMLINE_REASON_*`, the [reason codes](../wire/handshakes/comline@2.md#reason-codes). | | `side` | `uint8_t` | Which end refused: `COMLINE_SIDE_*`. | | `has_protocol` | `uint8_t` | 1 when the refusal is about one protocol. | | `_reserved` | `uint8_t` | 0. | @@ -491,8 +491,8 @@ cbindgen --config cbindgen.toml --output include/comline.h The CI jobs are `ffi`, `ffi miri`, `wasm32 core` and `fuzz (ffi_feed)`, `fuzz (ffi_vs_core)` and `fuzz (stream_split)`; all are required to merge. See -[Verification](../proposals/shared-runtime-core/verification.md) for how the -fuzz targets and the benchmark gate work, and [Branch rules](../conduct/branch-rules.md). +[Verification](../../proposals/shared-runtime-core/verification.md) for how the +fuzz targets and the benchmark gate work, and [Branch rules](../../conduct/branch-rules.md). ## What crossing costs @@ -539,9 +539,9 @@ rustc 1.98.0. Message mode, a 16-byte payload: few instructions between machines, because the overlap check compares addresses, so the CI figures above differ slightly from the PR's. - **Not in these tables:** the wasm32 build and the JS glue. Their cost was - measured end to end for [stage 4](../proposals/shared-runtime-core/index.md#stage-4-moving-typescript), + measured end to end for [stage 4](../../proposals/shared-runtime-core/index.md#stage-4-moving-typescript), with a local tool and not a CI gate; the results are on the - [TypeScript page](../languages/typescript/index.md#what-it-costs). + [TypeScript page](../../languages/typescript/index.md#what-it-costs). The gate compares each call with its base on every pull request and fails at a 1% rise, like the core's benches. Both sides of each pair are gated. @@ -551,5 +551,5 @@ a 1% rise, like the core's benches. Both sides of each pair are gated. - **`comline_connection_size` and `comline_connection_init`,** for a host that supplies the handle's memory. A later addition. - **More bindings.** TypeScript binds the ABI through the JS glue (stage 4, - see [TypeScript on the core](../languages/typescript/index.md#on-the-shared-core)). No + see [TypeScript on the core](../../languages/typescript/index.md#on-the-shared-core)). No other language does yet. diff --git a/docs/docs/reference/languages/rust.md b/docs/docs/reference/languages/rust.md index e23ad64..c7d05b9 100644 --- a/docs/docs/reference/languages/rust.md +++ b/docs/docs/reference/languages/rust.md @@ -91,7 +91,7 @@ are compared against (see [Verification](../../proposals/shared-runtime-core/ver - Seven [`cargo-fuzz`](https://github.com/ComlineProject/runtime/tree/main/runtime/fuzz) targets: `decode_no_panic`, `packed_canonical`, `framing_decode`, `seed_vs_derive` and `stream_split`, and, for the - [C ABI](../c-abi.md), `ffi_feed` and `ffi_vs_core`. + [C ABI](c-abi.md), `ffi_feed` and `ffi_vs_core`. - [Property tests](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/tests/properties.rs) (`proptest`) over random types and values, run with `cargo test`. - CI runs each target for a minute on every pull request, and for ten minutes diff --git a/docs/docs/reference/languages/typescript.md b/docs/docs/reference/languages/typescript.md index 70c88bc..80f9983 100644 --- a/docs/docs/reference/languages/typescript.md +++ b/docs/docs/reference/languages/typescript.md @@ -135,3 +135,11 @@ branch. The latest, fetched when this site was built: Every published run, and the ratio to Rust per workload: [Observatory → Benchmarks](https://comlineproject.github.io/observatory/bench/typescript-bench/) and [ratio to Rust](https://comlineproject.github.io/observatory/ratio/). + +### Conformance + +comline-typescript's nightly workflow (03:17 UTC) installs `comline-reference` +at the corpus's `VECTORS_REV` and runs its property tests +(`runtime/test/prop/`) against it at many more runs than a pull request's. +Connection logic is compared through the `session` op. A failure opens or +comments on an issue. Coverage-guided fuzzing (Jazzer.js) isn't set up yet. diff --git a/docs/docs/reference/package/congregation/comline-toml.md b/docs/docs/reference/package/comline-toml.md similarity index 92% rename from docs/docs/reference/package/congregation/comline-toml.md rename to docs/docs/reference/package/comline-toml.md index f4ea9f8..60df494 100644 --- a/docs/docs/reference/package/congregation/comline-toml.md +++ b/docs/docs/reference/package/comline-toml.md @@ -7,20 +7,20 @@ never frozen, and never part of a published package. With no `comline.toml` (or an all-commented one) the defaults below apply and targets come from `config.idp`'s `code_generation.languages`. -For the task-oriented walkthrough see [Generating code](../../../guides/codegen/generating-code.md). +For the task-oriented walkthrough see [Generating code](../../guides/codegen/generating-code.md). ## `[generate]` Defaults for every target. | Key | Type | Default | Meaning | -|---|---|---|---| +| --- | --- | --- | --- | | `out` | string (path) | `"generated"` | Output root, relative to `comline.toml`. | | `layout` | string (template) | `"{{language}}/{{namespace}}.{{ext}}"` | Path of each generated file under `out`. Handlebars, strict. | -| `mode` | `"code"` \| `"lib"` \| `"dylib"` | `"code"` | Emit form. `code` and `lib` are implemented for `rust` and `typescript`; `dylib` is planned — see [Library generation](../../../guides/codegen/library-generation.md). | +| `mode` | `"code"` \| `"lib"` \| `"dylib"` | `"code"` | Emit form. `code` and `lib` are implemented for `rust` and `typescript`; `dylib` is planned — see [Library generation](../../guides/codegen/library-generation.md). | | `package_versions` | `"latest"` \| `"all"` \| list | `"latest"` | Which package versions to emit — see [below](#package_versions). | | `external_std` | bool | on in `lib`, off in `code` | Reference `std` as its own package instead of inlining it — see [below](#external_std). | -| `validate_on_decode` | `"collect"` \| `"first"` \| `{ off = "why" }` | `"collect"` | Whether decoding a value runs its [`@validators`](../../../guides/idl/validator.md); off is a give-up with its reason — see [below](#validate_on_decode). | +| `validate_on_decode` | `"collect"` \| `"first"` \| `{ off = "why" }` | `"collect"` | Whether decoding a value runs its [`@validators`](../../guides/idl/validator.md); off is a give-up with its reason — see [below](#validate_on_decode). | | `async` | bool | `false` | Generate protocols on a connection for an async driver — see [below](#async). | | `encodings` | table | — | Which type implements each encoding (format, framing, handshake) in the generated code — see [below](#encodings). | | `only` | list of strings | — | Generate just these languages, out of the ones `config.idp` declares — see [below](#only). | @@ -38,9 +38,9 @@ declare is an error. To generate *fewer* than the package declares, use [`only`](#only). | Key | Type | Meaning | -|---|---|---| +| --- | --- | --- | | `language` | string | **Required.** Must be one the manifest declares under `code_generation.languages`. | -| `lang_version` | string | Optional; the toolchain version to generate for. Defaults to what `config.idp` declared, and may raise it but never lower it ([Language versions](../../../guides/codegen/generating-code.md#language-versions)). | +| `lang_version` | string | Optional; the toolchain version to generate for. Defaults to what `config.idp` declared, and may raise it but never lower it ([Language versions](../../guides/codegen/generating-code.md#language-versions)). | | `out`, `layout`, `mode`, `package_versions`, `external_std`, `validate_on_decode`, `async` | — | Per-target override of the `[generate]` value. | | `generator` | string (path) | A generator outside the CLI, a `.wasm` or a native executable, relative to `comline.toml`'s directory. Unset, the built-in generator for `language` is used. See [below](#generator). | | `timeout` | integer (seconds) | With `generator` only: how long each call to it may take. At least 1; default 60. | @@ -68,11 +68,11 @@ timeout = 120 ``` Runs a generator that is its own program, speaking the -[generator protocol](../../generator-protocol.md). `language` picks among the +[generator protocol](../generator-protocol.md). `language` picks among the languages it offers, and may be left out if it offers one; it must still be one `config.idp` declares. A `.wasm` runs sandboxed; anything else runs as a native process, unsandboxed. The limits and path rules are in -[Generators](../../../developer/extending/generators.md#limits). +[Generators](../../developer/extending/generators.md#limits). ### `encodings` @@ -91,7 +91,7 @@ language. A value is the type alone, or a table with `type` and optionally that answers wins: this table (the target's over `[generate]`'s), then the `map` the declaring package suggests in its `config.idp`. An encoding that is used and has no mapping is a generation error. See -[Map it to a type](../../../developer/extending/encodings.md#5-map-it-to-a-type). +[Map it to a type](../../developer/extending/encodings.md#5-map-it-to-a-type). ### `flatten` @@ -124,7 +124,7 @@ decides whether their code is **copied into** your generated output or **referenced** from std's own pre-generated package: | Language | Package | -|---|---| +| --- | --- | | `rust` | `comline_std` (a crate, pinned by git rev) | | `typescript` | `@comline/std` (an npm package, by semver range) | @@ -141,12 +141,12 @@ already true of `@comline/runtime` for any package with a protocol. ### `validate_on_decode` -Generated code always carries a schema's [`@validators`](../../../guides/idl/validator.md) +Generated code always carries a schema's [`@validators`](../../guides/idl/validator.md) checks as `validate()` / `validate_first()`, for you to call. This decides whether **decoding** calls them too: | Value | Decoding a value that fails its checks | -|---|---| +| --- | --- | | `"collect"` (the default) | fails, with every failed check | | `"first"` | fails, with the first failed check | | `{ off = "why" }` | succeeds; nothing runs until you call `validate()` | @@ -158,7 +158,7 @@ stop early want different things from the same type. A provider also refuses a call whose arguments fail their checks, the same way it refuses arguments that don't decode. -Not running them is a **give-up**, like [`@unverified`](../../../guides/idl/protocol.md), +Not running them is a **give-up**, like [`@unverified`](../../guides/idl/protocol.md), and it carries its reason: ```toml @@ -177,14 +177,14 @@ its reason is kept. ### `async` -A protocol on a [connection](../../../guides/idl/protocol.md#calls-in-both-directions) +A protocol on a [connection](../../guides/idl/protocol.md#calls-in-both-directions) is generated for a blocking driver by default. `async = true` generates it for an async one instead: - **`rust`:** async handlers, `AsyncDispatcher` and `AsyncSender`, and an async package `serve` / `accept` / `connect`, for the runtime's `AsyncConnection` on any executor — see - [Rust connections](../../../languages/rust/connections.md#async). A `lib` + [Rust connections](../../languages/rust/connections.md#async). A `lib` package turns on the runtime's `async` feature. - **`typescript`:** nothing changes; its handlers and senders return Promises either way. @@ -216,11 +216,11 @@ you want fewer. ## `[doc]` -Settings for [`comline doc`](../../tools/cli/doc.md). Each key but `renderer`, +Settings for [`comline doc`](../tools/cli/doc.md). Each key but `renderer`, `timeout` and `links` has a flag that overrides it. | Key | Type | Default | Meaning | -|---|---|---|---| +| --- | --- | --- | --- | | `out` | string (path) | `"docs/api"` | Output directory, relative to `comline.toml`. `--out` overrides it. | | `formats` | array of strings | `["markdown"]` | What to write, each into `//` in one run: `markdown` (pages), `html` (a static site), `json` (the documentation model, `model.json`), or another renderer id `renderer` offers. `--format` overrides it. Replaces `format`, which is no longer accepted. | | `versions` | `"all"` or table | `"all"` | Which versions to document. `"all"`: every release, every dev build and the unreleased working state. A table trims the list, and its keys combine, applied in this order: `releases_only = true` keeps releases alone (no dev builds, no unreleased state), `since = "0.3.0"` keeps that version and newer, `last = N` keeps the newest N (the unreleased state counts as one). `--version` documents one version instead. | @@ -240,7 +240,7 @@ links = { std = "https://example.com/std" } ## Layout variables | Variable | Value | -|---|---| +| --- | --- | | `{{language}}` | target language, e.g. `rust` | | `{{namespace}}` | the schema's namespace, `/`-joined | | `{{ext}}` | the generator's file extension, e.g. `rs` | @@ -251,7 +251,7 @@ links = { std = "https://example.com/std" } ## `package_versions` | Value | Effect | -|---|---| +| --- | --- | | `"latest"` (default) | The working tree only. No `.comline/` read. `{{package_version}}` is the last committed version (empty if never built). | | `"all"` | Every release in the project's history (not the builds between them). | | `["0.3.0", "0.4.0"]` | Exactly those versions: releases, builds (`0.3.0-dev.2`) or commit hashes. | @@ -271,7 +271,7 @@ layout = "{{language}}/{{package_version}}/{{namespace}}.{{ext}}" The framing is part of the schema, since both ends have to agree on it, so it moved to the package: `config.idp`'s `wire = { framing = "..." }` -([Wire defaults](../../../guides/packages/index.md#wire-defaults)). A +([Wire defaults](../../guides/packages/index.md#wire-defaults)). A `default_framing`, in `[generate]` or in a target, is an error that names the line to write there instead: diff --git a/docs/docs/reference/package/congregation/index.md b/docs/docs/reference/package/congregation/index.md deleted file mode 100644 index 70cd666..0000000 --- a/docs/docs/reference/package/congregation/index.md +++ /dev/null @@ -1,26 +0,0 @@ -# Package files - -A package (historically a *congregation*) has two configuration files with -different owners. Keeping them apart is what lets one package be generated -into many projects, each its own way. - -```mermaid -flowchart LR - A["package author"] --> IDP["config.idp
identity, languages,
dependencies, wire, settings"] - IDP -->|"frozen into
every version"| CAS[(".comline/")] - C["whoever generates"] --> TOML["comline.toml
output paths, layout,
mode, versions"] - CAS --> GEN["comline generation"] - IDP --> GEN - TOML --> GEN -``` - -| | [`config.idp`](config-id.md) | [`comline.toml`](comline-toml.md) | -|---|---|---| -| Owner | the package author | whoever runs `comline generation` | -| Says | what the package is: name, identity, languages it supports, dependencies, wire defaults, settings | where generated code goes, in what layout and mode, for which versions | -| Frozen into versions | yes | no | -| Changes the version | can | never | - -The reasoning behind the split is in -[Consumer generation configuration](../../../design/consumer-generation-config.md). -How the version follows from changes is in [Versioning rules](../../versioning.md). diff --git a/docs/docs/reference/package/congregation/config-id.md b/docs/docs/reference/package/idp.md similarity index 70% rename from docs/docs/reference/package/congregation/config-id.md rename to docs/docs/reference/package/idp.md index 3340d92..370399d 100644 --- a/docs/docs/reference/package/congregation/config-id.md +++ b/docs/docs/reference/package/idp.md @@ -5,7 +5,7 @@ it can be generated as, and the rules and wire defaults its schemas share. It sits at the package root, beside `src/`. The [package author](index.md) owns it, and it's frozen into every built version. -For a walk through it, see [Packages & dependencies](../../../guides/packages/index.md). +For a walk through it, see [Packages & dependencies](../../guides/packages/index.md). ```idp //! The shop's API: orders, customers and payments. @@ -47,22 +47,22 @@ settings = { - **Keys don't take dots.** `.ids` settings blocks accept `validators.allowed = false`; here the same key is written with braces. - **Comments** are `//`. `//!` lines at the top - [document the package](../../../guides/idl/docstrings.md#module-docs). + [document the package](../../guides/idl/docstrings.md#module-docs). ## Keys | Key | Value | Details | |---|---|---| | `congregation` | the package's name | Renaming it is fine: `id` is what identifies the package. | -| `id` | 32 lowercase hex digits, random | The package's identity. `comline new` writes it, and `comline build` adds one to a package without it. A copy of the package must change it. Changing it is a major change. See [Compatibility facts](../../versioning.md#compatibility-facts). | -| `specification_version` | an integer: `1`, the only edition so far | **Required.** The [edition](../../versioning.md#editions) of the schema language the files are written in. Leaving it out, or naming an edition this Comline doesn't know, is an error with a fix that `comline fix` applies. | -| `code_generation` | `{ languages = { # = {} ... } }` | The languages the package can be generated as, each with the oldest toolchain its output must build on. `comline build` proves each one. See [Language versions](../../../guides/codegen/generating-code.md#language-versions). | -| `wire` | `{ format = "..." framing = "..." }` | The format and framing every protocol uses unless it says otherwise. Defaults: `packed@1` and `datagram@3`. See [Wire defaults](../../../guides/packages/index.md#wire-defaults). | -| `handshake` | `"comline@1"`, `"comline@2"` or `"none"`, or `{ name = "comline@2" scope = "package" }` to set [its parameters](../../wire/handshakes/index.md#parameters) | What a protocol with no `@handshake` runs before its first call. Default `comline@2`, or `comline@1` in a package whose protocols aren't all on `datagram@3`. See [Giving up a guarantee](../../../guides/idl/protocol.md#giving-up-a-guarantee). | -| `dependencies` | a dictionary of entries by name | Other packages this one imports, from a path or a git commit, each optionally pinned by `hash`, and what it accepts of their [`@unverified`](../../../guides/idl/protocol.md#giving-up-a-guarantee) protocols (`accept_unverified`). See [Dependencies](../../../guides/packages/index.md#dependencies). | -| `settings` | a dictionary, or `::settings::` | The package-wide authoring policy every schema starts from. See [Settings](../../../guides/idl/settings.md). | -| `encodings` | a dictionary of entries by name | The formats, framings and handshakes this package defines. See [Formats and framings](../../../developer/extending/encodings.md). | -| `publish_registries` | a dictionary of registries by name | Where the package can be published. Planned; see [Publishing](../../../guides/packages/index.md#publishing). | +| `id` | 32 lowercase hex digits, random | The package's identity. `comline new` writes it, and `comline build` adds one to a package without it. A copy of the package must change it. Changing it is a major change. See [Compatibility facts](../versioning.md#compatibility-facts). | +| `specification_version` | an integer: `1`, the only edition so far | **Required.** The [edition](../versioning.md#editions) of the schema language the files are written in. Leaving it out, or naming an edition this Comline doesn't know, is an error with a fix that `comline fix` applies. | +| `code_generation` | `{ languages = { # = {} ... } }` | The languages the package can be generated as, each with the oldest toolchain its output must build on. `comline build` proves each one. See [Language versions](../../guides/codegen/generating-code.md#language-versions). | +| `wire` | `{ format = "..." framing = "..." }` | The format and framing every protocol uses unless it says otherwise. Defaults: `packed@1` and `datagram@3`. See [Wire defaults](../../guides/packages/index.md#wire-defaults). | +| `handshake` | `"comline@1"`, `"comline@2"` or `"none"`, or `{ name = "comline@2" scope = "package" }` to set [its parameters](../wire/handshakes/comline@2.md#parameters) | What a protocol with no `@handshake` runs before its first call. Default `comline@2`, or `comline@1` in a package whose protocols aren't all on `datagram@3`. See [Giving up a guarantee](../../guides/idl/protocol.md#giving-up-a-guarantee). | +| `dependencies` | a dictionary of entries by name | Other packages this one imports, from a path or a git commit, each optionally pinned by `hash`, and what it accepts of their [`@unverified`](../../guides/idl/protocol.md#giving-up-a-guarantee) protocols (`accept_unverified`). See [Dependencies](../../guides/packages/index.md#dependencies). | +| `settings` | a dictionary, or `::settings::` | The package-wide authoring policy every schema starts from. See [Settings](../../guides/idl/settings.md). | +| `encodings` | a dictionary of entries by name | The formats, framings and handshakes this package defines. See [Formats and framings](../../developer/extending/encodings.md). | +| `publish_registries` | a dictionary of registries by name | Where the package can be published. Planned; see [Publishing](../../guides/packages/index.md#publishing). | An unknown key, a value of the wrong kind, or a dependency that doesn't resolve fails `comline check` and `comline build`, and the editor reports it @@ -73,7 +73,7 @@ as you type. `comline build` lowers the manifest to IR units and stores them with every version, so a version carries its own declarations. A change to the manifest counts like a schema change; the rules are in -[Manifest changes](../../versioning.md#manifest-configidp-changes). +[Manifest changes](../versioning.md#manifest-configidp-changes). It never holds an output path: where generated code lands is the consumer's choice, in [`comline.toml`](comline-toml.md). diff --git a/docs/docs/reference/package/index.md b/docs/docs/reference/package/index.md index e69de29..d6b6607 100644 --- a/docs/docs/reference/package/index.md +++ b/docs/docs/reference/package/index.md @@ -0,0 +1,26 @@ +# Package + +A package (historically a *congregation*) has two configuration files with +different owners. Keeping them apart is what lets one package be generated +into many projects, each its own way. + +```mermaid +flowchart LR + A["package author"] --> IDP["config.idp
identity, languages,
dependencies, wire, settings"] + IDP -->|"frozen into
every version"| CAS[(".comline/")] + C["whoever generates"] --> TOML["comline.toml
output paths, layout,
mode, versions"] + CAS --> GEN["comline generation"] + IDP --> GEN + TOML --> GEN +``` + +| | [`config.idp`](idp.md) | [`comline.toml`](comline-toml.md) | +| --- | --- | --- | +| Owner | the package author | whoever runs `comline generation` | +| Says | what the package is: name, identity, languages it supports, dependencies, wire defaults, settings | where generated code goes, in what layout and mode, for which versions | +| Frozen into versions | yes | no | +| Changes the version | can | never | + +The reasoning behind the split is in +[Consumer generation configuration](../../design/consumer-generation-config.md). +How the version follows from changes is in [Versioning rules](../versioning.md). diff --git a/docs/docs/reference/schema/index.md b/docs/docs/reference/package/schema/index.md similarity index 100% rename from docs/docs/reference/schema/index.md rename to docs/docs/reference/package/schema/index.md diff --git a/docs/docs/reference/schema/std/http.md b/docs/docs/reference/package/schema/std/http.md similarity index 100% rename from docs/docs/reference/schema/std/http.md rename to docs/docs/reference/package/schema/std/http.md diff --git a/docs/docs/reference/schema/std/index.md b/docs/docs/reference/package/schema/std/index.md similarity index 100% rename from docs/docs/reference/schema/std/index.md rename to docs/docs/reference/package/schema/std/index.md diff --git a/docs/docs/reference/schema/std/validators.md b/docs/docs/reference/package/schema/std/validators.md similarity index 100% rename from docs/docs/reference/schema/std/validators.md rename to docs/docs/reference/package/schema/std/validators.md diff --git a/docs/docs/reference/package/schema/types.md b/docs/docs/reference/package/schema/types.md new file mode 100644 index 0000000..99243f5 --- /dev/null +++ b/docs/docs/reference/package/schema/types.md @@ -0,0 +1,65 @@ +# Types + +The built-in types of the schema language. What they become in each +language is in [the guide](../../../guides/idl/types.md#in-generated-code), and +their bytes are in each [format](../../wire/formats/index.md). Bounds +(`T[max N]`) are covered in [the guide](../../../guides/idl/types.md#bounds). + +## `bool` { #bool } + +`true` or `false`, the only two literals of the type. + +## Integers + +### Unsigned { #unsigned } + +| Type | Range | +| --- | --- | +| `u8` | 0 to 255 | +| `u16` | 0 to 65 535 | +| `u32` | 0 to 2^32 − 1 | +| `u64` | 0 to 2^64 − 1 | +| `u128` | 0 to 2^128 − 1 | + +### Signed { #signed } + +Two's complement. + +| Type | Range | +| --- | --- | +| `s8` | −128 to 127 | +| `s16` | −32 768 to 32 767 | +| `s32` | −2^31 to 2^31 − 1 | +| `s64` | −2^63 to 2^63 − 1 | +| `s128` | −2^127 to 2^127 − 1 | + +Signed integers are written `s`, not `i`: `i32` is not Comline syntax. + +## Floats { #floats } + +| Type | Precision | +| --- | --- | +| `f32` | IEEE 754 single precision | +| `f64` | IEEE 754 double precision | + +## Text: `string` and `char` { #text-string-and-char } + + +- **`char`** is one Unicode scalar value: a single character, not a byte or a + UTF-16 unit. Its literal is a string of exactly one character, `"a"`; any + other length is a build error. +- **`string`** is the text type: UTF-8, the built-in name for `char[]`. An + array of `char` is always text, so `char[max 64]` is a string of at most 64 + characters, not a list of 64 `char` values. +- **`str`** was an accepted spelling until it was removed. Writing it is now a + build error that tells you to use `string`. +- **`String`** (capitalised) is **not** Comline syntax. It is the Rust type that + `string` generates into, so you see it in generated Rust, never in a `.ids` + file. Writing it gets the same hint. + +## `()` { #unit } + +The unit type: no value, and no bytes on the wire. It's what a function that +waits for its provider but returns nothing gives back, `-> ()`, unlike a +function with no `->`, which gets no reply at all +([One-way, or acknowledged](../../../guides/idl/protocol.md#one-way-or-acknowledged)). diff --git a/docs/docs/reference/tools/cli/doc.md b/docs/docs/reference/tools/cli/doc.md index 744ce5c..349a44d 100644 --- a/docs/docs/reference/tools/cli/doc.md +++ b/docs/docs/reference/tools/cli/doc.md @@ -40,7 +40,7 @@ reads it. `comline doc --open` is the same after a render, as `cargo doc --open` is. -The `[doc]` keys are in [`comline.toml`](../../package/congregation/comline-toml.md#doc), +The `[doc]` keys are in [`comline.toml`](../../package/comline-toml.md#doc), including `versions`, which trims the list of versions. ## Versions and paths @@ -124,7 +124,7 @@ Each declaration has an anchor, its name in lowercase; a function has `-`. A type links to its declaration's anchor. A type from a dependency links to `//#` when `[doc] links` gives the dependency a base URL; `std` defaults to the hosted std docs, -`https://comlineproject.github.io/docs/reference/schema/std/` (`std::http::HttpMethod` links to +`https://comlineproject.github.io/docs/reference/package/schema/std/` (`std::http::HttpMethod` links to `.../std/http/#httpmethod`). Otherwise it links to the dependency's own pages, `deps//.md`, and is plain text only when the package doesn't reach that declaration. @@ -167,7 +167,7 @@ types Dependencies (collapsed) localdeps greeting -> deps/localdeps/greeting.html - std ↗ -> https://comlineproject.github.io/docs/reference/schema/std/ + std ↗ -> https://comlineproject.github.io/docs/reference/package/schema/std/ http -> .../std/http/ Changelog ``` diff --git a/docs/docs/reference/tools/cli/index.md b/docs/docs/reference/tools/cli/index.md index 714e7ce..c4b85ba 100644 --- a/docs/docs/reference/tools/cli/index.md +++ b/docs/docs/reference/tools/cli/index.md @@ -45,11 +45,11 @@ itself the command that writes code, and has `clean` beneath it to remove it. Ru | `comline fix [--edition ] [--dry-run]` | Apply the fixes Comline attaches to what it reports about the language: add a missing `specification_version` to `config.idp`, and replace each use of a keyword or type the package's [edition](../../versioning.md#editions) deprecates or removes, when it has a replacement; uses without one are listed to change by hand. `--edition ` checks and fixes the schemas under edition `n` and moves `config.idp` to it, unless a removed word without a replacement is left (exit `1`). Moving back is refused. `--dry-run` lists the fixes and writes nothing. | | `comline inspect [--json]` | Show each protocol's format, framing and handshake, with their parameters and where each came from (`@framing`/`@handshake`, `config.idp`, or core's default), then every guarantee, held or given up, with the cause, the `@unverified` reason and whether it's accepted. No `.comline/` write. `--json` prints the same as JSON. See [comline inspect](inspect.md). | | `comline build [--watch] [--reconcile ]` | Compile, validate, and freeze a build into `.comline/` under a pre-release version (`0.3.0-dev.2`); print the changelog, the bump, and the next release. `--reconcile ` puts a head dropped in a Git merge back beside the current one, so the build [reconciles the two](../../../guides/ir/cas.md); the output says when a build reconciled. A project with [unbounded strings or lists](../../../guides/idl/types.md#every-string-and-list-has-a-maximum) fails with each position listed and the `settings` block to add; on a terminal, the build offers to write it with maxima you type in. Before freezing, proves every declared `code_generation.languages` target against this binary's generators ([Language versions](../../../guides/codegen/generating-code.md#language-versions)). Releasing is `comline release`. | -| `comline generation` (short: `gen`) `[--target ] [--out ] [--layout ] [--mode ] [--external-std] [--no-external-std] [--validate-on-decode ] [--dry-run] [--list] [--watch]` | Validate (no freeze), then write generated code. Wire numbers and the rest of the [facts](../../versioning.md#number) come from the last build when nothing changed since, else are worked out as the next build would assign them, provisional until a release; no build is needed. Location/layout from [`comline.toml`](../../package/congregation/comline-toml.md); targets from there or `config.idp`. A target can use an [external generator](../../../developer/extending/generators.md) (`generator = ""`) and map [encodings](../../../developer/extending/encodings.md#5-map-it-to-a-type). | +| `comline generation` (short: `gen`) `[--target ] [--out ] [--layout ] [--mode ] [--external-std] [--no-external-std] [--validate-on-decode ] [--dry-run] [--list] [--watch]` | Validate (no freeze), then write generated code. Wire numbers and the rest of the [facts](../../versioning.md#number) come from the last build when nothing changed since, else are worked out as the next build would assign them, provisional until a release; no build is needed. Location/layout from [`comline.toml`](../../package/comline-toml.md); targets from there or `config.idp`. A target can use an [external generator](../../../developer/extending/generators.md) (`generator = ""`) and map [encodings](../../../developer/extending/encodings.md#5-map-it-to-a-type). | | `comline release [1.0] [--raise --reason ] [--yes]` | Build, then release: create the version from the highest bump since the last release ([builds and releases](../../versioning.md#builds-and-releases)). `--raise patch\|minor\|major` goes higher, with a `--reason` recorded with the release; it never lowers. `1.0` makes a `0.x` package stable after showing what that commits to (its protocols, and what's worth settling first). Raising to major and `1.0` ask on a terminal; without one they refuse (exit `2`) unless `--yes` (`-y`). | | `comline diff []` | Show schema changes between two built versions. Each argument is a version (`0.2.0`), a commit hash (4+ chars), or `HEAD` (`` default). | -| `comline doc [--out ] [--version ] [--changelog-from ] [--format ] [--open]` | Write the package's documentation for every version (each release, dev build and the unreleased working state), each into `///` with the changelog from the version before it: Markdown pages (an index, one page per module with its `//!` header, every declaration's `///` docs and each protocol's wire summary, a changelog, an mkdocs/zensical `nav.yml`), `--format html` for a self-contained static site with the same content, a search box, a version picker and a page comparing versions, or `--format json` for the documentation model. Built versions are cached, so a run re-renders only the unreleased state and new versions; `--version` documents one built version. Defaults from [`comline.toml`](../../package/congregation/comline-toml.md#doc)'s `[doc]`. `comline doc open []` (or `--open` after a render) opens the result with the system's default handler, rendering first if it isn't there. No `.comline/` write. See [comline doc](doc.md). | -| `comline conformance --impl [--encoding ]... [--corpus ] [--timeout ] [--json]` | Run an implementation of an encoding against its vectors. The package's declared [`encodings`](../../../developer/extending/encodings.md) ship vectors beside their profiles (`profiles/x.vectors.json`); `--impl` is a process speaking the [ops protocol](../../conformance/wire.md#the-ops-protocol), run once with a cleared environment, a time limit (`--timeout`, 60 s) and a capped stdout. `--encoding` limits the run to one encoding (repeatable); `--corpus` runs a corpus laid out like the runtime's `runtime/conformance` instead, which is how the official encodings are proven. A failed or unimplemented case, or an unproven profile fact, exits `1`. `--json` prints the report as JSON on stdout instead (nothing else on stdout; the exit code is the same): `format` (`1`), `implementation` (`command` and the `version` it reported, with its `protocol`), `encodings` (each with `name`, `kind`, `listed_by_implementation`, `passed`, `total`, `facts` as `{fact, value, proven, reason}` and `cases` as `{id, outcome, detail}` where `outcome` is `pass`, `fail` or `not_implemented`), `totals` (`encodings`, `cases`, `passed`, `problems`) and `verdict` (`conforms` or `fails`). See [Formats and framings](../../../developer/extending/encodings.md#3-write-vectors-and-prove-an-implementation). | +| `comline doc [--out ] [--version ] [--changelog-from ] [--format ] [--open]` | Write the package's documentation for every version (each release, dev build and the unreleased working state), each into `///` with the changelog from the version before it: Markdown pages (an index, one page per module with its `//!` header, every declaration's `///` docs and each protocol's wire summary, a changelog, an mkdocs/zensical `nav.yml`), `--format html` for a self-contained static site with the same content, a search box, a version picker and a page comparing versions, or `--format json` for the documentation model. Built versions are cached, so a run re-renders only the unreleased state and new versions; `--version` documents one built version. Defaults from [`comline.toml`](../../package/comline-toml.md#doc)'s `[doc]`. `comline doc open []` (or `--open` after a render) opens the result with the system's default handler, rendering first if it isn't there. No `.comline/` write. See [comline doc](doc.md). | +| `comline conformance --impl [--encoding ]... [--corpus ] [--timeout ] [--json]` | Run an implementation of an encoding against its vectors. The package's declared [`encodings`](../../../developer/extending/encodings.md) ship vectors beside their profiles (`profiles/x.vectors.json`); `--impl` is a process speaking the [ops protocol](../../conformance/wire.md#the-ops-protocol), run once with a cleared environment, a time limit (`--timeout`, 60 s) and a capped stdout. `--encoding` limits the run to one encoding (repeatable); `--corpus` runs a corpus laid out like the runtime's `runtime/conformance` instead, which is how the official encodings are proven. A failed or unimplemented case, or an unproven profile fact, exits `1`. `--json` prints the report as JSON on stdout instead (nothing else on stdout; the exit code is the same): `format` (`1`), `implementation` (`command` and the `version` it reported, with its `protocol`), `encodings` (each with `name`, `kind`, `listed_by_implementation`, `passed`, `total`, `facts` as `{fact, value, proven, reason}` and `cases` as `{id, outcome, detail}` where `outcome` is `pass`, `fail` or `not_implemented`), `totals` (`encodings`, `cases`, `passed`, `problems`) and `verdict` (`conforms` or `fails`). See [Formats and framings](../../../developer/extending/encodings.md#3-writing-vectors). | | `comline generation clean [--dry-run] [--target ]` | Remove what `generation` wrote: the output root, or the individual generated files. `--target ` (`-t`) removes only that language's output; a language no target matches is an error, as it is for `generation --target`. Never touches `.comline/`. | | `comline reset [--force] [--dry-run]` | **Destructive.** Delete `.comline/` — the whole version history, no undo — *and* generated code. Next `build` starts over at `0.1.0-dev.1`. On a terminal it shows how many versions will be lost and makes you type `reset`; without a terminal it refuses (exit `2`) unless `--force`. | | `comline completions ` | Print a completion script to stdout. `bash` `zsh` `fish` `powershell` `elvish`. | @@ -60,10 +60,10 @@ several are configured. `COMLINE_GENERATE_OUT` / `_LAYOUT` / `_MODE` do the same from the environment, for every target. `--external-std` makes generated code reference std's own package — `comline_std` for rust, `@comline/std` for typescript — instead of inlining it, and `--no-external-std` inlines it; -either overrides [`comline.toml`](../../package/congregation/comline-toml.md)'s +either overrides [`comline.toml`](../../package/comline-toml.md)'s `[generate] external_std`, which itself defaults to on in `lib` mode and off in `code` mode. `--validate-on-decode collect|first` overrides -[`[generate] validate_on_decode`](../../package/congregation/comline-toml.md#validate_on_decode), +[`[generate] validate_on_decode`](../../package/comline-toml.md#validate_on_decode), whether decoding runs the schema's `@validators` (also `COMLINE_GENERATE_VALIDATE_ON_DECODE`). The default is `collect`; turning it off is a give-up written in `comline.toml` with its reason diff --git a/docs/docs/reference/versioning.md b/docs/docs/reference/versioning.md index 70e2a7e..583fecb 100644 --- a/docs/docs/reference/versioning.md +++ b/docs/docs/reference/versioning.md @@ -113,7 +113,7 @@ play. | respell an argument or return type through an equivalent alias | patch | | add an error to `!` | minor (an old client receives it as `Remote`) | | remove an error from `!` | major (a generated error variant disappears) | -| `@framing` or `@handshake` changes (compared by meaning: writing out the default isn't a change) | major, except a handshake's [policies](wire/handshakes/index.md#hashing-and-changing-them) | +| `@framing` or `@handshake` changes (compared by meaning: writing out the default isn't a change) | major, except a handshake's [policies](wire/handshakes/comline@2.md#hashing-and-changing-them) | | a guarantee given up (`@unverified`, or one more joining the set) | major | | a given-up guarantee restored | minor (a patch for version honesty, which only `comline@2`'s `same_version_differs` gives up today) | | the `@unverified` reason changed | patch | @@ -362,7 +362,7 @@ computed when asked for, not stored. Each format, framing and handshake in play gets blake3 over the CBOR of its kind, canonical name and parameters. The parameters are a framing's `{ name = ..., ... }` entries other than `name`, a handshake's -[wire parameters](wire/handshakes/index.md#hashing-and-changing-them) that aren't +[wire parameters](wire/handshakes/comline@2.md#hashing-and-changing-them) that aren't at their default (its policies never), plus, for an encoding the package declares in `encodings`, its profile's hash. diff --git a/docs/docs/reference/wire/formats/packed@1.md b/docs/docs/reference/wire/formats/packed@1.md index a6bf449..9bbae60 100644 --- a/docs/docs/reference/wire/formats/packed@1.md +++ b/docs/docs/reference/wire/formats/packed@1.md @@ -12,13 +12,13 @@ declared, it's this one. ## Layout | Type | Bytes | -|---|---| -| `bool` | 1 byte: `00` or `01`; anything else is an error | -| `u8` `u16` `u32` `u64` `u128` | 1, 2, 4, 8, 16 bytes, little-endian | -| `s8` `s16` `s32` `s64` `s128` | 1, 2, 4, 8, 16 bytes, little-endian two's complement | -| `f32` `f64` | 4, 8 bytes, IEEE 754 little-endian | -| `string`, `char[max N]`, `char[N]` | length in bytes as a varint, then that many bytes of UTF-8 (invalid UTF-8 is an error). An array of `char` is always text, so `char[N]` sends its length too | -| `char` | its UTF-8 bytes alone, 1 to 4, no length: the first byte says how many follow | +| --- | --- | +| [`#!ids bool`](../../package/schema/types.md#bool) | 1 byte: `00` or `01`; anything else is an error | +| [`#!ids u8 u16 u32 u64 u128`](../../package/schema/types.md#unsigned) | 1, 2, 4, 8, 16 bytes, little-endian | +| [`#!ids s8 s16 s32 s64 s128`](../../package/schema/types.md#signed) | 1, 2, 4, 8, 16 bytes, little-endian two's complement | +| [`#!ids f32 f64`](../../package/schema/types.md#floats) | 4, 8 bytes, IEEE 754 little-endian | +| [`#!ids char`](../../package/schema/types.md#char) | its UTF-8 bytes alone, 1 to 4, no length: the first byte says how many follow | +| [`#!ids string` `#!ids char[max N]` `#!ids char[N]`](../../package/schema/types.md#text-string-and-char) | length in bytes as a varint, then that many bytes of UTF-8 (invalid UTF-8 is an error). An array of `char` is always text, so `char[N]` sends its length too | | `T[]`, `T[max N]` | element count as a varint, then the elements | | `T[N]` | the N elements; the length is in the schema (except `char[N]`, above) | | struct, error body, a call's arguments | the fields in declaration order | @@ -36,34 +36,39 @@ bytes are an error. A length larger than the bytes left is an error before anything is allocated, so a hostile length can't make the decoder reserve memory. -## `char` and bounds +## `char` **A single `char`** is one Unicode scalar value, sent as its UTF-8 bytes with no prefix: `61` for `a`, `c3 a9` for `é`, `e2 82 ac` for `€`, `f0 9f 98 80` for `😀`. The decoder reads the first byte's width and runs the strict UTF-8 check. It refuses: + - a continuation byte or `c0`, `c1`, `f5`..`ff` in first place; - an overlong form; - a surrogate; - a value past U+10FFFF; - a character cut short. -**Bounds** (`char[max N]`, `T[max N]`, `@max_depth`) don't change the bytes, +## Bounds + +On a schema, **bounds** (`#!ids char[max N]`, `#!ids T[max N]`, `#!ids @max_depth`) don't change the bytes, only what a decoder accepts. A value over its bound is refused like malformed input, and an encoder refuses to write one. | Bound | Decoding refuses | -|---|---| -| `T[max N]` | a count above N, before anything is allocated | -| `char[max N]` | a byte length above 4 × N before reading the string, since N characters take at most 4 × N bytes; then, during the UTF-8 check, a string of more than N characters. Characters are counted, never bytes, and never normalized | -| `@max_depth = D` | a value of the recursive type nested more than D deep (the outermost is level 1), before reading any deeper | +| --- | --- | +| `#!ids T[max N]` | a count above N, before anything is allocated | +| `#!ids char[max N]` | a byte length above 4 × N before reading the string, since N characters take at most 4 × N bytes; then, during the UTF-8 check, a string of more than N characters. Characters are counted, never bytes, and never normalized | +| `#!ids @max_depth = D` | a value of the recursive type nested more than D deep (the outermost is level 1), before reading any deeper | -`"é€😀a"` fits `char[max 4]`: 4 characters in 10 bytes. `"ééééé"` doesn't: +`"é€😀a"` fits `#!ids char[max 4]`: 4 characters in 10 bytes. `"ééééé"` doesn't: 5 characters, though only 10 bytes. ## Example -```ids +Given the following snippet + +```ids title="snippet.ids" enum Class { Warrior Mage @@ -79,8 +84,12 @@ struct Player { } ``` +With the following values: + `Player { id: 7, hp: 100, name: "ab", guild: none, class: Mage, pos: [1.0, 2.0] }`: +Is equivalent to: + ```text 07 00 00 00 id 64 00 hp @@ -118,9 +127,11 @@ Changing a package's `wire.format` is itself a major change. - **Rust:** `comline_runtime::format::Packed`, a `WireFormat` built as a serde serializer and deserializer. It needs neither `std` nor an allocator, so it works on targets the JSON and MessagePack formats don't. + - **TypeScript:** `PackedCodec`. A JS value doesn't say whether a number is a `u16` or an `f64`, so the generator emits a `Layout` per type, which the codec walks. `u64`, `s64` and wider are `bigint`. + - **Conformance:** byte-exact vectors in the [runtime's wire corpus](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-packed.json) pin each type's bytes and each decoding error, a single `char`, and each diff --git a/docs/docs/reference/wire/framings/datagram@1.md b/docs/docs/reference/wire/framings/datagram@1.md new file mode 100644 index 0000000..f4a017e --- /dev/null +++ b/docs/docs/reference/wire/framings/datagram@1.md @@ -0,0 +1,27 @@ +# `datagram@1` + +The first, lockstep datagram framing, also spelled `datagram`. A client sends +a request and reads its reply before sending the next, so frames carry no +kind byte and only the client starts a call. + +| Frame | Bytes | +| --- | --- | +| request | call id (`u16` LE), request id (`u64` LE), arguments | +| reply | request id (`u64` LE), [envelope](index.md#the-envelope) | + +- The **call id** is the function's position in its protocol. Both ends + share it because `comline@1` refuses a peer whose IR differs, so the + profile states `calls = by_index`. +- With no kind byte and no protocol number, a connection carries one + protocol, called in one direction. [`datagram@2`](datagram@2.md) adds both. + +For example, a call to function 3, request id 42, with the JSON arguments +`[7,"x"]`, and an ok reply to request 7: + +```text +0300 2a00000000000000 5b372c2278225d +0700000000000000 00 7b22626f6479223a226869227d +``` + +Proven by the `framing-datagram` vectors in the runtime's +[corpus](https://github.com/ComlineProject/runtime/tree/main/runtime/conformance/spec/vectors). diff --git a/docs/docs/reference/wire/framings/datagram@2.md b/docs/docs/reference/wire/framings/datagram@2.md new file mode 100644 index 0000000..619769d --- /dev/null +++ b/docs/docs/reference/wire/framings/datagram@2.md @@ -0,0 +1,12 @@ +# `datagram@2` + +| Frame | Bytes | +| --- | --- | +| request | `00`, protocol (varint), call id (`u16` LE), request id (`u64` LE), arguments | +| reply | `01`, protocol (varint), request id (`u64` LE), envelope | + +- The **call id** is the function's position in its protocol, and the + **protocol** number is the protocol's position in the package. Both ends + share them because `comline@1` refuses a peer whose IR differs. +- Reordering or inserting functions therefore changes what a call + addresses: the profile states `calls = by_index`. diff --git a/docs/docs/reference/wire/framings/datagram@3.md b/docs/docs/reference/wire/framings/datagram@3.md new file mode 100644 index 0000000..32028c5 --- /dev/null +++ b/docs/docs/reference/wire/framings/datagram@3.md @@ -0,0 +1,23 @@ +# `datagram@3` + +| Frame | Bytes | +| --- | --- | +| request | `00`, protocol number (varint), function number (varint), request id (`u64` LE), arguments | +| reply | `01`, protocol number (varint), request id (`u64` LE), envelope (as `datagram@2`) | + +- The numbers are the **wire numbers** core gives each protocol and + function (core#127): unique in their scope, binding from `comline + release`, and never reused once retired. Reordering or inserting + functions doesn't move them, so the profile states `calls = by_id`. +- A reply is byte for byte a `datagram@2` reply. +- It's meant for [`comline@2`](../handshakes/comline@2.md), which checks + compatibility by those same numbers. + +For example, a call to function 3 of protocol 0, request id 42, with the +JSON arguments `[7,"x"]`: + +```text +00 00 03 2a00000000000000 5b372c2278225d +``` + +Function 300 of protocol 300 is `00 ac02 ac02 …`, and 65535 is `ff ff 03`. diff --git a/docs/docs/reference/wire/framings/index.md b/docs/docs/reference/wire/framings/index.md index 9ebdd65..39f7c91 100644 --- a/docs/docs/reference/wire/framings/index.md +++ b/docs/docs/reference/wire/framings/index.md @@ -18,43 +18,6 @@ byte, low bits first, high bit set on every byte but the last. A protocol or function number is a `u16`, so it takes 1 to 3 bytes. Only the shortest form is accepted: `80 00` for 0 is refused, and so is a value above 65535. -## `datagram@2` - -| Frame | Bytes | -| --- | --- | -| request | `00`, protocol (varint), call id (`u16` LE), request id (`u64` LE), arguments | -| reply | `01`, protocol (varint), request id (`u64` LE), envelope | - -- The **call id** is the function's position in its protocol, and the - **protocol** number is the protocol's position in the package. Both ends - share them because `comline@1` refuses a peer whose IR differs. -- Reordering or inserting functions therefore changes what a call - addresses: the profile states `calls = by_index`. - -## `datagram@3` - -| Frame | Bytes | -| --- | --- | -| request | `00`, protocol number (varint), function number (varint), request id (`u64` LE), arguments | -| reply | `01`, protocol number (varint), request id (`u64` LE), envelope (as `datagram@2`) | - -- The numbers are the **wire numbers** core gives each protocol and - function (core#127): unique in their scope, binding from `comline - release`, and never reused once retired. Reordering or inserting - functions doesn't move them, so the profile states `calls = by_id`. -- A reply is byte for byte a `datagram@2` reply. -- It's meant for [`comline@2`](../handshakes/index.md#comline2), which checks - compatibility by those same numbers. - -For example, a call to function 3 of protocol 0, request id 42, with the -JSON arguments `[7,"x"]`: - -```text -00 00 03 2a00000000000000 5b372c2278225d -``` - -Function 300 of protocol 300 is `00 ac02 ac02 …`, and 65535 is `ff ff 03`. - ## The envelope A reply's outcome, the same in both framings: diff --git a/docs/docs/reference/wire/handshakes/comline@1.md b/docs/docs/reference/wire/handshakes/comline@1.md index e69de29..27243b4 100644 --- a/docs/docs/reference/wire/handshakes/comline@1.md +++ b/docs/docs/reference/wire/handshakes/comline@1.md @@ -0,0 +1,17 @@ +# `comline@1` + +Each end sends one fixed 31-byte frame and refuses unless the peer's +matches: + +| Bytes | Field | +| --- | --- | +| 2 | magic, `"CO"` | +| 1 | version, `01` | +| 8 | the schema fingerprint (IR hash), `u64` LE | +| 8 | the format's name hash (FNV-1a 64), `u64` LE | +| 8 | the framing's name hash, `u64` LE | +| 4 | capability bits, `u32` LE (advisory) | + +Any edit to the schema changes the fingerprint, so the two ends must have +been built from the same schema. + diff --git a/docs/docs/reference/wire/handshakes/comline@2.md b/docs/docs/reference/wire/handshakes/comline@2.md index e69de29..fbf7f9f 100644 --- a/docs/docs/reference/wire/handshakes/comline@2.md +++ b/docs/docs/reference/wire/handshakes/comline@2.md @@ -0,0 +1,179 @@ +# `comline@2` + +`comline@2` compares what each version's history records +([core#119](https://github.com/ComlineProject/core/issues/119)), not the +whole IR: + +- the **package id** (`id` in `config.idp`); +- the **package version**; +- for each protocol: its **wire number**, its **line** (the release at + which it last broke) and its **wire-shape hash**; +- for each function: its **wire number**; +- for each encoding (format, framing, the handshake itself): its + **parameter hash**, which covers its name and parameters. + +The namespace isn't compared, so renaming one doesn't stop peers from +connecting. + +## What decides + +1. Different package ids: refused. +2. Different encodings (format, framing or handshake, by parameter hash): + refused. +3. Per protocol both ends have: + - **Lines differ:** the protocol broke between the two versions, so it's + refused. A line moves at a major release from 1.0, and at a minor + release while the package is 0.x. + - **Same package version, but the shape hash differs:** one end isn't + what its version claims. `same_version_differs` decides. + - **The consumer has functions the provider lacks**, by function number: + `newer_consumer` decides. +4. A protocol only one end has is `unknown_protocol`, refused on its own. + +## Parameters + +Written in `@handshake` or `handshake`'s dictionary form +([usage resolution](index.md#usage-resolution)), each value a string or a bare word in `@handshake`, +and a string in `config.idp`. Defaults first: + +| Parameter | Values | Kind | Meaning | +| --- | --- | --- | --- | +| `scope` | `protocol` \| `package` | policy | A protocol's refusal refuses that protocol alone, or the whole connection. `unknown_protocol` never refuses more than its protocol. | +| `newer_consumer` | `degrade` \| `refuse` | policy | `degrade` connects, and calls to the functions the peer lacks fail locally without reaching the wire. `refuse` refuses the protocol. | +| `same_version_differs` | `refuse` \| `warn` | policy | Refuse the protocol, or connect and report a warning. `warn` gives up version honesty. | +| `exchange` | `offer` \| `symmetric` | wire | Who sends what (below). Both ends must agree: it's part of the handshake's parameter hash and of every hello. | + +Each end applies its own policies, and either end's refusal counts. With +`scope = protocol`, a connection whose every protocol is refused is +refused. + +An unknown parameter or value is a build error, and so is any parameter on a +handshake that takes none: `comline@1`, `none`, and for now a handshake a +package declares, whose profile can't state parameters. + +### Hashing and changing them + +Only `exchange` is in the handshake's parameter hash. The policies aren't, so +two ends with different policies still agree. A parameter written as its default +is the same as one left out, in the hash and in the comparison of two versions: + +| Change | Version bump | Line | +| --- | --- | --- | +| `exchange` | major | moves | +| `scope` or `newer_consumer`, either way | patch | stays | +| `same_version_differs`, `warn` to `refuse` | patch | stays | +| `same_version_differs`, `refuse` to `warn` | major | stays | +| writing a default out, or dropping it | none | stays | + +`same_version_differs = warn` gives up **version honesty**: two peers of the same +version may turn out to have different shapes. As with +[any guarantee given up](../../../guides/idl/protocol.md#giving-up-a-guarantee), it +needs `@unverified` on the protocol to build, and each consumer accepts it. The +major bump tells consumers so, but nothing on the wire changed, so the line stays +and peers on it keep connecting. + +## The exchange + +=== "offer" + + ```mermaid + sequenceDiagram + participant C as Initiator (consumer) + participant S as Responder (provider) + C->>S: Hello: package, encodings, protocols + S->>C: Answer: per protocol, accepted (+ missing functions) or refused + C->>S: Verdict: accept (+ protocols it refuses) or refuse + ``` + +=== "symmetric" + + ```mermaid + sequenceDiagram + participant C as Initiator + participant S as Responder + C->>S: Hello + S->>C: Hello + C->>S: Verdict + S->>C: Verdict + ``` + +There is one handshake per connection. A refusal ends it: the refusing end +sends its refusal and stops. + +## Messages + +Every message starts with `"CO"`, the version byte `02` and a type byte. +Numbers and counts are shortest-form varints, a hash is 16 bytes (the +first 128 bits of core's blake3 hash), and a release is three varints: +major, minor, patch. Bytes after a message's last field make it malformed. + +**Hello** (type `01`): + +| Field | Bytes | +| --- | --- | +| `exchange` | 1: `00` offer, `01` symmetric | +| package id | 16 | +| package version | a release, then the pre-release (`dev.2`): length varint + UTF-8, `00` for none | +| encodings | count, then each: kind (1: `00` format, `01` framing, `02` handshake) + parameter hash (16), ascending | +| protocols | count, then each: number, line (a release), shape hash (16), function numbers (count + numbers, ascending), ascending by number | + +**Answer** (type `02`, from the responder under `offer`): + +- `00` accept: the responder's package version, then a count and, for every + offered protocol in the offer's order, its number and a status: + - `00` accepted, or `01` accepted with the shape hash differing, each + followed by the offered functions the responder lacks (count + + numbers); + - `02` refused, followed by the reason code. +- `01` refuse, followed by a refusal. + +**Verdict** (type `03`): + +- `00` accept, followed by the protocols this end refuses: count, then each + number + reason code, ascending; +- `01` refuse, followed by a refusal. + +**Refusal:** the reason code (1 byte), the refusing side (1 byte: `00` +initiator, `01` responder), then the protocol number when the reason is +about one protocol. + +For example, an initiator refusing the connection over protocol 1's line: + +```text +434f 02 03 01 06 00 01 +"CO" v2 verdict refuse line_mismatch initiator protocol 1 +``` + +## Reason codes + +Stable: a code is never renumbered or reused. + +| Code | Reason | About | When | +| --- | --- | --- | --- | +| 1 | `malformed` | connection | The peer's message didn't parse. | +| 2 | `handshake_version` | connection | The peer speaks another handshake version, such as a `comline@1` frame. | +| 3 | `exchange_mismatch` | connection | The two ends declare different `exchange` values. | +| 4 | `package_mismatch` | connection | The package ids differ. | +| 5 | `encoding_mismatch` | connection | The encodings' parameter hashes differ. | +| 6 | `line_mismatch` | a protocol | The protocol's lines differ. | +| 7 | `same_version_differs` | a protocol | The versions are equal but the shape hash differs, under `same_version_differs = refuse`. | +| 8 | `newer_consumer` | a protocol | The consumer has functions the provider lacks, under `newer_consumer = refuse`. | +| 9 | `unknown_protocol` | a protocol | Only one end has the protocol. | + +In the Rust runtime, a refusal is `RuntimeError::Refused { reason, side }` +on both ends. After a successful handshake, `Negotiated::check_call` +returns `RuntimeError::Unavailable` for a call the peer can't take, either +because it lacks the function or because the protocol was refused. + +## Proven by + +The `handshake-comline2` vectors in the runtime's corpus +([vectors](https://github.com/ComlineProject/runtime/tree/main/runtime/conformance/vectors)): + +- every message type, with byte-exact encodings and malformed variants; +- every step of both exchanges under every parameter value; +- a case for every reason code. + +`comline@2`'s profile states `schema_agreement` and `encoding_agreement`, +proven by an accepted peer and by peers refused for a moved line, another +package, other encodings or another exchange. diff --git a/docs/docs/reference/wire/handshakes/index.md b/docs/docs/reference/wire/handshakes/index.md index b84ff37..63d2491 100644 --- a/docs/docs/reference/wire/handshakes/index.md +++ b/docs/docs/reference/wire/handshakes/index.md @@ -33,7 +33,7 @@ A protocol's `@handshake` is picked first, then its package's `handshake` key in protocol Gateway { /* ... */ } ``` -A handshake that takes [parameters](#parameters) is written as a dictionary +A handshake that takes [parameters](comline@2.md#parameters) is written as a dictionary instead, its name under `name`, as `@framing` does. The plain name is shorthand for a dictionary with only a name: @@ -49,203 +49,6 @@ handshake = { name = "comline@2" scope = "package" } A protocol's own `@handshake` replaces the package's whole: the two aren't merged. -## `comline@1` - -Each end sends one fixed 31-byte frame and refuses unless the peer's -matches: - -| Bytes | Field | -| --- | --- | -| 2 | magic, `"CO"` | -| 1 | version, `01` | -| 8 | the schema fingerprint (IR hash), `u64` LE | -| 8 | the format's name hash (FNV-1a 64), `u64` LE | -| 8 | the framing's name hash, `u64` LE | -| 4 | capability bits, `u32` LE (advisory) | - -Any edit to the schema changes the fingerprint, so the two ends must have -been built from the same schema. - -## `comline@2` - -`comline@2` compares what each version's history records -([core#119](https://github.com/ComlineProject/core/issues/119)), not the -whole IR: - -- the **package id** (`id` in `config.idp`); -- the **package version**; -- for each protocol: its **wire number**, its **line** (the release at - which it last broke) and its **wire-shape hash**; -- for each function: its **wire number**; -- for each encoding (format, framing, the handshake itself): its - **parameter hash**, which covers its name and parameters. - -The namespace isn't compared, so renaming one doesn't stop peers from -connecting. - -### What decides - -1. Different package ids: refused. -2. Different encodings (format, framing or handshake, by parameter hash): - refused. -3. Per protocol both ends have: - - **Lines differ:** the protocol broke between the two versions, so it's - refused. A line moves at a major release from 1.0, and at a minor - release while the package is 0.x. - - **Same package version, but the shape hash differs:** one end isn't - what its version claims. `same_version_differs` decides. - - **The consumer has functions the provider lacks**, by function number: - `newer_consumer` decides. -4. A protocol only one end has is `unknown_protocol`, refused on its own. - -### Parameters - -Written in `@handshake` or `handshake`'s dictionary form -([above](#usage-resolution)), each value a string or a bare word in `@handshake`, -and a string in `config.idp`. Defaults first: - -| Parameter | Values | Kind | Meaning | -| --- | --- | --- | --- | -| `scope` | `protocol` \| `package` | policy | A protocol's refusal refuses that protocol alone, or the whole connection. `unknown_protocol` never refuses more than its protocol. | -| `newer_consumer` | `degrade` \| `refuse` | policy | `degrade` connects, and calls to the functions the peer lacks fail locally without reaching the wire. `refuse` refuses the protocol. | -| `same_version_differs` | `refuse` \| `warn` | policy | Refuse the protocol, or connect and report a warning. `warn` gives up version honesty. | -| `exchange` | `offer` \| `symmetric` | wire | Who sends what (below). Both ends must agree: it's part of the handshake's parameter hash and of every hello. | - -Each end applies its own policies, and either end's refusal counts. With -`scope = protocol`, a connection whose every protocol is refused is -refused. - -An unknown parameter or value is a build error, and so is any parameter on a -handshake that takes none: `comline@1`, `none`, and for now a handshake a -package declares, whose profile can't state parameters. - -#### Hashing and changing them - -Only `exchange` is in the handshake's parameter hash. The policies aren't, so -two ends with different policies still agree. A parameter written as its default -is the same as one left out, in the hash and in the comparison of two versions: - -| Change | Version bump | Line | -| --- | --- | --- | -| `exchange` | major | moves | -| `scope` or `newer_consumer`, either way | patch | stays | -| `same_version_differs`, `warn` to `refuse` | patch | stays | -| `same_version_differs`, `refuse` to `warn` | major | stays | -| writing a default out, or dropping it | none | stays | - -`same_version_differs = warn` gives up **version honesty**: two peers of the same -version may turn out to have different shapes. As with -[any guarantee given up](../../../guides/idl/protocol.md#giving-up-a-guarantee), it -needs `@unverified` on the protocol to build, and each consumer accepts it. The -major bump tells consumers so, but nothing on the wire changed, so the line stays -and peers on it keep connecting. - -### The exchange - -=== "offer" - - ```mermaid - sequenceDiagram - participant C as Initiator (consumer) - participant S as Responder (provider) - C->>S: Hello: package, encodings, protocols - S->>C: Answer: per protocol, accepted (+ missing functions) or refused - C->>S: Verdict: accept (+ protocols it refuses) or refuse - ``` - -=== "symmetric" - - ```mermaid - sequenceDiagram - participant C as Initiator - participant S as Responder - C->>S: Hello - S->>C: Hello - C->>S: Verdict - S->>C: Verdict - ``` - -There is one handshake per connection. A refusal ends it: the refusing end -sends its refusal and stops. - -### Messages - -Every message starts with `"CO"`, the version byte `02` and a type byte. -Numbers and counts are shortest-form varints, a hash is 16 bytes (the -first 128 bits of core's blake3 hash), and a release is three varints: -major, minor, patch. Bytes after a message's last field make it malformed. - -**Hello** (type `01`): - -| Field | Bytes | -| --- | --- | -| `exchange` | 1: `00` offer, `01` symmetric | -| package id | 16 | -| package version | a release, then the pre-release (`dev.2`): length varint + UTF-8, `00` for none | -| encodings | count, then each: kind (1: `00` format, `01` framing, `02` handshake) + parameter hash (16), ascending | -| protocols | count, then each: number, line (a release), shape hash (16), function numbers (count + numbers, ascending), ascending by number | - -**Answer** (type `02`, from the responder under `offer`): - -- `00` accept: the responder's package version, then a count and, for every - offered protocol in the offer's order, its number and a status: - - `00` accepted, or `01` accepted with the shape hash differing, each - followed by the offered functions the responder lacks (count + - numbers); - - `02` refused, followed by the reason code. -- `01` refuse, followed by a refusal. - -**Verdict** (type `03`): - -- `00` accept, followed by the protocols this end refuses: count, then each - number + reason code, ascending; -- `01` refuse, followed by a refusal. - -**Refusal:** the reason code (1 byte), the refusing side (1 byte: `00` -initiator, `01` responder), then the protocol number when the reason is -about one protocol. - -For example, an initiator refusing the connection over protocol 1's line: - -```text -434f 02 03 01 06 00 01 -"CO" v2 verdict refuse line_mismatch initiator protocol 1 -``` - -### Reason codes - -Stable: a code is never renumbered or reused. - -| Code | Reason | About | When | -| --- | --- | --- | --- | -| 1 | `malformed` | connection | The peer's message didn't parse. | -| 2 | `handshake_version` | connection | The peer speaks another handshake version, such as a `comline@1` frame. | -| 3 | `exchange_mismatch` | connection | The two ends declare different `exchange` values. | -| 4 | `package_mismatch` | connection | The package ids differ. | -| 5 | `encoding_mismatch` | connection | The encodings' parameter hashes differ. | -| 6 | `line_mismatch` | a protocol | The protocol's lines differ. | -| 7 | `same_version_differs` | a protocol | The versions are equal but the shape hash differs, under `same_version_differs = refuse`. | -| 8 | `newer_consumer` | a protocol | The consumer has functions the provider lacks, under `newer_consumer = refuse`. | -| 9 | `unknown_protocol` | a protocol | Only one end has the protocol. | - -In the Rust runtime, a refusal is `RuntimeError::Refused { reason, side }` -on both ends. After a successful handshake, `Negotiated::check_call` -returns `RuntimeError::Unavailable` for a call the peer can't take, either -because it lacks the function or because the protocol was refused. - -### Proven by - -The `handshake-comline2` vectors in the runtime's corpus -([vectors](https://github.com/ComlineProject/runtime/tree/main/runtime/conformance/vectors)): - -- every message type, with byte-exact encodings and malformed variants; -- every step of both exchanges under every parameter value; -- a case for every reason code. - -`comline@2`'s profile states `schema_agreement` and `encoding_agreement`, -proven by an accepted peer and by peers refused for a moved line, another -package, other encodings or another exchange. - ## Defaults before `comline@2` Until `comline@2` became the default, a protocol that named no handshake ran @@ -268,4 +71,3 @@ handshake = "comline@1" A package that states them sees no change: not in its IR, its facts or its version. Either can also be stated on its own, per protocol, with `@framing` and `@handshake`. - diff --git a/docs/docs/reference/wire/official-encodings.md b/docs/docs/reference/wire/official-encodings.md index 6a15e71..fcb0bc8 100644 --- a/docs/docs/reference/wire/official-encodings.md +++ b/docs/docs/reference/wire/official-encodings.md @@ -66,7 +66,7 @@ to write it. | Format | One `char` | `T[max N]` | `char[max N]` | | --- | --- | --- | --- | -| [`packed@1`](formats/packed@1.md#char-and-bounds) | its UTF-8 bytes, 1 to 4, no prefix, checked by the strict UTF-8 check | a count prefix above N refused before allocating | a byte length above 4 × N refused before reading; then counted exactly during the UTF-8 check | +| [`packed@1`](formats/packed@1.md#char) | its UTF-8 bytes, 1 to 4, no prefix, checked by the strict UTF-8 check | a count prefix above N refused before allocating | a byte length above 4 × N refused before reading; then counted exactly during the UTF-8 check | | `msgpack@1` | a one-character string (`fixstr`, 2 to 5 bytes) | a count prefix above N refused before allocating | a byte length above 4 × N refused; then counted exactly | | `json@1` | a one-character string, `"a"` | refused at item N + 1, which isn't built | counted after unescaping; the frame limit caps the allocation | diff --git a/docs/docs/resources.md b/docs/docs/resources.md index 2c27214..d37db58 100755 --- a/docs/docs/resources.md +++ b/docs/docs/resources.md @@ -3,7 +3,7 @@ ## Comline repositories | Repo | What | -|---|---| +| --- | --- | | [`ComlineProject/core`](https://github.com/ComlineProject/core) | the compiler, IR, `.comline/` store, diff & auto-versioning | | [`ComlineProject/cli`](https://github.com/ComlineProject/cli) | the `comline` CLI — `build`, `generation`, `diff`, `generation clean`, `reset` ([full guide](https://github.com/ComlineProject/cli/blob/main/docs/cli.md)) | | [`ComlineProject/comline`](https://github.com/ComlineProject/comline) | runtime experiments | @@ -33,7 +33,6 @@ Systems that Comline is inspired, compares against or borrows from: - [MessagePack](https://msgpack.org/) — a compact binary serialization Comline can target - ## Background - [Git Internals — Git Objects](https://git-scm.com/book/en/v2/Git-Internals-Git-Objects) diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 746ada9..232b3fb 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -55,6 +55,8 @@ nav: - Client & server: 'languages/typescript/client-and-server.md' - Framing & wire format: 'languages/typescript/framing-and-wire-format.md' - Connections: 'languages/typescript/connections.md' + + - C: 'languages/c/index.md' - Developer Guides: - 'developer/index.md' @@ -70,20 +72,40 @@ nav: - Reference: - "reference/index.md" + - Package: + - "reference/package/index.md" + - 'reference/package/idp.md' + - 'reference/package/comline-toml.md' + + - Schema: + - 'reference/package/schema/index.md' + - Types: 'reference/package/schema/types.md' + + # std:start + - Standard library: + - std: 'reference/package/schema/std/index.md' + - http: 'reference/package/schema/std/http.md' + - validators: 'reference/package/schema/std/validators.md' + # std:end + - Languages: - 'reference/languages/index.md' - Rust: 'reference/languages/rust.md' - TypeScript: 'reference/languages/typescript.md' + - C ABI: 'reference/languages/c-abi.md' - Wire: - 'reference/wire/index.md' - Framings: - 'reference/wire/framings/index.md' - #- datagram@2: 'reference/wire/framings/datagram.md' + - datagram@3: 'reference/wire/framings/datagram@3.md' + - datagram@2: 'reference/wire/framings/datagram@2.md' + - datagram@1: 'reference/wire/framings/datagram@1.md' - Handshakes: - 'reference/wire/handshakes/index.md' - #- handshake@1: '' + - comline@2: 'reference/wire/handshakes/comline@2.md' + - comline@1: 'reference/wire/handshakes/comline@1.md' - Formats: - 'reference/wire/formats/index.md' @@ -93,23 +115,6 @@ nav: - Official encodings: 'reference/wire/official-encodings.md' - - Runtime: - - C ABI: 'reference/c-abi.md' - - - Package: - - 'reference/package/congregation/index.md' - - config.idp: 'reference/package/congregation/config-id.md' - - comline.toml: 'reference/package/congregation/comline-toml.md' - - - Schema: - - 'reference/schema/index.md' - # std:start - - Standard library: - - std: 'reference/schema/std/index.md' - - http: 'reference/schema/std/http.md' - - validators: 'reference/schema/std/validators.md' - # std:end - - Tools: - Tools: 'reference/tools/index.md' - CLI: diff --git a/scripts/check-idl-drift.py b/scripts/check-idl-drift.py index c4c767e..4d177ea 100755 --- a/scripts/check-idl-drift.py +++ b/scripts/check-idl-drift.py @@ -36,7 +36,7 @@ # Each grammar and the docs that are expected to cover it. # # `core/src/package/config/idl/grammar.rs` (the `config.idp` grammar) is not -# listed yet: `reference/package/congregation/config-id.md` is still an empty stub, so +# listed yet: `reference/package/idp.md` is still an empty stub, so # checking it would report `false` as undocumented every run for a reason that # has nothing to do with drift — and a check that is always red gets ignored. # Add it here once that page is written; it is only three keywords diff --git a/scripts/sync-std-docs.py b/scripts/sync-std-docs.py index 8f424b4..e489f26 100755 --- a/scripts/sync-std-docs.py +++ b/scripts/sync-std-docs.py @@ -3,15 +3,15 @@ `std` is the package embedded in core, at `core_stdlib/packages/std/`. Its `//!` headers and `///` docstrings are the documentation; `comline doc` turns -them into the pages under `docs/docs/reference/schema/std/`, and their `nav` into the block +them into the pages under `docs/docs/reference/package/schema/std/`, and their `nav` into the block between the `std:start` and `std:end` markers in `docs/mkdocs.yml`. Nothing -under `docs/docs/reference/schema/std/` is written by hand: edit std's docstrings in core and +under `docs/docs/reference/package/schema/std/` is written by hand: edit std's docstrings in core and re-run this. -Other docs link here (`https://comlineproject.github.io/docs/reference/schema/std/http/#httpmethod` +Other docs link here (`https://comlineproject.github.io/docs/reference/package/schema/std/http/#httpmethod` is what `[doc] links = { std = ... }` produces), so the pages' paths are part -of the site's contract: `reference/schema/std/.md` publishes at -`/reference/schema/std//`. +of the site's contract: `reference/package/schema/std/.md` publishes at +`/reference/package/schema/std//`. Needs `comline` on PATH (or `--comline`) and `git` when no `--checkout` is given. The docs CI has no Rust toolchain by default, so the drift workflow @@ -34,14 +34,14 @@ from pathlib import Path ROOT = Path(__file__).resolve().parent.parent -PAGES = ROOT / "docs/docs/reference/schema/std" +PAGES = ROOT / "docs/docs/reference/package/schema/std" MKDOCS = ROOT / "docs/mkdocs.yml" CORE = "https://github.com/ComlineProject/core" PACKAGE = "core_stdlib/packages/std" START = "# std:start" END = "# std:end" # Where the block sits in mkdocs.yml, so its entries are indented to match. -INDENT = " " +INDENT = " " # ` - "http": http.md` NAV_RE = re.compile(r'^\s*-\s*"(?P[^"]+)":\s*(?P<path>\S+)\s*$') @@ -76,7 +76,7 @@ def nav_block(nav_yml: Path) -> str: for line in nav_yml.read_text(encoding="utf-8").splitlines(): match = NAV_RE.match(line) if match: - lines.append(f"{INDENT} - {match['title']}: 'reference/schema/std/{match['path']}'") + lines.append(f"{INDENT} - {match['title']}: 'reference/package/schema/std/{match['path']}'") lines.append(f"{INDENT}{END}") return "\n".join(lines)