From 1fe79c434882600325b0d466fd616c096ce1f781 Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 9 Nov 2026 12:47:57 +0800 Subject: [PATCH 1/3] Reference: a Package section, Conformance in its own section, the Observatory page - reference/congregation moves under reference/package (config.idp and comline.toml), with the schema reference beside it. - reference/conformance.md splits into reference/conformance/: the pipeline, wire and generator conformance, fuzzing, benchmarks, timings, size and coverage, and reports and the Observatory. - reference/tools/cli.md becomes reference/tools/cli/ beside its subcommand pages; a new reference/tools/observatory.md. - Links across the guides, the developer pages and the reference follow the moves; the drift workflows and scripts point at the new paths. --- .github/workflows/cli-drift.yml | 2 +- .github/workflows/std-drift.yml | 4 +- docs/docs/conduct/readiness.md | 4 +- docs/docs/design/validators.md | 4 +- docs/docs/developer/extending/encodings.md | 4 +- docs/docs/developer/extending/handshakes.md | 4 +- docs/docs/developer/extending/index.md | 9 +- docs/docs/developer/extending/transports.md | 2 +- docs/docs/developer/index.md | 7 - docs/docs/getting-started.md | 24 ++- docs/docs/guides/codegen/generating-code.md | 4 +- docs/docs/guides/codegen/index.md | 2 +- .../docs/guides/codegen/library-generation.md | 6 +- docs/docs/guides/documenting.md | 8 +- docs/docs/guides/idl/index.md | 5 +- docs/docs/guides/idl/types.md | 22 +-- docs/docs/guides/idl/use.md | 6 +- docs/docs/guides/idl/validator.md | 6 +- docs/docs/guides/ir/cas.md | 32 ++-- docs/docs/guides/packages/index.md | 24 +-- docs/docs/guides/runtime/index.md | 2 +- docs/docs/languages/rust/connections.md | 2 +- docs/docs/languages/rust/index.md | 2 +- docs/docs/proposals/auto-documentation.md | 8 +- docs/docs/proposals/extensibility.md | 2 +- docs/docs/reference/conformance/benchmarks.md | 117 +++++++++++ docs/docs/reference/conformance/fuzzing.md | 93 +++++++++ docs/docs/reference/conformance/generators.md | 51 +++++ docs/docs/reference/conformance/index.md | 59 ++++++ docs/docs/reference/conformance/reports.md | 99 ++++++++++ docs/docs/reference/conformance/timings.md | 80 ++++++++ .../{conformance.md => conformance/wire.md} | 181 +++++++++++++----- docs/docs/reference/generator-protocol.md | 4 +- docs/docs/reference/languages/index.md | 10 +- .../congregation/comline-toml.md | 26 +-- .../{ => package}/congregation/config-id.md | 24 +-- .../{ => package}/congregation/index.md | 4 +- docs/docs/reference/package/index.md | 0 .../reference/{ => package}/schema/index.md | 0 .../{ => package}/schema/std/http.md | 0 .../{ => package}/schema/std/index.md | 0 .../{ => package}/schema/std/validators.md | 0 docs/docs/reference/tools/cli/doc.md | 6 +- .../reference/tools/{cli.md => cli/index.md} | 38 ++-- .../tools/editor-extensions/vscode.md | 4 + docs/docs/reference/tools/index.md | 4 +- docs/docs/reference/tools/language-server.md | 3 + docs/docs/reference/tools/observatory.md | 4 + docs/docs/reference/versioning.md | 20 +- docs/docs/reference/wire/formats/json@1.md | 1 + docs/docs/reference/wire/formats/msgpack@1.md | 1 + docs/docs/reference/wire/handshakes/index.md | 2 +- docs/docs/stylesheets/nav.css | 14 ++ docs/mkdocs.yml | 31 +-- scripts/check-cli-drift.py | 4 +- scripts/check-idl-drift.py | 2 +- scripts/sync-std-docs.py | 14 +- 57 files changed, 848 insertions(+), 243 deletions(-) create mode 100644 docs/docs/reference/conformance/benchmarks.md create mode 100644 docs/docs/reference/conformance/fuzzing.md create mode 100644 docs/docs/reference/conformance/generators.md create mode 100644 docs/docs/reference/conformance/index.md create mode 100644 docs/docs/reference/conformance/reports.md create mode 100644 docs/docs/reference/conformance/timings.md rename docs/docs/reference/{conformance.md => conformance/wire.md} (54%) rename docs/docs/reference/{ => package}/congregation/comline-toml.md (92%) rename docs/docs/reference/{ => package}/congregation/config-id.md (70%) rename docs/docs/reference/{ => package}/congregation/index.md (91%) create mode 100644 docs/docs/reference/package/index.md rename docs/docs/reference/{ => package}/schema/index.md (100%) rename docs/docs/reference/{ => package}/schema/std/http.md (100%) rename docs/docs/reference/{ => package}/schema/std/index.md (100%) rename docs/docs/reference/{ => package}/schema/std/validators.md (100%) rename docs/docs/reference/tools/{cli.md => cli/index.md} (60%) create mode 100644 docs/docs/reference/tools/observatory.md diff --git a/.github/workflows/cli-drift.yml b/.github/workflows/cli-drift.yml index 58f726b..9c39e82 100644 --- a/.github/workflows/cli-drift.yml +++ b/.github/workflows/cli-drift.yml @@ -16,7 +16,7 @@ on: - cron: '0 7 * * 1' pull_request: paths: - - 'docs/docs/reference/tools/cli.md' + - 'docs/docs/reference/tools/cli/index.md' - 'scripts/check-cli-drift.py' - '.github/workflows/cli-drift.yml' workflow_dispatch: diff --git a/.github/workflows/std-drift.yml b/.github/workflows/std-drift.yml index 4905e48..a5cf97e 100644 --- a/.github/workflows/std-drift.yml +++ b/.github/workflows/std-drift.yml @@ -1,6 +1,6 @@ name: Std drift -# The pages under `docs/docs/reference/schema/std/` are not written by hand: they are `comline +# The pages under `docs/docs/reference/package/schema/std/` are not written by hand: they are `comline # doc` run over the standard library package core embeds # (`core_stdlib/packages/std`), and the std block of `docs/mkdocs.yml` is its # generated nav. This regenerates both and fails on a difference, so a @@ -19,7 +19,7 @@ on: - cron: '15 7 * * 1' pull_request: paths: - - 'docs/docs/reference/schema/std/**' + - 'docs/docs/reference/package/schema/std/**' - 'docs/mkdocs.yml' - 'scripts/sync-std-docs.py' - '.github/workflows/std-drift.yml' diff --git a/docs/docs/conduct/readiness.md b/docs/docs/conduct/readiness.md index 4cf6c2a..8e4baba 100644 --- a/docs/docs/conduct/readiness.md +++ b/docs/docs/conduct/readiness.md @@ -80,11 +80,11 @@ Semver for a user's schema package means nothing until a break is defined. reviewed and numbered for 1.0: - the [generator protocol](../reference/generator-protocol.md), including its render request (version 0 today); - - encoding profiles and the [conformance](../reference/conformance.md) + - encoding profiles and the [conformance](../reference/conformance/index.md) corpus and ops protocol; - the documentation model behind [`comline doc`](../reference/tools/cli/doc.md) (version 1 today); - - the [`comline.toml` schema](../reference/congregation/comline-toml.md) + - the [`comline.toml` schema](../reference/package/congregation/comline-toml.md) (`0.x` today); - the JSON from [`comline inspect --json`](../reference/tools/cli/inspect.md). - The extension API is part of 1.0, so it is designed, built and frozen diff --git a/docs/docs/design/validators.md b/docs/docs/design/validators.md index 492b9aa..989901e 100644 --- a/docs/docs/design/validators.md +++ b/docs/docs/design/validators.md @@ -257,7 +257,7 @@ left to right, parenthesising each comparison. Whether decode runs the checks is the **consumer's** call, not the schema's — it changes the shape and cost of generated decode (a clone per decode, no "parse now, validate later"), which is a codegen concern. So it's a `[generate]` key in -the consumer's [`comline.toml`](../reference/congregation/comline-toml.md), overridable per +the consumer's [`comline.toml`](../reference/package/congregation/comline-toml.md), overridable per `[[generate.target]]`, slotting into the existing `defaults → [generate] → [[generate.target]] → env → CLI` precedence: @@ -279,7 +279,7 @@ validate_on_decode = "collect" in its `TryFrom`); `"off"` leaves decode untouched. - Matching `COMLINE_GENERATE_VALIDATE_ON_DECODE` env var and `--validate-on-decode` flag, like the other `[generate]` keys. -- Folds into [`comline.toml`](../reference/congregation/comline-toml.md) when 4c ships. +- Folds into [`comline.toml`](../reference/package/congregation/comline-toml.md) when 4c ships. ### Decided diff --git a/docs/docs/developer/extending/encodings.md b/docs/docs/developer/extending/encodings.md index e897e35..36663c6 100644 --- a/docs/docs/developer/extending/encodings.md +++ b/docs/docs/developer/extending/encodings.md @@ -129,7 +129,7 @@ path with `.json` replaced by `.vectors.json`, so `profiles/lines.json` has - Every `proven_by` entry in the profile names a case in this file. The case format is a public, versioned spec: see the -[conformance reference](../../reference/conformance.md) for the ops, the value +[conformance reference](../../reference/conformance/wire.md) for the ops, the value encoding and the rules a case is judged by. To run it, write a small program that speaks the **ops protocol**: one JSON @@ -144,7 +144,7 @@ The report gives, per encoding, how many cases pass, whether each profile fact is **proven** (every case it names passes and shows its value), and a byte diff for each failed encode. A failed or unimplemented case, or an unproven fact, exits with `1`. The flags, limits and report are in the -[CLI reference](../../reference/tools/cli.md#commands). +[CLI reference](../../reference/tools/cli/index.md#commands). - `comline conformance` runs every encoding the package declares. An encoding with no vectors file is an error, so name the ones you mean with diff --git a/docs/docs/developer/extending/handshakes.md b/docs/docs/developer/extending/handshakes.md index c966b15..d82dc40 100644 --- a/docs/docs/developer/extending/handshakes.md +++ b/docs/docs/developer/extending/handshakes.md @@ -164,7 +164,7 @@ A fact stated `true` is **proven by vectors**: cases of the decode-only op agrees, which the handshake accepts. For each thing the fact covers there is a peer frame that differs in only that, which it refuses: the IR hash for `schema_agreement`, the format and the framing for `encoding_agreement`. See -the [`handshake_check` op](../../reference/conformance.md#ops). A fact stated +the [`handshake_check` op](../../reference/conformance/wire.md#ops). A fact stated `false` claims nothing, so it names no cases. A protocol **gives up every agreement its handshake's profile doesn't @@ -191,7 +191,7 @@ gives up schema agreement (handshake `com.acme.hello@1` doesn't establish it) For a handshake that states nothing, the `@unverified` marker is your word that something else covers both. -```ids +```ids title="giving_up_guarantee.ids" @handshake = "com.acme.hello@1" @unverified = "the gateway checks the schema itself in its HELLO reply" protocol Gateway { /* ... */ } diff --git a/docs/docs/developer/extending/index.md b/docs/docs/developer/extending/index.md index 66cc54b..0025a5a 100644 --- a/docs/docs/developer/extending/index.md +++ b/docs/docs/developer/extending/index.md @@ -1,5 +1,12 @@ # Extending Comline +In this section you will find guides that are for people _that want to make implementations +that extend Comline_, either for your own specific personal use cases or to extend the +Comline Project itself upstream. + +If you simply want to use Comline (writting schemas, configuring your package, implement generated +code), then you should go to the regular [Guides](../guides/index.md) instead + Comline has five places where you can bring your own code. This section is a guide to each. What a page describes is built and shipped; what isn't is listed under [What isn't built](#what-isnt-built). @@ -29,7 +36,7 @@ listed under [What isn't built](#what-isnt-built). `@` and a whole number. A bare name such as `json@1` is reserved for the official package, and `comline build` rejects it anywhere else. - **Proof is optional but visible.** An encoding can ship test vectors that - prove an implementation, run with [`comline conformance`](../../reference/tools/cli.md#commands). + prove an implementation, run with [`comline conformance`](../../reference/tools/cli/index.md#commands). Nothing forces you to write them, but a format or framing without them, and a handshake whose profile doesn't state what it establishes, give up a guarantee that the schema must mark with `@unverified`. See diff --git a/docs/docs/developer/extending/transports.md b/docs/docs/developer/extending/transports.md index e7b1d82..d9734d1 100644 --- a/docs/docs/developer/extending/transports.md +++ b/docs/docs/developer/extending/transports.md @@ -8,7 +8,7 @@ handshake never sees which one it is. A transport carries **whole frames**. A medium that has message boundaries carries one frame per message. A byte stream has none, so it adds a `u32` little-endian length prefix per frame, as the runtime's own TCP transport does. -The [conformance reference](../../reference/conformance.md#stream-framing) +The [conformance reference](../../reference/conformance/wire.md#stream-framing) has the exact rules. ## Rust diff --git a/docs/docs/developer/index.md b/docs/docs/developer/index.md index 6c14c89..07dd0c5 100644 --- a/docs/docs/developer/index.md +++ b/docs/docs/developer/index.md @@ -1,8 +1 @@ # Overview - -In this section you will find guides that are for people _that want to make implementations -that extend Comline_, either for your own specific personal use cases or to extend the -Comline Project itself upstream. - -If you simply want to use Comline (writting schemas, configuring your package, implement generated -code), then you should go to the regular [Guides](../guides/index.md) instead diff --git a/docs/docs/getting-started.md b/docs/docs/getting-started.md index 3d92f7a..fa29261 100755 --- a/docs/docs/getting-started.md +++ b/docs/docs/getting-started.md @@ -6,7 +6,7 @@ from source. You need a Rust toolchain ([rustup](https://rustup.rs/)) and `git`: -```bash +```shell git clone https://github.com/ComlineProject/cli cd cli cargo install --path . @@ -14,7 +14,7 @@ cargo install --path . This puts the `comline` binary in `~/.cargo/bin`. Check it's on your `PATH`: -```bash +```shell comline --help ``` @@ -24,14 +24,14 @@ planned; packaging is tracked in ## Create a project -```bash +```shell comline new my-api cd my-api ``` You get: -```text +```shell my-api/ ├── config.idp # package manifest ├── comline.toml # where generated code goes (all-commented by default) @@ -40,9 +40,7 @@ my-api/ └── .gitignore # ignores .comline/deps-cache/ ``` -`src/main.ids`: - -```ids +```ids title="src/main.ids" /// The language a greeting is written in. enum Language { English @@ -64,7 +62,7 @@ characters. To set one for the whole package instead, see ## Build -```bash +```shell comline build ``` @@ -77,21 +75,21 @@ since the last one. Validate without freezing — safe for editors and pre-commit hooks: -```bash +```shell comline check ``` ## Generate code -```bash +```shell comline generate ``` Targets come from `code_generation.languages` in `config.idp` (the scaffold declares `rust#1.85.0`, the Rust generator's floor); output lands under `generated/` by default. Change where -and how in [`comline.toml`](reference/congregation/comline-toml.md): +and how in [`comline.toml`](reference/package/congregation/comline-toml.md): -```toml +```toml title="comline.toml" [generate] out = "src/generated" layout = "{{language}}/{{namespace}}.{{ext}}" @@ -104,4 +102,4 @@ Today `rust` and `typescript` are the generators. - [Overview](guides/index.md) — the whole pipeline - [IDL guide](guides/idl/index.md) — the schema language - [Generating code](guides/codegen/generating-code.md) — the full `generate` surface -- [CLI reference](reference/tools/cli.md) — every command and flag +- [CLI reference](reference/tools/cli/index.md) — every command and flag diff --git a/docs/docs/guides/codegen/generating-code.md b/docs/docs/guides/codegen/generating-code.md index 32b8365..6ef4e20 100644 --- a/docs/docs/guides/codegen/generating-code.md +++ b/docs/docs/guides/codegen/generating-code.md @@ -267,7 +267,7 @@ using cross-schema references in `code` mode needs to declare matching `mod` statements itself. `std` is the exception, because -[`external_std`](../../reference/congregation/comline-toml.md#external_std) +[`external_std`](../../reference/package/congregation/comline-toml.md#external_std) is on by default in `lib` mode: a std type then resolves to `comline_std::http::Request` and the crate depends on `comline_std` rather than carrying a copy of it. @@ -293,7 +293,7 @@ An alias imports the name the other module actually exports nothing refers to produces no import at all. `std` again resolves to its own package when -[`external_std`](../../reference/congregation/comline-toml.md#external_std) +[`external_std`](../../reference/package/congregation/comline-toml.md#external_std) is on, by subpath import — the leading `std` segment drops, since `@comline/std` is its own package root: diff --git a/docs/docs/guides/codegen/index.md b/docs/docs/guides/codegen/index.md index 1385952..d7a1eba 100755 --- a/docs/docs/guides/codegen/index.md +++ b/docs/docs/guides/codegen/index.md @@ -20,7 +20,7 @@ detail that the generated code is pleasant to use directly rather than only as a ## Using it `comline generate` is the command; [Generating code](generating-code.md) is the -full guide, and [`comline.toml`](../../reference/congregation/comline-toml.md) is where a +full guide, and [`comline.toml`](../../reference/package/congregation/comline-toml.md) is where a consumer sets the output location, layout and mode. Today there are `rust` and `typescript` generators, and both do `code` and diff --git a/docs/docs/guides/codegen/library-generation.md b/docs/docs/guides/codegen/library-generation.md index ce120f9..2318f82 100755 --- a/docs/docs/guides/codegen/library-generation.md +++ b/docs/docs/guides/codegen/library-generation.md @@ -7,7 +7,7 @@ the module layout. A planned third mode, `dylib`, would compile it to a library any language's [runtime](../runtime/index.md) can load. `comline generate`'s `mode` chooses how far generation goes. It is set per target -in [`comline.toml`](../../reference/congregation/comline-toml.md) (`[generate] mode` or +in [`comline.toml`](../../reference/package/congregation/comline-toml.md) (`[generate] mode` or `[[generate.target]] mode`). | `mode` | Output | Status | @@ -71,7 +71,7 @@ import { Greeting } from "my-api/types"; Comline's standard library is implicitly part of every package, so without help a `lib` build would vendor `std/http` and `std/validators` into every generated package. It doesn't: -[`external_std`](../../reference/congregation/comline-toml.md#external_std) +[`external_std`](../../reference/package/congregation/comline-toml.md#external_std) defaults to **on** in `lib` mode, so std's schemas are left out and the manifest depends on std's own pre-generated package instead — `comline_std` for rust, `@comline/std` for typescript. Your own nested @@ -89,7 +89,7 @@ with a message naming the pair. In either language, `/` keeps this target's package from colliding with another `lib` target sharing the same `out`; a target with `out` to itself can set -[`flatten = true`](../../reference/congregation/comline-toml.md#flatten) to skip that and +[`flatten = true`](../../reference/package/congregation/comline-toml.md#flatten) to skip that and land the package straight into `out`. `layout` plays no part in any of this — it only applies to `code` mode. Multi-version `lib` output is not done yet. diff --git a/docs/docs/guides/documenting.md b/docs/docs/guides/documenting.md index 95c5c04..300cdc0 100644 --- a/docs/docs/guides/documenting.md +++ b/docs/docs/guides/documenting.md @@ -144,7 +144,7 @@ the declarations your own use, and what those use in turn, never the whole dependency. Each dependency gets its own pages (`deps//`), and a type from it links there. -`std` links to its hosted docs instead, `https://comlineproject.github.io/docs/reference/schema/std/`. +`std` links to its hosted docs instead, `https://comlineproject.github.io/docs/reference/package/schema/std/`. To link another dependency to its published docs rather than render pages for it, give it a base URL in `comline.toml`: @@ -211,7 +211,7 @@ and every path it returns is checked before anything is written. ## Settings -Every flag has a `[doc]` key in [`comline.toml`](../reference/congregation/comline-toml.md#doc), +Every flag has a `[doc]` key in [`comline.toml`](../reference/package/congregation/comline-toml.md#doc), and the flag wins: | Key | Flag | Default | @@ -226,8 +226,8 @@ and the flag wins: ## The standard library, rendered When rendering documentation of a package that uses any standard library types, -the built documentation will link to [`std` pages](../reference/schema/std/index.md), meaning +the built documentation will link to [`std` pages](../reference/package/schema/std/index.md), meaning that the links will always lead to online A type from `std` in your own docs links to them by default, so `std::http::Request` in a field's type links -to `https://comlineproject.github.io/docs/reference/schema/std/http/#request`. +to `https://comlineproject.github.io/docs/reference/package/schema/std/http/#request`. diff --git a/docs/docs/guides/idl/index.md b/docs/docs/guides/idl/index.md index 3e095df..e1aad2a 100755 --- a/docs/docs/guides/idl/index.md +++ b/docs/docs/guides/idl/index.md @@ -5,8 +5,7 @@ write by hand. A schema file has the extension `.ids` and lives under a project' `src/` directory. It defines the data shapes and the protocols that carry them, independent of any language. -```ids -// greeting.ids +```ids title="greeting.ids" const GREETING_MAX: u16 = 280 enum Language { @@ -57,7 +56,7 @@ next declaration. `//!` lines at the top of a file are its ## Imports -```ids +```ids title="imports.ids" use std::http::Request use mypackage::{User, Post} use external::uuid::Uuid as UUID diff --git a/docs/docs/guides/idl/types.md b/docs/docs/guides/idl/types.md index ec59c94..7db0e8d 100644 --- a/docs/docs/guides/idl/types.md +++ b/docs/docs/guides/idl/types.md @@ -3,7 +3,7 @@ Primitives: | Group | Types | -|---|---| +| --- | --- | | signed integers | `s8` `s16` `s32` `s64` `s128` | | unsigned integers | `u8` `u16` `u32` `u64` `u128` | | floats | `f32` `f64` | @@ -18,7 +18,7 @@ type or given by a [setting](settings.md#default-maximums). The examples in this carry maxima so they build as written; a bare `string` or `T[]` in one of your schemas builds only once a default is set. -### Text: `string` and `char` +## Text: `string` and `char` - **`char`** is one Unicode scalar value: a single character, not a byte or a UTF-16 unit. Its literal is a string of exactly one character, `"a"`; any @@ -79,7 +79,7 @@ A schema builds only when every string and dynamic array, at every position There is no built-in default, so a position with none of these is an error that names it and the fix: -``` +```sh struct `Greeting`, field `message`: `string` has no maximum Help: write `char[max N]`, or set `bounds.string.max` in settings ``` @@ -97,7 +97,7 @@ A [type alias](#type-aliases) carries its bound, and the limit is changed there. Where an alias of a string or of a dynamic array is used (a field, argument, return or union member), it takes **no `[…]` suffix** of its own: -```ids +```ids title="bounds.ids" type Name = char[max 32] type Names = Name[max 10] // a list of an alias is its own alias @@ -116,9 +116,9 @@ Writing an alias in a list inside another alias, as `Names` does, is allowed. ## Type aliases -`type NAME = TYPE` gives a type a second name: +Writting `#!ids type NAME = TYPE` gives a type a second name: -```ids +```ids title="type_alias.ids" type UserId = u64 type Handle = char[max 32] @@ -128,17 +128,17 @@ struct User { } ``` -An alias is transparent: `UserId` is a `u64` everywhere, in the wire format and -in generated code (`pub type UserId = u64;`, `export type UserId = number;`). +An alias is transparent: `#!ids UserId` is a `#!ids u64` everywhere, in the wire format and in generated code. (In Rust is: `#!rust pub type UserId = u64;`, in Typescript being: `#!typescript export type UserId = number;` etc). + An alias can name another alias. -An alias can carry [`@validators`](validator.md#on-a-type-alias), which then -check every value of it, and `@deprecated`, which warns wherever it's used. +An alias can carry [Validators](validator.md#on-a-type-alias), which then +check every value of it. ## In generated code | Comline | Rust | TypeScript | -|---|---|---| +| --- | --- | --- | | `s8` … `s32` | `i8` … `i32` | `number` | | `u8` … `u32` | `u8` … `u32` | `number` | | `s64`, `s128` | `i64`, `i128` | `bigint` | diff --git a/docs/docs/guides/idl/use.md b/docs/docs/guides/idl/use.md index a745a35..fabd488 100644 --- a/docs/docs/guides/idl/use.md +++ b/docs/docs/guides/idl/use.md @@ -102,9 +102,9 @@ struct Call { `std` and each of its modules carry [module docs](docstrings.md#module-docs): hover `std` or `validators` in the editor to read them, with what's in each. -The full reference is on this site, [rendered from those docs](../../reference/schema/std/index.md) -by [`comline doc`](../documenting.md): [`std::http`](../../reference/schema/std/http.md) and -[`std::validators`](../../reference/schema/std/validators.md), each declaration with its fields +The full reference is on this site, [rendered from those docs](../../reference/package/schema/std/index.md) +by [`comline doc`](../documenting.md): [`std::http`](../../reference/package/schema/std/http.md) and +[`std::validators`](../../reference/package/schema/std/validators.md), each declaration with its fields and variants. A build includes only the std schemas a package imports. Like a dependency's, diff --git a/docs/docs/guides/idl/validator.md b/docs/docs/guides/idl/validator.md index 1e4aa59..b7ee8b6 100644 --- a/docs/docs/guides/idl/validator.md +++ b/docs/docs/guides/idl/validator.md @@ -125,7 +125,7 @@ function there's no single value for it to check, so it's a build error. ## The standard validators -[`std::validators`](../../reference/schema/std/validators.md) ships these. A **maximum** belongs +[`std::validators`](../../reference/package/schema/std/validators.md) ships these. A **maximum** belongs in the type ([`char[max 32]`](types.md#bounds)), where a decoder checks it before allocating; validators check what a type can't say, a minimum or a range. @@ -199,7 +199,7 @@ an alias: innermost first), then by the field's own. - An alias is the only way to check a **function argument**: arguments take no annotations of their own, but `to: Handle` runs `Handle`'s checks. -`@deprecated` works on an alias too, and warns wherever the alias is used. +`#!ids @deprecated` works on an alias too, and warns wherever the alias is used. ## In generated code @@ -225,7 +225,7 @@ in `comline.toml`; `"first"` stops at the first failure), and a provider refuses a call whose arguments fail them. Turning that off is a give-up with its reason, `validate_on_decode = { off = "why" }`; then nothing runs until you call `validate()`. See -[`validate_on_decode`](../../reference/congregation/comline-toml.md#validate_on_decode). +[`validate_on_decode`](../../reference/package/congregation/comline-toml.md#validate_on_decode). Properties are filled in when the code is generated: the argument you wrote, else the property's default, with constants resolved. Nothing about the diff --git a/docs/docs/guides/ir/cas.md b/docs/docs/guides/ir/cas.md index 16595ba..364acf3 100755 --- a/docs/docs/guides/ir/cas.md +++ b/docs/docs/guides/ir/cas.md @@ -12,7 +12,7 @@ with differences that suit a schema package. They are collected in ## On disk -``` +```sh .comline/ ├── FORMAT_VERSION the IR format stamp, e.g. 3 ├── objects/ @@ -35,7 +35,7 @@ flowchart LR ``` | Path | Contents | -|---|---| +| --- | --- | | `objects//` | One object. The 64 hex characters of its hash are split: the first two name the directory, the other 62 name the file. The file holds the object's bytes, LZ4-compressed. | | `refs/heads/main` | The hash of the newest commit, as 64 hex characters. Updated at the end of every build and release. It is the only ref. | | `FORMAT_VERSION` | The [IR format stamp](#the-ir-format-stamp) as a decimal number. | @@ -51,7 +51,7 @@ without `.comline/` starts over with no recorded versions, at `0.1.0-dev.1`. `uri` and `commit` pinned in `config.idp`. The `.gitignore` that `comline new` writes ignores only that directory: -``` +```sh .comline/deps-cache/ ``` @@ -72,7 +72,7 @@ names below are the keys in the stored bytes. A blob holds raw bytes. | Field | Type | Meaning | -|---|---|---| +| --- | --- | --- | | `content` | bytes | The payload. For a schema unit, the CBOR of one frozen unit. | Blobs hold three things: one frozen unit each (a struct, an enum, a protocol, @@ -84,13 +84,13 @@ dependency's `manifest`. A tree lists named entries, like a directory. | Field | Type | Meaning | -|---|---|---| +| --- | --- | --- | | `entries` | list of entry | In the order they were added. | Each entry: | Field | Type | Meaning | -|---|---|---| +| --- | --- | --- | | `mode` | `Blob` or `Tree` | What `hash` points at. | | `name` | string | The entry's name in this tree. | | `hash` | 32 bytes | The object's blake3 hash. | @@ -105,7 +105,7 @@ declaration: `namespace_`, `struct_`, `enum_`, A commit is one version of the package. | Field | Type | Meaning | -|---|---|---| +| --- | --- | --- | | `tree` | hash | The version's root tree. | | `parents` | list of hash | The previous commit. Empty for the first commit. Today it holds at most one. | | `author` | string | The user name of whoever built, from the operating system. | @@ -124,7 +124,7 @@ The `release` field of a release commit says how the version was reached. See [Builds and releases](#builds-and-releases) for how the bumps are chosen. | Field | Type | Meaning | -|---|---|---| +| --- | --- | --- | | `computed` | bump | The bump the changes since the previous release call for: `None`, `Patch`, `Minor` or `Major`. | | `applied` | bump | The bump used. The computed one, or higher if `--raise` was given. | | `reason` | string, optional | Why the bump was raised or the package made stable. | @@ -152,7 +152,7 @@ flowchart TD ``` | Entry | Mode | Contents | -|---|---|---| +| --- | --- | --- | | `schema_` | tree | One per schema file, numbered in build order, holding that schema's frozen units. | | `dep_` | tree | A dependency vendored into the commit: its own `schema_` trees and a `manifest` blob with its verified content hash. Written by builds that include dependency support. | | `config` | blob | The frozen `config.idp`, as a list of units. | @@ -169,7 +169,7 @@ history (see [Compatibility facts](../../reference/versioning.md#compatibility-f It's CBOR like every other object, with fields by name: | Field | Type | Meaning | -|---|---|---| +| --- | --- | --- | | `protocols` | list | One entry per protocol of the package's own, in schema and declaration order. | | `protocols[].path` | string | `namespace::Protocol`, the key every fact is kept by. | | `protocols[].line` | string | The release at which the protocol last broke. | @@ -234,7 +234,7 @@ language edition the schemas are written in, and it is unrelated to how core stores its own IR. | Stamp | What changed | -|---|---| +| --- | --- | | 1 | CBOR, replacing a position-based encoding. | | 2 | A property's `expression` became a typed `value`. | | 3 | Types became structured (`TypeRef`, `Literal`) instead of text. | @@ -316,12 +316,12 @@ now. Then it adds one commit on top of the latest build: the highest the changes since the last release call for, raised with `--raise` if asked. It refuses when there is nothing to release. -See `comline release` in the [CLI reference](../../reference/tools/cli.md). +See `comline release` in the [CLI reference](../../reference/tools/cli/index.md). ## What reads the store | Reader | What it reads | -|---|---| +| --- | --- | | `comline build`, `comline release` | The ref and the previous commits, to compare and to number the next version. | | `comline diff ` | Two commits' root trees, compared with the same rules as a build: schema changes, changes within a dependency's reach, config changes and the bump they call for. | | `comline inspect` | The newest build's `facts`, shown beside each protocol's wire summary. It writes nothing. See [`comline inspect`](../../reference/tools/cli/inspect.md). | @@ -347,9 +347,9 @@ hash, and each encoding's name and parameter hash (both blake3, as hex). See [Compatibility facts](../../reference/versioning.md#compatibility-facts) for what they mean and how they are compared. The `config.idp` keys are in the -[config.idp reference](../../reference/congregation/config-id.md), and where +[config.idp reference](../../reference/package/congregation/config-id.md), and where generated code lands is set in -[`comline.toml`](../../reference/congregation/comline-toml.md). +[`comline.toml`](../../reference/package/congregation/comline-toml.md). ## Branches @@ -399,7 +399,7 @@ from, and which one it recorded as the second parent. ## Compared with Git | | Git | Comline | -|---|---|---| +| --- | --- | --- | | Hash | SHA-1 or SHA-256, over a `type size` header plus the content | blake3, over the object's serialized bytes | | Loose object compression | zlib | LZ4 | | Object encoding | Its own text and binary layouts | CBOR, with field names | diff --git a/docs/docs/guides/packages/index.md b/docs/docs/guides/packages/index.md index b3d1402..8251ff5 100644 --- a/docs/docs/guides/packages/index.md +++ b/docs/docs/guides/packages/index.md @@ -3,18 +3,18 @@ A Comline **package** (historically *congregation*) is a directory with a manifest and one or more schema files: -``` +```shell my-api/ -├── config.idp # the manifest — package identity + declarations +├── config.idp # the manifest — package identity + declarations ├── comline.toml # consumer-side: where generated code goes (see below) ├── src/ │ └── *.ids # the schemas └── .comline/ # content-addressable store (appears after the first build) ``` -## Manifest (`config.idp`) +## Manifest -```idp +```idp title="config.idp" congregation my_api id = "3f2a9c41d07e4b1c8a65f2e0d91b7c34" specification_version = 1 @@ -28,7 +28,7 @@ code_generation = { ``` Every key, its syntax and what it does is in the -[`config.idp` reference](../../reference/congregation/config-id.md). The ones +[`config.idp` reference](../../reference/package/congregation/config-id.md). The ones with more to them have their own sections below: [wire defaults](#wire-defaults) and [dependencies](#dependencies). @@ -47,7 +47,7 @@ wire = { format = "packed@1" framing = "datagram@3" } ``` | Key | Values | Default | -|---|---|---| +| --- | --- | --- | | `format` | `"packed@1"`, `"json@1"`, `"msgpack@1"` | `"packed@1"`: fixed-width fields with no tags or names — see [`packed@1`](../../reference/wire/formats/packed@1.md) | | `framing` | `"datagram@3"`, `"datagram@2"`, `"datagram"` (`"datagram@1"`), `"jsonrpc"` | `"datagram@3"`: every protocol on a [connection](../idl/protocol.md#calls-in-both-directions), calls addressed by [wire number](../../reference/versioning.md#number) | @@ -76,20 +76,19 @@ that says nothing gets. The values are strings, quoted. These used to be the consumer's choice, in `comline.toml`'s `default_framing`. That key is now an error that hands back the `wire` line to write instead. - ## Consumer Side (`comline.toml`) Where generated code lands, in what layout, for which versions — none of which belongs to the package author — is set by whoever runs `comline generate`, in a separate, non-frozen `comline.toml`. See [Generating code](../codegen/generating-code.md) and the -[`comline.toml` reference](../../reference/congregation/comline-toml.md). +[`comline.toml` reference](../../reference/package/congregation/comline-toml.md). ## Dependencies A package can import schemas from other packages it declares in `config.idp`: -```idp +```idp title="config.idp" dependencies = { shared_types = { path = "../shared-types" @@ -108,7 +107,7 @@ The key (`shared_types`, `net`) is a name you choose. What the entry contains decides where the package comes from: | Source | Written as | Resolved | -| --- | --- | --- | +| --- | --- | --- | | Path | `path` (optional `hash`) | relative to this package's directory | | Git | `version`, `uri`, `commit` (optional `hash`) | cloned once per pin into `.comline/deps-cache/`, using the `git` binary | | Registry | `version`, `uri` (optional `hash`, `signature`) | **not supported yet** — there's no registry server, so `comline check` / `build` reject it | @@ -166,7 +165,6 @@ comline add net --git https://github.com/acme/net --commit 4f2c9e1 --version 1.2 The rest of `config.idp` is left as written. `--no-hash` skips the pin, say for a path dependency you're still working on. - ## CAS (`.comline/`) After the first `comline build`, a `.comline/` directory holds the @@ -182,10 +180,8 @@ recorded versions. The `comline new` scaffold's `.gitignore` ignores only `comline reset` deletes `.comline/` (behind a confirmation). `comline clean` does not — see [Versioning rules](../../reference/versioning.md#history-model). - ## Publishing - !!! note Planned feature It is yet not possible to publish a package since there is no package registry or package registry publisher tooling ready yet. @@ -206,7 +202,7 @@ publish_registries = { Then: -``` +```shell comlinepm registry login comlinepm registry publish --registries "mainstream my_registry" ``` diff --git a/docs/docs/guides/runtime/index.md b/docs/docs/guides/runtime/index.md index f853e52..be53df6 100755 --- a/docs/docs/guides/runtime/index.md +++ b/docs/docs/guides/runtime/index.md @@ -31,7 +31,7 @@ the same and are tested the same. Codecs, generated types and transports stay in each language. Every runtime is checked against the same -[conformance vectors](../../reference/conformance.md), so a Rust peer and a +[conformance vectors](../../reference/conformance/wire.md), so a Rust peer and a TypeScript peer generated from one schema speak the same bytes. See [Language Guides](../../languages/index.md) for the practical, diff --git a/docs/docs/languages/rust/connections.md b/docs/docs/languages/rust/connections.md index 3a071da..0a077aa 100644 --- a/docs/docs/languages/rust/connections.md +++ b/docs/docs/languages/rust/connections.md @@ -212,7 +212,7 @@ another task while the connection serves, generate for the async driver. ## Async -With `async = true` in [`comline.toml`](../../reference/congregation/comline-toml.md#async), +With `async = true` in [`comline.toml`](../../reference/package/congregation/comline-toml.md#async), connection protocols are generated for the runtime's `AsyncConnection` instead, on any executor: diff --git a/docs/docs/languages/rust/index.md b/docs/docs/languages/rust/index.md index 47b7a3d..10806dc 100644 --- a/docs/docs/languages/rust/index.md +++ b/docs/docs/languages/rust/index.md @@ -23,7 +23,7 @@ its reply arrives, and nothing inside the runtime spawns a thread or takes a lock, so it works the same on a bare-metal target with no allocator as in a normal `std` binary. -Set [`async = true`](../../reference/congregation/comline-toml.md#async) in +Set [`async = true`](../../reference/package/congregation/comline-toml.md#async) in `comline.toml` and connection protocols are generated for the runtime's **async** driver instead: handlers are `async fn`s, the connection is a future you spawn on any executor, and handles call the peer from any task diff --git a/docs/docs/proposals/auto-documentation.md b/docs/docs/proposals/auto-documentation.md index 552480e..28d7309 100755 --- a/docs/docs/proposals/auto-documentation.md +++ b/docs/docs/proposals/auto-documentation.md @@ -36,7 +36,7 @@ modules. Versioning is not hand-maintained: every `comline build` freezes an immutable version into `.comline/`, the bump is derived from the schema change itself, and -[`comline diff`](../reference/tools/cli.md) already reports what changed between two +[`comline diff`](../reference/tools/cli/index.md) already reports what changed between two frozen versions. A documentation tool built on top of that can do what a conventional one cannot: @@ -58,7 +58,7 @@ IR", and it follows from the It is a tool a project opts into, not a step in the pipeline. Nothing here happens unless asked for. The consumer's -[`comline.toml`](../reference/congregation/comline-toml.md) owns the switches, +[`comline.toml`](../reference/package/congregation/comline-toml.md) owns the switches, the same way it already owns generation's output location, layout and mode. ## Decisions @@ -112,8 +112,8 @@ These were open questions until 2026-11-01. The rationale is in It lands in the product documentation, not here: a Guide page on documenting your schemas, a `comline doc` entry in the -[CLI reference](../reference/tools/cli.md), its `comline.toml` keys in the -[congregation reference](../reference/congregation/index.md), and the doc +[CLI reference](../reference/tools/cli/index.md), its `comline.toml` keys in the +[congregation reference](../reference/package/congregation/index.md), and the doc model's JSON Schema. The obvious first demonstration is diff --git a/docs/docs/proposals/extensibility.md b/docs/docs/proposals/extensibility.md index dd9233f..4e946df 100644 --- a/docs/docs/proposals/extensibility.md +++ b/docs/docs/proposals/extensibility.md @@ -521,7 +521,7 @@ None open. Both were decided on 2026-10-31: Built in October 2026. The guides are under [Extending Comline](../developer/extending/index.md), the [generator protocol](../reference/generator-protocol.md) and the -[conformance spec](../reference/conformance.md) are reference pages. +[conformance spec](../reference/conformance/wire.md) are reference pages. - **Declaring encodings:** `encodings` in `config.idp`, with a kind, a profile and suggested mappings diff --git a/docs/docs/reference/conformance/benchmarks.md b/docs/docs/reference/conformance/benchmarks.md new file mode 100644 index 0000000..f6bbec4 --- /dev/null +++ b/docs/docs/reference/conformance/benchmarks.md @@ -0,0 +1,117 @@ +# Benchmarks + +How performance is measured, gated and published. The source of truth for +the shared definitions is the runtime's +[`runtime/conformance/benches/README.md`](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/benches/README.md). + +The gates count **instructions** (iai-callgrind or Callgrind, under +Valgrind), not time. Instruction counts are deterministic, so a 1% limit +holds on shared CI machines without flaking. A count compares only against +the same runtime on another commit, never across languages or machines. + +## Flow + +```mermaid +flowchart LR + defs["shared definitions
benches/*.json"] --> rust["Rust benches"] + defs -->|vendored| ts["TypeScript benches"] + rust -->|pull request| gate["base vs head
1% gate + ledger"] + rust -->|push to main| rep["<lang>.bench.json"] + ts -->|push to main| rep + core["core parser benches"] --> gate + core --> rep + rep --> branch["reports branch"] --> obs["Observatory"] +``` + +## Shared definitions + +`runtime/conformance/benches/*.json` defines the work every runtime measures, +so ids line up across languages. Each file says what is inside the measured +region; anything it doesn't name (building inputs, allocating the output +buffer, I/O) is outside. + +| File | Measures | Ids | +| --- | --- | --- | +| `formats.json` | encode and decode per format, per shape (`small-scalar`, `string-heavy`, `nested`, `large-array`, `enum-heavy`) | `format///` | +| `framings.json` | framing encode and decode per framing, per payload size | `framing///` | +| `handshake.json` | the handshake frame: encode, decode, check | `handshake/` | +| `comline2.json` | the `comline@2` hello: encode, decode, answer | `comline2/` | +| `connection.json` | one connection step: issue a call, receive its reply | `connection/step` | + +The files use the [conformance vectors'](wire.md) value encoding, plus a +`generate` rule for large values. `specs/bench.cue` specifies them, and +`tests/bench_definitions.rs` checks them against the Rust mirrors. The Rust +benches are the reference: where the README and they disagree, one of them +has a bug. + +## Per repository + +| Repository | What | Metric | Pull request | Push to default branch | +| --- | --- | --- | --- | --- | +| runtime | the shared definitions (`runtime/core/benches`) and the C ABI (`runtime/ffi/bench`) | instructions | base and head in one job, **gated** | head benched, `rust.bench.json` published | +| core | the parser (`core/benches`, over `core/benches/inputs/`) | instructions (gated); wall time with Criterion (reported only) | base and head, **gated** | `core.bench.json` published | +| comline-typescript | the shared definitions, vendored | instructions in two tiers: `--no-opt` and optimizing after warm-up | each benchmark runs once, uncounted | counted, `typescript.bench.json` published | + +- The runtime's C ABI benches add `ffi/abi/` and `ffi/core/` + pairs. The job summary shows the crossing cost between them, for + information only. +- comline-typescript isn't gated yet. Its optimizing tier is what the + Observatory's ratio to Rust uses. +- Wall time is never gated: on shared runners it's noise. + +## The gate + +On a pull request, runtime and core bench the base and then the head in the +same job, on the same machine: + +- A benchmark that grows **more than 1%** fails. +- A benchmark the base had and the head doesn't run fails. +- A new benchmark has nothing to compare against, and is recorded only. + +A cost that is the price of a fix is accepted in the diff, by appending to +the repository's `accepted-regressions.toml`: + +```toml +[[accepted]] +pr = 123 # the pull request that pays it +bench = "handshake/check" # one benchmark id, exactly +max_percent = 2.37 # the most it may grow, in % +reason = "check the peer's ids in constant time (#120)" +``` + +- An entry counts only in the pull request that adds it (`pr` must be its + number), covers one benchmark up to `max_percent`, and must be used. +- `[[removed]]` (with an optional `renamed_to`) covers a dropped or renamed + benchmark. A renamed one is compared under its new id. +- The ledger is append-only. Nothing is accepted by a label, a setting or a + default. +- The job summary prints the entry to paste, with the measured growth rounded + up. + +The Observatory reads the ledger too: a step of more than 1% between two +published versions is marked **accepted** when a ledger entry explains it, and +**unexplained** otherwise. + +## Outputs + +Reports go to each repository's `reports` branch, one commit per publish: +`rust.bench.json`, `core.bench.json`, `typescript.bench.json`. A pull request +writes its report as an artifact only. + +```json +{ + "runtime": "rust", + "commit": "", + "toolchain": "rustc 1.100.0-nightly (787af2b8c 2026-08-25)", + "metric": "instructions", + "benches": { "format/encode/small-scalar/packed@1": { "ir": 123 } } +} +``` + +That is format 1. Format 2, used by core and comline-typescript, allows several +metrics per benchmark: `{"": {"value", "runner"?, "spread"?}}`. A +metric with a `runner` is charted per runner. See the Observatory's +[`DATA.md`](https://github.com/ComlineProject/observatory/blob/main/DATA.md#bench). + +The core parser also has local-only Criterion runs (`core/benches/run_bench.sh`) +that write HTML reports. diff --git a/docs/docs/reference/conformance/fuzzing.md b/docs/docs/reference/conformance/fuzzing.md new file mode 100644 index 0000000..b95eaca --- /dev/null +++ b/docs/docs/reference/conformance/fuzzing.md @@ -0,0 +1,93 @@ +# Fuzzing + +How the reference runtime is fuzzed, and how other runtimes are tested +against it on random inputs. The source of truth is the runtime's +[`runtime/fuzz/README.md`](https://github.com/ComlineProject/runtime/blob/main/runtime/fuzz/README.md). + +The Rust runtime is the reference every other runtime is checked against, so +its decoders must be strict, canonical and panic-free. Two kinds of test +check that: + +- **Coverage-guided fuzzing** (`cargo-fuzz`, libFuzzer, nightly Rust) on raw + bytes. +- **Property tests** (`proptest`) on structured inputs: round trips over + random types and values, and the C ABI against the core. + +## Flow + +```mermaid +flowchart LR + vec["conformance vectors"] -->|seed-corpus.sh| corpus["corpus/<target>/"] + corpus --> fuzz["cargo fuzz run"] + fuzz -->|finding| tmin["cargo fuzz tmin"] + tmin --> fix["fix + found-* vector"] + fix --> vec + fuzz --> rep["rust.fuzz.json"] + rep --> branch["reports branch"] +``` + +1. **Seed.** `seed-corpus.sh` writes every conformance vector's bytes, every + test mirror's samples and whole connection exchanges into + `corpus//`, encoded as each target reads its input. The fuzzer + starts from real frames, not from nothing. +2. **Fuzz.** Each target reads its input through a fixed grammar, one byte + per choice, so a seed can be written back in it. Memory is capped + (`-rss_limit_mb=2048 -malloc_limit_mb=512`), and going past a cap counts as + a crash. +3. **Report.** `.github/scripts/fuzz-report.py` turns libFuzzer's stats and + the property tests' verdicts into `rust.fuzz.json`. +4. **A finding becomes a vector.** Minimize it, fix the runtime (never weaken + the assertion), add the input as a `found-*` case and bless the vectors. + From then on every runtime checks it on every run. + +## Targets + +| Target | Checks | +| --- | --- | +| `decode_no_panic` | decoding never panics, stays within the memory caps, and anything it accepts passes `check` | +| `packed_canonical` | bytes that decode in `packed@1` re-encode to exactly themselves | +| `framing_decode` | every framing, envelope and handshake decoder: no panic; binary frames re-encode exactly; JSON-RPC decodes the same after a round trip | +| `seed_vs_derive` | the conformance crate's `TypeSeed` accepts, refuses and reads exactly what a `serde` derive does | +| `stream_split` | stream framing splits and prefixes exactly by the declared length and limit | +| `ffi_feed` | the C ABI keeps its own rules: no panic crosses, statuses and spans are valid, no allocation after `comline_connection_new` | +| `ffi_vs_core` | the C ABI gives the same results as the core called directly | +| `ffi_comline2` | a `comline@2` connection through the C ABI, under any side and parameters, fed any frames | + +Property tests: `conformance/tests/properties.rs` (round trips over random +types and values) and `ffi/tests/differential.rs` (the C ABI against the +core on random sequences). + +## When it runs + +| Run | Fuzzing | Property tests | On failure | +| --- | --- | --- | --- | +| Every pull request | 60 s per target, seeded corpus | default case count, with `cargo test` | the check fails | +| Nightly (03:17 UTC) and manual | 600 s per target, corpus cached between runs | `PROPTEST_CASES=100000` | the minimized crash is uploaded, and an issue titled after the target is opened (unless one is open). Nobody's pull request is blocked | + +## Other runtimes + +A runtime that isn't the reference compares itself with it through +`comline-reference`, which answers any [ops protocol](wire.md#the-ops-protocol) +question. + +comline-typescript's nightly workflow (03:17 UTC) installs `comline-reference` +at the corpus's `VECTORS_REV` and runs its property tests +(`runtime/test/prop/`) against it at many more runs than a pull request's. +Connection logic is compared through the `session` op. A failure opens or +comments on an issue. Coverage-guided fuzzing (Jazzer.js) isn't set up yet. + +## Outputs + +| Report | Written by | Published | +| --- | --- | --- | +| `rust.fuzz.json` | runtime `fuzz.yml`, every run | nightly or manual from `main` | +| `typescript.fuzz.json` | comline-typescript `nightly.yml`, pass or fail | every nightly run | + +Both go to their repository's `reports` branch. The +[Observatory](reports.md) charts executions, corpus size, findings and +property verdicts per target, and the +[Rust language page](../languages/rust.md) renders the latest Rust run. The report +format is in the +[fuzz README](https://github.com/ComlineProject/runtime/blob/main/runtime/fuzz/README.md#the-report). + +Pull requests write the report as an artifact only. diff --git a/docs/docs/reference/conformance/generators.md b/docs/docs/reference/conformance/generators.md new file mode 100644 index 0000000..aa4ba50 --- /dev/null +++ b/docs/docs/reference/conformance/generators.md @@ -0,0 +1,51 @@ +# Generator conformance + +How each code generator's output is pinned, and how its feature support is +proven. The source of truth is the generation repository's +[`conformance/README.md`](https://github.com/ComlineProject/generation/blob/master/conformance/README.md). + +## Flow + +1. **Fixtures.** Each fixture in `conformance/src/lib.rs` is one schema as + hand-built IR (`Vec`), not parsed source. The corpus pins IR + shape, not parser behaviour. +2. **Goldens.** Every registered generator runs over every fixture, and its + output is diffed against `tests/golden//.`. A golden is + a snapshot of what the generator does now, not what it should do: a change + in output shows up in that pull request's diff and is re-blessed + deliberately. +3. **Support claims.** Each generator states what it implements in its + `SUPPORT` table (`comline_codegen::features`). `support_is_proven` checks + every claim against the fixtures. +4. **Outputs.** The proven table is written to `FEATURES.md` and + `features.json` at the repository root, by the same run that blesses the + goldens. +5. **Consumers.** The [Languages](../languages/index.md) pages and the + [Observatory](reports.md)'s `codegen` matrix read `features.json` from the + default branch. + +```sh +cargo test -p comline-conformance # check +CONFORMANCE_BLESS=1 cargo test -p comline-conformance # regenerate goldens and FEATURES +``` + +## Support claims + +| Claim | Holds when | +| --- | --- | +| `supported`, `partial` | a fixture uses the feature, generates, and matches its golden; where the feature can be taken out, the output differs without it | +| `ignored` | the output is exactly what it would be without the feature | +| `unsupported` | generation is refused, with an error naming the feature | + +A claim no fixture can back fails the suite, so a new feature comes with a +fixture. + +## When it runs + +| Where | When | Checks | +| --- | --- | --- | +| generation | every pull request and push (`cargo test`) | goldens match; every claim is proven; `FEATURES.md` and `features.json` are current | +| docs | weekly (`language-support-drift.yml`) | the Languages pages match `features.json`, the wire corpus and each runtime's report | + +Editors and the CLI surface the same table: `comline targets --features` and +the language server's warnings for features a target doesn't support. diff --git a/docs/docs/reference/conformance/index.md b/docs/docs/reference/conformance/index.md new file mode 100644 index 0000000..cbfb608 --- /dev/null +++ b/docs/docs/reference/conformance/index.md @@ -0,0 +1,59 @@ +# Conformance + +How Comline proves its implementations agree, and how it keeps track of +their speed, size and health over time. Each subject has its own page. This +one shows how they fit together. + +## The pipeline + +```mermaid +flowchart LR + subgraph producers["Producers (each repository's CI)"] + rt["runtime"] + gen["generation"] + ts["comline-typescript"] + core["core"] + cli["cli"] + sim["simulator"] + end + producers -->|gates on pull requests| pr["pass / fail"] + producers -->|reports on main| rb["reports branch
or default branch"] + gh["GitHub API"] --> col + rb --> col["Observatory collector"] --> site["Observatory site"] + rb --> docs["docs: Languages pages"] +``` + +1. **Pull requests are gated** in each repository's own CI: vectors and + goldens must match, support claims must be proven, fuzz targets must + survive a short run, and instruction counts must not grow more than 1% + without a ledger entry. +2. **Pushes to the default branch publish reports**, one JSON file per + measurement, committed to the repository's `reports` branch. Conformance + reports are committed on the default branch itself. +3. **The Observatory collects** every report's history, so each published + version can be opened and any two compared. It shows results and gates + nothing. +4. **The docs render** the latest conformance, fuzz and bench results into the + [Languages](../languages/index.md) pages at build time. + +## Subjects + +| Page | Proves or measures | Gate on pull requests | Published as | +| --- | --- | --- | --- | +| [Wire conformance](wire.md) | every runtime reproduces the wire contract byte for byte | yes: vectors current, claims proven | `reports/.json` | +| [Generator conformance](generators.md) | each generator's output, and the features it claims | yes: goldens match, claims proven | `features.json` | +| [Fuzzing](fuzzing.md) | the reference's decoders are strict, canonical and panic-free; other runtimes match it on random inputs | yes: 60 s per target | `.fuzz.json` (nightly) | +| [Benchmarks](benchmarks.md) | instruction counts of shared and per-repository benchmarks | yes: 1% limit plus the ledger | `.bench.json` | +| [Timings, size and coverage](timings.md) | compile time, CI time, artifact sizes, test coverage | no | `.compile.json`, `.size.json`, `.coverage.json`; CI from the API | +| [Reports and the Observatory](reports.md) | how reports are published, collected and compared | — | — | + +## Schedule + +| When (UTC) | What | +| --- | --- | +| every pull request | conformance, short fuzz, bench gate | +| every push to a default branch | benches, size, coverage and compile reports published | +| 03:17 nightly | long fuzz and property runs (runtime), differential tests against the reference (comline-typescript) | +| 04:30 daily, and every 4 hours | Observatory rebuilt | +| 06:00 daily | docs rebuilt, picking up the nightly fuzz reports | +| Mondays 07:45 | docs drift check of the Languages pages against conformance | diff --git a/docs/docs/reference/conformance/reports.md b/docs/docs/reference/conformance/reports.md new file mode 100644 index 0000000..7e5f48c --- /dev/null +++ b/docs/docs/reference/conformance/reports.md @@ -0,0 +1,99 @@ +# Reports and the Observatory + +How every result on the other pages reaches the +[Observatory](https://comlineproject.github.io/observatory/), where each +published version can be opened and any two compared. The source of truth is +the Observatory's +[`README.md`](https://github.com/ComlineProject/observatory/blob/main/README.md) +and [`DATA.md`](https://github.com/ComlineProject/observatory/blob/main/DATA.md), +the contract between its collector and its site. + +The Observatory only shows results. Every gate stays in the producing +repository's own CI. + +## Flow + +```mermaid +flowchart LR + ci["producers' CI"] -->|publish-report| rb["reports branch
<name>.<kind>.json"] + ci -->|committed| main["default branch
conformance reports"] + api["GitHub API
workflow runs"] --> col + rb --> col["collector
sources.toml"] + main --> col + col --> data["static dataset"] --> site["site"] +``` + +1. **Producers publish.** A repository's CI writes a JSON report, and the + shared `publish-report` action commits it to the repository's `reports` + branch. +2. **Each commit is one version.** The collector walks every source's branch + first-parent, and each commit that changed the report becomes a version. + `git log -- ` on the `reports` branch is the series. +3. **Versions are labelled by release.** A version is `-dev.`, + where the cycle is the oldest release tag containing the measured commit, + or `unreleased`. Labels are recomputed on every run, so a new tag regroups + older versions. +4. **The site renders the dataset.** It's a static SvelteKit site, rebuilt + daily at 04:30 UTC, every 4 hours, on every push to the observatory and on + demand. + +## Publishing + +`ComlineProject/observatory/.github/actions/publish-report@v1`: + +- is called only from the default branch, a schedule or a manual run, never + from a pull request. Anywhere else, callers pass `dry-run: true`; +- needs `permissions: contents: write` and a checkout with `fetch-depth: 0`; +- requires each report to carry the `commit` it measured, and skips a report + older than the one already published, so a slow run never overwrites a + newer one; +- can publish a `readme` as the branch's `README.md`, which says what the + branch holds (`.github/reports-branch.md` in each repository). + +The conformance matrices are the exception: they're committed on the default +branch by the tests that prove them, and read from there. + +## Sources + +[`sources.toml`](https://github.com/ComlineProject/observatory/blob/main/sources.toml) +lists everything the collector reads. A source that fails (unreachable, no +versions, a report that doesn't parse) is shown as unpublished with a reason, +and the rest still build. + +| Kind | Page | Sources | +| --- | --- | --- | +| `conformance` | [Wire](wire.md), [Generators](generators.md) | `wire` matrix: Rust and TypeScript runtime reports; `codegen` matrix: generation's `features.json` | +| `fuzz` | [Fuzzing](fuzzing.md) | runtime, comline-typescript | +| `bench` | [Benchmarks](benchmarks.md) | runtime, core, comline-typescript | +| `compile` | [Timings](timings.md#compile-time) | runtime, core, cli | +| `ci` | [Timings](timings.md#ci-time) | every repository, from the GitHub API | +| `size` | [Timings](timings.md#size) | runtime, cli, simulator, comline-typescript | +| `coverage` | [Timings](timings.md#coverage) | runtime, core, cli, generation, comline-typescript | + +A conformance matrix combines its sources into one table. Their histories are +separate, so a matrix version is a point in time: each source at its newest +version on or before that date. + +## Reading the site + +- Every kind has an index page and a page per source. There is also `ratio`, + which shows TypeScript instruction counts relative to Rust's. +- **Compare:** pick two versions, and the URL keeps them (`?a=