diff --git a/docs/docs/design/index.md b/docs/docs/design/index.md index b51b273..2c18dbc 100644 --- a/docs/docs/design/index.md +++ b/docs/docs/design/index.md @@ -18,10 +18,11 @@ Each page carries a status and links the issues / PRs that carry it out. across Rust, Python, TypeScript, Luau. *Sketch.* - [Bounds on types](type-bounds.md) — a maximum in the type, `T[max N]`, with strings as `char[]`: what's checked where, the wire, generated code, and the - rollout. *Accepted, not implemented.* + rollout. *Accepted; core's checks and generation's feature are in, the + runtime and generators' checks are next.* - [Validators](validators.md) — `validator` declarations and the `@validators` field annotation: the intended shape and what has to change to get there. - *Not implemented.* + *Checked at build time and run in Rust; TypeScript next.* - [Generation](generation.md) — what codegen / libgen / runtime each mean, moving the generators out of `core`, and the `generation` repo de-rot plan. *Partly built.* diff --git a/docs/docs/getting-started.md b/docs/docs/getting-started.md index 266ac06..1397275 100755 --- a/docs/docs/getting-started.md +++ b/docs/docs/getting-started.md @@ -2,33 +2,26 @@ ## Install -At the moment the is no proper `comline` releases in crates.io, so install **From source** below. +`comline` isn't released on crates.io or as an OS package yet, so build it +from source. You need a Rust toolchain ([rustup](https://rustup.rs/)) and +`git`: -After installing from any of the two options, you should now have access to the `comline` binary (which should be at `~/.cargo/bin`). You can check by doing: - -```bash -comline --help -``` - - -### From Source (Github) ```bash git clone https://github.com/ComlineProject/cli cd cli cargo install --path . ``` -### From Cargo (crates.io) - -`comline` is on crates.io but there is no proper releases being done yet, so build from source by doing the following: +This puts the `comline` binary in `~/.cargo/bin`. Check it's on your `PATH`: +```bash +comline --help +``` -A `cargo install comline` from crates.io, and OS packages (Fedora, Arch, -Windows, macOS), are planned — packaging is tracked in +`cargo install comline` and OS packages (Fedora, Arch, Windows, macOS) are +planned; packaging is tracked in [`ComlineProject/distributions`](https://github.com/ComlineProject/distributions). - - ## Create a project ```bash diff --git a/docs/docs/guide/codegen/generating-code.md b/docs/docs/guide/codegen/generating-code.md index 112c486..32b8365 100644 --- a/docs/docs/guide/codegen/generating-code.md +++ b/docs/docs/guide/codegen/generating-code.md @@ -255,7 +255,7 @@ is a dedicated one, e.g. `generated/`). It leaves `.comline/` alone — |---|---| | `struct` | `#[derive(…)] pub struct` with `serde` derives | | `enum` | `pub enum` | -| `protocol` | `pub trait` with a method per function | +| `protocol` | `pub trait` with a method per function, a dispatcher and a sender; see [Rust connections](../../languages/rust/connections.md) | A cross-schema reference (a type pulled in via `use`) is emitted as a `crate`-qualified path (`crate::shared::Shared`), not a bare name — a @@ -279,7 +279,7 @@ than carrying a copy of it. | `struct` | `export interface` | | `error` | `export interface` (the payload) + `export class Error` | | `enum` | `export enum` with string values | -| `protocol` | a client, a dispatcher, and a `serve` helper against `@comline/runtime` | +| `protocol` | a provider interface and a sender, with a package-wide `serve` / `accept` / `connect` for [connections](../../languages/typescript/connections.md); a client, a dispatcher and a `serve` helper for one on the lockstep `datagram` framing | A cross-schema reference is emitted as an `import type` from the schema that declares it, by relative specifier: diff --git a/docs/docs/guide/codegen/library-generation.md b/docs/docs/guide/codegen/library-generation.md index 3ba6048..ce120f9 100755 --- a/docs/docs/guide/codegen/library-generation.md +++ b/docs/docs/guide/codegen/library-generation.md @@ -2,9 +2,9 @@ [Code generation](index.md) turns a schema into **source text** — the type definitions and protocol traits. **Library generation** goes further and wraps -that source into a **package**: a manifest (`Cargo.toml`, `pyproject.toml`, a -`.rockspec`), the module layout, and — when the consumer is in another language -— the FFI surface a [runtime](../runtime/index.md) loads and dispatches through. +that source into a **package**: a manifest (`Cargo.toml`, `package.json`) and +the module layout. A planned third mode, `dylib`, would compile it to a library +any language's [runtime](../runtime/index.md) can load. `comline generate`'s `mode` chooses how far generation goes. It is set per target in [`comline.toml`](../../reference/congregation/comline-toml.md) (`[generate] mode` or @@ -12,8 +12,8 @@ in [`comline.toml`](../../reference/congregation/comline-toml.md) (`[generate] m | `mode` | Output | Status | |---|---|---| -| `code` | bare source files you place in a project yourself | built | -| `lib` | a buildable package (manifest + module tree) | rust, typescript | +| `code` | bare source files you place in a project yourself | Rust, TypeScript | +| `lib` | a buildable package (manifest + module tree) | Rust, TypeScript | | `dylib` | a package built with a **stable ABI** and compiled to a dynamic library any language's runtime can load | planned | ## `code` — source you wire in yourself @@ -38,7 +38,7 @@ This is the common case, and where every language generator starts. A whole buildable package: the manifest with the right dependencies, a `src/` module tree, and a top-level module listing every schema. The result is -something you `cargo add` / `pip install` / `luarocks install`, or publish. +something you `cargo add` or `npm install`, or publish. **rust**: `/rust/` gets a `Cargo.toml` (package name from the congregation, `serde` dependency), `src/lib.rs`, and `src/.rs` per @@ -123,9 +123,9 @@ the object carries. ## Status -`lib` is built for Rust (see [Language Guides → -Rust](../../languages/rust/index.md) for using the result). `dylib` is -still planned for every language, Rust included. The open questions for +`code` and `lib` are built for Rust and TypeScript (see the +[Language Guides](../../languages/index.md) for using the result). `dylib` is +still planned for every language. The open questions for `dylib` — how the generated library is named and versioned, and what identity the runtime checks when it loads one — are worked through in [Consumer generation configuration](../../design/consumer-generation-config.md). diff --git a/docs/docs/guide/extending/encodings.md b/docs/docs/guide/extending/encodings.md index 2ca83d2..dad3155 100644 --- a/docs/docs/guide/extending/encodings.md +++ b/docs/docs/guide/extending/encodings.md @@ -5,6 +5,15 @@ wraps a call or a reply around those bytes (`datagram@2`, `jsonrpc@1`). You can add either one. This page covers the whole path: declare it, profile it, prove it, map it, use it. +```mermaid +flowchart LR + D["1. declare
config.idp encodings"] --> P["2. profile
its facts"] + P --> V["3. vectors
prove an implementation"] + D --> U["4. use
wire, @framing"] + U --> M["5. map
a type per language"] + M --> G["generated code
names that type"] +``` + ## 1. Declare it A package declares the encodings it defines under `encodings` in `config.idp`: @@ -140,6 +149,8 @@ exits with `1`. The flags, limits and report are in the - `comline conformance` runs every encoding the package declares. An encoding with no vectors file is an error, so name the ones you mean with `--encoding`. +- Official encodings are proven in the runtime's corpus. Run them with + `--corpus /runtime/conformance`. ### Shipping no vectors @@ -162,8 +173,6 @@ gives up encoding conformance (framing `com.acme.lines@1` ships no conformance v Adding the vectors file restores the guarantee, and the marker is then needless. -- Official encodings are proven in the runtime's corpus. Run them with - `--corpus /runtime/conformance`. ## 4. Use it in a schema diff --git a/docs/docs/guide/idl/annotations.md b/docs/docs/guide/idl/annotations.md new file mode 100644 index 0000000..fa3319c --- /dev/null +++ b/docs/docs/guide/idl/annotations.md @@ -0,0 +1,57 @@ +# Annotations + +An **annotation** is a `@key = value` line placed directly above what it +applies to: a declaration, a field or a function. + +```ids linenums="1" +@framing = "jsonrpc" +protocol Mail { + @timeout_ms = 1000 + function send(message: Message) -> bool; + + @idempotent + function fetch_inbox() -> Message[]; +} +``` + +## Values + +A value takes one of four forms: + +| Form | Example | +|---|---| +| none: a bare marker | `@idempotent` | +| a single value: an integer, a string, `true` / `false`, or a name | `@timeout_ms = 1000`, `@provider = Client` | +| a list of calls, for validators | `@validators = [StringBounds(min_chars = 3)]` | +| a dictionary of `key = value` entries | `@framing = { name = "jsonrpc", method_names = qualified }` | + +## The annotations Comline knows + +Each one is checked: its value against the form it takes, and its place +against where it applies. A wrong value or a misplaced annotation fails the +build. + +| Annotation | On | What it does | Details | +|---|---|---|---| +| `@validators` | a field, a `type` alias | runs named checks on the value | [Validators](validator.md#using-one-validators) | +| `@deprecated` | a struct, a `type` alias | warns wherever it's used; the value is `true` or a message | [Types](types.md#type-aliases) | +| `@settings` | any declaration | applies a named settings preset | [Settings](settings.md#named-presets-settings) | +| `@framing` | a protocol | how calls are framed on the wire | [Protocols](protocol.md#annotations) | +| `@handshake` | a protocol | what runs before the first call | [Protocols](protocol.md#annotations) | +| `@provider` | a protocol | which side implements it: `Server`, `Client` or `Multiple` | [Calls in both directions](protocol.md#calls-in-both-directions) | +| `@consumer` | a protocol | the other side; follows from `@provider` | [Protocols](protocol.md#annotations) | +| `@unverified` | a protocol | gives up a guarantee, with the reason | [Giving up a guarantee](protocol.md#giving-up-a-guarantee) | +| `@timeout_ms` | a function | how long a client waits for the reply; used by generated Rust | [Protocols](protocol.md#functions) | +| `@idempotent` | a function | marks a call as safe to repeat; advisory, nothing reads it yet | | + +`@framing` and `@unverified` are off until a package allows them in its +[settings](settings.md#the-built-in-default), since each changes what peers +can rely on. + +Any other `@key` is allowed and kept in the frozen schema, so a tool of your +own can read it. Comline doesn't check it. + +## In the editor + +Hovering an annotation shows what it does, where it applies, its default, and +what reads it. Completion offers the annotations that fit where the cursor is. diff --git a/docs/docs/guide/idl/index.md b/docs/docs/guide/idl/index.md index 9188eed..3fa1739 100755 --- a/docs/docs/guide/idl/index.md +++ b/docs/docs/guide/idl/index.md @@ -38,10 +38,17 @@ A schema is a flat list of declarations, in any order: | [`protocol`](protocol.md) | a set of callable [`function`s](protocol.md) | | [`error`](error.md) | a named failure with an interpolated `message` and fields, raised by functions with `!` | | [`const`](const.md) | a compile-time constant: `const NAME: type = value` | +| [`type`](types.md#type-aliases) | a second name for a type: `type UserId = u64` | | [`settings`](settings.md) | package/schema-wide authoring policy (`key = value`) — enforced at build time | -| [`validator`](validator.md) | a named, parameterised field check — recorded, not yet enforced | +| [`validator`](validator.md) | a named, parameterised check on a field's value, run by generated code | | [`use`](use.md) | pull declarations in from another schema or package | +## Annotations + +A line `@key = value` above a declaration, field or function configures it: +`@validators`, `@framing`, `@deprecated` and others. See +[Annotations](annotations.md). + ## Comments and docs `//` is a comment. `///` lines are [docstrings](docstrings.md) and attach to the diff --git a/docs/docs/guide/idl/protocol.md b/docs/docs/guide/idl/protocol.md index 4caacc8..2ce1120 100755 --- a/docs/docs/guide/idl/protocol.md +++ b/docs/docs/guide/idl/protocol.md @@ -53,6 +53,17 @@ protocol Sensors { } ``` +```mermaid +sequenceDiagram + participant C as Caller + participant P as Provider + C->>P: report(21.5) + Note over C: carries on at once + C->>P: calibrate() + Note over C: waits + P-->>C: () or an error +``` + | | `function report(…);` | `function calibrate() -> ();` | |---|---|---| | Reply | none: a one-way call | an empty reply | @@ -79,8 +90,9 @@ protocol Mail { } ``` -- **`@framing`** picks the wire framing the generated code uses. Without it, - the protocol takes the package's `wire.framing` from `config.idp` +- **`@framing`** picks the wire framing the generated code uses (what a + framing is: [Call systems](../runtime/call-system.md#the-layers)). Without + it, the protocol takes the package's `wire.framing` from `config.idp` ([Wire defaults](../packages/index.md#wire-defaults)), else `"datagram@2"`, which puts it on a [connection](#calls-in-both-directions). `"jsonrpc"` (`json-rpc`, `jsonrpc-2.0`) emits a @@ -128,6 +140,12 @@ protocol Events { } ``` +```mermaid +flowchart LR + C["client
(dialled)"] -->|"Chat.send"| S["server
(accepted)"] + S -->|"Events.message, Events.ping"| C +``` + - **`@provider = Client`**: the client implements it and the server calls it. **`@provider = Multiple`**: both sides implement it and either can call. - **The framing has to tell requests from replies,** since both ends now send diff --git a/docs/docs/guide/idl/settings.md b/docs/docs/guide/idl/settings.md index 6d10850..8160f3a 100755 --- a/docs/docs/guide/idl/settings.md +++ b/docs/docs/guide/idl/settings.md @@ -64,9 +64,20 @@ struct Payment { ## How the layers compose -For any one declaration, three layers merge in order — package default → -schema's own unnamed block → named preset (if `@settings` is used) — each one -overriding what came before it, recursively, key by key. +For any one declaration, the layers merge in order, each one overriding what +came before it, recursively, key by key: + +```mermaid +flowchart LR + B["built-in default
@framing, @unverified off"] --> P["package
config.idp settings"] + P --> S["schema
settings { }"] + S --> N["preset
@settings = Name"] + N --> D["what applies
to the declaration"] +``` + +The [built-in default](#the-built-in-default) is the base; a package's +`settings` in `config.idp` goes on top of it, then the schema's own unnamed +block, then a named preset if the declaration uses `@settings`. **Merge is the default**; a subtree can opt out with the reserved bare keyword `mode = replace`, which drops the layer below entirely instead of @@ -160,9 +171,7 @@ peers rely on is the same kind of decision. It is the base layer every package starts from, whether or not the package writes a `settings` block of its own: writing an unrelated setting does not -drop it. (It used to — the default was installed only when a package wrote -no `settings` key at all, so writing any setting silently removed the -protection entirely.) A package that genuinely wants no built-in policy can +drop it. A package that genuinely wants no built-in policy can drop the layer with `mode = replace` at the root, the same escape hatch any other layer has. diff --git a/docs/docs/guide/idl/structure.md b/docs/docs/guide/idl/structure.md index d69f169..2d32596 100755 --- a/docs/docs/guide/idl/structure.md +++ b/docs/docs/guide/idl/structure.md @@ -35,53 +35,27 @@ struct Message { ## Field types -| Kind | Examples | -|---|---| -| integers | `s8 s16 s32 s64`, `u8 u16 u32 u64` | -| floats | `f32`, `f64` | -| other | `bool`, `string` | -| named | another `struct` / `enum`, scoped as `pkg::module::Type` | -| array | `Message[]` (any length), `u8[16]` (fixed) | -| union | `union(string u32)` | +A field can be any [type](types.md): a primitive (`u32`, `bool`, `string`), a +`struct`, `enum` or `type` alias by name (scoped as `pkg::module::Type`), an +array (`Message[]`, `u8[16]`) or a union (`union(string u32)`). -Use `string` for text; `str` and `String` are not Comline types — see -[Text](types.md#text-string). +## Docstrings and annotations -## Docstrings - -A [docstring](docstrings.md) (`///` lines) and `@key=value` annotations attach to -a struct or an individual field. +A [docstring](docstrings.md) (`///` lines) documents the struct or a field. +[Annotations](annotations.md) (`@key = value`) above a field attach to it, as +[`@validators`](validator.md) does; above the struct, they attach to the +struct, as `@deprecated` and [`@settings`](settings.md#named-presets-settings) +do. ```ids linenums="1" /// A message routed through the mail protocol. +@deprecated = "use Envelope" struct Message { + /// Plain-text body, UTF-8. + @validators = [StringBounds(min_chars = 1)] body: string /// Shown in the client's list view. optional subject: string } ``` - - -## Annotations - -An annotation's value is one of three forms: - -- **A single value:** an integer, a string, `true`/`false` or a name - (`@timeout_ms = 1000`). -- **A list of calls,** for [`@validators`](validator.md) - (`@validators = [StringBounds(min_chars = 3)]`). -- **A dictionary of `key = value` entries,** for an annotation with several - settings, such as - [`@framing`](protocol.md#annotations) - (`@framing = { name = "jsonrpc", method_names = qualified }`). - -```ids -struct Message { - @optional // <- This is an annotation - body: string -} -``` - -TODO: annotations might not apply to structures, only fields - diff --git a/docs/docs/guide/idl/types.md b/docs/docs/guide/idl/types.md index 5784f67..5fb1608 100644 --- a/docs/docs/guide/idl/types.md +++ b/docs/docs/guide/idl/types.md @@ -4,8 +4,8 @@ Primitives: | Group | Types | |---|---| -| signed integers | `s8` `s16` `s32` `s64` | -| unsigned integers | `u8` `u16` `u32` `u64` | +| signed integers | `s8` `s16` `s32` `s64` `s128` | +| unsigned integers | `u8` `u16` `u32` `u64` `u128` | | floats | `f32` `f64` | | boolean | `bool` | | text | `string` (see below) | @@ -47,8 +47,10 @@ check every value of it, and `@deprecated`, which warns wherever it's used. | Comline | Rust | TypeScript | |---|---|---| -| `s8` … `s64` | `i8` … `i64` | `number` | -| `u8` … `u64` | `u8` … `u64` | `number` | +| `s8` … `s32` | `i8` … `i32` | `number` | +| `u8` … `u32` | `u8` … `u32` | `number` | +| `s64`, `s128` | `i64`, `i128` | `bigint` | +| `u64`, `u128` | `u64`, `u128` | `bigint` | | `f32` / `f64` | `f32` / `f64` | `number` | | `bool` | `bool` | `boolean` | | `string` | `String` | `string` | @@ -58,8 +60,8 @@ check every value of it, and `@deprecated`, which warns wherever it's used. | `union(string u32)` | not supported yet | `string \| number` | | `optional x: T` | `Option` | `x?: T` | -TypeScript's `number` holds integers exactly only up to 2^53, so an `s64` or -`u64` above that loses precision there. +TypeScript's `number` holds integers exactly only up to 2^53, so the 64- and +128-bit integers are `bigint` there, in generated types and in both codecs. ### Fixed arrays diff --git a/docs/docs/guide/idl/validator.md b/docs/docs/guide/idl/validator.md index f93ac66..937f196 100644 --- a/docs/docs/guide/idl/validator.md +++ b/docs/docs/guide/idl/validator.md @@ -136,12 +136,20 @@ protocol Accounts { } ``` +```mermaid +flowchart LR + A["Handle's validators
StringBounds(2, 8)"] --> F["the field's own
@validators"] + F --> OK["value accepted"] +``` + +A value of the alias is checked by the alias's validators first (an alias of +an alias: innermost first), then by the field's own. + - The alias stays the type it stands for: `Handle` is still a `String` in generated Rust. Adding a validator never changes a type's API or its wire format. - The validator's `for` target is checked once, against what the alias resolves to. -- A field of the alias runs the alias's checks first, then its own. - An alias of an alias collects the checks along the chain. - An array of the alias (`Handle[]`) checks each element. - An alias is the only way to check a **function argument**: arguments take no diff --git a/docs/docs/guide/index.md b/docs/docs/guide/index.md index 319dcb1..5d0c4c8 100644 --- a/docs/docs/guide/index.md +++ b/docs/docs/guide/index.md @@ -1,58 +1,76 @@ -# Guide Overview - -Comline takes a schema you write by hand and carries it through a fixed pipeline: - -**parse → resolve imports → validate → freeze → generate** +# Guide Overview + +Comline takes a schema you write by hand and carries it through a fixed +pipeline. Each stage has its own section in this guide; this page is the map. + +```mermaid +flowchart LR + subgraph author["You write"] + IDP["config.idp
the package"] + IDS["src/*.ids
the schemas"] + end + IDP --> P["parse"] + IDS --> P + P --> I["resolve imports
and dependencies"] + I --> V["validate"] + V -->|"comline build"| F["freeze"] + F --> CAS[(".comline/
version history")] + V -->|"comline generate"| G["generate"] + TOML["comline.toml
where code goes"] --> G + G --> OUT["generated/"] +``` -Each stage has its own page in this guide; this is the map. +`comline check` stops after validating. `comline build` freezes a version; +`comline generate` writes code without freezing anything. -## 1. Writing a Schema +## 1. Write a schema -A project is a directory with a package manifest (`config.idp`) and one or more -`.ids` [Schema / IDL](idl/index.md) files under `src/`. The schema language defines -[structures](idl/structure.md), [enums](idl/enum.md), [protocols](idl/protocol.md), -errors and [settings](idl/settings.md), with [docstrings](idl/docstrings.md) -attached to any of them. +A package is a directory with a manifest (`config.idp`) and one or more `.ids` +[schema](idl/index.md) files under `src/`: ``` comline new my-api # scaffold config.idp + comline.toml + src/main.ids ``` -## 2. Checking and Compiling +A schema declares [structs](idl/structure.md), [enums](idl/enum.md), +[errors](idl/error.md), [constants](idl/const.md), +[type aliases](idl/types.md#type-aliases), [protocols](idl/protocol.md) and +[validators](idl/validator.md), with [docstrings](idl/docstrings.md) on any of +them. [Annotations](idl/annotations.md) and [settings](idl/settings.md) tune +how they're checked and generated. -`comline check` parses, resolves imports and validates. +## 2. Check and build -`comline build` does that -and then **freezes** the result: the schema becomes a set of immutable [Intermediate Representation](ir/index.md) -written into a Content Addressable Store, [`.comline/`](ir/cas.md), as a new -commit in an append-only chain. +`comline check` parses, resolves imports and validates. `comline build` does +that and then **freezes** the result into an +[intermediate representation](ir/index.md), stored in the +[content-addressable store](ir/cas.md) under `.comline/` as a new commit in an +append-only chain. -Each build is recorded under a pre-release version leading to the next release; +Each build is a pre-release version leading to the next release; `comline release` creates that version, bumped by the largest change since the -last one — see [Versioning rules](../reference/versioning.md). +last one. See [Versioning rules](../reference/versioning.md). ## 3. [Generate code](codegen/generating-code.md) -`comline generate` turns the compiled schema into source for a target language. -*What* languages a package supports is declared in `config.idp`; *where* the code -lands and in what layout is the consumer's choice, in -[`comline.toml`](../reference/congregation/comline-toml.md). One schema history can be -generated across [several languages](../design/codegen-by-language.md) and -[several past versions](../design/multi-version-generation.md) at once. - -## 4. [Runtime](runtime/index.md) +`comline generate` turns the schemas into source for each target language. +The package declares *which* languages it supports, in `config.idp`; whoever +generates decides *where* the code lands, in `comline.toml` +([the two files](codegen/generating-code.md#the-two-files)). One history can be +generated for [several past versions](../design/multi-version-generation.md) +at once. -At run time a runtime library handles transport, message parsing and routing a -[protocol](idl/protocol.md) call to its implementation, over a pluggable -[call system](runtime/call-system.md) (JSON-RPC, a compact binary framing, or -your own). +## 4. [Run it](runtime/index.md) -## Where things are owned +Generated code links against a runtime library, which runs the handshake, +frames each call and encodes its values: the [call system](runtime/call-system.md). +The [language guides](../languages/index.md) show it in Rust and TypeScript. -| Concern | File | Owner | Frozen into CAS | -|---|---|---|---| -| Schema shape, package identity, which languages are supported | `config.idp` | package **author** | yes | -| Output location, layout, which versions to emit | `comline.toml` | **consumer** | no | +## Then -See [Packages & dependencies](packages/index.md) for the manifest, and the -[Design](../design/index.md) section for the reasoning behind the split. +- [Packages & dependencies](packages/index.md): the manifest, wire defaults, + and importing other packages. +- [Documenting your schemas](documenting.md): pages and a site from your + docstrings. +- [Extending Comline](extending/index.md): your own formats, framings, + handshakes, generators, transports and middleware. diff --git a/docs/docs/guide/ir/cas.md b/docs/docs/guide/ir/cas.md index 191f62d..da00393 100755 --- a/docs/docs/guide/ir/cas.md +++ b/docs/docs/guide/ir/cas.md @@ -290,17 +290,9 @@ now. Then it adds one commit on top of the latest build: version, so the release gets a root tree of its own. - Its `version` has no pre-release part, and it carries a [release record](#release-record). -- The bump is the highest the changes since the last release call for. - `--raise --reason ` goes higher and never lower, and the reason - is required. The first release is `0.1.0`. `comline release 1.0` moves a - `0.x` package to `1.0.0`. -- It refuses when there is nothing to release: the latest commit is already a - release, or nothing since the last release calls for a bump (a raise releases - anyway). - -Before 1.0 the minor version plays the major's part: a breaking change goes -`0.3.2` to `0.4.0`, and anything else `0.3.2` to `0.3.3`. A `0.0.z` version -moves to `0.1.0` on any change. +- The bump follows the [versioning rules](../../reference/versioning.md#builds-and-releases): + the highest the changes since the last release call for, raised with + `--raise` if asked. It refuses when there is nothing to release. See `comline release` in the [CLI reference](../../reference/cli.md). diff --git a/docs/docs/guide/ir/index.md b/docs/docs/guide/ir/index.md index 14cb2a6..89f72c0 100755 --- a/docs/docs/guide/ir/index.md +++ b/docs/docs/guide/ir/index.md @@ -10,18 +10,19 @@ looks at `.ids` source again. A compiled schema becomes a sequence of **frozen units** — one per declaration: -- `Namespace` — the schema's namespace path -- `Structure`, `Enum`, `Error` — the data shapes, with their fields / variants, - types, indices and defaults -- `Protocol` — functions with typed parameters and results -- `Validator`, `Settings` — rules and per-schema configuration +- `Namespace` and `Import`: where the schema sits, and what it brings in +- `Struct`, `Enum`, `Error`, `TypeAlias`, `Constant`: the data shapes, with + their fields or variants, structured types, defaults and annotations +- `Protocol`: its functions, with typed arguments, returns and the errors each + raises +- `Validator`, `Settings`: checks, and the schema's settings blocks The package manifest lowers to units too (`SpecificationVersion`, `Dependency`, `CodeGeneration`, `PublishRegistry`), so a frozen version carries both the schema API and the declarations around it. "Frozen" is literal: a unit is immutable once written, and it is addressed by the -hash of its content. +hash of its content. [What's stored](cas.md#objects) has the exact layout. ## Where it goes diff --git a/docs/docs/guide/packages/index.md b/docs/docs/guide/packages/index.md index 63d3db6..53b8bcd 100644 --- a/docs/docs/guide/packages/index.md +++ b/docs/docs/guide/packages/index.md @@ -27,18 +27,10 @@ code_generation = { } ``` -| Field | Meaning | -|---|---| -| `congregation` | the package name — a Comline identifier (letters, digits, `_`) | -| `id` | the package's identity: 32 lowercase hex digits, random. `comline new` writes it, and `comline build` adds one to a package without it, as one line after `congregation`, with a note. A rename keeps it; a copy of the package must change it. Changing it is a major change. See [Compatibility facts](../../reference/versioning.md#compatibility-facts) | -| `specification_version` | which version of the schema language these files are written against | -| `code_generation.languages` | the **capability list**: `language#lang_version` targets this package *can* be generated as, each version the oldest toolchain the output must build on, which `comline build` proves ([Language versions](../codegen/generating-code.md#language-versions)). Bare entries — `= {}` is required and takes no options. | -| `dependencies` | other packages this one imports — see [Dependencies](#dependencies) | -| `settings` | package-wide authoring policy, enforced at build time — see [Settings](../idl/settings.md) | -| `wire` | the package's wire format and framing — see [Wire defaults](#wire-defaults) | -| `handshake` | what a protocol with no `@handshake` runs before its first call: `"comline@1"` (the default) or `"none"` — see [Giving up a guarantee](../idl/protocol.md#giving-up-a-guarantee) | -| `encodings` | the formats, framings and handshakes this package defines, each with a profile — see [Extending Comline](../extending/encodings.md) | -| `publish_registries` | named registries this package publishes to (planned — [see below](#publishing-planned)) | +Every key, its syntax and what it does is in the +[`config.idp` reference](../../reference/congregation/config-id.md). The ones +with more to them have their own sections below: [wire defaults](#wire-defaults) +and [dependencies](#dependencies). The manifest is **frozen**: it lowers to IR units and is content-addressed into every built version, so a published version carries its own declarations. It @@ -157,6 +149,16 @@ decides where the package comes from: | Git | `version`, `uri`, `commit` (optional `hash`) | cloned once per pin into `.comline/deps-cache/`, using the `git` binary | | Registry | `version`, `uri` (optional `hash`, `signature`) | **not supported yet** — there's no registry server, so `comline check` / `build` reject it | +```mermaid +flowchart LR + E["config.idp
dependencies"] -->|path| L["../shared-types"] + E -->|"git commit"| G[".comline/deps-cache/"] + L --> C["compiled under
its own settings"] + G --> C + C -->|"hash checked"| U["your schemas
use shared_types::…"] + C -->|"comline build"| V[("vendored into
the version")] +``` + `comline check` and `comline build` resolve every dependency, then compile it like any other package. Your package uses that compile as it is: a dependency's schemas aren't compiled again under your settings, its own diff --git a/docs/docs/guide/runtime/call-system.md b/docs/docs/guide/runtime/call-system.md index eed647f..f99bc9f 100644 --- a/docs/docs/guide/runtime/call-system.md +++ b/docs/docs/guide/runtime/call-system.md @@ -1,59 +1,97 @@ # Call systems & serialization The **call system** (sometimes *CaSy*) is the part of the [runtime](index.md) -that turns an invocation of a [protocol](../idl/protocol.md) function into bytes -on a transport and back again. - -It is pluggable along two axes: - -- **Call format (framing)** — how a request/response is framed. Three ship - today: **`datagram@2`** (the default), a compact binary framing for - [connections](#connections); its lockstep first version, **`datagram`**; - and [JSON-RPC 2.0](https://www.jsonrpc.org/specification). Framing is - chosen when code is generated (a protocol's `@framing` annotation, else the - package's `wire.framing`), not swapped at the call site. -- **Message serialization (wire format)** — how the - [structures](../idl/structure.md) carried by a call are encoded: - **[`packed@1`](../../reference/wire/packed.md)** (the default: fixed-width - fields, no tags or names), JSON, and - [MessagePack](https://msgpack.org/) (Rust only, for now). The package - declares it as `wire.format`, and generated code names it `Format`; a - connection made by hand can pass another. - -Both are declared in `config.idp`'s `wire` key -([Wire defaults](../packages/index.md#wire-defaults)). - -The runtime handles routing between a caller and the protocol implementation; -you pick the formats that fit your deployment. +that turns a call to a [protocol](../idl/protocol.md) function into bytes on a +transport and back again. This page is the one place that explains its parts; +the rest of the docs link here. + +## The layers + +```mermaid +flowchart LR + subgraph caller["Caller"] + A["send(text)"] --> F1["format
encodes the arguments"] + F1 --> R1["framing
wraps them as a request"] + end + R1 --> T(["transport
TCP, in-memory, yours"]) + T --> R2 + subgraph provider["Provider"] + R2["framing
reads the request"] --> F2["format
decodes the arguments"] + F2 --> H["your handler"] + end +``` + +The reply goes back the same way. Each layer is chosen separately: + +| Layer | What it does | Built in | Chosen by | +|---|---|---|---| +| **Format** | turns values into bytes | **`packed@1`** (the default: fixed-width fields, no tags or names, see [`packed@1`](../../reference/wire/packed.md)), `json@1`, `msgpack@1` (Rust only, for now) | the package's `wire.format` | +| **Framing** | wraps a call or a reply around those bytes | **`datagram@2`** (the default, for [connections](#connections)), its lockstep first version `datagram`, and [JSON-RPC 2.0](https://www.jsonrpc.org/specification) | a protocol's `@framing`, else the package's `wire.framing` | +| **Handshake** | checks both ends agree, before the first call | **`comline@1`** (the default), `none` | a protocol's `@handshake`, else the package's `handshake` | +| **Transport** | carries the frames | TCP and in-memory in Rust, in-memory in TypeScript | your code, when you connect | + +The format, framing and handshake are part of the schema: both ends have to +agree on them, so they're declared once, in `config.idp` +([Wire defaults](../packages/index.md#wire-defaults)), and generated code uses +them without being told. What's built in, and what each one guarantees, is in +[Official encodings](../../reference/wire/official-encodings.md); you can +[add your own](../extending/encodings.md). The transport doesn't change the +bytes, so it's plain code you pass in. + +JSON-RPC carries JSON, so a protocol on it uses `json@1` whatever the package +says. Which combinations each language supports is on its +[reference page](../../reference/languages/index.md#runtimes-side-by-side). ## Handshake -Before either side makes a real call, `connect` / `serve` run the -protocol's handshake. Comline's own, `comline@1` (the default), exchanges a -small fixed-size frame declaring the -schema fingerprint both ends were generated from, plus the wire format -and framing in use. A mismatch on any of those — different schema -versions, one end MessagePack and the other JSON, incompatible framing — -is refused immediately, rather than surfacing later as a confusing decode -error partway through a real call. Capability bits are exchanged too, but -a difference there is advisory, not a refusal. - -A protocol talking to a peer that can't take part — a legacy service, say — -declares `@handshake = "none"` and says why with `@unverified` (see [Giving up -a guarantee](../idl/protocol.md#giving-up-a-guarantee)); its generated -`connect` / `serve` then skip the exchange. In the runtimes the handshake is -a trait (`Handshake`; an interface in TypeScript), so a custom one can be -passed to `Client::connect_with` / `Server::serve_with` by hand. Hand-wired -code can also skip it at the call site with `Client::new_unverified` / -`Server::serve_unverified` (`Client.unverified` / `serveUnverified` in -TypeScript) — named so that skipping is as visible there as in a schema. +Before the first call, `connect` and `serve` run the protocol's handshake. +`comline@1` exchanges one small fixed-size frame each way: + +```mermaid +sequenceDiagram + participant C as Client + participant S as Server + C->>S: hello: schema fingerprint, format, framing + S->>C: hello: schema fingerprint, format, framing + alt all three match + C->>S: first call + else any differs + Note over C,S: both refuse, before any call is made + end +``` + +A mismatch (different schema versions, one end MessagePack and the other JSON, +a different framing) is refused at once, rather than surfacing later as a +confusing decode error partway through a real call. Capability bits are +exchanged too, but a difference there is advisory. + +A protocol talking to a peer that can't take part (a legacy service, say) +declares `@handshake = "none"` and says why with `@unverified`; see +[Giving up a guarantee](../idl/protocol.md#giving-up-a-guarantee). In the +runtimes the handshake is a trait (`Handshake`; an interface in TypeScript), +so a [custom one](../extending/handshakes.md) can be passed to +`Client::connect_with` / `Server::serve_with`. Hand-wired code can also skip it +at the call site with `Client::new_unverified` / `Server::serve_unverified` +(`Client.unverified` / `serveUnverified` in TypeScript), named so that skipping +is as visible there as in a schema. ## Connections -A client and a server (the `datagram` framing) work in lockstep: the client -sends a request and reads the next frame as its reply. A **connection**, which +With the lockstep `datagram` framing, a client sends a request and reads the +next frame as its reply, one protocol per transport. A **connection**, which every protocol uses by default, lifts that limit: +```mermaid +sequenceDiagram + participant C as Client + participant S as Server + C->>S: request 1: Chat.send("hi") + S->>C: request 7: Events.message("hi") + Note over C: serves the server's call
while waiting on its own + C->>S: reply to 7: true + S->>C: reply to 1: 42 +``` + - **Either side can call,** since `datagram@2` marks each frame as a request or a reply. - **Several calls can be in flight.** A reply is matched to its call by @@ -71,9 +109,8 @@ the package, which also fixes the protocols' numbers. Which protocols run on a connection is the schema's choice: see [Calls in both directions](../idl/protocol.md#calls-in-both-directions). -## Concrete API +## In code -This page stays the language-agnostic concept; [Language -Guides](../../languages/index.md) has the concrete, per-language API — -constructing a wire format, picking a framing, and what the handshake +[Language Guides](../../languages/index.md) has the concrete API in each +language: constructing a format, picking a framing, and what the handshake looks like in code. diff --git a/docs/docs/guide/runtime/index.md b/docs/docs/guide/runtime/index.md index 14a8f8e..8534efa 100755 --- a/docs/docs/guide/runtime/index.md +++ b/docs/docs/guide/runtime/index.md @@ -6,24 +6,37 @@ that does this, so each project does not reinvent it. ## Per-language runtimes -Each language ships its own runtime implementation — not a thin shim -calling into a shared core, but a from-scratch implementation of the same -contract (handshake, framing, wire format), wire-compatible with the -others. The Rust runtime (`comline-runtime`) is the reference; a -TypeScript peer and a Rust peer generated from the same schema negotiate -the same handshake and speak the same request/response bytes, because -each runtime's framing and handshake code is cross-checked against the -others' reference vectors, not because one calls into the other at run -time. - -What that means in practice: there is no FFI boundary between a -TypeScript program and a "core" written in Rust. Each runtime is a normal -dependency in its own language, built and shipped like any other -library. +Each language has its own runtime, a normal dependency built and shipped like +any other library: `comline-runtime` for Rust, `@comline/runtime` for +TypeScript. Generated code calls into it; you rarely do. + +```mermaid +flowchart TB + H["your handlers and calls"] --> GC["generated code
types, senders, dispatch"] + GC -->|Rust| RS["comline-runtime"] + GC -->|TypeScript| TS["@comline/runtime"] + TS -->|"WebAssembly,
through the C ABI"| RS + RS --> TR["transport
TCP, in-memory, yours"] + TS --> TR +``` + +The protocol logic that has to be byte-identical everywhere (framing, request +ids, the handshake's frames) is written once, in Rust: `comline-runtime` is +the [shared runtime core](../../proposals/shared-runtime-core/index.md), and +its [C ABI](../../reference/c-abi.md) exposes it to other languages. The +TypeScript runtime loads it as WebAssembly for its built-in framings, and falls +back to its own TypeScript implementation where the core can't load (a strict +content security policy, say) or for a framing of your own; both paths behave +the same and are tested the same. Codecs, generated types and transports stay +in each language. + +Every runtime is checked against the same +[conformance vectors](../../reference/conformance.md), so a Rust peer and a +TypeScript peer generated from one schema speak the same bytes. See [Language Guides](../../languages/index.md) for the practical, -per-language next step — spawning a client, serving a protocol, and the -concrete framing/wire-format API. This page stays the conceptual +per-language next step: spawning a client, serving a protocol, and the +concrete framing and wire-format API. This page stays the conceptual overview. ## Call system diff --git a/docs/docs/index.md b/docs/docs/index.md index ed4b386..e924578 100755 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -2,50 +2,59 @@ !!! warning "Early days" Comline is in heavy development and changes constantly. Much of this - documentation is working design material rather than a polished reference — - parts are dense and read closer to notes than prose. It will be rewritten - for people, end to end, once the design settles. - -Comline is an agnostic schema, RPC/IPC (or any other similar terminologies) -library, similar to ones like: - - - [Protobuf](https://protobuf.dev/), [Capn'Proto](https://capnproto.org/), [Fuchsia IDL (FIDL)](https://fuchsia.dev/fuchsia-src/get-started/sdk/learn/fidl) - - [Apache Avro](https://avro.apache.org/), [Apache Thrift](https://thrift.apache.org/docs/idl) - - -## What constitutes Comline? - -### [Interface Definition Language (IDL or also named Schema)](./guide/idl/index.md) -Some of the distinct differences to other libraries is that this -[IDL/Schema](./guide/idl/index.md) allows to be more flexible, optional or robust, -detailed and specific when setting options, rules, message structures, -protocols (named services in protobuf, interfaces in capn'proto) and so on... - - -### [Intermediate Representation (IR)](./guide/ir/index.md) -Schemas compile into an Intermiate Representation that solidifies the structure as a sort of "mini-spec", -which then can output to other formats like a rpc schema such as -[OpenRPC](https://open-rpc.org/), a custom text format, or better a binary -compact format like [msgpack](https://msgpack.org/) which is ideal to reduce wire load - - -### [Code Generation (Codegen)](./guide/codegen/index.md) -Codegen is a common feature that similar libraries have, however -some of them lack details at development time and might make the experience -more indicated towards "try-it-out" and not provide so much detail, - -Comline tries to provide more language specific generation that fits as best, -detailed and information specific to them as possible, be it for compiled -or dynamic languages, it's believed to be important to have as much information -detail as possible at development type. - - -### [Runtime & call system](./guide/runtime/index.md) -A runtime handles transport, message parsing and routing a call to its -implementation. The **call system** part of it is pluggable: specs like -[json-rpc](https://www.jsonrpc.org/) can be used, or a custom format that is -binary compact like msgpack, or any custom call format you need to implement. - -The message serialization is likewise specifiable, so you can use any data -format, from JSON to other schema libraries' formats, or once again binary -minimalist formats like msgpack. + documentation is working design material rather than a polished reference: + parts are dense and read closer to notes than prose. + +Comline is a schema language and RPC toolkit. You describe your data and the +calls between programs once, in a schema, and Comline checks it, versions it, +and generates typed code for each language that talks over it. It sits in the +same family as [Protocol Buffers](https://protobuf.dev/), +[Cap'n Proto](https://capnproto.org/), +[FIDL](https://fuchsia.dev/fuchsia-src/get-started/sdk/learn/fidl), +[Apache Avro](https://avro.apache.org/) and +[Apache Thrift](https://thrift.apache.org/docs/idl). + +```mermaid +flowchart LR + S["schema
.ids files"] --> C["comline build
check and freeze"] + C --> V[("version history
.comline/")] + C --> G["comline generate"] + G --> R["Rust code"] + G --> T["TypeScript code"] + R <-->|"calls over a
transport"| T +``` + +## What makes up Comline + +### [The schema language](./guide/idl/index.md) +Structs, enums, errors, constants and protocols, written in `.ids` files. +Beyond the shapes, a schema says how values are checked +([validators](./guide/idl/validator.md)), how calls travel +([framing and format](./guide/runtime/call-system.md)), and which rules a +package enforces on itself ([settings](./guide/idl/settings.md)). + +### [Versions you don't write by hand](./reference/versioning.md) +`comline build` compiles the schemas into a frozen +[intermediate representation](./guide/ir/index.md) and stores it in +`.comline/`, an append-only history. The version number follows from what +changed since the last release, so a breaking change can't ship as a patch. + +### [Generated code](./guide/codegen/index.md) +`comline generate` writes idiomatic code for each target language: types, a +trait or interface per protocol, clients, servers and connections. Rust and +TypeScript are supported today, each in two forms: source files to drop into a +project, or a buildable package. + +### [A runtime per language](./guide/runtime/index.md) +Generated code links against a small runtime that runs the handshake, frames +calls and encodes values. The pieces are pluggable: the +[wire format](./reference/wire/official-encodings.md), the framing, the +handshake and the transport, including [your own](./guide/extending/index.md). + +## Where to start + +- [Getting started](getting-started.md): install the CLI and build a first + package. +- [Guide overview](guide/index.md): the whole pipeline, stage by stage. +- [Language guides](languages/index.md): using the generated code in Rust or + TypeScript. diff --git a/docs/docs/languages/index.md b/docs/docs/languages/index.md index 23dfc6a..10c9f05 100644 --- a/docs/docs/languages/index.md +++ b/docs/docs/languages/index.md @@ -12,29 +12,21 @@ Each language gets its own tree. Pick one from the tabs on the left. | Language | Generator | Modes | Runtime | |---|---|---|---| -| [Rust](rust/index.md) | shipped | `code`, `lib` | `comline-runtime`, sync, `no_std`-first | -| [TypeScript](typescript/index.md) | shipped | `code` | `@comline/runtime`, async/Promise-based | +| [Rust](rust/index.md) | shipped | `code`, `lib` | `comline-runtime`, `no_std`-first; sync, or async with `async = true` | +| [TypeScript](typescript/index.md) | shipped | `code`, `lib` | `@comline/runtime`, async/Promise-based | | Python, Lua, Luau, C | none yet | — | — | Python, Lua, Luau and C appear in example `config.idp` manifests and design sketches, but no `comline-codegen-*` crate exists for them today — there is nothing to generate and no runtime to link against. -## Capability matrix +## What each runtime speaks -What each shipped runtime actually supports, independent of what the call -system [could in principle](../guide/runtime/call-system.md) support: - -| | Transports | Wire formats | Framings | -|---|---|---|---| -| Rust | in-memory, TCP | packed, JSON, MessagePack | datagram@2, datagram, JSON-RPC | -| TypeScript | in-memory only | packed, JSON | datagram@2, datagram, JSON-RPC | - -Not every combination of wire format and framing ships — JSON-RPC's own spec -is JSON text, so a `jsonrpc` + `msgpack`/`packed` pairing doesn't exist on -either side. MessagePack is Rust only, until TypeScript gets a codec for it. -`packed@1` and `datagram@2` are the built-in defaults; see -[Wire defaults](../guide/packages/index.md#wire-defaults). +Both runtimes speak every built-in framing, `packed@1` and `json@1`, and are +byte-compatible with each other. MessagePack is Rust only for now, and so is +TCP: TypeScript ships an in-memory transport and you bring your own for +anything else. The full matrix, proven against the conformance corpus, is in +the [language reference](../reference/languages/index.md#runtimes-side-by-side). ## Who maintains this diff --git a/docs/docs/languages/rust/client-and-server.md b/docs/docs/languages/rust/client-and-server.md index eeeaa22..a748e58 100644 --- a/docs/docs/languages/rust/client-and-server.md +++ b/docs/docs/languages/rust/client-and-server.md @@ -106,10 +106,10 @@ use comline_runtime::transport::Tcp; struct MyMail; impl Mail for MyMail { - fn send_message(&self, message: Message) -> Result { + fn send_message(&self, message: Message) -> Result> { Ok(format!("sent: {}", message.body)) } - fn fetch_inbox(&self) -> Result, MailFetchInboxError> { + fn fetch_inbox(&self) -> Result, Fault> { Ok(vec![]) } } diff --git a/docs/docs/languages/rust/concurrency.md b/docs/docs/languages/rust/concurrency.md index b0511c1..ffb97d8 100644 --- a/docs/docs/languages/rust/concurrency.md +++ b/docs/docs/languages/rust/concurrency.md @@ -1,9 +1,15 @@ # Sync, async & concurrency -The runtime itself makes no decision here — as [the overview -page](index.md) covers, it's plain sync Rust, no threads, no `Arc` -anywhere inside it. Everything below is a pattern for *your* code, not a -feature of `comline-runtime`. +For [connections](connections.md), the default, concurrency is the runtime's +job: generate with [`async = true`](connections.md#async) and handlers run as +`async fn`s on your executor, with several calls in flight on one connection. +The blocking `Connection` handles one thing at a time +([Threads](connections.md#threads)). + +This page is about the lockstep `Client` and `Server` that a protocol on the +`datagram` framing gets. They're plain sync Rust, with no threads and no `Arc` +inside, so everything below is a pattern for *your* code, not a feature of +`comline-runtime`. ## Plain sync — the common case @@ -29,25 +35,20 @@ thread::spawn(move || { }); ``` -This isn't just "the safe option because the borrow checker demands it" — -it's a reasonable fit for what's actually happening underneath. [One call -is outstanding at a time](client-and-server.md#connecting-a-client) on -any single `Client` regardless of locking, since a call borrows the -client until its response is consumed. The mutex isn't adding an -artificial bottleneck on top of a runtime that could otherwise pipeline -several calls at once — there's nothing to pipeline yet either way. +The lock costs nothing the lockstep framing wasn't already costing: [one call +is outstanding at a time](client-and-server.md#connecting-a-client) on any +single `Client`, since a call borrows the client until its response is +consumed, and the framing matches a reply to whatever request came before it. -If you need actual concurrent calls in flight, that means several -connections (several `Client`s, each with its own transport), not several -threads contending for one. +For several calls in flight, use a [connection](connections.md#async), which +matches replies by request id, or several `Client`s, each with its own +transport. ## Calling from async code -There's no async client or server today — the crate's own stated -intent (quoted on [the overview page](index.md)) is an async layer -wrapping the sync core behind the `std` feature, later. Until that ships, -calling a sync `Client` from inside an async task means not blocking the -reactor: +The lockstep `Client` has no async form. If you can, move the protocol to a +connection (drop `@framing = "datagram"`) and generate with `async = true`. +Otherwise, call the sync `Client` without blocking the reactor: ```rust let client = Arc::new(Mutex::new(client)); @@ -60,12 +61,9 @@ let body = tokio::task::spawn_blocking(move || { .unwrap()?; ``` -`spawn_blocking` (or your executor's equivalent — `async-std`'s -`spawn_blocking`, a dedicated thread pool) runs the blocking call on a -thread meant for blocking work, so it doesn't stall the async runtime's -other tasks while it waits on the network. Treat this as the bridge until -an async layer exists, not a permanent design — don't build deep -abstractions on top of `spawn_blocking` that you'll need to unwind later. +`spawn_blocking` (or your executor's equivalent) runs the blocking call on a +thread meant for blocking work, so it doesn't stall the async runtime's other +tasks while it waits on the network. ## Serving several connections concurrently diff --git a/docs/docs/languages/rust/framing-and-wire-format.md b/docs/docs/languages/rust/framing-and-wire-format.md index 6ffe65b..168baf0 100644 --- a/docs/docs/languages/rust/framing-and-wire-format.md +++ b/docs/docs/languages/rust/framing-and-wire-format.md @@ -102,16 +102,7 @@ schema, so they get the same `Format`. ## Shipped combinations -What the runtime is proven to speak, feature by feature, is on the -[Rust reference page](../../reference/languages/rust.md#runtime). - -| Framing | `packed@1` | `json@1` | `msgpack@1` | -|---|---|---|---| -| `datagram@2` | ✅ | ✅ | ✅ | -| `datagram` | ✅ | ✅ | ✅ | -| `jsonrpc` | — | ✅ | — | - -JSON-RPC's own spec is JSON text, so a package with a JSON-RPC protocol can't -declare another `wire.format`: that's a build error naming the protocol. -Leaving `wire.format` out is fine — the default then applies only to the -package's other protocols. +Every framing works with every format the runtime has, except JSON-RPC, which +carries JSON only: a package with a JSON-RPC protocol can't declare another +`wire.format`. What the runtime is proven to speak, feature by feature, is on +the [Rust reference page](../../reference/languages/rust.md#runtime). diff --git a/docs/docs/languages/rust/index.md b/docs/docs/languages/rust/index.md index 90fdc9b..37769b4 100644 --- a/docs/docs/languages/rust/index.md +++ b/docs/docs/languages/rust/index.md @@ -16,22 +16,19 @@ worked example): comline-runtime = { git = "https://github.com/ComlineProject/runtime", rev = "…" } ``` -## The one fact everything else here builds on +## Sync by default, async when asked -The runtime is **sync**, `&mut self`-based, and `no_std`-first. There is -no `Arc`, `Rc`, `Mutex`, or thread spawned anywhere inside it — every call -blocks the caller until the response arrives, by design, so it works the -same on a bare-metal target with no allocator as it does in a normal -`std` binary. The crate's own intent, stated in -`contract/dispatch.rs`, is explicit about this being deliberate for now: +The runtime is `no_std`-first and **sync** by default: a call blocks until +its reply arrives, and nothing inside the runtime spawns a thread or takes a +lock, so it works the same on a bare-metal target with no allocator as in a +normal `std` binary. -> Sync — no boxed futures on the `no_std` path. An async server layer -> wraps this behind the `std` feature. - -That layer doesn't exist yet. Concurrency — running calls on several -threads, or from an async runtime — is entirely something *you* build on -top, not something the runtime hands you. [Sync, async & -concurrency](concurrency.md) covers the patterns. +Set [`async = true`](../../reference/congregation/comline-toml.md#async) in +`comline.toml` and connection protocols are generated for the runtime's +**async** driver instead: handlers are `async fn`s, the connection is a future +you spawn on any executor, and handles call the peer from any task +([Connections → Async](connections.md#async)). Protocols on the lockstep +`datagram` framing keep their blocking client and server either way. ## Where to go next @@ -40,5 +37,7 @@ concurrency](concurrency.md) covers the patterns. - [Framing & wire format](framing-and-wire-format.md) — `Packed`, `Json` and `MsgPack`; `Datagram2Framing`, `DatagramFraming` and `JsonRpcFraming`; and how the package's `wire` key picks both. -- [Sync, async & concurrency](concurrency.md) — using a `Client` from more - than one thread, or from async code, today. +- [Connections](connections.md) — the default: every protocol on one + connection, calls in both directions, sync or async. +- [Sync, async & concurrency](concurrency.md) — sharing a lockstep `Client` + between threads, and calling it from async code. diff --git a/docs/docs/languages/typescript/client-and-server.md b/docs/docs/languages/typescript/client-and-server.md index c525c0b..2deef35 100644 --- a/docs/docs/languages/typescript/client-and-server.md +++ b/docs/docs/languages/typescript/client-and-server.md @@ -141,10 +141,8 @@ This is why a function that declares no `!` is still worth calling defensively — and why it needs no `!` to be able to fail. Declare a `!` for the failures a caller should *handle*; leave the rest to this channel. -It also means one bad call no longer costs the connection. Before, a -handler that threw something undeclared unwound the server loop: the -connection died and the caller, who was never sent a frame, waited out its -timeout instead. Now that call is answered and the next one still works. +One bad call doesn't cost the connection: that call is answered with the +failure, and the next one still works. ## Serving diff --git a/docs/docs/languages/typescript/framing-and-wire-format.md b/docs/docs/languages/typescript/framing-and-wire-format.md index 4b8c42c..8d2db5e 100644 --- a/docs/docs/languages/typescript/framing-and-wire-format.md +++ b/docs/docs/languages/typescript/framing-and-wire-format.md @@ -62,16 +62,11 @@ dependency. Whoever encodes the types has the runtime already. ## Shipped combinations -What the runtime is proven to speak, feature by feature, is on the -[TypeScript reference page](../../reference/languages/typescript.md#runtime). - -| Framing | `packed@1` | `json@1` | `msgpack@1` | -|---|---|---|---| -| `datagram@2` | ✅ | ✅ | — no codec yet | -| `datagram` | ✅ | ✅ | — no codec yet | -| `jsonrpc` | — | ✅ | — | - -A TypeScript client and a Rust server (or vice versa) interoperate on any -row both sides support — the handshake, framings and codecs are -byte-compatible with the Rust runtime by design, cross-checked against -its reference vectors. +Every framing works with every format the runtime has, except JSON-RPC, which +carries JSON only: a package with a JSON-RPC protocol can't declare another +`wire.format`. What the runtime is proven to speak, feature by feature, is on +the [TypeScript reference page](../../reference/languages/typescript.md#runtime). + +A TypeScript peer and a Rust peer interoperate on anything both support: the +handshake, framings and codecs are byte-compatible, checked against the same +conformance vectors. diff --git a/docs/docs/languages/typescript/index.md b/docs/docs/languages/typescript/index.md index 8f902cd..62f5f1e 100644 --- a/docs/docs/languages/typescript/index.md +++ b/docs/docs/languages/typescript/index.md @@ -1,8 +1,8 @@ # TypeScript -Status: shipped, `code` mode only — there's no `lib` mode for TypeScript -yet (Rust is the only language that has it; see [Library -generation](../../guide/codegen/library-generation.md)). Generated code +Status: shipped. `comline-codegen-typescript` generates `code` mode (plain +source files) and `lib` mode (an npm package); see [Library +generation](../../guide/codegen/library-generation.md). Generated code depends on **`@comline/runtime`** (MPL-2.0, ESM, Node). Not published to npm yet. Depend on it from a git checkout or workspace @@ -23,7 +23,8 @@ it from async code is simply using it as it's already written, with Part of the runtime is the Rust [shared runtime core](../../proposals/shared-runtime-core/index.md), loaded through wasm (the [C ABI](../../reference/c-abi.md#wasm32-and-the-js-glue)), -and part is still TypeScript. The core is vendored at `runtime` `257bc02` +and part is still TypeScript. The core is vendored into the package, pinned +to a `runtime` revision by `scripts/vendor-core.sh` ([ts#87](https://github.com/ComlineProject/comline-typescript/pull/87)). | Runs on the core, once it has loaded | Stays in TypeScript | @@ -62,5 +63,7 @@ median of 5. They are not a CI gate; see - [Client & server](client-and-server.md) — connecting, serving, and the shape codegen actually emits. +- [Connections](connections.md) — the default: every protocol on one + connection, calls in both directions. - [Framing & wire format](framing-and-wire-format.md) — what's pluggable today and what isn't yet. diff --git a/docs/docs/reference/cli.md b/docs/docs/reference/cli.md index 6976ceb..c944d4e 100644 --- a/docs/docs/reference/cli.md +++ b/docs/docs/reference/cli.md @@ -43,7 +43,7 @@ parse → resolve imports → validate → (freeze into .comline/) → (generate | `comline doc [--out ] [--version ] [--changelog-from ] [--format ] [--open]` | Write the package's documentation for every version (each release, dev build and the unreleased working state), each into `///` with the changelog from the version before it: Markdown pages (an index, one page per module with its `//!` header, every declaration's `///` docs and each protocol's wire summary, a changelog, an mkdocs/zensical `nav.yml`), `--format html` for a self-contained static site with the same content, a search box, a version picker and a page comparing versions, or `--format json` for the documentation model. Built versions are cached, so a run re-renders only the unreleased state and new versions; `--version` documents one built version. Defaults from [`comline.toml`](congregation/comline-toml.md#doc)'s `[doc]`. `comline doc open []` (or `--open` after a render) opens the result with the system's default handler, rendering first if it isn't there. No `.comline/` write. See [comline doc](tools/doc.md). | | `comline conformance --impl [--encoding ]... [--corpus ] [--timeout ]` | Run an implementation of an encoding against its vectors. The package's declared [`encodings`](../guide/extending/encodings.md) ship vectors beside their profiles (`profiles/x.vectors.json`); `--impl` is a process speaking the [ops protocol](conformance.md#the-ops-protocol), run once with a cleared environment, a time limit (`--timeout`, 60 s) and a capped stdout. `--encoding` limits the run to one encoding (repeatable); `--corpus` runs a corpus laid out like the runtime's `runtime/conformance` instead, which is how the official encodings are proven. A failed or unimplemented case, or an unproven profile fact, exits `1`. See [Formats and framings](../guide/extending/encodings.md#3-write-vectors-and-prove-an-implementation). | | `comline clean [--dry-run]` | Remove generated code (the `generate` output root or files). Never touches `.comline/`. | -| `comline reset [--force] [--dry-run]` | **Destructive.** Delete `.comline/` — the whole version history, no undo — *and* generated code. Next `build` starts over at `0.0.1`. On a terminal it shows how many versions will be lost and makes you type `reset`; without a terminal it refuses (exit `2`) unless `--force`. | +| `comline reset [--force] [--dry-run]` | **Destructive.** Delete `.comline/` — the whole version history, no undo — *and* generated code. Next `build` starts over at `0.1.0-dev.1`. On a terminal it shows how many versions will be lost and makes you type `reset`; without a terminal it refuses (exit `2`) unless `--force`. | | `comline completions ` | Print a completion script to stdout. `bash` `zsh` `fish` `powershell` `elvish`. | | `comline targets [--features [--json]]` | Print one `language#version` per line to stdout — the token syntax `config.idp`'s `code_generation.languages` expects, at the generator's floor — then the versions it handles and has verified, for every generator this binary was compiled with. Ignores `--path`: it reflects the build, not a project. `--features` also lists what each generator supports, feature by feature ([Languages](languages/index.md)); `--json` prints that as JSON. | diff --git a/docs/docs/reference/congregation/comline-toml.md b/docs/docs/reference/congregation/comline-toml.md index afd7f85..75d8b2e 100644 --- a/docs/docs/reference/congregation/comline-toml.md +++ b/docs/docs/reference/congregation/comline-toml.md @@ -307,13 +307,3 @@ missing language in `config.idp`. It never runs a generator. In other editors, any TOML language server that reads the `#:schema` directive picks the schema up, such as [Taplo](https://taplo.tamasfe.dev/). - -## Forward-looking - -```toml -# Generate bindings for a registry dependency rather than the local schemas. -# Parsed but not yet operational — needs dependency resolution (core#6). -[generate.dependencies.stdlib] -out = "src/gen/stdlib" -targets = ["rust"] -``` diff --git a/docs/docs/reference/congregation/compliance_features/index.md b/docs/docs/reference/congregation/compliance_features/index.md deleted file mode 100644 index f78bc1c..0000000 --- a/docs/docs/reference/congregation/compliance_features/index.md +++ /dev/null @@ -1,2 +0,0 @@ -# Compliance Features - diff --git a/docs/docs/reference/congregation/config-id.md b/docs/docs/reference/congregation/config-id.md index e69de29..bc2cab2 100644 --- a/docs/docs/reference/congregation/config-id.md +++ b/docs/docs/reference/congregation/config-id.md @@ -0,0 +1,79 @@ +# `config.idp` + +The package manifest: who the package is, what it depends on, which languages +it can be generated as, and the rules and wire defaults its schemas share. It +sits at the package root, beside `src/`. The [package author](index.md) owns +it, and it's frozen into every built version. + +For a walk through it, see [Packages & dependencies](../../guide/packages/index.md). + +```idp +//! The shop's API: orders, customers and payments. +congregation shop +id = "3f2a9c41d07e4b1c8a65f2e0d91b7c34" +specification_version = 1 + +code_generation = { + languages = { + rust#1.85.0 = {} + typescript#5.0 = {} + } +} + +wire = { format = "packed@1" framing = "datagram@2" } + +dependencies = { + shared_types = { + path = "../shared-types" + } +} + +settings = { + annotations = { + framing = { + allowed = true + } + } +} +``` + +## Syntax + +- **The first line** is `congregation `, a Comline identifier (letters, + digits, `_`). Everything after it is `key = value`. +- **Values** are strings (`"packed@1"`), integers, `true` / `false`, bare + names, or dictionaries in braces. Entries in a dictionary are separated by + new lines or spaces; there are no commas. +- **Keys don't take dots.** `.ids` settings blocks accept + `validators.allowed = false`; here the same key is written with braces. +- **Comments** are `//`. `//!` lines at the top + [document the package](../../guide/idl/docstrings.md#module-docs). + +## Keys + +| Key | Value | Details | +|---|---|---| +| `congregation` | the package's name | Renaming it is fine: `id` is what identifies the package. | +| `id` | 32 lowercase hex digits, random | The package's identity. `comline new` writes it, and `comline build` adds one to a package without it. A copy of the package must change it. Changing it is a major change. See [Compatibility facts](../versioning.md#compatibility-facts). | +| `specification_version` | an integer, `1` | The edition of the schema language the files are written in. | +| `code_generation` | `{ languages = { # = {} ... } }` | The languages the package can be generated as, each with the oldest toolchain its output must build on. `comline build` proves each one. See [Language versions](../../guide/codegen/generating-code.md#language-versions). | +| `wire` | `{ format = "..." framing = "..." }` | The format and framing every protocol uses unless it says otherwise. Defaults: `packed@1` and `datagram@2`. See [Wire defaults](../../guide/packages/index.md#wire-defaults). | +| `handshake` | `"comline@1"` or `"none"` | What a protocol with no `@handshake` runs before its first call. Default `comline@1`. See [Giving up a guarantee](../../guide/idl/protocol.md#giving-up-a-guarantee). | +| `dependencies` | a dictionary of entries by name | Other packages this one imports, from a path or a git commit, each optionally pinned by `hash`, and what it accepts of their [`@unverified`](../../guide/idl/protocol.md#giving-up-a-guarantee) protocols (`accept_unverified`). See [Dependencies](../../guide/packages/index.md#dependencies). | +| `settings` | a dictionary, or `::settings::` | The package-wide authoring policy every schema starts from. See [Settings](../../guide/idl/settings.md). | +| `encodings` | a dictionary of entries by name | The formats, framings and handshakes this package defines. See [Formats and framings](../../guide/extending/encodings.md). | +| `publish_registries` | a dictionary of registries by name | Where the package can be published. Planned; see [Publishing](../../guide/packages/index.md#publishing-planned). | + +An unknown key, a value of the wrong kind, or a dependency that doesn't +resolve fails `comline check` and `comline build`, and the editor reports it +as you type. + +## What's frozen, and what changes the version + +`comline build` lowers the manifest to IR units and stores them with every +version, so a version carries its own declarations. A change to the manifest +counts like a schema change; the rules are in +[Manifest changes](../versioning.md#manifest-configidp-changes). + +It never holds an output path: where generated code lands is the consumer's +choice, in [`comline.toml`](comline-toml.md). diff --git a/docs/docs/reference/congregation/index.md b/docs/docs/reference/congregation/index.md index 180b1a0..981c112 100644 --- a/docs/docs/reference/congregation/index.md +++ b/docs/docs/reference/congregation/index.md @@ -1,2 +1,26 @@ -# Congregation Overview +# Package files +A package (historically a *congregation*) has two configuration files with +different owners. Keeping them apart is what lets one package be generated +into many projects, each its own way. + +```mermaid +flowchart LR + A["package author"] --> IDP["config.idp
identity, languages,
dependencies, wire, settings"] + IDP -->|"frozen into
every version"| CAS[(".comline/")] + C["whoever generates"] --> TOML["comline.toml
output paths, layout,
mode, versions"] + CAS --> GEN["comline generate"] + IDP --> GEN + TOML --> GEN +``` + +| | [`config.idp`](config-id.md) | [`comline.toml`](comline-toml.md) | +|---|---|---| +| Owner | the package author | whoever runs `comline generate` | +| Says | what the package is: name, identity, languages it supports, dependencies, wire defaults, settings | where generated code goes, in what layout and mode, for which versions | +| Frozen into versions | yes | no | +| Changes the version | can | never | + +The reasoning behind the split is in +[Consumer generation configuration](../../design/consumer-generation-config.md). +How the version follows from changes is in [Versioning rules](../versioning.md). diff --git a/docs/docs/reference/tools/index.md b/docs/docs/reference/tools/index.md index e69de29..ab1e5f9 100644 --- a/docs/docs/reference/tools/index.md +++ b/docs/docs/reference/tools/index.md @@ -0,0 +1,9 @@ +# Tools + +| Tool | What it's for | +|---|---| +| [`comline`](../cli.md) | The CLI: scaffold, check, build, release, diff, generate, document. | +| [`comline inspect`](inspect.md) | What each protocol resolves to on the wire, and the guarantees it keeps or gives up. | +| [`comline doc`](doc.md) | Documentation pages, a static site or a JSON model, for every version. | +| The language server | Diagnostics, hover, completion, go-to-definition and quick fixes in `.ids`, `config.idp` and `comline.toml`, through any LSP editor. | +| The [playground](https://github.com/ComlineProject/playground) | Compile, generate and simulate calls in the browser. | diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 68663f8..6e359b9 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -20,6 +20,7 @@ nav: - Constants: 'guide/idl/const.md' - Imports: 'guide/idl/use.md' - Docstrings: 'guide/idl/docstrings.md' + - Annotations: 'guide/idl/annotations.md' - Settings: 'guide/idl/settings.md' - 'Intermediate Representation': @@ -76,14 +77,11 @@ nav: - Runtime: - C ABI: 'reference/c-abi.md' - - Congregation: - - Congregation: 'reference/congregation/index.md' - - comline.toml: 'reference/congregation/comline-toml.md' + - Packages: + - Package files: 'reference/congregation/index.md' - config.idp: 'reference/congregation/config-id.md' - - - Compliance features: - - Compliance features: 'reference/congregation/compliance_features/index.md' - - Versioning rules: 'reference/versioning.md' + - comline.toml: 'reference/congregation/comline-toml.md' + - Versioning rules: 'reference/versioning.md' # std:start - Standard library: