From 569754123ab99eacd23f9dbd15b9b64385e38d1f Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 26 Oct 2026 23:23:13 +0800 Subject: [PATCH 1/4] docs: shared runtime core stage 2, steps 1-3 built; C ABI reference The core-interface proposal is accepted, with what the build changed recorded: names, Call unchanged, COMLINE_ERR_BUSY, exact BUFFER_TOO_SMALL sizes, the frame-limit design, one-way by mode, the jsonrpc feature, and the deferrals. Stale lines (the API sketch, the toolchain, the JS glue's panic handling) are fixed. A new reference page documents the C ABI 0.1: version, status codes, structs, functions, the rules, wasm32 and the JS glue, how it is verified, and the measured crossing cost. The verification page gains the FFI vectors, fuzz targets, differential proptest, benches and the generalized gate, and the open finding runtime#58. --- docs/docs/proposals/index.md | 4 +- .../shared-runtime-core/core-interface.md | 480 +++++++++++------- .../proposals/shared-runtime-core/index.md | 7 +- .../shared-runtime-core/verification.md | 98 +++- docs/docs/reference/c-abi.md | 437 ++++++++++++++++ docs/mkdocs.yml | 3 + 6 files changed, 831 insertions(+), 198 deletions(-) create mode 100644 docs/docs/reference/c-abi.md diff --git a/docs/docs/proposals/index.md b/docs/docs/proposals/index.md index eeb3ed9..7811c75 100755 --- a/docs/docs/proposals/index.md +++ b/docs/docs/proposals/index.md @@ -72,7 +72,9 @@ reference. One proposal in three stages, read in order: fuzz targets and benchmark gates are built for Rust. - [Stage 2 — core interface](shared-runtime-core/core-interface.md): the handshake, framings and connection state machine behind a stable C ABI and - WASM, built from one crate. *under discussion.* + WASM, built from one crate. ***accepted***, steps 1–3 built: the core + changes, the C ABI at 0.1 ([reference](../reference/c-abi.md)), and its + vectors, fuzzing and benchmarks. Bindings and ABI 1.0 are next. - [Stage 3 — generated codecs](shared-runtime-core/generated-codecs.md): native per-type codecs and protocol glue rendered from one plan in the codegen kernel. *under discussion.* diff --git a/docs/docs/proposals/shared-runtime-core/core-interface.md b/docs/docs/proposals/shared-runtime-core/core-interface.md index 2eb8147..c1e8e88 100644 --- a/docs/docs/proposals/shared-runtime-core/core-interface.md +++ b/docs/docs/proposals/shared-runtime-core/core-interface.md @@ -1,6 +1,8 @@ # One protocol core: a C ABI and WASM from one crate -Status: **under discussion** · part of [Shared runtime core](index.md) · stage 2 +Status: **accepted** · part of [Shared runtime core](index.md) · stage 2 · +rollout steps 1–3 done (see [What's built](#whats-built)); steps 4–6 are future +work **The question:** what exactly the shared core owns, and what its boundary looks like, so that ten or more languages can bind it with no loss of @@ -17,8 +19,178 @@ So every choice below is judged on all three at once. Where a host makes a cost unavoidable, the page says so, and the [stage 1 gates](verification.md) decide by measurement. +## What's built + +Built 2026-10-25 and 2026-10-26, in [`runtime`](https://github.com/ComlineProject/runtime) except where noted. These +are [rollout](#rollout) steps 1–3. The as-built API is the +[C ABI reference](../../reference/c-abi.md). + +**Step 1: the core changes** + +- **JSON-RPC without `std`** ([runtime#49](https://github.com/ComlineProject/runtime/pull/49)): a `jsonrpc` + feature that needs `alloc` and no `std`. Request ids are written by a stack + writer, not through `to_string`. +- **Body-bytes entry points** ([runtime#50](https://github.com/ComlineProject/runtime/pull/50)): + `encode_request_body` on `Framing` and `ConnectionFraming`. `Call` is + unchanged. +- **The stream layer and the handshake step** + ([runtime#51](https://github.com/ComlineProject/runtime/pull/51)): + - `contract::stream`: `DEFAULT_MAX_FRAME` (16 MiB), `put_prefix` and + `split_frame`; + - `CountBuf`; + - `HandshakeFrame::from_facts` and `check_bytes`; + - `RuntimeError::FrameTooLarge`. +- **Fixed room for waiting calls** ([runtime#52](https://github.com/ComlineProject/runtime/pull/52)): + - `State::with_capacity` and `unordered_with_capacity`; + - `RuntimeError::Busy` and `BufferTooSmall`; + - `State::encode_request_body`, which commits nothing on overflow. +- **Frame limits in the transports** ([runtime#55](https://github.com/ComlineProject/runtime/pull/55)): the + transport owns the limit, with one default for everyone. See + [the frame-limit design](#what-changed-from-the-proposal). +- **The simulator** ([simulator#29](https://github.com/ComlineProject/simulator/pull/29)) + frames through the runtime's JSON-RPC and `encode_request_body`. Its vendored + JSON-RPC copy and `Preencoded` are deleted. It still keeps its own request + ids instead of using `State`, because it plays both ends on one simulated + clock. + +**Step 2: the C ABI** ([runtime#53](https://github.com/ComlineProject/runtime/pull/53)), crate +`comline-runtime-ffi`: + +- ABI 0.1, with `include/comline.h` generated by cbindgen 0.29.2 and checked + in. CI fails when it drifts. +- A `cdylib` and a `staticlib` natively, and the same exports on wasm32, plus + `comline_wasm_reserve_in` and `comline_wasm_reserve_out`. +- Status codes, an opaque `comline_connection`, a 48-byte `comline_event`, the + `NAME_ESCAPED` flag and `comline_method_name_unescape`. +- Every function runs under `catch_unwind` and poisons the handle on a panic. + Nothing allocates after `comline_connection_new`, except `jsonrpc@1` + decoding (see [Found along the way](#found-along-the-way)). A call with + overlapping buffers, or a length over `isize::MAX`, is refused. +- CI jobs `ffi` and `ffi miri`. + +**Step 3: proving the boundary** + +- **The vectors, from Rust, C and Node** ([runtime#56](https://github.com/ComlineProject/runtime/pull/56)): + the `export_vectors` example, a C harness (static, dynamic, and under ASan and + UBSan), the JS glue `js/comline.mjs` with `comline.d.mts` (no `wasm-bindgen`), + and a Node harness. The same 65 cases pass in all three. +- **Fuzzing and a differential test** ([runtime#57](https://github.com/ComlineProject/runtime/pull/57)): the + `ffi_feed` and `ffi_vs_core` fuzz targets, and `tests/differential.rs`. +- **Benchmarks** ([runtime#59](https://github.com/ComlineProject/runtime/pull/59)): the C ABI's calls against the + same work on the core, gated with the runtime's. +- **Cheaper crossing** ([runtime#60](https://github.com/ComlineProject/runtime/pull/60)): every ABI call costs + fewer instructions. + +[Verification](verification.md#whats-built) has the detail of step 3's checks. +The `main` ruleset requires `ffi`, `ffi miri`, `wasm32 core` and the fuzz +targets; see [Branch rules](../../conduct/branch-rules.md). + +## What changed from the proposal + +The implementation changed these, each checked against the source. Everything +else on this page is as proposed. + +- **Method names follow [runtime#45](https://github.com/ComlineProject/runtime/pull/45).** The sketch said `name` must + be an identifier, and that a received method with an escape fails to decode. + As built: + - `name` is any UTF-8 text, and `jsonrpc@1` writes it escaped as serde_json + does. A name that isn't UTF-8 is `COMLINE_ERR_INVALID_ARGUMENT`; + - a received name is kept as the frame spelled it, and an event sets + `COMLINE_EVENT_NAME_ESCAPED` when it has escapes; + - `comline_method_name_unescape` turns it into UTF-8. +- **`Call` is unchanged.** The proposal borrowed `Call::name` (`&'a str`). The + new `encode_request_body(call_id, name: &str, …)` takes the name as a + borrowed `&str` instead, so `Call` keeps `&'static str` and nothing cascades + into generated code. +- **`COMLINE_ERR_BUSY` (23) is `max_waiting`'s overflow.** The proposal left it + open whether it is an error or the binding's job to queue. It is an error. + The request spends no id and writes nothing, and queueing is the binding's + choice. A one-way call needs no room, and `max_waiting` 0 serves requests and + sends one-way calls only. The code wasn't in the sketch. +- **`BUFFER_TOO_SMALL` reports the exact size with the connection unchanged.** + `CountBuf` measures the frame, and `State::encode_request_body` commits no id + and no waiting call on overflow. `*written` counts a stream's prefix too. + `start`, `encode_reply`, `handshake_encode` and `comline_method_name_unescape` + report their sizes the same way. +- **The frame-limit design.** It follows + [runtime#55](https://github.com/ComlineProject/runtime/pull/55): + - the transport owns the limit, with one default, 16 MiB, declared once in + `contract::stream`; + - in the ABI the limit is `comline_config.max_frame`: 0 is the default, and it + may be raised to `u32::MAX` less the prefix; + - sending over the limit is recoverable: nothing is written, no id is spent, + the connection works on; + - receiving over it closes the transport, in stream mode. In message mode the + frame is refused and the connection works on; + - the Rust drivers answer an oversized handler reply with a `Fault`. The ABI + reports `COMLINE_ERR_FRAME_TOO_LARGE` from `encode_reply` and leaves that + choice to the host; + - there is no `config.idp` key for it. +- **One `encode_reply`, built on the envelope.** The lockstep `Framing` has + three (`encode_response_ok`, `_err` and `_fault`). The ABI has one, as + `ConnectionFraming::encode_reply` does, which takes the envelope kind. A + `Fault` takes no body. +- **One-way depends on the mode, and a connection uses only the connection + framings.** + - `datagram@1` is lockstep and has no connection form, so + `comline_connection_new` returns `COMLINE_ERR_UNKNOWN_FRAMING` for it. The + ABI carries `datagram@2` and `jsonrpc@1`. + - `datagram@2` has no way to say a call is one-way. On `jsonrpc@1` a one-way + call is a notification, with no `id`. + - An event sets `COMLINE_EVENT_ONE_WAY` only when the frame says so, never from + a request id of 0. +- **Which fields each framing reads** is stated per function and set out in + [the reference](../../reference/c-abi.md#what-each-framing-reads). + `datagram@2` reads `protocol` and `call_id`. `jsonrpc@1` reads `name`, writes + an error's `error_id` as the JSON-RPC `code`, and writes a fault as `-32603`. +- **JSON-RPC without `std` is confirmed.** `serde_json` with `raw_value` builds + without `std`, and CI's `wasm32 core` job builds `alloc,jsonrpc` for + `wasm32-unknown-unknown`. +- **Smaller differences from the sketch,** all in `comline.h`: + - `comline_status` is an `int32_t` with constants, not an enum; + - names are `const uint8_t *` with a length, not `const char *`; + - `max_waiting` is a `uint32_t` and `max_frame` a `uint64_t`; + - `comline_config` has a `_reserved` byte that must be 0; + - `comline_event` carries `capabilities`, for the handshake event; + - a `comline_handshake` struct is what `comline_handshake_check` returns; + - the constants are `COMLINE_DEFAULT_MAX_FRAME` and + `COMLINE_MAX_FRAME_CEILING`, where the sketch had `COMLINE_MAX_FRAME`. +- **The JS glue does not catch a panic.** The sketch said it would. The trap is + thrown as a `WebAssembly.RuntimeError` and the host discards the instance. + The glue also reserves its regions on every call, not up front, and returns + copies, so no view of wasm memory outlives a call. +- **Deferred:** + - `comline_connection_size` and `comline_connection_init` come after 1.0, as a + minor addition; + - `comline@2`'s negotiation comes before 1.0 if it changes the event struct. + +### Found along the way + +- **[runtime#58](https://github.com/ComlineProject/runtime/issues/58): `jsonrpc@1` decoding allocates after + `comline_connection_new`.** The ABI promises no allocation, but the framing + parses with serde_json, which allocates: + - on any refused frame, because it boxes its errors; + - when a member name has an escape; + - when a value nests brackets three deep, as in `"params":[[1]]`, which is an + ordinary frame. + + The planned fix is a hand-written strict JSON scanner in the core that + allocates nothing, checked against serde_json by a differential fuzz target. + **It is deferred:** Comline's own formats come first, and JSON is left for + later. The exemption is narrow and written down: the test + `jsonrpc_decode_allocates_nothing` is ignored, and the fuzz targets and the + differential proptest let through only those cases, on `jsonrpc@1` feeds. + Every `datagram@2` feed and every encode must still allocate nothing. + +The rest of this page is the proposal as written. Where the code has since +changed it, the text says so. + ## Where things stand +As of the proposal, before the work in [What's built](#whats-built). The simulator +and the core have both moved since: simulator#29 deleted the vendored JSON-RPC +and `Preencoded` named below. + The core already exists. It is `runtime/core` (crate `comline-runtime`) in `ComlineProject/runtime`, and most of what a shared core needs is already sans-IO, `no_std` and allocation-free. @@ -98,8 +270,9 @@ that is a host value.** - A sans-IO form of the exchange: "write our frame", then "check the peer's frame". The binding moves the bytes. 2. **Envelopes:** the tag byte, the `Err` ordinal, and `Fault`. -3. **All framings:** `datagram@1`, `datagram@2`, `jsonrpc@1`, and later - versions as they are written. +3. **All connection framings:** `datagram@2`, `jsonrpc@1`, and later versions + as they are written. (`datagram@1` is lockstep and has no connection form; + see [What changed](#what-changed-from-the-proposal).) - Request and reply encode go **around a body encoded elsewhere** (the `Preencoded` shape, made a first-class entry point). - Decode returns the body as a span of the input. @@ -111,7 +284,8 @@ that is a host value.** - `abandon` on timeout. The host's clock decides *when*; the core decides what a late reply means. 5. **Byte-level limits.** - - The stream length prefix (`u32` LE) and `MAX_FRAME`, declared once. + - The stream length prefix (`u32` LE) and the default frame limit, + declared once, in `contract::stream`. - The varint rules: shortest form only, and fits in `u16`. - JSON-RPC id range. Today these are per transport and per language. In the core they are one @@ -141,12 +315,16 @@ that is a host value.** ``` runtime/ ComlineProject/runtime ├── runtime/core/ comline-runtime: the reference, unchanged in role -└── runtime/ffi/ comline-runtime-ffi (new) +└── runtime/ffi/ comline-runtime-ffi ├── Cargo.toml crate-type = ["cdylib", "staticlib", "rlib"] ├── cbindgen.toml ├── include/comline.h generated by cbindgen, checked in ├── src/lib.rs the extern "C" surface: handles, spans, codes - └── js/ ES module glue for the wasm32 build, plus .d.ts + ├── js/ ES module glue for the wasm32 build, plus .d.mts + │ └── test/ the Node harness + ├── tests/ Rust tests, the C harness (tests/c), the differential test + ├── examples/ export_vectors: the vectors, written out for C and Node + └── bench/ the instruction-count benches (a package of its own) ``` - **One crate, two outputs.** On native targets it builds a `cdylib` (for @@ -160,136 +338,34 @@ runtime/ ComlineProject/runtime calls the same functions with the same semantics. The simulator uses `wasm-bindgen` because its surface is JS objects; this one isn't. - **It depends on `comline-runtime` without `std`** (`default-features = - false, features = ["alloc"]`, plus a `jsonrpc` feature once that moves; see + false, features = ["alloc", "jsonrpc"]`; see [What changes in the core](#what-changes-in-the-core)). So `package_abi` never enters the build, and wasm32 works before stage 5. The FFI crate itself uses `std` for `catch_unwind`, and wasm32 has `std`. - **The header is generated, then checked in.** - - cbindgen is pinned, and CI fails when `include/comline.h` differs from a - fresh run, as the docs' `cli` and `idl` drift jobs do. + - cbindgen is pinned (0.29.2), and CI's `ffi` job fails when + `include/comline.h` differs from a fresh run, as the docs' `cli` and `idl` + drift jobs do. - Reviewers then see every ABI change as a header diff. - **Why in `runtime` and not its own repo:** the FFI layer is a thin translation of `comline-runtime`. Splitting it would put every core change through a two-repo cascade. The core's tests, vectors and fuzz targets already live there. -### API sketch - -```c -/* comline.h (sketch). All integers are fixed width; no struct is - passed by value across the boundary except the small ones shown. */ - -#define COMLINE_ABI_MAJOR 0 /* 1 once frozen; see Rollout, step 5 */ -#define COMLINE_ABI_MINOR 1 - -uint32_t comline_abi_version(void); /* (major << 16) | minor */ - -typedef enum comline_status { /* append-only, stable values */ - COMLINE_OK = 0, - COMLINE_NEED_MORE = 1, /* stream mode: no whole frame yet */ - COMLINE_BUFFER_TOO_SMALL = 2, /* *written holds the size needed */ - COMLINE_ERR_FRAMING = 16, /* RuntimeError::Framing */ - COMLINE_ERR_HANDSHAKE = 17, /* RuntimeError::Handshake */ - COMLINE_ERR_FRAME_TOO_LARGE = 18, - COMLINE_ERR_UNKNOWN_FRAMING = 19, - COMLINE_ERR_INVALID_ARGUMENT = 20, /* null pointer, bad name bytes */ - COMLINE_ERR_STATE = 21, /* e.g. a call before the handshake */ - COMLINE_ERR_POISONED = 22, /* an earlier call panicked; drop the handle */ -} comline_status; - -typedef struct comline_connection comline_connection; /* opaque */ - -typedef struct comline_span { uint32_t offset; uint32_t len; } comline_span; - /* into the caller's input buffer */ - -typedef enum comline_event_kind { - COMLINE_EVENT_HANDSHAKE_OK = 0, - COMLINE_EVENT_REQUEST = 1, - COMLINE_EVENT_REPLY = 2, - COMLINE_EVENT_STALE = 3, -} comline_event_kind; - -typedef enum comline_envelope_kind { - COMLINE_ENVELOPE_OK = 0, COMLINE_ENVELOPE_ERR = 1, COMLINE_ENVELOPE_FAULT = 2, -} comline_envelope_kind; - -typedef struct comline_event { - uint8_t kind; /* comline_event_kind */ - uint8_t flags; /* HAS_PROTOCOL, ONE_WAY, CALL_BY_NAME, BODY_ABSENT */ - uint16_t protocol; /* valid with HAS_PROTOCOL */ - uint16_t call_id; /* valid without CALL_BY_NAME */ - uint8_t envelope; /* comline_envelope_kind, for REPLY */ - uint8_t _reserved; - uint16_t error_id; /* the schema-global ordinal, for ENVELOPE_ERR */ - uint64_t request_id; - comline_span call_name; /* valid with CALL_BY_NAME (jsonrpc@1) */ - comline_span body; /* params for REQUEST, payload or error body for REPLY */ - comline_span frame; /* the whole frame, for logging or forwarding */ -} comline_event; - -typedef struct comline_config { - uint64_t ir_hash; - const char* wire_format; size_t wire_format_len; /* "packed@1" */ - const char* framing; size_t framing_len; /* "datagram@2" */ - uint32_t capabilities; - uint8_t handshake; /* COMLINE_HANDSHAKE_COMLINE or _NONE */ - uint8_t stream; /* 1: u32 LE length-prefixed byte stream; 0: one frame per message */ - uint8_t unordered; /* 1: replies in any order (async hosts) */ - uint16_t max_waiting; /* waiting-call capacity, fixed at create */ - uint32_t max_frame; /* <= COMLINE_MAX_FRAME (16 MiB); 0 = the default */ -} comline_config; - -comline_status comline_connection_new(const comline_config* cfg, comline_connection** out); -void comline_connection_free(comline_connection* conn); -``` +### The API -The three representative calls: - -```c -/* 1. Feed bytes, get the next event. One frame per call. - * - * Stream mode: `in` is whatever the host has read so far. If a whole - * length-prefixed frame is there, it is decoded, *consumed is set to the - * prefix + frame length, and *ev describes it, with spans relative to `in`. - * Otherwise COMLINE_NEED_MORE, *consumed = 0, and ev->frame.len holds the - * total bytes needed once the prefix itself is complete (0 before that). - * - * Message mode: `in` is exactly one frame. - * - * The first frame on a COMLINE_HANDSHAKE_COMLINE connection is the peer's - * handshake: it is checked and yields HANDSHAKE_OK or COMLINE_ERR_HANDSHAKE. */ -comline_status comline_connection_feed( - comline_connection* conn, - const uint8_t* in, size_t in_len, - size_t* consumed, - comline_event* ev); - -/* 2. Frame a request around a body the generated codec already wrote. - * - * `name` is used by name-oriented framings (jsonrpc@1) and ignored by the - * datagram framings. It must be an identifier (no JSON escaping), or the - * call fails with COMLINE_ERR_INVALID_ARGUMENT. In stream mode the length - * prefix is written too, so `out` is ready for one transport write. - * - * On COMLINE_BUFFER_TOO_SMALL, *written is the size needed and the - * connection is unchanged: no request id is spent and nothing is waited on. */ -comline_status comline_connection_encode_request( - comline_connection* conn, - uint16_t protocol, uint16_t call_id, - const uint8_t* name, size_t name_len, - uint8_t expects_reply, - const uint8_t* body, size_t body_len, - uint8_t* out, size_t cap, size_t* written, - uint64_t* request_id); - -/* 3. Frame the reply to a request this side was given. */ -comline_status comline_connection_encode_reply( - comline_connection* conn, - uint16_t protocol, uint64_t request_id, - uint8_t envelope, uint16_t error_id, - const uint8_t* body, size_t body_len, - uint8_t* out, size_t cap, size_t* written); -``` +The header and the [C ABI reference](../../reference/c-abi.md) hold the +as-built API. The C sketch that stood here is gone. Its shape held: + +- a version, `comline_abi_version()`, and a header that carries it; +- an opaque `comline_connection`, made from a `comline_config`; +- `comline_connection_feed`, `comline_connection_encode_request` and + `comline_connection_encode_reply`, one whole frame per call; +- event spans as offsets into the caller's input; +- append-only status codes. + +[What changed](#what-changed-from-the-proposal) lists where the built API +differs from the sketch. The rest of the surface is small: @@ -297,10 +373,11 @@ The rest of the surface is small: handshake frame. - `comline_connection_abandon(conn, request_id)` gives up on a call after a timeout. -- `comline_connection_is_waiting(conn)`. +- `comline_connection_is_waiting(conn, waiting)`. - Free functions with no handle: `comline_name_hash`, `comline_handshake_encode` and `comline_handshake_check`, for hosts that - run the handshake themselves. + run the handshake themselves; `comline_supports`; and + `comline_method_name_unescape`. A call from Python, Go or Java is then: @@ -332,16 +409,18 @@ The FFI layer is for every other language, and Rust never pays for it. `comline_connection_new`, sized by `max_waiting`, and frees it in `comline_connection_free`. The host never frees library memory and the library never frees host memory. -- After the handle exists, the happy path allocates nothing. That needs - `State.waiting` to become a fixed-capacity array and `jsonrpc@1` to stop - formatting ids through `String`. Both are listed under - [What changes in the core](#what-changes-in-the-core), and both already - apply to Rust callers under - [§4.6's budget](../../design/core-target-contract.md#the-budget) - ("**zero** allocations on the happy path"). +- After the handle exists, the happy path allocates nothing. + - `State::with_capacity` reserves the waiting calls' room in + `comline_connection_new` (a `Vec` reserved to `max_waiting`, not an + array), and `jsonrpc@1` writes ids on the stack. Both apply to Rust + callers too, under + [§4.6's budget](../../design/core-target-contract.md#the-budget) + ("**zero** allocations on the happy path"). + - **One exemption, as built:** `jsonrpc@1` decoding allocates through + serde_json. See [Found along the way](#found-along-the-way). - A host with no allocator story of its own (embedded C, Zig) can get a later `comline_connection_size(cfg)` / `comline_connection_init(mem, …)` - pair. That is a minor ABI addition, not part of 1.0. + pair. That is a minor ABI addition, **after 1.0**, not part of it. **Panics never cross.** - Since Rust 1.81, a panic unwinding out of an `extern "C"` function aborts @@ -355,11 +434,13 @@ The FFI layer is for every other language, and Rust never pays for it. - Every later call on that handle returns the same code, so the binding closes the connection. - **On wasm32,** a panic is a trap (`WebAssembly.RuntimeError`). The JS glue - catches it and discards the instance. Unwinding on wasm32 needs the - exception-handling proposal, which this does not depend on. + does not catch it: the trap reaches the host, which discards the instance. + Unwinding on wasm32 needs the exception-handling proposal, which this does + not depend on. - **Panics are bugs, not a control path.** The decode paths are already - written as `Option`-returning `split_at_checked` chains. Stage 1's fuzz - targets run against the FFI entry points with "no panic" as the check. + written as `Option`-returning `split_at_checked` chains. The `ffi_feed` and + `ffi_vs_core` fuzz targets run against the FFI entry points and fail if a + panic crosses. **Thread safety.** - A `comline_connection*` is **movable, not shareable**. Any thread may use @@ -393,9 +474,10 @@ The FFI layer is for every other language, and Rust never pays for it. - **Status codes and flag bits are append-only.** A binding maps an unknown status to a generic error carrying the number. `RuntimeError` is already `#[non_exhaustive]`, for the same reason. -- `comline_supports(name, len)` answers whether this library build knows an - encoding name (`"datagram@2"`). A binding can then refuse a schema's - `@framing` at setup, not at the first frame. +- `comline_supports(name, len)` answers whether this library build carries a + framing or handshake name (`"datagram@2"`, `"jsonrpc@1"`, `"comline@1"`, + `"none"`). Wire formats are the host's codecs, so none is listed. A binding + can then refuse a schema's `@framing` at setup, not at the first frame. **Zero-copy where the host allows.** - **Input is borrowed.** Events point into it by `comline_span` offsets, so @@ -408,15 +490,20 @@ The FFI layer is for every other language, and Rust never pays for it. reallocates its buffer between the call and the decode. A garbage-collected host (Java heap arrays, .NET with pinning only during the call, Go) usually does exactly that. -- **`u32` spans are enough:** `MAX_FRAME` is 16 MiB. They are also the same - width on wasm32. +- **`u32` spans are enough:** the default frame limit is 16 MiB, and a + connection's limit can't pass `u32::MAX` less the prefix. They are also the + same width on wasm32. - **Two JSON-RPC edge cases surface as flags, not copies:** - - Absent `params` or `result` decode today as the literal `null` (a static + - Absent `params` or `result` decode as the literal `null` (a static slice, not part of the input). The event sets `BODY_ABSENT` with an empty - span, and the JSON codec reads `null`. + span, and the JSON codec reads `null`. A `null` the frame spelled out sets + it too, since the framing can't tell the two apart. - `serde_json` will not borrow a string that contains escapes, so a - `method` with a `\u` escape fails to decode today. The FFI keeps that - behaviour. A vector should pin it before anything changes it. + `method` with an escape failed to decode. **As built + ([runtime#45](https://github.com/ComlineProject/runtime/pull/45)):** the + name is kept as the frame spelled it and the event sets `NAME_ESCAPED`; + `comline_method_name_unescape` reads it. The `found-request-method-*` + vectors pin it. - **Output is written once,** straight into the caller's `out`. The body passed to `encode_*` is copied exactly once, into the frame. That is the same single copy the Rust path makes when `Packed` encodes into the send @@ -426,26 +513,28 @@ The FFI layer is for every other language, and Rust never pays for it. All of these are small. Each one also helps Rust callers. -| Change | Why | -|---|---| -| Body-bytes entry points on `Framing` / `ConnectionFraming` (`encode_request_body(…, body: &[u8], …)`), the generic `Serialize` form becoming a wrapper | The `Preencoded` shim, made first-class. The simulator can drop its copy. | -| `Call::name` borrowed (`&'a str`) rather than `&'static str` | Host-owned names; the simulator's `WireFraming` workaround goes away. | -| `jsonrpc@1` behind `alloc` or its own `jsonrpc` feature, not `std`; ids written without `to_string` | The simulator vendors it today. Whether `serde_json`'s `raw_value` builds without `std` has to be checked first. | -| `State.waiting` with a capacity fixed at construction | No allocation on the happy path; an overflow becomes an error code, not a reallocation. | -| A sans-IO stream deframer: the `u32` LE prefix plus one `MAX_FRAME` constant | Replaces the two declarations in `transport.rs` and the tokio `Tcp`. | -| A sans-IO handshake step (`ours.encode`, `check(peer_bytes)`) beside `ComlineHandshake` | `ComlineHandshake::exchange` is `Transport`-bound. | -| A counting `BufMut` | So `BUFFER_TOO_SMALL` can report the exact size needed without mutating state. | +| Change | Why | Status | +|---|---|---| +| Body-bytes entry points on `Framing` / `ConnectionFraming` (`encode_request_body(…, body: &[u8], …)`), the generic `Serialize` form becoming a wrapper | The `Preencoded` shim, made first-class. The simulator can drop its copy. | Done ([runtime#50](https://github.com/ComlineProject/runtime/pull/50)). The generic form is kept as it was; the simulator dropped `Preencoded` (simulator#29). | +| `Call::name` borrowed (`&'a str`) rather than `&'static str` | Host-owned names; the simulator's `WireFraming` workaround goes away. | **Not done, not needed.** `encode_request_body` takes `name: &str`, so `Call` is unchanged. | +| `jsonrpc@1` behind `alloc` or its own `jsonrpc` feature, not `std`; ids written without `to_string` | The simulator vendors it today. Whether `serde_json`'s `raw_value` builds without `std` has to be checked first. | Done ([runtime#49](https://github.com/ComlineProject/runtime/pull/49)): a `jsonrpc` feature. `raw_value` builds without `std`. | +| `State.waiting` with a capacity fixed at construction | No allocation on the happy path; an overflow becomes an error code, not a reallocation. | Done ([runtime#52](https://github.com/ComlineProject/runtime/pull/52)): `State::with_capacity`; overflow is `RuntimeError::Busy`. | +| A sans-IO stream deframer: the `u32` LE prefix plus one `MAX_FRAME` constant | Replaces the two declarations in `transport.rs` and the tokio `Tcp`. | Done ([runtime#51](https://github.com/ComlineProject/runtime/pull/51)): `contract::stream`. Both transports read it ([runtime#55](https://github.com/ComlineProject/runtime/pull/55)). | +| A sans-IO handshake step (`ours.encode`, `check(peer_bytes)`) beside `ComlineHandshake` | `ComlineHandshake::exchange` is `Transport`-bound. | Done (runtime#51): `HandshakeFrame::from_facts` and `check_bytes`. | +| A counting `BufMut` | So `BUFFER_TOO_SMALL` can report the exact size needed without mutating state. | Done (runtime#51, #52): `CountBuf`, and `BufMut::overflowed`. | ## WASM **The same API, compiled to `wasm32-unknown-unknown`.** - `comline-runtime-ffi` exports the `comline_*` functions as they are. - `js/` holds a small ES module, hand-written, with no generated glue. It - - instantiates the module; + - instantiates the module, and checks its ABI version and the + `comline_event` layout; - reserves an input region and an output region in linear memory, through two wasm-only exports (`comline_wasm_reserve_in(len)`, `comline_wasm_reserve_out(len)`) that return pointers into memory the - module owns; + module owns. It does so on every call, which costs nothing when the + region is already big enough; - reads `comline_event` with a `DataView`. - **A size reference point:** the simulator's 2026-09-03 spike, built on `comline-runtime` with `alloc` only, was an 88 KB `.wasm`. @@ -457,12 +546,12 @@ All of these are small. Each one also helps Rust callers. | Bytes from the socket (a `Uint8Array` on the JS heap) into linear memory | one `set()`, a memcpy linear in frame size, per frame in | | The call | a JS→WASM call; small in current engines, not free | | Spans back | read through `DataView`; no copy | -| The body, for the generated JS codec | read in place through a view on `memory.buffer`, **or** copied out if the codec keeps it past the next call | +| The body, for the generated JS codec | **As built:** a subarray of the `Uint8Array` the host fed, at the span's offset. The glue hands out no view of wasm memory. | | A frame out | the codec's body is copied in, the frame is copied out: two memcpys | -- **Views on `memory.buffer` are invalidated when memory grows.** The glue - reserves its regions up front and only grows between calls. Even so, a - generated codec must not hold a view across calls. +- **Views on `memory.buffer` are invalidated when memory grows.** As built, + the glue takes fresh views after every reserve and returns copies, so a + generated codec holds none across calls. - **For small frames this can lose** to hand-written TypeScript framing, which works on the socket's `Uint8Array` with no copy. A `datagram@2` request header is 12 bytes for a protocol number below 128; the copy and @@ -538,8 +627,9 @@ addition, not a replacement. ## Distribution -**This is a sketch and an open question.** It needs its own pass before -stage 2 ships. +**This is a sketch and an open question.** It needs its own pass before the +release workflow and prebuilt binaries ([rollout](#rollout) step 5). Nothing +in it is built. **Targets.** Prebuilt binaries per triple, from `runtime`'s release workflow: @@ -566,9 +656,9 @@ workflow: | Dart, PHP, Lua | native libs inside the package; specifics per ecosystem | **Reproducibility.** -- Release builds use a **pinned stable toolchain**. The runtime repo's - `rust-toolchain` currently says `nightly`, unpinned, and that can't stay - for shipped binaries. +- Release builds use a **pinned stable toolchain**. The runtime repo pins stable + 1.98.0 in `rust-toolchain.toml` ([runtime#30](https://github.com/ComlineProject/runtime/pull/30)), + which replaced an unpinned `nightly`. - Builds run with `--locked`, `--remap-path-prefix` and a fixed `SOURCE_DATE_EPOCH`, and the release publishes checksums. - Each binding records the core release it bundles in a file, as @@ -621,35 +711,43 @@ stays a separate idea. ## Open questions -- **Handle memory for no-allocator hosts:** whether - `comline_connection_size` / `comline_connection_init` belongs in 1.0 or - 1.1. -- **`max_waiting` overflow:** whether it is an error to the caller of - `encode_request`, as proposed, or the binding's job to queue. +**Settled by the build:** + +- **Handle memory for no-allocator hosts:** `comline_connection_size` / + `comline_connection_init` come after 1.0, as a minor addition. +- **`max_waiting` overflow:** an error to the caller of `encode_request`, + `COMLINE_ERR_BUSY`. Queueing is the binding's job. +- **Whether `jsonrpc@1`'s `serde_json` dependency fits the no-`std` FFI + build:** it does, as is. Its decoding allocates, which breaks the ABI's + no-allocation rule; that is [runtime#58](https://github.com/ComlineProject/runtime/issues/58), + deferred. + +**Still open:** + +- **Whether `comline@2`'s negotiation** (protocol numbers handed out at the + handshake) changes the event struct. If so, it should land before ABI 1.0 + freezes. - **`pyo3` vs `cffi` for Python,** and whether a second binding surface is ever worth it. Decided by gates, but the default needs picking. - **Go without cgo:** wazero against pure Go, measured on the same benchmarks. -- **Whether `jsonrpc@1`'s `serde_json` dependency** fits the no-`std` FFI - build as is, or needs a narrower parser for the envelope fields. - **Distribution:** everything flagged in [Distribution](#distribution). -- **Whether `comline@2`'s negotiation** (protocol numbers handed out at the - handshake) changes the event struct. If so, it should land before ABI 1.0 - freezes. ## Rollout -1. **The core changes** in `comline-runtime`, each with vectors: body-bytes - framing entry points, borrowed `Call::name`, `jsonrpc@1` out of `std`, - fixed-capacity `waiting`, the stream deframer with one `MAX_FRAME`, and - the sans-IO handshake step. The simulator moves onto them and deletes its - vendored JSON-RPC and `Preencoded`. -2. **`comline-runtime-ffi`** with `comline.h` at ABI `0.x`: native and wasm32 - builds, the header drift check, and FFI fuzz targets in stage 1's +Steps 1–3 are done ([What's built](#whats-built)). + +1. **The core changes** in `comline-runtime`, each with vectors: **done.** + Body-bytes framing entry points, `jsonrpc@1` out of `std`, fixed-capacity + `waiting`, the stream deframer with one default limit, and the sans-IO + handshake step. Borrowed `Call::name` was not needed. The simulator moved + onto them and deleted its vendored JSON-RPC and `Preencoded`. +2. **`comline-runtime-ffi`** with `comline.h` at ABI `0.x`: **done.** Native and + wasm32 builds, the header drift check, and FFI fuzz targets in stage 1's harness. 3. **A C test binding** in `runtime` CI runs the conformance vectors through - the C ABI and the wasm32 build. This proves the boundary before any - language depends on it. + the C ABI and the wasm32 build: **done.** C and Node replay the same 65 + cases as Rust. This proves the boundary before any language depends on it. 4. **The first two real bindings, chosen to stress the boundary:** - **TypeScript**, through WASM and napi: stage 4's question; - **Go**, through cgo: the most expensive call. @@ -668,13 +766,13 @@ stays a separate idea. - FFI fuzz targets; - the release workflow; - the workspace `Cargo.toml`, where the `runtime-langs/c` stub (an `add` - function today) is superseded. -- **Simulator:** drops its vendored JSON-RPC and the `Preencoded` shim once - the core has body-bytes entry points. + function) was removed by runtime#53. +- **Simulator:** dropped its vendored JSON-RPC and the `Preencoded` shim + (simulator#29). - **comline-typescript:** stage 4, behind the gates. - **Each future `comline-`:** a binding, its packaging, and the gates. - **Docs:** - the [runtime guide](../../guide/runtime/index.md)'s "no FFI boundary"; - [`core-target-contract.md`](../../design/core-target-contract.md) §4.7 (answered here) and §4.6's `no_std` layering table; - - a `comline.h` reference page. + - a `comline.h` reference page: done, the [C ABI reference](../../reference/c-abi.md). diff --git a/docs/docs/proposals/shared-runtime-core/index.md b/docs/docs/proposals/shared-runtime-core/index.md index 57ca4c8..18bd420 100644 --- a/docs/docs/proposals/shared-runtime-core/index.md +++ b/docs/docs/proposals/shared-runtime-core/index.md @@ -3,8 +3,9 @@ Status: **under discussion** · tracked in [runtime#29](https://github.com/ComlineProject/runtime/issues/29) · direction agreed 2026-10-25; stage 1 accepted and in progress (see -[what's built](verification.md#whats-built)); stages 2–3 are proposals below, -stages 4–5 are agreed +[what's built](verification.md#whats-built)); stage 2 accepted, with its first +three rollout steps built (see [what's built](core-interface.md#whats-built)); +stage 3 is a proposal below, stages 4–5 are agreed **The question:** today every Comline runtime is written from scratch in its own language, and every runtime feature is built once per language. That @@ -111,7 +112,7 @@ Split the work by what can cross a language boundary cheaply. | Stage | Page | State | |---|---|---| | 1 | [Verification: differential fuzzing and benchmark gates](verification.md) | accepted, in progress | -| 2 | [One protocol core: a C ABI and WASM from one crate](core-interface.md) | proposal | +| 2 | [One protocol core: a C ABI and WASM from one crate](core-interface.md) | accepted; rollout steps 1–3 built, bindings and ABI 1.0 next; see the [C ABI reference](../../reference/c-abi.md) | | 3 | [Codecs and protocol glue generated from one plan](generated-codecs.md) | proposal | | 4 | Move TypeScript onto the core where the benchmarks allow | agreed | | 5 | Clean up the leftovers and the documents that contradict this | agreed | diff --git a/docs/docs/proposals/shared-runtime-core/verification.md b/docs/docs/proposals/shared-runtime-core/verification.md index 5d08464..0be59bd 100644 --- a/docs/docs/proposals/shared-runtime-core/verification.md +++ b/docs/docs/proposals/shared-runtime-core/verification.md @@ -1,6 +1,7 @@ # Verification: differential fuzzing and benchmark gates -Status: **accepted** · part of [Shared runtime core](index.md) · stage 1 in progress +Status: **accepted** · part of [Shared runtime core](index.md) · stage 1 in progress; +the C ABI's checks (stage 2, steps 2–3) are built **The question:** with ten or more language runtimes and generators, how does each one stay byte-identical to the others, exactly as strict as the others @@ -121,6 +122,68 @@ except where noted. length; `json@1` and `jsonrpc@1` replaced invalid UTF-8 and skipped a byte-order mark; the JSON reader accepted an escaped lone surrogate. +**For the C ABI** (stage 2, steps 2–3 of the +[core interface](core-interface.md#whats-built)), 2026-10-26: + +- **Frame-limit vectors and a fuzz target** + ([runtime#55](https://github.com/ComlineProject/runtime/pull/55)): `vectors/stream.json`, 24 cases, and the + `stream_split` fuzz target, which checks `split_frame` and `put_prefix` + against each other and against the prefix's own rules. It also fixed the + transports' frame-limit bugs (see the + [core interface](core-interface.md#what-changed-from-the-proposal)). +- **The ABI's vectors, from Rust, C and Node** + ([runtime#56](https://github.com/ComlineProject/runtime/pull/56)): one harness per host, on the same cases and + with the same checks, against the six vector files the ABI can carry + (`name-hash`, `handshake`, `envelope`, `framing-datagram`, + `framing-datagram2`, `framing-jsonrpc`). + - The cases are 65. The nine `datagram@1` cases have no connection form, so + for them the harness checks that the ABI refuses it. + - C is built against `comline.h` and runs on the static library, the dynamic + library, and the static one under ASan and UBSan. Node runs against the + wasm32 build, through the JS glue. + - C and Node can't run a codec, so the `export_vectors` example writes the + cases out with their params already encoded. Each harness fails on a + mismatch, and on a case count that differs from the vectors'. + - CI job `ffi`; Miri runs the Rust side separately as `ffi miri`. +- **Two fuzz targets and a differential proptest** + ([runtime#57](https://github.com/ComlineProject/runtime/pull/57)): + - `ffi_feed` runs random configs and sequences (calls, replies, abandons, peer + frames, chunked deliveries, flipped bytes) through a `comline_connection` + and holds every call to the ABI's rules: no panic crosses, only documented + statuses, spans inside the frame, exact retry sizes, refusals spend no + request id, nothing allocates after `comline_connection_new`. + - `ffi_vs_core` adds the comparison: each call is also made through `State`, + the connection framings and `HandshakeFrame` directly, and must give the + same status, bytes, request id, `*consumed` and event. + - `tests/differential.rs` runs the same driver on proptest's sequences, with + 100,000 cases nightly. Both targets share the driver, so a sequence fails the + same way under either. + - The first nightly run with them (main at `b773057`) ran each target for 10 + minutes: 4,154,468 executions for `ffi_feed` and 2,465,895 for + `ffi_vs_core`, no findings, and the differential property passed. + - They found [runtime#58](#found-along-the-way), deferred. +- **FFI benchmarks and a gate over every target** + ([runtime#59](https://github.com/ComlineProject/runtime/pull/59), [runtime#60](https://github.com/ComlineProject/runtime/pull/60)): + - `runtime/ffi/bench` is a package of its own, so the ABI is built as a host + links it, without the crate's test hooks. Each call is measured twice: + through its `extern "C"` function (`ffi/abi/...`) and as the same work on + `State` and the framings (`ffi/core/...`). That is 104 benchmarks. + - The gate now runs every bench target, listed once as `TARGETS` in + `bench-report.py`. The limit (1%), the ledger and the rename rules are the + same for all. Ids are unique across targets: the runtime's keep the shared + definitions' names and every other target's starts with its name. + - A target the base doesn't have yet is recorded only, as is the whole report + when the base has no benches. + - Base and head now build at paths of the same length, because a longer path + moved 16 of the 102 FFI counts by a few instructions. + - The job summary pairs each ABI call with its core twin, the crossing cost. + It informs and isn't gated on its own. + - runtime#60 cut the crossing: every ABI call is cheaper, none dearer. The + numbers are in the [C ABI reference](../../reference/c-abi.md#what-crossing-costs). +- **Required checks.** The `main` ruleset requires `ffi`, `ffi miri`, + `wasm32 core` and the fuzz jobs `fuzz (ffi_feed)`, `fuzz (ffi_vs_core)` and + `fuzz (stream_split)`. See [Branch rules](../../conduct/branch-rules.md). + ## Found along the way Open issues the work turned up: @@ -140,6 +203,26 @@ Open issues the work turned up: name that needs one unescaped, producing invalid JSON. - [runtime#42](https://github.com/ComlineProject/runtime/issues/42): `jsonrpc@1` accepts invalid UTF-8 inside members it ignores. +- [runtime#58](https://github.com/ComlineProject/runtime/issues/58): `jsonrpc@1` decoding allocates after + `comline_connection_new`, which the C ABI promises it doesn't. The framing + parses with serde_json, which allocates: + - on any refused frame, because it boxes its errors; + - when a member name has an escape; + - when a value nests brackets three deep, as in `"params":[[1]]`, which is an + ordinary frame. + + **Deferred (decided 2026-10-26):** Comline's own formats come first, and JSON + is left for later. The planned fix is a hand-written strict JSON scanner in the + core that allocates nothing, with a differential fuzz target against + serde_json. Until then the exemption is narrow and written down: + - the test `jsonrpc_decode_allocates_nothing` (`runtime/ffi/tests/rules.rs`) + is ignored, and fails with `--ignored`; + - `serde_json_allocates` in `tests/common/drive.rs` lets the fuzz targets and + the differential proptest through on exactly those three cases, on + `jsonrpc@1` feeds only; + - every `datagram@2` feed and every encode must still allocate nothing; + - the fuzz README lists it under "Open", and the fix deletes the exemption and + un-ignores the test. - In TypeScript, each listed as a known divergence: [comline-typescript#73](https://github.com/ComlineProject/comline-typescript/issues/73) (`json@1` decoding doesn't check a value's shape against its type), @@ -161,6 +244,10 @@ Open issues the work turned up: `rust.bench.json` (every push to `main`) and `rust.fuzz.json` (nightly) to the runtime's `reports` branch, and the pages render them. TypeScript's benchmark results fill in when it checks in its `conformance-bench.json`. +- **The `jsonrpc@1` allocation** ([runtime#58](https://github.com/ComlineProject/runtime/issues/58)), deferred. +- **The wasm32 build and the JS glue are not benchmarked.** The ABI's benches + are native. A JavaScript call into WebAssembly, and the copies in and out, + are what [stage 4](index.md#stages) has to measure. - **A re-baseline path for the gate.** A correctness fix that costs time needs a deliberate way to move the baseline (see [Budgets](#budgets)). It isn't built yet. @@ -247,8 +334,9 @@ implementation. commit in `VECTORS_REV`, so a language's CI answers the same way on every run. Every language CI that vendors the corpus already has, or can install, a Rust toolchain (comline-typescript's CI has one for its codegen). -- **(b) comes later as an option, not a replacement.** Once stage 2 exists, a - language may link the reference in-process for speed. Codecs stay native +- **(b) comes later as an option, not a replacement.** Once stage 2 exists (the + C ABI is built, [runtime#53](https://github.com/ComlineProject/runtime/pull/53); no language's fuzzer links it + yet), a language may link the reference in-process for speed. Codecs stay native ([stage 3](generated-codecs.md)), so the differential check stays meaningful there. For framing it then checks the binding, not the logic. - **(c) falls out of (a)** for free: the reference can dump a batch of cases to a @@ -446,6 +534,10 @@ The last row is what [stage 4](index.md#stages) needs. For small frames, a TypeScript call into WASM or napi may cost more than the work it replaces. Only this row can say where that line is. +As built ([runtime#59](https://github.com/ComlineProject/runtime/pull/59)), the row covers the native ABI only: `ffi/abi/...` +against `ffi/core/...`, in both stream and message mode, and at the payload +sizes above for the calls that carry a frame. The wasm32 build and the JS glue aren't measured yet. + ### Judging a regression without flaky CI Shared CI runners are noisy. Wall-time differences of several percent between diff --git a/docs/docs/reference/c-abi.md b/docs/docs/reference/c-abi.md new file mode 100644 index 0000000..39bb8c7 --- /dev/null +++ b/docs/docs/reference/c-abi.md @@ -0,0 +1,437 @@ +# C ABI + +The C ABI of `comline-runtime-ffi`: what a language other than Rust binds to +for the `comline@1` handshake, the connection framings (`datagram@2`, +`jsonrpc@1`), envelopes, connection state and frame limits. Codecs, +transports, async and dispatch stay in each host. + +- **ABI version 0.1.** It is not frozen; see [Version](#version). +- **Source:** [`runtime/ffi`](https://github.com/ComlineProject/runtime/tree/main/runtime/ffi) + in the runtime repository. +- **Header:** [`include/comline.h`](https://github.com/ComlineProject/runtime/blob/main/runtime/ffi/include/comline.h). + cbindgen 0.29.2 generates it from `src/lib.rs`, and it is checked in. Its + comments are the doc comments of the Rust source. +- **Why it exists and what was decided:** [one protocol core](../proposals/shared-runtime-core/core-interface.md). + +This page was written against runtime `main` at `cb41d31`. Where it and the +header disagree, the header is right. + +## Version + +- The header defines `COMLINE_ABI_MAJOR` (0) and `COMLINE_ABI_MINOR` (1). +- `comline_abi_version()` returns the library's, as `(major << 16) | minor`. +- **A host checks it at load, before any other call.** + + ```c + if (comline_abi_version() != ((COMLINE_ABI_MAJOR << 16) | COMLINE_ABI_MINOR)) { + /* refuse the library, naming both versions */ + } + ``` + +- **Before 1.0, a minor counts as a major.** A host built against `0.1` accepts + only `0.1`. The JS glue does exactly this. +- The major changes when a signature, a struct layout or the meaning of a code + changes. The minor changes when functions, codes or flag bits are added. +- Structs are frozen within a major. New data comes through new functions or new + flag bits. +- Status codes, event kinds and flag bits are append-only. A host maps a code + it doesn't know to a generic error carrying the number. +- `comline_supports(name, len)` says whether this build carries a framing or + handshake by name, so a host can refuse a schema's `@framing` at setup, not + at the first frame. + +## A connection's life + +1. `comline_connection_new` with a [`comline_config`](#comline_config). +2. `comline_connection_start` writes this side's handshake frame. Send it + first. +3. `comline_connection_feed` with what arrives. The first frame is the peer's + handshake and gives `COMLINE_EVENT_HANDSHAKE_OK`. It may arrive before or + after `start`. A request or reply needs both. +4. Calls and replies: `comline_connection_encode_request`, + `comline_connection_feed`, `comline_connection_encode_reply`. When the + host's clock says a call timed out, `comline_connection_abandon`. +5. `comline_connection_free`. + +With `COMLINE_HANDSHAKE_NONE` there is nothing to send or check: the +connection is open from the start, and `start` writes nothing. + +## Status codes + +`comline_status` is an `int32_t`. Below 16 is an outcome, 16 and up is an error. + +| Code | Value | Meaning | +|---|---|---| +| `COMLINE_OK` | 0 | Done. | +| `COMLINE_NEED_MORE` | 1 | Stream mode: no whole frame in the input yet. Nothing was consumed. `ev->frame.len` is the total bytes the frame takes, prefix included, once the prefix is complete (0 before that). | +| `COMLINE_BUFFER_TOO_SMALL` | 2 | The output didn't fit `cap`. `*written` is the exact size needed, and nothing changed: no request id spent, nothing waits. | +| `COMLINE_ERR_FRAMING` | 16 | A malformed frame, or (ordered connections) a reply that skips the innermost waiting call. The frame was consumed and the connection stays usable. | +| `COMLINE_ERR_HANDSHAKE` | 17 | The peer's `comline@1` frame didn't parse, or names another schema, wire format or framing. The connection is closed. | +| `COMLINE_ERR_FRAME_TOO_LARGE` | 18 | A frame over `max_frame`. See [Frame limits](#frame-limits). | +| `COMLINE_ERR_UNKNOWN_FRAMING` | 19 | A framing name this library has no connection form of. | +| `COMLINE_ERR_INVALID_ARGUMENT` | 20 | A null pointer where one isn't allowed, a name that isn't UTF-8, a field outside its values, buffers that overlap, or a length over `isize::MAX`. | +| `COMLINE_ERR_STATE` | 21 | The call doesn't fit the connection's state: a request or reply before the handshake completed, a second `start`, or any call on a closed connection. | +| `COMLINE_ERR_POISONED` | 22 | An earlier call on this handle panicked. Drop the handle. | +| `COMLINE_ERR_BUSY` | 23 | The request wasn't made: `max_waiting` calls already wait. Nothing written, no request id spent. Queueing is the host's choice. | + +Every call that takes a handle can also return `COMLINE_ERR_INVALID_ARGUMENT` +(a null handle or pointer), `COMLINE_ERR_POISONED`, and `COMLINE_ERR_STATE` on a +closed connection. The tables below list only what is particular to a call. + +## Constants + +| Constant | Value | Meaning | +|---|---|---| +| `COMLINE_DEFAULT_MAX_FRAME` | 16 MiB | The frame limit when `max_frame` is 0, prefix not counted. | +| `COMLINE_MAX_FRAME_CEILING` | 4294967291 | The most `max_frame` can be: `u32::MAX` less the prefix. | +| `COMLINE_PREFIX_LEN` | 4 | A stream's length prefix: a `u32`, little-endian. | +| `COMLINE_HANDSHAKE_LEN` | 31 | The `comline@1` frame. | +| `COMLINE_HANDSHAKE_COMLINE` / `_NONE` | 0 / 1 | `comline_config.handshake`. | +| `COMLINE_ENVELOPE_OK` / `_ERR` / `_FAULT` | 0 / 1 / 2 | A reply's outcome. | + +## Structs + +### `comline_config` + +Read once by `comline_connection_new`. The name pointers are borrowed for that +call. Zero every field you don't set, including `_reserved`. + +| Field | Type | Meaning | +|---|---|---| +| `ir_hash` | `uint64_t` | The schema fingerprint the generated code was built from. Only the handshake reads it. | +| `wire_format`, `wire_format_len` | `const uint8_t *`, `size_t` | The wire format's name (`"packed@1"`), UTF-8, not NUL-terminated. The codecs are the host's, so any name is taken; only the handshake reads it, hashed. | +| `framing`, `framing_len` | `const uint8_t *`, `size_t` | `"datagram@2"` or `"jsonrpc@1"`. UTF-8, not NUL-terminated. | +| `max_frame` | `uint64_t` | The largest frame sent or taken, prefix not counted. 0 is the default. Above `COMLINE_MAX_FRAME_CEILING` is `COMLINE_ERR_INVALID_ARGUMENT`. | +| `capabilities` | `uint32_t` | Transport capability bits, sent in the handshake. Advisory: peers that differ still agree. | +| `max_waiting` | `uint32_t` | How many calls may wait for a reply at once. The room is reserved when the handle is made. 0 is none: the connection serves requests and sends one-way calls only. | +| `handshake` | `uint8_t` | `COMLINE_HANDSHAKE_COMLINE` or `COMLINE_HANDSHAKE_NONE`. | +| `stream` | `uint8_t` | 1: a byte stream, each frame behind a `u32` LE length prefix (TCP, a pipe, stdio). 0: one frame per message, boundaries kept by the transport (WebSocket, UDP). | +| `unordered` | `uint8_t` | 1: replies may come in any order (a host with several calls in flight). 0: calls nest, and a reply must answer the innermost waiting call. | +| `_reserved` | `uint8_t` | Must be 0. | + +`stream` and `unordered` take 0 or 1; any other value is +`COMLINE_ERR_INVALID_ARGUMENT`. The struct is 64 bytes on a 64-bit target and 48 +on wasm32, because it holds pointers. + +### `comline_span` + +`{ uint32_t offset; uint32_t len; }`: a part of the input, `offset` bytes from +its start. 8 bytes. + +### `comline_event` + +What one fed frame means. 48 bytes, aligned to 8. A field that isn't valid for +the kind and flags is 0. + +| Offset | Field | Type | Valid for | +|---|---|---|---| +| 0 | `kind` | `uint8_t` | always: `COMLINE_EVENT_*` | +| 1 | `flags` | `uint8_t` | always: see the flags below | +| 2 | `protocol` | `uint16_t` | a request with `HAS_PROTOCOL` | +| 4 | `call_id` | `uint16_t` | a request without `CALL_BY_NAME` | +| 6 | `envelope` | `uint8_t` | a reply: `COMLINE_ENVELOPE_*` | +| 7 | `_reserved` | `uint8_t` | | +| 8 | `error_id` | `uint16_t` | a reply with `COMLINE_ENVELOPE_ERR`: the schema-global ordinal | +| 10 | `_reserved2` | `uint16_t` | | +| 12 | `call_name` | `comline_span` | a request with `CALL_BY_NAME` | +| 20 | `body` | `comline_span` | a request's params, or a reply's result or error body | +| 28 | `frame` | `comline_span` | every frame: the whole frame, prefix not included | +| 36 | `capabilities` | `uint32_t` | `HANDSHAKE_OK`: the peer's capability bits | +| 40 | `request_id` | `uint64_t` | a request's id (0 for a one-way `jsonrpc@1` request), to answer it with; or the id of the call a reply answers | + +Event kinds: + +| Kind | Value | Meaning | +|---|---|---| +| `COMLINE_EVENT_HANDSHAKE_OK` | 0 | The peer's handshake arrived and agrees with ours. | +| `COMLINE_EVENT_REQUEST` | 1 | The peer calls this side. | +| `COMLINE_EVENT_REPLY` | 2 | The reply to a call this side waits on, which no longer waits. | +| `COMLINE_EVENT_STALE` | 3 | A reply to a call nobody waits on any more (it was abandoned). Drop it. | + +Flags: + +| Flag | Bit | Meaning | +|---|---|---| +| `COMLINE_EVENT_HAS_PROTOCOL` | `1 << 0` | `protocol` holds the protocol the request is for. Set on `datagram@2` requests. A `jsonrpc@1` request names only its method. | +| `COMLINE_EVENT_ONE_WAY` | `1 << 1` | The request wants no reply. Set whenever the frame says so (a `jsonrpc@1` notification), never inferred from a request id of 0. `datagram@2` can't say it, so there the schema tells one-way calls apart. | +| `COMLINE_EVENT_CALL_BY_NAME` | `1 << 2` | The request names its call by `call_name`, not `call_id`. | +| `COMLINE_EVENT_BODY_ABSENT` | `1 << 3` | The frame has no body: JSON-RPC `params`, `result` or `error.data` absent or `null`. `body` is empty, and a JSON codec reads it as `null`. | +| `COMLINE_EVENT_NAME_ESCAPED` | `1 << 4` | `call_name` is JSON string text with escapes in it, as the frame spelled it (`app.Chat\/count`). Read it with `comline_method_name_unescape`, or compare it as JSON text. | + +### `comline_handshake` + +A decoded `comline@1` frame, written by `comline_handshake_check`. 32 bytes. + +| Field | Type | Meaning | +|---|---|---| +| `ir_hash` | `uint64_t` | The peer's schema fingerprint. | +| `wire_format` | `uint64_t` | The wire format's name, hashed (`comline_name_hash`). | +| `framing` | `uint64_t` | The framing's name, hashed. | +| `capabilities` | `uint32_t` | The peer's capability bits. | +| `_reserved` | `uint32_t` | 0. | + +## Functions + +### Connection + +| Function | Purpose | Errors | +|---|---|---| +| `comline_connection_new(cfg, &out)` | Make a connection into `*out`. The one allocation the library makes: the handle, with room for `max_waiting` waiting calls. | `INVALID_ARGUMENT` (a null pointer, a name that isn't UTF-8, a field outside its values, `max_frame` above the ceiling, `*out` overlapping the config or its names), `UNKNOWN_FRAMING`. `*out` is null on any error, or left as it was if it overlaps an input. | +| `comline_connection_free(conn)` | Free it. Null is a no-op. Works on a poisoned or closed handle. | none | +| `comline_connection_start(conn, out, cap, &written)` | Write this side's handshake frame, with its prefix in stream mode. Without a handshake it writes nothing. | `BUFFER_TOO_SMALL` (`*written` is 31, or 35 in stream mode), `FRAME_TOO_LARGE` (`max_frame` below 31), `STATE` (already written). | +| `comline_connection_feed(conn, in, in_len, &consumed, &ev)` | Feed received bytes and get the event for the next frame. One frame per call. | `NEED_MORE`, `FRAMING`, `HANDSHAKE` (closes the connection), `FRAME_TOO_LARGE`. | +| `comline_connection_encode_request(conn, protocol, call_id, name, name_len, expects_reply, body, body_len, out, cap, &written, &request_id)` | Frame a request around a body the host's codec already wrote. `*request_id` is the id it was given. `expects_reply` is 1 for a call that waits, 0 for a one-way call. | `BUFFER_TOO_SMALL`, `FRAME_TOO_LARGE`, `BUSY`, `INVALID_ARGUMENT` (a name that isn't UTF-8, `expects_reply` not 0 or 1), `STATE` (handshake not complete). None of them spends a request id or leaves a call waiting. | +| `comline_connection_encode_reply(conn, protocol, request_id, envelope, error_id, body, body_len, out, cap, &written)` | Frame the reply to a request this side was given. | `BUFFER_TOO_SMALL`, `FRAME_TOO_LARGE`, `INVALID_ARGUMENT` (a `FAULT` with a body, an unknown envelope), `STATE` (handshake not complete). | +| `comline_connection_abandon(conn, request_id)` | Stop waiting on a call that timed out. A late reply to it is `STALE`. An id nobody waits on is a no-op. | none | +| `comline_connection_is_waiting(conn, &waiting)` | `*waiting` is 1 if any call waits for a reply, else 0. | none | + +### Without a handle + +These are pure and re-entrant. + +| Function | Purpose | Errors | +|---|---|---| +| `comline_abi_version()` | The ABI version: the major in the high 16 bits, the minor in the low 16. | none | +| `comline_supports(name, len)` | 1 if this build carries the named framing or handshake: `datagram@2`, `jsonrpc@1`, `comline@1`, `none`. 0 otherwise, `datagram@1` and wire formats included. | none | +| `comline_name_hash(name, len, &hash)` | The FNV-1a-64 hash of an encoding name, as the handshake carries it. | `INVALID_ARGUMENT` | +| `comline_handshake_encode(ir_hash, wire_format, wire_format_len, framing, framing_len, capabilities, out, cap, &written)` | Write a 31-byte `comline@1` frame with no prefix, for a host that runs the handshake itself. Any UTF-8 name is taken. | `BUFFER_TOO_SMALL` (`*written` is 31), `INVALID_ARGUMENT` | +| `comline_handshake_check(ours, ours_len, peer, peer_len, peer_out)` | Check a peer's frame against ours. They must agree on schema, wire format and framing; capability bits may differ. `*peer_out`, unless null, gets the peer's frame whenever it parses, so a host can say what the peer declared. Bytes after the 31-byte frame are ignored ([runtime#33](https://github.com/ComlineProject/runtime/issues/33)). | `HANDSHAKE` (they disagree, or `peer` doesn't parse), `INVALID_ARGUMENT` (`ours` doesn't parse) | +| `comline_method_name_unescape(in, len, out, cap, &written)` | Turn a `call_name` span with `NAME_ESCAPED` into UTF-8. Text with no escapes is copied as it is. | `BUFFER_TOO_SMALL` (`*written` is the exact size, nothing written), `INVALID_ARGUMENT` (not a valid JSON string's text) | + +## What each framing reads + +A connection always speaks a connection framing, whose frames say whether they +are a request or a reply. `datagram@1` is lockstep and has no connection form: +`comline_connection_new` returns `COMLINE_ERR_UNKNOWN_FRAMING` for it. + +| | `datagram@2` | `jsonrpc@1` | +|---|---|---| +| `encode_request` reads | `protocol` and `call_id`; `name` is ignored | `name`, escaped as serde_json escapes a string; `protocol` and `call_id` are ignored | +| A one-way request out | Looks like any other: the frame can't say it | A notification, with no `id` | +| An empty body is written as | nothing | `null` | +| `feed` of a request gives | `HAS_PROTOCOL`, `protocol`, `call_id`, `request_id` | `CALL_BY_NAME`, `call_name` (with `NAME_ESCAPED` if it has escapes), `request_id` | +| A one-way request in | not marked | A notification: `ONE_WAY`, `request_id` 0 | +| `encode_reply` writes | `protocol` and the request id | the id; `protocol` is ignored. An error is a JSON-RPC `error` object with `code` set to `error_id`; a fault is `-32603` | +| `BODY_ABSENT` | never | `params`, `result` or `error.data` absent or `null` | + +`name` is any UTF-8 text. It may be empty for `datagram@2`. + +## Frame limits + +`max_frame` is the connection's limit, in bytes, not counting a stream's +prefix. It may be lowered, or raised up to `COMLINE_MAX_FRAME_CEILING`. + +| Case | Result | +|---|---| +| An encode over the limit | `COMLINE_ERR_FRAME_TOO_LARGE`. Nothing written, no request id spent, the connection keeps working. | +| `feed`, stream mode, a prefix over the limit | `COMLINE_ERR_FRAME_TOO_LARGE`. The stream can't be resynchronised, so the connection is closed and later calls return `COMLINE_ERR_STATE`. | +| `feed`, message mode, a frame over the limit | `COMLINE_ERR_FRAME_TOO_LARGE`. The frame is refused and the connection keeps working. | + +`encode_reply` returns `FRAME_TOO_LARGE` for a reply over the limit and leaves +the connection usable. The Rust drivers answer such a reply with a fault; the +ABI leaves that to the host. + +## Rules every function keeps + +- **One crossing per frame.** `feed`, `encode_request` and `encode_reply` each + handle one whole frame. Nothing calls back into the host. +- **Buffers are the caller's.** Every output goes to an `out`, `cap`, + `*written` triple. Every input is borrowed for the call only. The library + never frees host memory and the host never frees library memory. +- **Spans are offsets, not pointers.** They count from the start of the input + passed to the `feed` that produced them, so they stay valid if the host + moves its buffer afterwards. They fit a `u32` under the frame ceiling. + Nothing is copied out of the input. +- **No allocation after `comline_connection_new`.** The handle is the one + thing the library allocates. The exemption is `jsonrpc@1` decoding, which + allocates through serde_json when it refuses a frame, when a member name has + an escape, and when a value nests brackets three deep (`"params":[[1]]`). It + is [runtime#58](https://github.com/ComlineProject/runtime/issues/58), deferred. + Every `datagram@2` feed and every encode, under either framing, allocates + nothing. +- **Null pointers.** A data pointer may be null only when its length or + capacity is 0. A length or capacity over `isize::MAX` is + `COMLINE_ERR_INVALID_ARGUMENT`. +- **Inputs and outputs must not overlap.** No buffer a call writes (`out`, + `*written`, `*ev`, the handle) may share a byte with any buffer or + out-pointer the same call takes. A call given overlapping ones returns + `COMLINE_ERR_INVALID_ARGUMENT` and touches nothing. Inputs may overlap each + other. +- **Panics never cross.** Every function runs inside `catch_unwind`. A panic is + a bug. The call returns `COMLINE_ERR_POISONED`, the handle is poisoned and + every later call on it returns the same, so the host drops the connection. + `comline_connection_free` still works. A free function returns + `COMLINE_ERR_POISONED` for the call that panicked. +- **Threads.** A handle may move between threads, but calls on one handle must + not overlap: the host locks it. Different handles are independent. +- **Refusals change nothing.** `BUFFER_TOO_SMALL`, `BUSY` and `FRAME_TOO_LARGE` + on an encode spend no request id and leave no call waiting. After + `BUFFER_TOO_SMALL`, retry with exactly `*written` bytes of room. +- **A closed connection** (a refused handshake, or an oversized stream prefix) + answers every call but `comline_connection_free` with `COMLINE_ERR_STATE`. + +The layouts of `comline_span`, `comline_event`, `comline_handshake` and +`comline_config` are asserted at compile time in `src/lib.rs`. + +## wasm32 and the JS glue + +The crate builds for `wasm32-unknown-unknown`, and the same `extern "C"` +functions are the module's exports. There is no `wasm-bindgen`: the `.wasm` is +the C ABI, so a non-JS WebAssembly host (wazero, wasmtime) calls the same +functions with the same meaning. + +```sh +cargo build -p comline-runtime-ffi --release --target wasm32-unknown-unknown +``` + +- **Layouts.** A pointer and `size_t` are 4 bytes, so `comline_config` is 48 + bytes. `comline_event` and `comline_handshake` don't change. +- **Two extra exports,** not in the header, on wasm32 without the `atomics` + feature: + - `comline_wasm_reserve_in(len)` and `comline_wasm_reserve_out(len)` return a + pointer to a region of at least `len` bytes that the module owns, or null if + memory can't grow. + - A host copies input into the `in` region and reads output from the `out` + region. + - Reserving more may move a region and grow memory, which detaches the host's + views. Take the pointer again after each reserve. +- **A panic is a trap.** The host discards the instance. + +[`js/comline.mjs`](https://github.com/ComlineProject/runtime/blob/main/runtime/ffi/js/comline.mjs) +and [`comline.d.mts`](https://github.com/ComlineProject/runtime/blob/main/runtime/ffi/js/comline.d.mts) +are a hand-written ES module over that build, with no dependencies. + +- **Thin.** Every method is one ABI call. It returns the status with whatever + the call wrote, retries nothing, never throws because of a status and keeps no + state of its own. A binding builds on it. +- **`instantiate(source)`** compiles the module, checks that its ABI version is + exactly the glue's, and feeds one `datagram@2` request through to check the + `comline_event` layout. +- **Memory.** Each call reserves its regions, copies inputs in and copies + outputs back, so a byte output is a fresh copy and nothing returned points + into wasm memory. A span is an offset into the array the host fed, so the + host reads a body from its own array. +- **Types.** A `u64` is a `BigInt` both ways. A name is a string, encoded as + UTF-8, or a `Uint8Array` of the bytes as they are. +- **A panic** is thrown as a `WebAssembly.RuntimeError`. The glue doesn't catch + it. + +```js +import { instantiate, Handshake } from "./comline.mjs"; + +const comline = await instantiate(await readFile("comline_runtime_ffi.wasm")); +const { status, connection } = comline.connectionNew({ + framing: "datagram@2", + handshake: Handshake.NONE, + maxWaiting: 16, +}); +``` + +## Building natively + +```sh +cargo build -p comline-runtime-ffi --release +``` + +This writes `libcomline_runtime_ffi.so` (a `cdylib`) and `libcomline_runtime_ffi.a` +(a `staticlib`) under `target/release/`. Linking the static library on Linux +needs what a Rust `staticlib` needs from the system: `-lgcc_s -lutil -lrt +-lpthread -lm -ldl -lc`. + +To regenerate the header, from `runtime/ffi`: + +```sh +cargo install cbindgen --version 0.29.2 --locked +cbindgen --config cbindgen.toml --output include/comline.h +``` + +## How it is verified + +- **The conformance vectors, from three hosts, on the same cases.** Rust + (`tests/vectors.rs`, through the `extern "C"` functions only), C + (`tests/c/vectors.c`, C11 against `comline.h`) and Node (`js/test/vectors.test.mjs`, + against the wasm32 build). C runs against the static library, the dynamic + library, and the static library under ASan and UBSan. + - The cases are every one in `name-hash` (8), `handshake` (9), `envelope` + (8), `framing-datagram` (9), `framing-datagram2` (12) and `framing-jsonrpc` + (19): 65 in all. Each harness fails on any mismatch, and on a case count + that differs from the vectors'. + - The nine `datagram@1` cases have no connection form. For them the harness + checks that the ABI refuses `datagram@1`. + - C and Node have no codec, so `examples/export_vectors.rs` writes the cases + out with their params already encoded. +- **The ABI against the core, on random sequences.** `tests/differential.rs` + runs random configs and sequences (calls, replies, abandons, peer frames, + chunked deliveries, flipped bytes) through a `comline_connection` and through + `State` and the framings directly. They must give the same frames, events and + refusals. Every call is also held to the rules above. The nightly run uses + 100,000 cases. +- **Fuzz.** `ffi_feed` checks the rules alone; `ffi_vs_core` also compares with + the core. Both share their driver with the proptest. +- **Miri.** The ABI's tests run under Miri, which checks aliasing, bounds and + leaks at the `extern "C"` boundary. +- **Header drift.** CI regenerates `comline.h` with the pinned cbindgen and + fails on any difference. +- **Benchmarks.** Instruction counts of each call, below. + +The CI jobs are `ffi`, `ffi miri`, `wasm32 core` and `fuzz (ffi_feed)`, +`fuzz (ffi_vs_core)` and `fuzz (stream_split)`; all are required to merge. See +[Verification](../proposals/shared-runtime-core/verification.md) for how the +fuzz targets and the benchmark gate work, and [Branch rules](../conduct/branch-rules.md). + +## What crossing costs + +The ABI's benches count instructions (Callgrind, through `iai-callgrind`) for +each call through its `extern "C"` function (`ffi/abi/...`) and for the same +work on `State` and the framings directly (`ffi/core/...`). The difference is +what crossing the boundary costs. + +These numbers are from the latest published `rust.bench.json` on the runtime's +[`reports`](https://github.com/ComlineProject/runtime/tree/reports) branch, which +has the `ffi/*` ids: main at `cb41d31`, the merge of +[runtime#60](https://github.com/ComlineProject/runtime/pull/60), built with +rustc 1.98.0. Message mode, a 16-byte payload: + +| Call | Core | ABI | Crossing | +|---|---:|---:|---:| +| `feed`, `datagram@2` | 141 | 294 | +153 | +| `feed`, `jsonrpc@1` | 1,550 | 1,703 | +153 | +| `encode_request`, `datagram@2` | 336 | 724 | +388 | +| `encode_request`, `jsonrpc@1` | 518 | 906 | +388 | +| `encode_reply`, `datagram@2` | 320 | 462 | +142 | +| `encode_reply`, `jsonrpc@1` | 331 | 473 | +142 | +| `name_hash` | 49 | 198 | +149 | +| `handshake_encode` | 437 | 772 | +335 | +| `handshake_check` | 100 | 138 | +38 | +| `handshake_check` with `peer_out` | 130 | 210 | +80 | + +- **The crossing is a constant per call, not per byte.** `feed` costs the same + extra at 16 B, 256 B, 4 KiB and 64 KiB. The two encode calls cost 48 more at + 64 KiB (436 and 190). +- **Stream mode** adds a few instructions to the core side and changes the + crossing little: `feed` +149, `encode_request` +388, `encode_reply` +141 at + 16 B. +- **`jsonrpc@1` decoding is its own cost.** It scans the frame with serde_json + at about 2.8 instructions a byte (185,825 for 64 KiB, on the core). The + crossing adds the same 153 on top. +- **runtime#60 cut it.** The overlap check was most of `encode_request`'s + crossing. From the PR's body, measured locally at the same paths: + `encode_request` `datagram@2` 965 to 742, `feed` `datagram@2` 319–327 to + 294–299, `handshake_check` 152 to 138. No call got dearer. Counts move by a + few instructions between machines, because the overlap check compares + addresses, so the CI figures above differ slightly from the PR's. +- **Not measured:** the wasm32 build and the JS glue. A JavaScript call into + WebAssembly and the copies in and out are the cost [stage 4](../proposals/shared-runtime-core/index.md#stages) + has to measure before moving TypeScript. + +The gate compares each call with its base on every pull request and fails at +a 1% rise, like the core's benches. Both sides of each pair are gated. + +## Not here yet + +- **`comline_connection_size` and `comline_connection_init`,** for a host that + supplies the handle's memory. After 1.0. +- **`comline@2`'s negotiation** hands protocol numbers out at the handshake and + may change the event struct. If it does, it lands before 1.0 freezes the ABI. +- **Bindings.** No language binds the ABI yet; the JS glue is the only host + code, and it is a thin wrapper, not a binding. diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 3302169..ffcc99f 100755 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -62,6 +62,9 @@ nav: - Wire formats: - packed@1: 'reference/wire/packed.md' + - Runtime: + - C ABI: 'reference/c-abi.md' + - Congregation: - Congregation: 'reference/congregation/index.md' - comline.toml: 'reference/congregation/comline-toml.md' From 10801283db1e1ebf9476c5951273b45a45858dd6 Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 26 Oct 2026 23:24:12 +0800 Subject: [PATCH 2/4] docs: the Rust page counts seven fuzz targets --- docs/docs/reference/languages/rust.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/docs/reference/languages/rust.md b/docs/docs/reference/languages/rust.md index 98aebf8..b358862 100644 --- a/docs/docs/reference/languages/rust.md +++ b/docs/docs/reference/languages/rust.md @@ -78,9 +78,10 @@ are compared against (see [Verification](../../proposals/shared-runtime-core/ver ### Fuzzing and properties -- Four [`cargo-fuzz`](https://github.com/ComlineProject/runtime/tree/main/runtime/fuzz) - targets: `decode_no_panic`, `packed_canonical`, `framing_decode` and - `seed_vs_derive`. +- Seven [`cargo-fuzz`](https://github.com/ComlineProject/runtime/tree/main/runtime/fuzz) + targets: `decode_no_panic`, `packed_canonical`, `framing_decode`, + `seed_vs_derive` and `stream_split`, and, for the + [C ABI](../c-abi.md), `ffi_feed` and `ffi_vs_core`. - [Property tests](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/tests/properties.rs) (`proptest`) over random types and values, run with `cargo test`. - CI runs each target for a minute on every pull request, and for ten minutes From 16c22fa3ee0ae383188f13aafe1a1672f40f6ebf Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 26 Oct 2026 23:57:25 +0800 Subject: [PATCH 3/4] docs: render the stream feature in the language support tables --- docs/docs/reference/languages/index.md | 1 + docs/docs/reference/languages/rust.md | 1 + docs/docs/reference/languages/typescript.md | 1 + 3 files changed, 3 insertions(+) diff --git a/docs/docs/reference/languages/index.md b/docs/docs/reference/languages/index.md index c1634f2..d6b7c27 100644 --- a/docs/docs/reference/languages/index.md +++ b/docs/docs/reference/languages/index.md @@ -108,6 +108,7 @@ commit. A runtime's suite runs every example against it: | envelope
`envelope` | ✅ supported | ✅ supported | | datagram framing
`framing-datagram` | ✅ supported | ✅ supported | | datagram@2 framing
`framing-datagram2` | ✅ supported | ✅ supported | +| stream framing
`stream` | ✅ supported | ❌ unsupported: no length-prefixed framing or frame limit yet: the transports are message-oriented ([comline-typescript#80](https://github.com/ComlineProject/comline-typescript/issues/80)) | | JSON-RPC framing
`framing-jsonrpc` | ✅ supported | ✅ supported | | JSON wire format
`format-json` | ✅ supported | 🟡 partial: decoding doesn't check enum variants or integer ranges ([comline-typescript#57](https://github.com/ComlineProject/comline-typescript/issues/57)) | | MessagePack wire format
`format-msgpack` | ✅ supported | ❌ unsupported: JSON and packed@1 ship | diff --git a/docs/docs/reference/languages/rust.md b/docs/docs/reference/languages/rust.md index b358862..05cb080 100644 --- a/docs/docs/reference/languages/rust.md +++ b/docs/docs/reference/languages/rust.md @@ -63,6 +63,7 @@ Runtime `rust`. | envelope
`envelope` | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) | | datagram framing
`framing-datagram` | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) | | datagram@2 framing
`framing-datagram2` | the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection | ✅ supported | [12 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram2.json) | +| stream framing
`stream` | frames on a byte stream (TCP, a pipe): each behind a u32 little-endian length prefix, refused over the limit, by default 16 MiB (16777216 bytes) in every runtime | ✅ supported | [24 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/stream.json) | | JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [19 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | | JSON wire format
`format-json` | values as UTF-8 JSON: a struct as an object | ✅ supported | [46 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) | | MessagePack wire format
`format-msgpack` | values as MessagePack: a struct as an array | ✅ supported | [45 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-msgpack.json) | diff --git a/docs/docs/reference/languages/typescript.md b/docs/docs/reference/languages/typescript.md index ee46c15..36fe672 100644 --- a/docs/docs/reference/languages/typescript.md +++ b/docs/docs/reference/languages/typescript.md @@ -63,6 +63,7 @@ Runtime `typescript`. | envelope
`envelope` | a response's outcome: ok with a payload, an error ordinal with its body, or a provider fault | ✅ supported | [8 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/envelope.json) | | datagram framing
`framing-datagram` | Comline's own binary framing of requests and responses | ✅ supported | [9 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram.json) | | datagram@2 framing
`framing-datagram2` | the datagram framing with a request/reply kind and a protocol number, for calls in both directions and several protocols on one connection | ✅ supported | [12 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-datagram2.json) | +| stream framing
`stream` | frames on a byte stream (TCP, a pipe): each behind a u32 little-endian length prefix, refused over the limit, by default 16 MiB (16777216 bytes) in every runtime | ❌ unsupported: no length-prefixed framing or frame limit yet: the transports are message-oriented ([comline-typescript#80](https://github.com/ComlineProject/comline-typescript/issues/80)) | — | | JSON-RPC framing
`framing-jsonrpc` | JSON-RPC 2.0 requests and responses, a fault as -32603 | ✅ supported | [19 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/framing-jsonrpc.json) | | JSON wire format
`format-json` | values as UTF-8 JSON: a struct as an object | 🟡 partial: decoding doesn't check enum variants or integer ranges ([comline-typescript#57](https://github.com/ComlineProject/comline-typescript/issues/57))
fails: `format-json/decode-unknown-variant`, `format-json/decode-wider-integer-overflow` | [44 cases](https://github.com/ComlineProject/runtime/blob/main/runtime/conformance/vectors/format-json.json) | | MessagePack wire format
`format-msgpack` | values as MessagePack: a struct as an array | ❌ unsupported: JSON and packed@1 ship | — | From 10f8e323cc361761236ece44390668ccabba38d3 Mon Sep 17 00:00:00 2001 From: Kinflou Date: Mon, 26 Oct 2026 23:58:21 +0800 Subject: [PATCH 4/4] docs: call jsonrpc@1's allocation a known bug, as comline.h does --- docs/docs/reference/c-abi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/docs/reference/c-abi.md b/docs/docs/reference/c-abi.md index 39bb8c7..aefa533 100644 --- a/docs/docs/reference/c-abi.md +++ b/docs/docs/reference/c-abi.md @@ -243,7 +243,7 @@ ABI leaves that to the host. moves its buffer afterwards. They fit a `u32` under the frame ceiling. Nothing is copied out of the input. - **No allocation after `comline_connection_new`.** The handle is the one - thing the library allocates. The exemption is `jsonrpc@1` decoding, which + thing the library allocates. One known exception, a bug: `jsonrpc@1` decoding, which allocates through serde_json when it refuses a frame, when a member name has an escape, and when a value nests brackets three deep (`"params":[[1]]`). It is [runtime#58](https://github.com/ComlineProject/runtime/issues/58), deferred.